CDH 5.14.2 Release Notes
The following lists all Flume NG Jiras included in CDH 5.14.2
that are not included in the Flume NG base version 1.6.0. The
flume-ng-1.6.0-cdh5.14.2.CHANGES.txt
file lists all changes included in CDH 5.14.2. The patch for each
change can be found in the cloudera/patches directory in the release tarball.
Changes Not In Flume NG 1.6.0
Flume
Bug
- [FLUME-3201] - Fix SyslogUtil to handle RFC3164 format in december correctly
- [FLUME-3046] - Kafka Sink and Source Configuration Improvements
- [FLUME-3186] - Make asyncHbaseClient configuration parameters available from flume config
- [FLUME-3152] - Add Flume Metric for Backup Checkpoint Errors
- [FLUME-2752] - Flume AvroSource will leak the memory and the OOM will be happened.
- [FLUME-2905] - NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks
- [FLUME-3085] - HDFS Sink can skip flushing some BucketWriters, might lead to data loss
- [FLUME-3080] - Close failure in HDFS Sink might cause data loss
- [FLUME-3049] - Wrapping the exception into SecurityException in UGIExecutor.execute hides the original one
- [FLUME-2649] - Elasticsearch sink doesn't handle JSON fields correctly
- [FLUME-2857] - Kafka Source/Channel/Sink does not restore default values when live update config
- [FLUME-2966] - NULL text in a TextMessage from a JMS source in Flume can lead to NPE
- [FLUME-2974] - Some tests are broken in TestReliableSpoolingFileEventReader and TestSpoolingFileLineReader
- [FLUME-2318] - SpoolingDirectory is unable to handle empty files
- [FLUME-2619] - Spooldir source does not log channel exceptions
- [FLUME-2761] - Move Hive sink out of preview mode
- [FLUME-2514] - Some TestFileChannelRestart tests are extremely slow
- [FLUME-2952] - SyslogAgent possible NPE on stop()
- [FLUME-2725] - HDFS Sink does not use configured timezone for rounding
- [FLUME-2950] - ReliableSpoolingFileEventReader.rollCurrentFile is broken
- [FLUME-2936] - KafkaSource tests arbitrarily fail
- [FLUME-2949] - Flume fails to build on Windows
- [FLUME-2939] - Upgrade recursive SpoolDir to use Java7 features
- [FLUME-2567] - Remove unneeded repository declarations in pom.xml
- [FLUME-2881] - Windows Launch Script fails in plugins dir code
- [FLUME-2806] - flume-ng.ps1 Error running script to start an agent on Windows
- [FLUME-2835] - Hive Sink tests need to create table with transactional property set
- [FLUME-2593] - ResettableFileInputStream returns negate values from read() method
- [FLUME-2797] - SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource
- [FLUME-2798] - Malformed Syslog messages can lead to OutOfMemoryException
- [FLUME-2812] - Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded
- [FLUME-2889] - Fixes to DateTime computations
- [FLUME-3031] - sequence source should reset its counter for event body on channel exception
- [FLUME-3027] - Kafka Channel should clear offsets map after commit
- [FLUME-2844] - ChannelCounter of SpillableMemoryChannel doesn't register actually.
- [FLUME-2983] - Handle offset migration in the new Kafka Source
- [FLUME-2901] - Document Kerberos setup for Kafka channel
- [FLUME-2948] - Docs: Fixed parameters on Replicating Channel Selector documentation example
- [FLUME-2652] - Documented transaction handling semantics incorrect
- [FLUME-2972] - Handle offset migration in the new Kafka Channel
- [FLUME-2922] - HDFSSequenceFile Should Sync Writer
- [FLUME-2923] - Bump AsyncHBase version
- [FLUME-2913] - Flume classpath too long
- [FLUME-2920] - Kafka Channel Should Not Commit Offsets When Stopping
- [FLUME-2915] - The kafka channel using new APIs will be stuck when the sink is avro sink
- [FLUME-2908] - NetcatSource - SocketChannel not closed when session is broken
- [FLUME-2897] - AsyncHBase sink NPE when Channel.getTransaction() fails
- [FLUME-2886] - Optional Channels can cause OOMs
- [FLUME-2632] - High CPU on KafkaSink
- [FLUME-2712] - Optional channel errors slows down the Source to Main channel event rate
- [FLUME-2841] - Upgrade commons-collections to 3.2.2
- [FLUME-2734] - Kafka Channel timeout property is overridden by default value
- [FLUME-2751] - Upgrade Derby version to 10.11.1.1
- [FLUME-2804] - Hive sink - abort remaining transactions on shutdown
- [FLUME-2773] - TailDirSource throws FileNotFound Exception if ~/.flume directory is not created already
- [FLUME-2672] - NPE in KafkaSourceCounter
- [FLUME-2754] - Hive Sink skipping first transaction in each Batch of Hive Transactions
- [FLUME-2763] - flume_env script should handle jvm parameters like -javaagent -agentpath -agentlib
- [FLUME-2753] - Error when specifying empty replace string in Search and Replace Interceptor
- [FLUME-2485] - Thrift Source tests fail on Oracle JDK 8
- [FLUME-2660] - Add documentation for EventValidator
- [FLUME-2749] - Kerberos configuration error when using short names in multiple HDFS Sinks
- [FLUME-2738] - Async HBase sink FD leak on client shutdown
- [FLUME-2732] - Make maximum tolerated failures before shutting down and recreating client in AsyncHbaseSink configurable
- [FLUME-2548] - Enable SSLv2Hello for Avro Source and NettyAvroRpcClient
- [FLUME-2215] - ResettableFileInputStream can't support ucs-4 character
Documentation
- [FLUME-2971] - Document secure Kafka Sink/Source/Channel setup
- [FLUME-2998] - Add missing configuration parameter to SequenceSource docs
- [FLUME-2890] - Typo in Twitter source warning
- [FLUME-2975] - Minor mistake in NetCat Source example in documentation
- [FLUME-2963] - FlumeUserGuide - error in Kafka Source properties table
- [FLUME-2934] - Document new cachePatternMatching option for TaildirSource
- [FLUME-2713] - Document Fault Tolerant Config parameters in FlumeUserGuide
Improvement
- [FLUME-3158] - Upgrade surefire version and config
- [FLUME-3100] - Support arbitrary header substitution for topic of Kafka
- [FLUME-199] - Unit tests should hunt for available ports if defaults are in use
- [FLUME-3144] - Improve Log4jAppender's performance by allowing logging collection of messages
- [FLUME-3092] - Extend the FileChannel's monitoring metrics
- [FLUME-2911] - Add includePattern option in SpoolDirectorySource configuration
- [FLUME-2959] - Fix issues with flume-checkstyle module
- [FLUME-2955] - Add file path to the header in TaildirSource
- [FLUME-2810] - Add static Schema URL to AvroEventSerializer configuration
- [FLUME-2941] - Integrate checkstyle for test classes
- [FLUME-2937] - Integrate checkstyle for non-test classes
- [FLUME-2875] - Allow RollingFileSink to specify a file prefix and a file extension.
- [FLUME-2704] - Configurable poll delay for spooling directory source
- [FLUME-2718] - HTTP Source to support generic Stream Handler
- [FLUME-2801] - Performance improvement on TailDir source
- [FLUME-2999] - Kafka channel and sink should enable statically assigned partition per event via header
- [FLUME-3025] - Expose FileChannel.open on JMX
- [FLUME-3020] - Improve HDFSEventSink Escape Ingestion by more then 10x by not getting InetAddress on every record
- [FLUME-2982] - Add localhost escape sequence to HDFS sink
- [FLUME-2954] - make raw data appearing in log messages explicit
- [FLUME-1899] - Make SpoolDir work with Sub-Directories
- [FLUME-2918] - TaildirSource is underperforming with huge parent directories
- [FLUME-2910] - AsyncHBaseSink - Failure callbacks should log the exception that caused them
- [FLUME-2909] - Bump Rat version
- [FLUME-2852] - Kafka Source/Sink should optionally read/write Flume records
- [FLUME-2823] - Flume-Kafka-Channel with new APIs
- [FLUME-2822] - Flume-Kafka-Sink with new Producer
- [FLUME-2821] - Flume-Kafka Source with new Consumer
- [FLUME-2781] - A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
- [FLUME-2628] - Add an optional parameter to specify the expected input text encoding for the netcat sourcef the netcat source
- [FLUME-2729] - Allow pollableSource backoff times to be configurable
New Feature
Question
Task
- [FLUME-3093] - Groundwork for version changes in root pom
- [FLUME-2935] - Bump java target version to 1.7
Test
- [FLUME-3195] - Split up the TestKafkaChannel class
- [FLUME-3003] - testSourceCounter in TestSyslogUdpSource is flaky
- [FLUME-2997] - Fix flaky junit test in SpillableMemoryChannel
- [FLUME-3002] - Some tests in TestBucketWriter are flaky
Wish
- [FLUME-2579] - JMS source support durable subscriptions and message listening