CDH 5.3.3 Release Notes
The following lists all Flume NG Jiras included in CDH 5.3.3
that are not included in the Flume NG base version 1.5.0. The
flume-ng-1.5.0-cdh5.3.3.CHANGES.txt
file lists all changes included in CDH 5.3.3. The patch for each
change can be found in the cloudera/patches directory in the release tarball.
Changes Not In Flume NG 1.5.0
Flume
Bug
- [FLUME-2586] - HDFS Sink should have an option to try rename even if close fails
- [FLUME-2568] - Additional fix for TestReliableSpoolingFileEventReader
- [FLUME-2556] - TestBucketPath.testDateFormatHours fails intermittently
- [FLUME-2525] - flume should handle a zero byte .flumespool-main.meta file for the spooldir source
- [FLUME-2488] - TestElasticSearchRestClient fails on Oracle JDK 8
- [FLUME-2557] - DatasetSink doesn't work with Kerberos
- [FLUME-2538] - TestResettableFileInputStream fails on JDK 8
- [FLUME-2560] - Kafka channel entries missing for output jar generation
- [FLUME-2548] - Enable SSLv2Hello for Avro Source and NettyAvroRpcClient
- [FLUME-2500] - Add a channel that uses Kafka
- [FLUME-2520] - HTTP Source should be able to block a prefixed set of protocols.
- [FLUME-2517] - Performance issue: SimpleDateFormat constructor takes 30% of HDFSEventSink.process()
- [FLUME-2486] - TestExecSource fails on some environments
- [FLUME-2482] - Race condition in File Channels' Log.removeOldLogs
- [FLUME-2126] - Problem in elasticsearch sink when the event body is a complex field
- [FLUME-2475] - toLowerCase/toUpperCase used without Locale parameter to manipulate Enum
- [FLUME-2472] - SyslogUtils fails when system locale is not English
- [FLUME-2346] - idLogFileMap in Log can lose track of file ids
- [FLUME-2466] - Embedded agent name cannot contain spaces
- [FLUME-2481] - TestFileChannelEncryption fails with System locale other than en_US
- [FLUME-2496] - Disable the Parquet format in DatasetSink
- [FLUME-2495] - Kafka Source may miss events when channel is not available
- [FLUME-2479] - Kafka property auto.commit.enable is incorrect for KafkaSource
- [FLUME-2492] - Flume's Kafka Source doesn't account time correctly
- [FLUME-2470] - Kafka Sink and Source must use camel case for all configs.
- [FLUME-2337] - export JAVA_HOME in flume-env.sh.template and increase heap size
- [FLUME-2420] - HDFSEventSink#stop might throw ConcurrentModificationException
- [FLUME-2450] - Improve replay index insertion speed.
- [FLUME-2425] - FileChannel should trim data and checkpoint directories
- [FLUME-2438] - Make Syslog source message body configurable
- [FLUME-2441] - Unit test TestHTTPSource.java failed with IBM JDK 1.7
- [FLUME-1793] - Unit test TestElasticSearchLogStashEventSerializer fails with IBM JDK
- [FLUME-2404] - Default maxReadBufferBytes might cause OOM and cause scribe source exit
- [FLUME-2024] - Add ExecSource flush timeout to Flume User Guide
- [FLUME-2432] - Update Kite dependency to 0.15.0
- [FLUME-2416] - Use CodecPool in compressed stream to prevent leak of direct buffers
- [FLUME-2245] - HDFS files with errors unable to close
- [FLUME-2400] - Dataset Sink is not built in hbase-98 profile
- [FLUME-2397] - HBase-98 compatibility
Improvement
- [FLUME-2502] - Spool source's directory listing is inefficient
- [FLUME-2499] - Include Kafka Message Key in Event Header, Updated Comments
- [FLUME-1710] - JSONEvent.getBody should not return null
- [FLUME-2237] - Example config in User Guide to use same notation for Agent name
- [FLUME-2469] - DatasetSink should load dataset when needed, not at startup
- [FLUME-2511] - Allow configuration of enabled protocols in Avro source and Rpc client
- [FLUME-2463] - Add support for Hive and HBase datasets to DatasetSink
- [FLUME-2462] - Remove use of deprecated methods in DatasetSink
- [FLUME-1491] - Dynamic configuration from Zookeeper watcher
- [FLUME-2439] - Update DatasetSink for Kite 0.15.0
- [FLUME-1920] - Test case TestFileChannel fails when flume is built from paths containing the string "hadoop"
- [FLUME-2401] - Optionally Compress Backup Checkpoint
- [FLUME-2273] - ElasticSearchSink: Add handling for header substitution in indexName
New Feature
- [FLUME-2250] - Add support for Kafka Source
- [FLUME-2251] - Add support for Kafka Sink
- [FLUME-2426] - Support interceptors in the Embedded Agent
- [FLUME-2431] - Add simple regex search-and-replace interceptor
Task
Test
- [FLUME-1501] - Flume Scribe Source needs unit tests.