CDH 5 Release Notes

The following lists all Flume NG Jiras included in CDH 5 that are not included in the Flume NG base version 1.4.0. The flume-ng-1.4.0-cdh5.0.5.CHANGES.txt file lists all changes included in CDH 5. The patch for each change can be found in the cloudera/patches directory in the release tarball.

Changes Not In Flume NG 1.4.0

Flume

Bug

[FLUME-2548] - Enable SSLv2Hello for Avro Source and NettyAvroRpcClient
[FLUME-2198] - Avro Source should disable itself if ipFilterRules contains invalid rules
[FLUME-2438] - Make Syslog source message body configurable
[FLUME-2416] - Use CodecPool in compressed stream to prevent leak of direct buffers
[FLUME-2245] - HDFS files with errors unable to close
[FLUME-2357] - HDFS sink should retry closing files that previously had close errors
[FLUME-2338] - Support coalescing increments in HBaseSink
[FLUME-2336] - HBase tests that pass in ZK configs must use a new context object
[FLUME-2335] - TestHBaseSink#testWithoutConfigurationObject() must delete the table at the end of the test
[FLUME-2324] - Support writing to multiple HBase clusters using HBaseSink
[FLUME-2329] - Add an alias for the Morphline Solr Sink
[FLUME-2283] - Spool Dir source must check interrupt flag before writing to channel
[FLUME-2323] - Morphline sink must increment eventDrainAttemptCount when it takes event from channel
[FLUME-2328] - FileChannel Dual Checkpoint Backup Thread not released on Application stop
[FLUME-2334] - Upgrade Asynchbase to 1.5.0 as this include hbase 0.96.x support
[FLUME-2320] - Deadlock in DatasetSink
[FLUME-2325] - BucketWriter might throw BucketClosedException incorrectly
[FLUME-2307] - Remove Log writetimeout
[FLUME-2314] - Upgrade to Mapdb 0.9.9
[FLUME-2311] - Use standard way of finding queue/topic
[FLUME-2312] - Add utility for adorning HTTP contexts in Jetty
[FLUME-2305] - BucketWriter#close must cancel idleFuture
[FLUME-1892] - IRC Sink NPE
[FLUME-2304] - DatasetSink test fails unexpectedly
[FLUME-2302] - TestHDFS Sink fails with Can't get Kerberos realm
[FLUME-2303] - HBaseSink tests can fail based on order of execution
[FLUME-2301] - Update HBaseSink tests to reflect sink returning backoff only on empty batches
[FLUME-2259] - transaction closure not happening for all the scenario in hbasesink
[FLUME-2289] - Disable maxUnderReplication test which is extremely flakey
[FLUME-2265] - Closed bucket writers should be removed from sfwriters map
[FLUME-2270] - Twitter Source Documentation Does not load properly
[FLUME-2239] - Clarify File Channel's dataDirs setting in User Guide
[FLUME-2264] - Log4j Appender + Avro Reflection on string results in an invalid avro schema
[FLUME-1679] - Add dependency on Guava to flume-ng-elasticsearch-sink POM
[FLUME-2266] - Update Morphline Sink to kite-0.10.0
[FLUME-2209] - AsyncHBaseSink will never recover if the column family does not exists for the first start
[FLUME-2238] - Provide option to configure worker threads in NettyAvroRpcClient
[FLUME-2262] - Log4j Appender should use timeStamp field not getTimestamp
[FLUME-2255] - Spooling Directory Source cannot handle channel exceptions
[FLUME-2182] - Spooling Directory Source will not ingest data completely when a wide character appears at the edge of a buffer
[FLUME-2235] - idleFuture should be cancelled at the start of append
[FLUME-2233] - MemoryChannel lock contention on every put due to bytesRemaining Semaphore
[FLUME-2229] - Backoff period gets reset too often in OrderSelector
[FLUME-2192] - AbstractSinkProcessor stop incorrectly calls start
[FLUME-2210] - UnresolvedAddressException when using multiple hostNames in Elasticsearch sink configuration
[FLUME-1666] - Syslog source strips timestamp and hostname from log message body
[FLUME-2159] - Sporadic failures in TestNettyAvroRpcClient.spinThreadsCrazily()
[FLUME-2208] - Jetty's default SocketSelector leaks File descriptors
[FLUME-2200] - HTTP Source should be able to use "port" parameter if SSL is enabled
[FLUME-2172] - Update protocol buffer from 2.4.1 to 2.5.0
[FLUME-2184] - flume-ng-morphline-solr-sink Build failing due to incorrect hadoop-common dependency declaration
[FLUME-2174] - Integration of morphline solr sink puts Lucene/Solr dependencies on the default classpath
[FLUME-2161] - Flume does not support spaces in -X java-opt command line args
[FLUME-2156] - Unregister then re-register MonitoredCounterGroup JMX MBeans on reconfigure
[FLUME-2157] - Spool directory source does not shut down correctly when Flume is reconfigured
[FLUME-2134] - AsyncHbase Sink bugfix plus tests errors on Windows
[FLUME-2142] - HTTPS tests for http source
[FLUME-2109] - HTTPS support in HTTP Source
[FLUME-2110] - Scribe Source must check if category is null before inserting the headers
[FLUME-2088] - Minor typo in Flume User Guide JSON Reporting section

Documentation

[FLUME-2319] - Incorrect property name of Ganglia Reporting documentation
[FLUME-2278] - Incorrect documentation for write-timeout of File Channel
[FLUME-2065] - Regex Extractor Interceptor config agent name inconsistent with rest of docs
[FLUME-1851] - User Guide grammar mistake
[FLUME-2064] - Typo/Grammar in flume main user doc under Scribe

Improvement

[FLUME-2511] - Allow configuration of enabled protocols in Avro source and Rpc client
[FLUME-2243] - AvroSource to use TransceiverThreadFactory for Thread naming while initializing NioServerSocketChannelFactory
[FLUME-2130] - Handle larger payloads via SyslogUDPSource
[FLUME-2292] - Upgrade mapdb to 0.9.8
[FLUME-2275] - Improve scalability of MorphlineInterceptor under contention
[FLUME-2267] - Increase default transactionCapacity for FileChannel to 10000 from 1000
[FLUME-2155] - Improve replay time
[FLUME-2217] - Preserve priority, timestamp and hostname fields in MultiportSyslogTcp and Udp sources
[FLUME-2202] - AsyncHBaseSink should coalesce increments to reduce RPC roundtrips
[FLUME-2052] - Spooling directory source should be able to replace or ignore malformed characters
[FLUME-2231] - Add details in Flume Ganglia config in User Guide
[FLUME-2206] - ElasticSearchSink ttl field modification to mimic Elasticsearch way of specifying TTL
[FLUME-2213] - MorphlineInterceptor should share metric registry across threads for better (aggregate) reporting
[FLUME-2190] - add a source capable of feeding off of the Twitter Streaming API
[FLUME-2154] - Reducing duplicate events caused by reset-connection-interval
[FLUME-2139] - upgrade morphline library dependency to a 0.5.0 release

New Feature

[FLUME-2189] - Add support for IP filtering on AvroSource
[FLUME-2294] - Add a sink for Kite Datasets
[FLUME-2056] - Allow SpoolDir to pass just the filename that is the source of an event
[FLUME-2185] - Upgrade morphlines to 0.7.0