CDH 3 Release Notes
The following lists all Flume NG Jiras included in CDH 3
that are not included in the Flume NG base version 1.2.0. The
file lists all changes included in CDH 3. The patch for each
change can be found in the cloudera/patches directory in the release tarball.
Changes Not In Flume NG 1.2.0
- [FLUME-1955] - fileSuffix does not work with compressed streams
- [FLUME-1939] - FlumeEventQueue must check if file is open before setting the length of the file
- [FLUME-1937] - Issue with maxUnderReplication in HDFS sink
- [FLUME-1931] - HDFS Sink has a commons-lang dependency which is missing in pom
- [FLUME-1776] - Several modules require commons-lang but do not declare this in the pom
- [FLUME-1916] - HDFS sink should poll for # of active replicas. If less than required, roll the file.
- [FLUME-1925] - HDFS timeouts should not starve other threads
- [FLUME-1922] - HDFS Sink should optionally insert the timestamp at the sink
- [FLUME-1880] - Double-logging of created HDFS files
- [FLUME-1865] - Rename the Sequence File formatters to Serializer to be consistent with the rest of Flume
- [FLUME-1844] - HDFSEventSink should have option to use RawLocalFileSystem
- [FLUME-1748] - HDFS Sink should check if the thread is interrupted before performing any HDFS operations
- [FLUME-1709] - HDFS CompressedDataStream doesn't support serializer parameter
- [FLUME-1820] - Should not be possible for RPC client to block indefinitely on close()
- [FLUME-1869] - Request to add "HTTP" source type to SourceType.java
- [FLUME-1659] - JSON Handler should return simple events, not JSONEvents.
- [FLUME-1562] - TestLoadBalancingSinkProcessor.testRoundRobinBackoffFailureRecovery is flaky, fails every once in a while...
- [FLUME-1616] - FileChannel will lose data in when rollback fails with IOException
- [FLUME-1567] - Avro source should expose the number of active connections through JMX
- [FLUME-1610] - HDFSEventSink and bucket writer have a race condition
- [FLUME-1606] - Rollbacks of Put transactions does not clear the transaction from inflight puts.
- [FLUME-1583] - FileChannel fast full replay will always be used if enabled
- [FLUME-1577] - CHECKPOINT_INCOMPLETE should be synced to disk before starting the checkpoint.
- [FLUME-1572] - Add batching to FILE_ROLL sink
- [FLUME-1557] - It would be nice if SequenceGeneratorSource could do batching
- [FLUME-1555] - StressSource outputs bad log messages that reference (Sequence generator)
- [FLUME-1545] - File channel missing implicit dependency on commons-lang
- [FLUME-1541] - Implement a SinkSelector for LoadBalancingSinkProcessor that includes failover mechanics
- [FLUME-1482] - Flume should support exposing metrics via HTTP in JSON/some other web service format.
- [FLUME-1500] - Upgrade flume to use latest version of Avro - v1.7
- [FLUME-1534] - CheckpointRebuilder$ComparableFlumeEventPointer#equals does not work correctly.
- [FLUME-1527] - Seed TransactionID/WriteOrderID in the case no logs need to be replayed
- [FLUME-1513] - File Channel log close() method should not be synchronized
- [FLUME-1428] - File Channel should not consider a file as inactive until all takes are committed.
- [FLUME-1512] - File Channel should not stop during a checkpoint.
- [FLUME-1504] - Test file channel times out randomly
- [FLUME-1498] - File channel - Log updates and queue updates should be atomic
- [FLUME-1437] - Checkpoint can miss pending takes.
- [FLUME-1432] - FileChannel should replay logs in the order they were written
- [FLUME-1417] - File Channel checkpoint can be bad leading to the channel being unable to start.
- [FLUME-1401] - Asynchbase sink should be configurable to support timeout
- [FLUME-1398] - Improve concurrency for async hbase sink
- [FLUME-1414] - VersionInfo should not create a log instance
- [FLUME-1412] - Commons collections is used in file channel - even though it is not in pom.xml
- [FLUME-1386] - Ganglia metrics plugin should filter MBean name
- [FLUME-1373] - Remove hardcoded file separator in HDFSEventSink
- [FLUME-1380] - File channel log can record the op code and not the operation in some cases
- [FLUME-1377] - ChannelProcessor clobbers exception with NPE if Channel.getTransaction() throws
- [FLUME-1376] - StaticInterceptor doc update
- [FLUME-1374] - Support ganglia reporting
- [FLUME-1369] - Code and user guide documentation out of sync for interceptors
- [FLUME-1368] - In user guide, property sink.directory for file roller sink should be bold
- [FLUME-1362] - Port retrying in TestThriftLegacySource not working
- [FLUME-1364] - Document the necessity of the timestamp header when using time-related escapes for hdfs sink paths
- [FLUME-1363] - flume-ng-node - TestNetcatSource doesn't try multiple ports
- [FLUME-1310] - Make Asynch hbase sink test work with other versions of Hbase
- [FLUME-1276] - Create a static header interceptor
- [FLUME-1496] - TestFileChannel is bloated
- [FLUME-1508] - TestFileChannel times out maven periodically
- [FLUME-1864] - Allow hdfs idle callback to clean up closed bucket writers
- [FLUME-1848] - HDFSDataStream logger is actually for a sequence file
- [FLUME-1100] - HDFSWriterFactory and HDFSFormatterFactory should allow extension
- [FLUME-1702] - HDFSEventSink should write to a hidden file as opposed to a .tmp file
- [FLUME-1730] - HDFSEventSink testCloseOnIdle appears to be racy
- [FLUME-1660] - Close "idle" hdfs handles
- [FLUME-1645] - add hdfs.fileSuffix property to HDFSEventSink
- [FLUME-1631] - Retire hdfs.txnEventMax in HDFS sink
- [FLUME-1509] - Flume HDFS sink should allow for the use of different timezones when resolving sink paths
- [FLUME-1408] - ScheduledExecutorService does not log uncaught Throwables, we should log them
- [FLUME-1383] - Improve various log messages in FileChannel and HDFSSink
- [FLUME-1821] - Support configuration of hbase instances to be used in AsyncHBaseSink from flume config
- [FLUME-1523] - Allow -X java opts to be passed to the agent on the command line
- [FLUME-1538] - Channels should expose channel fill ratio through JMX
- [FLUME-1536] - Support for batch size in StressSource
- [FLUME-1526] - LogFile log message is scary when no harm has been done
- [FLUME-1490] - Option to limit number of events sent in Stress source
- [FLUME-1528] - File Channel replay when no checkpoint is present can be faster.
- [FLUME-1505] - TestFileChannel needs to be able to force a checkpoint
- [FLUME-1434] - Distinguish background worker with channel name
- [FLUME-1418] - Improvement for Log4j configuration
- [FLUME-1387] - Ganglia graphs should work with Flume
- [FLUME-1360] - Provide documentation for static interceptor
- [FLUME-1361] - Add event batching to ExecSource
- [FLUME-1629] - Add Audience/Stability annotations
- [FLUME-1199] - Add HTTP Post Source
- [FLUME-1537] - Dump RollingFileSink's counter status when agent stops
- [FLUME-1488] - Load Balancing RPC client should support exponential backoff of failed nodes