CDH 4 Release Notes

Changes Not In Flume NG 1.2.0

Flume

Bug

• [FLUME-1820] - Should not be possible for RPC client to block indefinitely on close()
• [FLUME-1616] - FileChannel will lose data in when rollback fails with IOException
• [FLUME-1613] - All of the sink examples in the user guide are broken
• [FLUME-1611] - LogUtils regex can be precompiled
• [FLUME-1547] - avro-sink example is wrong in user guide
• [FLUME-1610] - HDFSEventSink and bucket writer have a race condition
• [FLUME-1606] - Rollbacks of Put transactions does not clear the transaction from inflight puts.
• [FLUME-1595] - HDFS SequenceFile implementation is not durable due to not using syncFs()
• [FLUME-1570] - StressSource batching does not work unless maxTotalEvents is specified
• [FLUME-1582] - flume-ng script should set LD_LIBRARY_PATH
• [FLUME-1583] - FileChannel fast full replay will always be used if enabled
• [FLUME-1578] - Proposal to modify file channel encryption config
• [FLUME-1577] - CHECKPOINT_INCOMPLETE should be synced to disk before starting the checkpoint.
• [FLUME-1572] - Add batching to FILE_ROLL sink
• [FLUME-1565] - FileChannel Decryption in RandomReader is not thread safe
• [FLUME-1557] - It would be nice if SequenceGeneratorSource could do batching
• [FLUME-1470] - Syslog source does not parse facility correctly
• [FLUME-1562] - TestLoadBalancingSinkProcessor.testRoundRobinBackoffFailureRecovery is flaky, fails every once in a while...
• [FLUME-1553] - TestFileChannelEncryption should be refactored to use TestFileChannelBase
• [FLUME-1555] - StressSource outputs bad log messages that reference (Sequence generator)
• [FLUME-1513] - File Channel log close() method should not be synchronized
• [FLUME-1506] - Child poms pull in specific versions of packages not in top level pom
• [FLUME-1552] - TestFileChannelEncryption fails without a high encryption policy file
• [FLUME-1544] - Update dev guide to reflect the protoc requirement
• [FLUME-1545] - File channel missing implicit dependency on commons-lang
• [FLUME-1541] - Implement a SinkSelector for LoadBalancingSinkProcessor that includes failover mechanics
• [FLUME-1534] - CheckpointRebuilder$ComparableFlumeEventPointer#equals does not work correctly.
• [FLUME-1532] - Increase test timeouts even further.
• [FLUME-1527] - Seed TransactionID/WriteOrderID in the case no logs need to be replayed
• [FLUME-1525] - On some (slow) machines TestFileChannel can fail
• [FLUME-1524] - TestMonitoredCounterGroup is flaky
• [FLUME-1428] - File Channel should not consider a file as inactive until all takes are committed.
• [FLUME-1512] - File Channel should not stop during a checkpoint.
• [FLUME-1500] - Upgrade flume to use latest version of Avro - v1.7
• [FLUME-1489] - scribe-source sources not included in packaged source tarball
• [FLUME-1504] - Test file channel times out randomly
• [FLUME-1498] - File channel - Log updates and queue updates should be atomic
• [FLUME-1437] - Checkpoint can miss pending takes.
• [FLUME-1482] - Flume should support exposing metrics via HTTP in JSON/some other web service format.
• [FLUME-1432] - FileChannel should replay logs in the order they were written
• [FLUME-1417] - File Channel checkpoint can be bad leading to the channel being unable to start.
• [FLUME-1422] - Fix "BarSource" Class Signature in Flume Developer Guide
• [FLUME-1421] - PollableSourceRunner does not name it's thread
• [FLUME-1416] - Version Info should have hardcoded git repo address
• [FLUME-1414] - VersionInfo should not create a log instance
• [FLUME-1412] - Commons collections is used in file channel - even though it is not in pom.xml
• [FLUME-1389] - Flume gives opaque error if interceptor type not specified
• [FLUME-1391] - Use sync() instead of syncFs() in HDFS Sink to be compatible with hadoop 0.20.2
• [FLUME-1401] - Asynchbase sink should be configurable to support timeout
• [FLUME-1354] - Update docs to show that recoverable memory channel is deprecated
• [FLUME-1392] - Inactive channels get added to source channels list causing NPE
• [FLUME-1398] - Improve concurrency for async hbase sink
• [FLUME-1386] - Ganglia metrics plugin should filter MBean name
• [FLUME-1373] - Remove hardcoded file separator in HDFSEventSink
• [FLUME-1380] - File channel log can record the op code and not the operation in some cases
• [FLUME-1377] - ChannelProcessor clobbers exception with NPE if Channel.getTransaction() throws
• [FLUME-1376] - StaticInterceptor doc update
• [FLUME-1374] - Support ganglia reporting
• [FLUME-1369] - Code and user guide documentation out of sync for interceptors
• [FLUME-1368] - In user guide, property sink.directory for file roller sink should be bold
• [FLUME-1362] - Port retrying in TestThriftLegacySource not working
• [FLUME-1364] - Document the necessity of the timestamp header when using time-related escapes for hdfs sink paths
• [FLUME-1363] - flume-ng-node - TestNetcatSource doesn't try multiple ports
• [FLUME-1310] - Make Asynch hbase sink test work with other versions of Hbase
• [FLUME-1276] - Create a static header interceptor

Improvement

• [FLUME-1762] - File Channel should recover automatically if the checkpoint is incomplete or bad by deleting the contents of the checkpoint directory
• [FLUME-1607] - FileChannel We should use a regex as opposed to simple filename filter when finding logs
• [FLUME-1600] - FileChannel metadata files should be written to a temp file and then moved over existing files
• [FLUME-1603] - FileChannel capacity reached message is unclear
• [FLUME-1599] - FileChannel error message and documentation is stale
• [FLUME-1590] - ExecSource doesn't kill it's child when shutting down
• [FLUME-1575] - FIleChannel Encryption should disallow a null key
• [FLUME-1538] - Channels should expose channel fill ratio through JMX
• [FLUME-1564] - FileChannel log file creation could be clarified and tested
• [FLUME-1563] - FileChannel Encryption KeyProvider configuration properties should be more consistent
• [FLUME-1554] - FileChannel fails to build on machines with old protocol buffer compiler
• [FLUME-1556] - It would be nice if NullSink logged the number of event processed every 10K or so
• [FLUME-1480] - Replace object descriptor with detailed component type plus name
• [FLUME-1519] - LifecycleController prints tons of DEBUG messages
• [FLUME-1560] - TestFileChannel* tests which fill up the channel should use larger batch size than 1
• [FLUME-1536] - Support for batch size in StressSource
• [FLUME-1550] - Use maven-antrun-plugin to save version
• [FLUME-1548] - Build dies due to older protocol buffers compiler
• [FLUME-1543] - TestFileChannel should be factored into many tests
• [FLUME-1546] - File channel encryption: trim() passwords and warn user if he doesn't have JCE policy file
• [FLUME-1487] - FileChannel format needs to be extensible
• [FLUME-1496] - TestFileChannel is bloated
• [FLUME-1509] - Flume HDFS sink should allow for the use of different timezones when resolving sink paths
• [FLUME-1526] - LogFile log message is scary when no harm has been done
• [FLUME-1507] - Have "Topology Design Considerations" in User Guide
• [FLUME-1531] - Flume User Guide should provide more details on configuring the timestamp interceptor
• [FLUME-1528] - File Channel replay when no checkpoint is present can be faster.
• [FLUME-1523] - Allow -X java opts to be passed to the agent on the command line
• [FLUME-1508] - TestFileChannel times out maven periodically
• [FLUME-1505] - TestFileChannel needs to be able to force a checkpoint
• [FLUME-1490] - Option to limit number of events sent in Stress source
• [FLUME-1430] - FileChannel Log.writeCheckpoint needs better documentation and variable names
• [FLUME-1434] - Distinguish background worker with channel name
• [FLUME-1418] - Improvement for Log4j configuration
• [FLUME-1408] - ScheduledExecutorService does not log uncaught Throwables, we should log them
• [FLUME-1358] - Add a regex-based filtering interceptor
• [FLUME-1383] - Improve various log messages in FileChannel and HDFSSink
• [FLUME-1387] - Ganglia graphs should work with Flume
• [FLUME-1360] - Provide documentation for static interceptor
• [FLUME-1361] - Add event batching to ExecSource

New Feature

• [FLUME-1385] - Add a multiport syslog source
• [FLUME-1537] - Dump RollingFileSink's counter status when agent stops
• [FLUME-1488] - Load Balancing RPC client should support exponential backoff of failed nodes
• [FLUME-1424] - File Channel should support encryption
• [FLUME-1382] - Flume adopt message from existing local Scribe

Test

• [FLUME-1492] - Create integration test for file channel