Release Notes - Flume - Version v0.9.4

Bug

[FLUME-48] - collectorSink writes files with .seq suffix even though text files are written out.
[FLUME-129] - Find and fix bug/race that caused TestTailSourceCursor.testCursorRotatePrexistingSameSizeWithNewModtime to fail.
[FLUME-203] - Fix TestDiskFailoverAgent flakiness
[FLUME-265] - FanOut Collector does not support multiple collectorSink attributes
[FLUME-305] - Fix code that causes TestCollectorSink to fail intermittently.
[FLUME-474] - flume executable doesn't support extending java.library.path with flume-env.sh
[FLUME-482] - Compression producing NullPointerException
[FLUME-496] - Handle 64-bit vs 32-bit JVM during windows install.
[FLUME-511] - Output format args can't be given via the configuration language
[FLUME-513] - Typo fixes
[FLUME-515] - Master Node does not bootstrap custom formats extending from AbstractOutputFormat
[FLUME-516] - bin/flume script does not properly load pre-existing hadoop jars
[FLUME-523] - Missing jar reference in helloworld plugin's build.xml
[FLUME-540] - Large number of duplicates with agentE2ESink due to old log files
[FLUME-548] - TestEvent has a time zone dependency
[FLUME-549] - Uncaught doctext warnings from Thrift
[FLUME-564] - avro-compiler-1.5.0.jar is not needed
[FLUME-571] - Test for DefaultCodec in TestEscapedCustomOutputDfs does not actually work
[FLUME-586] - Fix TestDiskFailoverSource and TestNaiveFileWALSource flakyness
[FLUME-588] - Fix TestDiskFailoverBehavior flakyness
[FLUME-593] - Node should only spawn logical node specified on master.
[FLUME-595] - Node can fork multiple drivers from subsequent heartbeats when there only should be one.
[FLUME-597] - Fix handling interruption of blocked WALSource
[FLUME-600] - Have collector source create names that are both lexographically and chronologically ordered
[FLUME-602] - Fix flaky TestChokeDeco tests.
[FLUME-606] - Update TestFileSource and TestMultiGrep to get data from the classpath
[FLUME-607] - Maven update breaks hello plugin
[FLUME-608] - Maven update removed windows build option
[FLUME-609] - Maven build introduces new test failures.
[FLUME-614] - Build for HBase plugin broken due to maven.
[FLUME-622] - Escape sequences gets stuck with "Illegal group reference" error if replacements have '$' chars.
[FLUME-623] - Hbase plugin build missing hadoop-test jar
[FLUME-624] - Fix intermittent TestAgentSink
[FLUME-630] - Maven build breaks doc build.
[FLUME-642] - fix path of package-info.java annotation.
[FLUME-646] - Add support to the FlumeNode windows installer to detect 32-bit Java Installations (read JAVA_HOME properly) when running on 64 bit windows machines.
[FLUME-661] - Remove duplicate copy of documentation source in ./src/docs
[FLUME-662] - flume dump and dumplog command line options are broken.
[FLUME-667] - stacks webapp is missing from node and master
[FLUME-669] - Flogger fails to compile
[FLUME-673] - Flume distribution problems.

Epic

[FLUME-6] - Add HBase Sink
[FLUME-296] - Mavenize Flume

Improvement

[FLUME-31] - ant tar-binaries target
[FLUME-226] - Parts of the logical nodes section of the manual are out of date.
[FLUME-250] - Allow compression in Avro output formats
[FLUME-299] - Add support for subarguments in flume config elements
[FLUME-414] - Improve syntax of HBase sink
[FLUME-418] - Add batching and compression arguments to agent and collectors
[FLUME-420] - Change Context object to accept Objects.
[FLUME-427] - make compression codec selection case-insensitive
[FLUME-509] - flume-daemon script does not source flume-env script
[FLUME-519] - Move FlumeNode.loadOutputFormatPlugins to FormatFactory and add tests.
[FLUME-520] - Remove various main methods that exist to test
[FLUME-521] - Remove deprecated Thrift code
[FLUME-550] - Factor out Cursor from TailSource
[FLUME-561] - Allow using a file to exclude certain tests from build
[FLUME-569] - Make Drivers handle source/sink open and close lifecycle.
[FLUME-589] - Improve Cancellation to take advantage of InterruptedException
[FLUME-596] - Simplify driver by removing unneeded DriverListener.
[FLUME-598] - Update RPC to send CLOSING state information from LogicalNode Driver.
[FLUME-618] - Test output should be squelched
[FLUME-627] - Make all temp files created by File.createTempFile or FileUtil.mktmpdir live in one dir.
[FLUME-632] - Support arguments in output format configuration
[FLUME-633] - Upgrade to Avro 1.5.1
[FLUME-635] - Allow the compression codec to be specified in SequenceFiles
[FLUME-639] - Remove LET sink remnants.
[FLUME-641] - Support aliases for compression codecs
[FLUME-644] - Remove unused parts of the Tagger
[FLUME-660] - Add seqfile output format to documentation
[FLUME-665] - Consolidate version numbers in maven build.

New Feature

[FLUME-27] - Create log4j appender.
[FLUME-81] - Support for 1 RegExp extractor that populates multi event attributes
[FLUME-92] - Extractor to handle extracting of a date's components from an event field
[FLUME-247] - Add Efficient HBase Sink whith Flexible Event's Attributes Writing
[FLUME-373] - Flume windows installer should be build target in build.
[FLUME-467] - Merge cloudera/hbase with trunk cloudera/master
[FLUME-524] - Create a Lo4j / Logback / etc.. Avro Log Appender
[FLUME-546] - File reading sources should allow for some sort of parameterized record delimiter
[FLUME-572] - Allow using a file to only include certain tests in a build
[FLUME-579] - various rpcSources / Sinks should expose port information in metrics data.
[FLUME-651] - Hook in support for output format arguments into relevant sinks.
[FLUME-663] - add 'flume sink' and 'flume source' commands to ease debugging.

Task

[FLUME-464] - Removed deprecated GZIPCompression setting.
[FLUME-504] - Update Thrift to 0.6.0
[FLUME-530] - Rename "Flume Windows Guide" to be "Installing Flume Nodes on Windows"
[FLUME-545] - Upgrade to Avro 1.5.0
[FLUME-551] - Document tail custom delimiter options.

Sub-task

[FLUME-584] - Change version to 0.9.4-SNAPSHOT (instead of 0.9.10-SNAPSHOT)
[FLUME-585] - Precompile JSP pages
[FLUME-587] - Fix test failures due to relocated files
[FLUME-590] - Figure out why not all tests are running
[FLUME-591] - Support the original URLs in the maven build

Release Notes - Flume - Version v0.9.3

Bug

[FLUME-63] - Master complains about logical nodes contexts with multiple WALs
[FLUME-99] - Errors in default ports documentation
[FLUME-164] - empty files cause NullPointerException, agent stops collecting further data but keeps running in background
[FLUME-280] - '+=' Operation not work. bash-3.0
[FLUME-320] - Race condition in new NIO based TailSource
[FLUME-333] - 'ant tar' did not include LICENSE, RELEASENOTES, README, and NOTICE
[FLUME-336] - Fix Tests that fail due to permissions in windows
[FLUME-337] - TestAvroDataFile tests fail
[FLUME-339] - Fix Tests that break due to windows path escaping.
[FLUME-345] - Generated html from ASCIIDOC contains GPL code.
[FLUME-347] - Developer guide asciidoc does not build.
[FLUME-348] - ThriftEventAdaptor should return null when a nonexistent attribute is fetched
[FLUME-349] - Fix tests that break due to file semantics in windows.
[FLUME-350] - Typo in conf/log4j.properties
[FLUME-362] - Typos in man pages
[FLUME-365] - 'refresh' command with auto*Chain replaces autoCollectorSource with logicalSource.
[FLUME-376] - Multiple WALs not isolated when using multiple agentE2ESinks.
[FLUME-382] - All logical nodes in multi master mode are stuck in DECOMISSIONED mode.
[FLUME-395] - Incorrect zip-compression codec name inside flume documentation
[FLUME-411] - Events sent down each branch of a FanOut sink should be independent
[FLUME-416] - CollectorSink hangs due to ConcurrentModificationException in RollSink
[FLUME-430] - E2E Mode recovery doesn't deliver all events.
[FLUME-433] - Agents end up in ERROR mode when all collectors are killed in BE or DFO mode
[FLUME-435] - Be nice when there is a null body in ThriftEventAdapter
[FLUME-439] - Auto Find Port in StatusHttpServer doesn't work.
[FLUME-466] - Documentation typos
[FLUME-483] - Some code in src/javatest-torture and plugins fails to build.
[FLUME-508] - Fix TestMasterReportPusher tests

Improvement

[FLUME-17] - Remove LET sink expressions
[FLUME-122] - Make all Source/Sink/Decos API calls cancellable by throwing InterruptedExceptions
[FLUME-131] - Handle source/sink/deco cancellation cleanly.
[FLUME-240] - Flume master status page keeps accumulating information
[FLUME-307] - Auto-generate all build version strings during build.
[FLUME-327] - StatusHttpServers should use create paths in a OS agnostic way
[FLUME-377] - Node statuses should be displayed in a sorted order.
[FLUME-379] - Better error message when mapping already mapped logical node.
[FLUME-391] - Changing sleep from 100ms to 10ms in exec source
[FLUME-398] - Node Configurations should be displayed in sorted order.
[FLUME-399] - Update Source API to use InterruptedExceptions
[FLUME-405] - Parameterize gossip ack max age
[FLUME-407] - Remove dead code from FlumeConfiguration
[FLUME-412] - Text Sink should use KW Args.
[FLUME-422] - Remove unsupported FanInSource.
[FLUME-423] - Remove deprecated ExecEventSource
[FLUME-428] - Get rid of FAILCHAIN from language and flume config parser.
[FLUME-434] - Remove multi source from configuration grammer
[FLUME-436] - Reduce the severity of being unable to connect to a master
[FLUME-437] - Reduce the severity of being unable to connect to a master gossip port
[FLUME-438] - Add ${user-name} escape sequence to default agent/master log dirs in FlumeConfiguration.
[FLUME-444] - Remove flumereporter.jsp file from node webapp.
[FLUME-445] - Remove the use of LET in agentDFOChain sink.
[FLUME-446] - Reactivate tests in TestAgentSink.
[FLUME-469] - node table should show flow-id
[FLUME-470] - Flume code generator should regenerate kwargs
[FLUME-479] - Lexer error is inconsistent
[FLUME-487] - Update Thrift to 0.5.0
[FLUME-488] - Fix some tests that have wrong names

New Feature

[FLUME-58] - Builders and configuration language should support keyword arguments
[FLUME-107] - Allow Flume to run on Windows
[FLUME-165] - Allow collector sink to output to other sinks.
[FLUME-208] - add "startFromEnd" args to tailDir
[FLUME-338] - Service'ify flume node/master for windows.
[FLUME-356] - Installer for Windows Flume.
[FLUME-360] - Generic JSON output format
[FLUME-361] - bin/flume dump: Allow specifying output format
[FLUME-367] - JSON interface for master
[FLUME-385] - Flume node read-only JSON interface
[FLUME-408] - Expose ack information in a master for debugging
[FLUME-410] - Add ability to tail subdirs to tailDir sink
[FLUME-413] - Sources should have a Context as an argument.
[FLUME-424] - Make Multi master startup not depend on special config vars or special command line flags.
[FLUME-425] - Comprehensive node metrics traversal mechanism.
[FLUME-426] - Use Jettison Library to generate JSON objects in Master
[FLUME-440] - Add 'dumplog' command to bin/flume
[FLUME-443] - Simplify how decorators are specified.
[FLUME-471] - Add metrics to batching/unbatching/gzip/gunzip decorators
[FLUME-505] - Message Digest decorator

Task

[FLUME-291] - Support using pmd or checkstyle with Flume
[FLUME-309] - Logging Developer Guidelines
[FLUME-315] - Create Developer Guide documentation
[FLUME-328] - Identify Tests that break/fail in Windows
[FLUME-335] - Include asciidoc artifacts in repo and only build docs when necessary
[FLUME-340] - Check that all files use unix style newlines instead of windows style newlines
[FLUME-341] - Link to the DeveloperGuide from the index.html
[FLUME-352] - Document running flume on windows.
[FLUME-415] - Properly handle license for aho-corasick code.
[FLUME-461] - Document auto find port option of flume node
[FLUME-462] - Cleanup type casting warnings in core build.
[FLUME-489] - Update docs to explain new 0.9.3 syntax.