# Hadoop 0.20.2 Release Notes

These release notes include new developer and user-facing incompatibilities, features, and major improvements. The table below is sorted by Component.

## Changes Since Hadoop 0.20.1

### Common

#### Bug

- [HADOOP-4802] - RPC Server send buffer retains size of largest response ever sent
- [HADOOP-5611] - C++ libraries do not build on Debian Lenny
- [HADOOP-5612] - Some c++ scripts are not chmodded before ant execution
- [HADOOP-5623] - Streaming: process provided status messages are overwritten every 10 seoncds
- [HADOOP-5759] - IllegalArgumentException when CombineFileInputFormat is used as job InputFormat
- [HADOOP-6097] - Multiple bugs w/ Hadoop archives
- [HADOOP-6231] - Allow caching of filesystem instances to be disabled on a per-instance basis
- [HADOOP-6269] - Missing synchronization for defaultResources in Configuration.addResource
- [HADOOP-6315] - GzipCodec should not represent BuiltInZlibInflater as decompressorType
- [HADOOP-6386] - NameNode's HttpServer can't instantiate InetSocketAddress: IllegalArgumentException is thrown
- [HADOOP-6428] - HttpServer sleeps with negative values
- [HADOOP-6460] - Namenode runs of out of memory due to memory leak in ipc Server
- [HADOOP-6498] - IPC client bug may cause rpc call hang
- [HADOOP-6506] - Failing tests prevent the rest of test targets from execution.
- [HADOOP-6524] - Contrib tests are failing Clover'ed build
- [HADOOP-6575] - Tests do not run on 0.20 branch
- [HADOOP-6576] - TestStreamingStatus is failing on 0.20 branch

#### Task

- [HADOOP-6328] - Hadoop 0.20 Docs - backport changes for streaming and m/r tutorial docs

### HDFS

#### Bug

- [HDFS-101] - DFS write pipeline : DFSClient sometimes does not detect second datanode failure
- [HDFS-185] - Chown , chgrp , chmod operations allowed when namenode is in safemode .
- [HDFS-187] - TestStartup fails if hdfs is running in the same machine
- [HDFS-495] - Hadoop FSNamesystem startFileInternal() getLease() has bug
- [HDFS-579] - HADOOP-3792 update of DfsTask incomplete
- [HDFS-596] - Memory leak in libhdfs: hdfsFreeFileInfo() in libhdfs does not free memory for mOwner and mGroup
- [HDFS-677] - Rename failure due to quota results in deletion of src directory
- [HDFS-723] - Deadlock in DFSClient#DFSOutputStream
- [HDFS-732] - HDFS files are ending up truncated
- [HDFS-734] - TestDatanodeBlockScanner times out in branch 0.20
- [HDFS-745] - TestFsck timeout on 0.20.
- [HDFS-761] - Failure to process rename operation from edits log due to quota verification
- [HDFS-781] - Metrics PendingDeletionBlocks is not decremented
- [HDFS-793] - DataNode should first receive the whole packet ack message before it constructs and sends its own ack message for the packet
- [HDFS-795] - DFS Write pipeline does not detect defective datanode correctly in some cases (HADOOP-3339)
- [HDFS-872] - DFSClient 0.20.1 is incompatible with HDFS 0.20.2
- [HDFS-927] - DFSInputStream retries too many times for new block locations

#### Test

- [HDFS-784] - TestFsck times out on branch 0.20.1
- [HDFS-907] - Add tests for getBlockLocations and totalLoad metrics.
- [HDFS-919] - Create test to validate the BlocksVerified metric

### MapReduce

#### Bug

- [MAPREDUCE-112] - Reduce Input Records and Reduce Output Records counters are not being set when using the new Mapreduce reducer API
- [MAPREDUCE-433] - TestReduceFetch failed.
- [MAPREDUCE-826] - harchive doesn't use ToolRunner / harchive returns 0 even if the job fails with exception
- [MAPREDUCE-979] - JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
- [MAPREDUCE-1010] - Adding tests for changes in archives.
- [MAPREDUCE-1068] - In hadoop-0.20.0 streaming job do not throw proper verbose error message if file is not present
- [MAPREDUCE-1070] - Deadlock in FairSchedulerServlet
- [MAPREDUCE-1112] - Fix CombineFileInputFormat for hadoop 0.20
- [MAPREDUCE-1147] - Map output records counter missing for map-only jobs in new API
- [MAPREDUCE-1163] - hdfsJniHelper.h: Yahoo! specific paths are encoded
- [MAPREDUCE-1182] - Reducers fail with OutOfMemoryError while copying Map outputs
- [MAPREDUCE-1251] - c++ utils doesn't compile
- [MAPREDUCE-1328] - contrib/index - modify build / ivy files as appropriate

#### Improvement

- [MAPREDUCE-623] - Resolve javac warnings in mapred

#### New Feature

- [MAPREDUCE-1145] - Multiple Outputs doesn't work with new API in 0.20 branch
- [MAPREDUCE-1170] - MultipleInputs doesn't work with new API in 0.20 branch

## Changes Since Hadoop 0.20.0

### Common

#### Sub-task

- [HADOOP-6213] - Remove commons dependency on commons-cli2

#### Bug

- [HADOOP-4626] - API link in forrest doc should point to the same version of hadoop.
- [HADOOP-4674] - hadoop fs -help should list detailed help info for the following commands: test, text, tail, stat & touchz
- [HADOOP-4856] - Document JobInitializationPoller configuration in capacity scheduler forrest documentation.
- [HADOOP-4931] - Document TaskTracker's memory management functionality and CapacityScheduler's memory based scheduling.
- [HADOOP-5210] - Reduce Task Progress shows > 100% when the total size of map outputs (for a single reducer) is high
- [HADOOP-5213] - BZip2CompressionOutputStream NullPointerException
- [HADOOP-5349] - When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
- [HADOOP-5533] - Recovery duration shown on the jobtracker webpage is inaccurate
- [HADOOP-5539] - o.a.h.mapred.Merger not maintaining map out compression on intermediate files
- [HADOOP-5636] - Job is left in Running state after a killJob
- [HADOOP-5641] - Possible NPE in CapacityScheduler's MemoryMatcher
- [HADOOP-5646] - TestQueueCapacities is failing Hudson tests for the last few builds
- [HADOOP-5648] - Not able to generate gridmix.jar on already compiled version of hadoop
- [HADOOP-5654] - TestReplicationPolicy.<init> fails on java.net.BindException
- [HADOOP-5655] - TestMRServerPorts fails on java.net.BindException
- [HADOOP-5688] - HftpFileSystem.getChecksum(..) does not work for the paths with scheme and authority
- [HADOOP-5691] - org.apache.hadoop.mapreduce.Reducer should not be abstract.
- [HADOOP-5711] - Change Namenode file close log to info
- [HADOOP-5718] - Capacity Scheduler should not check for presence of default queue while starting up.
- [HADOOP-5719] - Jobs failed during job initalization are never removed from Capacity Schedulers waiting list
- [HADOOP-5736] - Update CapacityScheduler documentation to reflect latest changes
- [HADOOP-5746] - Errors encountered in MROutputThread after the last map/reduce call can go undetected
- [HADOOP-5796] - DFS Write pipeline does not detect defective datanode correctly in some cases (HADOOP-3339)
- [HADOOP-5828] - Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
- [HADOOP-5850] - map/reduce doesn't run jobs with 0 maps
- [HADOOP-5863] - mapred metrics shows negative count of waiting maps and reduces
- [HADOOP-5869] - TestQueueCapacitisues.apache.org/jjira/browse/HADOOP-OP-6017] - NameNode and SecondaryNameNode fail to restart because of abnormal filenames.
- [HADOOP-6097] - Multiple bugs w/ Hadoop archives
- [HADOOP-6139] - Incomplete help message is displayed for rm and rmr options.
- [HADOOP-6141] - hadoop 0.20 branch "test-patch" is broken
- [HADOOP-6145] - No error message for deleting non-existant file or directory.
- [HADOOP-6215] - fix GenericOptionParser to deal with -D with '=' in the value

#### Improvement

- [HADOOP-5726] - Remove pre-emption from the capacity scheduler code base

#### New Feature

- [HADOOP-3315] - New binary file format
- [HADOOP-5714] - Metric to show number of fs.exists (or number of getFileInfo) calls
- [HADOOP-6080] - Handling of Trash with quota

### HDFS

#### Bug

- [HDFS-26] - HADOOP-5862 for version .20 (Namespace quota exceeded message unclear)
- [HDFS-167] - DFSClient continues to retry indefinitely
- [HDFS-438] - Improve help message for quotas
- [HDFS-442] - dfsthroughput in test.jar throws NPE
- [HDFS-485] - error : too many fetch failures
- [HDFS-495] - Hadoop FSNamesystem startFileInternal() getLease() has bug
- [HDFS-525] - ListPathsServlet.java uses static SimpleDateFormat that has threading issues

#### Improvement

- [HDFS-504] - HDFS updates the modification time of a file when the file is closed.
- [HDFS-527] - Refactor DFSClient constructors

### Map/Reduce

#### Bug

- [MAPREDUCE-2] - ArrayOutOfIndex error in KeyFieldBasedPartitioner on empty key
- [MAPREDUCE-18] - Under load the shuffle sometimes gets incorrect data
- [MAPREDUCE-40] - Memory management variables need a backwards compatibility option after HADOOP-5881
- [MAPREDUCE-112] - Reduce Input Records and Reduce Output Records counters are not being set when using the new Mapreduce reducer API
- [MAPREDUCE-124] - When abortTask of OutputCommitter fails with an Exception for a map-only job, the task is marked as success
- [MAPREDUCE-130] - Delete the jobconf copy from the log directory of the JobTracker when the job is retired
- [MAPREDUCE-179] - setProgress not called for new RecordReaders
- [MAPREDUCE-383] - pipes combiner does not reset properly after a spill
- [MAPREDUCE-421] - mapred pipes might return exit code 0 even when failing
- [MAPREDUCE-430] - Task stuck in cleanup with OutOfMemoryErrors
- [MAPREDUCE-565] - Partitioner does not work with new API
- [MAPREDUCE-657] - CompletedJobStatusStore hardcodes filesystem to hdfs
- [MAPREDUCE-687] - TestMiniMRMapRedDebugScript fails sometimes
- [MAPREDUCE-735] - ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
- [MAPREDUCE-745] - TestRecoveryManager fails sometimes
- [MAPREDUCE-796] - Encountered "ClassCastException" on tasktracker while running wordcount with MultithreadedMapRunner
- [MAPREDUCE-805] - Deadlock in Jobtracker
- [MAPREDUCE-806] - WordCount example does not compile given the current instructions
- [MAPREDUCE-807] - Stray user files in mapred.system.dir with permissions other than 777 can prevent the jobtracker from starting up.
- [MAPREDUCE-818] - org.apache.hadoop.mapreduce.Counters.getGroup returns null if the group name doesnt exist.
- [MAPREDUCE-827] - "hadoop job -status <jobid>" command should display job's completion status also.
- [MAPREDUCE-832] - Too many WARN messages about deprecated memorty config variables in JobTacker log
- [MAPREDUCE-834] - When TaskTracker config use old memory management values its memory monitoring is diabled.
- [MAPREDUCE-838] - Task succeeds even when committer.commitTask fails with IOException
- [MAPREDUCE-911] - TestTaskFail fail sometimes
- [MAPREDUCE-924] - TestPipes crashes on trunk

#### Improvement

- [MAPREDUCE-465] - Deprecate org.apache.hadoop.mapred.lib.MultithreadedMapRunner
- [MAPREDUCE-487] - DBInputFormat support for Oracle
- [MAPREDUCE-767] - to remove mapreduce dependency on commons-cli2

## Changes Since Hadoop 0.19.1

Issue | Component | Notes
--- | --- | ---
HADOOP-3344 | build | Changed build procedure for libhdfs to build correctly for different platforms. Build instructions are in the Jira item.
HADOOP-4253 | conf | Removed from class org.apache.hadoop.fs.RawLocalFileSystem deprecated methods public String getName(), public void lock(Path p, boolean shared) and public void release(Path p).
HADOOP-4454 | conf | Changed processing of conf/slaves file to allow # to begin a comment.
HADOOP-4631 | conf | Split hadoop-default.xml into core-default.xml, hdfs-default.xml and mapreduce-default.xml.
HADOOP-4035 | contrib/capacity-sched | Changed capacity scheduler policy to take note of task memory requirements and task tracker memory availability.
HADOOP-4445 | contrib/capacity-sched | Changed JobTracker UI to better present the number of active tasks.
HADOOP-4576 | contrib/capacity-sched | Changed capacity scheduler UI to better present number of running and pending tasks.
HADOOP-4179 | contrib/chukwa | Introduced Vaidya rule based performance diagnostic tool for Map/Reduce jobs.
HADOOP-4827 | contrib/chukwa | Improved framework for data aggregation in Chuckwa.
HADOOP-4843 | contrib/chukwa | Introduced Chuckwa collection of job history.
HADOOP-5030 | contrib/chukwa | Changed RPM install location to the value specified by build.properties file.
HADOOP-5531 | contrib/chukwa | Disabled Chukwa unit tests for 0.20 branch only.
HADOOP-4789 | contrib/fair-share | Changed fair scheduler to divide resources equally between pools, not jobs.
HADOOP-4873 | contrib/fair-share | Changed fair scheduler UI to display minMaps and minReduces variables.
HADOOP-3750 | dfs | Removed deprecated method parseArgs from org.apache.hadoop.fs.FileSystem.
HADOOP-4029 | dfs | Added name node storage information to the dfshealth page, and moved data node information to a separated page.
HADOOP-4103 | dfs | Modified dfsadmin -report to report under replicated blocks. blocks with corrupt replicas, and missing blocks".
HADOOP-4567 | dfs | Changed GetFileBlockLocations to return topology information for nodes that host the block replicas.
HADOOP-4572 | dfs | Moved org.apache.hadoop.hdfs.{CreateEditsLog, NNThroughputBenchmark} to org.apache.hadoop.hdfs.server.namenode.
HADOOP-4618 | dfs | Moved HTTP server from FSNameSystem to NameNode. Removed FSNamesystem.getNameNodeInfoPort(). Replaced FSNamesystem.getDFSNameNodeMachine() and FSNamesystem.getDFSNameNodePort() with new method FSNamesystem.getDFSNameNodeAddress(). Removed constructor NameNode(bindAddress, conf).
HADOOP-4826 | dfs | Introduced new dfsadmin command saveNamespace to command the name service to do an immediate save of the file system image.
HADOOP-4970 | dfs | Changed trash facility to use absolute path of the deleted file.
HADOOP-5468 | documentation | Reformatted HTML documentation for Hadoop to use submenus at the left column.
HADOOP-3497 | fs | Changed the semantics of file globbing with a PathFilter (using the globStatus method of FileSystem). Previously, the filtering was too restrictive, so that a glob of /*/* and a filter that only accepts /a/b would not have matched /a/b. With this change /a/b does match.
HADOOP-4234 | fs | Changed KFS glue layer to allow applications to interface with multiple KFS metaservers.
HADOOP-4422 | fs/s3 | Modified Hadoop file system to no longer create S3 buckets. Applications can create buckets for their S3 file systems by other means, for example, using the JetS3t API.
HADOOP-3063 | io | Introduced BloomMapFile subclass of MapFile that creates a Bloom filter from all keys.
HADOOP-1230 | mapred | Replaced parameters with context obejcts in Mapper, Reducer, Partitioner, InputFormat, and OutputFormat classes.
HADOOP-1650 | mapred | Upgraded all core servers to use Jetty 6
HADOOP-3923 | mapred | Moved class org.apache.hadoop.mapred.StatusHttpServer to org.apache.hadoop.http.HttpServer.
HADOOP-3986 | mapred | Removed classes org.apache.hadoop.mapred.JobShell and org.apache.hadoop.mapred.TestJobShell. Removed from JobClient methods static void setCommandLineConfig(Configuration conf) and public static Configuration getCommandLineConfig().
HADOOP-4188 | mapred | Removed Task's dependency on concrete file systems by taking list from FileSystem class. Added statistics table to FileSystem class. Deprecated FileSystem method getStatistics(Class<? extends FileSystem> cls).
HADOOP-4210 | mapred | Changed public class org.apache.hadoop.mapreduce.ID to be an abstract class. Removed from class org.apache.hadoop.mapreduce.ID the methods public static ID read(DataInput in) and public static ID forName(String str).
HADOOP-4305 | mapred | Improved TaskTracker blacklisting strategy to better exclude faulty tracker from executing tasks.
HADOOP-4435 | mapred | Changed JobTracker web status page to display the amount of heap memory in use. This changes the JobSubmissionProtocol.
HADOOP-4565 | mapred | Improved MultiFileInputFormat so that multiple blocks from the same node or same rack can be combined into a single split.
HADOOP-4749 | mapred | Added a new counter REDUCE_INPUT_BYTES.
HADOOP-4783 | mapred | Changed history directory permissions to 750 and history file permissions to 740.
HADOOP-3422 | metrics | Changed names of ganglia metrics to avoid conflicts and to better identify source function.
HADOOP-4284 | security | Introduced HttpServer method to support global filters.
HADOOP-4575 | security | Introduced independent HSFTP proxy server for authenticated access to clusters.
HADOOP-4661 | tools/distcp | Introduced distch tool for parallel ch{mod, own, grp}.