CDH 4 Release Notes

Changes Not In Apache Hadoop 0.23.0

Common

Bug

• [HADOOP-7999] - "hadoop archive" fails with ClassNotFoundException
• [HADOOP-8012] - hadoop-daemon.sh and yarn-daemon.sh are trying to mkdir and chow log/pid dirs which can fail
• [HADOOP-8000] - fetchdt command not available in bin/hadoop
• [HADOOP-7993] - Hadoop ignores old-style config options for enabling compressed output
• [HADOOP-7997] - SequenceFile.createWriter(...createParent...) no longer works on existing file
• [HADOOP-7981] - Improve documentation for org.apache.hadoop.io.compress.Decompressor.getRemaining
• [HADOOP-7986] - Add config for History Server protocol in hadoop-policy for service level authorization.
• [HADOOP-7982] - UserGroupInformation fails to login if thread's context classloader can't load HadoopLoginModule
• [HADOOP-7971] - hadoop <job/queue/pipes> removed - should be added back, but deprecated
• [HADOOP-7968] - Errant println left in RPC.getHighestSupportedProtocol
• [HADOOP-7975] - Add entry to XML defaults for new LZ4 codec
• [HADOOP-7972] - HAServiceProtocol exceptions need to be unwrapped before it can be used
• [HADOOP-2776] - Web interface uses internal hostnames on EC2
• [HADOOP-7964] - Deadlock in class init.
• [HADOOP-7974] - TestViewFsTrash incorrectly determines the user's home directory
• [HADOOP-7963] - test failures: TestViewFileSystemWithAuthorityLocalFileSystem and TestViewFileSystemLocalFileSystem

Improvement

• [HADOOP-7965] - Support for protocol version and signature in PB

New Feature

• [HADOOP-7991] - HA: the FailoverController should check the standby is ready before failing over
• [HADOOP-7938] - HA: the FailoverController should optionally fence the active during failover
• [HADOOP-7983] - HA: failover should be able to pass args to fencers
• [HADOOP-7992] - Add ZK client for leader election
• [HADOOP-7970] - All HAServiceProtocol methods have to throw IOException since they are invoked over the network

HDFS

Bug

• [HDFS-2923] - Namenode IPC handler count uses the wrong configuration key
• [HDFS-2893] - The start/stop scripts don't start/stop the 2NN when using the default configuration
• [HDFS-2889] - getNumCurrentReplicas is package private but should be public on 0.23 (see HDFS-2408)
• [HDFS-2877] - If locking of a storage dir fails, it will remove the other NN's lock file on exit
• [HDFS-2759] - Pre-allocate HDFS edit log files after writing version number
• [HDFS-2791] - If block report races with closing of file, replica is incorrectly marked corrupt
• [HDFS-2840] - TestHostnameFilter should work with localhost or localhost.localdomain
• [HDFS-2837] - mvn javadoc:javadoc not seeing LimitedPrivate class
• [HDFS-2836] - HttpFSServer still has 2 javadoc warnings in trunk
• [HDFS-2838] - NPE in FSNamesystem when in safe mode
• [HDFS-442] - dfsthroughput in test.jar throws NPE
• [HDFS-2818] - dfshealth.jsp missing space between role and node name
• [HDFS-2816] - Fix missing license header in hadoop-hdfs-project/hadoop-hdfs-httpfs/dev-support/findbugsExcludeFile.xml
• [HDFS-2751] - Datanode drops OS cache behind reads even for short reads
• [HDFS-2810] - Leases not properly getting renewed by clients
• [HDFS-2768] - BackupNode stop can not close proxy connections because it is not a proxy instance.
• [HDFS-2790] - FSNamesystem.setTimes throws exception with wrong configuration name in the message
• [HDFS-69] - Improve dfsadmin command line help
• [HDFS-2739] - SecondaryNameNode doesn't start up

Improvement

• [HDFS-2397] - Undeprecate SecondaryNameNode
• [HDFS-2817] - Combine the two TestSafeMode test suites
• [HDFS-2803] - Adding logging to LeaseRenewer for better lease expiration triage.
• [HDFS-2788] - HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code
• [HDFS-362] - FSEditLog should not writes long and short as UTF8 and should not use ArrayWritable for writing non-array items

New Feature

• [HDFS-2579] - Starting delegation token manager during safemode fails
• [HDFS-2794] - HA: Active NN may purge edit log files before standby NN has a chance to read them
• [HDFS-2733] - Document HA configuration and CLI
• [HDFS-2894] - HA: automatically determine the nameservice Id if only one nameservice is configured
• [HDFS-2752] - HA: exit if multiple shared dirs are configured
• [HDFS-2819] - Document new HA-related configs in hdfs-default.xml
• [HDFS-2808] - HA: haadmin should use namenode ids
• [HDFS-2792] - HA: Make fsck work
• [HDFS-2890] - HA: DFSUtil#getSuffixIDs should skip unset configurations
• [HDFS-2874] - HA: edit log should log to shared dirs before local dirs
• [HDFS-2863] - Failures observed if dfs.edits.dir and shared.edits.dir have same directories.
• [HDFS-2769] - HA: When HA is enabled with a shared edits dir, that dir should be marked required
• [HDFS-2860] - HA: TestDFSRollback#testRollback is failing
• [HDFS-2861] - HA: checkpointing should verify that the dfs.http.address has been configured to a non-loopback for peer NN
• [HDFS-2859] - LOCAL_ADDRESS_MATCHER.match has NPE when called from DFSUtil.getSuffixIDs when the host is incorrect
• [HDFS-2870] - HA: Remove some INFO level logging accidentally left around
• [HDFS-2742] - HA: observed dataloss in replication stress test
• [HDFS-2845] - SBN should not allow browsing of the file system via web UI
• [HDFS-2853] - HA: NN fails to start if the shared edits dir is marked required
• [HDFS-2824] - HA: failover does not succeed if prior NN died just after creating an edit log segment
• [HDFS-2691] - HA: Tests and fixes for pipeline targets and replica recovery
• [HDFS-2841] - HA: HAAdmin does not work if security is enabled
• [HDFS-2805] - HA: Add a test for a federated cluster with HA NNs
• [HDFS-2809] - Add test to verify that delegation tokens are honored after failover.
• [HDFS-2807] - Service level authorizartion for HAServiceProtocol
• [HDFS-2804] - SBN should not mark blocks under-replicated when exiting safemode
• [HDFS-2688] - HA: write tests for quota tracking and HA
• [HDFS-2820] - Add a simple sanity check for HA config
• [HDFS-2737] - HA: Automatically trigger log rolls periodically on the active NN
• [HDFS-2812] - When becoming active, NN should treat all leases as freshly renewed
• [HDFS-2367] - HA: Enable the configuration of multiple HA cluster addresses
• [HDFS-2592] - HA: Balancer support for HA namenodes
• [HDFS-2795] - HA: Standby NN takes a long time to recover from a dead DN starting up
• [HDFS-2767] - HA: ConfiguredFailoverProxyProvider should support NameNodeProtocol
• [HDFS-2772] - HA: On transition to active, standby should not swallow ELIE
• [HDFS-2747] - HA: entering safe mode after starting SBN can NPE
• [HDFS-2789] - TestHAAdmin.testFailover is failing
• [HDFS-2738] - FSEditLog.selectinputStreams is reading through in-progress streams even when non-in-progress are requested
• [HDFS-2766] - HA: test for case where standby partially reads log and then performs checkpoint

MapReduce

Bug

• [MAPREDUCE-3716] - java.io.File.createTempFile fails in map/reduce tasks
• [MAPREDUCE-3749] - ConcurrentModificationException in counter groups
• [MAPREDUCE-3740] - Mapreduce Trunk compilation fails
• [MAPREDUCE-3742] - "yarn logs" command fails with ClassNotFoundException
• [MAPREDUCE-3736] - Variable substitution depth too large for fs.default.name causes jobs to fail
• [MAPREDUCE-3717] - JobClient test jar has missing files to run all the test programs.
• [MAPREDUCE-3712] - The mapreduce tar does not contain the hadoop-mapreduce-client-jobclient-tests.jar.
• [MAPREDUCE-3714] - Reduce hangs in a corner case
• [MAPREDUCE-3710] - last split generated by FileInputFormat.getSplits may not have the best locality
• [MAPREDUCE-3505] - yarn APPLICATION_CLASSPATH needs to be overridable
• [MAPREDUCE-3681] - capacity scheduler LeafQueues calculate used capacity wrong
• [MAPREDUCE-3646] - Remove redundant URL info from "mapred job" output
• [MAPREDUCE-3702] - internal server error trying access application master via proxy with filter enabled
• [MAPREDUCE-3691] - webservices add support to compress response
• [MAPREDUCE-3705] - ant build fails on 0.23 branch
• [MAPREDUCE-3549] - write api documentation for web service apis for RM, NM, mapreduce app master, and job history server
• [MAPREDUCE-3689] - RM web UI doesn't handle newline in job name
• [MAPREDUCE-3698] - Client cannot talk to the history server in secure mode
• [MAPREDUCE-3582] - Move successfully passing MR1 tests to MR2 maven tree.
• [MAPREDUCE-3684] - LocalDistributedCacheManager does not shut down its thread pool
• [MAPREDUCE-3669] - Getting a lot of PriviledgedActionException / SaslException when running a job
• [MAPREDUCE-2450] - Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
• [MAPREDUCE-3641] - CapacityScheduler should be more conservative assigning off-switch requests
• [MAPREDUCE-3657] - State machine visualize build fails
• [MAPREDUCE-3649] - Job End notification gives an error on calling back.
• [MAPREDUCE-3404] - Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false
• [MAPREDUCE-3532] - When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM
• [MAPREDUCE-3656] - Sort job on 350 scale is consistently failing with latest MRV2 code
• [MAPREDUCE-3596] - Sort benchmark got hang after completion of 99% map phase
• [MAPREDUCE-3664] - HDFS Federation Documentation has incorrect configuration example
• [MAPREDUCE-3512] - Batch jobHistory disk flushes
• [MAPREDUCE-3625] - CapacityScheduler web-ui display of queue's used capacity is broken
• [MAPREDUCE-3652] - org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails
• [MAPREDUCE-3645] - TestJobHistory fails
• [MAPREDUCE-3651] - TestQueueManagerRefresh fails
• [MAPREDUCE-3648] - TestJobConf failing
• [MAPREDUCE-3618] - TaskHeartbeatHandler holds a global lock for all task-updates
• [MAPREDUCE-3380] - Token infrastructure for running clients which are not kerberos authenticated
• [MAPREDUCE-3382] - Network ACLs can prevent AMs to ping the Job-end notification URL
• [MAPREDUCE-3312] - Make MR AM not send a stopContainer w/o corresponding start container
• [MAPREDUCE-3639] - TokenCache likely broken for FileSystems which don't issue delegation tokens
• [MAPREDUCE-3528] - The task timeout check interval should be configurable independent of mapreduce.task.timeout
• [MAPREDUCE-3616] - Thread pool for launching containers in MR AM not expanding as expected
• [MAPREDUCE-3511] - Counters occupy a good part of AM heap
• [MAPREDUCE-3434] - Nightly build broken
• [MAPREDUCE-3436] - JobHistory webapp address should use the host from the jobhistory address

Improvement

• [MAPREDUCE-3692] - yarn-resourcemanager out and log files can get big
• [MAPREDUCE-3553] - Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON
• [MAPREDUCE-3299] - Add AMInfo table to the AM job page

Task

• [MAPREDUCE-3630] - NullPointerException running teragen
• [MAPREDUCE-3251] - Network ACLs can prevent some clients to talk to MR ApplicationMaster