CDH 4 Release Notes

Changes Not In Apache Hadoop 0.23.1

Common

Bug

• [HADOOP-8282] - start-all.sh refers incorrectly start-dfs.sh existence for starting start-yarn.sh
• [HADOOP-8199] - Fix issues in start-all.sh and stop-all.sh
• [HADOOP-8270] - hadoop-daemon.sh stop action should return 0 for an already stopped service
• [HADOOP-8261] - Har file system doesn't deal with FS URIs with a host but no port
• [HADOOP-8257] - Auto-HA: TestZKFailoverControllerStress occasionally fails with Mockito error
• [HADOOP-8249] - invalid hadoop-auth cookies should trigger authentication if info is avail before returning HTTP 401
• [HADOOP-8245] - Fix flakiness in TestZKFailoverController
• [HADOOP-8243] - Security support broken in CLI (manual) failover controller
• [HADOOP-8218] - RPC.closeProxy shouldn't throw error when closing a mock
• [HADOOP-8202] - stopproxy() is not closing the proxies correctly
• [HADOOP-8220] - ZKFailoverController doesn't handle failure to become active correctly
• [HADOOP-8159] - NetworkTopology: getLeaf should check for invalid topologies
• [HADOOP-8238] - NetUtils#getHostNameOfIP blows up if given ip:port string w/o port
• [HADOOP-8208] - Disallow self failover
• [HADOOP-8204] - TestHealthMonitor fails occasionally
• [HADOOP-8197] - Configuration logs WARNs on every use of a deprecated key
• [HADOOP-8191] - SshFenceByTcpPort uses netcat incorrectly
• [HADOOP-8189] - LdapGroupsMapping shouldn't throw away IOException
• [HADOOP-8167] - Configuration deprecation logic breaks backwards compatibility
• [HADOOP-8154] - DNS#getIPs shouldn't silently return the local host IP for bogus interface names
• [HADOOP-7888] - TestFailoverProxy fails intermittently on trunk
• [HADOOP-2430] - [hbase] Master will not shut down if there are no active region servers
• [HADOOP-8050] - Deadlock in metrics
• [HADOOP-7680] - TestHardLink fails on Mac OS X, when gnu stat is in path
• [HADOOP-8057] - hadoop-setup-conf.sh not working because of some extra spaces.
• [HADOOP-8046] - Revert StaticMapping semantics to the existing ones, add DNS mapping diagnostics in progress

Improvement

• [HADOOP-8247] - Auto-HA: add a config to enable auto-HA, which disables manual FC
• [HADOOP-8246] - Auto-HA: automatically scope znode by nameservice ID
• [HADOOP-8077] - HA: fencing method should be able to be configured on a per-NN or per-NS basis
• [HADOOP-8242] - AbstractDelegationTokenIdentifier: add getter methods for owner and realuser
• [HADOOP-8007] - HA: use substitution token for fencing argument
• [HADOOP-8215] - Security support for ZK Failover controller
• [HADOOP-8236] - haadmin should have configurable timeouts for failover commands
• [HADOOP-8149] - cap space usage of default log4j rolling policy
• [HADOOP-8216] - address log4j.properties inconsistencies btw main and template dirs
• [HADOOP-8200] - Remove HADOOP_[JOBTRACKER|TASKTRACKER]_OPTS
• [HADOOP-8212] - Improve ActiveStandbyElector's behavior when session expires
• [HADOOP-8193] - Refactor FailoverController/HAAdmin code to add an abstract class for "target" services
• [HADOOP-8163] - Improve ActiveStandbyElector to provide hooks for fencing old active
• [HADOOP-7557] - Make IPC header be extensible
• [HADOOP-8108] - move method getHostPortString() from NameNode to NetUtils
• [HADOOP-7358] - Improve log levels when exceptions caught in RPC handler
• [HADOOP-7729] - Send back valid HTTP response if user hits IPC port with HTTP GET
• [HADOOP-7717] - Move handling of concurrent client fail-overs to RetryInvocationHandler
• [HADOOP-8141] - Add method to init krb5 cipher suites

New Feature

• [HADOOP-7030] - Add TableMapping topology implementation to read host to rack mapping from a file
• [HADOOP-8206] - Common portion of ZK-based failover controller
• [HADOOP-8210] - Common side of HDFS-3148
• [HADOOP-8211] - Update commons-net version to 3.1
• [HADOOP-7788] - HA: Simple HealthMonitor class to watch an HAService
• [HADOOP-8121] - Active Directory Group Mapping Service
• [HADOOP-7806] - Support binding to sub-interfaces

Task

• [HADOOP-8142] - Update versions from 0.23.2 to 0.23.3

Test

• [HADOOP-8260] - Auto-HA: Replace ClientBaseWithFixes with our own modified copy of the class
• [HADOOP-8228] - Auto HA: Refactor tests and add stress tests
• [HADOOP-8157] - TestRPCCallBenchmark#testBenchmarkWithWritable fails with RTE

HDFS

Bug

• [HDFS-3255] - HA DFS returns wrong token service
• [HDFS-3305] - GetImageServlet should consider SBN a valid requestor in a secure HA setup
• [HDFS-3165] - HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
• [HDFS-3284] - bootstrapStandby fails in secure cluster
• [HDFS-3280] - DFSOutputStream.sync should not be synchronized
• [HDFS-2765] - TestNameEditsConfigs is incorrectly swallowing IOE
• [HDFS-2799] - Trim fs.checkpoint.dir values
• [HDFS-3256] - HDFS considers blocks under-replicated if topology script is configured with only 1 rack
• [HDFS-3260] - TestDatanodeRegistration should set minimum DN version in addition to minimum NN version
• [HDFS-3202] - NamespaceInfo PB translation drops build version
• [HDFS-3261] - TestHASafeMode fails on HDFS-3042 branch
• [HDFS-3037] - TestMulitipleNNDataBlockScanner#testBlockScannerAfterRestart is racy
• [HDFS-3234] - Accidentally left log message in GetConf after HDFS-3226
• [HDFS-2696] - Fix the fuse-fds build
• [HDFS-3236] - NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
• [HDFS-3214] - InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery
• [HDFS-3208] - Bogus entries in hosts files are incorrectly displayed in the report
• [HDFS-3210] - JsonUtil#toJsonMap for for a DatanodeInfo should use "ipAddr" instead of "name"
• [HDFS-3199] - TestValidateConfigurationSettings is failing
• [HDFS-3156] - TestDFSHAAdmin is failing post HADOOP-8202
• [HDFS-2976] - Remove unnecessary method (tokenRefetchNeeded) in DFSClient
• [HDFS-3174] - Fix assert in TestPendingDataNodeMessages
• [HDFS-3100] - failed to append data
• [HDFS-3101] - cannot read empty file using webhdfs
• [HDFS-3005] - ConcurrentModificationException in FSDataset$FSVolume.getDfsUsed(..)
• [HDFS-2995] - start-dfs.sh should only start the 2NN for namenodes with dfs.namenode.secondary.http-address configured
• [HDFS-3070] - HDFS balancer doesn't ensure that hdfs-site.xml is loaded
• [HDFS-3160] - httpfs should exec catalina instead of forking it
• [HDFS-3132] - Findbugs warning on HDFS trunk
• [HDFS-3083] - Cannot run an MR job with HA and security enabled when second-listed NN active
• [HDFS-3062] - Fail to submit mapred job on a secured-HA-HDFS: logic URI cannot be picked up by job submission.
• [HDFS-3099] - SecondaryNameNode does not properly initialize metrics system
• [HDFS-3057] - httpfs and hdfs launcher scripts should honor CATALINA_HOME and HADOOP_LIBEXEC_DIR
• [HDFS-3093] - TestAllowFormat is trying to be interactive
• [HDFS-1624] - Append: The condition is incorrect for checking whether the last block is full
• [HDFS-2878] - TestBlockRecovery does not compile
• [HDFS-2764] - TestBackupNode is racy
• [HDFS-2285] - BackupNode should reject requests trying to modify namespace
• [HDFS-1765] - Block Replication should respect under-replication block priority
• [HDFS-3038] - Add FSEditLog.metrics to findbugs exclude list
• [HDFS-3020] - Auto-logSync based on edit log buffer size broken
• [HDFS-2991] - failure to load edits: ClassCastException
• [HDFS-3008] - Negative caching of local addrs doesn't work
• [HDFS-2725] - hdfs script usage information is missing the information about "dfs" command
• [HDFS-2968] - Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery

Improvement

• [HDFS-3263] - HttpFS should read HDFS config from Hadoop site.xml files
• [HDFS-2708] - Stats for the # of blocks per DN
• [HDFS-3259] - NameNode#initializeSharedEdits should populate shared edits dir with edit log segments
• [HDFS-2983] - Relax the build version check to permit rolling upgrades within a release
• [HDFS-3247] - Improve bootstrapStandby behavior when original NN is not active
• [HDFS-3240] - Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
• [HDFS-3223] - Auto HA: add zkfc to hadoop-daemon.sh script
• [HDFS-3226] - Allow GetConf tool to print arbitrary keys
• [HDFS-3050] - rework OEV to share more code with the NameNode
• [HDFS-3084] - FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
• [HDFS-3130] - Move FSDataset implemenation to a package
• [HDFS-3144] - Refactor DatanodeID#getName by use
• [HDFS-3171] - The DatanodeID "name" field is overloaded
• [HDFS-3164] - Move DatanodeInfo#hostName to DatanodeID
• [HDFS-3138] - Move DatanodeInfo#ipcPort to DatanodeID
• [HDFS-3155] - Clean up FSDataset implemenation related code.
• [HDFS-3139] - Minor Datanode logging improvement
• [HDFS-3088] - Move FSDatasetInterface inner classes to a package
• [HDFS-3082] - Clean up FSDatasetInterface
• [HDFS-3056] - Add an interface for DataBlockScanner logging
• [HDFS-3120] - Enable hsync and hflush by default
• [HDFS-3172] - dfs.upgrade.permission is dead code
• [HDFS-3137] - Bump LAST_UPGRADABLE_LAYOUT_VERSION to -16
• [HDFS-3066] - cap space usage of default log4j rolling policy (hdfs specific changes)
• [HDFS-3044] - fsck move should be non-destructive by default
• [HDFS-2413] - Add public APIs for safemode
• [HDFS-3158] - LiveNodes member of NameNodeMXBean should list non-DFS used space and capacity per DN
• [HDFS-3071] - haadmin failover command does not provide enough detail for when target NN is not ready to be active
• [HDFS-2303] - Unbundle jsvc
• [HDFS-3036] - Remove unused method DFSUtil#isDefaultNamenodeAddress
• [HDFS-3014] - FSEditLogOp and its subclasses should have toString() method
• [HDFS-3003] - Remove getHostPortString() from NameNode, replace it with NetUtils.getHostPortString()
• [HDFS-208] - name node should warn if only one dir is listed in dfs.name.dir
• [HDFS-2410] - Further clean up hard-coded configuration keys
• [HDFS-2507] - HA: Allow saveNamespace operations to be canceled
• [HDFS-2477] - Optimize computing the diff between a block report and the namenode state.
• [HDFS-1580] - Add interface for generic Write Ahead Logging mechanisms
• [HDFS-2476] - More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks
• [HDFS-2495] - Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lock
• [HDFS-2334] - Add Closeable to JournalManager
• [HDFS-2188] - HDFS-1580: Make FSEditLog create its journals from a list of URIs rather than NNStorage
• [HDFS-2158] - Add JournalSet to manage the set of journals.
• [HDFS-2992] - Edit log failure trace should include transaction ID of error
• [HDFS-3024] - Improve performance of stringification in addStoredBlock
• [HDFS-2985] - Improve logging when replicas are marked as corrupt

New Feature

• [HDFS-3159] - Document NN auto-failover setup and configuration
• [HDFS-3200] - Auto-HA: Scope all ZKFC configs by nameservice
• [HDFS-3102] - Add CLI tool to initialize the shared-edits dir
• [HDFS-3000] - Add a public API for setting quotas
• [HDFS-2185] - HA: HDFS portion of ZK-based FailoverController
• [HDFS-3148] - The client should be able to use multiple local interfaces for data transfer
• [HDFS-3086] - Change Datanode not to send storage list in registration - it will be sent in block report
• [HDFS-3105] - Add DatanodeStorage information to block recovery
• [HDFS-2941] - Add an administrative command to download a copy of the fsimage from the NN
• [HDFS-2731] - HA: Autopopulate standby name dirs if they're empty
• [HDFS-1623] - High Availability Framework for HDFS NN
• [HDFS-2978] - The NameNode should expose name dir statuses via JMX

Task

• [HDFS-3111] - Missing license headers in trunk

Test

• [HDFS-3129] - NetworkTopology: add test that getLeaf should check for invalid topologies
• [HDFS-3060] - Bump TestDistributedUpgrade#testDistributedUpgrade timeout

MapReduce

Bug

• [MAPREDUCE-4140] - mapreduce classes incorrectly importing "clover.org.apache.*" classes
• [MAPREDUCE-4105] - Yarn RackResolver ignores rack configurations
• [MAPREDUCE-3999] - Tracking link gives an error if the AppMaster hasn't started yet
• [MAPREDUCE-4097] - tools testcases fail because missing mrapp-generated-classpath file in classpath
• [MAPREDUCE-4043] - Secret keys set in Credentials are not seen by tasks
• [MAPREDUCE-4098] - TestMRApps testSetClasspath fails
• [MAPREDUCE-4061] - RM only has 1 AM launcher thread
• [MAPREDUCE-4034] - Unable to view task logs on history server with mapreduce.job.acl-view-job=*
• [MAPREDUCE-3353] - Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes
• [MAPREDUCE-4025] - AM can crash if task attempt reports bogus progress value
• [MAPREDUCE-4091] - tools testcases failing because of MAPREDUCE-4082
• [MAPREDUCE-3916] - various issues with running yarn proxyserver
• [MAPREDUCE-4005] - AM container logs URL is broken for completed apps when log aggregation is enabled
• [MAPREDUCE-4006] - history server container log web UI sometimes combines stderr/stdout/syslog contents together
• [MAPREDUCE-3992] - Reduce fetcher doesn't verify HTTP status code of response
• [MAPREDUCE-3431] - NPE in Resource Manager shutdown
• [MAPREDUCE-4010] - TestWritableJobConf fails on trunk
• [MAPREDUCE-4007] - JobClient getJob(JobID) should return NULL if the job does not exist (for backwards compatibility)
• [MAPREDUCE-3982] - TestEmptyJob fails with FileNotFound
• [MAPREDUCE-3348] - mapred job -status fails to give info even if the job is present in History
• [MAPREDUCE-3954] - Clean up passing HEAPSIZE to yarn and mapred commands.
• [MAPREDUCE-3009] - RM UI -> Applications -> Application(Job History) -> Map Tasks -> Task ID -> Node link is not working
• [MAPREDUCE-3614] - finalState UNDEFINED if AM is killed by hand
• [MAPREDUCE-3792] - job -list displays only the jobs submitted by a particular user
• [MAPREDUCE-3931] - MR tasks failing due to changing timestamps on Resources to download
• [MAPREDUCE-2855] - ResourceBundle lookup during counter name resolution takes a lot of time
• [MAPREDUCE-3686] - history server web ui - job counter values for map/reduce not shown properly
• [MAPREDUCE-3910] - user not allowed to submit jobs even though queue -showacls shows it allows
• [MAPREDUCE-2793] - [MR-279] Maintain consistency in naming appIDs, jobIDs and attemptIDs
• [MAPREDUCE-3866] - bin/yarn prints the command line unnecessarily
• [MAPREDUCE-3634] - All daemons should crash instead of hanging around when their EventHandlers get exceptions

Improvement

• [MAPREDUCE-4103] - Fix HA docs for changes to shell command fencer args
• [MAPREDUCE-3989] - cap space usage of default log4j rolling policy (mr specific changes)
• [MAPREDUCE-3922] - Fix the potential problem compiling 32 bit binaries on a x86_64 host.
• [MAPREDUCE-3901] - lazy load JobHistory Task and TaskAttempt details
• [MAPREDUCE-3909] - javadoc the Service interfaces
• [MAPREDUCE-3730] - Allow restarted NM to rejoin cluster before RM expires it

Test

• [MAPREDUCE-3798] - TestJobCleanup testCustomCleanup is failing
• [MAPREDUCE-3877] - Add a test to formalise the current state transitions of the yarn lifecycle