CDH 5.1.5 Release Notes

The following lists all Apache Hadoop Jiras included in CDH 5.1.5 that are not included in the Apache Hadoop base version 2.3.0. The hadoop-2.3.0-cdh5.1.5.CHANGES.txt file lists all changes included in CDH 5.1.5. The patch for each change can be found in the cloudera/patches directory in the release tarball.

Changes Not In Apache Hadoop 2.3.0

MapReduce

Bug

[MAPREDUCE-6012] - DBInputSplit creates invalid ranges on Oracle
[MAPREDUCE-5968] - Work directory is not deleted when downloadCacheObject throws IOException
[MAPREDUCE-5966] - MR1 FairScheduler use of custom weight adjuster is not thread safe for comparisons
[MAPREDUCE-5979] - FairScheduler: zero weight can cause sort failures
[MAPREDUCE-5707] - JobClient does not allow setting RPC timeout for communications with JT/RM
[MAPREDUCE-5375] - Delegation Token renewal exception in jobtracker logs
[MAPREDUCE-5875] - Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[MAPREDUCE-5813] - YarnChild does not load job.xml with mapreduce.job.classloader=true
[MAPREDUCE-5814] - fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
[MAPREDUCE-5888] - Failed job leaves hung AM after it unregisters
[MAPREDUCE-5835] - Killing Task might cause the job to go to ERROR state
[MAPREDUCE-5841] - uber job doesn't terminate on getting mapred job kill
[MAPREDUCE-5795] - Job should be marked as Failed if it is recovered from commit.
[MAPREDUCE-5746] - Job diagnostics can implicate wrong task for a failed job
[MAPREDUCE-5769] - Unregistration to RM should not be called if AM is crashed before registering with RM
[MAPREDUCE-5751] - MR app master fails to start in some cases if mapreduce.job.classloader is true
[MAPREDUCE-5757] - ConcurrentModificationException in JobControl.toList
[MAPREDUCE-5670] - CombineFileRecordReader should report progress when moving to the next file
[MAPREDUCE-5770] - Redirection from AM-URL is broken with HTTPS_ONLY policy
[MAPREDUCE-5862] - Line records longer than 2x split size aren't handled correctly
[MAPREDUCE-4052] - Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[MAPREDUCE-5028] - Maps fail when io.sort.mb is set to high value
[MAPREDUCE-5768] - TestMRJobs.testContainerRollingLog fails on trunk
[MAPREDUCE-5699] - Allow setting tags on MR jobs
[MAPREDUCE-2779] - JobSplitWriter.java can't handle large job.split file
[MAPREDUCE-2324] - Job should fail if a reduce task can't be scheduled anywhere
[MAPREDUCE-5702] - TaskLogServlet#printTaskLog has spurious HTML closing tags
[MAPREDUCE-4383] - HadoopPipes.cc needs to include unistd.h
[MAPREDUCE-5206] - JT can show the same job multiple times in Retired Jobs section
[MAPREDUCE-5508] - JobTracker memory leak caused by unreleased FileSystem objects in JobInProgress#cleanupJob
[MAPREDUCE-5066] - JobTracker should set a timeout when calling into job.end.notification.url
[MAPREDUCE-5198] - Race condition in cleanup during task tracker renint with LinuxTaskController
[MAPREDUCE-4914] - TestMiniMRDFSSort fails with openJDK7
[MAPREDUCE-5450] - Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[MAPREDUCE-4366] - mapred metrics shows negative count of waiting maps and reduces
[MAPREDUCE-5351] - JobTracker memory leak caused by CleanupQueue reopening FileSystem
[MAPREDUCE-5364] - Deadlock between RenewalTimerTask methods cancel() and run()
[MAPREDUCE-5038] - old API CombineFileInputFormat missing fixes that are in new API
[MAPREDUCE-2359] - Distributed cache doesn't use non-default FileSystems correctly
[MAPREDUCE-5250] - Searching for ';' in JobTracker History throws ArrayOutOfBoundException
[MAPREDUCE-4970] - Child tasks (try to) create security audit log files
[MAPREDUCE-323] - Improve the way job history files are managed
[MAPREDUCE-4576] - Large dist cache can block tasktracker heartbeat
[MAPREDUCE-3824] - Distributed caches are not removed properly
[MAPREDUCE-2409] - Distributed Cache does not differentiate between file /archive for files with the same path
[MAPREDUCE-5218] - Annotate (comment) internal classes as Private
[MAPREDUCE-5193] - A few MR tests use block sizes which are smaller than the default minimum block size
[MAPREDUCE-5154] - staging directory deletion fails because delegation tokens have been cancelled
[MAPREDUCE-5133] - TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race
[MAPREDUCE-2817] - MiniRMCluster hardcodes 'mapred.local.dir' configuration to 'build/test/mapred/local'
[MAPREDUCE-5008] - Merger progress miscounts with respect to EOF_MARKER
[MAPREDUCE-5073] - TestJobStatusPersistency.testPersistency fails on JDK7
[MAPREDUCE-5072] - TestDelegationTokenRenewal.testDTRenewal fails in MR1 on jdk7
[MAPREDUCE-5070] - TestClusterStatus.testClusterMetrics fails on JDK7
[MAPREDUCE-2187] - map tasks timeout during sorting
[MAPREDUCE-5047] - keep.failed.task.files=true causes job failure on secure clusters
[MAPREDUCE-4843] - When using DefaultTaskController, JobLocalizer not thread safe
[MAPREDUCE-4643] - Make job-history cleanup-period configurable
[MAPREDUCE-4888] - NLineInputFormat drops data in 1.1 and beyond
[MAPREDUCE-4962] - jobdetails.jsp uses display name instead of real name to get counters
[MAPREDUCE-4963] - StatisticsCollector improperly keeps track of "Last Day" and "Last Hour" statistics for new TaskTrackers
[MAPREDUCE-2264] - Job status exceeds 100% in some cases
[MAPREDUCE-4929] - mapreduce.task.timeout is ignored
[MAPREDUCE-4562] - Support for "FileSystemCounter" legacy counter group name for compatibility reasons is creating incorrect counter name
[MAPREDUCE-4923] - Add toString method to TaggedInputSplit
[MAPREDUCE-4315] - jobhistory.jsp throws 500 when a .txt file is found in /done
[MAPREDUCE-4925] - The pentomino option parser may be buggy
[MAPREDUCE-4924] - flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR
[MAPREDUCE-3475] - JT can't renew its own tokens
[MAPREDUCE-4933] - MR1 final merge asks for length of file it just wrote before flushing it
[MAPREDUCE-4850] - Job recovery may fail if staging directory has been deleted
[MAPREDUCE-4278] - cannot run two local jobs in parallel from the same gateway.
[MAPREDUCE-2217] - The expire launching task should cover the UNASSIGNED task
[MAPREDUCE-1238] - mapred metrics shows negative count of waiting maps and reduces
[MAPREDUCE-4860] - DelegationTokenRenewal attempts to renew token even after a job is removed
[MAPREDUCE-4859] - TestRecoveryManager fails on branch-1
[MAPREDUCE-4778] - Fair scheduler event log is only written if directory exists on HDFS
[MAPREDUCE-4800] - Cleanup o.a.h.mapred.MapTaskStatus - remove unused code
[MAPREDUCE-1905] - Context.setStatus() and progress() api are ignored
[MAPREDUCE-4629] - Remove JobHistory.DEBUG_MODE
[MAPREDUCE-4765] - Restarting the JobTracker programmatically can cause DelegationTokenRenewal to throw an exception
[MAPREDUCE-1806] - CombineFileInputFormat does not work with paths not on default FS
[MAPREDUCE-4451] - fairscheduler fail to init job with kerberos authentication configured
[MAPREDUCE-4706] - FairScheduler#dump(): Computing of # running maps and reduces is commented out
[MAPREDUCE-4652] - ValueAggregatorJob sets the wrong job jar
[MAPREDUCE-2185] - Infinite loop at creating splits using CombineFileInputFormat
[MAPREDUCE-4470] - Fix TestCombineFileInputFormat.testForEmptyFile
[MAPREDUCE-4610] - Support deprecated mapreduce.job.counters.limit property in MR2
[MAPREDUCE-2764] - Fix renewal of dfs delegation tokens
[MAPREDUCE-2452] - Delegation token cancellation shouldn't hold global JobTracker lock
[MAPREDUCE-4595] - TestLostTracker failing - possibly due to a race in JobHistory.JobHistoryFilesManager#run()
[MAPREDUCE-2374] - "Text File Busy" errors launching MR tasks
[MAPREDUCE-4385] - FairScheduler.maxTasksToAssign() should check for fairscheduler.assignmultiple.maps < TaskTracker.availableSlots
[MAPREDUCE-3993] - Graceful handling of codec errors during decompression
[MAPREDUCE-1505] - Cluster class should create the rpc client only when needed
[MAPREDUCE-4463] - JobTracker recovery fails with HDFS permission issue
[MAPREDUCE-4399] - Fix performance regression in shuffle
[MAPREDUCE-4195] - With invalid queueName request param, jobqueue_details.jsp shows NPE
[MAPREDUCE-3674] - If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[MAPREDUCE-2839] - MR Jobs fail on a secure cluster with viewfs
[MAPREDUCE-4226] - ConcurrentModificationException in FileSystemCounterGroup
[MAPREDUCE-4129] - Lots of unneeded counters log messages
[MAPREDUCE-3827] - Counters aggregation slowed down significantly after MAPREDUCE-3749
[MAPREDUCE-3809] - Tasks may take upto 3 seconds to exit after completion
[MAPREDUCE-2450] - Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[MAPREDUCE-4095] - TestJobInProgress#testLocality uses a bogus topology
[MAPREDUCE-4012] - Hadoop Job setup error leaves no useful info to users (when LinuxTaskController is used)
[MAPREDUCE-3583] - ProcfsBasedProcessTree#constructProcessInfo() may throw NumberFormatException
[MAPREDUCE-1740] - NPE in getMatchingLevelForNodes when node locations are variable depth
[MAPREDUCE-4036] - Streaming TestUlimit fails on CentOS 6
[MAPREDUCE-3727] - jobtoken location property in jobconf refers to wrong jobtoken file
[MAPREDUCE-4010] - TestWritableJobConf fails on trunk
[MAPREDUCE-3789] - CapacityTaskScheduler may perform unnecessary reservations in heterogenous tracker environments
[MAPREDUCE-1109] - ConcurrentModificationException in jobtracker.jsp
[MAPREDUCE-3997] - jobhistory.jsp cuts off the job name at the first underscore of the job name
[MAPREDUCE-3697] - Hadoop Counters API limits Oozie's working across different hadoop versions
[MAPREDUCE-3639] - TokenCache likely broken for FileSystems which don't issue delegation tokens
[MAPREDUCE-3749] - ConcurrentModificationException in counter groups
[MAPREDUCE-3563] - LocalJobRunner doesn't handle Jobs using o.a.h.mapreduce.OutputCommitter
[MAPREDUCE-2531] - org.apache.hadoop.mapred.jobcontrol.getAssignedJobID throw class cast exception
[MAPREDUCE-3542] - Support "FileSystemCounter" legacy counter group name for compatibility
[MAPREDUCE-3433] - Finding counters by legacy group name returns empty counters
[MAPREDUCE-895] - FileSystem::ListStatus will now throw FileNotFoundException, MapRed needs updated

Improvement

[MAPREDUCE-5777] - Support utf-8 text with BOM (byte order marker)
[MAPREDUCE-5754] - Preserve Job diagnostics in history
[MAPREDUCE-5014] - Extending DistCp through a custom CopyListing is not possible
[MAPREDUCE-5773] - Provide dedicated MRAppMaster syslog length limit
[MAPREDUCE-5732] - Report proper queue when job has been automatically placed
[MAPREDUCE-5651] - Backport Fair Scheduler queue placement policies to branch-1
[MAPREDUCE-3310] - Custom grouping comparator cannot be set for Combiners
[MAPREDUCE-3169] - Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[MAPREDUCE-5609] - Add debug log message when sending job end notification
[MAPREDUCE-5367] - Local jobs all use same local working directory
[MAPREDUCE-5379] - Include token tracking ids in jobconf
[MAPREDUCE-2494] - Make the distributed cache delete entires using LRU priority
[MAPREDUCE-2495] - The distributed cache cleanup thread has no monitoring to check to see if it has died for some reason
[MAPREDUCE-1568] - TrackerDistributedCacheManager should clean up cache in a background thread
[MAPREDUCE-4777] - In TestIFile, testIFileReaderWithCodec relies on testIFileWriterWithCodec
[MAPREDUCE-4977] - Documentation for pluggable shuffle and pluggable sort
[MAPREDUCE-2931] - CLONE - LocalJobRunner should support parallel mapper execution
[MAPREDUCE-2492] - [MAPREDUCE] The new MapReduce API should make available task's progress to the task
[MAPREDUCE-4907] - TrackerDistributedCacheManager issues too many getFileStatus calls
[MAPREDUCE-4464] - Reduce tasks failing with NullPointerException in ConcurrentHashMap.get()
[MAPREDUCE-3607] - Port missing new API mapreduce lib classes to 1.x
[MAPREDUCE-4845] - ClusterStatus.getMaxMemory() and getUsedMemory() exist in MR1 but not MR2
[MAPREDUCE-4556] - FairScheduler: PoolSchedulable#updateDemand() has potential redundant computation
[MAPREDUCE-2786] - TestDFSIO should also test compression reading/writing from command-line.
[MAPREDUCE-4408] - allow jobs to set a JAR that is in the distributed cached
[MAPREDUCE-4511] - Add IFile readahead
[MAPREDUCE-4487] - Reduce job latency by removing hardcoded sleep statements
[MAPREDUCE-336] - The logging level of the tasks should be configurable by the job
[MAPREDUCE-2456] - Show the reducer taskid and map/reduce tasktrackers for "Failed fetch notification #_ for task attempt..." log messages
[MAPREDUCE-3289] - Make use of fadvise in the NM's shuffle handler
[MAPREDUCE-1221] - Kill tasks on a node if the free physical memory on that machine falls below a configured threshold
[MAPREDUCE-157] - Job History log file format is not friendly for external tools.
[MAPREDUCE-954] - The new interface's Context objects should be interfaces
[MAPREDUCE-368] - Change org.apache.hadoop.mapred.jobcontrol to use new api
[MAPREDUCE-1943] - Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
[MAPREDUCE-901] - Move Framework Counters into a TaskMetric structure
[MAPREDUCE-2455] - Remove deprecated JobTracker.State in favour of JobTrackerStatus

New Feature

[MAPREDUCE-4049] - plugin for generic shuffle service
[MAPREDUCE-4808] - Refactor MapOutput and MergeManager to facilitate reuse by Shuffle implementations
[MAPREDUCE-4807] - Allow MapOutputBuffer to be pluggable
[MAPREDUCE-4824] - Provide a mechanism for jobs to indicate they should not be recovered on restart
[MAPREDUCE-3678] - The Map tasks logs should have the value of input split it processed
[MAPREDUCE-4355] - Add RunningJob.getJobStatus()
[MAPREDUCE-4417] - add support for encrypted shuffle
[MAPREDUCE-3837] - Job tracker is not able to recover job in case of crash and after that no user can submit job.

Task

[MAPREDUCE-4246] - Failure in deleting user directories in Secure hadoop

Common

Bug

[HADOOP-11243] - SSLFactory shouldn't allow SSLv3
[HADOOP-11156] - DelegateToFileSystem should implement getFsStatus(final Path f).
[HADOOP-11035] - distcp on mr1(branch-1) fails with NPE using a short relative source path.
[HADOOP-10820] - Throw an exception in GenericOptionsParser when passed an empty Path
[HADOOP-10876] - The constructor of Path should not take an empty URL as a parameter
[HADOOP-10701] - NFS should not validate the access premission only based on the user's primary group
[HADOOP-10711] - Cleanup some extra dependencies from hadoop-auth
[HADOOP-10394] - TestAuthenticationFilter is flaky
[HADOOP-10301] - AuthenticationFilter should return Forbidden for failed authentication
[HADOOP-10158] - SPNEGO should work with multiple interfaces/SPNs.
[HADOOP-10568] - Add s3 server-side encryption
[HADOOP-10353] - FsUrlStreamHandlerFactory is not thread safe
[HADOOP-10332] - HttpServer's jetty audit log always logs 200 OK
[HADOOP-10425] - Incompatible behavior of LocalFileSystem:getContentSummary
[HADOOP-10414] - Incorrect property name for RefreshUserMappingProtocol in hadoop-policy.xml
[HADOOP-10475] - ConcurrentModificationException in AbstractDelegationTokenSelector.selectToken()
[HADOOP-8826] - Docs still refer to 0.20.205 as stable line
[HADOOP-10531] - hadoop-config.sh - bug in --hosts argument
[HADOOP-10500] - TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy user configuration.
[HADOOP-10658] - SSLFactory expects truststores being configured
[HADOOP-10609] - .gitignore should ignore .orig and .rej files
[HADOOP-10562] - Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[HADOOP-10638] - Updating hadoop-daemon.sh to work as expected when nfs is started as a privileged user.
[HADOOP-10639] - FileBasedKeyStoresFactory initialization is not using default for SSL_REQUIRE_CLIENT_CERT_KEY
[HADOOP-10612] - NFS failed to refresh the user group id mapping table
[HADOOP-10489] - UserGroupInformation#getTokens and UserGroupInformation#addToken can lead to ConcurrentModificationException
[HADOOP-10401] - ShellBasedUnixGroupsMapping#getGroups does not always return primary group first
[HADOOP-10442] - Group look-up can cause segmentation fault when certain JNI-based mapping module is used.
[HADOOP-10456] - Bug in Configuration.java exposed by Spark (ConcurrentModificationException)
[HADOOP-10459] - distcp V2 doesn't preserve root dir's attributes when -p is specified
[HADOOP-10418] - SaslRpcClient should not assume that remote principals are in the default_realm
[HADOOP-10378] - Typo in help printed by hdfs dfs -help
[HADOOP-10314] - The ls command help still shows outdated 0.16 format.
[HADOOP-10368] - InputStream is not closed in VersionInfo ctor
[HADOOP-10355] - TestLoadGenerator#testLoadGenerator fails
[HADOOP-10328] - loadGenerator exit code is not reliable
[HADOOP-10249] - LdapGroupsMapping should trim ldap password read from file
[HADOOP-10330] - TestFrameDecoder fails if it cannot bind port 12345
[HADOOP-10320] - Javadoc in InterfaceStability.java lacks final </ul>
[HADOOP-10070] - RPC client doesn't use per-connection conf to determine server's expected Kerberos principal name
[HADOOP-10346] - Deadlock while logging tokens
[HADOOP-10338] - Cannot get the FileStatus of the root inode from the new Globber
[HADOOP-10326] - M/R jobs can not access S3 if Kerberos is enabled
[HADOOP-8912] - adding .gitattributes file to prevent CRLF and LF mismatches for source and text files
[HADOOP-8329] - Build fails with Java 7
[HADOOP-8552] - Conflict: Same security.log.file for multiple users.
[HADOOP-7967] - Need generalized multi-token filesystem support
[HADOOP-8781] - hadoop-config.sh should add JAVA_LIBRARY_PATH to LD_LIBRARY_PATH
[HADOOP-7644] - Fix the delegation token tests to use the new style renewers
[HADOOP-8466] - hadoop-client POM incorrectly excludes avro
[HADOOP-8261] - Har file system doesn't deal with FS URIs with a host but no port

Improvement

[HADOOP-10379] - Protect authentication cookies with the HttpOnly and Secure flags
[HADOOP-10322] - Add ability to read principal names from a keytab
[HADOOP-10211] - Enable RPC protocol to negotiate SASL-QOP values between clients and servers
[HADOOP-10345] - Sanitize the the inputs (groups and hosts) for the proxyuser configuration
[HADOOP-10614] - CBZip2InputStream is not threadsafe
[HADOOP-10572] - Example NFS mount command must pass noacl as it isn't supported by the server yet
[HADOOP-10503] - Move junit up to v 4.11
[HADOOP-10556] - Add toLowerCase support to auth_to_local rules for service name
[HADOOP-9454] - Support multipart uploads for s3native
[HADOOP-10278] - Refactor to make CallQueue pluggable
[HADOOP-10348] - Deprecate hadoop.ssl.configuration in branch-2, and remove it in trunk
[HADOOP-10343] - Change info to debug log in LossyRetryInvocationHandler
[HADOOP-10295] - Allow distcp to automatically identify the checksum type of source files and use it for the target
[HADOOP-10139] - Update and improve the Single Cluster Setup document
[HADOOP-10333] - Fix grammatical error in overview.html document
[HADOOP-9756] - Additional cleanup RPC code
[HADOOP-8700] - Move the checksum type constants to an enum
[HADOOP-8968] - Add a flag to completely disable the worker version check
[HADOOP-8209] - Add option to relax build-version check for branch-1

New Feature

[HADOOP-10498] - Add support for proxy server
[HADOOP-10399] - FileContext API for ACLs.
[HADOOP-8581] - add support for HTTPS to the web UIs

Task

[HADOOP-4687] - Split map/reduce into sub-project

Test

[HADOOP-3679] - calls to junit Assert::assertEquals invert arguments, causing misleading error messages, other minor improvements.
[HADOOP-9051] - ant test will build failed for trying to delete a file

YARN

Bug

[YARN-570] - Time strings are formated in different timezone
[YARN-2251] - Avoid negative elapsed time in JHS/MRAM web UI and services
[YARN-2588] - Standby RM does not transitionToActive if previous transitionToActive is failed with ZK exception.
[YARN-2273] - NPE in ContinuousScheduling thread when we lose a node
[YARN-2608] - FairScheduler: Potential deadlocks in loading alloc files and clock access
[YARN-1458] - FairScheduler: Zero weight can lead to livelock
[YARN-2313] - Livelock can occur in FairScheduler when there are lots of running apps
[YARN-2359] - Application hangs when it fails to launch AM container
[YARN-2192] - TestRMHA fails when run with a mix of Schedulers
[YARN-2109] - Fix TestRM to work with both schedulers
[YARN-2204] - TestAMRestart#testAMRestartWithExistingContainers assumes CapacityScheduler
[YARN-2187] - FairScheduler: Disable max-AM-share check by default
[YARN-2155] - FairScheduler: Incorrect threshold check for preemption
[YARN-2132] - ZKRMStateStore.ZKAction#runWithRetries doesn't log the exception it encounters
[YARN-2128] - FairScheduler: Incorrect calculation of amResource usage
[YARN-2122] - In AllocationFileLoaderService, the reloadThread should be created in init() and started in start()
[YARN-1913] - With Fair Scheduler, cluster can logjam when all resources are consumed by AMs
[YARN-1550] - NPE in FairSchedulerAppsBlock#render
[YARN-2054] - Better defaults for YARN ZK configs for retries and retry-inteval when HA is enabled
[YARN-1929] - DeadLock in RM when automatic failover is enabled.
[YARN-1934] - Potential NPE in ZKRMStateStore caused by handling Disconnected event from ZK.
[YARN-1867] - NPE while fetching apps via the REST API
[YARN-1866] - YARN RM fails to load state store with delegation token parsing error
[YARN-1852] - Application recovery throws InvalidStateTransitonException for FAILED and KILLED jobs
[YARN-1849] - NPE in ResourceTrackerService#registerNodeManager for UAM
[YARN-1136] - Replace junit.framework.Assert with org.junit.Assert
[YARN-1590] - _HOST doesn't expand properly for RM, NM, ProxyServer and JHS
[YARN-2010] - RM can't transition to active if it can't recover an app attempt
[YARN-1837] - TestMoveApplication.testMoveRejectedByScheduler randomly fails
[YARN-1883] - TestRMAdminService fails due to inconsistent entries in UserGroups
[YARN-1490] - RM should optionally not kill all containers when an ApplicationMaster exits
[YARN-1566] - Change distributed-shell to retain containers from previous AppAttempt
[YARN-2073] - Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free
[YARN-1692] - ConcurrentModificationException in fair scheduler AppSchedulable
[YARN-1553] - Do not use HttpConfig.isSecure() in YARN
[YARN-2016] - Yarn getApplicationRequest start time range is not honored
[YARN-2036] - Document yarn.resourcemanager.hostname in ClusterSetup
[YARN-1932] - Javascript injection on the job status page
[YARN-1924] - STATE_STORE_OP_FAILED happens when ZKRMStateStore tries to update app(attempt) before storing it
[YARN-1824] - Make Windows client work with Linux/Unix cluster
[YARN-1784] - TestContainerAllocation assumes CapacityScheduler
[YARN-1846] - TestRM#testNMTokenSentForNormalContainer assumes CapacityScheduler
[YARN-1839] - Capacity scheduler preempts an AM out. AM attempt 2 fails to launch task container with SecretManager$InvalidToken: No NMToken sent
[YARN-1821] - NPE on registerNodeManager if the request has containers for UnmanagedAMs
[YARN-1793] - yarn application -kill doesn't kill UnmanagedAMs
[YARN-1790] - Fair Scheduler UI not showing apps table
[YARN-1774] - FS: Submitting to non-leaf queue throws NPE
[YARN-1785] - FairScheduler treats app lookup failures as ERRORs
[YARN-1758] - MiniYARNCluster broken post YARN-1666
[YARN-1398] - Deadlock in capacity scheduler leaf queue and parent queue for getQueueInfo and completedContainer call
[YARN-713] - ResourceManager can exit unexpectedly if DNS is unavailable
[YARN-1417] - RM may issue expired container tokens to AM while issuing new containers.
[YARN-1748] - hadoop-yarn-server-tests packages core-site.xml breaking downstream tests
[YARN-1760] - TestRMAdminService assumes CapacityScheduler
[YARN-1528] - Allow setting auth for ZK connections
[YARN-1429] - *nix: Allow a way for users to augment classpath of YARN daemons
[YARN-1718] - Fix a couple isTerminals in Fair Scheduler queue placement rules
[YARN-1724] - Race condition in Fair Scheduler when continuous scheduling is turned on
[YARN-1721] - When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
[YARN-1678] - Fair scheduler gabs incessantly about reservations
[YARN-1689] - RMAppAttempt is not killed when RMApp is at ACCEPTED
[YARN-1166] - YARN 'appsFailed' metric should be of type 'counter'
[YARN-1041] - Protocol changes for RM to bind and notify a restarted AM of existing containers
[YARN-1493] - Schedulers don't recognize apps separately from app-attempts
[YARN-1470] - Add audience annotation to MiniYARNCluster
[YARN-1736] - FS: AppSchedulable.assignContainer's priority argument is redundant
[YARN-1617] - Remove ancient comment and surround LOG.debug in AppSchedulingInfo.allocate
[YARN-1697] - NodeManager reports negative running containers
[YARN-1672] - YarnConfiguration is missing a default for yarn.nodemanager.log.retain-seconds

Improvement

[YARN-2641] - Decommission nodes on -refreshNodes instead of next NM-RM heartbeat
[YARN-2352] - FairScheduler: Collect metrics on duration of critical methods that affect performance
[YARN-2274] - FairScheduler: Add debug information about cluster capacity, availability and reservations
[YARN-2061] - Revisit logging levels in ZKRMStateStore
[YARN-2089] - FairScheduler: QueuePlacementPolicy and QueuePlacementRule are missing audience annotations
[YARN-1771] - many getFileStatus calls made from node manager for localizing a public distributed cache resource
[YARN-1461] - RM API and RM changes to handle tags for running jobs
[YARN-1171] - Add default queue properties to Fair Scheduler documentation

New Feature

[YARN-1764] - Handle RM fail overs after the submitApplication call.
[YARN-1861] - Both RM stuck in standby mode when automatic failover is enabled
[YARN-1864] - Fair Scheduler Dynamic Hierarchical User Queues
[YARN-1898] - Standby RM's conf, stacks, logLevel, metrics, jmx and logs links are redirecting to Active RM
[YARN-1893] - Make ApplicationMasterProtocol#allocate AtMostOnce
[YARN-1521] - Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[YARN-1640] - Manual Failover does not work in secure clusters
[YARN-1705] - Reset cluster-metrics on transition to standby
[YARN-1658] - Webservice should redirect to active RM when HA is enabled.
[YARN-1816] - Succeeded application remains in accepted after RM restart
[YARN-1812] - Job stays in PREP state for long time after RM Restarts
[YARN-1766] - When RM does the initiation, it should use loaded Configuration instead of bootstrap configuration.
[YARN-1641] - ZK store should attempt a write periodically to ensure it is still Active
[YARN-1811] - RM HA: AM link broken if the AM is on nodes other than RM
[YARN-1776] - renewDelegationToken should survive RM failover
[YARN-1525] - Web UI should redirect to active RM when HA is enabled.
[YARN-986] - RM DT token service should have service addresses of both RMs
[YARN-1734] - RM should get the updated Configurations when it transits from Standby to Active
[YARN-1666] - Make admin refreshNodes work across RM failover
[YARN-1676] - Make admin refreshUserToGroupsMappings of configuration work across RM failover
[YARN-1459] - RM services should depend on ConfigurationProvider during startup too
[YARN-1660] - add the ability to set yarn.resourcemanager.hostname.rm-id instead of setting all the various host:port properties for RM
[YARN-1665] - Set better defaults for HA configs for automatic failover
[YARN-1669] - Make admin refreshServiceAcls work across RM failover
[YARN-1667] - Make admin refreshSuperUserGroupsConfiguration work across RM failover
[YARN-1668] - Make admin refreshAdminAcls work across RM failover
[YARN-1611] - Make admin refresh of capacity scheduler configuration work across RM failover
[YARN-1639] - YARM RM HA requires different configs on different RM hosts
[YARN-1497] - Expose moving apps between queues on the command line
[YARN-1499] - Fair Scheduler changes for moving apps between queues
[YARN-1504] - RM changes for moving apps between queues
[YARN-1498] - Common scheduler changes for moving apps between queues
[YARN-1496] - Protocol additions to allow moving apps between queues

Task

[YARN-2566] - DefaultContainerExecutor should pick a working directory randomly

Test

[YARN-2018] - TestClientRMService.testTokenRenewalWrongUser fails after HADOOP-10562
[YARN-1970] - Prepare YARN codebase for JUnit 4.11.
[YARN-1854] - Race condition in TestRMHA#testStartAndTransitions

HDFS

Bug

[HDFS-7960] - The full block report should prune zombie storages even if they're not empty
[HDFS-6831] - Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
[HDFS-7596] - NameNode should prune dead storages from storageMap
[HDFS-7208] - NN doesn't schedule replication when a DN storage fails
[HDFS-7575] - Upgrade should generate a unique storage ID for each volume
[HDFS-7391] - Renable SSLv2Hello in HttpFS
[HDFS-7235] - DataNode#transferBlock should report blocks that don't exist using reportBadBlock
[HDFS-7274] - Disable SSLv3 in HttpFS
[HDFS-7005] - DFS input streams do not timeout
[HDFS-6376] - Distcp data between two HA clusters requires another configuration
[HDFS-6621] - Hadoop Balancer prematurely exits iterations
[HDFS-6776] - Using distcp to copy data between insecure and secure cluster via webdhfs doesn't work
[HDFS-6908] - incorrect snapshot directory diff generated by snapshot deletion
[HDFS-6825] - Edit log corruption due to delayed block removal
[HDFS-6703] - NFS: Files can be deleted from a read-only mount
[HDFS-6640] - [ Web HDFS ] Syntax for MKDIRS, CREATESYMLINK, and SETXATTR are given wrongly(missed webhdfs/v1).).
[HDFS-6114] - Block Scan log rolling will never happen if blocks written continuously leading to huge size of dncp_block_verification.log.curr
[HDFS-6788] - Improve synchronization in BPOfferService with read write lock
[HDFS-6647] - Edit log corruption when pipeline recovery occurs for deleted file present in snapshot
[HDFS-6622] - Rename and AddBlock may race and produce invalid edits
[HDFS-6618] - FSNamesystem#delete drops the FSN lock between removing INodes from the tree and deleting them from the inode map
[HDFS-6551] - Rename with OVERWRITE option may throw NPE when the target file/directory is a reference INode
[HDFS-5321] - Clean up the HTTP-related configuration in HDFS
[HDFS-6475] - WebHdfs clients fail without retry because incorrect handling of StandbyException
[HDFS-5891] - webhdfs should not try connecting the DN during redirection
[HDFS-6312] - WebHdfs HA failover is broken on secure clusters
[HDFS-6197] - Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
[HDFS-6527] - Edit log corruption due to defered INode removal
[HDFS-6294] - Use INode IDs to avoid conflicts when a file open for write is renamed
[HDFS-6549] - Add support for accessing the NFS gateway from the AIX NFS client
[HDFS-6563] - NameNode cannot save fsimage in certain circumstances when snapshots are in use
[HDFS-6553] - Add missing DeprecationDeltas for NFS Kerberos configurations
[HDFS-6439] - NFS should not reject NFS requests to the NULL procedure whether port monitoring is enabled or not
[HDFS-6500] - Snapshot shouldn't be removed silently after renaming to an existing snapshot
[HDFS-6379] - HTTPFS - Implement ACLs support
[HDFS-6325] - Append should fail if the last block has insufficient number of replicas
[HDFS-5522] - Datanode disk error check may be incorrectly skipped
[HDFS-6078] - TestIncrementalBlockReports is flaky
[HDFS-2882] - DN continues to start up, even if block pool fails to initialize
[HDFS-6245] - datanode fails to start with a bad disk even when failed volumes is set
[HDFS-6208] - DataNode caching can leak file descriptors.
[HDFS-4909] - Avoid protocol buffer RPC namespace clashes
[HDFS-6381] - Fix a typo in INodeReference.java
[HDFS-6355] - Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner
[HDFS-6219] - Proxy superuser configuration should use true client IP for address checks
[HDFS-6056] - Clean up NFS config settings
[HDFS-5705] - TestSecondaryNameNodeUpgrade#testChangeNsIDFails may fail due to ConcurrentModificationException
[HDFS-6362] - InvalidateBlocks is inconsistent in usage of DatanodeUuid and StorageID
[HDFS-6313] - WebHdfs may use the wrong NN when configured for multiple HA NNs
[HDFS-6077] - running slive with webhdfs on secure HA cluster fails with unkown host exception
[HDFS-6329] - WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
[HDFS-6217] - Webhdfs PUT operations may not work via a http proxy
[HDFS-6214] - Webhdfs has poor throughput for files >2GB
[HDFS-6215] - Wrong error message for upgrade
[HDFS-6140] - WebHDFS cannot create a file with spaces in the name after HA failover changes.
[HDFS-6127] - WebHDFS tokens cannot be renewed in HA setup
[HDFS-6100] - DataNodeWebHdfsMethods does not failover in HA mode
[HDFS-6400] - Cannot execute "hdfs oiv_legacy"
[HDFS-6293] - Issues with OIV processing PB-based fsimages
[HDFS-6274] - Cleanup javadoc warnings in HDFS code
[HDFS-6340] - DN can't finalize upgrade
[HDFS-6130] - NPE when upgrading namenode from fsimages older than -32
[HDFS-5846] - Assigning DEFAULT_RACK in resolveNetworkLocation method can break data resiliency
[HDFS-3087] - Decomissioning on NN restart can complete without blocks being replicated
[HDFS-6089] - Standby NN while transitioning to active throws a connection refused error when the prior active NN process is suspended
[HDFS-6411] - nfs-hdfs-gateway mount raises I/O error and hangs when a unauthorized user attempts to access it
[HDFS-6131] - Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[HDFS-5840] - Follow-up to HDFS-5138 to improve error handling during partial upgrade failures
[HDFS-6135] - In HDFS upgrade with HA setup, JournalNode cannot handle layout version bump when rolling back
[HDFS-6166] - revisit balancer so_timeout
[HDFS-6181] - Fix the wrong property names in NFS user guide
[HDFS-6206] - DFSUtil.substituteForWildcardAddress may throw NPE
[HDFS-6229] - Race condition in failover can cause RetryCache fail to work
[HDFS-6243] - HA NameNode transition to active or shutdown may leave lingering image transfer thread.
[HDFS-6361] - TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
[HDFS-6156] - Simplify the JMX API that provides snapshot information
[HDFS-6462] - NFS: fsstat request fails with the secure hdfs
[HDFS-6404] - HttpFS should use a 000 umask for mkdir and create operations
[HDFS-6402] - Suppress findbugs warning for failure to override equals and hashCode in FsAclPermission.
[HDFS-6326] - WebHdfs ACL compatibility is broken
[HDFS-6351] - Command "hdfs dfs -rm -r" can't remove empty directory
[HDFS-6345] - DFS.listCacheDirectives() should allow filtering based on cache directive ID
[HDFS-6288] - DFSInputStream Pread doesn't update ReadStatistics
[HDFS-6230] - Expose upgrade status through NameNode web UI
[HDFS-6289] - HA failover can fail if there are pending DN messages for DNs which no longer exist
[HDFS-6094] - The same block can be counted twice towards safe mode threshold
[HDFS-6160] - TestSafeMode occasionally fails
[HDFS-6153] - Document "fileId" and "childrenNum" fields in the FileStatus Json schema
[HDFS-6180] - dead node count / listing is very broken in JMX and old GUI
[HDFS-6232] - OfflineEditsViewer throws a NPE on edits containing ACL modifications
[HDFS-6231] - DFSClient hangs infinitely if using hedged reads and all eligible datanodes die.
[HDFS-5591] - Checkpointing should use monotonic time when calculating period
[HDFS-6068] - Disallow snapshot names that are also invalid directory names
[HDFS-6112] - NFS Gateway docs are incorrect for allowed hosts configuration
[HDFS-6106] - Reduce default for dfs.namenode.path.based.cache.refresh.interval.ms
[HDFS-6107] - When a block can't be cached due to limited space on the DataNode, that block becomes uncacheable
[HDFS-5944] - LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/ right and cause SecondaryNameNode failed do checkpoint
[HDFS-6102] - Lower the default maximum items per directory to fix PB fsimage loading
[HDFS-6097] - zero-copy reads are incorrectly disabled on file offsets above 2GB
[HDFS-6079] - Timeout for getFileBlockStorageLocations does not work
[HDFS-6063] - TestAclCLI fails intermittently when running test 24: copyFromLocal
[HDFS-6065] - HDFS zero-copy reads should return null on EOF when doing ZCR
[HDFS-6067] - TestPread.testMaxOutHedgedReadPool is flaky
[HDFS-6053] - Fix TestDecommissioningStatus and TestDecommission in branch-2
[HDFS-6039] - Uploading a File under a Dir with default acls throws "Duplicated ACLFeature"
[HDFS-5821] - TestHDFSCLI fails for user names with the dash character
[HDFS-6028] - Print clearer error message when user attempts to delete required mask entry from ACL.
[HDFS-5064] - Standby checkpoints should not block concurrent readers
[HDFS-6071] - BlockReaderLocal doesn't return -1 on EOF when doing a zero-length read on a short file
[HDFS-6047] - TestPread NPE inside in DFSInputStream hedgedFetchBlockByteRange
[HDFS-6059] - TestBlockReaderLocal fails if native library is not available
[HDFS-6057] - DomainSocketWatcher.watcherThread should be marked as daemon thread
[HDFS-6051] - HDFS cannot run on Windows since short-circuit shared memory segment changes.
[HDFS-5339] - WebHDFS URI does not accept logical nameservices when security is enabled
[HDFS-6033] - PBImageXmlWriter incorrectly handles processing cache directives
[HDFS-3969] - Small bug fixes and improvements for disk locations API
[HDFS-6008] - Namenode dead node link is giving HTTP error 500
[HDFS-5981] - PBImageXmlWriter generates malformed XML
[HDFS-5962] - Mtime and atime are not persisted for symbolic links
[HDFS-5742] - DatanodeCluster (mini cluster of DNs) fails to start
[HDFS-5961] - OIV cannot load fsimages containing a symbolic link
[HDFS-5759] - Web UI does not show up during the period of loading FSImage
[HDFS-5716] - Allow WebHDFS to use pluggable authentication filter
[HDFS-5943] - 'dfs.namenode.https-address.ns1' property is not used in federation setup
[HDFS-5941] - add dfs.namenode.secondary.https-address and dfs.namenode.secondary.https-address in hdfs-default.xml
[HDFS-5879] - Some TestHftpFileSystem tests do not close streams
[HDFS-4858] - HDFS DataNode to NameNode RPC should timeout
[HDFS-5886] - Potential null pointer deference in RpcProgramNfs3#readlink()
[HDFS-5881] - Fix skip() of the short-circuit local reader (legacy).
[HDFS-5791] - TestHttpsFileSystem should use a random port to avoid binding error during testing
[HDFS-5828] - BlockPlacementPolicyWithNodeGroup can place multiple replicas on the same node group when dfs.namenode.avoid.write.stale.datanode is true
[HDFS-5856] - DataNode.checkDiskError might throw NPE
[HDFS-5843] - DFSClient.getFileChecksum() throws IOException if checksum is disabled
[HDFS-5492] - Port HDFS-2069 (Incorrect default trash interval in the docs) to trunk
[HDFS-5922] - DN heartbeat thread can get stuck in tight loop
[HDFS-5982] - Need to update snapshot manager when applying editlog for deleting a snapshottable directory
[HDFS-5988] - Bad fsimage always generated after upgrade
[HDFS-5959] - Fix typo at section name in FSImageFormatProtobuf.java
[HDFS-5915] - Refactor FSImageFormatProtobuf to simplify cross section reads
[HDFS-5803] - TestBalancer.testBalancer0 fails
[HDFS-5780] - TestRBWBlockInvalidation times out intemittently on branch-2
[HDFS-5913] - Nfs3Utils#getWccAttr() should check attr parameter against null
[HDFS-5904] - TestFileStatus fails intermittently on trunk and branch2
[HDFS-5807] - TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails intermittently on Branch-2
[HDFS-5859] - DataNode#checkBlockToken should check block tokens even if security is not enabled
[HDFS-5901] - NameNode new UI doesn't support IE8 and IE9 on windows 7
[HDFS-5934] - New Namenode UI back button doesn't work as expected
[HDFS-5949] - New Namenode UI when trying to download a file, the browser doesn't know the file name
[HDFS-5948] - TestBackupNode flakes with port in use error
[HDFS-3128] - Unit tests should not use a test root in /tmp
[HDFS-5900] - Cannot set cache pool limit of "unlimited" via CacheAdmin
[HDFS-5138] - Support HDFS upgrade in HA
[HDFS-5895] - HDFS cacheadmin -listPools has exit_code of 1 when the command returns 0 result.
[HDFS-5767] - NFS implementation assumes userName userId mapping to be unique, which is not true sometimes
[HDFS-5790] - LeaseManager.findPath is very slow when many leases need recovery
[HDFS-5685] - DistCp will fail to copy with -delete switch
[HDFS-2264] - NamenodeProtocol has the wrong value for clientPrincipal in KerberosInfo annotation

Improvement

[HDFS-7217] - Better batching of IBRs
[HDFS-7278] - Add a command that allows sysadmins to manually trigger full block reports from a DN
[HDFS-6529] - Trace logging for RemoteBlockReader2 to identify remote datanode and file being read
[HDFS-6007] - Update documentation about short-circuit local reads
[HDFS-6403] - Add metrics for log warnings reported by JVM pauses
[HDFS-6499] - Use NativeIO#renameTo instead of File#renameTo in FileJournalManager
[HDFS-6268] - Better sorting in NetworkTopology#pseudoSortByDistance when no local node is found
[HDFS-5986] - Capture the number of blocks pending deletion on namenode webUI
[HDFS-6084] - Namenode UI - "Hadoop" logo link shouldn't go to hadoop homepage
[HDFS-4052] - BlockManager#invalidateWork should print logs outside the lock
[HDFS-6072] - Clean up dead code of FSImage
[HDFS-6070] - Cleanup use of ReadStatistics in DFSInputStream
[HDFS-6080] - Improve NFS gateway performance by making rtmax and wtmax configurable
[HDFS-5167] - Add metrics about the NameNode retry cache
[HDFS-6186] - Pause deletion of blocks when the namenode starts up
[HDFS-6050] - NFS does not handle exceptions correctly in a few places
[HDFS-5196] - Provide more snapshot information in WebUI
[HDFS-6304] - Consolidate the logic of path resolution in FSDirectory
[HDFS-5683] - Better audit log messages for caching operations
[HDFS-6295] - Add "decommissioning" state and node state filtering to dfsadmin
[HDFS-6191] - Disable quota checks when replaying edit log.
[HDFS-6086] - Fix a case where zero-copy or no-checksum reads were not allowed even when the block was cached
[HDFS-6085] - Improve CacheReplicationMonitor log messages a bit
[HDFS-6069] - Quash stack traces when ACLs are disabled
[HDFS-6044] - Add property for setting the NFS look up time for users
[HDFS-6043] - Give HDFS daemons NFS3 and Portmap their own OPTS
[HDFS-5956] - A file size is multiplied by the replication factor in 'hdfs oiv -p FileDistribution' option
[HDFS-3405] - Checkpointing should use HTTP POST or PUT instead of GET-GET to send merged fsimages
[HDFS-5776] - Support 'hedged' reads in DFSClient
[HDFS-6061] - Allow dfs.datanode.shared.file.descriptor.path to contain multiple entries and fall back when needed
[HDFS-6046] - add dfs.client.mmap.enabled
[HDFS-6040] - fix DFSClient issue without libhadoop.so and some other ShortCircuitShm cleanups
[HDFS-5950] - The DFSClient and DataNode should use shared memory segments to communicate short-circuit information
[HDFS-4200] - Reduce the size of synchronized sections in PacketResponder
[HDFS-5939] - WebHdfs returns misleading error code and logs nothing if trying to create a file with no DNs in cluster
[HDFS-6018] - Exception recorded in LOG when IPCLoggerChannel#close is called
[HDFS-6006] - Remove duplicate code in FSNameSystem#getFileInfo
[HDFS-5935] - New Namenode UI FS browser should throw smarter error messages
[HDFS-5973] - add DomainSocket#shutdown method
[HDFS-5868] - Make hsync implementation pluggable
[HDFS-5318] - Support read-only and read-write paths to shared replicas
[HDFS-5942] - Fix javadoc in OfflineImageViewer
[HDFS-5929] - Add Block pool % usage to HDFS federated nn page
[HDFS-5781] - Use an array to record the mapping between FSEditLogOpCode and the corresponding byte value
[HDFS-5153] - Datanode should send block reports for each storage in a separate message
[HDFS-5979] - Typo and logger fix for fsimage PB code
[HDFS-5847] - Consolidate INodeReference into a separate section
[HDFS-5698] - Use protobuf to serialize / deserialize FSImage
[HDFS-5775] - Consolidate the code for serialization in CacheManager
[HDFS-5768] - Consolidate the serialization code in DelegationTokenSecretManager
[HDFS-5726] - Fix compilation error in AbstractINodeDiff for JDK7
[HDFS-5632] - Add Snapshot feature to INodeDirectory
[HDFS-5647] - Merge INodeDirectory.Feature and INodeFile.Feature
[HDFS-5554] - Add Snapshot Feature to INodeFile
[HDFS-5537] - Remove FileWithSnapshot interface
[HDFS-5286] - Flatten INodeDirectory hierarchy: add DirectoryWithQuotaFeature
[HDFS-5285] - Flatten INodeFile hierarchy: Add UnderContruction Feature
[HDFS-5531] - Combine the getNsQuota() and getDsQuota() methods in INode
[HDFS-4370] - Fix typo Blanacer in DataNode
[HDFS-5940] - Minor cleanups to ShortCircuitReplica, FsDatasetCache, and DomainSocketWatcher
[HDFS-5938] - Make BlockReaderFactory#BlockReaderPeer a static class
[HDFS-5810] - Unify mmap cache and short-circuit file descriptor cache
[HDFS-4911] - Reduce PeerCache timeout to be commensurate with dfs.datanode.socket.reuse.keepalive
[HDFS-5746] - add ShortCircuitSharedMemorySegment
[HDFS-5709] - Improve NameNode upgrade with existing reserved paths and path components
[HDFS-4817] - make HDFS advisory caching configurable on a per-file basis
[HDFS-3910] - DFSTestUtil#waitReplication should timeout

New Feature

[HDFS-4257] - The ReplaceDatanodeOnFailure policies could have a forgiving option
[HDFS-5638] - HDFS implementation of FileContext API for ACLs.
[HDFS-6076] - SimulatedDataSet should not create DatanodeRegistration with namenode layout version and type
[HDFS-6435] - Add support for specifying a static uid/gid mapping for the NFS gateway
[HDFS-6406] - Add capability for NFS gateway to reject connections from unprivileged ports
[HDFS-6281] - Provide option to use the NFS Gateway without having to use the Hadoop portmapper
[HDFS-5898] - Allow NFS gateway to login/relogin from its kerberos keytab
[HDFS-6060] - NameNode should not check DataNode layout version
[HDFS-6041] - Downgrade/Finalize should rename the rollback image instead of purging it
[HDFS-6042] - Fix rolling upgrade documentation and error messges
[HDFS-6013] - add rollingUpgrade information to latest UI
[HDFS-6034] - DN registration should use DataNodeLayoutVersion instead of NameNodeLayoutVersion
[HDFS-6031] - NN running newer software rejects loading the fsimage during rolling upgrade.
[HDFS-6032] - -rollingUpgrade query hits NPE after the NN restarts
[HDFS-6029] - Secondary NN fails to checkpoint after -rollingUpgrade prepare
[HDFS-6024] - Test whether the NN will reject the downgrade if it has a fsimage from a newer release
[HDFS-6026] - Fix TestDFSUpgrade and TestDataNodeRollingUpgrade
[HDFS-6023] - Test whether the standby NN continues to checkpoint after the prepare stage
[HDFS-5535] - Umbrella jira for improved HDFS rolling upgrades
[HDFS-4685] - Implementation of ACLs in HDFS
[HDFS-5483] - NN should gracefully handle multiple block replicas on same DN
[HDFS-5804] - HDFS NFS Gateway fails to mount and proxy when using Kerberos
[HDFS-5511] - improve CacheManipulator interface to allow better unit testing
[HDFS-1623] - High Availability Framework for HDFS NN

Task

[HDFS-6030] - Remove an unused constructor in INode.java
[HDFS-6025] - Update findbugsExcludeFile.xml

Test

[HDFS-6419] - TestBookKeeperHACheckpoints#TestSBNCheckpoints fails on trunk
[HDFS-6287] - Add vecsum test of libhdfs read access times
[HDFS-6224] - Add a unit test to TestAuditLogger for file permissions passed to logAuditEvent
[HDFS-5936] - MiniDFSCluster does not clean data left behind by SecondaryNameNode.
[HDFS-5953] - TestBlockReaderFactory fails if libhadoop.so has not been built
[HDFS-5882] - TestAuditLogs is flaky