CDH 5.1.5 Release Notes
The following lists all Apache Hadoop Jiras included in CDH 5.1.5
that are not included in the Apache Hadoop base version 2.3.0. The
hadoop-2.3.0-cdh5.1.5.CHANGES.txt
file lists all changes included in CDH 5.1.5. The patch for each
change can be found in the cloudera/patches directory in the release tarball.
Changes Not In Apache Hadoop 2.3.0
MapReduce
Bug
- [MAPREDUCE-6012] - DBInputSplit creates invalid ranges on Oracle
- [MAPREDUCE-5968] - Work directory is not deleted when downloadCacheObject throws IOException
- [MAPREDUCE-5966] - MR1 FairScheduler use of custom weight adjuster is not thread safe for comparisons
- [MAPREDUCE-5979] - FairScheduler: zero weight can cause sort failures
- [MAPREDUCE-5707] - JobClient does not allow setting RPC timeout for communications with JT/RM
- [MAPREDUCE-5375] - Delegation Token renewal exception in jobtracker logs
- [MAPREDUCE-5875] - Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
- [MAPREDUCE-5813] - YarnChild does not load job.xml with mapreduce.job.classloader=true
- [MAPREDUCE-5814] - fat jar with *-default.xml may fail when mapreduce.job.classloader=true.
- [MAPREDUCE-5888] - Failed job leaves hung AM after it unregisters
- [MAPREDUCE-5835] - Killing Task might cause the job to go to ERROR state
- [MAPREDUCE-5841] - uber job doesn't terminate on getting mapred job kill
- [MAPREDUCE-5795] - Job should be marked as Failed if it is recovered from commit.
- [MAPREDUCE-5746] - Job diagnostics can implicate wrong task for a failed job
- [MAPREDUCE-5769] - Unregistration to RM should not be called if AM is crashed before registering with RM
- [MAPREDUCE-5751] - MR app master fails to start in some cases if mapreduce.job.classloader is true
- [MAPREDUCE-5757] - ConcurrentModificationException in JobControl.toList
- [MAPREDUCE-5670] - CombineFileRecordReader should report progress when moving to the next file
- [MAPREDUCE-5770] - Redirection from AM-URL is broken with HTTPS_ONLY policy
- [MAPREDUCE-5862] - Line records longer than 2x split size aren't handled correctly
- [MAPREDUCE-4052] - Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
- [MAPREDUCE-5028] - Maps fail when io.sort.mb is set to high value
- [MAPREDUCE-5768] - TestMRJobs.testContainerRollingLog fails on trunk
- [MAPREDUCE-5699] - Allow setting tags on MR jobs
- [MAPREDUCE-2779] - JobSplitWriter.java can't handle large job.split file
- [MAPREDUCE-2324] - Job should fail if a reduce task can't be scheduled anywhere
- [MAPREDUCE-5702] - TaskLogServlet#printTaskLog has spurious HTML closing tags
- [MAPREDUCE-4383] - HadoopPipes.cc needs to include unistd.h
- [MAPREDUCE-5206] - JT can show the same job multiple times in Retired Jobs section
- [MAPREDUCE-5508] - JobTracker memory leak caused by unreleased FileSystem objects in JobInProgress#cleanupJob
- [MAPREDUCE-5066] - JobTracker should set a timeout when calling into job.end.notification.url
- [MAPREDUCE-5198] - Race condition in cleanup during task tracker renint with LinuxTaskController
- [MAPREDUCE-4914] - TestMiniMRDFSSort fails with openJDK7
- [MAPREDUCE-5450] - Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
- [MAPREDUCE-4366] - mapred metrics shows negative count of waiting maps and reduces
- [MAPREDUCE-5351] - JobTracker memory leak caused by CleanupQueue reopening FileSystem
- [MAPREDUCE-5364] - Deadlock between RenewalTimerTask methods cancel() and run()
- [MAPREDUCE-5038] - old API CombineFileInputFormat missing fixes that are in new API
- [MAPREDUCE-2359] - Distributed cache doesn't use non-default FileSystems correctly
- [MAPREDUCE-5250] - Searching for ';' in JobTracker History throws ArrayOutOfBoundException
- [MAPREDUCE-4970] - Child tasks (try to) create security audit log files
- [MAPREDUCE-323] - Improve the way job history files are managed
- [MAPREDUCE-4576] - Large dist cache can block tasktracker heartbeat
- [MAPREDUCE-3824] - Distributed caches are not removed properly
- [MAPREDUCE-2409] - Distributed Cache does not differentiate between file /archive for files with the same path
- [MAPREDUCE-5218] - Annotate (comment) internal classes as Private
- [MAPREDUCE-5193] - A few MR tests use block sizes which are smaller than the default minimum block size
- [MAPREDUCE-5154] - staging directory deletion fails because delegation tokens have been cancelled
- [MAPREDUCE-5133] - TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race
- [MAPREDUCE-2817] - MiniRMCluster hardcodes 'mapred.local.dir' configuration to 'build/test/mapred/local'
- [MAPREDUCE-5008] - Merger progress miscounts with respect to EOF_MARKER
- [MAPREDUCE-5073] - TestJobStatusPersistency.testPersistency fails on JDK7
- [MAPREDUCE-5072] - TestDelegationTokenRenewal.testDTRenewal fails in MR1 on jdk7
- [MAPREDUCE-5070] - TestClusterStatus.testClusterMetrics fails on JDK7
- [MAPREDUCE-2187] - map tasks timeout during sorting
- [MAPREDUCE-5047] - keep.failed.task.files=true causes job failure on secure clusters
- [MAPREDUCE-4843] - When using DefaultTaskController, JobLocalizer not thread safe
- [MAPREDUCE-4643] - Make job-history cleanup-period configurable
- [MAPREDUCE-4888] - NLineInputFormat drops data in 1.1 and beyond
- [MAPREDUCE-4962] - jobdetails.jsp uses display name instead of real name to get counters
- [MAPREDUCE-4963] - StatisticsCollector improperly keeps track of "Last Day" and "Last Hour" statistics for new TaskTrackers
- [MAPREDUCE-2264] - Job status exceeds 100% in some cases
- [MAPREDUCE-4929] - mapreduce.task.timeout is ignored
- [MAPREDUCE-4562] - Support for "FileSystemCounter" legacy counter group name for compatibility reasons is creating incorrect counter name
- [MAPREDUCE-4923] - Add toString method to TaggedInputSplit
- [MAPREDUCE-4315] - jobhistory.jsp throws 500 when a .txt file is found in /done
- [MAPREDUCE-4925] - The pentomino option parser may be buggy
- [MAPREDUCE-4924] - flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR
- [MAPREDUCE-3475] - JT can't renew its own tokens
- [MAPREDUCE-4933] - MR1 final merge asks for length of file it just wrote before flushing it
- [MAPREDUCE-4850] - Job recovery may fail if staging directory has been deleted
- [MAPREDUCE-4278] - cannot run two local jobs in parallel from the same gateway.
- [MAPREDUCE-2217] - The expire launching task should cover the UNASSIGNED task
- [MAPREDUCE-1238] - mapred metrics shows negative count of waiting maps and reduces
- [MAPREDUCE-4860] - DelegationTokenRenewal attempts to renew token even after a job is removed
- [MAPREDUCE-4859] - TestRecoveryManager fails on branch-1
- [MAPREDUCE-4778] - Fair scheduler event log is only written if directory exists on HDFS
- [MAPREDUCE-4800] - Cleanup o.a.h.mapred.MapTaskStatus - remove unused code
- [MAPREDUCE-1905] - Context.setStatus() and progress() api are ignored
- [MAPREDUCE-4629] - Remove JobHistory.DEBUG_MODE
- [MAPREDUCE-4765] - Restarting the JobTracker programmatically can cause DelegationTokenRenewal to throw an exception
- [MAPREDUCE-1806] - CombineFileInputFormat does not work with paths not on default FS
- [MAPREDUCE-4451] - fairscheduler fail to init job with kerberos authentication configured
- [MAPREDUCE-4706] - FairScheduler#dump(): Computing of # running maps and reduces is commented out
- [MAPREDUCE-4652] - ValueAggregatorJob sets the wrong job jar
- [MAPREDUCE-2185] - Infinite loop at creating splits using CombineFileInputFormat
- [MAPREDUCE-4470] - Fix TestCombineFileInputFormat.testForEmptyFile
- [MAPREDUCE-4610] - Support deprecated mapreduce.job.counters.limit property in MR2
- [MAPREDUCE-2764] - Fix renewal of dfs delegation tokens
- [MAPREDUCE-2452] - Delegation token cancellation shouldn't hold global JobTracker lock
- [MAPREDUCE-4595] - TestLostTracker failing - possibly due to a race in JobHistory.JobHistoryFilesManager#run()
- [MAPREDUCE-2374] - "Text File Busy" errors launching MR tasks
- [MAPREDUCE-4385] - FairScheduler.maxTasksToAssign() should check for fairscheduler.assignmultiple.maps < TaskTracker.availableSlots
- [MAPREDUCE-3993] - Graceful handling of codec errors during decompression
- [MAPREDUCE-1505] - Cluster class should create the rpc client only when needed
- [MAPREDUCE-4463] - JobTracker recovery fails with HDFS permission issue
- [MAPREDUCE-4399] - Fix performance regression in shuffle
- [MAPREDUCE-4195] - With invalid queueName request param, jobqueue_details.jsp shows NPE
- [MAPREDUCE-3674] - If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
- [MAPREDUCE-2839] - MR Jobs fail on a secure cluster with viewfs
- [MAPREDUCE-4226] - ConcurrentModificationException in FileSystemCounterGroup
- [MAPREDUCE-4129] - Lots of unneeded counters log messages
- [MAPREDUCE-3827] - Counters aggregation slowed down significantly after MAPREDUCE-3749
- [MAPREDUCE-3809] - Tasks may take upto 3 seconds to exit after completion
- [MAPREDUCE-2450] - Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
- [MAPREDUCE-4095] - TestJobInProgress#testLocality uses a bogus topology
- [MAPREDUCE-4012] - Hadoop Job setup error leaves no useful info to users (when LinuxTaskController is used)
- [MAPREDUCE-3583] - ProcfsBasedProcessTree#constructProcessInfo() may throw NumberFormatException
- [MAPREDUCE-1740] - NPE in getMatchingLevelForNodes when node locations are variable depth
- [MAPREDUCE-4036] - Streaming TestUlimit fails on CentOS 6
- [MAPREDUCE-3727] - jobtoken location property in jobconf refers to wrong jobtoken file
- [MAPREDUCE-4010] - TestWritableJobConf fails on trunk
- [MAPREDUCE-3789] - CapacityTaskScheduler may perform unnecessary reservations in heterogenous tracker environments
- [MAPREDUCE-1109] - ConcurrentModificationException in jobtracker.jsp
- [MAPREDUCE-3997] - jobhistory.jsp cuts off the job name at the first underscore of the job name
- [MAPREDUCE-3697] - Hadoop Counters API limits Oozie's working across different hadoop versions
- [MAPREDUCE-3639] - TokenCache likely broken for FileSystems which don't issue delegation tokens
- [MAPREDUCE-3749] - ConcurrentModificationException in counter groups
- [MAPREDUCE-3563] - LocalJobRunner doesn't handle Jobs using o.a.h.mapreduce.OutputCommitter
- [MAPREDUCE-2531] - org.apache.hadoop.mapred.jobcontrol.getAssignedJobID throw class cast exception
- [MAPREDUCE-3542] - Support "FileSystemCounter" legacy counter group name for compatibility
- [MAPREDUCE-3433] - Finding counters by legacy group name returns empty counters
- [MAPREDUCE-895] - FileSystem::ListStatus will now throw FileNotFoundException, MapRed needs updated
Improvement
- [MAPREDUCE-5777] - Support utf-8 text with BOM (byte order marker)
- [MAPREDUCE-5754] - Preserve Job diagnostics in history
- [MAPREDUCE-5014] - Extending DistCp through a custom CopyListing is not possible
- [MAPREDUCE-5773] - Provide dedicated MRAppMaster syslog length limit
- [MAPREDUCE-5732] - Report proper queue when job has been automatically placed
- [MAPREDUCE-5651] - Backport Fair Scheduler queue placement policies to branch-1
- [MAPREDUCE-3310] - Custom grouping comparator cannot be set for Combiners
- [MAPREDUCE-3169] - Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
- [MAPREDUCE-5609] - Add debug log message when sending job end notification
- [MAPREDUCE-5367] - Local jobs all use same local working directory
- [MAPREDUCE-5379] - Include token tracking ids in jobconf
- [MAPREDUCE-2494] - Make the distributed cache delete entires using LRU priority
- [MAPREDUCE-2495] - The distributed cache cleanup thread has no monitoring to check to see if it has died for some reason
- [MAPREDUCE-1568] - TrackerDistributedCacheManager should clean up cache in a background thread
- [MAPREDUCE-4777] - In TestIFile, testIFileReaderWithCodec relies on testIFileWriterWithCodec
- [MAPREDUCE-4977] - Documentation for pluggable shuffle and pluggable sort
- [MAPREDUCE-2931] - CLONE - LocalJobRunner should support parallel mapper execution
- [MAPREDUCE-2492] - [MAPREDUCE] The new MapReduce API should make available task's progress to the task
- [MAPREDUCE-4907] - TrackerDistributedCacheManager issues too many getFileStatus calls
- [MAPREDUCE-4464] - Reduce tasks failing with NullPointerException in ConcurrentHashMap.get()
- [MAPREDUCE-3607] - Port missing new API mapreduce lib classes to 1.x
- [MAPREDUCE-4845] - ClusterStatus.getMaxMemory() and getUsedMemory() exist in MR1 but not MR2
- [MAPREDUCE-4556] - FairScheduler: PoolSchedulable#updateDemand() has potential redundant computation
- [MAPREDUCE-2786] - TestDFSIO should also test compression reading/writing from command-line.
- [MAPREDUCE-4408] - allow jobs to set a JAR that is in the distributed cached
- [MAPREDUCE-4511] - Add IFile readahead
- [MAPREDUCE-4487] - Reduce job latency by removing hardcoded sleep statements
- [MAPREDUCE-336] - The logging level of the tasks should be configurable by the job
- [MAPREDUCE-2456] - Show the reducer taskid and map/reduce tasktrackers for "Failed fetch notification #_ for task attempt..." log messages
- [MAPREDUCE-3289] - Make use of fadvise in the NM's shuffle handler
- [MAPREDUCE-1221] - Kill tasks on a node if the free physical memory on that machine falls below a configured threshold
- [MAPREDUCE-157] - Job History log file format is not friendly for external tools.
- [MAPREDUCE-954] - The new interface's Context objects should be interfaces
- [MAPREDUCE-368] - Change org.apache.hadoop.mapred.jobcontrol to use new api
- [MAPREDUCE-1943] - Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
- [MAPREDUCE-901] - Move Framework Counters into a TaskMetric structure
- [MAPREDUCE-2455] - Remove deprecated JobTracker.State in favour of JobTrackerStatus
New Feature
- [MAPREDUCE-4049] - plugin for generic shuffle service
- [MAPREDUCE-4808] - Refactor MapOutput and MergeManager to facilitate reuse by Shuffle implementations
- [MAPREDUCE-4807] - Allow MapOutputBuffer to be pluggable
- [MAPREDUCE-4824] - Provide a mechanism for jobs to indicate they should not be recovered on restart
- [MAPREDUCE-3678] - The Map tasks logs should have the value of input split it processed
- [MAPREDUCE-4355] - Add RunningJob.getJobStatus()
- [MAPREDUCE-4417] - add support for encrypted shuffle
- [MAPREDUCE-3837] - Job tracker is not able to recover job in case of crash and after that no user can submit job.
Task
- [MAPREDUCE-4246] - Failure in deleting user directories in Secure hadoop
Common
Bug
- [HADOOP-11243] - SSLFactory shouldn't allow SSLv3
- [HADOOP-11156] - DelegateToFileSystem should implement getFsStatus(final Path f).
- [HADOOP-11035] - distcp on mr1(branch-1) fails with NPE using a short relative source path.
- [HADOOP-10820] - Throw an exception in GenericOptionsParser when passed an empty Path
- [HADOOP-10876] - The constructor of Path should not take an empty URL as a parameter
- [HADOOP-10701] - NFS should not validate the access premission only based on the user's primary group
- [HADOOP-10711] - Cleanup some extra dependencies from hadoop-auth
- [HADOOP-10394] - TestAuthenticationFilter is flaky
- [HADOOP-10301] - AuthenticationFilter should return Forbidden for failed authentication
- [HADOOP-10158] - SPNEGO should work with multiple interfaces/SPNs.
- [HADOOP-10568] - Add s3 server-side encryption
- [HADOOP-10353] - FsUrlStreamHandlerFactory is not thread safe
- [HADOOP-10332] - HttpServer's jetty audit log always logs 200 OK
- [HADOOP-10425] - Incompatible behavior of LocalFileSystem:getContentSummary
- [HADOOP-10414] - Incorrect property name for RefreshUserMappingProtocol in hadoop-policy.xml
- [HADOOP-10475] - ConcurrentModificationException in AbstractDelegationTokenSelector.selectToken()
- [HADOOP-8826] - Docs still refer to 0.20.205 as stable line
- [HADOOP-10531] - hadoop-config.sh - bug in --hosts argument
- [HADOOP-10500] - TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy user configuration.
- [HADOOP-10658] - SSLFactory expects truststores being configured
- [HADOOP-10609] - .gitignore should ignore .orig and .rej files
- [HADOOP-10562] - Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
- [HADOOP-10638] - Updating hadoop-daemon.sh to work as expected when nfs is started as a privileged user.
- [HADOOP-10639] - FileBasedKeyStoresFactory initialization is not using default for SSL_REQUIRE_CLIENT_CERT_KEY
- [HADOOP-10612] - NFS failed to refresh the user group id mapping table
- [HADOOP-10489] - UserGroupInformation#getTokens and UserGroupInformation#addToken can lead to ConcurrentModificationException
- [HADOOP-10401] - ShellBasedUnixGroupsMapping#getGroups does not always return primary group first
- [HADOOP-10442] - Group look-up can cause segmentation fault when certain JNI-based mapping module is used.
- [HADOOP-10456] - Bug in Configuration.java exposed by Spark (ConcurrentModificationException)
- [HADOOP-10459] - distcp V2 doesn't preserve root dir's attributes when -p is specified
- [HADOOP-10418] - SaslRpcClient should not assume that remote principals are in the default_realm
- [HADOOP-10378] - Typo in help printed by hdfs dfs -help
- [HADOOP-10314] - The ls command help still shows outdated 0.16 format.
- [HADOOP-10368] - InputStream is not closed in VersionInfo ctor
- [HADOOP-10355] - TestLoadGenerator#testLoadGenerator fails
- [HADOOP-10328] - loadGenerator exit code is not reliable
- [HADOOP-10249] - LdapGroupsMapping should trim ldap password read from file
- [HADOOP-10330] - TestFrameDecoder fails if it cannot bind port 12345
- [HADOOP-10320] - Javadoc in InterfaceStability.java lacks final </ul>
- [HADOOP-10070] - RPC client doesn't use per-connection conf to determine server's expected Kerberos principal name
- [HADOOP-10346] - Deadlock while logging tokens
- [HADOOP-10338] - Cannot get the FileStatus of the root inode from the new Globber
- [HADOOP-10326] - M/R jobs can not access S3 if Kerberos is enabled
- [HADOOP-8912] - adding .gitattributes file to prevent CRLF and LF mismatches for source and text files
- [HADOOP-8329] - Build fails with Java 7
- [HADOOP-8552] - Conflict: Same security.log.file for multiple users.
- [HADOOP-7967] - Need generalized multi-token filesystem support
- [HADOOP-8781] - hadoop-config.sh should add JAVA_LIBRARY_PATH to LD_LIBRARY_PATH
- [HADOOP-7644] - Fix the delegation token tests to use the new style renewers
- [HADOOP-8466] - hadoop-client POM incorrectly excludes avro
- [HADOOP-8261] - Har file system doesn't deal with FS URIs with a host but no port
Improvement
- [HADOOP-10379] - Protect authentication cookies with the HttpOnly and Secure flags
- [HADOOP-10322] - Add ability to read principal names from a keytab
- [HADOOP-10211] - Enable RPC protocol to negotiate SASL-QOP values between clients and servers
- [HADOOP-10345] - Sanitize the the inputs (groups and hosts) for the proxyuser configuration
- [HADOOP-10614] - CBZip2InputStream is not threadsafe
- [HADOOP-10572] - Example NFS mount command must pass noacl as it isn't supported by the server yet
- [HADOOP-10503] - Move junit up to v 4.11
- [HADOOP-10556] - Add toLowerCase support to auth_to_local rules for service name
- [HADOOP-9454] - Support multipart uploads for s3native
- [HADOOP-10278] - Refactor to make CallQueue pluggable
- [HADOOP-10348] - Deprecate hadoop.ssl.configuration in branch-2, and remove it in trunk
- [HADOOP-10343] - Change info to debug log in LossyRetryInvocationHandler
- [HADOOP-10295] - Allow distcp to automatically identify the checksum type of source files and use it for the target
- [HADOOP-10139] - Update and improve the Single Cluster Setup document
- [HADOOP-10333] - Fix grammatical error in overview.html document
- [HADOOP-9756] - Additional cleanup RPC code
- [HADOOP-8700] - Move the checksum type constants to an enum
- [HADOOP-8968] - Add a flag to completely disable the worker version check
- [HADOOP-8209] - Add option to relax build-version check for branch-1
New Feature
Task
Test
- [HADOOP-3679] - calls to junit Assert::assertEquals invert arguments, causing misleading error messages, other minor improvements.
- [HADOOP-9051] - ant test will build failed for trying to delete a file
YARN
Bug
- [YARN-570] - Time strings are formated in different timezone
- [YARN-2251] - Avoid negative elapsed time in JHS/MRAM web UI and services
- [YARN-2588] - Standby RM does not transitionToActive if previous transitionToActive is failed with ZK exception.
- [YARN-2273] - NPE in ContinuousScheduling thread when we lose a node
- [YARN-2608] - FairScheduler: Potential deadlocks in loading alloc files and clock access
- [YARN-1458] - FairScheduler: Zero weight can lead to livelock
- [YARN-2313] - Livelock can occur in FairScheduler when there are lots of running apps
- [YARN-2359] - Application hangs when it fails to launch AM container
- [YARN-2192] - TestRMHA fails when run with a mix of Schedulers
- [YARN-2109] - Fix TestRM to work with both schedulers
- [YARN-2204] - TestAMRestart#testAMRestartWithExistingContainers assumes CapacityScheduler
- [YARN-2187] - FairScheduler: Disable max-AM-share check by default
- [YARN-2155] - FairScheduler: Incorrect threshold check for preemption
- [YARN-2132] - ZKRMStateStore.ZKAction#runWithRetries doesn't log the exception it encounters
- [YARN-2128] - FairScheduler: Incorrect calculation of amResource usage
- [YARN-2122] - In AllocationFileLoaderService, the reloadThread should be created in init() and started in start()
- [YARN-1913] - With Fair Scheduler, cluster can logjam when all resources are consumed by AMs
- [YARN-1550] - NPE in FairSchedulerAppsBlock#render
- [YARN-2054] - Better defaults for YARN ZK configs for retries and retry-inteval when HA is enabled
- [YARN-1929] - DeadLock in RM when automatic failover is enabled.
- [YARN-1934] - Potential NPE in ZKRMStateStore caused by handling Disconnected event from ZK.
- [YARN-1867] - NPE while fetching apps via the REST API
- [YARN-1866] - YARN RM fails to load state store with delegation token parsing error
- [YARN-1852] - Application recovery throws InvalidStateTransitonException for FAILED and KILLED jobs
- [YARN-1849] - NPE in ResourceTrackerService#registerNodeManager for UAM
- [YARN-1136] - Replace junit.framework.Assert with org.junit.Assert
- [YARN-1590] - _HOST doesn't expand properly for RM, NM, ProxyServer and JHS
- [YARN-2010] - RM can't transition to active if it can't recover an app attempt
- [YARN-1837] - TestMoveApplication.testMoveRejectedByScheduler randomly fails
- [YARN-1883] - TestRMAdminService fails due to inconsistent entries in UserGroups
- [YARN-1490] - RM should optionally not kill all containers when an ApplicationMaster exits
- [YARN-1566] - Change distributed-shell to retain containers from previous AppAttempt
- [YARN-2073] - Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free
- [YARN-1692] - ConcurrentModificationException in fair scheduler AppSchedulable
- [YARN-1553] - Do not use HttpConfig.isSecure() in YARN
- [YARN-2016] - Yarn getApplicationRequest start time range is not honored
- [YARN-2036] - Document yarn.resourcemanager.hostname in ClusterSetup
- [YARN-1932] - Javascript injection on the job status page
- [YARN-1924] - STATE_STORE_OP_FAILED happens when ZKRMStateStore tries to update app(attempt) before storing it
- [YARN-1824] - Make Windows client work with Linux/Unix cluster
- [YARN-1784] - TestContainerAllocation assumes CapacityScheduler
- [YARN-1846] - TestRM#testNMTokenSentForNormalContainer assumes CapacityScheduler
- [YARN-1839] - Capacity scheduler preempts an AM out. AM attempt 2 fails to launch task container with SecretManager$InvalidToken: No NMToken sent
- [YARN-1821] - NPE on registerNodeManager if the request has containers for UnmanagedAMs
- [YARN-1793] - yarn application -kill doesn't kill UnmanagedAMs
- [YARN-1790] - Fair Scheduler UI not showing apps table
- [YARN-1774] - FS: Submitting to non-leaf queue throws NPE
- [YARN-1785] - FairScheduler treats app lookup failures as ERRORs
- [YARN-1758] - MiniYARNCluster broken post YARN-1666
- [YARN-1398] - Deadlock in capacity scheduler leaf queue and parent queue for getQueueInfo and completedContainer call
- [YARN-713] - ResourceManager can exit unexpectedly if DNS is unavailable
- [YARN-1417] - RM may issue expired container tokens to AM while issuing new containers.
- [YARN-1748] - hadoop-yarn-server-tests packages core-site.xml breaking downstream tests
- [YARN-1760] - TestRMAdminService assumes CapacityScheduler
- [YARN-1528] - Allow setting auth for ZK connections
- [YARN-1429] - *nix: Allow a way for users to augment classpath of YARN daemons
- [YARN-1718] - Fix a couple isTerminals in Fair Scheduler queue placement rules
- [YARN-1724] - Race condition in Fair Scheduler when continuous scheduling is turned on
- [YARN-1721] - When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
- [YARN-1678] - Fair scheduler gabs incessantly about reservations
- [YARN-1689] - RMAppAttempt is not killed when RMApp is at ACCEPTED
- [YARN-1166] - YARN 'appsFailed' metric should be of type 'counter'
- [YARN-1041] - Protocol changes for RM to bind and notify a restarted AM of existing containers
- [YARN-1493] - Schedulers don't recognize apps separately from app-attempts
- [YARN-1470] - Add audience annotation to MiniYARNCluster
- [YARN-1736] - FS: AppSchedulable.assignContainer's priority argument is redundant
- [YARN-1617] - Remove ancient comment and surround LOG.debug in AppSchedulingInfo.allocate
- [YARN-1697] - NodeManager reports negative running containers
- [YARN-1672] - YarnConfiguration is missing a default for yarn.nodemanager.log.retain-seconds
Improvement
- [YARN-2641] - Decommission nodes on -refreshNodes instead of next NM-RM heartbeat
- [YARN-2352] - FairScheduler: Collect metrics on duration of critical methods that affect performance
- [YARN-2274] - FairScheduler: Add debug information about cluster capacity, availability and reservations
- [YARN-2061] - Revisit logging levels in ZKRMStateStore
- [YARN-2089] - FairScheduler: QueuePlacementPolicy and QueuePlacementRule are missing audience annotations
- [YARN-1771] - many getFileStatus calls made from node manager for localizing a public distributed cache resource
- [YARN-1461] - RM API and RM changes to handle tags for running jobs
- [YARN-1171] - Add default queue properties to Fair Scheduler documentation
New Feature
- [YARN-1764] - Handle RM fail overs after the submitApplication call.
- [YARN-1861] - Both RM stuck in standby mode when automatic failover is enabled
- [YARN-1864] - Fair Scheduler Dynamic Hierarchical User Queues
- [YARN-1898] - Standby RM's conf, stacks, logLevel, metrics, jmx and logs links are redirecting to Active RM
- [YARN-1893] - Make ApplicationMasterProtocol#allocate AtMostOnce
- [YARN-1521] - Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
- [YARN-1640] - Manual Failover does not work in secure clusters
- [YARN-1705] - Reset cluster-metrics on transition to standby
- [YARN-1658] - Webservice should redirect to active RM when HA is enabled.
- [YARN-1816] - Succeeded application remains in accepted after RM restart
- [YARN-1812] - Job stays in PREP state for long time after RM Restarts
- [YARN-1766] - When RM does the initiation, it should use loaded Configuration instead of bootstrap configuration.
- [YARN-1641] - ZK store should attempt a write periodically to ensure it is still Active
- [YARN-1811] - RM HA: AM link broken if the AM is on nodes other than RM
- [YARN-1776] - renewDelegationToken should survive RM failover
- [YARN-1525] - Web UI should redirect to active RM when HA is enabled.
- [YARN-986] - RM DT token service should have service addresses of both RMs
- [YARN-1734] - RM should get the updated Configurations when it transits from Standby to Active
- [YARN-1666] - Make admin refreshNodes work across RM failover
- [YARN-1676] - Make admin refreshUserToGroupsMappings of configuration work across RM failover
- [YARN-1459] - RM services should depend on ConfigurationProvider during startup too
- [YARN-1660] - add the ability to set yarn.resourcemanager.hostname.rm-id instead of setting all the various host:port properties for RM
- [YARN-1665] - Set better defaults for HA configs for automatic failover
- [YARN-1669] - Make admin refreshServiceAcls work across RM failover
- [YARN-1667] - Make admin refreshSuperUserGroupsConfiguration work across RM failover
- [YARN-1668] - Make admin refreshAdminAcls work across RM failover
- [YARN-1611] - Make admin refresh of capacity scheduler configuration work across RM failover
- [YARN-1639] - YARM RM HA requires different configs on different RM hosts
- [YARN-1497] - Expose moving apps between queues on the command line
- [YARN-1499] - Fair Scheduler changes for moving apps between queues
- [YARN-1504] - RM changes for moving apps between queues
- [YARN-1498] - Common scheduler changes for moving apps between queues
- [YARN-1496] - Protocol additions to allow moving apps between queues
Task
- [YARN-2566] - DefaultContainerExecutor should pick a working directory randomly
Test
- [YARN-2018] - TestClientRMService.testTokenRenewalWrongUser fails after HADOOP-10562
- [YARN-1970] - Prepare YARN codebase for JUnit 4.11.
- [YARN-1854] - Race condition in TestRMHA#testStartAndTransitions
HDFS
Bug
- [HDFS-7960] - The full block report should prune zombie storages even if they're not empty
- [HDFS-6831] - Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
- [HDFS-7596] - NameNode should prune dead storages from storageMap
- [HDFS-7208] - NN doesn't schedule replication when a DN storage fails
- [HDFS-7575] - Upgrade should generate a unique storage ID for each volume
- [HDFS-7391] - Renable SSLv2Hello in HttpFS
- [HDFS-7235] - DataNode#transferBlock should report blocks that don't exist using reportBadBlock
- [HDFS-7274] - Disable SSLv3 in HttpFS
- [HDFS-7005] - DFS input streams do not timeout
- [HDFS-6376] - Distcp data between two HA clusters requires another configuration
- [HDFS-6621] - Hadoop Balancer prematurely exits iterations
- [HDFS-6776] - Using distcp to copy data between insecure and secure cluster via webdhfs doesn't work
- [HDFS-6908] - incorrect snapshot directory diff generated by snapshot deletion
- [HDFS-6825] - Edit log corruption due to delayed block removal
- [HDFS-6703] - NFS: Files can be deleted from a read-only mount
- [HDFS-6640] - [ Web HDFS ] Syntax for MKDIRS, CREATESYMLINK, and SETXATTR are given wrongly(missed webhdfs/v1).).
- [HDFS-6114] - Block Scan log rolling will never happen if blocks written continuously leading to huge size of dncp_block_verification.log.curr
- [HDFS-6788] - Improve synchronization in BPOfferService with read write lock
- [HDFS-6647] - Edit log corruption when pipeline recovery occurs for deleted file present in snapshot
- [HDFS-6622] - Rename and AddBlock may race and produce invalid edits
- [HDFS-6618] - FSNamesystem#delete drops the FSN lock between removing INodes from the tree and deleting them from the inode map
- [HDFS-6551] - Rename with OVERWRITE option may throw NPE when the target file/directory is a reference INode
- [HDFS-5321] - Clean up the HTTP-related configuration in HDFS
- [HDFS-6475] - WebHdfs clients fail without retry because incorrect handling of StandbyException
- [HDFS-5891] - webhdfs should not try connecting the DN during redirection
- [HDFS-6312] - WebHdfs HA failover is broken on secure clusters
- [HDFS-6197] - Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
- [HDFS-6527] - Edit log corruption due to defered INode removal
- [HDFS-6294] - Use INode IDs to avoid conflicts when a file open for write is renamed
- [HDFS-6549] - Add support for accessing the NFS gateway from the AIX NFS client
- [HDFS-6563] - NameNode cannot save fsimage in certain circumstances when snapshots are in use
- [HDFS-6553] - Add missing DeprecationDeltas for NFS Kerberos configurations
- [HDFS-6439] - NFS should not reject NFS requests to the NULL procedure whether port monitoring is enabled or not
- [HDFS-6500] - Snapshot shouldn't be removed silently after renaming to an existing snapshot
- [HDFS-6379] - HTTPFS - Implement ACLs support
- [HDFS-6325] - Append should fail if the last block has insufficient number of replicas
- [HDFS-5522] - Datanode disk error check may be incorrectly skipped
- [HDFS-6078] - TestIncrementalBlockReports is flaky
- [HDFS-2882] - DN continues to start up, even if block pool fails to initialize
- [HDFS-6245] - datanode fails to start with a bad disk even when failed volumes is set
- [HDFS-6208] - DataNode caching can leak file descriptors.
- [HDFS-4909] - Avoid protocol buffer RPC namespace clashes
- [HDFS-6381] - Fix a typo in INodeReference.java
- [HDFS-6355] - Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner
- [HDFS-6219] - Proxy superuser configuration should use true client IP for address checks
- [HDFS-6056] - Clean up NFS config settings
- [HDFS-5705] - TestSecondaryNameNodeUpgrade#testChangeNsIDFails may fail due to ConcurrentModificationException
- [HDFS-6362] - InvalidateBlocks is inconsistent in usage of DatanodeUuid and StorageID
- [HDFS-6313] - WebHdfs may use the wrong NN when configured for multiple HA NNs
- [HDFS-6077] - running slive with webhdfs on secure HA cluster fails with unkown host exception
- [HDFS-6329] - WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
- [HDFS-6217] - Webhdfs PUT operations may not work via a http proxy
- [HDFS-6214] - Webhdfs has poor throughput for files >2GB
- [HDFS-6215] - Wrong error message for upgrade
- [HDFS-6140] - WebHDFS cannot create a file with spaces in the name after HA failover changes.
- [HDFS-6127] - WebHDFS tokens cannot be renewed in HA setup
- [HDFS-6100] - DataNodeWebHdfsMethods does not failover in HA mode
- [HDFS-6400] - Cannot execute "hdfs oiv_legacy"
- [HDFS-6293] - Issues with OIV processing PB-based fsimages
- [HDFS-6274] - Cleanup javadoc warnings in HDFS code
- [HDFS-6340] - DN can't finalize upgrade
- [HDFS-6130] - NPE when upgrading namenode from fsimages older than -32
- [HDFS-5846] - Assigning DEFAULT_RACK in resolveNetworkLocation method can break data resiliency
- [HDFS-3087] - Decomissioning on NN restart can complete without blocks being replicated
- [HDFS-6089] - Standby NN while transitioning to active throws a connection refused error when the prior active NN process is suspended
- [HDFS-6411] - nfs-hdfs-gateway mount raises I/O error and hangs when a unauthorized user attempts to access it
- [HDFS-6131] - Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
- [HDFS-5840] - Follow-up to HDFS-5138 to improve error handling during partial upgrade failures
- [HDFS-6135] - In HDFS upgrade with HA setup, JournalNode cannot handle layout version bump when rolling back
- [HDFS-6166] - revisit balancer so_timeout
- [HDFS-6181] - Fix the wrong property names in NFS user guide
- [HDFS-6206] - DFSUtil.substituteForWildcardAddress may throw NPE
- [HDFS-6229] - Race condition in failover can cause RetryCache fail to work
- [HDFS-6243] - HA NameNode transition to active or shutdown may leave lingering image transfer thread.
- [HDFS-6361] - TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
- [HDFS-6156] - Simplify the JMX API that provides snapshot information
- [HDFS-6462] - NFS: fsstat request fails with the secure hdfs
- [HDFS-6404] - HttpFS should use a 000 umask for mkdir and create operations
- [HDFS-6402] - Suppress findbugs warning for failure to override equals and hashCode in FsAclPermission.
- [HDFS-6326] - WebHdfs ACL compatibility is broken
- [HDFS-6351] - Command "hdfs dfs -rm -r" can't remove empty directory
- [HDFS-6345] - DFS.listCacheDirectives() should allow filtering based on cache directive ID
- [HDFS-6288] - DFSInputStream Pread doesn't update ReadStatistics
- [HDFS-6230] - Expose upgrade status through NameNode web UI
- [HDFS-6289] - HA failover can fail if there are pending DN messages for DNs which no longer exist
- [HDFS-6094] - The same block can be counted twice towards safe mode threshold
- [HDFS-6160] - TestSafeMode occasionally fails
- [HDFS-6153] - Document "fileId" and "childrenNum" fields in the FileStatus Json schema
- [HDFS-6180] - dead node count / listing is very broken in JMX and old GUI
- [HDFS-6232] - OfflineEditsViewer throws a NPE on edits containing ACL modifications
- [HDFS-6231] - DFSClient hangs infinitely if using hedged reads and all eligible datanodes die.
- [HDFS-5591] - Checkpointing should use monotonic time when calculating period
- [HDFS-6068] - Disallow snapshot names that are also invalid directory names
- [HDFS-6112] - NFS Gateway docs are incorrect for allowed hosts configuration
- [HDFS-6106] - Reduce default for dfs.namenode.path.based.cache.refresh.interval.ms
- [HDFS-6107] - When a block can't be cached due to limited space on the DataNode, that block becomes uncacheable
- [HDFS-5944] - LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/ right and cause SecondaryNameNode failed do checkpoint
- [HDFS-6102] - Lower the default maximum items per directory to fix PB fsimage loading
- [HDFS-6097] - zero-copy reads are incorrectly disabled on file offsets above 2GB
- [HDFS-6079] - Timeout for getFileBlockStorageLocations does not work
- [HDFS-6063] - TestAclCLI fails intermittently when running test 24: copyFromLocal
- [HDFS-6065] - HDFS zero-copy reads should return null on EOF when doing ZCR
- [HDFS-6067] - TestPread.testMaxOutHedgedReadPool is flaky
- [HDFS-6053] - Fix TestDecommissioningStatus and TestDecommission in branch-2
- [HDFS-6039] - Uploading a File under a Dir with default acls throws "Duplicated ACLFeature"
- [HDFS-5821] - TestHDFSCLI fails for user names with the dash character
- [HDFS-6028] - Print clearer error message when user attempts to delete required mask entry from ACL.
- [HDFS-5064] - Standby checkpoints should not block concurrent readers
- [HDFS-6071] - BlockReaderLocal doesn't return -1 on EOF when doing a zero-length read on a short file
- [HDFS-6047] - TestPread NPE inside in DFSInputStream hedgedFetchBlockByteRange
- [HDFS-6059] - TestBlockReaderLocal fails if native library is not available
- [HDFS-6057] - DomainSocketWatcher.watcherThread should be marked as daemon thread
- [HDFS-6051] - HDFS cannot run on Windows since short-circuit shared memory segment changes.
- [HDFS-5339] - WebHDFS URI does not accept logical nameservices when security is enabled
- [HDFS-6033] - PBImageXmlWriter incorrectly handles processing cache directives
- [HDFS-3969] - Small bug fixes and improvements for disk locations API
- [HDFS-6008] - Namenode dead node link is giving HTTP error 500
- [HDFS-5981] - PBImageXmlWriter generates malformed XML
- [HDFS-5962] - Mtime and atime are not persisted for symbolic links
- [HDFS-5742] - DatanodeCluster (mini cluster of DNs) fails to start
- [HDFS-5961] - OIV cannot load fsimages containing a symbolic link
- [HDFS-5759] - Web UI does not show up during the period of loading FSImage
- [HDFS-5716] - Allow WebHDFS to use pluggable authentication filter
- [HDFS-5943] - 'dfs.namenode.https-address.ns1' property is not used in federation setup
- [HDFS-5941] - add dfs.namenode.secondary.https-address and dfs.namenode.secondary.https-address in hdfs-default.xml
- [HDFS-5879] - Some TestHftpFileSystem tests do not close streams
- [HDFS-4858] - HDFS DataNode to NameNode RPC should timeout
- [HDFS-5886] - Potential null pointer deference in RpcProgramNfs3#readlink()
- [HDFS-5881] - Fix skip() of the short-circuit local reader (legacy).
- [HDFS-5791] - TestHttpsFileSystem should use a random port to avoid binding error during testing
- [HDFS-5828] - BlockPlacementPolicyWithNodeGroup can place multiple replicas on the same node group when dfs.namenode.avoid.write.stale.datanode is true
- [HDFS-5856] - DataNode.checkDiskError might throw NPE
- [HDFS-5843] - DFSClient.getFileChecksum() throws IOException if checksum is disabled
- [HDFS-5492] - Port HDFS-2069 (Incorrect default trash interval in the docs) to trunk
- [HDFS-5922] - DN heartbeat thread can get stuck in tight loop
- [HDFS-5982] - Need to update snapshot manager when applying editlog for deleting a snapshottable directory
- [HDFS-5988] - Bad fsimage always generated after upgrade
- [HDFS-5959] - Fix typo at section name in FSImageFormatProtobuf.java
- [HDFS-5915] - Refactor FSImageFormatProtobuf to simplify cross section reads
- [HDFS-5803] - TestBalancer.testBalancer0 fails
- [HDFS-5780] - TestRBWBlockInvalidation times out intemittently on branch-2
- [HDFS-5913] - Nfs3Utils#getWccAttr() should check attr parameter against null
- [HDFS-5904] - TestFileStatus fails intermittently on trunk and branch2
- [HDFS-5807] - TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails intermittently on Branch-2
- [HDFS-5859] - DataNode#checkBlockToken should check block tokens even if security is not enabled
- [HDFS-5901] - NameNode new UI doesn't support IE8 and IE9 on windows 7
- [HDFS-5934] - New Namenode UI back button doesn't work as expected
- [HDFS-5949] - New Namenode UI when trying to download a file, the browser doesn't know the file name
- [HDFS-5948] - TestBackupNode flakes with port in use error
- [HDFS-3128] - Unit tests should not use a test root in /tmp
- [HDFS-5900] - Cannot set cache pool limit of "unlimited" via CacheAdmin
- [HDFS-5138] - Support HDFS upgrade in HA
- [HDFS-5895] - HDFS cacheadmin -listPools has exit_code of 1 when the command returns 0 result.
- [HDFS-5767] - NFS implementation assumes userName userId mapping to be unique, which is not true sometimes
- [HDFS-5790] - LeaseManager.findPath is very slow when many leases need recovery
- [HDFS-5685] - DistCp will fail to copy with -delete switch
- [HDFS-2264] - NamenodeProtocol has the wrong value for clientPrincipal in KerberosInfo annotation
Improvement
- [HDFS-7217] - Better batching of IBRs
- [HDFS-7278] - Add a command that allows sysadmins to manually trigger full block reports from a DN
- [HDFS-6529] - Trace logging for RemoteBlockReader2 to identify remote datanode and file being read
- [HDFS-6007] - Update documentation about short-circuit local reads
- [HDFS-6403] - Add metrics for log warnings reported by JVM pauses
- [HDFS-6499] - Use NativeIO#renameTo instead of File#renameTo in FileJournalManager
- [HDFS-6268] - Better sorting in NetworkTopology#pseudoSortByDistance when no local node is found
- [HDFS-5986] - Capture the number of blocks pending deletion on namenode webUI
- [HDFS-6084] - Namenode UI - "Hadoop" logo link shouldn't go to hadoop homepage
- [HDFS-4052] - BlockManager#invalidateWork should print logs outside the lock
- [HDFS-6072] - Clean up dead code of FSImage
- [HDFS-6070] - Cleanup use of ReadStatistics in DFSInputStream
- [HDFS-6080] - Improve NFS gateway performance by making rtmax and wtmax configurable
- [HDFS-5167] - Add metrics about the NameNode retry cache
- [HDFS-6186] - Pause deletion of blocks when the namenode starts up
- [HDFS-6050] - NFS does not handle exceptions correctly in a few places
- [HDFS-5196] - Provide more snapshot information in WebUI
- [HDFS-6304] - Consolidate the logic of path resolution in FSDirectory
- [HDFS-5683] - Better audit log messages for caching operations
- [HDFS-6295] - Add "decommissioning" state and node state filtering to dfsadmin
- [HDFS-6191] - Disable quota checks when replaying edit log.
- [HDFS-6086] - Fix a case where zero-copy or no-checksum reads were not allowed even when the block was cached
- [HDFS-6085] - Improve CacheReplicationMonitor log messages a bit
- [HDFS-6069] - Quash stack traces when ACLs are disabled
- [HDFS-6044] - Add property for setting the NFS look up time for users
- [HDFS-6043] - Give HDFS daemons NFS3 and Portmap their own OPTS
- [HDFS-5956] - A file size is multiplied by the replication factor in 'hdfs oiv -p FileDistribution' option
- [HDFS-3405] - Checkpointing should use HTTP POST or PUT instead of GET-GET to send merged fsimages
- [HDFS-5776] - Support 'hedged' reads in DFSClient
- [HDFS-6061] - Allow dfs.datanode.shared.file.descriptor.path to contain multiple entries and fall back when needed
- [HDFS-6046] - add dfs.client.mmap.enabled
- [HDFS-6040] - fix DFSClient issue without libhadoop.so and some other ShortCircuitShm cleanups
- [HDFS-5950] - The DFSClient and DataNode should use shared memory segments to communicate short-circuit information
- [HDFS-4200] - Reduce the size of synchronized sections in PacketResponder
- [HDFS-5939] - WebHdfs returns misleading error code and logs nothing if trying to create a file with no DNs in cluster
- [HDFS-6018] - Exception recorded in LOG when IPCLoggerChannel#close is called
- [HDFS-6006] - Remove duplicate code in FSNameSystem#getFileInfo
- [HDFS-5935] - New Namenode UI FS browser should throw smarter error messages
- [HDFS-5973] - add DomainSocket#shutdown method
- [HDFS-5868] - Make hsync implementation pluggable
- [HDFS-5318] - Support read-only and read-write paths to shared replicas
- [HDFS-5942] - Fix javadoc in OfflineImageViewer
- [HDFS-5929] - Add Block pool % usage to HDFS federated nn page
- [HDFS-5781] - Use an array to record the mapping between FSEditLogOpCode and the corresponding byte value
- [HDFS-5153] - Datanode should send block reports for each storage in a separate message
- [HDFS-5979] - Typo and logger fix for fsimage PB code
- [HDFS-5847] - Consolidate INodeReference into a separate section
- [HDFS-5698] - Use protobuf to serialize / deserialize FSImage
- [HDFS-5775] - Consolidate the code for serialization in CacheManager
- [HDFS-5768] - Consolidate the serialization code in DelegationTokenSecretManager
- [HDFS-5726] - Fix compilation error in AbstractINodeDiff for JDK7
- [HDFS-5632] - Add Snapshot feature to INodeDirectory
- [HDFS-5647] - Merge INodeDirectory.Feature and INodeFile.Feature
- [HDFS-5554] - Add Snapshot Feature to INodeFile
- [HDFS-5537] - Remove FileWithSnapshot interface
- [HDFS-5286] - Flatten INodeDirectory hierarchy: add DirectoryWithQuotaFeature
- [HDFS-5285] - Flatten INodeFile hierarchy: Add UnderContruction Feature
- [HDFS-5531] - Combine the getNsQuota() and getDsQuota() methods in INode
- [HDFS-4370] - Fix typo Blanacer in DataNode
- [HDFS-5940] - Minor cleanups to ShortCircuitReplica, FsDatasetCache, and DomainSocketWatcher
- [HDFS-5938] - Make BlockReaderFactory#BlockReaderPeer a static class
- [HDFS-5810] - Unify mmap cache and short-circuit file descriptor cache
- [HDFS-4911] - Reduce PeerCache timeout to be commensurate with dfs.datanode.socket.reuse.keepalive
- [HDFS-5746] - add ShortCircuitSharedMemorySegment
- [HDFS-5709] - Improve NameNode upgrade with existing reserved paths and path components
- [HDFS-4817] - make HDFS advisory caching configurable on a per-file basis
- [HDFS-3910] - DFSTestUtil#waitReplication should timeout
New Feature
- [HDFS-4257] - The ReplaceDatanodeOnFailure policies could have a forgiving option
- [HDFS-5638] - HDFS implementation of FileContext API for ACLs.
- [HDFS-6076] - SimulatedDataSet should not create DatanodeRegistration with namenode layout version and type
- [HDFS-6435] - Add support for specifying a static uid/gid mapping for the NFS gateway
- [HDFS-6406] - Add capability for NFS gateway to reject connections from unprivileged ports
- [HDFS-6281] - Provide option to use the NFS Gateway without having to use the Hadoop portmapper
- [HDFS-5898] - Allow NFS gateway to login/relogin from its kerberos keytab
- [HDFS-6060] - NameNode should not check DataNode layout version
- [HDFS-6041] - Downgrade/Finalize should rename the rollback image instead of purging it
- [HDFS-6042] - Fix rolling upgrade documentation and error messges
- [HDFS-6013] - add rollingUpgrade information to latest UI
- [HDFS-6034] - DN registration should use DataNodeLayoutVersion instead of NameNodeLayoutVersion
- [HDFS-6031] - NN running newer software rejects loading the fsimage during rolling upgrade.
- [HDFS-6032] - -rollingUpgrade query hits NPE after the NN restarts
- [HDFS-6029] - Secondary NN fails to checkpoint after -rollingUpgrade prepare
- [HDFS-6024] - Test whether the NN will reject the downgrade if it has a fsimage from a newer release
- [HDFS-6026] - Fix TestDFSUpgrade and TestDataNodeRollingUpgrade
- [HDFS-6023] - Test whether the standby NN continues to checkpoint after the prepare stage
- [HDFS-5535] - Umbrella jira for improved HDFS rolling upgrades
- [HDFS-4685] - Implementation of ACLs in HDFS
- [HDFS-5483] - NN should gracefully handle multiple block replicas on same DN
- [HDFS-5804] - HDFS NFS Gateway fails to mount and proxy when using Kerberos
- [HDFS-5511] - improve CacheManipulator interface to allow better unit testing
- [HDFS-1623] - High Availability Framework for HDFS NN
Task
- [HDFS-6030] - Remove an unused constructor in INode.java
- [HDFS-6025] - Update findbugsExcludeFile.xml
Test
- [HDFS-6419] - TestBookKeeperHACheckpoints#TestSBNCheckpoints fails on trunk
- [HDFS-6287] - Add vecsum test of libhdfs read access times
- [HDFS-6224] - Add a unit test to TestAuditLogger for file permissions passed to logAuditEvent
- [HDFS-5936] - MiniDFSCluster does not clean data left behind by SecondaryNameNode.
- [HDFS-5953] - TestBlockReaderFactory fails if libhadoop.so has not been built
- [HDFS-5882] - TestAuditLogs is flaky