MapReduce Commands Guide

Overview

MapReduce commands are invoked by the bin/mapred script. Running the script without any arguments prints the description for all commands.

Usage: mapred [--config confdir] COMMAND

MapReduce has an option parsing framework that employs parsing generic options as well as running classes.

COMMAND_OPTIONS Description
--config confdir Overwrites the default Configuration directory. Default is ${HADOOP_PREFIX}/conf.
COMMAND COMMAND_OPTIONS Various commands with their options are described in the following sections. The commands have been grouped into User Commands and Administration Commands.

User Commands

Commands useful for users of a hadoop cluster.

pipes

Runs a pipes job.

Usage: mapred pipes [-conf <path>] [-jobconf <key=value>, <key=value>, ...] [-input <path>] [-output <path>] [-jar <jar file>] [-inputformat <class>] [-map <class>] [-partitioner <class>] [-reduce <class>] [-writer <class>] [-program <executable>] [-reduces <num>]

COMMAND_OPTION Description
-conf path Configuration for job
-jobconf key=value, key=value, ... Add/override configuration for job
-input path Input directory
-output path Output directory
-jar jar file Jar filename
-inputformat class InputFormat class
-map class Java Map class
-partitioner class Java Partitioner
-reduce class Java Reduce class
-writer class Java RecordWriter
-program executable Executable URI
-reduces num Number of reduces

job

Command to interact with Map Reduce Jobs.

Usage: mapred job | [GENERIC_OPTIONS] | [-submit <job-file>] | [-status <job-id>] | [-counter <job-id> <group-name> <counter-name>] | [-kill <job-id>] | [-events <job-id> <from-event-#> <#-of-events>] | [-history [all] <jobHistoryFile|jobId> [-outfile <file>] [-format <human|json>]] | [-list [all]] | [-kill-task <task-id>] | [-fail-task <task-id>] | [-set-priority <job-id> <priority>] | [-config <job-id> <file>]

COMMAND_OPTION Description
-submit job-file Submits the job.
-status job-id Prints the map and reduce completion percentage and all job counters.
-counter job-id group-name counter-name Prints the counter value.
-kill job-id Kills the job.
-events job-id from-event-# #-of-events Prints the events' details received by jobtracker for the given range.
-history [all] jobHistoryFile|jobId [-outfile file] [-format human|json] Prints job details, failed and killed task details. More details about the job such as successful tasks, task attempts made for each task, task counters, etc can be viewed by specifying the [all] option. An optional file output path (instead of stdout) can be specified. The format defaults to human-readable but can also be changed to JSON with the [-format] option.
-list [all] Displays jobs which are yet to complete. -list all displays all jobs.
-kill-task task-id Kills the task. Killed tasks are NOT counted against failed attempts.
-fail-task task-id Fails the task. Failed tasks are counted against failed attempts.
-set-priority job-id priority Changes the priority of the job. Allowed priority values are VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW
-config job-id file Download the job configuration file.

queue

command to interact and view Job Queue information

Usage: mapred queue [-list] | [-info <job-queue-name> [-showJobs]] | [-showacls]

COMMAND_OPTION Description
-list Gets list of Job Queues configured in the system. Along with scheduling information associated with the job queues.
-info job-queue-name [-showJobs] Displays the job queue information and associated scheduling information of particular job queue. If -showJobs options is present a list of jobs submitted to the particular job queue is displayed.
-showacls Displays the queue name and associated queue operations allowed for the current user. The list consists of only those queues to which the user has access.

classpath

Prints the class path needed to get the Hadoop jar and the required libraries.

Usage: mapred classpath

distcp

Copy file or directories recursively. More information can be found at Hadoop DistCp Guide.

archive

Creates a hadoop archive. More information can be found at Hadoop Archives Guide.

archive-logs

A tool to combine YARN aggregated logs into Hadoop archives to reduce the number of files in HDFS. More information can be found at Hadoop Archive Logs Guide.

Administration Commands

Commands useful for administrators of a hadoop cluster.

historyserver

Start JobHistoryServer.

Usage: mapred historyserver

hsadmin

Runs a MapReduce hsadmin client for execute JobHistoryServer administrative commands.

Usage: mapred hsadmin [-refreshUserToGroupsMappings] | [-refreshSuperUserGroupsConfiguration] | [-refreshAdminAcls] | [-refreshLoadedJobCache] | [-refreshLogRetentionSettings] | [-refreshJobRetentionSettings] | [-getGroups [username]] | [-help [cmd]]

COMMAND_OPTION Description
-refreshUserToGroupsMappings Refresh user-to-groups mappings
-refreshSuperUserGroupsConfiguration Refresh superuser proxy groups mappings
-refreshAdminAcls Refresh acls for administration of Job history server
-refreshLoadedJobCache Refresh loaded job cache of Job history server
-refreshJobRetentionSettings Refresh job history period, job cleaner settings
-refreshLogRetentionSettings Refresh log retention period and log retention check interval
-getGroups [username] Get the groups which given user belongs to
-help [cmd] Displays help for the given command or all commands if none is specified.