TaskTracker (Hadoop 0.20.2+737 API)
Date: 2010-10-11

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.apache.hadoop.mapred
Class TaskTracker

    java.lang.Object
      org.apache.hadoop.mapred.TaskTracker

All Implemented Interfaces:
  Runnable, VersionedProtocol, TaskUmbilicalProtocol

    public class TaskTracker

extends Object
implements TaskUmbilicalProtocol, Runnable

TaskTracker is a process that starts and tracks MR Tasks in a networked environment. It contacts the JobTracker for Task assignments and reporting results.

Nested Class Summary
`static class` | `TaskTracker.MapOutputServlet` This class is used in TaskTracker's Jetty to serve the map outputs to other nodes.

Field Summary
`static org.apache.commons.logging.Log` | `ClientTraceLog`
`static long` | `COUNTER_UPDATE_INTERVAL`
`static int` | `FILE_NOT_FOUND`
`static String` | `FOR_REDUCE_TASK` The reduce task number for which this map output is being transferred
`static String` | `FROM_MAP_TASK` The map task from which the map output data is being transferred
`static int` | `HEARTBEAT_INTERVAL_MIN`
`static org.apache.commons.logging.Log` | `LOG`
`static String` | `MAP_OUTPUT_LENGTH` The custom http header used for the map output length.
`static String` | `MR_CLIENTTRACE_FORMAT`
`static String` | `RAW_MAP_OUTPUT_LENGTH` The custom http header used for the "raw" map output length.
`static String` | `SUBDIR`
`static int` | `SUCCESS`
`static String` | `TT_KEYTAB_FILE`
`static String` | `TT_USER_NAME`
`static String` | `WORKDIR`

Fields inherited from interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
`versionID`

Constructor Summary
`TaskTracker(JobConf conf)` Start with the local machine name, and the default JobTracker

Method Summary
`boolean` | `canCommit(TaskAttemptID taskid)` Child checking whether it can commit
`void` | `cleanupStorage()` Deprecated.
`void` | `close()` Close down the TaskTracker and all its components.
`void` | `commitPending(TaskAttemptID taskid, TaskStatus taskStatus)` Task is reporting that it is in commit_pending and it is waiting for the commit Response
`void` | `done(TaskAttemptID taskid)` The task is done.
`void` | `fatalError(TaskAttemptID taskId, String msg)` A child task had a fatal error.
`void` | `fsError(TaskAttemptID taskId, String message)` A child task had a local filesystem error.
`static Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation>` | `getInstrumentationClass(Configuration conf)`
`static String` | `getJobCacheSubdir(String user)`
`org.apache.hadoop.mapred.InterTrackerProtocol` | `getJobClient()` The connection to the JobTracker, used by the TaskRunner for locating remote files.
`org.apache.hadoop.mapred.JvmManager` | `getJvmManagerInstance()`
`static String` | `getLocalJobDir(String user, String jobid)`
`static String` | `getLocalTaskDir(String user, String jobid, String taskid, boolean isCleanupAttempt)`
`MapTaskCompletionEventsUpdate` | `getMapCompletionEvents(JobID jobId, int fromEventId, int maxLocs, TaskAttemptID id)` Called by a reduce task to get the map output locations for finished maps.
`static String` | `getPrivateDistributedCacheDir(String user)`
`long` | `getProtocolVersion(String protocol, long clientVersion)` Return protocol version corresponding to protocol interface.
`static String` | `getPublicDistributedCacheDir()`
`JvmTask` | `getTask(org.apache.hadoop.mapred.JvmContext context)` Called upon startup by the child process, to fetch Task data.
`TaskController` | `getTaskController()`
`org.apache.hadoop.mapred.TaskMemoryManagerThread` | `getTaskMemoryManager()`
`org.apache.hadoop.mapred.TaskTrackerInstrumentation` | `getTaskTrackerInstrumentation()`
`InetSocketAddress` | `getTaskTrackerReportAddress()` Return the port at which the tasktracker bound to
`static String` | `getUserDir(String user)`
`boolean` | `isIdle()` Is this task tracker idle?
`boolean` | `isTaskMemoryManagerEnabled()` Is the TaskMemoryManager Enabled on this system?
`static void` | `main(String[] argv)` Start the TaskTracker, point toward the indicated JobTracker
`void` | `mapOutputLost(TaskAttemptID taskid, String errorMsg)` A completed map task's output has been lost.
`boolean` | `ping(TaskAttemptID taskid)` Child checking to see if we're alive.
`void` | `reportDiagnosticInfo(TaskAttemptID taskid, String info)` Called when the task dies before completion, and we want to report back diagnostic info
`void` | `reportNextRecordRange(TaskAttemptID taskid, org.apache.hadoop.mapred.SortedRanges.Range range)` Report the record range which is going to process next by the Task.
`void` | `run()` The server retry loop.
`static void` | `setInstrumentationClass(Configuration conf, Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> t)`
`void` | `shuffleError(TaskAttemptID taskId, String message)` A reduce-task failed to shuffle the map-outputs.
`void` | `shutdown()`
`boolean` | `statusUpdate(TaskAttemptID taskid, TaskStatus taskStatus)` Called periodically to report Task progress, from 0.0 to 1.0.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### LOG

    public static final org.apache.commons.logging.Log LOG

### MR_CLIENTTRACE_FORMAT

    public static final String MR_CLIENTTRACE_FORMAT

See Also:
  Constant Field Values

### ClientTraceLog

    public static final org.apache.commons.logging.Log ClientTraceLog

### SUBDIR

    public static final String SUBDIR

See Also:
  Constant Field Values

### TT_USER_NAME

    public static final String TT_USER_NAME

See Also:
  Constant Field Values

### TT_KEYTAB_FILE

    public static final String TT_KEYTAB_FILE

See Also:
  Constant Field Values

### HEARTBEAT_INTERVAL_MIN

    public static final int HEARTBEAT_INTERVAL_MIN

See Also:
  Constant Field Values

### COUNTER_UPDATE_INTERVAL

    public static final long COUNTER_UPDATE_INTERVAL

See Also:
  Constant Field Values

### SUCCESS

    public static final int SUCCESS

See Also:
  Constant Field Values

### FILE_NOT_FOUND

    public static final int FILE_NOT_FOUND

See Also:
  Constant Field Values

### MAP_OUTPUT_LENGTH

    public static final String MAP_OUTPUT_LENGTH

  The custom http header used for the map output length.

  See Also:
    Constant Field Values

### RAW_MAP_OUTPUT_LENGTH

    public static final String RAW_MAP_OUTPUT_LENGTH

  The custom http header used for the "raw" map output length.

  See Also:
    Constant Field Values

### FROM_MAP_TASK

    public static final String FROM_MAP_TASK

  The map task from which the map output data is being transferred

  See Also:
    Constant Field Values

### FOR_REDUCE_TASK

    public static final String FOR_REDUCE_TASK

  The reduce task number for which this map output is being transferred

  See Also:
    Constant Field Values

### WORKDIR

    public static final String WORKDIR

See Also:
  Constant Field Values

Constructor Detail

### TaskTracker

    public TaskTracker(JobConf conf)
                throws IOException,
                       InterruptedException

  Start with the local machine name, and the default JobTracker

  Throws:
    `IOException`
    `InterruptedException`

Method Detail

### getTaskTrackerInstrumentation

    public org.apache.hadoop.mapred.TaskTrackerInstrumentation getTaskTrackerInstrumentation()

### getTaskController

    public TaskController getTaskController()

### getUserDir

    public static String getUserDir(String user)

### getPrivateDistributedCacheDir

    public static String getPrivateDistributedCacheDir(String user)

### getPublicDistributedCacheDir

    public static String getPublicDistributedCacheDir()

### getJobCacheSubdir

    public static String getJobCacheSubdir(String user)

### getLocalJobDir

    public static String getLocalJobDir(String user,
                                        String jobid)

### getLocalTaskDir

    public static String getLocalTaskDir(String user,
                                         String jobid,
                                         String taskid,
                                         boolean isCleanupAttempt)

### getProtocolVersion

    public long getProtocolVersion(String protocol,
                                   long clientVersion)
                            throws IOException

  Description copied from interface: `VersionedProtocol`
  Return protocol version corresponding to protocol interface.

  Specified by:
    `getProtocolVersion` in interface `VersionedProtocol`
  Parameters:
    `protocol` - The classname of the protocol interface
    `clientVersion` - The version of the protocol that the client speaks
  Returns:
    the version that the server will speak
  Throws:
    `IOException`

### getInstrumentationClass

    public static Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> getInstrumentationClass(Configuration conf)

### setInstrumentationClass

    public static void setInstrumentationClass(Configuration conf,
                                               Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> t)

### cleanupStorage

    @Deprecated
    public void cleanupStorage()
                        throws IOException

  Deprecated.

  Removes all contents of temporary storage. Called upon startup, to remove any leftovers from previous run. Use MRAsyncDiskService.moveAndDeleteAllVolumes instead.

  Throws:
    `IOException`
  See Also:
    `org.apache.hadoop.mapreduce.util.MRAsyncDiskService#cleanupAllVolumes()`

### shutdown

    public void shutdown()
                  throws IOException

  Throws:
    `IOException`

### close

    public void close()
               throws IOException

  Close down the TaskTracker and all its components. We must also shutdown any running tasks or threads, and cleanup disk space. A new TaskTracker within the same process space might be restarted, so everything must be clean.

  Throws:
    `IOException`

### getJobClient

    public org.apache.hadoop.mapred.InterTrackerProtocol getJobClient()

  The connection to the JobTracker, used by the TaskRunner for locating remote files.

### getTaskTrackerReportAddress

    public InetSocketAddress getTaskTrackerReportAddress()

  Return the port at which the tasktracker bound to

### getJvmManagerInstance

    public org.apache.hadoop.mapred.JvmManager getJvmManagerInstance()

### run

    public void run()

  The server retry loop. This while-loop attempts to connect to the JobTracker. It only loops when the old TaskTracker has gone bad (its state is stale somehow) and we need to reinitialize everything.

  Specified by:
    `run` in interface `Runnable`

### getTask

    public JvmTask getTask(org.apache.hadoop.mapred.JvmContext context)
                    throws IOException

  Called upon startup by the child process, to fetch Task data.

  Specified by:
    `getTask` in interface `TaskUmbilicalProtocol`
  Parameters:
    `context` - the JvmContext of the JVM w.r.t the TaskTracker that launched it
  Returns:
    Task object
  Throws:
    `IOException`

### statusUpdate

    public boolean statusUpdate(TaskAttemptID taskid,
                                TaskStatus taskStatus)
                         throws IOException

  Called periodically to report Task progress, from 0.0 to 1.0.

  Specified by:
    `statusUpdate` in interface `TaskUmbilicalProtocol`
  Parameters:
    `taskid` - task-id of the child
    `taskStatus` - status of the child
  Returns:
    True if the task is known
  Throws:
    `IOException`

### reportDiagnosticInfo

    public void reportDiagnosticInfo(TaskAttemptID taskid,
                                     String info)
                              throws IOException

  Called when the task dies before completion, and we want to report back diagnostic info

  Specified by:
    `reportDiagnosticInfo` in interface `TaskUmbilicalProtocol`
  Parameters:
    `taskid` - the id of the task involved
    `info` - the text to report
  Throws:
    `IOException`

### reportNextRecordRange

    public void reportNextRecordRange(TaskAttemptID taskid,
                                      org.apache.hadoop.mapred.SortedRanges.Range range)
                               throws IOException

  Description copied from interface: `TaskUmbilicalProtocol`
  Report the record range which is going to process next by the Task.

  Specified by:
    `reportNextRecordRange` in interface `TaskUmbilicalProtocol`
  Parameters:
    `taskid` - the id of the task involved
    `range` - the range of record sequence nos
  Throws:
    `IOException`

### ping

    public boolean ping(TaskAttemptID taskid)
                 throws IOException

  Child checking to see if we're alive. Normally does nothing.

  Specified by:
    `ping` in interface `TaskUmbilicalProtocol`
  Returns:
    True if the task is known
  Throws:
    `IOException`

### commitPending

    public void commitPending(TaskAttemptID taskid,
                              TaskStatus taskStatus)
                       throws IOException

  Task is reporting that it is in commit_pending and it is waiting for the commit Response

  Specified by:
    `commitPending` in interface `TaskUmbilicalProtocol`
  Parameters:
    `taskid` - task's id
    `taskStatus` - status of the child
  Throws:
    `IOException`

### canCommit

    public boolean canCommit(TaskAttemptID taskid)
                      throws IOException

  Child checking whether it can commit

  Specified by:
    `canCommit` in interface `TaskUmbilicalProtocol`
  Returns:
    true/false
  Throws:
    `IOException`

### done

    public void done(TaskAttemptID taskid)
              throws IOException

  The task is done.

  Specified by:
    `done` in interface `TaskUmbilicalProtocol`
  Parameters:
    `taskid` - task's id
  Throws:
    `IOException`

### shuffleError

    public void shuffleError(TaskAttemptID taskId,
                             String message)
                      throws IOException

  A reduce-task failed to shuffle the map-outputs. Kill the task.

  Specified by:
    `shuffleError` in interface `TaskUmbilicalProtocol`
  Throws:
    `IOException`

### fsError

    public void fsError(TaskAttemptID taskId,
                        String message)
                 throws IOException

  A child task had a local filesystem error. Kill the task.

  Specified by:
    `fsError` in interface `TaskUmbilicalProtocol`
  Throws:
    `IOException`

### fatalError

    public void fatalError(TaskAttemptID taskId,
                           String msg)
                    throws IOException

  A child task had a fatal error. Kill the task.

  Specified by:
    `fatalError` in interface `TaskUmbilicalProtocol`
  Throws:
    `IOException`

### getMapCompletionEvents

    public MapTaskCompletionEventsUpdate getMapCompletionEvents(JobID jobId,
                                                                int fromEventId,
                                                                int maxLocs,
                                                                TaskAttemptID id)
                                                         throws IOException

  Description copied from interface: `TaskUmbilicalProtocol`
  Called by a reduce task to get the map output locations for finished maps. Returns an update centered around the map-task-completion-events. The update also piggybacks the information whether the events copy at the task-tracker has changed or not. This will trigger some action at the child-process.

  Specified by:
    `getMapCompletionEvents` in interface `TaskUmbilicalProtocol`
  Parameters:
    `jobId` - the reducer job id
    `fromEventId` - the index starting from which the locations should be fetched
    `maxLocs` - the max number of locations to fetch
    `id` - The attempt id of the task that is trying to communicate
  Returns:
    A `MapTaskCompletionEventsUpdate`
  Throws:
    `IOException`

### mapOutputLost

    public void mapOutputLost(TaskAttemptID taskid,
                              String errorMsg)
                       throws IOException

  A completed map task's output has been lost.

  Throws:
    `IOException`

### isIdle

    public boolean isIdle()

  Is this task tracker idle?

  Returns:
    has this task tracker finished and cleaned up all of its tasks?

### main

    public static void main(String[] argv)
                     throws Exception

  Start the TaskTracker, point toward the indicated JobTracker

  Throws:
    `Exception`

### isTaskMemoryManagerEnabled

    public boolean isTaskMemoryManagerEnabled()

  Is the TaskMemoryManager Enabled on this system?

  Returns:
    true if enabled, false otherwise.

### getTaskMemoryManager

    public org.apache.hadoop.mapred.TaskMemoryManagerThread getTaskMemoryManager()

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2009 The Apache Software Foundation