org.apache.hadoop.hive.ql.exec.mr
Class ExecDriver

java.lang.Object
  extended by org.apache.hadoop.hive.ql.exec.Task<MapredWork>
      extended by org.apache.hadoop.hive.ql.exec.mr.ExecDriver
All Implemented Interfaces:
Serializable, HadoopJobExecHook, Node
Direct Known Subclasses:
MapRedTask

public class ExecDriver
extends Task<MapredWork>
implements Serializable, HadoopJobExecHook

ExecDriver is the central class in co-ordinating execution of any map-reduce task. It's main responsabilities are: - Converting the plan (MapredWork) into a MR Job (JobConf) - Submitting a MR job to the cluster via JobClient and ExecHelper - Executing MR job in local execution mode (where applicable)

See Also:
Serialized Form

Nested Class Summary
 
Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Task
Task.FeedType
 
Field Summary
protected  JobConf job
           
protected  HadoopJobExecHelper jobExecHelper
           
protected static org.apache.commons.logging.Log LOG
           
static MemoryMXBean memoryMXBean
           
 
Fields inherited from class org.apache.hadoop.hive.ql.exec.Task
BACKUP_COMMON_JOIN, backupChildrenTasks, backupTask, childTasks, clonedConf, COMMON_JOIN, conf, console, CONVERTED_LOCAL_MAPJOIN, CONVERTED_MAPJOIN, CONVERTED_SORTMERGEJOIN, db, driverContext, feedSubscribers, id, initialized, isdone, jobID, LOCAL_MAPJOIN, MAPJOIN_ONLY_NOBACKUP, NO_TAG, parentTasks, queryPlan, queued, started, taskCounters, taskHandle, taskTag, work
 
Constructor Summary
ExecDriver()
          Constructor when invoked from QL.
ExecDriver(MapredWork plan, JobConf job, boolean isSilent)
          Constructor/Initialization for invocation as independent utility.
 
Method Summary
 boolean checkFatalErrors(Counters ctrs, StringBuilder errMsg)
          Fatal errors are those errors that cannot be recovered by retries.
 int execute(DriverContext driverContext)
          Execute a query plan using Hadoop.
static String generateCmdLine(HiveConf hconf, Context ctx)
          Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver.
 String getName()
          Gets the name of the node.
 Collection<Operator<? extends OperatorDesc>> getTopOperators()
           
 StageType getType()
          Should be overridden to return the type of the specific task among the types in StageType.
 boolean hasReduce()
           
 void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext driverContext)
          Initialization when invoked from QL.
 boolean isMapRedTask()
           
 void logPlanProgress(SessionState ss)
           
static void main(String[] args)
           
 boolean mapDone()
           
 boolean mapStarted()
           
 boolean reduceDone()
           
 boolean reduceStarted()
           
 boolean requireLock()
           
protected  void setInputAttributes(Configuration conf)
          Set hive input format, and input format file if necessary.
 void shutdown()
           
 void updateCounters(Counters ctrs, RunningJob rj)
           
 
Methods inherited from class org.apache.hadoop.hive.ql.exec.Task
addDependentTask, cloneConf, done, executeTask, fetch, getAndInitBackupTask, getBackupChildrenTasks, getBackupTask, getChildren, getChildTasks, getCounters, getDependentTasks, getFeedSubscribers, getId, getInitialized, getJobID, getParentTasks, getQueryPlan, getQueued, getReducer, getResultSchema, getTaskHandle, getTaskTag, getWork, ifRetryCmdWhenFail, isLocalMode, isMapRedLocalTask, isRunnable, pushFeed, receiveFeed, removeDependentTask, removeFromChildrenTasks, setBackupChildrenTasks, setBackupTask, setChildTasks, setDone, setFeedSubscribers, setId, setInitialized, setLocalMode, setParentTasks, setQueryPlan, setQueued, setRetryCmdWhenFail, setStarted, setTaskTag, setWork, started, subscribeFeed
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

job

protected transient JobConf job

memoryMXBean

public static MemoryMXBean memoryMXBean

jobExecHelper

protected HadoopJobExecHelper jobExecHelper

LOG

protected static final transient org.apache.commons.logging.Log LOG
Constructor Detail

ExecDriver

public ExecDriver()
Constructor when invoked from QL.


ExecDriver

public ExecDriver(MapredWork plan,
                  JobConf job,
                  boolean isSilent)
           throws HiveException
Constructor/Initialization for invocation as independent utility.

Throws:
HiveException
Method Detail

requireLock

public boolean requireLock()
Overrides:
requireLock in class Task<MapredWork>

initialize

public void initialize(HiveConf conf,
                       QueryPlan queryPlan,
                       DriverContext driverContext)
Initialization when invoked from QL.

Overrides:
initialize in class Task<MapredWork>

checkFatalErrors

public boolean checkFatalErrors(Counters ctrs,
                                StringBuilder errMsg)
Fatal errors are those errors that cannot be recovered by retries. These are application dependent. Examples of fatal errors include: - the small table in the map-side joins is too large to be feasible to be handled by one mapper. The job should fail and the user should be warned to use regular joins rather than map-side joins. Fatal errors are indicated by counters that are set at execution time. If the counter is non-zero, a fatal error occurred. The value of the counter indicates the error type.

Specified by:
checkFatalErrors in interface HadoopJobExecHook
Returns:
true if fatal errors happened during job execution, false otherwise.

execute

public int execute(DriverContext driverContext)
Execute a query plan using Hadoop.

Specified by:
execute in class Task<MapredWork>
Returns:
status of executing the task

setInputAttributes

protected void setInputAttributes(Configuration conf)
Set hive input format, and input format file if necessary.


mapStarted

public boolean mapStarted()

reduceStarted

public boolean reduceStarted()

mapDone

public boolean mapDone()

reduceDone

public boolean reduceDone()

main

public static void main(String[] args)
                 throws IOException,
                        HiveException
Throws:
IOException
HiveException

generateCmdLine

public static String generateCmdLine(HiveConf hconf,
                                     Context ctx)
                              throws IOException
Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver.

Throws:
IOException

isMapRedTask

public boolean isMapRedTask()
Overrides:
isMapRedTask in class Task<MapredWork>

getTopOperators

public Collection<Operator<? extends OperatorDesc>> getTopOperators()
Overrides:
getTopOperators in class Task<MapredWork>

hasReduce

public boolean hasReduce()
Overrides:
hasReduce in class Task<MapredWork>

getType

public StageType getType()
Description copied from class: Task
Should be overridden to return the type of the specific task among the types in StageType.

Specified by:
getType in class Task<MapredWork>
Returns:
StageType.* or null if not overridden

getName

public String getName()
Description copied from interface: Node
Gets the name of the node. This is used in the rule dispatchers.

Specified by:
getName in interface Node
Returns:
String

updateCounters

public void updateCounters(Counters ctrs,
                           RunningJob rj)
                    throws IOException
Specified by:
updateCounters in interface HadoopJobExecHook
Throws:
IOException

logPlanProgress

public void logPlanProgress(SessionState ss)
                     throws IOException
Specified by:
logPlanProgress in interface HadoopJobExecHook
Throws:
IOException

shutdown

public void shutdown()
Overrides:
shutdown in class Task<MapredWork>


Copyright © 2012 The Apache Software Foundation