org.apache.hadoop.hive.ql.optimizer
Class GenMapRedUtils

java.lang.Object
  extended by org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils

public final class GenMapRedUtils
extends Object

General utility common functions for the Processor to convert operator into map-reduce tasks.


Method Summary
static MapredWork getMapRedWork(ParseContext parseCtx)
          create a new plan and return.
static MapredWork getMapRedWorkFromConf(HiveConf conf)
          create a new plan and return.
static void initPlan(ReduceSinkOperator op, GenMRProcContext opProcCtx)
          Initialize the current plan by adding it to root tasks.
static void initUnionPlan(GenMRProcContext opProcCtx, UnionOperator currUnionOp, Task<? extends Serializable> currTask, boolean local)
           
static void initUnionPlan(ReduceSinkOperator op, UnionOperator currUnionOp, GenMRProcContext opProcCtx, Task<? extends Serializable> unionTask)
          Initialize the current union plan.
static void joinPlan(Task<? extends Serializable> currTask, Task<? extends Serializable> oldTask, GenMRProcContext opProcCtx)
          Merge the current task into the old task for the reducer
static void joinUnionPlan(GenMRProcContext opProcCtx, UnionOperator currUnionOp, Task<? extends Serializable> currentUnionTask, Task<? extends Serializable> existingTask, boolean local)
           
static Operator<? extends OperatorDesc> putOpInsertMap(Operator<? extends OperatorDesc> op, RowResolver rr, ParseContext parseCtx)
          insert in the map for the operator to row resolver.
static void replaceMapWork(String sourceAlias, String targetAlias, MapWork source, MapWork target)
          Replace the Map-side operator tree associated with targetAlias in target with the Map-side operator tree associated with sourceAlias in source.
static void setKeyAndValueDesc(ReduceWork plan, Operator<? extends OperatorDesc> topOp)
          set key and value descriptor.
static void setKeyAndValueDescForTaskTree(Task<? extends Serializable> task)
          Set the key and value description for all the tasks rooted at the given task.
static void setTaskPlan(String alias_id, Operator<? extends OperatorDesc> topOp, Task<?> task, boolean local, GenMRProcContext opProcCtx)
          set the current task in the mapredWork.
static void setTaskPlan(String alias_id, Operator<? extends OperatorDesc> topOp, Task<?> task, boolean local, GenMRProcContext opProcCtx, PrunedPartitionList pList)
          set the current task in the mapredWork.
static void setTaskPlan(String path, String alias, Operator<? extends OperatorDesc> topOp, MapWork plan, boolean local, TableDesc tt_desc)
          set the current task in the mapredWork.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Method Detail

initPlan

public static void initPlan(ReduceSinkOperator op,
                            GenMRProcContext opProcCtx)
                     throws SemanticException
Initialize the current plan by adding it to root tasks.

Parameters:
op - the reduce sink operator encountered
opProcCtx - processing context
Throws:
SemanticException

initUnionPlan

public static void initUnionPlan(ReduceSinkOperator op,
                                 UnionOperator currUnionOp,
                                 GenMRProcContext opProcCtx,
                                 Task<? extends Serializable> unionTask)
                          throws SemanticException
Initialize the current union plan.

Parameters:
op - the reduce sink operator encountered
opProcCtx - processing context
Throws:
SemanticException

initUnionPlan

public static void initUnionPlan(GenMRProcContext opProcCtx,
                                 UnionOperator currUnionOp,
                                 Task<? extends Serializable> currTask,
                                 boolean local)
                          throws SemanticException
Throws:
SemanticException

joinUnionPlan

public static void joinUnionPlan(GenMRProcContext opProcCtx,
                                 UnionOperator currUnionOp,
                                 Task<? extends Serializable> currentUnionTask,
                                 Task<? extends Serializable> existingTask,
                                 boolean local)
                          throws SemanticException
Throws:
SemanticException

joinPlan

public static void joinPlan(Task<? extends Serializable> currTask,
                            Task<? extends Serializable> oldTask,
                            GenMRProcContext opProcCtx)
                     throws SemanticException
Merge the current task into the old task for the reducer

Parameters:
currTask - the current task for the current reducer
oldTask - the old task for the current reducer
opProcCtx - processing context
Throws:
SemanticException

setTaskPlan

public static void setTaskPlan(String alias_id,
                               Operator<? extends OperatorDesc> topOp,
                               Task<?> task,
                               boolean local,
                               GenMRProcContext opProcCtx)
                        throws SemanticException
set the current task in the mapredWork.

Parameters:
alias_id - current alias
topOp - the top operator of the stack
plan - current plan
local - whether you need to add to map-reduce or local work
opProcCtx - processing context
Throws:
SemanticException

setTaskPlan

public static void setTaskPlan(String alias_id,
                               Operator<? extends OperatorDesc> topOp,
                               Task<?> task,
                               boolean local,
                               GenMRProcContext opProcCtx,
                               PrunedPartitionList pList)
                        throws SemanticException
set the current task in the mapredWork.

Parameters:
alias_id - current alias
topOp - the top operator of the stack
plan - current plan
local - whether you need to add to map-reduce or local work
opProcCtx - processing context
pList - pruned partition list. If it is null it will be computed on-the-fly.
Throws:
SemanticException

setTaskPlan

public static void setTaskPlan(String path,
                               String alias,
                               Operator<? extends OperatorDesc> topOp,
                               MapWork plan,
                               boolean local,
                               TableDesc tt_desc)
                        throws SemanticException
set the current task in the mapredWork.

Parameters:
alias - current alias
topOp - the top operator of the stack
plan - current plan
local - whether you need to add to map-reduce or local work
tt_desc - table descriptor
Throws:
SemanticException

setKeyAndValueDesc

public static void setKeyAndValueDesc(ReduceWork plan,
                                      Operator<? extends OperatorDesc> topOp)
set key and value descriptor.

Parameters:
plan - current plan
topOp - current top operator in the path

setKeyAndValueDescForTaskTree

public static void setKeyAndValueDescForTaskTree(Task<? extends Serializable> task)
Set the key and value description for all the tasks rooted at the given task. Loops over all the tasks recursively.

Parameters:
task -

getMapRedWork

public static MapredWork getMapRedWork(ParseContext parseCtx)
create a new plan and return.

Returns:
the new plan

getMapRedWorkFromConf

public static MapredWork getMapRedWorkFromConf(HiveConf conf)
create a new plan and return. The pan won't contain the name to split sample information in parse context.

Returns:
the new plan

putOpInsertMap

public static Operator<? extends OperatorDesc> putOpInsertMap(Operator<? extends OperatorDesc> op,
                                                              RowResolver rr,
                                                              ParseContext parseCtx)
insert in the map for the operator to row resolver.

Parameters:
op - operator created
rr - row resolver
parseCtx - parse context

replaceMapWork

public static void replaceMapWork(String sourceAlias,
                                  String targetAlias,
                                  MapWork source,
                                  MapWork target)
Replace the Map-side operator tree associated with targetAlias in target with the Map-side operator tree associated with sourceAlias in source.

Parameters:
sourceAlias -
targetAlias -
source -
target -


Copyright © 2012 The Apache Software Foundation