org.apache.hadoop.hive.ql.optimizer.correlation
Class ReduceSinkDeDuplication.AbsctractReducerReducerProc

java.lang.Object
  extended by org.apache.hadoop.hive.ql.optimizer.correlation.ReduceSinkDeDuplication.AbsctractReducerReducerProc
All Implemented Interfaces:
NodeProcessor
Enclosing class:
ReduceSinkDeDuplication

public abstract static class ReduceSinkDeDuplication.AbsctractReducerReducerProc
extends Object
implements NodeProcessor


Constructor Summary
ReduceSinkDeDuplication.AbsctractReducerReducerProc()
           
 
Method Summary
protected  Integer checkNumReducer(int creduce, int preduce)
          If number of reducers for RS is -1, the RS can have any number of reducers.
protected  Integer checkOrder(String corder, String porder)
           
protected  boolean merge(ReduceSinkOperator cRS, JoinOperator pJoin, int minReducer)
           
protected  boolean merge(ReduceSinkOperator cRS, ReduceSinkOperator pRS, int minReducer)
          Current RSDedup remove/replace child RS.
 Object process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx, Object... nodeOutputs)
          Generic process for all ops that don't have specific implementations.
protected abstract  Object process(ReduceSinkOperator cRS, GroupByOperator cGBY, ReduceSinkDeDuplication.ReduceSinkDeduplicateProcCtx dedupCtx)
           
protected abstract  Object process(ReduceSinkOperator cRS, ReduceSinkDeDuplication.ReduceSinkDeduplicateProcCtx dedupCtx)
           
protected  Integer sameKeys(List<ExprNodeDesc> cexprs, List<ExprNodeDesc> pexprs, Operator<?> child, Operator<?> parent)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

ReduceSinkDeDuplication.AbsctractReducerReducerProc

public ReduceSinkDeDuplication.AbsctractReducerReducerProc()
Method Detail

process

public Object process(Node nd,
                      Stack<Node> stack,
                      NodeProcessorCtx procCtx,
                      Object... nodeOutputs)
               throws SemanticException
Description copied from interface: NodeProcessor
Generic process for all ops that don't have specific implementations.

Specified by:
process in interface NodeProcessor
Parameters:
nd - operator to process
procCtx - operator processor context
nodeOutputs - A variable argument list of outputs from other nodes in the walk
Returns:
Object to be returned by the process call
Throws:
SemanticException

process

protected abstract Object process(ReduceSinkOperator cRS,
                                  ReduceSinkDeDuplication.ReduceSinkDeduplicateProcCtx dedupCtx)
                           throws SemanticException
Throws:
SemanticException

process

protected abstract Object process(ReduceSinkOperator cRS,
                                  GroupByOperator cGBY,
                                  ReduceSinkDeDuplication.ReduceSinkDeduplicateProcCtx dedupCtx)
                           throws SemanticException
Throws:
SemanticException

merge

protected boolean merge(ReduceSinkOperator cRS,
                        JoinOperator pJoin,
                        int minReducer)
                 throws SemanticException
Throws:
SemanticException

merge

protected boolean merge(ReduceSinkOperator cRS,
                        ReduceSinkOperator pRS,
                        int minReducer)
                 throws SemanticException
Current RSDedup remove/replace child RS. For key columns, sorting order, and the number of reducers, copy more specific part of configurations of child RS to that of parent RS. For partitioning columns, if both child RS and parent RS have been assigned partitioning columns, we will choose the more general partitioning columns. If parent RS has not been assigned any partitioning column, we will use partitioning columns (if exist) of child RS.

Throws:
SemanticException

sameKeys

protected Integer sameKeys(List<ExprNodeDesc> cexprs,
                           List<ExprNodeDesc> pexprs,
                           Operator<?> child,
                           Operator<?> parent)
                    throws SemanticException
Throws:
SemanticException

checkOrder

protected Integer checkOrder(String corder,
                             String porder)

checkNumReducer

protected Integer checkNumReducer(int creduce,
                                  int preduce)
If number of reducers for RS is -1, the RS can have any number of reducers. It's generally true except for order-by or forced bucketing cases. if both of num-reducers are not -1, those number should be the same.



Copyright © 2012 The Apache Software Foundation