org.apache.pig.backend.hadoop.executionengine.mapReduceLayer
Class JobControlCompiler

java.lang.Object
  extended by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler

public class JobControlCompiler
extends Object

This is compiler class that takes an MROperPlan and converts it into a JobControl object with the relevant dependency info maintained. The JobControl Object is made up of Jobs each of which has a JobConf. The MapReduceOper corresponds to a Job and the getJobCong method returns the JobConf that is configured as per the MapReduceOper

Comparator Design

A few words on how comparators are chosen. In almost all cases we use raw comparators (the one exception being when the user provides a comparison function for order by). For order by queries the PigTYPERawComparator functions are used, where TYPE is Int, Long, etc. These comparators are null aware and asc/desc aware. The first byte of each of the NullableTYPEWritable classes contains info on whether the value is null. Asc/desc is written as an array into the JobConf with the key pig.sortOrder so that it can be read by each of the comparators as part of their setConf call.

For non-order by queries, PigTYPEWritableComparator classes are used. These are all just type specific instances of WritableComparator.


Nested Class Summary
static class JobControlCompiler.PigBagWritableComparator
           
static class JobControlCompiler.PigCharArrayWritableComparator
           
static class JobControlCompiler.PigDBAWritableComparator
           
static class JobControlCompiler.PigDoubleWritableComparator
           
static class JobControlCompiler.PigFloatWritableComparator
           
static class JobControlCompiler.PigIntWritableComparator
           
static class JobControlCompiler.PigLongWritableComparator
           
static class JobControlCompiler.PigTupleWritableComparator
           
static class JobControlCompiler.PigWritableComparator
           
 
Field Summary
static String LOG_DIR
           
 
Constructor Summary
JobControlCompiler()
           
 
Method Summary
 org.apache.hadoop.mapred.jobcontrol.JobControl compile(MROperPlan plan, String grpName, org.apache.hadoop.conf.Configuration conf, PigContext pigContext)
          Top level compile method that issues a call to the recursive compile method.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

LOG_DIR

public static final String LOG_DIR
See Also:
Constant Field Values
Constructor Detail

JobControlCompiler

public JobControlCompiler()
Method Detail

compile

public org.apache.hadoop.mapred.jobcontrol.JobControl compile(MROperPlan plan,
                                                              String grpName,
                                                              org.apache.hadoop.conf.Configuration conf,
                                                              PigContext pigContext)
                                                       throws JobCreationException
Top level compile method that issues a call to the recursive compile method.

Parameters:
plan - - The MROperPlan to be compiled
grpName - - The name given to the JobControl
conf - - The Configuration object having the various properties
pigContext - - PigContext passed on from the execution engine
Returns:
JobControl object
Throws:
JobCreationException


Copyright © ${year} The Apache Software Foundation