org.apache.pig.backend.hadoop.executionengine.mapReduceLayer
Class JobControlCompiler
java.lang.Object
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
public class JobControlCompiler
- extends Object
This is compiler class that takes an MROperPlan and converts
it into a JobControl object with the relevant dependency info
maintained. The JobControl Object is made up of Jobs each of
which has a JobConf. The MapReduceOper corresponds to a Job
and the getJobCong method returns the JobConf that is configured
as per the MapReduceOper
Comparator Design
A few words on how comparators are chosen. In almost all cases we use raw
comparators (the one exception being when the user provides a comparison
function for order by). For order by queries the PigTYPERawComparator
functions are used, where TYPE is Int, Long, etc. These comparators are
null aware and asc/desc aware. The first byte of each of the
NullableTYPEWritable classes contains info on whether the value is null.
Asc/desc is written as an array into the JobConf with the key pig.sortOrder
so that it can be read by each of the comparators as part of their
setConf call.
For non-order by queries, PigTYPEWritableComparator classes are used.
These are all just type specific instances of WritableComparator.
Method Summary |
org.apache.hadoop.mapred.jobcontrol.JobControl |
compile(MROperPlan plan,
String grpName,
org.apache.hadoop.conf.Configuration conf,
PigContext pigContext)
Top level compile method that issues a call to the recursive
compile method. |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
LOG_DIR
public static final String LOG_DIR
- See Also:
- Constant Field Values
JobControlCompiler
public JobControlCompiler()
compile
public org.apache.hadoop.mapred.jobcontrol.JobControl compile(MROperPlan plan,
String grpName,
org.apache.hadoop.conf.Configuration conf,
PigContext pigContext)
throws JobCreationException
- Top level compile method that issues a call to the recursive
compile method.
- Parameters:
plan
- - The MROperPlan to be compiledgrpName
- - The name given to the JobControlconf
- - The Configuration object having the various propertiespigContext
- - PigContext passed on from the execution engine
- Returns:
- JobControl object
- Throws:
JobCreationException
Copyright © ${year} The Apache Software Foundation