com.cloudera.sqoop.mapreduce
Class MySQLExportMapper<KEYIN,VALIN>

java.lang.Object
  extended by org.apache.hadoop.mapreduce.Mapper<KEYIN,VALIN,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.NullWritable>
      extended by com.cloudera.sqoop.mapreduce.MySQLExportMapper<KEYIN,VALIN>
Direct Known Subclasses:
MySQLRecordExportMapper, MySQLTextExportMapper

public class MySQLExportMapper<KEYIN,VALIN>
extends org.apache.hadoop.mapreduce.Mapper<KEYIN,VALIN,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.NullWritable>

Mapper that starts a 'mysqlimport' process and uses that to export rows from HDFS to a MySQL database at high speed. map() methods are actually provided by subclasses that read from SequenceFiles (containing existing SqoopRecords) or text files (containing delimited lines) and deliver these results to the fifo used to interface with mysqlimport.


Nested Class Summary
 
Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Mapper
org.apache.hadoop.mapreduce.Mapper.Context
 
Field Summary
protected  long checkpointDistInBytes
           
protected  org.apache.hadoop.conf.Configuration conf
           
static long DEFAULT_CHECKPOINT_BYTES
           
protected  AsyncSink errSink
           
protected  java.io.File fifoFile
          The FIFO being used to communicate with mysqlimport.
protected  java.io.OutputStream importStream
          The stream to write to stdin for mysqlimport.
static org.apache.commons.logging.Log LOG
           
static java.lang.String MYSQL_CHECKPOINT_BYTES_KEY
          Configuration key that specifies the number of bytes before which it commits the current export transaction and opens a new one.
protected  java.lang.String mysqlCharSet
          Character set used to write to mysqlimport.
protected  java.lang.Process mysqlImportProcess
          The process object representing the active connection to mysqlimport.
protected  AsyncSink outSink
           
protected  java.io.File passwordFile
          File object where we wrote the user's password to pass to mysqlimport.
 
Constructor Summary
MySQLExportMapper()
           
 
Method Summary
 void run(org.apache.hadoop.mapreduce.Mapper.Context context)
           
protected  void setup(org.apache.hadoop.mapreduce.Mapper.Context context)
           
protected  void writeRecord(java.lang.String record, java.lang.String terminator)
          Takes a delimited text record (e.g., the output of a 'Text' object), re-encodes it for consumption by mysqlimport, and writes it to the pipe.
 
Methods inherited from class org.apache.hadoop.mapreduce.Mapper
cleanup, map
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

MYSQL_CHECKPOINT_BYTES_KEY

public static final java.lang.String MYSQL_CHECKPOINT_BYTES_KEY
Configuration key that specifies the number of bytes before which it commits the current export transaction and opens a new one. Default is 32 MB; setting this to 0 will use no checkpoints.

See Also:
Constant Field Values

DEFAULT_CHECKPOINT_BYTES

public static final long DEFAULT_CHECKPOINT_BYTES
See Also:
Constant Field Values

checkpointDistInBytes

protected long checkpointDistInBytes

conf

protected org.apache.hadoop.conf.Configuration conf

fifoFile

protected java.io.File fifoFile
The FIFO being used to communicate with mysqlimport.


mysqlImportProcess

protected java.lang.Process mysqlImportProcess
The process object representing the active connection to mysqlimport.


importStream

protected java.io.OutputStream importStream
The stream to write to stdin for mysqlimport.


outSink

protected AsyncSink outSink

errSink

protected AsyncSink errSink

passwordFile

protected java.io.File passwordFile
File object where we wrote the user's password to pass to mysqlimport.


mysqlCharSet

protected java.lang.String mysqlCharSet
Character set used to write to mysqlimport.

Constructor Detail

MySQLExportMapper

public MySQLExportMapper()
Method Detail

run

public void run(org.apache.hadoop.mapreduce.Mapper.Context context)
         throws java.io.IOException,
                java.lang.InterruptedException
Overrides:
run in class org.apache.hadoop.mapreduce.Mapper<KEYIN,VALIN,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.NullWritable>
Throws:
java.io.IOException
java.lang.InterruptedException

setup

protected void setup(org.apache.hadoop.mapreduce.Mapper.Context context)
Overrides:
setup in class org.apache.hadoop.mapreduce.Mapper<KEYIN,VALIN,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.NullWritable>

writeRecord

protected void writeRecord(java.lang.String record,
                           java.lang.String terminator)
                    throws java.io.IOException,
                           java.lang.InterruptedException
Takes a delimited text record (e.g., the output of a 'Text' object), re-encodes it for consumption by mysqlimport, and writes it to the pipe.

Parameters:
record - A delimited text representation of one record.
terminator - an optional string that contains delimiters that terminate the record (if not included in 'record' itself).
Throws:
java.io.IOException
java.lang.InterruptedException


Copyright © 2010 Cloudera, Inc.