|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.hadoop.mapreduce.Mapper<KEYIN,VALIN,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.NullWritable>
com.cloudera.sqoop.mapreduce.MySQLExportMapper<KEYIN,VALIN>
public class MySQLExportMapper<KEYIN,VALIN>
Mapper that starts a 'mysqlimport' process and uses that to export rows from HDFS to a MySQL database at high speed. map() methods are actually provided by subclasses that read from SequenceFiles (containing existing SqoopRecords) or text files (containing delimited lines) and deliver these results to the fifo used to interface with mysqlimport.
Nested Class Summary |
---|
Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Mapper |
---|
org.apache.hadoop.mapreduce.Mapper.Context |
Field Summary | |
---|---|
protected long |
checkpointDistInBytes
|
protected org.apache.hadoop.conf.Configuration |
conf
|
static long |
DEFAULT_CHECKPOINT_BYTES
|
protected AsyncSink |
errSink
|
protected java.io.File |
fifoFile
The FIFO being used to communicate with mysqlimport. |
protected java.io.OutputStream |
importStream
The stream to write to stdin for mysqlimport. |
static org.apache.commons.logging.Log |
LOG
|
static java.lang.String |
MYSQL_CHECKPOINT_BYTES_KEY
Configuration key that specifies the number of bytes before which it commits the current export transaction and opens a new one. |
protected java.lang.String |
mysqlCharSet
Character set used to write to mysqlimport. |
protected java.lang.Process |
mysqlImportProcess
The process object representing the active connection to mysqlimport. |
protected AsyncSink |
outSink
|
protected java.io.File |
passwordFile
File object where we wrote the user's password to pass to mysqlimport. |
Constructor Summary | |
---|---|
MySQLExportMapper()
|
Method Summary | |
---|---|
void |
run(org.apache.hadoop.mapreduce.Mapper.Context context)
|
protected void |
setup(org.apache.hadoop.mapreduce.Mapper.Context context)
|
protected void |
writeRecord(java.lang.String record,
java.lang.String terminator)
Takes a delimited text record (e.g., the output of a 'Text' object), re-encodes it for consumption by mysqlimport, and writes it to the pipe. |
Methods inherited from class org.apache.hadoop.mapreduce.Mapper |
---|
cleanup, map |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final org.apache.commons.logging.Log LOG
public static final java.lang.String MYSQL_CHECKPOINT_BYTES_KEY
public static final long DEFAULT_CHECKPOINT_BYTES
protected long checkpointDistInBytes
protected org.apache.hadoop.conf.Configuration conf
protected java.io.File fifoFile
protected java.lang.Process mysqlImportProcess
protected java.io.OutputStream importStream
protected AsyncSink outSink
protected AsyncSink errSink
protected java.io.File passwordFile
protected java.lang.String mysqlCharSet
Constructor Detail |
---|
public MySQLExportMapper()
Method Detail |
---|
public void run(org.apache.hadoop.mapreduce.Mapper.Context context) throws java.io.IOException, java.lang.InterruptedException
run
in class org.apache.hadoop.mapreduce.Mapper<KEYIN,VALIN,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.NullWritable>
java.io.IOException
java.lang.InterruptedException
protected void setup(org.apache.hadoop.mapreduce.Mapper.Context context)
setup
in class org.apache.hadoop.mapreduce.Mapper<KEYIN,VALIN,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.NullWritable>
protected void writeRecord(java.lang.String record, java.lang.String terminator) throws java.io.IOException, java.lang.InterruptedException
record
- A delimited text representation of one record.terminator
- an optional string that contains delimiters that
terminate the record (if not included in 'record' itself).
java.io.IOException
java.lang.InterruptedException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |