Introducing Hue Job Designer

The Job Designer application enables you to create and submit Hadoop Map/Reduce jobs to the Hadoop cluster. You can include variables with your jobs to enable you and other users to enter values for the variables when they run your job. The Job Designer supports streaming, MapReduce, and Java jobs.

Note
  • The Job Designer uses Oozie to submit MapReduce jobs. Therefore, Oozie must be installed and configured before you can use JobDesigner.
  • A job's input files must be uploaded to the cluster before you can submit the job.

Contents


Job Designer Installation and Configuration

Job Designer is one of the applications that is installed as part of Hue. For more information about installing Hue, see Hue Installation. For information about installing Oozie, see Oozie installation.

Starting Job Designer

To start Job Designer, click the Job Designer tab in the navigation bar at the top of the Hue web page. The Job Designs page opens in the browser.

Installing the Job Designer Samples

The Job Designer sample jobs can help you learn how to use Job Designer.

To install the Job Designer samples, click Install Samples in the Job Designs window and then click Yes. The sample jobs are displayed in the Job Designs window. Job Designer removes the Install Samples button after the samples are installed so you can only install the samples once.

Working with Job Designs

In the Job Designer, a job design specifies several meta-level properties of a MapReduce job, including the job design name, description, the MapReduce executable scripts or classes, and any parameters for those scripts or classes. You can create three types of job designs: a MapReduce Job, a streaming job and a Java job design.

Creating a MapReduce Job Design

A Hadoop JAR consists of MapReduce functions written in Java.

To create a MapReduce job design:

  1. In the Job Designs window, click Create MapReduce Design.

    The Job Design (MapReduce type) window opens where you can specify information about the MapReduce job.

  2. In the Job Design (MapReduce type) window, specify the following information.


Setting Description
Name The Name identifies the MapReduce job and it's collection of parameters.
Description Specify a description of the MapReduce job. The description is displayed in the dialog box that appears if you specify variables for the job.
Jar path Specify the name of the JAR file, including the path.
Job Properties Click Add Property, to set a property value:
»   For Property name, enter a configuration variable name. This field provides auto-completion, so you can type the first few characters of a property name and then select the one you want from a list.
»   For Value, enter the value you want to use for the variable.
Files Click Add File, then enter a file name or browse for a file to upload.
Archives Click Add Archive, then enter an archive name or browse for a


  1. Click Save to save the job settings.

Creating a Streaming Job Design

Hadoop streaming jobs enable you to create MapReduce functions in any non-Java language that reads standard Unix input and writes standard Unix output. For more information about Hadoop streaming jobs, see needs updated link to info.

To create a streaming job design:

  1. In the Job Designs window, click Create Streaming Design.

    The Job Design (streaming type) window opens to enable you to specify information about the streaming job.

  2. In the Job Design (streaming type) window, specify the following information.

    Note: You can use variables of the form $variable_name for the Mapper, and Reducer settings described in the following table. When the streaming job is run, a dialog box will appear to enable you or users to specify the values of the variables.


    Setting Description
    Name The Name identifies the streaming job design including the associated properties and parameters.
    Description Specify a description of the streaming job design. The description is displayed in the dialog box that appears if you specify variables for the job.
    Mapper Specify the path to the mapper script or class. If the mapper file is not on the machines on the cluster, use the Files option to pack it as a part of job submission. Equivalent to the Hadoop -mapper option.
    Reducer Specify the path to the reducer script or class. If the reducer file is not on the machines on the cluster, use the Files option to pack it as a part of job submission. Equivalent to the Hadoop -reducer option.
    Job Properties Click Add Property, to set a property value:
    »   For Property name, enter a configuration variable name.
    »   For Value, enter the value you want to use for the variable.
    Files Click Add File, then enter a file name what files?
    Archives Click Add Archive, then enter an archive name what is this?



  1. Click Save to save the job settings.

Creating a Java Job Design

A Hadoop JAR consists of MapReduce functions written in Java.
how is this different from a MapReduce job?

To create a Java job design:

  1. In the Job Designs window, click Create Java Design.

    The Job Design (java type) window opens where you can specify information about the JAR job.

  2. In the Job Design (java type) window, specify the following information.

    Note: You can use variables of the form $variable_name for the Args setting described in the following table. When the JAR job is run, a dialog box will appear to enable you or users to specify the values of the variables.


    Setting Description
    Name The Name identifies the Java job and it's collection of parameters.
    Description Specify a description of the java job. The description is displayed in the dialog box that appears if you specify variables for the job.
    Jar path Specify the name of the JAR file, including the path.
    Main class Specify the (name of?) the Main class to invoke the program.
    Args Specify the arguments you want to pass to the running JAR job.
    Java opts Specify any Java options you want to specify as part of the job.
    Job Properties Click Add Property, to set a property value:
    »   For Property name, enter a configuration variable name.
    »   For Value, enter the value you want to use for the variable.
    Files Click Add File, then enter a file name what files?
    Archives Click Add Archive, then enter an archive name what is this?


  1. Click Save to save the job settings.

Submitting a Job to a Cluster

To submit a job to a cluster:

  1. In the Job Designs window, click Designs in the upper left corner.
    Your jobs and other users' jobs are displayed in the Job Designs window.

  2. In the Job Design window, click the Submit button for the job you want to submit.

  3. If the job contains variables, enter the information requested in the dialog box that appears.
    For example, the sample streaming PI Calculator job displays the following dialog box to enable you to specify the settings for Iterations per Mapper and Num of mappers.

  4. Click Submit to submit the job.

    After the job is complete, the Job Designer displays the results of the job including the last 10 KB of stdout and stderr for a streaming job. For example, after the sample streaming PI Calculator job is complete, the following results appear.

    For information about displaying job results, see Displaying Job Results.

Copying, Editing, and Deleting a Job Design

If you want to edit and use a job but you don't own it, you can make a copy of it and then edit and use the copied job.

To copy a job design:

  1. In the Job Designs window, click Designs.

    The jobs are displayed in the Job Designs window.

  2. Click the Clone button to the right of the job you want to copy.

  3. In the Job Design Editor window, change the settings and then click Save to save the job settings.

To edit a job design:

  1. In the Job Designs window, click Designs.

    The jobs are displayed in the Job Designs window.

  2. Click the Edit button to the right of the job you want to copy.

  3. In the Job Design window, change the settings and then click Save to save the job settings.

To delete a job design:

  1. In the Job Designs window, click job designs.

    The jobs are displayed in the Job Designs window.

  2. In the Job Designs window, select the job, right click, and choose Delete from the context menu.

  3. Click Ok to confirm the deletion.

Filtering the Job Designs

You can filter the Job Designs by owner, by job name, or both.

To filter the Job Designs list:

  1. In the Job Designs window, click job designs.

  2. Enter the name of the owner, job name, or both in the query search boxes at the top of the Job Designs window.

    Job Designer displays the jobs that match the filter criteria.

Displaying Job Results (the Job Submission History)

To display the Job Submission History

  1. In the Job Designs window, click the History tab.

    The jobs are displayed in the Job Submissions History list by Oozie job ID.

  2. To display details about a job, double-click the Oozie Job ID.

    The results of the job appear.

  3. To display details about the job, click the job name under the Launched Jobs heading at the top of the results window.

    The following Job Browser screen appears with details about the job.

  4. To display details about a task, double-click a task under Recent Tasks or click this ?? next to the task.

  5. To view other types of job information, click the Metadata and Counters tabs.

  6. To view the output of the job, click the link next to Output at the top of the screen.

  7. To view status information of all jobs, click the view all tasks link.

    Click here for more information about using Job Browser.