<?xml version="1.0"?>
<!--
  Copyright 2002-2004 The Apache Software Foundation

  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd">

<document>
  
  <header>
    <title>Native Libraries Guide</title>
  </header>
  
  <body>
  
    <section>
      <title>Purpose</title>
      
      <p>Hadoop has native implementations of certain components for reasons of 
      both performance and non-availability of Java implementations. These 
      components are available in a single, dynamically-linked, native library. 
      On the *nix platform it is <em>libhadoop.so</em>. This document describes 
      the usage and details on how to build the native libraries.</p>
    </section>
    
    <section>
      <title>Components</title>
      
      <p>Hadoop currently has the following 
      <a href="ext:api/org/apache/hadoop/io/compress/compressioncodec">
      compression codecs</a> as the native components:</p>
      <ul>
        <li><a href="ext:zlib">zlib</a></li>
        <li><a href="ext:gzip">gzip</a></li>
        <li><a href="ext:bzip">bzip2</a></li>
      </ul>
      
      <p>Of the above, the availability of native hadoop libraries is imperative 
      for the gzip and bzip2 compression codecs to work.</p>
    </section>

    <section>
      <title>Usage</title>
      
      <p>It is fairly simple to use the native hadoop libraries:</p>

      <ul>
        <li>
          Take a look at the 
          <a href="#Supported+Platforms">supported platforms</a>.
        </li>
        <li>
          Either <a href="ext:releases/download">download</a> the pre-built 
          32-bit i386-Linux native hadoop libraries (available as part of hadoop 
          distribution in <code>lib/native</code> directory) or 
          <a href="#Building+Native+Hadoop+Libraries">build</a> them yourself.
        </li>
        <li>
          Make sure you have any of or all of <strong>&gt;zlib-1.2</strong>,
          <strong>&gt;gzip-1.2</strong>, and <strong>&gt;bzip2-1.0</strong>
          packages for your platform installed; 
          depending on your needs.
        </li>
      </ul>
      
      <p>The <code>bin/hadoop</code> script ensures that the native hadoop 
      library is on the library path via the system property 
      <em>-Djava.library.path=&lt;path&gt;</em>.</p>

      <p>To check everything went alright check the hadoop log files for:</p>
      
      <p>
        <code>
          DEBUG util.NativeCodeLoader - Trying to load the custom-built 
          native-hadoop library... 
        </code><br/>
        <code>
          INFO  util.NativeCodeLoader - Loaded the native-hadoop library
        </code>
      </p>

      <p>If something goes wrong, then:</p>
      <p>
        <code>
          INFO util.NativeCodeLoader - Unable to load native-hadoop library for 
          your platform... using builtin-java classes where applicable
        </code>
      </p>
    </section>
    
    <section>
      <title>Supported Platforms</title>
      
      <p>Hadoop native library is supported only on *nix platforms only.
      Unfortunately it is known not to work on <a href="ext:cygwin">Cygwin</a> 
      and <a href="ext:osx">Mac OS X</a> and has mainly been used on the 
      GNU/Linux platform.</p>

      <p>It has been tested on the following GNU/Linux distributions:</p>
      <ul>
        <li>
          <a href="http://www.redhat.com/rhel/">RHEL4</a>/<a href="http://fedora.redhat.com/">Fedora</a>
        </li>
        <li><a href="http://www.ubuntu.com/">Ubuntu</a></li>
        <li><a href="http://www.gentoo.org/">Gentoo</a></li>
      </ul>

      <p>On all the above platforms a 32/64 bit Hadoop native library will work 
      with a respective 32/64 bit jvm.</p>
    </section>
    
    <section>
      <title>Building Native Hadoop Libraries</title>
      
      <p>Hadoop native library is written in 
      <a href="http://en.wikipedia.org/wiki/ANSI_C">ANSI C</a> and built using 
      the GNU autotools-chain (autoconf, autoheader, automake, autoscan, libtool). 
      This means it should be straight-forward to build them on any platform with 
      a standards compliant C compiler and the GNU autotools-chain. 
      See <a href="#Supported+Platforms">supported platforms</a>.</p>

      <p>In particular the various packages you would need on the target 
      platform are:</p>
      <ul>
        <li>
          C compiler (e.g. <a href="http://gcc.gnu.org/">GNU C Compiler</a>)
        </li>
        <li>
          GNU Autools Chain: 
          <a href="http://www.gnu.org/software/autoconf/">autoconf</a>, 
          <a href="http://www.gnu.org/software/automake/">automake</a>, 
          <a href="http://www.gnu.org/software/libtool/">libtool</a>
        </li>
        <li> 
          zlib-development package (stable version >= 1.2.0)
        </li>
      </ul>

      <p>Once you have the pre-requisites use the standard <code>build.xml</code> 
      and pass along the <code>compile.native</code> flag (set to 
      <code>true</code>) to build the native hadoop library:</p>

      <p><code>$ ant -Dcompile.native=true &lt;target&gt;</code></p>

      <p>The native hadoop library is not built by default since not everyone is 
      interested in building them.</p>

      <p>You should see the newly-built native hadoop library in:</p>

      <p><code>$ build/native/&lt;platform&gt;/lib</code></p>

      <p>where &lt;platform&gt; is combination of the system-properties: 
      <code>${os.name}-${os.arch}-${sun.arch.data.model}</code>; for e.g. 
      Linux-i386-32.</p>

      <section>
        <title>Notes</title>
        
        <ul>
          <li>
            It is <strong>mandatory</strong> to have the 
            zlib, gzip, and bzip2
            development packages on the target platform for building the 
            native hadoop library; however for deployment it is sufficient to 
            install one of them if you wish to use only one of them.
          </li>
          <li>
            It is necessary to have the correct 32/64 libraries of both zlib 
            depending on the 32/64 bit jvm for the target platform for 
            building/deployment of the native hadoop library.
          </li>
        </ul>
      </section>
    </section>
    <section>
      <title> Loading native libraries through DistributedCache </title>
      <p>User can load native shared libraries through  
      <a href="mapred_tutorial.html#DistributedCache">DistributedCache</a>
      for <em>distributing</em> and <em>symlinking</em> the library files</p>
      
      <p>Here is an example, describing how to distribute the library and
      load it from map/reduce task. </p>
      <ol>
      <li> First copy the library to the HDFS. <br/>
      <code>bin/hadoop fs -copyFromLocal mylib.so.1 /libraries/mylib.so.1</code>
      </li>
      <li> The job launching program should contain the following: <br/>
      <code> DistributedCache.createSymlink(conf); </code> <br/>
      <code> DistributedCache.addCacheFile("hdfs://host:port/libraries/mylib.so.1#mylib.so", conf);
      </code>
      </li>
      <li> The map/reduce task can contain: <br/>
      <code> System.loadLibrary("mylib.so"); </code>
      </li>
      </ol>
    </section>
  </body>
  
</document>
