ChecksumFileSystem (Hadoop 0.20.2+320 API)
Date: 2010-06-28

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.apache.hadoop.fs
Class ChecksumFileSystem

    java.lang.Object
      org.apache.hadoop.conf.Configured
          org.apache.hadoop.fs.FileSystem
              org.apache.hadoop.fs.FilterFileSystem
                  org.apache.hadoop.fs.ChecksumFileSystem

All Implemented Interfaces:
  Closeable, Configurable

Direct Known Subclasses:
  InMemoryFileSystem, LocalFileSystem

    public abstract class ChecksumFileSystem

extends FilterFileSystem

Abstract Checksumed FileSystem. It provide a basice implementation of a Checksumed FileSystem, which creates a checksum file for each raw file. It generates & verifies checksums at the client side.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem
`FileSystem.Statistics`

Field Summary

Fields inherited from class org.apache.hadoop.fs.FilterFileSystem
`fs`

Fields inherited from class org.apache.hadoop.fs.FileSystem
`FS_DEFAULT_NAME_KEY, LOG, statistics`

Constructor Summary
`ChecksumFileSystem(FileSystem fs)`

Method Summary
`FSDataOutputStream` | `append(Path f, int bufferSize, Progressable progress)` Append to an existing file (optional operation).
`void` | `completeLocalOutput(Path fsOutputFile, Path tmpLocalFile)` Called when we're all done writing to the target.
`void` | `copyFromLocalFile(boolean delSrc, Path src, Path dst)` The src file is on the local disk.
`void` | `copyToLocalFile(boolean delSrc, Path src, Path dst)` The src file is under FS, and the dst is on the local disk.
`void` | `copyToLocalFile(Path src, Path dst, boolean copyCrc)` The src file is under FS, and the dst is on the local disk.
`FSDataOutputStream` | `create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress)` Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
`boolean` | `delete(Path f, boolean recursive)` Implement the delete(Path, boolean) in checksum file system.
`static double` | `getApproxChkSumLength(long size)`
`int` | `getBytesPerSum()` Return the bytes Per Checksum
`Path` | `getChecksumFile(Path file)` Return the name of the checksum file associated with a file.
`long` | `getChecksumFileLength(Path file, long fileSize)` Return the length of the checksum file given the size of the actual file.
`static long` | `getChecksumLength(long size, int bytesPerSum)` Calculated the length of the checksum file in bytes.
`FileSystem` | `getRawFileSystem()` get the raw file system
`static boolean` | `isChecksumFile(Path file)` Return true iff file is a checksum file name.
`FileStatus[]` | `listStatus(Path f)` List the statuses of the files/directories in the given path if the path is a directory.
`boolean` | `mkdirs(Path f)` Call `FileSystem.mkdirs(Path, FsPermission)` with default permission.
`FSDataInputStream` | `open(Path f, int bufferSize)` Opens an FSDataInputStream at the indicated Path.
`boolean` | `rename(Path src, Path dst)` Rename files/dirs
`boolean` | `reportChecksumFailure(Path f, FSDataInputStream in, long inPos, FSDataInputStream sums, long sumsPos)` Report a checksum error to the file system.
`void` | `setConf(Configuration conf)` Set the configuration to be used by this object.
`boolean` | `setReplication(Path src, short replication)` Set replication for an existing file.
`void` | `setVerifyChecksum(boolean verifyChecksum)` Set whether to verify checksum.
`Path` | `startLocalOutput(Path fsOutputFile, Path tmpLocalFile)` Returns a local File that the user can write output to.

Methods inherited from class org.apache.hadoop.fs.FilterFileSystem
`checkPath, close, delete, getConf, getDefaultBlockSize, getDefaultReplication, getFileBlockLocations, getFileChecksum, getFileStatus, getHomeDirectory, getName, getStatus, getUri, getWorkingDirectory, initialize, makeQualified, mkdirs, setOwner, setPermission, setWorkingDirectory`

Methods inherited from class org.apache.hadoop.fs.FileSystem
`append, append, clearStatistics, closeAll, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, createNewFile, deleteOnExit, exists, get, get, getAllStatistics, getBlockSize, getContentSummary, getDefaultUri, getLength, getLocal, getNamed, getReplication, getStatistics, getStatistics, getStatus, getUsed, globStatus, globStatus, isDirectory, isFile, listStatus, listStatus, listStatus, mkdirs, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstanceLocal, open, printStatistics, processDeleteOnExit, setDefaultUri, setDefaultUri, setTimes`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

### ChecksumFileSystem

    public ChecksumFileSystem(FileSystem fs)

Method Detail

### getApproxChkSumLength

    public static double getApproxChkSumLength(long size)

### setConf

    public void setConf(Configuration conf)

  Description copied from interface: `Configurable`
  Set the configuration to be used by this object.

  Specified by:
    `setConf` in interface `Configurable`
  Overrides:
    `setConf` in class `Configured`

### setVerifyChecksum

    public void setVerifyChecksum(boolean verifyChecksum)

  Set whether to verify checksum.

  Overrides:
    `setVerifyChecksum` in class `FilterFileSystem`

### getRawFileSystem

    public FileSystem getRawFileSystem()

  get the raw file system

### getChecksumFile

    public Path getChecksumFile(Path file)

  Return the name of the checksum file associated with a file.

### isChecksumFile

    public static boolean isChecksumFile(Path file)

  Return true iff file is a checksum file name.

### getChecksumFileLength

    public long getChecksumFileLength(Path file,
                                      long fileSize)

  Return the length of the checksum file given the size of the actual file.

### getBytesPerSum

    public int getBytesPerSum()

  Return the bytes Per Checksum

### open

    public FSDataInputStream open(Path f,
                                  int bufferSize)
                           throws IOException

  Opens an FSDataInputStream at the indicated Path.

  Overrides:
    `open` in class `FilterFileSystem`
  Parameters:
    `f` - the file name to open
    `bufferSize` - the size of the buffer to be used.
  Throws:
    `IOException`

### append

    public FSDataOutputStream append(Path f,
                                     int bufferSize,
                                     Progressable progress)
                              throws IOException

  Append to an existing file (optional operation).

  Overrides:
    `append` in class `FilterFileSystem`
  Parameters:
    `f` - the existing file to be appended.
    `bufferSize` - the size of the buffer to be used.
    `progress` - for reporting progress if it is not null.
  Throws:
    `IOException`

### getChecksumLength

    public static long getChecksumLength(long size,
                                         int bytesPerSum)

  Calculated the length of the checksum file in bytes.

  Parameters:
    `size` - the length of the data file in bytes
    `bytesPerSum` - the number of bytes in a checksum block
  Returns:
    the number of bytes in the checksum file

### create

    public FSDataOutputStream create(Path f,
                                     FsPermission permission,
                                     boolean overwrite,
                                     int bufferSize,
                                     short replication,
                                     long blockSize,
                                     Progressable progress)
                              throws IOException

  Opens an FSDataOutputStream at the indicated Path with write-progress reporting.

  Overrides:
    `create` in class `FilterFileSystem`
  Parameters:
    `f` - the file name to open
    `overwrite` - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
    `bufferSize` - the size of the buffer to be used.
    `replication` - required block replication for the file.
  Throws:
    `IOException`
  See Also:
    `FileSystem.setPermission(Path, FsPermission)`

### setReplication

    public boolean setReplication(Path src,
                                  short replication)
                           throws IOException

  Set replication for an existing file. Implement the abstract `setReplication` of `FileSystem`

  Overrides:
    `setReplication` in class `FilterFileSystem`
  Parameters:
    `src` - file name
    `replication` - new replication
  Returns:
    true if successful; false if file does not exist or is a directory
  Throws:
    `IOException`

### rename

    public boolean rename(Path src,
                          Path dst)
                   throws IOException

  Rename files/dirs

  Overrides:
    `rename` in class `FilterFileSystem`
  Throws:
    `IOException`

### delete

    public boolean delete(Path f,
                          boolean recursive)
                   throws IOException

  Implement the delete(Path, boolean) in checksum file system.

  Overrides:
    `delete` in class `FilterFileSystem`
  Parameters:
    `f` - the path to delete.
    `recursive` - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.
  Returns:
    true if delete is successful else false.
  Throws:
    `IOException`

### listStatus

    public FileStatus[] listStatus(Path f)
                            throws IOException

  List the statuses of the files/directories in the given path if the path is a directory.

  Overrides:
    `listStatus` in class `FilterFileSystem`
  Parameters:
    `f` - given path
  Returns:
    the statuses of the files/directories in the given patch
  Throws:
    `IOException`

### mkdirs

    public boolean mkdirs(Path f)
                   throws IOException

  Description copied from class: `FileSystem`
  Call `FileSystem.mkdirs(Path, FsPermission)` with default permission.

  Overrides:
    `mkdirs` in class `FileSystem`
  Throws:
    `IOException`

### copyFromLocalFile

    public void copyFromLocalFile(boolean delSrc,
                                  Path src,
                                  Path dst)
                           throws IOException

  Description copied from class: `FilterFileSystem`
  The src file is on the local disk. Add it to FS at the given dst name. delSrc indicates if the source should be removed

  Overrides:
    `copyFromLocalFile` in class `FilterFileSystem`
  Throws:
    `IOException`

### copyToLocalFile

    public void copyToLocalFile(boolean delSrc,
                                Path src,
                                Path dst)
                         throws IOException

  The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name.

  Overrides:
    `copyToLocalFile` in class `FilterFileSystem`
  Throws:
    `IOException`

### copyToLocalFile

    public void copyToLocalFile(Path src,
                                Path dst,
                                boolean copyCrc)
                         throws IOException

  The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name. If src and dst are directories, the copyCrc parameter determines whether to copy CRC files.

  Throws:
    `IOException`

### startLocalOutput

    public Path startLocalOutput(Path fsOutputFile,
                                 Path tmpLocalFile)
                          throws IOException

  Description copied from class: `FilterFileSystem`
  Returns a local File that the user can write output to. The caller provides both the eventual FS target name and the local working file. If the FS is local, we write directly into the target. If the FS is remote, we write into the tmp local area.

  Overrides:
    `startLocalOutput` in class `FilterFileSystem`
  Throws:
    `IOException`

### completeLocalOutput

    public void completeLocalOutput(Path fsOutputFile,
                                    Path tmpLocalFile)
                             throws IOException

  Description copied from class: `FilterFileSystem`
  Called when we're all done writing to the target. A local FS will do nothing, because we've written to exactly the right place. A remote FS will copy the contents of tmpLocalFile to the correct target at fsOutputFile.

  Overrides:
    `completeLocalOutput` in class `FilterFileSystem`
  Throws:
    `IOException`

### reportChecksumFailure

    public boolean reportChecksumFailure(Path f,
                                         FSDataInputStream in,
                                         long inPos,
                                         FSDataInputStream sums,
                                         long sumsPos)

  Report a checksum error to the file system.

  Parameters:
    `f` - the file name containing the error
    `in` - the stream open on the file
    `inPos` - the position of the beginning of the bad data in the file
    `sums` - the stream open on the checksum file
    `sumsPos` - the position of the beginning of the bad data in the checksum file
  Returns:
    if retry is neccessary

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2009 The Apache Software Foundation