org.apache.hadoop.hbase.regionserver
Class Store

java.lang.Object
  extended by org.apache.hadoop.hbase.regionserver.metrics.SchemaConfigured
      extended by org.apache.hadoop.hbase.regionserver.Store
All Implemented Interfaces:
HeapSize, SchemaMetrics.SchemaAware

public class Store
extends SchemaConfigured
implements HeapSize

A Store holds a column family in a Region. Its a memstore and a set of zero or more StoreFiles, which stretch backwards over time.

There's no reason to consider append-logging at this level; all logging and locking is handled at the HRegion level. Store just provides services to manage sets of StoreFiles. One of the most important of those services is compaction services where files are aggregated once they pass a configurable threshold.

The only thing having to do with logs that Store needs to deal with is the reconstructionLog. This is a segment of an HRegion's log that might NOT be present upon startup. If the param is NULL, there's nothing to do. If the param is non-NULL, we need to process the log to reconstruct a TreeMap that might not have been written to disk before the process died.

It's assumed that after this constructor returns, the reconstructionLog file will be deleted (by whoever has instantiated the Store).

Locking and transactions are handled at a higher level. This API should not be called directly but by an HRegion manager.


Nested Class Summary
static class Store.ScanInfo
          Immutable information for scans over a store.
 
Field Summary
static String BLOCKING_STOREFILES_KEY
           
static long DEEP_OVERHEAD
           
static int DEFAULT_BLOCKING_STOREFILE_COUNT
           
static long FIXED_OVERHEAD
           
protected  MemStore memstore
           
static int NO_PRIORITY
           
static int PRIORITY_USER
           
 
Fields inherited from class org.apache.hadoop.hbase.regionserver.metrics.SchemaConfigured
SCHEMA_CONFIGURED_UNALIGNED_HEAP_SIZE
 
Constructor Summary
protected Store(org.apache.hadoop.fs.Path basedir, HRegion region, HColumnDescriptor family, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration confParam)
          Constructor
 
Method Summary
protected  long add(KeyValue kv)
          Adds a value to the memstore
 void bulkLoadHFile(String srcPathStr, long seqNum)
          This method should only be called from HRegion.
 boolean canSplit()
           
 void compactRecentForTesting(int N)
          Compact the most recent N files.
 StoreFile.Writer createWriterInTmp(int maxKeyCount, Compression.Algorithm compression, boolean isCompaction, boolean includeMVCCReadpoint)
           
protected  long delete(KeyValue kv)
          Adds a value to the memstore
 void finishRequest(CompactionRequest cr)
           
protected  org.apache.hadoop.fs.Path flushCache(long logCacheFlushId, SortedSet<KeyValue> snapshot, TimeRangeTracker snapshotTimeRangeTracker, AtomicLong flushedSize, MonitoredTask status)
          Write out current snapshot.
static int getBytesPerChecksum(org.apache.hadoop.conf.Configuration conf)
          Returns the configured bytesPerChecksum value.
 CacheConfig getCacheConfig()
          Used for tests.
static ChecksumType getChecksumType(org.apache.hadoop.conf.Configuration conf)
          Returns the configured checksum algorithm.
 CompactionProgress getCompactionProgress()
          getter for CompactionProgress object
 int getCompactPriority()
           
 int getCompactPriority(int priority)
           
 KeyValue.KVComparator getComparator()
           
 HFileDataBlockEncoder getDataBlockEncoder()
           
 HColumnDescriptor getFamily()
           
 HRegion getHRegion()
           
 long getLastCompactSize()
           
static long getLowestTimestamp(List<StoreFile> candidates)
           
 long getMaxMemstoreTS()
           
 int getNumberOfStoreFiles()
           
 Store.ScanInfo getScanInfo()
           
 KeyValueScanner getScanner(Scan scan, NavigableSet<byte[]> targetCols)
          Return a scanner for both the memstore and the HStore files.
protected  List<KeyValueScanner> getScanners(boolean cacheBlocks, boolean usePread, boolean isCompaction, ScanQueryMatcher matcher)
          Get all scanners with no filtering based on TTL (that happens further down the line).
 long getSize()
           
 byte[] getSplitPoint()
          Determines if Store should be split
 List<StoreFile> getStorefiles()
           
 org.apache.hadoop.hbase.regionserver.StoreFlusher getStoreFlusher(long cacheFlushId)
           
static org.apache.hadoop.fs.Path getStoreHomedir(org.apache.hadoop.fs.Path parentRegionDirectory, byte[] family)
           
static org.apache.hadoop.fs.Path getStoreHomedir(org.apache.hadoop.fs.Path tabledir, String encodedName, byte[] family)
           
static org.apache.hadoop.fs.Path getStoreHomedir(org.apache.hadoop.fs.Path tabledir, String encodedName, String family)
           
 boolean hasTooManyStoreFiles()
           
 long heapSize()
           
 boolean needsCompaction()
          See if there's too much store files in this store
 CompactionRequest requestCompaction()
           
 CompactionRequest requestCompaction(int priority, CompactionRequest request)
           
protected  void rollback(KeyValue kv)
          Removes a kv from the memstore.
 com.google.common.collect.ImmutableList<StoreFile> sortAndClone(List<StoreFile> storeFiles)
           
 long timeOfOldestEdit()
          When was the oldest edit done in the memstore
 String toString()
           
 void triggerMajorCompaction()
           
 long updateColumnValue(byte[] row, byte[] f, byte[] qualifier, long newValue)
          Increments the value for the given row/family/qualifier.
 long upsert(List<KeyValue> kvs)
          Adds or replaces the specified KeyValues.
 
Methods inherited from class org.apache.hadoop.hbase.regionserver.metrics.SchemaConfigured
createUnknown, getColumnFamilyName, getSchemaMetrics, getTableName, isSchemaConfigured, passSchemaMetricsTo, resetSchemaMetricsConf, schemaConfAsJSON, schemaConfigurationChanged
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

BLOCKING_STOREFILES_KEY

public static final String BLOCKING_STOREFILES_KEY
See Also:
Constant Field Values

DEFAULT_BLOCKING_STOREFILE_COUNT

public static final int DEFAULT_BLOCKING_STOREFILE_COUNT
See Also:
Constant Field Values

memstore

protected final MemStore memstore

PRIORITY_USER

public static final int PRIORITY_USER
See Also:
Constant Field Values

NO_PRIORITY

public static final int NO_PRIORITY
See Also:
Constant Field Values

FIXED_OVERHEAD

public static final long FIXED_OVERHEAD

DEEP_OVERHEAD

public static final long DEEP_OVERHEAD
Constructor Detail

Store

protected Store(org.apache.hadoop.fs.Path basedir,
                HRegion region,
                HColumnDescriptor family,
                org.apache.hadoop.fs.FileSystem fs,
                org.apache.hadoop.conf.Configuration confParam)
         throws IOException
Constructor

Parameters:
basedir - qualified path under which the region directory lives; generally the table subdirectory
region -
family - HColumnDescriptor for this column
fs - file system object
confParam - configuration object failed. Can be null.
Throws:
IOException
Method Detail

getBytesPerChecksum

public static int getBytesPerChecksum(org.apache.hadoop.conf.Configuration conf)
Returns the configured bytesPerChecksum value.

Parameters:
conf - The configuration
Returns:
The bytesPerChecksum that is set in the configuration

getChecksumType

public static ChecksumType getChecksumType(org.apache.hadoop.conf.Configuration conf)
Returns the configured checksum algorithm.

Parameters:
conf - The configuration
Returns:
The checksum algorithm that is set in the configuration

getFamily

public HColumnDescriptor getFamily()

getMaxMemstoreTS

public long getMaxMemstoreTS()
Returns:
The maximum memstoreTS in all store files.

getStoreHomedir

public static org.apache.hadoop.fs.Path getStoreHomedir(org.apache.hadoop.fs.Path tabledir,
                                                        String encodedName,
                                                        byte[] family)
Parameters:
tabledir -
encodedName - Encoded region name.
family -
Returns:
Path to family/Store home directory.

getStoreHomedir

public static org.apache.hadoop.fs.Path getStoreHomedir(org.apache.hadoop.fs.Path tabledir,
                                                        String encodedName,
                                                        String family)
Parameters:
tabledir -
encodedName - Encoded region name.
family -
Returns:
Path to family/Store home directory.

getStoreHomedir

public static org.apache.hadoop.fs.Path getStoreHomedir(org.apache.hadoop.fs.Path parentRegionDirectory,
                                                        byte[] family)
Parameters:
parentRegionDirectory - directory for the parent region
family - family name of this store
Returns:
Path to the family/Store home directory

getDataBlockEncoder

public HFileDataBlockEncoder getDataBlockEncoder()
Returns:
the data block encoder

add

protected long add(KeyValue kv)
Adds a value to the memstore

Parameters:
kv -
Returns:
memstore size delta

timeOfOldestEdit

public long timeOfOldestEdit()
When was the oldest edit done in the memstore


delete

protected long delete(KeyValue kv)
Adds a value to the memstore

Parameters:
kv -
Returns:
memstore size delta

rollback

protected void rollback(KeyValue kv)
Removes a kv from the memstore. The KeyValue is removed only if its key & memstoreTS matches the key & memstoreTS value of the kv parameter.

Parameters:
kv -

getStorefiles

public List<StoreFile> getStorefiles()
Returns:
All store files.

bulkLoadHFile

public void bulkLoadHFile(String srcPathStr,
                          long seqNum)
                   throws IOException
This method should only be called from HRegion. It is assumed that the ranges of values in the HFile fit within the stores assigned region. (assertBulkLoadHFileOk checks this)

Throws:
IOException

flushCache

protected org.apache.hadoop.fs.Path flushCache(long logCacheFlushId,
                                               SortedSet<KeyValue> snapshot,
                                               TimeRangeTracker snapshotTimeRangeTracker,
                                               AtomicLong flushedSize,
                                               MonitoredTask status)
                                        throws IOException
Write out current snapshot. Presumes snapshot() has been called previously.

Parameters:
logCacheFlushId - flush sequence number
snapshot -
snapshotTimeRangeTracker -
flushedSize - The number of bytes flushed
status -
Returns:
Path The path name of the tmp file to which the store was flushed
Throws:
IOException

createWriterInTmp

public StoreFile.Writer createWriterInTmp(int maxKeyCount,
                                          Compression.Algorithm compression,
                                          boolean isCompaction,
                                          boolean includeMVCCReadpoint)
                                   throws IOException
Throws:
IOException

getScanners

protected List<KeyValueScanner> getScanners(boolean cacheBlocks,
                                            boolean usePread,
                                            boolean isCompaction,
                                            ScanQueryMatcher matcher)
                                     throws IOException
Get all scanners with no filtering based on TTL (that happens further down the line).

Returns:
all scanners for this store
Throws:
IOException

compactRecentForTesting

public void compactRecentForTesting(int N)
                             throws IOException
Compact the most recent N files. Used in testing.

Throws:
IOException

getLowestTimestamp

public static long getLowestTimestamp(List<StoreFile> candidates)
                               throws IOException
Throws:
IOException

getCompactionProgress

public CompactionProgress getCompactionProgress()
getter for CompactionProgress object

Returns:
CompactionProgress object; can be null

requestCompaction

public CompactionRequest requestCompaction()
                                    throws IOException
Throws:
IOException

requestCompaction

public CompactionRequest requestCompaction(int priority,
                                           CompactionRequest request)
                                    throws IOException
Throws:
IOException

finishRequest

public void finishRequest(CompactionRequest cr)

sortAndClone

public com.google.common.collect.ImmutableList<StoreFile> sortAndClone(List<StoreFile> storeFiles)

getNumberOfStoreFiles

public int getNumberOfStoreFiles()
Returns:
the number of files in this store

canSplit

public boolean canSplit()

getSplitPoint

public byte[] getSplitPoint()
Determines if Store should be split

Returns:
byte[] if store should be split, null otherwise.

getLastCompactSize

public long getLastCompactSize()
Returns:
aggregate size of all HStores used in the last compaction

getSize

public long getSize()
Returns:
aggregate size of HStore

triggerMajorCompaction

public void triggerMajorCompaction()

getScanner

public KeyValueScanner getScanner(Scan scan,
                                  NavigableSet<byte[]> targetCols)
                           throws IOException
Return a scanner for both the memstore and the HStore files. Assumes we are not in a compaction.

Throws:
IOException

toString

public String toString()
Overrides:
toString in class Object

getCompactPriority

public int getCompactPriority()

getCompactPriority

public int getCompactPriority(int priority)
Parameters:
priority -
Returns:
The priority that this store should have in the compaction queue

getHRegion

public HRegion getHRegion()

updateColumnValue

public long updateColumnValue(byte[] row,
                              byte[] f,
                              byte[] qualifier,
                              long newValue)
                       throws IOException
Increments the value for the given row/family/qualifier. This function will always be seen as atomic by other readers because it only puts a single KV to memstore. Thus no read/write control necessary.

Parameters:
row -
f -
qualifier -
newValue - the new value to set into memstore
Returns:
memstore size delta
Throws:
IOException

upsert

public long upsert(List<KeyValue> kvs)
            throws IOException
Adds or replaces the specified KeyValues.

For each KeyValue specified, if a cell with the same row, family, and qualifier exists in MemStore, it will be replaced. Otherwise, it will just be inserted to MemStore.

This operation is atomic on each KeyValue (row/family/qualifier) but not necessarily atomic across all of them.

Parameters:
kvs -
Returns:
memstore size delta
Throws:
IOException

getStoreFlusher

public org.apache.hadoop.hbase.regionserver.StoreFlusher getStoreFlusher(long cacheFlushId)

needsCompaction

public boolean needsCompaction()
See if there's too much store files in this store

Returns:
true if number of store files is greater than the number defined in minFilesToCompact

getCacheConfig

public CacheConfig getCacheConfig()
Used for tests. Get the cache configuration for this Store.


heapSize

public long heapSize()
Specified by:
heapSize in interface HeapSize
Overrides:
heapSize in class SchemaConfigured
Returns:
Approximate 'exclusive deep size' of implementing object. Includes count of payload and hosting object sizings.

getComparator

public KeyValue.KVComparator getComparator()

getScanInfo

public Store.ScanInfo getScanInfo()

hasTooManyStoreFiles

public boolean hasTooManyStoreFiles()


Copyright © 2013 The Apache Software Foundation. All Rights Reserved.