Class

com.qubole.spark.hiveacid

HiveAcidOperationDelegate

Related Doc: package hiveacid

Permalink

class HiveAcidOperationDelegate extends AcidOperationDelegate with Logging

HiveAcidTable uses Delegate pattern to delegate it's API to this object

Linear Supertypes
Logging, AcidOperationDelegate, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. HiveAcidOperationDelegate
  2. Logging
  3. AcidOperationDelegate
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new HiveAcidOperationDelegate(sparkSession: SparkSession, hiveAcidMetadata: HiveAcidMetadata, parameters: Map[String, String])

    Permalink

    sparkSession

    - spark session object

    hiveAcidMetadata

    - metadata object

    parameters

    - additional parameters

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. def addBatch(df: DataFrame, curTxn: HiveAcidTxn): Unit

    Permalink

    Used by streaming query to add a datframe to hive acid table.

    Used by streaming query to add a datframe to hive acid table.

    df

    - dataframe to insert

    curTxn

    - Transaction under which the operation is being done

    Definition Classes
    HiveAcidOperationDelegateAcidOperationDelegate
  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  7. def delete(condition: Column, curTxn: HiveAcidTxn): Unit

    Permalink

    Delete rows from the table based on condtional expression.

    Delete rows from the table based on condtional expression.

    Note: This API is transactional in nature.

    condition

    - Boolean SQL Expression filtering rows to be deleted

    curTxn

    - Transaction under which DELETE will be performed

    Definition Classes
    HiveAcidOperationDelegateAcidOperationDelegate
  8. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  12. def getRdd(requiredColumns: Array[String], filters: Array[Filter], readConf: SparkAcidConf, curTxn: HiveAcidTxn): RDD[Row]

    Permalink

    Return an RDD on top of Hive ACID table

    Return an RDD on top of Hive ACID table

    requiredColumns

    - columns needed

    filters

    - filters that can be pushed down to file format

    readConf

    - read conf

    curTxn

    - current Transaction under which ead will be done

    Definition Classes
    HiveAcidOperationDelegateAcidOperationDelegate
  13. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  14. val hiveAcidMetadata: HiveAcidMetadata

    Permalink

    - metadata object

  15. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  16. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  17. def insertInto(df: DataFrame, curTxn: HiveAcidTxn, statementId: Option[Int] = None): Unit

    Permalink

    Appends a given dataframe df into the hive acid table

    Appends a given dataframe df into the hive acid table

    Note: This API is transactional in nature.

    df

    - dataframe to insert

    curTxn

    - current transaction underwhich INSERT operation needs to run.

    statementId

    Optional. In a same transaction, multiple statements like INSERT/UPDATE/DELETE (like in case of MERGE) can be issued. statementId has to be different for them to ensure delta collision is avoided for them during writes.

    Definition Classes
    HiveAcidOperationDelegateAcidOperationDelegate
  18. def insertOverwrite(df: DataFrame, curTxn: HiveAcidTxn, statementId: Option[Int] = None): Unit

    Permalink

    Overwrites a given dataframe df onto the hive acid table

    Overwrites a given dataframe df onto the hive acid table

    Note: This API is transactional in nature.

    df

    - dataframe to insert

    curTxn

    - current transaction under which insertOverWrite need to be run

    statementId

    Optional. In a same transaction, multiple statements like INSERT/UPDATE/DELETE (like in case of MERGE) can be issued. statementId has to be different for them to ensure delta collision is avoided for them during writes.

    Definition Classes
    HiveAcidOperationDelegateAcidOperationDelegate
  19. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  20. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  21. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  22. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  23. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  24. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  25. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  26. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  27. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  28. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  29. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  30. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  31. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  32. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  33. def merge(sourceDf: DataFrame, hiveAcidMetadata: HiveAcidMetadata, mergeExpression: Expression, matchedClause: Seq[MergeWhenClause], notMatched: Option[MergeWhenNotInsert], sourceAlias: Option[AliasIdentifier], targetAlias: Option[AliasIdentifier], curTxn: HiveAcidTxn): Unit

    Permalink

    Merge from sourceDf to the current Table

    Merge from sourceDf to the current Table

    Definition Classes
    HiveAcidOperationDelegateAcidOperationDelegate
  34. def mergeDelete(deleteDf: DataFrame, curTxn: HiveAcidTxn, statementId: Option[Int] = None): Unit

    Permalink

    NOT TO BE USED EXTERNALLY Not protected by transactionality, it is assumed curTxn is already set before calling this

    NOT TO BE USED EXTERNALLY Not protected by transactionality, it is assumed curTxn is already set before calling this

    deleteDf

    DataFrame to be used to update is supposed to have same schema as tableSchemaWithRowId

    curTxn

    Transaction under which this operation will be performed

    statementId

    Optional. In a same transaction, multiple statements like INSERT/UPDATE/DELETE (like in case of MERGE) can be issued. statementId has to be different for them to ensure delta collision is avoided for them during writes.

    Definition Classes
    HiveAcidOperationDelegateAcidOperationDelegate
  35. def mergeUpdate(updateDf: DataFrame, curTxn: HiveAcidTxn, statementId: Option[Int] = None): Unit

    Permalink

    Uodated Df is written to the table

    Uodated Df is written to the table

    updateDf

    DataFrame to be used to update is supposed to have same schema as tableSchemaWithRowId

    curTxn

    - Transaction under which this operation will be performed

    statementId

    Optional. In a same transaction, multiple statements like INSERT/UPDATE/DELETE (like in case of MERGE) can be issued. statementId has to be different for them to ensure delta collision is avoided for them during writes.

    Definition Classes
    HiveAcidOperationDelegateAcidOperationDelegate
  36. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  37. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  38. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  39. val parameters: Map[String, String]

    Permalink

    - additional parameters

  40. val sparkSession: SparkSession

    Permalink

    - spark session object

  41. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  42. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  43. def update(condition: Option[Column], newValues: Map[String, Column], curTxn: HiveAcidTxn): Unit

    Permalink

    Update rows in the hive acid table based on condition and newValues

    Update rows in the hive acid table based on condition and newValues

    Note: This API is transactional in nature.

    condition

    - Optional condition string to identify rows which needs to be updated, if not specified then it means complete table.

    newValues

    - Map of (column, value) to set

    Definition Classes
    HiveAcidOperationDelegateAcidOperationDelegate
  44. def update(condition: Option[String], newValues: Map[String, String], curTxn: HiveAcidTxn): Unit

    Permalink

    Update rows in the hive acid table based on condition and newValues

    Update rows in the hive acid table based on condition and newValues

    Note: This API is transactional in nature.

    condition

    - condition string to identify rows which needs to be updated

    newValues

    - Map of (column, value) to set

    Definition Classes
    HiveAcidOperationDelegateAcidOperationDelegate
  45. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  46. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  47. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from AcidOperationDelegate

Inherited from AnyRef

Inherited from Any

Ungrouped