Trait

com.qubole.spark.hiveacid

AcidOperationDelegate

Related Doc: package hiveacid

Permalink

trait AcidOperationDelegate extends AnyRef

Delegate to perform ACID operations: UPDATE/DELETE/INSERT/READ These APIs are not protected under Transactions, however most of them may need transactions to be already open.

Linear Supertypes
AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AcidOperationDelegate
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def addBatch(df: DataFrame, curTxn: HiveAcidTxn): Unit

    Permalink

    Used by streaming query to add a datframe to hive acid table.

    Used by streaming query to add a datframe to hive acid table.

    df

    - dataframe to insert

    curTxn

    - Transaction under which the operation is being done

  2. abstract def delete(condition: Column, curTxn: HiveAcidTxn): Unit

    Permalink

    Delete rows from the table based on condtional expression.

    Delete rows from the table based on condtional expression.

    Note: This API is transactional in nature.

    condition

    - Boolean SQL Expression filtering rows to be deleted

    curTxn

    - Transaction under which DELETE will be performed

  3. abstract def getRdd(requiredColumns: Array[String], filters: Array[Filter], readConf: SparkAcidConf, curTxn: HiveAcidTxn): RDD[Row]

    Permalink

    Return an RDD on top of Hive ACID table

    Return an RDD on top of Hive ACID table

    requiredColumns

    - columns needed

    filters

    - filters that can be pushed down to file format

    readConf

    - read conf

    curTxn

    - current Transaction under which ead will be done

  4. abstract def insertInto(df: DataFrame, curTxn: HiveAcidTxn, statementId: Option[Int] = None): Unit

    Permalink

    Appends a given dataframe df into the hive acid table

    Appends a given dataframe df into the hive acid table

    Note: This API is transactional in nature.

    df

    - dataframe to insert

    curTxn

    - current transaction underwhich INSERT operation needs to run.

    statementId

    Optional. In a same transaction, multiple statements like INSERT/UPDATE/DELETE (like in case of MERGE) can be issued. statementId has to be different for them to ensure delta collision is avoided for them during writes.

  5. abstract def insertOverwrite(df: DataFrame, curTxn: HiveAcidTxn, statementId: Option[Int] = None): Unit

    Permalink

    Overwrites a given dataframe df onto the hive acid table

    Overwrites a given dataframe df onto the hive acid table

    Note: This API is transactional in nature.

    df

    - dataframe to insert

    curTxn

    - current transaction under which insertOverWrite need to be run

    statementId

    Optional. In a same transaction, multiple statements like INSERT/UPDATE/DELETE (like in case of MERGE) can be issued. statementId has to be different for them to ensure delta collision is avoided for them during writes.

  6. abstract def merge(sourceDf: DataFrame, hiveAcidMetadata: HiveAcidMetadata, mergeExpression: Expression, matchedClause: Seq[MergeWhenClause], notMatched: Option[MergeWhenNotInsert], sourceAlias: Option[AliasIdentifier], targetAlias: Option[AliasIdentifier], txn: HiveAcidTxn): Unit

    Permalink

    Merge from sourceDf to the current Table

  7. abstract def mergeDelete(mergeDeleteDf: DataFrame, curTxn: HiveAcidTxn, statementId: Option[Int] = None): Unit

    Permalink

    Delete dataframe of rows specified by MERGE Delete statement

    Delete dataframe of rows specified by MERGE Delete statement

    mergeDeleteDf

    dataframe of rows specified by MERGE Delete statement

    curTxn

    Transaction under which this operation will be performed

    statementId

    Specify Statement Id, for multi operation command like Merge statement Id need to be unique for every operation

  8. abstract def mergeUpdate(mergeUpdateDf: DataFrame, curTxn: HiveAcidTxn, statementId: Option[Int] = None): Unit

    Permalink

    Update dataframe with updated rows specified by MERGE update statement

    Update dataframe with updated rows specified by MERGE update statement

    mergeUpdateDf

    - Dataframe with updated rows specified by MERGE update statement

    curTxn

    - Transaction under which this operation will be performed

    statementId

    - Specify Statement Id, for multi operation command like Merge statement Id need to be unique for every operation

  9. abstract def update(condition: Option[Column], newValues: Map[String, Column], curTxn: HiveAcidTxn): Unit

    Permalink
  10. abstract def update(condition: Option[String], newValues: Map[String, String], curTxn: HiveAcidTxn): Unit

    Permalink

    Update rows in the hive acid table based on condition and newValues

    Update rows in the hive acid table based on condition and newValues

    Note: This API is transactional in nature.

    condition

    - condition string to identify rows which needs to be updated

    newValues

    - Map of (column, value) to set

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  10. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  11. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  13. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  14. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  15. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  16. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  17. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped