Package org.apache.spark.sql

Class MergeIntoWriter<T>

Object
org.apache.spark.sql.MergeIntoWriter<T>
Type Parameters:
T - the type of data in the Dataset. param: table the name of the target table for the merge operation. param: ds the source Dataset to merge into the target table. param: on the merge condition. param: schemaEvolutionEnabled whether to enable automatic schema evolution for this merge operation. Default is false.

public class MergeIntoWriter<T> extends Object
MergeIntoWriter provides methods to define and execute merge actions based on specified conditions.

Since:
4.0.0

  • Method Details

    • whenMatched

      public WhenMatched<T> whenMatched()
      Initialize a WhenMatched action without any condition.

      This WhenMatched action will be executed when a source row matches a target table row based on the merge condition.

      This WhenMatched can be followed by one of the following merge actions: - updateAll: Update all the matched target table rows with source dataset rows. - update(Map): Update all the matched target table rows while changing only a subset of columns based on the provided assignment. - delete: Delete all target rows that have a match in the source table.

      Returns:
      a new WhenMatched object.

    • whenMatched

      public WhenMatched<T> whenMatched(Column condition)
      Initialize a WhenMatched action with a condition.

      This WhenMatched action will be executed when a source row matches a target table row based on the merge condition and the specified condition is satisfied.

      This WhenMatched can be followed by one of the following merge actions: - updateAll: Update all the matched target table rows with source dataset rows. - update(Map): Update all the matched target table rows while changing only a subset of columns based on the provided assignment. - delete: Delete all target rows that have a match in the source table.

      Parameters:
      condition - a Column representing the condition to be evaluated for the action.
      Returns:
      a new WhenMatched object configured with the specified condition.

    • whenNotMatched

      public WhenNotMatched<T> whenNotMatched()
      Initialize a WhenNotMatched action without any condition.

      This WhenNotMatched action will be executed when a source row does not match any target row based on the merge condition.

      This WhenNotMatched can be followed by one of the following merge actions: - insertAll: Insert all rows from the source that are not already in the target table. - insert(Map): Insert all rows from the source that are not already in the target table, with the specified columns based on the provided assignment.

      Returns:
      a new WhenNotMatched object.

    • whenNotMatched

      public WhenNotMatched<T> whenNotMatched(Column condition)
      Initialize a WhenNotMatched action with a condition.

      This WhenNotMatched action will be executed when a source row does not match any target row based on the merge condition and the specified condition is satisfied.

      This WhenNotMatched can be followed by one of the following merge actions: - insertAll: Insert all rows from the source that are not already in the target table. - insert(Map): Insert all rows from the source that are not already in the target table, with the specified columns based on the provided assignment.

      Parameters:
      condition - a Column representing the condition to be evaluated for the action.
      Returns:
      a new WhenNotMatched object configured with the specified condition.

    • whenNotMatchedBySource

      public WhenNotMatchedBySource<T> whenNotMatchedBySource()
      Initialize a WhenNotMatchedBySource action without any condition.

      This WhenNotMatchedBySource action will be executed when a target row does not match any rows in the source table based on the merge condition.

      This WhenNotMatchedBySource can be followed by one of the following merge actions: - updateAll: Update all the not matched target table rows with source dataset rows. - update(Map): Update all the not matched target table rows while changing only the specified columns based on the provided assignment. - delete: Delete all target rows that have no matches in the source table.

      Returns:
      a new WhenNotMatchedBySource object.

    • whenNotMatchedBySource

      public WhenNotMatchedBySource<T> whenNotMatchedBySource(Column condition)
      Initialize a WhenNotMatchedBySource action with a condition.

      This WhenNotMatchedBySource action will be executed when a target row does not match any rows in the source table based on the merge condition and the specified condition is satisfied.

      This WhenNotMatchedBySource can be followed by one of the following merge actions: - updateAll: Update all the not matched target table rows with source dataset rows. - update(Map): Update all the not matched target table rows while changing only the specified columns based on the provided assignment. - delete: Delete all target rows that have no matches in the source table.

      Parameters:
      condition - a Column representing the condition to be evaluated for the action.
      Returns:
      a new WhenNotMatchedBySource object configured with the specified condition.

    • withSchemaEvolution

      public MergeIntoWriter<T> withSchemaEvolution()
      Enable automatic schema evolution for this merge operation.
      Returns:
      A MergeIntoWriter instance with schema evolution enabled.

    • merge

      public void merge()
      Executes the merge operation.