[Spark] Python DeltaTableBuilder API for Identity Columns

[zhipengmao-db]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

This PR is part of #1959

In this PR, we extend the addColumn interface in DeltaTableBuilder to allow for Identity Columns creation.

Resolves #1072

How was this patch tested?

New tests.

Does this PR introduce any user-facing changes?

We update the arguments of addColumn method:

• Support a new data type for parameter generatedAlwaysAs. Users can specify generatedAlwaysAs as IdentityGenerator to add an identity column that is GENERATED ALWAYS.

• Add a new parameter generatedByDefaultAs. Users can specify generatedByDefaultAs as IdentityGenerator to add an identity column that is GENERATED BY DEFAULT.

• Users can optionally pass in start (default = 1) and step (default = 1) values to construct IdentityGenerator object, which specify the start and step value to generate the identity column.

Interface

 def addColumn(
        self,
        colName: str,
        dataType: Union[str, DataType],
        nullable: bool = True,
        generatedAlwaysAs: Optional[Union[str, IdentityGenerator]] = None,
        generatedByDefaultAs: Optional[IdentityGenerator] = None,
        comment: Optional[str] = None,
) -> "DeltaTableBuilder"

Example Usage

 DeltaTable.create()
    .tableName("tableName")
    .addColumn("id", dataType=LongType(), generatedAlwaysAs=IdentityGenerator())
    .execute()

 DeltaTable.create()
    .tableName("tableName")
    .addColumn("id", dataType=LongType(), generatedAlwaysAs=IdentityGenerator(start=1, step=1))
    .execute()

 DeltaTable.create()
    .tableName("tableName")
    .addColumn("id", dataType=LongType(), generatedByDefaultAs=IdentityGenerator())
    .execute()

 DeltaTable.create()
    .tableName("tableName")
    .addColumn("id", dataType=LongType(), generatedByDefaultAs=IdentityGenerator(start=1, step=1))
    .execute()

[bart-samwel]

API looks nice like this BTW!

[c27kwan]

Nice!

[c27kwan]

lgtm! 🎉