[UNIFORM] Remove the setSchema patch

[Fokko]

This patch aims to align the field-ID assignment between uniform and Iceberg. This way we can use the public Iceberg APIs to evolve the schema, instead of enforcing the schema through a private API.

The difference is that Iceberg does not have true pre-order traversal, but when it encounters a struct, it reserves an ID for each field in the struct before continuing the traversal.

Iceberg:

1: struct
  2: struct
    4: long
    5: timestamp
  3: int

Delta:

1: struct
  2: struct
    3: long
    4: timestamp
  5: int

This PR changes Delta's assignment behavior, so we don't need the patch anymore.

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

[Fokko]

cc @amogh-jahagirdar

[tomvanbussel]

Left some comments on the code.

In general I am a bit worried about the brittleness of this solution. It relies on every code path assigning field ids in the exact same way as Iceberg. It's highly likely that someone adding a new code path in the future might not be aware of this requirement. At the very least we should add some verification logic to the UniForm exporter to confirm that the field ids are as expected. Would it instead be possible to add some interfaces to Iceberg that can help us with evolving the schema?

One open question for me is how this will work with existing tables that have already assigned field ids in an incompatible way.

[tomvanbussel]

Style Nit:

Suggested change

	def transformColumns(schema: StructType,
	traverseStructsAtOnce: Boolean = false)(
	def transformColumns(
	schema: StructType,
	traverseStructsAtOnce: Boolean = false)(

[tomvanbussel]

"At once" is a little bit ambiguous. Could we call this something like traverseAllStructFieldsBeforeRecursing?

[tomvanbussel]

Style Nit: Either add both branches inline, or create a block for both branches

Suggested change

	} else traversedField
	} else {
	traversedField
	}

[tomvanbussel]

This change could benefit from a few tests in DeltaColumnMappingSuite that create a table and add some columns and then check the assigned field ids in the log.