[Enhancement] Support cache delta lake delta log metadata

[Youngwb]

Why I'm doing:

Query delta lake table will need to access delta log, it would take long time to read a lot of json/parquet files, thees metadata files are not going to change, so they are suitable for caching to avoid repeated reading.

What I'm doing:

1. add jsonCache for json files in delta log
2. add checkpointCache for parquet files in delta log
3. add catalog properties for delta log cache

Property                                                   default value
enable_deltalake_table_cache                                  true
enable_deltalake_json_meta_cache                              true
deltalake_json_meta_cache_ttl_sec                             48 * 60 * 60
deltalake_json_meta_cache_max_num                             1000
enable_deltalake_checkpoint_meta_cache                        false
deltalake_checkpoint_meta_cache_ttl_sec                       48 * 60 * 60
deltalake_checkpoint_meta_cache_max_num                       100

4. add DeltaLakeEngine to override DefaultEngine, this allows us implement a customized reading process for Parquet and JSON.

Fixes #issue

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.3
  • 3.2
  • 3.1
  • 3.0
  • 2.5

[stephen-shelby]

why we don't use predicate

[Youngwb]

we cannot cache the entire parquet data when use predicate

[stephen-shelby]

is it the same as iceberg that caching all manifest file content?

[sonarcloud]

Quality Gate failed

Failed conditions
E Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[github-actions]

[FE Incremental Coverage Report]

✅ pass : 161 / 173 (93.06%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/connector/delta/DeltaLakeMetadataFactory.java	3	4	75.00%	[79]
🔵	com/starrocks/connector/delta/DeltaLakeParquetHandler.java	36	42	85.71%	[81, 90, 91, 92, 99, 122]
🔵	com/starrocks/connector/delta/DeltaLakeEngine.java	10	11	90.91%	[52]
🔵	com/starrocks/connector/delta/DeltaLakeMetastore.java	16	17	94.12%	[74]
🔵	com/starrocks/connector/delta/DeltaLakeJsonHandler.java	57	60	95.00%	[136, 174, 175]
🔵	com/starrocks/connector/delta/DeltaLakeInternalMgr.java	6	6	100.00%	[]
🔵	com/starrocks/connector/delta/CachingDeltaLakeMetastore.java	4	4	100.00%	[]
🔵	com/starrocks/connector/delta/HMSBackedDeltaMetastore.java	1	1	100.00%	[]
🔵	com/starrocks/connector/delta/DeltaLakeConnector.java	1	1	100.00%	[]
🔵	com/starrocks/connector/delta/DeltaLakeCatalogProperties.java	27	27	100.00%	[]

[github-actions]

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[Youngwb]

@mergify backport branch-3.3

[mergify]

backport branch-3.3

✅ Backports have been created

• #49479 [Enhancement] Support cache delta lake delta log metadata (backport #49069) has been created for branch branch-3.3