Removes support for Hadoop 2 #14763
Merged
Removes support for Hadoop 2 #14763
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
.github/workflows/cron-job-its.yml
Outdated
| @@ -112,7 +112,6 @@ jobs: | |||
| strategy: | |||
| fail-fast: false | |||
| matrix: | |||
There was a problem hiding this comment.
this line can be removed as well?
There was a problem hiding this comment.
did we move these files or just simply deleted them?
There was a problem hiding this comment.
deleted original tutorial/hadoop/docker/* files and moved tutorial/hadoop3/docker/* files to tutorial/hadoop/docker/ folder
ektravel
reviewed
Aug 7, 2023
docs/configuration/index.md
Outdated
| @@ -1534,7 +1534,7 @@ Additional peon configs include: | |||
| |`druid.indexer.task.baseDir`|Base temporary working directory.|`System.getProperty("java.io.tmpdir")`| | |||
| |`druid.indexer.task.baseTaskDir`|Base temporary working directory for tasks.|`${druid.indexer.task.baseDir}/persistent/task`| | |||
| |`druid.indexer.task.batchProcessingMode`| Batch ingestion tasks have three operating modes to control construction and tracking for intermediary segments: `OPEN_SEGMENTS`, `CLOSED_SEGMENTS`, and `CLOSED_SEGMENT_SINKS`. `OPEN_SEGMENTS` uses the streaming ingestion code path and performs a `mmap` on intermediary segments to build a timeline to make these segments available to realtime queries. Batch ingestion doesn't require intermediary segments, so the default mode, `CLOSED_SEGMENTS`, eliminates `mmap` of intermediary segments. `CLOSED_SEGMENTS` mode still tracks the entire set of segments in heap. The `CLOSED_SEGMENTS_SINKS` mode is the most aggressive configuration and should have the smallest memory footprint. It eliminates in-memory tracking and `mmap` of intermediary segments produced during segment creation. `CLOSED_SEGMENTS_SINKS` mode isn't as well tested as other modes so is currently considered experimental. You can use `OPEN_SEGMENTS` mode if problems occur with the 2 newer modes. |`CLOSED_SEGMENTS`| | |||
| |`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|org.apache.hadoop:hadoop-client:2.8.5| | |||
| |`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|org.apache.hadoop:hadoop-client-api:3.3.6,org.apache.hadoop:hadoop-client-runtime:3.3.6| | |||
There was a problem hiding this comment.
Suggested change
| |`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|org.apache.hadoop:hadoop-client-api:3.3.6,org.apache.hadoop:hadoop-client-runtime:3.3.6| | |
| |`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|`org.apache.hadoop:hadoop-client-api:3.3.6`, `org.apache.hadoop:hadoop-client-runtime:3.3.6`| |
ektravel
reviewed
Aug 7, 2023
docs/configuration/index.md
Outdated
| @@ -1605,7 +1605,7 @@ then the value from the configuration below is used: | |||
| |`druid.worker.numConcurrentMerges`|Maximum number of segment persist or merge operations that can run concurrently across all tasks.|`druid.worker.capacity` / 2, rounded down| | |||
| |`druid.indexer.task.baseDir`|Base temporary working directory.|`System.getProperty("java.io.tmpdir")`| | |||
| |`druid.indexer.task.baseTaskDir`|Base temporary working directory for tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`| | |||
| |`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|org.apache.hadoop:hadoop-client:2.8.5| | |||
| |`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|org.apache.hadoop:hadoop-client-api:3.3.6,org.apache.hadoop:hadoop-client-runtime:3.3.6| | |||
There was a problem hiding this comment.
Suggested change
| |`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|org.apache.hadoop:hadoop-client-api:3.3.6,org.apache.hadoop:hadoop-client-runtime:3.3.6| | |
| |`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|`org.apache.hadoop:hadoop-client-api:3.3.6`, `org.apache.hadoop:hadoop-client-runtime:3.3.6`| |
ektravel
reviewed
Aug 7, 2023
docs/operations/other-hadoop.md
Outdated
| @@ -89,7 +89,7 @@ classloader. | |||
| 2. Batch ingestion uses jars from `hadoop-dependencies/` to submit Map/Reduce jobs (location customizable via the | |||
| `druid.extensions.hadoopDependenciesDir` runtime property; see [Configuration](../configuration/index.md#extensions)). | |||
|
|
|||
| `hadoop-client:2.8.5` is the default version of the Hadoop client bundled with Druid for both purposes. This works with | |||
| `hadoop-client-api:3.3.6, hadoop-client-runtime:3.3.6` is the default version of the Hadoop client bundled with Druid for both purposes. This works with | |||
There was a problem hiding this comment.
Suggested change
| `hadoop-client-api:3.3.6, hadoop-client-runtime:3.3.6` is the default version of the Hadoop client bundled with Druid for both purposes. This works with | |
| The default version of the Hadoop client bundled with Druid is 3.3.6. This works with |
ektravel
reviewed
Aug 7, 2023
|
Thanks @abhishekagarwal87, @ektravel for the review. I have addressed your comments in the latest commit. |
abhishekagarwal87
approved these changes
Aug 8, 2023
clintropolis
approved these changes
Aug 9, 2023
xvrl
added a commit
to xvrl/druid
that referenced
this pull request
Dec 5, 2023
Several dependabot ignore directives are no longer relevant. Unpin them to ensure we get again get timely updates via dependabot. * support for Hadoop 2 was dropped as part of apache#14763 * Guava was upgraded to 31 as part of apache#14767 * Calcite was upgraded to 1.35 as part of apache#14510
xvrl
added a commit
to xvrl/druid
that referenced
this pull request
Dec 5, 2023
Several dependabot ignore directives are no longer relevant. Unpin them to ensure we get again get timely updates via dependabot. * support for Hadoop 2 was dropped as part of apache#14763 * Guava was upgraded to 31 as part of apache#14767 * Calcite was upgraded to 1.35 as part of apache#14510
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Release note
Hadoop 2 support is formally deprecated in Druid 27.0.0. Hadoop 2 support is now terminated.