Skip to content

Commit

Permalink
[SPARK-49752][YARN] Remove workaround for YARN-3350
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

Remove the logic of forcibly setting the log level to WARN for `org.apache.hadoop.yarn.util.RackResolver`.

### Why are the changes needed?

The removed code was introduced in SPARK-5393 as a workaround for YARN-3350, which is already fixed on the YARN 2.8.0/3.0.0.

### Does this PR introduce _any_ user-facing change?

Yes, previously, the log level of RackResolver is hardcoded as WARN even if the user explicitly sets it to DEBUG.

### How was this patch tested?

Review.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#48201 from pan3793/SPARK-49752.

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
pan3793 authored and dongjoon-hyun committed Sep 22, 2024
1 parent 067f8f1 commit 719b57a
Showing 1 changed file with 0 additions and 9 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,6 @@ import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.CommonConfigurationKeysPublic
import org.apache.hadoop.net._
import org.apache.hadoop.util.ReflectionUtils
import org.apache.hadoop.yarn.util.RackResolver
import org.apache.logging.log4j.{Level, LogManager}
import org.apache.logging.log4j.core.Logger

import org.apache.spark.internal.{Logging, MDC}
import org.apache.spark.internal.LogKeys.NODE_LOCATION
Expand All @@ -39,12 +36,6 @@ import org.apache.spark.internal.LogKeys.NODE_LOCATION
*/
private[spark] class SparkRackResolver(conf: Configuration) extends Logging {

// RackResolver logs an INFO message whenever it resolves a rack, which is way too often.
val logger = LogManager.getLogger(classOf[RackResolver])
if (logger.getLevel != Level.WARN) {
logger.asInstanceOf[Logger].setLevel(Level.WARN)
}

private val dnsToSwitchMapping: DNSToSwitchMapping = {
val dnsToSwitchMappingClass =
conf.getClass(CommonConfigurationKeysPublic.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY,
Expand Down

0 comments on commit 719b57a

Please sign in to comment.