incr.comp.: Make sure dependencies are recorded when feeding queries during eval-always queries. #109935

michaelwoerister · 2023-04-04T13:02:33Z

This PR makes sure we don't drop dependency edges when feeding queries during an eval-always query.

Background: During eval-always queries, no dependencies are recorded because the system knows to unconditionally re-evaluate them regardless of any actual dependencies. This works fine for these queries themselves but leads to a problem when feeding other queries: When queries are fed, we set up their dependency edges by copying the current set of dependencies of the feeding query. But because this set is empty for eval-always queries, we record no edges at all -- which has the effect that the fed query instances always look "green" to the system, although they should always be "red".

The fix is to explicitly add a dependency on the artificial "always red" dep-node when feeding during eval-always queries.

Fixes #108481
Maybe also fixes issue #88488.

cc @jyn514

r? @cjgillot or @oli-obk

…during eval-always queries.

cjgillot · 2023-04-04T17:32:20Z

compiler/rustc_query_system/src/dep_graph/graph.rs

+                TaskDepsRef::EvalAlways => {
+                    edges.push(DepNodeIndex::FOREVER_RED_NODE);
+                }
+                TaskDepsRef::Ignore => {}


Should we insert this red dependency in both cases? Or should we ICE when feeding in ignore mode?

Yeah, I've been wondering about that too. I think adding a dependency would be surprising. We are inside a with_ignore() after all. But ICEing might be useful?

One factor that might influence a decision here is that we seem to wrap everything inside of TaskDepsRef::Ignore by default:

rust/compiler/rustc_middle/src/ty/context/tls.rs

Lines 40 to 49 in a412564

pub fn new(gcx: &'tcx GlobalCtxt<'tcx>) -> Self {

let tcx = TyCtxt { gcx };

ImplicitCtxt {

tcx,

query: None,

diagnostics: None,

query_depth: 0,

task_deps: TaskDepsRef::Ignore,

}

}

If we changed that to TaskDepsRef::Forbid there would be less chance of things being silently ignored and as a result less harm in treating TaskDepsRef::Ignore as "yes, really turn off dependency tracking". We use DepGraph::with_ignore() in a few places where we really don't want any dep-tracking to happen, e.g. when re-computing the value of a green node.

But I haven't tried what the fallout of switching to TaskDepsRef::Forbid would be. It might need some cleanup.

Overall, I think DepGraph::with_ignore() should really reliable turn off dep-tracking -- but we should use it in as few places as possible. This PR would be an example of replacing it with something more targeted.

cjgillot · 2023-04-04T17:32:45Z

compiler/rustc_passes/src/hir_id_validator.rs

@@ -8,34 +8,34 @@ use rustc_middle::hir::nested_filter;
 use rustc_middle::ty::TyCtxt;

 pub fn check_crate(tcx: TyCtxt<'_>) {
-    tcx.dep_graph.assert_ignored();


Why this change?

This is actually running inside of an eval-always query, not in with_ignore(). The two kinds both used TaskDepsRef::Ignore before this PR. Now that there is a TaskDepsRef::EvalAlways, the assertion fails.

Looking at the code again, I think we might actually want to just run it normally. I think the original with_ignore() scope was just a performance optimization when this check was run for release builds too.

I removed the DepGraph::with_ignore() scope in 5f52a96. It shouldn't make a difference for performance since the function doesn't do much in release builds.

oli-obk · 2023-04-05T08:16:11Z

@bors try @rust-timer queue

bors · 2023-04-05T08:16:21Z

⌛ Trying commit 6117c06 with merge 2e6dc6945d8cfad0f6fd06ef2fc221f8930a8942...

bors · 2023-04-05T09:59:40Z

☀️ Try build successful - checks-actions
Build commit: 2e6dc6945d8cfad0f6fd06ef2fc221f8930a8942 (2e6dc6945d8cfad0f6fd06ef2fc221f8930a8942)

rust-timer · 2023-04-05T14:18:06Z

Finished benchmarking commit (2e6dc6945d8cfad0f6fd06ef2fc221f8930a8942): comparison URL.

Overall result: ❌✅ regressions and improvements - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.3%	[0.3%, 0.3%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.4%	[-0.4%, -0.4%]	1
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.8%	[3.8%, 3.8%]	1
Regressions ❌ (secondary)	2.3%	[1.5%, 3.2%]	5
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	3.8%	[3.8%, 3.8%]	1

Cycles

This benchmark run did not return any relevant results for this metric.

michaelwoerister · 2023-04-11T13:34:41Z

According to the release schedule the next beta will be branched off on Friday. Maybe we can get this merged before then? Are there open questions you'd like me to address, @cjgillot?

cjgillot · 2023-04-11T15:17:47Z

@bors r+

bors · 2023-04-11T15:17:49Z

📌 Commit 5f52a96 has been approved by cjgillot

It is now in the queue for this repository.

bors · 2023-04-12T11:16:38Z

⌛ Testing commit 5f52a96 with merge 661b33f...

bors · 2023-04-12T13:37:47Z

☀️ Test successful - checks-actions
Approved by: cjgillot
Pushing 661b33f to master...

rust-timer · 2023-04-12T14:55:21Z

Finished benchmarking commit (661b33f): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.2%	[-3.2%, -0.5%]	4
Improvements ✅ (secondary)	-2.1%	[-2.7%, -1.5%]	2
All ❌✅ (primary)	-1.2%	[-3.2%, -0.5%]	4

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.5%	[-0.5%, -0.5%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.5%	[-0.5%, -0.5%]	1

incr.comp.: Make sure dependencies are recorded when feeding queries …

6117c06

…during eval-always queries.

rustbot assigned cjgillot Apr 4, 2023

cjgillot reviewed Apr 4, 2023

View reviewed changes

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 5, 2023

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 5, 2023

incr.comp.: Don't ignore dep-tracking during HirId validation.

5f52a96

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 11, 2023

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 12, 2023

bors merged commit 661b33f into rust-lang:master Apr 12, 2023

rustbot added this to the 1.70.0 milestone Apr 12, 2023

bors mentioned this pull request Apr 12, 2023

test the parallel compiler #109776

Closed

jyn514 mentioned this pull request Jul 2, 2023

ICE: "no entry found for key" while building stage0 tools #88488

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

incr.comp.: Make sure dependencies are recorded when feeding queries during eval-always queries. #109935

incr.comp.: Make sure dependencies are recorded when feeding queries during eval-always queries. #109935

michaelwoerister commented Apr 4, 2023

cjgillot Apr 4, 2023

michaelwoerister Apr 5, 2023

cjgillot Apr 4, 2023

michaelwoerister Apr 5, 2023

michaelwoerister Apr 5, 2023

oli-obk commented Apr 5, 2023

This comment has been minimized.

bors commented Apr 5, 2023

bors commented Apr 5, 2023

This comment has been minimized.

rust-timer commented Apr 5, 2023

michaelwoerister commented Apr 11, 2023

cjgillot commented Apr 11, 2023

bors commented Apr 11, 2023

bors commented Apr 12, 2023

bors commented Apr 12, 2023

rust-timer commented Apr 12, 2023

	pub fn new(gcx: &'tcx GlobalCtxt<'tcx>) -> Self {
	let tcx = TyCtxt { gcx };
	ImplicitCtxt {
	tcx,
	query: None,
	diagnostics: None,
	query_depth: 0,
	task_deps: TaskDepsRef::Ignore,
	}
	}

incr.comp.: Make sure dependencies are recorded when feeding queries during eval-always queries. #109935

incr.comp.: Make sure dependencies are recorded when feeding queries during eval-always queries. #109935

Conversation

michaelwoerister commented Apr 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oli-obk commented Apr 5, 2023

This comment has been minimized.

bors commented Apr 5, 2023

bors commented Apr 5, 2023

This comment has been minimized.

rust-timer commented Apr 5, 2023

Overall result: ❌✅ regressions and improvements - no action needed

michaelwoerister commented Apr 11, 2023

cjgillot commented Apr 11, 2023

bors commented Apr 11, 2023

bors commented Apr 12, 2023

bors commented Apr 12, 2023

rust-timer commented Apr 12, 2023

Overall result: no relevant changes - no action needed