Make OnDiskCache thread-safer #49396
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rust-highfive
added
the
S-waiting-on-review
oli-obk
reviewed
Mar 27, 2018
| @@ -604,6 +610,7 @@ impl<'a, 'tcx, 'x> SpecializedDecoder<interpret::AllocId> for CacheDecoder<'a, ' | |||
| this.with_position(shorthand, |this| interpret::AllocId::decode(this)) | |||
| } | |||
| )?; | |||
| // FIXME: Looks like overwrites aren't allowed here | |||
There was a problem hiding this comment.
We could asseet that the result is the same. That would be fine I guess.
There was a problem hiding this comment.
That won't work for the alloc cache though. This code was not written with threading in mind. I think it could work if you made the creation of the alloc_id atomic with the insertion into the cache. But then you'd be back at troubles with the end position. So if there's an entry in the alloc_cache but not the end position, you need to decode the allocation, but throw away the result in order to make the deserializer move forward.
Uuuuh. It's a little messy but should be doable. You wanna do a live hacking together to hash out the details or is the above info enoughfor you?
|
I removed the Miri parts of this PR. The on disk cache for alloc IDs doesn't seem to actually work on |
Zoxc
force-pushed
the
sync-on-disk-cache
branch
2 times, most recently
from
b94e5af
to
174521f
Compare
|
Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem. Click to expand the log.I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
Zoxc
force-pushed
the
sync-on-disk-cache
branch
3 times, most recently
from
6daf665
to
e1498b3
Compare
|
Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem. Click to expand the log.I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
|
Ping from triage, @michaelwoerister ! |
| file_index_to_file: RefCell<FxHashMap<FileMapIndex, Lrc<FileMap>>>, | ||
| synthetic_expansion_infos: RefCell<FxHashMap<AbsoluteBytePos, SyntaxContext>>, | ||
| file_index_to_file: Lock<FxHashMap<FileMapIndex, Lrc<FileMap>>>, | ||
| synthetic_expansion_infos: Lock<FxHashMap<AbsoluteBytePos, SyntaxContext>>, |
There was a problem hiding this comment.
These two fields are locked for every span value that's encoded. I wonder if we can do better here. They are just simple caches.
| @@ -866,7 +866,7 @@ pub struct GlobalCtxt<'tcx> { | |||
| maybe_unused_extern_crates: Vec<(DefId, Span)>, | |||
|
|
|||
| // Internal cache for metadata decoding. No need to track deps on this. | |||
| pub rcache: RefCell<FxHashMap<ty::CReaderCacheKey, Ty<'tcx>>>, | |||
| pub rcache: Lock<FxHashMap<ty::CReaderCacheKey, Ty<'tcx>>>, | |||
There was a problem hiding this comment.
This field is also used very often, also during metadata encoding. Might be a good candidate for a thread-local cache in the future.
|
I've added an item to #48685 about making those fields thread-locals. |
|
I made use of |
|
☔ The latest upstream changes (presumably #49390) made this pull request unmergeable. Please resolve the merge conflicts. |
Zoxc
force-pushed
the
sync-on-disk-cache
branch
from
9b7a707
to
4f32dda
Compare
|
Looks good to me. r=me with the overlap with #49558 resolved. |
|
☔ The latest upstream changes (presumably #49558) made this pull request unmergeable. Please resolve the merge conflicts. |
Zoxc
force-pushed
the
sync-on-disk-cache
branch
from
4f32dda
to
fa06023
Compare
|
Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem. Click to expand the log.I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
Zoxc
force-pushed
the
sync-on-disk-cache
branch
from
fa06023
to
807c1a0
Compare
|
@bors r=michaelwoerister |
|
📌 Commit 807c1a0 has been approved by |
bors
added
S-waiting-on-bors
kennytm
added a commit
to kennytm/rust
that referenced
this pull request
Apr 13, 2018
…oerister Make OnDiskCache thread-safer I'm not sure if `synthetic_expansion_infos` is handled correctly. `interpret_alloc_cache` and `interpret_alloc_size` seems to be wrong though, since the code may now decode two `AllocId`s in parallel. I'd like some input on how to fix that. cc @oli-obk r? @michaelwoerister
bors
added a commit
that referenced
this pull request
Apr 14, 2018
Make OnDiskCache thread-safer I'm not sure if `synthetic_expansion_infos` is handled correctly. `interpret_alloc_cache` and `interpret_alloc_size` seems to be wrong though, since the code may now decode two `AllocId`s in parallel. I'd like some input on how to fix that. cc @oli-obk r? @michaelwoerister
|
☀️ Test successful - status-appveyor, status-travis |
This was referenced Apr 14, 2018
37 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm not sure if
synthetic_expansion_infosis handled correctly.interpret_alloc_cacheandinterpret_alloc_sizeseems to be wrong though, since the code may now decode twoAllocIds in parallel. I'd like some input on how to fix that.cc @oli-obk
r? @michaelwoerister