Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rewrite enumerate children async to be less fragile #3571

Merged
merged 3 commits into from
Jan 17, 2017
Merged

rewrite enumerate children async to be less fragile #3571

merged 3 commits into from
Jan 17, 2017

Conversation

whyrusleeping
Copy link
Member

#3550 made us realize how terribly fragile the EnumerateChildrenAsync code was, so this is a cleaner rewrite of it from scratch.

License: MIT
Signed-off-by: Jeromy why@ipfs.io

@whyrusleeping whyrusleeping added the status/in-progress In progress label Jan 7, 2017
@whyrusleeping whyrusleeping requested review from a user and Kubuxu January 7, 2017 13:46
@whyrusleeping whyrusleeping added this to the ipfs 0.4.5 milestone Jan 7, 2017
@whyrusleeping whyrusleeping added the need/review Needs a review label Jan 7, 2017
@whyrusleeping whyrusleeping mentioned this pull request Jan 7, 2017
Closed
@whyrusleeping
Copy link
Member Author

closes #3505

@whyrusleeping
Copy link
Member Author

CHANGELOG:
rewrite EnumerateChildrenAsync to be less fragile.

@whyrusleeping whyrusleeping mentioned this pull request Jan 9, 2017
Closed
@Voker57
Copy link
Contributor

Voker57 commented Jan 10, 2017

I've tried pinning my large directory with this code and following error happened: http://dump.bitcheese.net/files/lokydaf/daemon.log

@Kubuxu Kubuxu added need/author-input Needs input from the original author and removed need/review Needs a review labels Jan 10, 2017
@whyrusleeping
Copy link
Member Author

@Voker57 Whoops, forgot that these keysets arent threadsafe. Did some locking and ran the merkledag tests with the race detector (would be great to have faster CI so we can run the race detector in CI). Should be good now.

@whyrusleeping whyrusleeping added need/review Needs a review and removed need/author-input Needs input from the original author labels Jan 10, 2017
Kubuxu
Kubuxu approved these changes Jan 10, 2017
View reviewed changes
Copy link
Member

@Kubuxu Kubuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM, but I am not 100% sure of my knowledge and understanding side-effects of this code.

@Voker57
Copy link
Contributor

Voker57 commented Jan 10, 2017

This forkbombed or something my PC (extreme load, unresponsive until I killed ipfs). Unbounded concurrency again?

@whyrusleeping
Copy link
Member Author

@Voker57 are you able to run ipfs refs -r <YOUR HASH> ?

@whyrusleeping
Copy link
Member Author

@Voker57 I just removed concurrency from the pin enumeration. This should help, it might be slower, but depending on the situation, it might actually be faster because of disk contention issues with async...

@Kubuxu
Copy link
Member

Kubuxu commented Jan 11, 2017

There are merge conflicts.

@whyrusleeping
rewrite enumerate children async to be less fragile
8fc4257 
License: MIT
Signed-off-by: Jeromy <why@ipfs.io>
@whyrusleeping
make pinning use serial graph enumeration
eac1fa2 
License: MIT
Signed-off-by: Jeromy <why@ipfs.io>
@whyrusleeping
Copy link
Member Author

merge conflicts resolved. Awaiting feedback from @Voker57. My 100GB dataset was able to be pinned just fine, but its on a rather beefy machine.

@whyrusleeping
update test to print error from EnumerateChildren
397a35a 
License: MIT
Signed-off-by: Jeromy <why@ipfs.io>
@Voker57
Copy link
Contributor

Voker57 commented Jan 13, 2017

Serial fetching works, but it's very slow: 1.15 hrs vs ~40 minutes using my patch. it's utilizing neither CPU nor IO to full capacity.

@Voker57
Copy link
Contributor

Voker57 commented Jan 14, 2017 •
edited
Loading

I've submitted PR on this PR (is this the best way to do this?) which brings RAM usage to adequate levels. Also, I've rechecked and parallelizing doesn't win anything significant here on traditional HDD, except if disk storage is split into several IO channels somehow.

@whyrusleeping
Copy link
Member Author

@Voker57 I'm going to merge this one for now so its shipped in 0.4.5 (despite the decreased performance). We will get your patches into the next release.

@whyrusleeping whyrusleeping merged commit 75cce80 into master Jan 17, 2017
@whyrusleeping whyrusleeping removed the status/in-progress In progress label Jan 17, 2017
@ghost ghost deleted the feat/better-enum-async branch January 22, 2017 04:03
@ghost ghost mentioned this pull request Jan 27, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Kubuxu Kubuxu Kubuxu approved these changes

Assignees
No one assigned
Labels
need/review Needs a review
Projects
None yet
Milestone
ipfs 0.4.5
Development

Successfully merging this pull request may close these issues.
3 participants
@whyrusleeping @Voker57 @Kubuxu