-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
batch uploads error using walk #99
Comments
Thanks for reporting. The latest version on GitHub (v0.2.3) will re-attempt the request a few times if the API throws a 500 error. Can you try updating and see if the problem persists? I haven't tried uploading that many files before so I'll do a little testing on my end as well. |
Updated to v0.2.3 and tried uploading again two ways. In first attempt, tried to simply use the update in batch code and got this error:
So I then deleted the folder via the web interface and tried the first code from above which makes the directory and then uploads files to it. Got the 500 error again and only 400 files uploaded.
Let me know if I can try other approaches and help. Thanks! |
Thanks. I ran a couple of tests that attempted to upload 1500 files and was able to reproduce the same error. Unfortunately, sometimes it worked and sometimes it failed. I'm going to leave this open for now. These HTTP codes > 500 correspond to "unexpected errors" on the server, so we may need to loop in one of the OSF devs to ultimately solve it. In the meantime, this highlighted some inefficiencies in |
Are many of these files relatively small? Like in the "seconds or less to upload" size? |
Hi @brianjgeiger, thanks for checking into this. I used hundreds of small text files in my testing. Are you thinking it's rate limiting issue? |
Hi, @aaronwolen, no, I think it's because we have an inefficiency or two on capturing provenance data for file uploads, and it's causing the thread to eventually time out. It should be fixed in an upcoming version, but I don't have a date on that yet. But slowing down the requests will definitely keep you from seeing the error. |
Thanks for the info.
Is there a relevant PR or Issue I can monitor to determine when it's fixed? In the meantime, do you have recommendations for parameters I should use to moderate requests (eg, delay |
When using the osf_upload function in combination with purrr::walk I received an inconsistent error. I could upload ~50 files with no problems.
Later when trying the same basic code with a much larger directory (~1000) files only 700 files uploaded before I received this message:
Code that failed with error above:
I adjusted the code figuring it was a timeout issue and tried to complete the upload with:
and got this error:
Note:
overwrite = FALSE
failed to work, which is why overwrite is set to TRUEAs I stated originally, the same basic code worked for 50 files, but larger uploads failed to fully complete.
Have enjoyed using the package and this won't stop me from using it, just 1000 files is a standard size project for me so batch uploads without having to use the web interface is really useful.
The text was updated successfully, but these errors were encountered: