-
-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rust http client idea: stream rather than chunk #94
Comments
If anyone has any thoughts on this, I'm having a go at it. |
While I'm still (forever?) a rust-noob it sounds like a good idea to me. :) @pka? |
My gut feeling says that only the first point, reduce the latency-to-first-feature, will be a measurable win. And then we need also a streaming FGB client, which does e.g. render the first feature immediately, to make the performance win also measurable for the end user. |
I would expect measurable wins anywhere the cost of a waiting on a request roundtrip is measurably faster than transmitting (currently 1MB). Conversely, I'd expect no real impact on small files or lower bandwidth connections, especially if they have relatively low latency. Some motivating (but naive) math for transferring a 10MB file: ![]() check my math: https://docs.google.com/spreadsheets/d/1eu35b1mFwKTN-LzpmJFjwC4n4uP9AwfuPxuSh5OwO2k/edit#gid=0 This is all just theoretical numbers, I'm sure it won't be that good, but I'm reasonably optimistic that there will be more wins than only "time to first feature", especially for larger files on fast connections, without requiring users to hand-tune their chunk size. It also has the upside that you don't have to hold chunk_size in memory all at once. I'm not yet sure how much this will benefit bbox selection. The request response model there is more complicated. I'd expect a benefit, but the implementation might get complicated.
Oh that's an interesting idea. Anecdotally, I've seen a lot of benefit from compressing fgb. I could see how something like this would speed things up. Though, it seems like the chunk vs. stream question could equally apply to perusing a remote SOzip file. |
The current streaming strategy is based on chunks:
It seems like we could increase throughput, and in particular reduce the latency-to-first-feature, by utilizing a continuous byte stream.
Our http lib, reqwest supports
response.byte_stream
if you enable thestream
feature which might be useful.I'm personally not familiar with creating Futures and consuming Stream apis in rust, but this seems conceptually straight forward for the feature-fetching portion of the http reader.
There might be some way to also utilize it for the header+index downloads as well, but that seems a more complicated cost to benefit ratio.
The text was updated successfully, but these errors were encountered: