Restore: fix performance regression on .NET Core 2.1+

[dtivel]

Resolve NuGet/Home#7314.

The core problem with the original code is that it forces a buffer to be flushed synchronously rather than asynchronously (and, actually worse than that, as a sync-over-async operation, meaning it’s performing an asynchronous operation and then blocking waiting for it to complete). That would occur as part of the Seek, or if the validation wasn’t there, as part of Dispose’ing the FileStream, both synchronous operations.

The right fix would be to add an “await FileStream.FlushAsync();” before doing the seeking/validation/closing. That flush would ensure that the buffer is empty when Seek/Dispose/etc. are closed, so there’s nothing more to flush. However, there’s a bug in FlushAsync that’s causing it to also do the key part of the flush synchronously as well.

So, the workaround is to avoid buffering, such that nothing needs to be flushed as part of Seek/Dispose. That’s what bufferSize=1 does.

However, the concern expressed was that disabling that buffering would avoid this problem for the writing but would slow down the reading/validation. So, the fix splits it into two streams: one used for writing with buffering disabled, and then one used for reading that has buffering enabled still.

CC @rrelyea

[jainaashish]

Will it add any performance impact for desktop frameworks since it will always open separate streams to read and write?

Or shouldn't we just restrict these separate streams for netcore?

[jainaashish]

Discussed offline and got this resolved. Updated the PR description accordingly