HttpClient uploading files with NTLM: An existing connection was forcibly closed by the remote host -- .NET Core 2.1/3.0 difference against .NET Framework

[Rookian]

We try to upload a file to a windows hosted web server.
Using the Fullframework everything works fine.
We use the follwing code:

using (var httpClient = new HttpClient(new HttpClientHandler {UseDefaultCredentials = true}) {BaseAddress = new Uri($"{Host}/rest-ws/service/")})
{
    var requestUri = $"dms/{itemId}/content?filename={fileName}.{fileExtension}";

    var requestContent = new MultipartFormDataContent();

    var byteArrayContent = new ByteArrayContent(fileData);
    byteArrayContent.Headers.ContentType = MediaTypeHeaderValue.Parse("text/plain");
    requestContent.Add(byteArrayContent, fileName, "{fileName}.{fileExtension}");

    var responseMessage = await httpClient.PostAsync(requestUri, requestContent);
}

When we use the exact code in .NET Core we receive an error.

System.Net.Http.HttpRequestException
HResult=0x80131620
Message=An error occurred while sending the request.
Source=System.Net.Http
StackTrace:
at System.Net.Http.HttpConnection.d__53.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.AuthenticationHelper.<SendWithNtAuthAsync>d__47.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.HttpConnectionPool.d__48.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.HttpConnectionPool.<SendWithRetryAsync>d__47.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.AuthenticationHelper.d__17.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.DiagnosticsHandler.d__2.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__70.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Core.Console.Program.d__2.MoveNext() in C:\git\econ-main\src\PoC\ReproHttpClient\ReproHttpClient\Core.Console\Program.cs:line 49
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Core.Console.Program.

d__1.MoveNext() in C:\git\econ-main\src\PoC\ReproHttpClient\ReproHttpClient\Core.Console\Program.cs:line 18

Inner Exception 1:
IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..

Inner Exception 2:
SocketException: An existing connection was forcibly closed by the remote host.

HTTP/TCP Stream for full framework:

POST /rest-ws/service/dms/42863508DFF541C49B0C298C7C4A3EA3/content?filename=test.txt HTTP/1.1
Content-Type: multipart/form-data; boundary="af9cb1b0-4999-47f6-95ac-b91ec572edf3"
Authorization: NTLM TlRMTVNTUAABAAAAB7IIogMAAwA2AAAADgAOACgAAAAKAO5CAAAAD0RFRTRFNzQ5M0I2NUNFUENX
Host: enaio-web-dev.pcw.local
Content-Length: 0

HTTP/1.1 401 Unauthorized
WWW-Authenticate: NTLM TlRMTVNTUAACAAAABgAGADgAAAAFgomiuIB1rGBkz1kAAAAAAAAAAIYAhgA+AAAABgOAJQAAAA9QAEMAVwACAAYAUABDAFcAAQASAEQARQBMAEUASQA0ADQANgAxAAQAEgBwAGMAdwAuAGwAbwBjAGEAbAADACYARABFAEwARQBJADQANAA2ADEALgBwAGMAdwAuAGwAbwBjAGEAbAAFABIAcABjAHcALgBsAG8AYwBhAGwABwAIAEb+j2i6ftUBAAAAAA==
Connection: keep-alive
Transfer-Encoding: chunked

POST /rest-ws/service/dms/42863508DFF541C49B0C298C7C4A3EA3/content?filename=test.txt HTTP/1.1
Content-Type: multipart/form-data; boundary="af9cb1b0-4999-47f6-95ac-b91ec572edf3"
Authorization: NTLM TlRMTVNTUAADAAAAGAAYAJYAAABGAUYBrgAAAAYABgBYAAAAHAAcAF4AAAAcABwAegAAAAAAAAD0AQAABYKIogoA7kIAAAAPEbCaOQL9OyvKrDhOAwzVmVAAQwBXAEEAbABlAHgAYQBuAGQAZQByAC4AVABhAG4AawBEAEUARQA0AEUANwA0ADkAMwBCADYANQBDAEUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZCUzvh3aE3JHGgoP/w5ihQEBAAAAAAAARv6PaLp+1QH1umYM8m/M/wAAAAACAAYAUABDAFcAAQASAEQARQBMAEUASQA0ADQANgAxAAQAEgBwAGMAdwAuAGwAbwBjAGEAbAADACYARABFAEwARQBJADQANAA2ADEALgBwAGMAdwAuAGwAbwBjAGEAbAAFABIAcABjAHcALgBsAG8AYwBhAGwABwAIAEb+j2i6ftUBBgAEAAIAAAAIADAAMAAAAAAAAAAAAAAAADAAAA3oSg1RakvSibPBI+7YwrVkmqjdyu0lOsOQM14VA0j1CgAQAAAAAAAAAAAAAAAAAAAAAAAJADgASABUAFQAUAAvAGUAbgBhAGkAbwAtAHcAZQBiAC0AZABlAHYALgBwAGMAdwAuAGwAbwBjAGEAbAAAAAAAAAAAAAAAAAA=
Host: enaio-web-dev.pcw.local
Content-Length: 116336
Expect: 100-continue

--af9cb1b0-4999-47f6-95ac-b91ec572edf3
Content-Type: text/plain
Content-Disposition: form-data; name=test; filename="{fileName}.{fileExtension}"; filename*=utf-8''%7BfileName%7D.%7BfileExtension%7D

...UPLOADED FILE CONTENT
--af9cb1b0-4999-47f6-95ac-b91ec572edf3--

HTTP/1.1 201 Created
Set-Cookie: GWSESSIONID=node015onigrgvwlxy1sylpb0trlbwk182.node0;Path=/
Expires: Thu, 01 Jan 1970 00:00:00 GMT
X-Application-Context: gateway:prodsso,cloud,red:80
X-Powered-By: Undertow/1
Set-Cookie: JSESSIONID_10.49.3.203_8080=jVf77qn98PARc89FAEeX4ciuRCqd3xVu1tN_xD3j.delei4461; path=/rest-ws
Server: WildFly/9
Location: http://***/rest-ws/service/dms/42863508DFF541C49B0C298C7C4A3EA3/content?index=0&type=contractdocument
Date: Wed, 09 Oct 2019 15:58:36 GMT
Connection: keep-alive
X-Os-Id: 42863508DFF541C49B0C298C7C4A3EA3
Content-Type: application/json;charset=utf-8
Transfer-Encoding: chunked

{"id":"42863508DFF541C49B0C298C7C4A3EA3","uri":"http://***/rest-ws/service/dms/42863508DFF541C49B0C298C7C4A3EA3/content?index=0&type=contractdocument"}

HTTP/TCP Stream for .NET Core 3.0:

POST /rest-ws/service/dms/42863508DFF541C49B0C298C7C4A3EA3/content?filename=test.txt HTTP/1.1
Host: enaio-web-dev.pcw.local
Content-Type: multipart/form-data; boundary="0145d6df-9419-4027-9095-629778dbbd72"
Content-Length: 116336

--0145d6df-9419-4027-9095-629778dbbd72
Content-Type: text/plain
Content-Disposition: form-data; name=test; filename="{fileName}.{fileExtension}"; filename*=utf-8''%7BfileName%7D.%7BfileExtension%7D

...FILE CONTENT

HTTP/1.1 401 Unauthorized
WWW-Authenticate: NTLM
Connection: keep-alive
Content-Language: en-US
Content-Type: text/html;charset=utf-8
Transfer-Encoding: chunked

...

As you can see the full framework is sending 2 additional headers:

• Authorization: NTLM

• Expect: 100-continue

We also noticed that when we send a very small file or an empty byte array the authentication is working for .NET Core 3.0

Our workaround right now is to first send a small Get request to authenticate against the server and receive an authentication cookie and then upload the file. So all following requests just use the authentication cookie and no NTLM.

[davidsh]

Sounds very similar to another 'Expect: 100-continue' and Negotiate/NTLM bug we had. But I thought we fixed it in .NET Core 3.0.

[davidsh]

@Rookian .NET Core doesn't send 'Expect: 100-continue' by default (.NET Framework does).

Can you try your scenario with manually adding 'Expect: 100-continue' to your HTTP request? Would like to see what difference that makes. Thanks.

[Rookian]

@davidsh Yes, .NET Core doesn't send Expect:100-continue.

I now use: httpClient.DefaultRequestHeaders.ExpectContinue = true;

HTTP/TCP Stream:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: NTLM
Connection: keep-alive
Content-Language: en-US
Content-Type: text/html;charset=utf-8
Connection: close

POST /rest-ws/service/dms/42863508DFF541C49B0C298C7C4A3EA3/content?filename=test.txt HTTP/1.1
Host: ***
Expect: 100-continue
Content-Type: multipart/form-data; boundary="31115bdf-60a2-44cd-ab89-039fb327f216"
Content-Length: 116290

--31115bdf-60a2-44cd-ab89-039fb327f216
Content-Type: text/plain
Content-Disposition: form-data; name=test; filename=test.txt; filename*=utf-8''test.txt

...UPLOAD CONTENT
--31115bdf-60a2-44cd-ab89-039fb327f216--

Error message:

System.Net.Http.HttpRequestException
HResult=0x80131500
Message=Authentication failed because the connection could not be reused.
Source=System.Net.Http
StackTrace:
at System.Net.Http.HttpConnection.d__101.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.AuthenticationHelper.d__47.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.HttpConnectionPool.<SendWithNtConnectionAuthAsync>d__48.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.HttpConnectionPool.d__47.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.AuthenticationHelper.<SendWithAuthAsync>d__17.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.RedirectHandler.d__4.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.DiagnosticsHandler.<SendAsync>d__2.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.HttpClient.d__70.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Core.Console.Program.<UploadDocument>d__2.MoveNext() in C:\git\econ-main\src\PoC\ReproHttpClient\ReproHttpClient\Core.Console\Program.cs:line 50 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Core.Console.Program.

d__1.MoveNext() in C:\git\econ-main\src\PoC\ReproHttpClient\ReproHttpClient\Core.Console\Program.cs:line 18
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Core.Console.Program.()

If you need a pcap file or so let me know.

[davidsh]

I now use: httpClient.DefaultRequestHeaders.ExpectContinue = true;
Message=Authentication failed because the connection could not be reused.

Is this using .NET Core 2.1 or .NET Core 3.0. We've seen error like "" in 2.1. So, I'd be interested in seeing .NET Core 3.0 results using ExpectContinue=true.

If you need a pcap file or so let me know.

Thanks. If the pcap file doesn't have any confidential information, then, yes, please zip it up and attach it. Otherwise, we'll need to figure out a different way for you to give us this information.

[Rookian]

This is using .NET Core 3.0.
netcore3expect100true.zip

@davidsh If you need anything else let me know.

[davidsh]

@mconnew Do you think this issue is related at all to #26662? We fixed most of that in .NET Core 3.0.

Perhaps this current issue is related to your comments about "optimizations" we haven't done yet compared with .NET Framework?

[davidsh]

Our workaround right now is to first send a small Get request to authenticate against the server and receive an authentication cookie and then upload the file. So all following requests just use the authentication cookie and no NTLM.

That workaround works because it "warms up" the connection for Windows auth. Instead of a GET request, you can also send a HEAD request. It will be more performant since a server will send back a response to a HEAD requests with no response body payload.

[mconnew]

@davidsh , I believe this is related to the optimizations I mentioned. When using NTLM, .NET Framework sends a 0-byte content length as it knows there will be a challenge due to the passed credentials. The server will tolerate draining a certain amount of request but if the request is too big, it will close the connection.
The HEAD request is only a best effort solution and is only guaranteed to work if you only have a request concurrency of 1. Otherwise you risk warming up a connection, then another thread picks up that warmed up connection and your current thread then tries to send the real request which creates a new "clean" connection which needs authentication and you get the same problem. It turns the problem into a race condition.

[wfurt]

I know this is very old issue, but can somebody confirm if this is still broken and relevant?