Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failure: System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http2.Http2_PingKeepAlive(keepAlivePingDelay: 00:00:01, keepAlivePingPolicy: WithActiveRequests, expectRequestFail: False) #41929

Closed
v-haren opened this issue Sep 7, 2020 · 15 comments · Fixed by #56736
Closed

Test failure: System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http2.Http2_PingKeepAlive(keepAlivePingDelay: 00:00:01, keepAlivePingPolicy: WithActiveRequests, expectRequestFail: False) #41929

v-haren opened this issue Sep 7, 2020 · 15 comments · Fixed by #56736
Assignees
@antonfirsov
Labels
area-System.Net.Http disabled-test The test is disabled in source code against the issue test-bug Problem in test source code (most likely) test-run-core Test failures in .NET Core test runs
Milestone
6.0.0

Comments

@v-haren
Copy link

v-haren commented Sep 7, 2020

failed in job: runtime-libraries-coreclr outerloop 20200906.1

net5.0-Linux-Release-arm64-CoreCLR_release-(Alpine.312.Arm64.Open)ubuntu.1804.armarch.open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e

Error message

System.AggregateException : One or more errors occurred. (One or more errors occurred. (An error occurred while sending the request.)) (One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.))
---- System.AggregateException : One or more errors occurred. (An error occurred while sending the request.)
-------- System.Net.Http.HttpRequestException : An error occurred while sending the request.
------------ System.IO.IOException : The request was aborted.
---------------- System.Net.Http.Http2ConnectionException : The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).
---- System.AggregateException : One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.)
-------- System.IO.IOException : Unable to write data to the transport connection: Broken pipe.
------------ System.Net.Sockets.SocketException : Broken pipe


Stack trace
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 122
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 84
   at System.Net.Test.Common.Http2LoopbackServer.CreateClientAndServerAsync(Func`2 clientFunc, Func`2 serverFunc, Http2Options http2Options, Int32 timeout) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 172
   at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http2.Http2_PingKeepAlive(TimeSpan keepAlivePingDelay, HttpKeepAlivePingPolicy keepAlivePingPolicy, Boolean expectRequestFail) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http2.cs:line 1672
--- End of stack trace from previous location ---
----- Inner Stack Trace #1 (System.AggregateException) -----

----- Inner Stack Trace -----
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1917
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 892
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 736
   at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 251
   at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http2.<>c__DisplayClass62_0.<<Http2_PingKeepAlive>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http2.cs:line 1697
----- Inner Stack Trace -----
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 2019
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 808
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 837
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 887
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1917
----- Inner Stack Trace -----
   at System.Net.Http.Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 2027
   at System.Net.Http.Http2Connection.ThrowProtocolError() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 2023
   at System.Net.Http.Http2Connection.VerifyKeepAlive() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1988
   at System.Net.Http.Http2Connection.HeartBeat() in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1027
----- Inner Stack Trace #2 (System.AggregateException) -----

----- Inner Stack Trace -----
   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|177_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 724
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1030
   at System.Net.Test.Common.Http2LoopbackConnection.WriteFrameAsync(Frame frame) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 117
   at System.Net.Test.Common.Http2LoopbackConnection.SendDefaultResponseAsync(Int32 streamId) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 738
   at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http2.<>c__DisplayClass62_0.<<Http2_PingKeepAlive>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http2.cs:line 1747
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 96
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 924
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsyncForNetworkStream(Socket socket, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 727
   at System.Net.Sockets.Socket.SendAsyncForNetworkStream(ReadOnlyMemory`1 buffer, SocketFlags socketFlags, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 299
   at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs:line 720
   at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 672
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1012
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 807
   at System.Net.Security.SslStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 802
   at System.IO.Stream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 656
   at System.Net.Test.Common.Http2LoopbackConnection.WriteFrameAsync(Frame frame) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 117
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Test.Common.Http2LoopbackConnection.WriteFrameAsync(Frame frame)
   at System.Net.Test.Common.Http2LoopbackConnection.SendDefaultResponseAsync(Int32 streamId) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 738
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Test.Common.Http2LoopbackConnection.SendDefaultResponseAsync(Int32 streamId)
   at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http2.<>c__DisplayClass62_0.<<Http2_PingKeepAlive>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http2.cs:line 1747
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 289
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 186
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 327
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 305
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 186
   at System.Threading.Tasks.Task.InnerInvoke() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2376
   at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2358
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 186
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2316
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2252
   at System.Threading.Tasks.ThreadPoolTaskScheduler.<>c.<.cctor>b__10_0(Object s) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs:line 35
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state) in /_/src/coreclr/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 46
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 186
   at System.Threading.ThreadHelper.ThreadStart(Object obj) in /_/src/coreclr/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 74
--- End of stack trace from previous location ---
@ghost
Copy link

ghost commented Sep 7, 2020

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Sep 7, 2020
@v-haren

This comment has been minimized.

Sign in to view
@ManickaP
Copy link
Member

Triage: probably test issue, it's timing dependent since it's testing background process.

@ManickaP ManickaP removed the untriaged New issue has not been triaged by the area owner label Sep 18, 2020
@ManickaP ManickaP added this to the 6.0.0 milestone Sep 18, 2020
@ManickaP ManickaP added the test-bug Problem in test source code (most likely) label Sep 18, 2020
This was referenced Sep 25, 2020
Merged
Merged
@ManickaP ManickaP mentioned this issue Oct 9, 2020
Closed
@stephentoub stephentoub mentioned this issue Nov 4, 2020
Merged
@v-haren

This comment has been minimized.

Sign in to view
@ManickaP ManickaP self-assigned this Nov 6, 2020
@antonfirsov antonfirsov mentioned this issue Nov 6, 2020
Merged
@antonfirsov antonfirsov added the blocking-clean-ci-optional Blocking optional rolling runs label Nov 6, 2020
@antonfirsov
Copy link
Member

According to our reports and experience, this (and probably #41531) are the most annoying NCL test issues today. It's almost impossible to get an outerloop run without one or both of them failing.

@karelz how about kicking priority here?

/cc @wfurt

@antonfirsov
Copy link
Member

image

Totally missed this in my previous comment.

@runfoapp runfoapp bot mentioned this issue Nov 6, 2020
Closed
@ManickaP
Copy link
Member

ManickaP commented Nov 6, 2020 •
edited
Loading

There's been spike of errors in the last 2 days:

after 5.11.202	34	2020-11-05T11:51:30.497Z
before 5.11.202	46	2020-09-07T11:53:27.273Z

34 errors in the last two days.

@v-haren

This comment has been minimized.

Sign in to view
@stephentoub stephentoub added disabled-test The test is disabled in source code against the issue and removed blocking-clean-ci-optional Blocking optional rolling runs labels Nov 25, 2020
@ManickaP ManickaP removed their assignment Nov 26, 2020
@ManickaP
Copy link
Member

Disabled in #45214

@v-haren

This comment has been minimized.

Sign in to view
@wfurt
Copy link
Member

wfurt commented Dec 2, 2020

it seems like the last failure is on release/5.0 Is there guidance @danmosemsft how much effort we want to put into that branch? One strategy could be to disable all flaky tests and focus on stabilization in master. If we car, I would also expect we should log 5.0 bugs separate?

@danmoseley
Copy link
Member

Is there guidance @danmosemsft how much effort we want to put into that branch

We will be servicing 5.0 until about Feb 2022 so we will still need to run PR validation a few 100 times over that time. In general I think this means if a test is significantly flaky we generally either disable it (as long as we still get reasonably coverage) or backport a test fix if any.

@VincentBu

This comment has been minimized.

Sign in to view
@karelz karelz mentioned this issue Feb 12, 2021
Merged
@VincentBu

This comment has been minimized.

Sign in to view
@karelz karelz added the test-run-core Test failures in .NET Core test runs label Apr 23, 2021
@karelz
Copy link
Member

karelz commented Jul 22, 2021 •
edited
Loading

Summary of details and history:

@antonfirsov antonfirsov mentioned this issue Aug 2, 2021
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Aug 2, 2021
antonfirsov added a commit that referenced this issue Aug 6, 2021
@antonfirsov
Redisign HTTP/2 KeepAlive PING tests (#56736)
a8baca1 
Completely redesign tests for HTTP/2 KeepAlive PING, so they:
- Work well with RTT pings introduced in Implement dynamic HTTP/2 window scaling #54755
- Run sequentially, reducing the chance of failing because of timing issues caused by parallel workloads
- Are better organized: multiple test cases for different scenarios, instead of one theory with complex branches on parameters

Instead of reading / reacting to frames inline, there is a separate Task for processing incoming frames, responding to PING immediately and pushing other frames to a Channel<Frame>.

Fixes #41929
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Aug 6, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Sep 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@antonfirsov antonfirsov

Labels
area-System.Net.Http disabled-test The test is disabled in source code against the issue test-bug Problem in test source code (most likely) test-run-core Test failures in .NET Core test runs
Projects
None yet
Milestone
6.0.0
Development

Successfully merging a pull request may close this issue.

Redisign HTTP/2 KeepAlive PING tests antonfirsov/runtime
9 participants
@stephentoub @karelz @danmoseley @antonfirsov @ManickaP @wfurt @v-haren @VincentBu @Dotnet-GitSync-Bot