[HTTP] An error rate gets occurred in perf test done for http/1.1 scenario (H1_H1)

[ldclakmal]

Description:
An error rate gets occurred in perf test done for the h1_h1 scenario, under high load (1000 concurrency, 50 or 1024 bytes payload). The test was run for 10 mins in 3 different machines for JMeter, Ballerina passthrough and Netty echo backend. But roughly an hour after stopping the test, OOM was occurred at Ballerina passthrough service.

Request failures can be seen with the following errors.

• At JMeter:
  

• At passthrough (h1_h1.bal): internal_log_h1_h1.log

[2019-02-25 13:05:39,563] WARN  {io.netty.channel.DefaultChannelPipeline} - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. 
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received close_notify during handshake

[2019-02-25 13:26:10,471] WARN  {io.netty.channel.ChannelInitializer} - Failed to initialize a channel. Closing: [id: 0xc6ae626e] 
java.lang.OutOfMemoryError: GC overhead limit exceeded

• At Netty backend: netty_log_h1_h1.txt

Affected Versions:
0.990.3

[ldclakmal]

The issue is fixed with multiple improvements done for netty backend [1] and the latest test results can be found at [2].

[1] https://github.com/wso2/performance-common
[2] https://github.com/ballerina-platform/ballerina-lang/blob/performance-test-162-2019-03-18_10-05-56/performance/benchmarks/summary.md