Crashes when initializing System.Net.Security.Native

[jkoritzinsky]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=206370
Build error leg or test failing: System.Net.Mail.Functional.Tests.WorkItemExecution
Pull request: #82867

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "(at NetSecurityNative:<InitSecContext>g____PInvoke|error 6 in gssntlmssp.so)",
  "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[ghost]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=206370
Build error leg or test failing: System.Net.Mail.Functional.Tests.WorkItemExecution
Pull request: #82867

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "at NetSecurityNative:<InitSecContext>g____PInvoke|16_0",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Author:	jkoritzinsky
Assignees:	-
Labels:	area-Infrastructure-libraries, blocking-clean-ci, Known Build Error
Milestone:	-

[ghost]

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=206370
Build error leg or test failing: System.Net.Mail.Functional.Tests.WorkItemExecution
Pull request: #82867

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "at NetSecurityNative:<InitSecContext>g____PInvoke|16_0",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Author:	jkoritzinsky
Assignees:	-
Labels:	area-System.Net.Security, blocking-clean-ci, untriaged, Known Build Error
Milestone:	-

[CarnaViire]

Seems like a dupe of #83482 and/or #83481...
cc @wfurt

[wfurt]

This seems to be specific to RedHat 7. It just got updated package so I assume the updated rolled in yesterday.
As far as I can tell disabling test runs on RedHat.7 is only one way how to stabilize CI.

cc: @simo5 @tmds @omajid

[toweinfu@toweinfu-rh7 System.Net.Mail.Functional.Tests]$ rpm -qi  gssntlmssp
Name        : gssntlmssp
Version     : 1.2.0
Release     : 1.el7
Architecture: x86_64
Install Date: Thu 09 Mar 2023 06:29:25 PM UTC
Group       : System Environment/Libraries
Size        : 137341
License     : LGPLv3+
Signature   : RSA/SHA256, Tue 21 Feb 2023 04:30:30 PM UTC, Key ID 6a2faea2352c64e5
Source RPM  : gssntlmssp-1.2.0-1.el7.src.rpm
Build Date  : Tue 21 Feb 2023 02:14:11 PM UTC
Build Host  : buildhw-x86-12.iad2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://fedorahosted.org/gss-ntlmssp
Bug URL     : https://bugz.fedoraproject.org/gssntlmssp
Summary     : GSSAPI NTLMSSP Mechanism
Description :
A GSSAPI Mechanism that implements NTLMSSP

(lldb) r
Process 2230 launched: '../../correlation-payload/dotnet' (x86_64)
  Discovering: System.Net.Mail.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Mail.Functional.Tests (found 155 of 156 test cases)
  Starting:    System.Net.Mail.Functional.Tests (parallel test collections = on, max threads = 2)
Process 2230 stopped
* thread #11: tid = 2256, 0x00007fff74e9331f gssntlmssp.so`ntlm_decode_u16l_str_hdr(str_hdr=0x00007fbeb001266c, buffer=0x00007fbeb0009ba0, payload_offs=56, str=0x00007fbecabfbd40, ctx=0x0000000000000000) + 111 at ntlm.c:328, name = '.NET Long Runni', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x00007fff74e9331f gssntlmssp.so`ntlm_decode_u16l_str_hdr(str_hdr=0x00007fbeb001266c, buffer=0x00007fbeb0009ba0, payload_offs=56, str=0x00007fbecabfbd40, ctx=0x0000000000000000) + 111 at ntlm.c:328
   325 	        safefree(out);
   326 	    } else {
   327 	        /* make sure to terminate output string */
-> 328 	        out[outlen] = '\0';
   329 	    }
   330
   331 	    *str = out;
(lldb) bt
* thread #11: tid = 2256, 0x00007fff74e9331f gssntlmssp.so`ntlm_decode_u16l_str_hdr(str_hdr=0x00007fbeb001266c, buffer=0x00007fbeb0009ba0, payload_offs=56, str=0x00007fbecabfbd40, ctx=0x0000000000000000) + 111 at ntlm.c:328, name = '.NET Long Runni', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x00007fff74e9331f gssntlmssp.so`ntlm_decode_u16l_str_hdr(str_hdr=0x00007fbeb001266c, buffer=0x00007fbeb0009ba0, payload_offs=56, str=0x00007fbecabfbd40, ctx=0x0000000000000000) + 111 at ntlm.c:328
    frame #1: 0x00007fff74e94b2a gssntlmssp.so`ntlm_decode_chal_msg(ctx=<unavailable>, buffer=0x00007fbeb0009ba0, _flags=0x00007fbecabfbdd4, target_name=0x00007fbecabfbe08, challenge=0x00007fbecabfbe10, target_info=0x00007fbecabfbe20) + 186 at ntlm.c:1125
    frame #2: 0x00007fff74e9b394 gssntlmssp.so`gssntlm_init_sec_context(minor_status=0x00007fbecabfcb00, claimant_cred_handle=0x00007fbeb0012c30, context_handle=<unavailable>, target_name=<unavailable>, mech_type=<unavailable>, req_flags=<unavailable>, time_req=0, input_chan_bindings=0x0000000000000000, input_token=0x00007fbeb0009900, actual_mech_type=0x00007fbeb0013100, output_token=0x00007fbecabfc120, ret_flags=0x00007fbeb00130f0, time_rec=0x0000000000000000) + 2436 at gss_sec_ctx.c:290
    frame #3: 0x00007fff76297ecb libgssapi_krb5.so.2`gss_init_sec_context + 555
    frame #4: 0x00007fff762be6ef libgssapi_krb5.so.2`___lldb_unnamed_symbol372$$libgssapi_krb5.so.2 + 239
    frame #5: 0x00007fff762c046a libgssapi_krb5.so.2`___lldb_unnamed_symbol374$$libgssapi_krb5.so.2 + 1178
    frame #6: 0x00007fff76297ecb libgssapi_krb5.so.2`gss_init_sec_context + 555
    frame #7: 0x00007fffeeefacd7 libSystem.Net.Security.Native.so`NetSecurityNative_InitSecContextEx(minorStatus=0x00007fbecabfcb00, claimantCredHandle=0x00007fbeb00127e0, contextHandle=0x00007fbecabfc760, packageType=0, cbt=0x0000000000000000, cb

[jkotas]

Likely introduced by this security fix: gssapi/gss-ntlmssp@c753000

When str_len is 0, the code takes goto done here, all ret, out and outlen are 0 and the code crashes here

[simo5]

That's unfortunate, can you open an issue against gssntlmssp?
I should be able to fix shortly and then try to release a new package in EPEL to deal with the regression.

[wfurt]

Thanks @simo5, I opened gssapi/gss-ntlmssp#90. Let me know if you need anything. I can also test new binaries if useful.

[simo5]

Nice, unfortunately I may need some time to get back to it, unless I find a couple of hours today.

[simo5]

What do you know, I found some time: gssapi/gss-ntlmssp#91

[simo5]

@wfurt here are EPEL7 builds if you need to test https://koji.fedoraproject.org/koji/taskinfo?taskID=98819730
Otherwise in a few weeks they will bubble through the update system (faster is someone tests and give karma):
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-6361864f28
https://bodhi.fedoraproject.org/updates/FEDORA-2023-d23a377bf8
https://bodhi.fedoraproject.org/updates/FEDORA-2023-a76142a62f

[wfurt]

Thanks for the links. I still have VM running so I can give try try @simo5.

[wfurt]

I tested the linked binaries (on Centos7) and everything looks good to me @simo5. Thanks for quic fix.
Do you know if there is story for Debian & Ubuntu? It seems to be still on 0.7 and that probably suffer same issue found by fuzzing and does not have your other fix for OpenSSL 3.