Reconfiguring the client (via Start Config packet) breaks it sometimes

[VidTu]

Describe the bug
Reconfiguring a client with Start Configuration packet breaks the mod until the game is restarted.

To Reproduce
Steps to reproduce the behavior:

1. Download Velocity (or any proxy that used RECONFIG packets to switch between servers) for 1.20.5.
2. Configure it for two backend (1.20.5 "vanilla" servers used in testing without ViaVersion or any other protocol hacks, will possibly break all 1.20.2+ with ViaVersion) servers.
3. Join the proxy. (connect to the first backend through Velocity)
4. Attempt to switch to second backend.
5. It kicks you. Sometimes it does, sometimes it doesn't. Sometimes it locks you from accessing the worls. Fully relogging into the proxy and then switching servers a few times (both relogging and switching) should reliably reproduce the bug.

Expected behavior
This shouldn't happen like in vanilla client.

Screenshots

Runtime info (please complete the following information):

• OS: Win11
• Minecraft version: 1.20.5 (Fabric)
• Mod version: 0.2.0+beta.7.151 (509a207)
• Mod branch: ver/1.20.6 (ig?...)

Crash reports / logs
https://paste.gg/p/VidTu/00091feda94d4637876a8084c1039bec (both deobf and obf)

Other mods
First log paste has StackDeobfuscator used for meaningful logs.
Second paste has only Fabric (w/o API) + VMP.

Checklist

• I am using the official version of the mod.
• I tried the latest development version but the issue persists.
• I searched for similar open issues and could not find an existing bug report on this.

Additional context
This doesn't happen in Vanilla no matter how hard I try.
Dev version wasn't tested but there are no significant changes.