Add reconnection to Gateway (form nb2kg)

[oyvsyo]

Mirror logic from
jupyter/nb2kg#45
that implements an exponential backoff algorithm and retry count
limit to prevent too many false API calls in error trying to reconnect to an orphaned session

@kevin-bates @esevan Can we add it here?

[kevin-bates]

Thank you @oyvsyo - these changes look good. Just had one comment on the wording.

Once this is merged, we will want to apply this to applicable code in jupyter_server/gateway as well. Are you able to make that contribution also?

[kevin-bates]

But I have not tested it yet, so maybe we should put it on WIP and I will install notebook from this branch and try to run the gateway?

No worries - thanks! Go ahead and place this into DRAFT mode if it's going to be multiple days, otherwise, just ping back when you're ready.

[oyvsyo]

was able to reproduce a lost session and get EnterpriseGatewayApp logs listed below:

[I 2021-01-06 11:58:00.621 EnterpriseGatewayApp] Jupyter Enterprise Gateway 2.2.0.dev0 is available at http://0.0.0.0:6666
[W 2021-01-06 11:58:04.375 EnterpriseGatewayApp] No session ID specified
[W 210106 11:58:04 web:1786] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1): Kernel does not exist: ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9
[W 210106 11:58:04 web:2250] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1) 3.58ms
[W 2021-01-06 11:58:06.236 EnterpriseGatewayApp] No session ID specified
[W 210106 11:58:06 web:1786] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1): Kernel does not exist: ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9
[W 210106 11:58:06 web:2250] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1) 3.32ms
[W 2021-01-06 11:58:08.606 EnterpriseGatewayApp] No session ID specified
[W 210106 11:58:08 web:1786] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1): Kernel does not exist: ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9
[W 210106 11:58:08 web:2250] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1) 3.27ms
[W 2021-01-06 11:58:12.771 EnterpriseGatewayApp] No session ID specified
[W 210106 11:58:12 web:1786] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1): Kernel does not exist: ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9
[W 210106 11:58:12 web:2250] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1) 3.76ms
[W 2021-01-06 11:58:20.920 EnterpriseGatewayApp] No session ID specified
[W 210106 11:58:20 web:1786] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1): Kernel does not exist: ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9
[W 210106 11:58:20 web:2250] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1) 2.83ms
[W 2021-01-06 11:58:37.837 EnterpriseGatewayApp] No session ID specified
[W 210106 11:58:37 web:1786] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1): Kernel does not exist: ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9
[W 210106 11:58:37 web:2250] 404 GET /api/kernels/ce5cdbbb-a718-4f6c-b91a-09b0ac51a3a9/channels (127.0.0.1) 2.48ms

There are timings in power of two so that seems to work proper

[kevin-bates]

Looks good Dvoiak - thank you. Please post a similar PR to jupyter_server when you can. (I believe you had confirmed that previously, but I no longer see that comment.)