hooks: Hooks continue failing after plugin crashes

[Acconut]

When the process for a hook plugin crashes (e.g. because of a panic), the current hook invocation fails which is expected and acceptable. However, all future hook invocations will also fail because the plugin has died. We should investigate ways to restart the plugin to allow hooks again.

First invocation when the plugin crashes:

2024/05/21 10:19:14.635445 level=INFO event=RequestIncoming method=POST path="" requestId=""
2024/05/21 10:19:14.635493 level=DEBUG event=HookInvocationStart type=pre-create id=""
2024/05/21 10:19:14.635806 [DEBUG] plugin.hook_handler: 2024/05/21 10:19:14 MyHookHandler.InvokeHook is invoked
2024/05/21 10:19:14.638372 [DEBUG] plugin.hook_handler: panic: oh no
2024/05/21 10:19:14.638379 [DEBUG] plugin.hook_handler
2024/05/21 10:19:14.638383 [DEBUG] plugin.hook_handler: goroutine 26 [running]:
2024/05/21 10:19:14.638429 [DEBUG] plugin.hook_handler: main.(*MyHookHandler).InvokeHook(0xb8?, {{0x140000d2bf0, 0xa}, {{0x0, 0x0}, {{0x0, 0x0}, 0xa, 0x0, 0x0, ...}, ...}})
2024/05/21 10:19:14.638433 [DEBUG] plugin.hook_handler:         /Users/marius/workspace/tus/tusd/examples/hooks/plugin/hook_handler.go:29 +0xa4
2024/05/21 10:19:14.638499 [DEBUG] plugin.hook_handler: github.com/tus/tusd/v2/pkg/hooks/plugin.(*HookHandlerRPCServer).InvokeHook(0xc0?, {{0x140000d2bf0, 0xa}, {{0x0, 0x0}, {{0x0, 0x0}, 0xa, 0x0, 0x0, ...}, ...}}, ...)
2024/05/21 10:19:14.638505 [DEBUG] plugin.hook_handler:         /Users/marius/workspace/tus/tusd/pkg/hooks/plugin/plugin.go:123 +0x58
2024/05/21 10:19:14.638581 [DEBUG] plugin.hook_handler: reflect.Value.call({0x140001ae480?, 0x140002840d0?, 0x13?}, {0x1024b0ee1, 0x4}, {0x14000235ef8, 0x3, 0x3?})
2024/05/21 10:19:14.638591 [DEBUG] plugin.hook_handler:         /opt/homebrew/Cellar/go/1.21.1/libexec/src/reflect/value.go:596 +0x994
2024/05/21 10:19:14.638619 [DEBUG] plugin.hook_handler: reflect.Value.Call({0x140001ae480?, 0x140002840d0?, 0x1?}, {0x1400028fef8?, 0x1400028fe88?, 0x10205b49c?})
2024/05/21 10:19:14.638624 [DEBUG] plugin.hook_handler:         /opt/homebrew/Cellar/go/1.21.1/libexec/src/reflect/value.go:380 +0x94
2024/05/21 10:19:14.638672 [DEBUG] plugin.hook_handler: net/rpc.(*service).call(0x14000286340, 0x0?, 0x0?, 0x1400029c080, 0x140002a4200, 0x14000212360?, {0x102633a20?, 0x14000210180?, 0x0?}, {0x1025bfb00?, ...}, ...)
2024/05/21 10:19:14.638677 [DEBUG] plugin.hook_handler:         /opt/homebrew/Cellar/go/1.21.1/libexec/src/net/rpc/server.go:382 +0x204
2024/05/21 10:19:14.638706 [DEBUG] plugin.hook_handler: created by net/rpc.(*Server).ServeCodec in goroutine 9
2024/05/21 10:19:14.638708 [DEBUG] plugin.hook_handler:         /opt/homebrew/Cellar/go/1.21.1/libexec/src/net/rpc/server.go:479 +0x304
2024/05/21 10:19:14.639243 level=ERROR event=HookInvocationError type=pre-create id="" error="unexpected EOF"
2024/05/21 10:19:14.639255 level=ERROR event=InternalServerError method=POST path="" requestId="" message="unexpected EOF"
2024/05/21 10:19:14.639269 level=INFO event=ResponseOutgoing method=POST path="" requestId="" status=500 body="ERR_INTERNAL_SERVER_ERROR: unexpected EOF\n"
2024/05/21 10:19:14.639304 [ERROR] plugin: plugin process exited: plugin=./examples/hooks/plugin/hook_handler id=61777 error="exit status 2"

All future hook invocations:

2024/05/21 10:19:20.836569 level=INFO event=RequestIncoming method=POST path="" requestId=""
2024/05/21 10:19:20.836609 level=DEBUG event=HookInvocationStart type=pre-create id=""
2024/05/21 10:19:20.836620 level=ERROR event=HookInvocationError type=pre-create id="" error="connection is shut down"
2024/05/21 10:19:20.836636 level=ERROR event=InternalServerError method=POST path="" requestId="" message="connection is shut down"
2024/05/21 10:19:20.836647 level=INFO event=ResponseOutgoing method=POST path="" requestId="" status=500 body="ERR_INTERNAL_SERVER_ERROR: connection is shut down\n"

According to hashicorp/go-plugin#31, there is not built-in way inside go-plugin and we would have to detect such errors on our own and then initialize a new client.