Panic extension/storage/filestorage after upgrate OS

[yanamg7]

Describe the bug
After upgrading os from 1.7.0 to 1.8.0 with two tenants seeing otel core and otel pod are in crashloopbackoff state

Steps to reproduce

1. Bringup up Vanquish High with 1.7.0 RC build and downgrade the firmwares
2. Now deploy two tenants and configure networks and verify traffic
3. Continuously send traffic over the tenants and upgrade the appliance to 1.8.0 RC build
4. Once firmware upgrades and cluster upgrades is completed check for tenants' status
5. We see on of the next tenant has OTEL pod not coming up with state crashloopbackoff
6. If we check for cores under /system//core/ we see OTEL core generated

What did you expect to see?
We should not see cores and pods should come up properly

What did you see instead?
OTEL core file and OTEL node in the crash loop

What version did you use?
Version: 0.91.0

Environment
OS: Velos upgrating f5os from 1.7.0 to 1.8.0

Additional context
the panic log:
panic: runtime error: invalid memory address or nil pointer dereference","ct":"f5-toda-otel-collector"}
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3804c60]","ct":"f5-toda-otel-collector"}
goroutine 1 [running]:","ct":"f5-toda-otel-collector"}
panic({0x6cf9900?, 0xdc346d0?})","ct":"f5-toda-otel-collector"}
runtime/panic.go:1017 +0x3ac fp=0xc002722738 sp=0xc002722688 pc=0x43be0c","ct":"f5-toda-otel-collector"}
runtime.panicmem(...)","ct":"f5-toda-otel-collector"}
runtime/panic.go:261","ct":"f5-toda-otel-collector"}
runtime.sigpanic()","ct":"f5-toda-otel-collector"}
runtime/signal_unix.go:861 +0x378 fp=0xc002722798 sp=0xc002722738 pc=0x4534b8","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt.(*Bucket).Sequence(0xc0027227f0?)","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt@v1.3.8/bucket.go:338 fp=0xc0027227a0 sp=0xc002722798 pc=0x3804c60","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt.Compact.walk.func3.1({0x7efc9c74f057, 0x15, 0x15}, 0x7efce38d25b8?)","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt@v1.3.8/compact.go:94 +0x3c fp=0xc002722820 sp=0xc0027227a0 pc=0x380699c","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt.(*Tx).ForEach.func1({0x7efc9c74f057, 0x15, 0x15}, {0x0?, 0x0?, 0x3fe0000000000000?})","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt@v1.3.8/tx.go:130 +0x62 fp=0xc002722858 sp=0xc002722820 pc=0x3814902","ct":"f5-toda-otel-collector"}
"go.etcd.io/bbolt.(*Bucket).ForEach(0xffffffffffffffff?, 0xc0027228c8)","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt@v1.3.8/bucket.go:389 +0x90 fp=0xc0027228b8 sp=0xc002722858 pc=0x3804e50","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt.(*Tx).ForEach(0xc002772f60?, 0xc002722910?)","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt@v1.3.8/tx.go:129 +0x45 fp=0xc0027228f0 sp=0xc0027228b8 pc=0x3814865","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt.Compact.walk.func3(0xc001ce5ce0?)","ct":"f5-toda-otel-collector"}
log":"\tgo.etcd.io/bbolt@v1.3.8/compact.go:93 +0x33 fp=0xc002722920 sp=0xc0027228f0 pc=0x3806933","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt.(*DB).View(0x3b9aca00?, 0xc0027229d8)","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt@v1.3.8/db.go:917 +0x72 fp=0xc002722988 sp=0xc002722920 pc=0x380b9f2","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt.walk(...)","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt@v1.3.8/compact.go:92","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt.Compact(0xc002772fc0, 0x15?, 0x10000)","ct":"f5-toda-otel-collector"}
go.etcd.io/bbolt@v1.3.8/compact.go:21 +0x146 fp=0xc002722a48 sp=0xc002722988 pc=0x3806826","ct":"f5-toda-otel-collector"}
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage.(*fileStorageClient).Compact(0xc0034f4230, {0xc002d18bc8?, 0xc002722ed0?}, 0x3b9aca00, 0xc00314b9c0?)","ct":"f5-toda-otel-collector"}
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage@v0.91.0/filestorage/client.go:196 +0x418 fp=0xc002722e50 sp=0xc002722a48 pc=0x381aeb8","ct":"f5-toda-otel-collector"}
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage.(*localFileStorage).GetClient(0xc0031d8df0, {0xc1a8daf58a602983?, 0x0?}, 0xc1a8daf58a602983?, {{0xc0031a8e30?, 0xdccb460?}, {0xc0031a8e38?, 0x0?}}, {0x0, 0x0})","ct":"f5-toda-otel-collector"}
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage@v0.91.0/filestorage/extension.go:75 +0x3c5 fp=0xc002722fd0 sp=0xc002722e50 pc=0x381cec5","ct":"f5-toda-otel-collector"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.GetStorageClient({0x8d77e70, 0xdd0eec0}, {0x8d79440?, 0xc0030f5c00?}, 0xc0031ab760, {{0xc0031a8e30?, 0x0?}, {0xc0031a8e38?, 0x0?}})","ct":"f5-toda-otel-collector"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.91.0/adapter/storage.go:29 +0xfc fp=0xc002723060 sp=0xc002722fd0 pc=0x3ed509c","ct":"f5-toda-otel-collector"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).setStorageClient(0xc0033f8b40, {0x8d77e70?, 0xdd0eec0?}, {0x8d79440?, 0xc0030f5c00?})","ct":"f5-toda-otel-collector"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.91.0/adapter/storage.go:34 +0x52 fp=0xc0027230b8 sp=0xc002723060 pc=0x3ed5212","ct":"f5-toda-otel-collector"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).Start(0xc0033f8b40, {0x8d77e70, 0xdd0eec0}, {0x8d79440, 0xc0030f5c00})","ct":"f5-toda-otel-collector"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.91.0/adapter/receiver.go:47 +0xb0 fp=0xc002723128 sp=0xc0027230b8 pc=0x3ed42b0","ct":"f5-toda-otel-collector"}

core attached
core.f5-toda-otel-c.otelcol-contrib.37536.1724052566.core.gz

[github-actions]

Pinging code owners for extension/storage/filestorage: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Thanks for filing @yanamg7, would you be able to test on a more recent version of OpenTelemetry Collector? A code owner may be able to respond with more information, but it's hard to know if this issue has already been resolved or if it's still relevant today since the v0.91.0 release was created 9 months ago.

[yanamg7]

Thanks for filing @yanamg7, would you be able to test on a more recent version of OpenTelemetry Collector? A code owner may be able to respond with more information, but it's hard to know if this issue has already been resolved or if it's still relevant today since the v0.91.0 release was created 9 months ago.

OK, I'll try

[andrzej-stencel]

Any update here @yanamg7?