recover from filestorage panic on corrupted DB #36840
Description
Component(s)
extension/storage/filestorage
Is your feature request related to a problem? Please describe.
we are using filelog receiver to collect logs and file_storage extension to persist read states, we have a large number of collectors running and occasionally we saw collectors are crashing while opening the DB, not sure why and how it ran into this state, at the mean time, should we add recover to prevent collector from crashing because of this?
(masked receiver name)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb5f2a0]
goroutine 95 [running]:
go.etcd.io/bbolt.(*Bucket).Sequence(0x7eff026ea057?)
go.etcd.io/[email protected]/bucket.go:346
go.etcd.io/bbolt.Compact.walk.func3.1({0x7eff026ea057, 0x15, 0x15}, 0x0)
go.etcd.io/[email protected]/compact.go:94 +0x3c
go.etcd.io/bbolt.(*Tx).ForEach.func1({0x7eff026ea057, 0x15, 0x15}, {0x0?, 0x0?, 0x3fe0000000000000?})
go.etcd.io/[email protected]/tx.go:131 +0x62
go.etcd.io/bbolt.(*Bucket).ForEach(0xffffffffffffffff?, 0xc001d1cd58)
go.etcd.io/[email protected]/bucket.go:397 +0x90
go.etcd.io/bbolt.(*Tx).ForEach(0xc000c75f28?, 0xc001d1cda0?)
go.etcd.io/[email protected]/tx.go:130 +0x45
go.etcd.io/bbolt.Compact.walk.func3(0x0?)
go.etcd.io/[email protected]/compact.go:93 +0x33
go.etcd.io/bbolt.(*DB).View(0x3b9aca00?, 0xc001d1cea8)
go.etcd.io/[email protected]/db.go:917 +0x72
go.etcd.io/bbolt.walk(...)
go.etcd.io/[email protected]/compact.go:92
go.etcd.io/bbolt.Compact(0xc000954248, 0xc000c75d48, 0x10000)
go.etcd.io/[email protected]/compact.go:21 +0x14c
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage.(*fileStorageClient).Compact(0xc000886d70, {0xc0000479b0?, 0xc001d1d3e8?}, 0x3b9aca00, 0x10000)
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/[email protected]/client.go:198 +0x425
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage.(*localFileStorage).GetClient(0xc0009e8a80, {0x7c8e00?, 0xc00094cc80?}, 0x0?, {{{0xc000a9b920?, 0x82dc76e33?}}, {0xc000a9b928?, 0x0?}}, {0x0, 0x0})
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/[email protected]/extension.go:82 +0x405
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.GetStorageClient({0x1b722b0, 0x29c2000}, {0x1b5c3c0?, 0xc0009f7160?}, 0xc001b3de20, {{{0xc000a9b920?, 0x0?}}, {0xc000a9b928?, 0x0?}})
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/adapter/storage.go:29 +0x173
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).setStorageClient(0xc001c87420, {0x1b722b0?, 0x29c2000?}, {0x1b5c3c0?, 0xc0009f7160?})
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/adapter/storage.go:34 +0x56
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).Start(0xc001c87420, {0x1b722b0, 0x29c2000}, {0x1b5c3c0, 0xc0009f7160})
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/adapter/receiver.go:49 +0xaf
Describe the solution you'd like
recover on DB error
Describe alternatives you've considered
No response
Additional context
No response