Categories
DevOps

How to fix broken Prometheus service in GitLab arm64 14.4.2

Fix broken Prometheus service in GitLab arm64 14.4.2.

The issue

Prometheus in GitLab arm64 14.4.2 is not working after installation.

Inspect GitLab status. The Prometheus service is running, which is a misleading as it does this for several seconds and simply restarts.

$ sudo gitlab-ctl status
run: alertmanager: (pid 13795) 80007s; run: log: (pid 12997) 80256s
run: gitaly: (pid 13918) 80000s; run: log: (pid 12432) 80542s
run: gitlab-exporter: (pid 13761) 80013s; run: log: (pid 12902) 80289s
run: gitlab-workhorse: (pid 13739) 80014s; run: log: (pid 12821) 80318s
run: grafana: (pid 13813) 80006s; run: log: (pid 13539) 80083s
run: logrotate: (pid 184838) 1366s; run: log: (pid 12359) 80565s
run: nginx: (pid 12832) 80314s; run: log: (pid 12845) 80311s
run: node-exporter: (pid 13753) 80013s; run: log: (pid 12884) 80300s
run: postgres-exporter: (pid 13805) 80007s; run: log: (pid 13028) 80244s
run: postgresql: (pid 12490) 80529s; run: log: (pid 12578) 80526s
run: prometheus: (pid 187452) 6s; run: log: (pid 12953) 80267s
run: puma: (pid 23295) 76950s; run: log: (pid 12764) 80339s
run: redis: (pid 12387) 80556s; run: log: (pid 12396) 80555s
run: redis-exporter: (pid 13763) 80012s; run: log: (pid 12930) 80278s
run: sidekiq: (pid 23268) 76958s; run: log: (pid 12781) 80330s

Inspect Prometheus version.

$ /opt/gitlab/embedded/bin/prometheus --version
prometheus, version 2.25.0 (branch: master, revision: )
  build user:       GitLab-Omnibus
  build date:       
  go version:       go1.16.4
  platform:         linux/arm64

Inspect Prometheus logs.

$ sudo gitlab-ctl tail prometheus
[...]
2021-11-16_22:20:55.31546 unexpected fault address 0x75625f736484fb
2021-11-16_22:20:55.31565 fatal error: fault
2021-11-16_22:20:55.31797 [signal SIGSEGV: segmentation violation code=0x1 addr=0x75625f736484fb pc=0x187ae90]
2021-11-16_22:20:55.31805 
2021-11-16_22:20:55.31810 goroutine 24783 [running]:
2021-11-16_22:20:55.32110 runtime.throw(0x21fdccc, 0x5)
2021-11-16_22:20:55.32127       /usr/local/go/src/runtime/panic.go:1117 +0x54 fp=0x4002827560 sp=0x4002827530 pc=0x45f94
2021-11-16_22:20:55.32132 runtime.sigpanic()
2021-11-16_22:20:55.32137       /usr/local/go/src/runtime/signal_unix.go:741 +0x230 fp=0x40028275a0 sp=0x4002827560 pc=0x5da70
2021-11-16_22:20:55.32141 github.com/golang/snappy.encodeBlock(0x40027d4003, 0x5dfe, 0x5dfe, 0x4002800000, 0x5078, 0x6a80, 0x3)
2021-11-16_22:20:55.32146       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/golang/snappy@v0.0.2/encode_arm64.s:666 +0x360 fp=0x400282f640 sp=0x40028275b0 pc=0x187ae90
2021-11-16_22:20:55.32150 github.com/golang/snappy.Encode(0x40027d4000, 0x5e01, 0x5e01, 0x0, 0x0, 0x0, 0x5078, 0x400269e900, 0x4000178500)
2021-11-16_22:20:55.32154       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/golang/snappy@v0.0.2/encode.go:39 +0x17c fp=0x400282f690 sp=0x400282f640 pc=0x187a3cc
2021-11-16_22:20:55.32159 github.com/prometheus/prometheus/tsdb/wal.(*WAL).log(0x40001443f0, 0x4002800000, 0x5078, 0x6a80, 0x1, 0x8c, 0x400269e94c)
2021-11-16_22:20:55.32163       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/wal/wal.go:634 +0x368 fp=0x400282f730 sp=0x400282f690 pc=0x18819a8
2021-11-16_22:20:55.32168 github.com/prometheus/prometheus/tsdb/wal.(*WAL).Log(0x40001443f0, 0x400282f838, 0x1, 0x1, 0x0, 0x0)
2021-11-16_22:20:55.32180       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/wal/wal.go:596 +0xc8 fp=0x400282f7c0 sp=0x400282f730 pc=0x1881588
2021-11-16_22:20:55.32194 github.com/prometheus/prometheus/tsdb.(*headAppender).log(0x40061c1680, 0x0, 0x0)
2021-11-16_22:20:55.32199       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/head.go:1259 +0x26c fp=0x400282f870 sp=0x400282f7c0 pc=0x18a2e0c
2021-11-16_22:20:55.32203 github.com/prometheus/prometheus/tsdb.(*headAppender).Commit(0x40061c1680, 0x0, 0x0)
2021-11-16_22:20:55.32207       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/head.go:1279 +0x74 fp=0x400282f9b0 sp=0x400282f870 pc=0x18a2f64
2021-11-16_22:20:55.32211 github.com/prometheus/prometheus/tsdb.dbAppender.Commit(0x29c3560, 0x40061c1680, 0x4000bb40e0, 0x3, 0x256065fde24b4dba)
2021-11-16_22:20:55.32215       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/db.go:794 +0x30 fp=0x400282f9f0 sp=0x400282f9b0 pc=0x1894390
2021-11-16_22:20:55.32299 github.com/prometheus/prometheus/tsdb.(*dbAppender).Commit(0x40028f8d38, 0x4002474940, 0x4002474940)
2021-11-16_22:20:55.32306       :1 +0x48 fp=0x400282fa30 sp=0x400282f9f0 pc=0x18be9f8
2021-11-16_22:20:55.32318 github.com/prometheus/prometheus/storage.(*fanoutAppender).Commit(0x4002474b80, 0x29c2fb0, 0x40028f8d50)
2021-11-16_22:20:55.32322       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/storage/fanout.go:174 +0x34 fp=0x400282fae0 sp=0x400282fa30 pc=0x18371e4
2021-11-16_22:20:55.32327 github.com/prometheus/prometheus/scrape.(*timeLimitAppender).Commit(0x40028f8d50, 0x3a9, 0x3a9)
2021-11-16_22:20:55.32330       :1 +0x3c fp=0x400282fb10 sp=0x400282fae0 pc=0x1947f6c
2021-11-16_22:20:55.32334 github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport.func1(0x400282fce0, 0x400224bcf0, 0x4002476420)
2021-11-16_22:20:55.32338       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:1086 +0x40 fp=0x400282fb90 sp=0x400282fb10 pc=0x19465a0
2021-11-16_22:20:55.32342 github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport(0x4002476420, 0x37e11d600, 0x37e11d600, 0x0, 0x0, 0x0, 0xc05d2991cee2727f, 0x5606599e2, 0x3af9040, 0x0, ...)
2021-11-16_22:20:55.32347       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:1153 +0x7a4 fp=0x400282fe00 sp=0x400282fb90 pc=0x193f984
2021-11-16_22:20:55.32359 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4002476420, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.32363       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:1039 +0x268 fp=0x400282ffb0 sp=0x400282fe00 pc=0x193eec8
2021-11-16_22:20:55.32367 runtime.goexit()
2021-11-16_22:20:55.32371       /usr/local/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x400282ffb0 sp=0x400282ffb0 pc=0x7d3c4
2021-11-16_22:20:55.32375 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.32378       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
2021-11-16_22:20:55.32382 
2021-11-16_22:20:55.32386 goroutine 1 [chan receive]:
2021-11-16_22:20:55.32390 github.com/oklog/run.(*Group).Run(0x4000b83b18, 0x400023ec40, 0x8)
2021-11-16_22:20:55.32393       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:43 +0xd4
2021-11-16_22:20:55.32397 main.main()
2021-11-16_22:20:55.32402       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:864 +0x6308
2021-11-16_22:20:55.32407 
2021-11-16_22:20:55.32411 goroutine 45 [select]:
2021-11-16_22:20:55.32414 go.opencensus.io/stats/view.(*worker).start(0x400052c280)
2021-11-16_22:20:55.32418       /var/cache/omnibus/src/prometheus/pkg/mod/go.opencensus.io@v0.22.5/stats/view/worker.go:276 +0x98
2021-11-16_22:20:55.32430 created by go.opencensus.io/stats/view.init.0
2021-11-16_22:20:55.32434       /var/cache/omnibus/src/prometheus/pkg/mod/go.opencensus.io@v0.22.5/stats/view/worker.go:34 +0x70
2021-11-16_22:20:55.32438 
2021-11-16_22:20:55.32441 goroutine 64 [select]:
2021-11-16_22:20:55.32445 github.com/prometheus/prometheus/pkg/logging.(*Deduper).run(0x400043b740)
2021-11-16_22:20:55.32461       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/pkg/logging/dedupe.go:75 +0x160
2021-11-16_22:20:55.32466 created by github.com/prometheus/prometheus/pkg/logging.Dedupe
2021-11-16_22:20:55.32470       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/pkg/logging/dedupe.go:61 +0xcc
2021-11-16_22:20:55.32474 
2021-11-16_22:20:55.32478 goroutine 103 [chan receive]:
2021-11-16_22:20:55.32481 github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0x4000255220)
2021-11-16_22:20:55.32485       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/storage/remote/write.go:96 +0x98
2021-11-16_22:20:55.32489 created by github.com/prometheus/prometheus/storage/remote.NewWriteStorage
2021-11-16_22:20:55.32493       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/storage/remote/write.go:89 +0x288
2021-11-16_22:20:55.32497 
2021-11-16_22:20:55.32501 goroutine 83 [syscall]:
2021-11-16_22:20:55.32504 os/signal.signal_recv(0x0)
2021-11-16_22:20:55.32508       /usr/local/go/src/runtime/sigqueue.go:168 +0xcc
2021-11-16_22:20:55.32520 os/signal.loop()
2021-11-16_22:20:55.32525       /usr/local/go/src/os/signal/signal_unix.go:23 +0x20
2021-11-16_22:20:55.33213 created by os/signal.Notify.func1.1
2021-11-16_22:20:55.33216       /usr/local/go/src/os/signal/signal.go:151 +0x44
2021-11-16_22:20:55.33218 
2021-11-16_22:20:55.33220 goroutine 84 [select]:
2021-11-16_22:20:55.33223 main.main.func6(0x6, 0x8)
2021-11-16_22:20:55.33225       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:627 +0x84
2021-11-16_22:20:55.33227 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40001a5b00, 0x40006c8230)
2021-11-16_22:20:55.33230       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28
2021-11-16_22:20:55.33232 created by github.com/oklog/run.(*Group).Run
2021-11-16_22:20:55.33234       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8
2021-11-16_22:20:55.33236 
2021-11-16_22:20:55.33239 goroutine 85 [chan receive]:
2021-11-16_22:20:55.33241 github.com/prometheus/prometheus/discovery.(*Manager).Run(0x40002555e0, 0x18600000007, 0x0)
2021-11-16_22:20:55.33243       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:142 +0x68
2021-11-16_22:20:55.33246 main.main.func8(0x6, 0x8)
2021-11-16_22:20:55.33248       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:647 +0x38
2021-11-16_22:20:55.33250 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40006a8180, 0x40006a81a0)
2021-11-16_22:20:55.33262       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28
2021-11-16_22:20:55.33264 created by github.com/oklog/run.(*Group).Run
2021-11-16_22:20:55.33267       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8
2021-11-16_22:20:55.33269 
2021-11-16_22:20:55.33271 goroutine 86 [chan receive]:
2021-11-16_22:20:55.33274 github.com/prometheus/prometheus/discovery.(*Manager).Run(0x4000255860, 0x16600000007, 0x0)
2021-11-16_22:20:55.33276       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:142 +0x68
2021-11-16_22:20:55.33278 main.main.func10(0x6, 0x8)
2021-11-16_22:20:55.33281       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:661 +0x38
2021-11-16_22:20:55.33283 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40006a81e0, 0x40006a8200)
2021-11-16_22:20:55.33285       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28
2021-11-16_22:20:55.33288 created by github.com/oklog/run.(*Group).Run
2021-11-16_22:20:55.33290       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8
2021-11-16_22:20:55.33292 
2021-11-16_22:20:55.33294 goroutine 87 [select]:
2021-11-16_22:20:55.33296 github.com/prometheus/prometheus/scrape.(*Manager).Run(0x400015cd20, 0x400009cba0, 0x0, 0x14600000007)
2021-11-16_22:20:55.33310       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/manager.go:142 +0x8c
2021-11-16_22:20:55.33312 main.main.func12(0x6, 0x8)
2021-11-16_22:20:55.33315       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:681 +0x68
2021-11-16_22:20:55.33317 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x4000169200, 0x40006a8220)
2021-11-16_22:20:55.33319       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28
2021-11-16_22:20:55.33322 created by github.com/oklog/run.(*Group).Run
2021-11-16_22:20:55.33324       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8
2021-11-16_22:20:55.33326 
2021-11-16_22:20:55.33328 goroutine 88 [select]:
2021-11-16_22:20:55.33330 main.main.func14(0x6, 0x8)
2021-11-16_22:20:55.33333       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:706 +0xd0
2021-11-16_22:20:55.33335 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40005ae360, 0x400023eb00)
2021-11-16_22:20:55.33337       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28
2021-11-16_22:20:55.33340 created by github.com/oklog/run.(*Group).Run
2021-11-16_22:20:55.33342       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8
2021-11-16_22:20:55.33357 
2021-11-16_22:20:55.33359 goroutine 89 [chan receive]:
2021-11-16_22:20:55.33362 main.main.func16(0x6, 0x8)
2021-11-16_22:20:55.33364       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:752 +0x2b4
2021-11-16_22:20:55.33366 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40005ae3c0, 0x400023eb50)
2021-11-16_22:20:55.33368       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28
2021-11-16_22:20:55.33371 created by github.com/oklog/run.(*Group).Run
2021-11-16_22:20:55.33373       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8
2021-11-16_22:20:55.33375 
2021-11-16_22:20:55.33378 goroutine 90 [chan receive]:
2021-11-16_22:20:55.33380 github.com/prometheus/prometheus/rules.(*Manager).Run(...)
2021-11-16_22:20:55.33383       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:912
2021-11-16_22:20:55.33385 main.main.func18(0x0, 0x0)
2021-11-16_22:20:55.33387       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:765 +0x64
2021-11-16_22:20:55.33390 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40001982a0, 0x400023eba0)
2021-11-16_22:20:55.33392       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28
2021-11-16_22:20:55.33394 created by github.com/oklog/run.(*Group).Run
2021-11-16_22:20:55.33396       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8
2021-11-16_22:20:55.33403 
2021-11-16_22:20:55.33406 goroutine 256 [select]:
2021-11-16_22:20:55.33408 github.com/prometheus/prometheus/discovery.(*Manager).sender(0x4000255860)
2021-11-16_22:20:55.33410       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:234 +0xb4
2021-11-16_22:20:55.33413 created by github.com/prometheus/prometheus/discovery.(*Manager).Run
2021-11-16_22:20:55.33415       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:141 +0x3c
2021-11-16_22:20:55.33417 
2021-11-16_22:20:55.33419 goroutine 91 [chan receive]:
2021-11-16_22:20:55.33422 main.main.func20(0x0, 0x0)
2021-11-16_22:20:55.33424       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:817 +0x850
2021-11-16_22:20:55.33426 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x4000161080, 0x4000473f50)
2021-11-16_22:20:55.33429       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28
2021-11-16_22:20:55.33431 created by github.com/oklog/run.(*Group).Run
2021-11-16_22:20:55.33433       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8
2021-11-16_22:20:55.33439 
2021-11-16_22:20:55.33442 goroutine 92 [select]:
2021-11-16_22:20:55.33444 github.com/prometheus/prometheus/web.(*Handler).Run(0x40006679e0, 0x29c28b0, 0x400043b840, 0x29b0aa0, 0x4000acb830, 0x0, 0x0, 0x0, 0x0)
2021-11-16_22:20:55.33451       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/web/web.go:588 +0x6e4
2021-11-16_22:20:55.33454 main.main.func22(0x0, 0x0)
2021-11-16_22:20:55.33456       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:832 +0x5c
2021-11-16_22:20:55.33458 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40001610c0, 0x400023ebf0)
2021-11-16_22:20:55.33461       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28
2021-11-16_22:20:55.33463 created by github.com/oklog/run.(*Group).Run
2021-11-16_22:20:55.33465       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8
2021-11-16_22:20:55.33467 
2021-11-16_22:20:55.33469 goroutine 93 [select]:
2021-11-16_22:20:55.33472 github.com/prometheus/prometheus/notifier.(*Manager).Run(0x400052d400, 0x400009cc60)
2021-11-16_22:20:55.33474       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/notifier/notifier.go:308 +0x90
2021-11-16_22:20:55.33476 main.main.func24(0x0, 0x0)
2021-11-16_22:20:55.33478       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:855 +0x68
2021-11-16_22:20:55.33481 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x4000473f80, 0x400023ec40)
2021-11-16_22:20:55.33483       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28
2021-11-16_22:20:55.33490 created by github.com/oklog/run.(*Group).Run
2021-11-16_22:20:55.33493       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8
2021-11-16_22:20:55.33495 
2021-11-16_22:20:55.33497 goroutine 257 [select]:
2021-11-16_22:20:55.33499 github.com/prometheus/prometheus/discovery.(*Manager).sender(0x40002555e0)
2021-11-16_22:20:55.33502       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:234 +0xb4
2021-11-16_22:20:55.33504 created by github.com/prometheus/prometheus/discovery.(*Manager).Run
2021-11-16_22:20:55.33506       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:141 +0x3c
2021-11-16_22:20:55.33509 
2021-11-16_22:20:55.33511 goroutine 344 [IO wait]:
2021-11-16_22:20:55.33513 internal/poll.runtime_pollWait(0x7f653c6098, 0x72, 0x0)
2021-11-16_22:20:55.33515       /usr/local/go/src/runtime/netpoll.go:222 +0x44
2021-11-16_22:20:55.33517 internal/poll.(*pollDesc).wait(0x400039d418, 0x72, 0x0, 0x0, 0x2200c12)
2021-11-16_22:20:55.33520       /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38
2021-11-16_22:20:55.33522 internal/poll.(*pollDesc).waitRead(...)
2021-11-16_22:20:55.33524       /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
2021-11-16_22:20:55.33526 internal/poll.(*FD).Accept(0x400039d400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2021-11-16_22:20:55.33529       /usr/local/go/src/internal/poll/fd_unix.go:401 +0x1b0
2021-11-16_22:20:55.33535 net.(*netFD).accept(0x400039d400, 0x4000078ba0, 0x0, 0x1)
2021-11-16_22:20:55.33796       /usr/local/go/src/net/fd_unix.go:172 +0x2c
2021-11-16_22:20:55.33807 net.(*TCPListener).accept(0x4000acb7d0, 0x1000000000001, 0x4000abfd40, 0x4000078ba0)
2021-11-16_22:20:55.33812       /usr/local/go/src/net/tcpsock_posix.go:139 +0x2c
2021-11-16_22:20:55.33816 net.(*TCPListener).Accept(0x4000acb7d0, 0x1e201, 0x4000bb2630, 0x30, 0x30)
2021-11-16_22:20:55.33820       /usr/local/go/src/net/tcpsock.go:261 +0x5c
2021-11-16_22:20:55.33824 golang.org/x/net/netutil.(*limitListener).Accept(0x4000ad97a0, 0x30, 0x30, 0x7f8f28df18, 0x30)
2021-11-16_22:20:55.33828       /var/cache/omnibus/src/prometheus/pkg/mod/golang.org/x/net@v0.0.0-20210119194325-5f4716e94777/netutil/listen.go:48 +0x48
2021-11-16_22:20:55.33832 github.com/mwitkow/go-conntrack.(*connTrackListener).Accept(0x4000acb830, 0x4000bb2630, 0x2670f00, 0x4000bb4080, 0x1d49240)
2021-11-16_22:20:55.33845       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/mwitkow/go-conntrack@v0.0.0-20190716064945-2f068394615f/listener_wrapper.go:100 +0x74
2021-11-16_22:20:55.33849 net/http.(*Server).Serve(0x4000bb4000, 0x29b0aa0, 0x4000acb830, 0x0, 0x0)
2021-11-16_22:20:55.33853       /usr/local/go/src/net/http/server.go:2981 +0x234
2021-11-16_22:20:55.33857 github.com/prometheus/exporter-toolkit/web.Serve(0x29b0aa0, 0x4000acb830, 0x4000bb4000, 0x0, 0x0, 0x29788d8, 0x4000a57170, 0x0, 0x0)
2021-11-16_22:20:55.34037       /var/cache/omnibus/src/prometheus/pkg/mod/github.com/prometheus/exporter-toolkit@v0.5.1/web/tls_config.go:192 +0x168
2021-11-16_22:20:55.34068 github.com/prometheus/prometheus/web.(*Handler).Run.func2(0x4000bac3c0, 0x29b0aa0, 0x4000acb830, 0x4000bb4000, 0x0, 0x0, 0x40006679e0)
2021-11-16_22:20:55.34074       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/web/web.go:585 +0x5c
2021-11-16_22:20:55.34078 created by github.com/prometheus/prometheus/web.(*Handler).Run
2021-11-16_22:20:55.34082       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/web/web.go:584 +0x684
2021-11-16_22:20:55.34086 
2021-11-16_22:20:55.34089 goroutine 368 [select]:
2021-11-16_22:20:55.34124 github.com/prometheus/prometheus/tsdb/wal.(*WAL).run(0x40001443f0)
2021-11-16_22:20:55.34129       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/wal/wal.go:332 +0x88
2021-11-16_22:20:55.34134 created by github.com/prometheus/prometheus/tsdb/wal.NewSize
2021-11-16_22:20:55.34137       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/wal/wal.go:301 +0x2b8
2021-11-16_22:20:55.34141 
2021-11-16_22:20:55.34145 goroutine 25145 [select]:
2021-11-16_22:20:55.34149 net/http.(*persistConn).writeLoop(0x400318fc20)
2021-11-16_22:20:55.34244       /usr/local/go/src/net/http/transport.go:2382 +0xa8
2021-11-16_22:20:55.34252 created by net/http.(*Transport).dialConn
2021-11-16_22:20:55.34256       /usr/local/go/src/net/http/transport.go:1744 +0x9bc
2021-11-16_22:20:55.34260 
2021-11-16_22:20:55.34472 goroutine 24742 [select]:
2021-11-16_22:20:55.34480 github.com/prometheus/prometheus/tsdb.(*DB).run(0x4000bb40e0)
2021-11-16_22:20:55.34484       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/db.go:748 +0x128
2021-11-16_22:20:55.34488 created by github.com/prometheus/prometheus/tsdb.open
2021-11-16_22:20:55.34492       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/db.go:698 +0x66c
2021-11-16_22:20:55.34496 
2021-11-16_22:20:55.34500 goroutine 25140 [select]:
2021-11-16_22:20:55.34504 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4002002210, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.34508       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8
2021-11-16_22:20:55.34512 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.34516       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
2021-11-16_22:20:55.34520 
2021-11-16_22:20:55.34524 goroutine 24785 [select]:
2021-11-16_22:20:55.34528 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4002476580, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.34532       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8
2021-11-16_22:20:55.34547 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.34551       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
2021-11-16_22:20:55.34555 
2021-11-16_22:20:55.34559 goroutine 24784 [select]:
2021-11-16_22:20:55.34563 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x40024764d0, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.34567       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8
2021-11-16_22:20:55.34571 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.34575       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
2021-11-16_22:20:55.34588 
2021-11-16_22:20:55.34593 goroutine 24781 [select]:
2021-11-16_22:20:55.34597 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x40024762c0, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.34601       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8
2021-11-16_22:20:55.34604 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.34608       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
2021-11-16_22:20:55.34620 
2021-11-16_22:20:55.34624 goroutine 25168 [select]:
2021-11-16_22:20:55.34628 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x40003116b0, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.34632       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8
2021-11-16_22:20:55.34636 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.34640       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
2021-11-16_22:20:55.34643 
2021-11-16_22:20:55.34647 goroutine 24780 [select]:
2021-11-16_22:20:55.34650 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4002476210, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.34654       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8
2021-11-16_22:20:55.34658 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.34662       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
2021-11-16_22:20:55.34666 
2021-11-16_22:20:55.34669 goroutine 25141 [select]:
2021-11-16_22:20:55.34673 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x40020022c0, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.34677       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8
2021-11-16_22:20:55.34688 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.34692       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
2021-11-16_22:20:55.34749 
2021-11-16_22:20:55.34755 goroutine 25106 [select]:
2021-11-16_22:20:55.34759 github.com/prometheus/prometheus/scrape.(*Manager).reloader(0x400015cd20)
2021-11-16_22:20:55.34763       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/manager.go:162 +0xb4
2021-11-16_22:20:55.34767 created by github.com/prometheus/prometheus/scrape.(*Manager).Run
2021-11-16_22:20:55.34771       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/manager.go:140 +0x3c
2021-11-16_22:20:55.34775 
2021-11-16_22:20:55.34779 goroutine 25085 [select]:
2021-11-16_22:20:55.34782 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f6c0, 0x29c28e8, 0x400018a000)
2021-11-16_22:20:55.34786       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110
2021-11-16_22:20:55.34790 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x400317a300, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f6c0)
2021-11-16_22:20:55.34795       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74
2021-11-16_22:20:55.34799 created by github.com/prometheus/prometheus/rules.(*Manager).Update
2021-11-16_22:20:55.34811       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460
2021-11-16_22:20:55.34815 
2021-11-16_22:20:55.34819 goroutine 25086 [select]:
2021-11-16_22:20:55.34823 github.com/prometheus/prometheus/rules.(*Group).run(0x40016256c0, 0x29c28e8, 0x400018a000)
2021-11-16_22:20:55.34827       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:385 +0x6ec
2021-11-16_22:20:55.34831 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x4001ed4000, 0x0, 0x4003186510, 0x40000518b0, 0x40016256c0)
2021-11-16_22:20:55.34842       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74
2021-11-16_22:20:55.34846 created by github.com/prometheus/prometheus/rules.(*Manager).Update
2021-11-16_22:20:55.34850       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460
2021-11-16_22:20:55.34858 
2021-11-16_22:20:55.34864 goroutine 25087 [select]:
2021-11-16_22:20:55.34867 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f180, 0x29c28e8, 0x400018a000)
2021-11-16_22:20:55.34871       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110
2021-11-16_22:20:55.34875 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x400317a300, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f180)
2021-11-16_22:20:55.34887       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74
2021-11-16_22:20:55.34892 created by github.com/prometheus/prometheus/rules.(*Manager).Update
2021-11-16_22:20:55.34895       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460
2021-11-16_22:20:55.34898 
2021-11-16_22:20:55.34903 goroutine 25088 [select]:
2021-11-16_22:20:55.34906 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f260, 0x29c28e8, 0x400018a000)
2021-11-16_22:20:55.34908       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110
2021-11-16_22:20:55.34911 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x40000a9e00, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f260)
2021-11-16_22:20:55.34914       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74
2021-11-16_22:20:55.34917 created by github.com/prometheus/prometheus/rules.(*Manager).Update
2021-11-16_22:20:55.34920       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460
2021-11-16_22:20:55.34923 
2021-11-16_22:20:55.34926 goroutine 25089 [select]:
2021-11-16_22:20:55.34929 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f340, 0x29c28e8, 0x400018a000)
2021-11-16_22:20:55.34937       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110
2021-11-16_22:20:55.34941 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x40000a9e00, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f340)
2021-11-16_22:20:55.34944       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74
2021-11-16_22:20:55.34947 created by github.com/prometheus/prometheus/rules.(*Manager).Update
2021-11-16_22:20:55.34949       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460
2021-11-16_22:20:55.34952 
2021-11-16_22:20:55.34955 goroutine 25090 [select]:
2021-11-16_22:20:55.34958 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f420, 0x29c28e8, 0x400018a000)
2021-11-16_22:20:55.34961       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110
2021-11-16_22:20:55.34964 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x40000a9f00, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f420)
2021-11-16_22:20:55.34967       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74
2021-11-16_22:20:55.34970 created by github.com/prometheus/prometheus/rules.(*Manager).Update
2021-11-16_22:20:55.34978       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460
2021-11-16_22:20:55.34982 
2021-11-16_22:20:55.34984 goroutine 25091 [select]:
2021-11-16_22:20:55.34987 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f500, 0x29c28e8, 0x400018a000)
2021-11-16_22:20:55.34990       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110
2021-11-16_22:20:55.34993 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x40000a9f00, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f500)
2021-11-16_22:20:55.35002       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74
2021-11-16_22:20:55.35005 created by github.com/prometheus/prometheus/rules.(*Manager).Update
2021-11-16_22:20:55.35008       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460
2021-11-16_22:20:55.35011 
2021-11-16_22:20:55.35014 goroutine 25092 [select]:
2021-11-16_22:20:55.35017 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f5e0, 0x29c28e8, 0x400018a000)
2021-11-16_22:20:55.35020       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110
2021-11-16_22:20:55.35023 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x4001ede000, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f5e0)
2021-11-16_22:20:55.35032       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74
2021-11-16_22:20:55.35035 created by github.com/prometheus/prometheus/rules.(*Manager).Update
2021-11-16_22:20:55.35038       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460
2021-11-16_22:20:55.35041 
2021-11-16_22:20:55.35044 goroutine 25169 [select]:
2021-11-16_22:20:55.35047 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4000311760, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.35050       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8
2021-11-16_22:20:55.35053 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.35056       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
2021-11-16_22:20:55.35059 
2021-11-16_22:20:55.35061 goroutine 25144 [IO wait]:
2021-11-16_22:20:55.35064 internal/poll.runtime_pollWait(0x7f653c5fb0, 0x72, 0xffffffffffffffff)
2021-11-16_22:20:55.35067       /usr/local/go/src/runtime/netpoll.go:222 +0x44
2021-11-16_22:20:55.35070 internal/poll.(*pollDesc).wait(0x4003de7598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
2021-11-16_22:20:55.35073       /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38
2021-11-16_22:20:55.35082 internal/poll.(*pollDesc).waitRead(...)
2021-11-16_22:20:55.35085       /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
2021-11-16_22:20:55.35088 internal/poll.(*FD).Read(0x4003de7580, 0x4002056000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
2021-11-16_22:20:55.35091       /usr/local/go/src/internal/poll/fd_unix.go:166 +0x180
2021-11-16_22:20:55.35094 net.(*netFD).Read(0x4003de7580, 0x4002056000, 0x1000, 0x1000, 0x0, 0x17e94, 0x4002214478)
2021-11-16_22:20:55.35097       /usr/local/go/src/net/fd_posix.go:55 +0x44
2021-11-16_22:20:55.35100 net.(*conn).Read(0x4000164668, 0x4002056000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
2021-11-16_22:20:55.35103       /usr/local/go/src/net/net.go:183 +0x74
2021-11-16_22:20:55.35106 net/http.(*persistConn).Read(0x400318fc20, 0x4002056000, 0x1000, 0x1000, 0x40005f5cf8, 0x2af0bc, 0x4000abee40)
2021-11-16_22:20:55.35109       /usr/local/go/src/net/http/transport.go:1922 +0x60
2021-11-16_22:20:55.35112 bufio.(*Reader).fill(0x40063ce900)
2021-11-16_22:20:55.35114       /usr/local/go/src/bufio/bufio.go:101 +0xf8
2021-11-16_22:20:55.35117 bufio.(*Reader).Peek(0x40063ce900, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3)
2021-11-16_22:20:55.35120       /usr/local/go/src/bufio/bufio.go:139 +0x40
2021-11-16_22:20:55.35123 net/http.(*persistConn).readLoop(0x400318fc20)
2021-11-16_22:20:55.35125       /usr/local/go/src/net/http/transport.go:2083 +0x170
2021-11-16_22:20:55.35128 created by net/http.(*Transport).dialConn
2021-11-16_22:20:55.35131       /usr/local/go/src/net/http/transport.go:1743 +0x99c
2021-11-16_22:20:55.35140 
2021-11-16_22:20:55.35143 goroutine 24782 [select]:
2021-11-16_22:20:55.35146 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4002476370, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.35149       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8
2021-11-16_22:20:55.35152 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.35160       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
2021-11-16_22:20:55.35164 
2021-11-16_22:20:55.35176 goroutine 25172 [select]:
2021-11-16_22:20:55.35180 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x400052ec60, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.35184       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8
2021-11-16_22:20:55.35187 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.35190       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
2021-11-16_22:20:55.35193 
2021-11-16_22:20:55.35195 goroutine 25173 [select]:
2021-11-16_22:20:55.35198 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x400052edc0, 0x37e11d600, 0x37e11d600, 0x0)
2021-11-16_22:20:55.35201       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8
2021-11-16_22:20:55.35211 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
2021-11-16_22:20:55.35217       /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790
[...]

The golang/snappy package pointed me directly to use golang/snappy v0.0.3 #8593, but there are other similar reports like prometheus: 2.22.0 failing on RPI4 #15236 and unexpected fault address on raspberry pi4 and openwrt #8589.

The solution

The fix in pull request #8593 was merged on 13th March and released a day later in version 2.25.1.

The solution

The solution is download at least Prometheus 2.25.1 and replace the binary file.

Download Prometheus release.

$ wget https://github.com/prometheus/prometheus/releases/download/v2.25.1/prometheus-2.25.1.linux-arm64.tar.gz

Inspect archive file.

$ tar tvfz prometheus-2.25.1.linux-arm64.tar.gz 
drwxr-xr-x circleci/circleci 0 2021-03-14 14:23 prometheus-2.25.1.linux-arm64/
drwxr-xr-x circleci/circleci 0 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/
-rw-r--r-- circleci/circleci 616 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/index.html.example
-rw-r--r-- circleci/circleci 2675 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/node-cpu.html
-rw-r--r-- circleci/circleci 3513 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/node-disk.html
-rw-r--r-- circleci/circleci 5794 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/node-overview.html
-rw-r--r-- circleci/circleci 1444 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/node.html
-rw-r--r-- circleci/circleci 4103 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/prometheus-overview.html
-rw-r--r-- circleci/circleci 1334 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/prometheus.html
drwxr-xr-x circleci/circleci    0 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/console_libraries/
-rw-r--r-- circleci/circleci 2888 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/console_libraries/menu.lib
-rw-r--r-- circleci/circleci 6152 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/console_libraries/prom.lib
-rw-r--r-- circleci/circleci  926 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/prometheus.yml
-rw-r--r-- circleci/circleci 11357 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/LICENSE
-rw-r--r-- circleci/circleci  3420 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/NOTICE
-rwxr-xr-x circleci/circleci 85921133 2021-03-14 13:32 prometheus-2.25.1.linux-arm64/prometheus
-rwxr-xr-x circleci/circleci 76259866 2021-03-14 13:34 prometheus-2.25.1.linux-arm64/promtool

Extract binary.

$ tar xvfz prometheus-2.25.1.linux-arm64.tar.gz prometheus-2.25.1.linux-arm64/prometheus

Stop Prometheus service.

$ sudo gitlab-ctl stop prometheus
ok: down: prometheus: 1s, normally up

Move original binary file.

$ mv /opt/gitlab/embedded/bin/{prometheus,prometheus.orig}

Copy the replacement.

$ sudo cp prometheus-2.25.1.linux-arm64/prometheus /opt/gitlab/embedded/bin/

Start service.

$ sudo gitlab-ctl start prometheus
ok: run: prometheus: (pid 190553) 0s

Check service status.

$ sudo gitlab-ctl status prometheus
run: prometheus: (pid 190553) 31s; run: log: (pid 12953) 81946s

Inspect service logs.

$ sudo gitlab-ctl tail prometheus
[...]
2021-11-16_22:43:14.47308 level=info ts=2021-11-16T22:43:14.472Z caller=main.go:751 msg="Server is ready to receive web requests."
[...]

Check out Grafana graphs.

GitLab bug report #345859.