Fix broken Prometheus service in GitLab arm64 14.4.2.
The issue
Prometheus in GitLab arm64 14.4.2 is not working after installation.
Inspect GitLab status. The Prometheus service is running, which is a misleading as it does this for several seconds and simply restarts.
$ sudo gitlab-ctl status
run: alertmanager: (pid 13795) 80007s; run: log: (pid 12997) 80256s run: gitaly: (pid 13918) 80000s; run: log: (pid 12432) 80542s run: gitlab-exporter: (pid 13761) 80013s; run: log: (pid 12902) 80289s run: gitlab-workhorse: (pid 13739) 80014s; run: log: (pid 12821) 80318s run: grafana: (pid 13813) 80006s; run: log: (pid 13539) 80083s run: logrotate: (pid 184838) 1366s; run: log: (pid 12359) 80565s run: nginx: (pid 12832) 80314s; run: log: (pid 12845) 80311s run: node-exporter: (pid 13753) 80013s; run: log: (pid 12884) 80300s run: postgres-exporter: (pid 13805) 80007s; run: log: (pid 13028) 80244s run: postgresql: (pid 12490) 80529s; run: log: (pid 12578) 80526s run: prometheus: (pid 187452) 6s; run: log: (pid 12953) 80267s run: puma: (pid 23295) 76950s; run: log: (pid 12764) 80339s run: redis: (pid 12387) 80556s; run: log: (pid 12396) 80555s run: redis-exporter: (pid 13763) 80012s; run: log: (pid 12930) 80278s run: sidekiq: (pid 23268) 76958s; run: log: (pid 12781) 80330s
Inspect Prometheus version.
$ /opt/gitlab/embedded/bin/prometheus --version
prometheus, version 2.25.0 (branch: master, revision: ) build user: GitLab-Omnibus build date: go version: go1.16.4 platform: linux/arm64
Inspect Prometheus logs.
$ sudo gitlab-ctl tail prometheus
[...] 2021-11-16_22:20:55.31546 unexpected fault address 0x75625f736484fb 2021-11-16_22:20:55.31565 fatal error: fault 2021-11-16_22:20:55.31797 [signal SIGSEGV: segmentation violation code=0x1 addr=0x75625f736484fb pc=0x187ae90] 2021-11-16_22:20:55.31805 2021-11-16_22:20:55.31810 goroutine 24783 [running]: 2021-11-16_22:20:55.32110 runtime.throw(0x21fdccc, 0x5) 2021-11-16_22:20:55.32127 /usr/local/go/src/runtime/panic.go:1117 +0x54 fp=0x4002827560 sp=0x4002827530 pc=0x45f94 2021-11-16_22:20:55.32132 runtime.sigpanic() 2021-11-16_22:20:55.32137 /usr/local/go/src/runtime/signal_unix.go:741 +0x230 fp=0x40028275a0 sp=0x4002827560 pc=0x5da70 2021-11-16_22:20:55.32141 github.com/golang/snappy.encodeBlock(0x40027d4003, 0x5dfe, 0x5dfe, 0x4002800000, 0x5078, 0x6a80, 0x3) 2021-11-16_22:20:55.32146 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/golang/snappy@v0.0.2/encode_arm64.s:666 +0x360 fp=0x400282f640 sp=0x40028275b0 pc=0x187ae90 2021-11-16_22:20:55.32150 github.com/golang/snappy.Encode(0x40027d4000, 0x5e01, 0x5e01, 0x0, 0x0, 0x0, 0x5078, 0x400269e900, 0x4000178500) 2021-11-16_22:20:55.32154 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/golang/snappy@v0.0.2/encode.go:39 +0x17c fp=0x400282f690 sp=0x400282f640 pc=0x187a3cc 2021-11-16_22:20:55.32159 github.com/prometheus/prometheus/tsdb/wal.(*WAL).log(0x40001443f0, 0x4002800000, 0x5078, 0x6a80, 0x1, 0x8c, 0x400269e94c) 2021-11-16_22:20:55.32163 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/wal/wal.go:634 +0x368 fp=0x400282f730 sp=0x400282f690 pc=0x18819a8 2021-11-16_22:20:55.32168 github.com/prometheus/prometheus/tsdb/wal.(*WAL).Log(0x40001443f0, 0x400282f838, 0x1, 0x1, 0x0, 0x0) 2021-11-16_22:20:55.32180 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/wal/wal.go:596 +0xc8 fp=0x400282f7c0 sp=0x400282f730 pc=0x1881588 2021-11-16_22:20:55.32194 github.com/prometheus/prometheus/tsdb.(*headAppender).log(0x40061c1680, 0x0, 0x0) 2021-11-16_22:20:55.32199 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/head.go:1259 +0x26c fp=0x400282f870 sp=0x400282f7c0 pc=0x18a2e0c 2021-11-16_22:20:55.32203 github.com/prometheus/prometheus/tsdb.(*headAppender).Commit(0x40061c1680, 0x0, 0x0) 2021-11-16_22:20:55.32207 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/head.go:1279 +0x74 fp=0x400282f9b0 sp=0x400282f870 pc=0x18a2f64 2021-11-16_22:20:55.32211 github.com/prometheus/prometheus/tsdb.dbAppender.Commit(0x29c3560, 0x40061c1680, 0x4000bb40e0, 0x3, 0x256065fde24b4dba) 2021-11-16_22:20:55.32215 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/db.go:794 +0x30 fp=0x400282f9f0 sp=0x400282f9b0 pc=0x1894390 2021-11-16_22:20:55.32299 github.com/prometheus/prometheus/tsdb.(*dbAppender).Commit(0x40028f8d38, 0x4002474940, 0x4002474940) 2021-11-16_22:20:55.32306 <autogenerated>:1 +0x48 fp=0x400282fa30 sp=0x400282f9f0 pc=0x18be9f8 2021-11-16_22:20:55.32318 github.com/prometheus/prometheus/storage.(*fanoutAppender).Commit(0x4002474b80, 0x29c2fb0, 0x40028f8d50) 2021-11-16_22:20:55.32322 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/storage/fanout.go:174 +0x34 fp=0x400282fae0 sp=0x400282fa30 pc=0x18371e4 2021-11-16_22:20:55.32327 github.com/prometheus/prometheus/scrape.(*timeLimitAppender).Commit(0x40028f8d50, 0x3a9, 0x3a9) 2021-11-16_22:20:55.32330 <autogenerated>:1 +0x3c fp=0x400282fb10 sp=0x400282fae0 pc=0x1947f6c 2021-11-16_22:20:55.32334 github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport.func1(0x400282fce0, 0x400224bcf0, 0x4002476420) 2021-11-16_22:20:55.32338 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:1086 +0x40 fp=0x400282fb90 sp=0x400282fb10 pc=0x19465a0 2021-11-16_22:20:55.32342 github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport(0x4002476420, 0x37e11d600, 0x37e11d600, 0x0, 0x0, 0x0, 0xc05d2991cee2727f, 0x5606599e2, 0x3af9040, 0x0, ...) 2021-11-16_22:20:55.32347 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:1153 +0x7a4 fp=0x400282fe00 sp=0x400282fb90 pc=0x193f984 2021-11-16_22:20:55.32359 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4002476420, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.32363 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:1039 +0x268 fp=0x400282ffb0 sp=0x400282fe00 pc=0x193eec8 2021-11-16_22:20:55.32367 runtime.goexit() 2021-11-16_22:20:55.32371 /usr/local/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x400282ffb0 sp=0x400282ffb0 pc=0x7d3c4 2021-11-16_22:20:55.32375 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.32378 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 2021-11-16_22:20:55.32382 2021-11-16_22:20:55.32386 goroutine 1 [chan receive]: 2021-11-16_22:20:55.32390 github.com/oklog/run.(*Group).Run(0x4000b83b18, 0x400023ec40, 0x8) 2021-11-16_22:20:55.32393 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:43 +0xd4 2021-11-16_22:20:55.32397 main.main() 2021-11-16_22:20:55.32402 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:864 +0x6308 2021-11-16_22:20:55.32407 2021-11-16_22:20:55.32411 goroutine 45 [select]: 2021-11-16_22:20:55.32414 go.opencensus.io/stats/view.(*worker).start(0x400052c280) 2021-11-16_22:20:55.32418 /var/cache/omnibus/src/prometheus/pkg/mod/go.opencensus.io@v0.22.5/stats/view/worker.go:276 +0x98 2021-11-16_22:20:55.32430 created by go.opencensus.io/stats/view.init.0 2021-11-16_22:20:55.32434 /var/cache/omnibus/src/prometheus/pkg/mod/go.opencensus.io@v0.22.5/stats/view/worker.go:34 +0x70 2021-11-16_22:20:55.32438 2021-11-16_22:20:55.32441 goroutine 64 [select]: 2021-11-16_22:20:55.32445 github.com/prometheus/prometheus/pkg/logging.(*Deduper).run(0x400043b740) 2021-11-16_22:20:55.32461 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/pkg/logging/dedupe.go:75 +0x160 2021-11-16_22:20:55.32466 created by github.com/prometheus/prometheus/pkg/logging.Dedupe 2021-11-16_22:20:55.32470 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/pkg/logging/dedupe.go:61 +0xcc 2021-11-16_22:20:55.32474 2021-11-16_22:20:55.32478 goroutine 103 [chan receive]: 2021-11-16_22:20:55.32481 github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0x4000255220) 2021-11-16_22:20:55.32485 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/storage/remote/write.go:96 +0x98 2021-11-16_22:20:55.32489 created by github.com/prometheus/prometheus/storage/remote.NewWriteStorage 2021-11-16_22:20:55.32493 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/storage/remote/write.go:89 +0x288 2021-11-16_22:20:55.32497 2021-11-16_22:20:55.32501 goroutine 83 [syscall]: 2021-11-16_22:20:55.32504 os/signal.signal_recv(0x0) 2021-11-16_22:20:55.32508 /usr/local/go/src/runtime/sigqueue.go:168 +0xcc 2021-11-16_22:20:55.32520 os/signal.loop() 2021-11-16_22:20:55.32525 /usr/local/go/src/os/signal/signal_unix.go:23 +0x20 2021-11-16_22:20:55.33213 created by os/signal.Notify.func1.1 2021-11-16_22:20:55.33216 /usr/local/go/src/os/signal/signal.go:151 +0x44 2021-11-16_22:20:55.33218 2021-11-16_22:20:55.33220 goroutine 84 [select]: 2021-11-16_22:20:55.33223 main.main.func6(0x6, 0x8) 2021-11-16_22:20:55.33225 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:627 +0x84 2021-11-16_22:20:55.33227 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40001a5b00, 0x40006c8230) 2021-11-16_22:20:55.33230 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28 2021-11-16_22:20:55.33232 created by github.com/oklog/run.(*Group).Run 2021-11-16_22:20:55.33234 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8 2021-11-16_22:20:55.33236 2021-11-16_22:20:55.33239 goroutine 85 [chan receive]: 2021-11-16_22:20:55.33241 github.com/prometheus/prometheus/discovery.(*Manager).Run(0x40002555e0, 0x18600000007, 0x0) 2021-11-16_22:20:55.33243 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:142 +0x68 2021-11-16_22:20:55.33246 main.main.func8(0x6, 0x8) 2021-11-16_22:20:55.33248 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:647 +0x38 2021-11-16_22:20:55.33250 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40006a8180, 0x40006a81a0) 2021-11-16_22:20:55.33262 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28 2021-11-16_22:20:55.33264 created by github.com/oklog/run.(*Group).Run 2021-11-16_22:20:55.33267 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8 2021-11-16_22:20:55.33269 2021-11-16_22:20:55.33271 goroutine 86 [chan receive]: 2021-11-16_22:20:55.33274 github.com/prometheus/prometheus/discovery.(*Manager).Run(0x4000255860, 0x16600000007, 0x0) 2021-11-16_22:20:55.33276 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:142 +0x68 2021-11-16_22:20:55.33278 main.main.func10(0x6, 0x8) 2021-11-16_22:20:55.33281 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:661 +0x38 2021-11-16_22:20:55.33283 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40006a81e0, 0x40006a8200) 2021-11-16_22:20:55.33285 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28 2021-11-16_22:20:55.33288 created by github.com/oklog/run.(*Group).Run 2021-11-16_22:20:55.33290 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8 2021-11-16_22:20:55.33292 2021-11-16_22:20:55.33294 goroutine 87 [select]: 2021-11-16_22:20:55.33296 github.com/prometheus/prometheus/scrape.(*Manager).Run(0x400015cd20, 0x400009cba0, 0x0, 0x14600000007) 2021-11-16_22:20:55.33310 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/manager.go:142 +0x8c 2021-11-16_22:20:55.33312 main.main.func12(0x6, 0x8) 2021-11-16_22:20:55.33315 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:681 +0x68 2021-11-16_22:20:55.33317 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x4000169200, 0x40006a8220) 2021-11-16_22:20:55.33319 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28 2021-11-16_22:20:55.33322 created by github.com/oklog/run.(*Group).Run 2021-11-16_22:20:55.33324 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8 2021-11-16_22:20:55.33326 2021-11-16_22:20:55.33328 goroutine 88 [select]: 2021-11-16_22:20:55.33330 main.main.func14(0x6, 0x8) 2021-11-16_22:20:55.33333 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:706 +0xd0 2021-11-16_22:20:55.33335 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40005ae360, 0x400023eb00) 2021-11-16_22:20:55.33337 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28 2021-11-16_22:20:55.33340 created by github.com/oklog/run.(*Group).Run 2021-11-16_22:20:55.33342 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8 2021-11-16_22:20:55.33357 2021-11-16_22:20:55.33359 goroutine 89 [chan receive]: 2021-11-16_22:20:55.33362 main.main.func16(0x6, 0x8) 2021-11-16_22:20:55.33364 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:752 +0x2b4 2021-11-16_22:20:55.33366 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40005ae3c0, 0x400023eb50) 2021-11-16_22:20:55.33368 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28 2021-11-16_22:20:55.33371 created by github.com/oklog/run.(*Group).Run 2021-11-16_22:20:55.33373 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8 2021-11-16_22:20:55.33375 2021-11-16_22:20:55.33378 goroutine 90 [chan receive]: 2021-11-16_22:20:55.33380 github.com/prometheus/prometheus/rules.(*Manager).Run(...) 2021-11-16_22:20:55.33383 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:912 2021-11-16_22:20:55.33385 main.main.func18(0x0, 0x0) 2021-11-16_22:20:55.33387 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:765 +0x64 2021-11-16_22:20:55.33390 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40001982a0, 0x400023eba0) 2021-11-16_22:20:55.33392 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28 2021-11-16_22:20:55.33394 created by github.com/oklog/run.(*Group).Run 2021-11-16_22:20:55.33396 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8 2021-11-16_22:20:55.33403 2021-11-16_22:20:55.33406 goroutine 256 [select]: 2021-11-16_22:20:55.33408 github.com/prometheus/prometheus/discovery.(*Manager).sender(0x4000255860) 2021-11-16_22:20:55.33410 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:234 +0xb4 2021-11-16_22:20:55.33413 created by github.com/prometheus/prometheus/discovery.(*Manager).Run 2021-11-16_22:20:55.33415 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:141 +0x3c 2021-11-16_22:20:55.33417 2021-11-16_22:20:55.33419 goroutine 91 [chan receive]: 2021-11-16_22:20:55.33422 main.main.func20(0x0, 0x0) 2021-11-16_22:20:55.33424 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:817 +0x850 2021-11-16_22:20:55.33426 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x4000161080, 0x4000473f50) 2021-11-16_22:20:55.33429 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28 2021-11-16_22:20:55.33431 created by github.com/oklog/run.(*Group).Run 2021-11-16_22:20:55.33433 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8 2021-11-16_22:20:55.33439 2021-11-16_22:20:55.33442 goroutine 92 [select]: 2021-11-16_22:20:55.33444 github.com/prometheus/prometheus/web.(*Handler).Run(0x40006679e0, 0x29c28b0, 0x400043b840, 0x29b0aa0, 0x4000acb830, 0x0, 0x0, 0x0, 0x0) 2021-11-16_22:20:55.33451 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/web/web.go:588 +0x6e4 2021-11-16_22:20:55.33454 main.main.func22(0x0, 0x0) 2021-11-16_22:20:55.33456 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:832 +0x5c 2021-11-16_22:20:55.33458 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x40001610c0, 0x400023ebf0) 2021-11-16_22:20:55.33461 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28 2021-11-16_22:20:55.33463 created by github.com/oklog/run.(*Group).Run 2021-11-16_22:20:55.33465 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8 2021-11-16_22:20:55.33467 2021-11-16_22:20:55.33469 goroutine 93 [select]: 2021-11-16_22:20:55.33472 github.com/prometheus/prometheus/notifier.(*Manager).Run(0x400052d400, 0x400009cc60) 2021-11-16_22:20:55.33474 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/notifier/notifier.go:308 +0x90 2021-11-16_22:20:55.33476 main.main.func24(0x0, 0x0) 2021-11-16_22:20:55.33478 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:855 +0x68 2021-11-16_22:20:55.33481 github.com/oklog/run.(*Group).Run.func1(0x40005ae4e0, 0x4000473f80, 0x400023ec40) 2021-11-16_22:20:55.33483 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x28 2021-11-16_22:20:55.33490 created by github.com/oklog/run.(*Group).Run 2021-11-16_22:20:55.33493 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xa8 2021-11-16_22:20:55.33495 2021-11-16_22:20:55.33497 goroutine 257 [select]: 2021-11-16_22:20:55.33499 github.com/prometheus/prometheus/discovery.(*Manager).sender(0x40002555e0) 2021-11-16_22:20:55.33502 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:234 +0xb4 2021-11-16_22:20:55.33504 created by github.com/prometheus/prometheus/discovery.(*Manager).Run 2021-11-16_22:20:55.33506 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/discovery/manager.go:141 +0x3c 2021-11-16_22:20:55.33509 2021-11-16_22:20:55.33511 goroutine 344 [IO wait]: 2021-11-16_22:20:55.33513 internal/poll.runtime_pollWait(0x7f653c6098, 0x72, 0x0) 2021-11-16_22:20:55.33515 /usr/local/go/src/runtime/netpoll.go:222 +0x44 2021-11-16_22:20:55.33517 internal/poll.(*pollDesc).wait(0x400039d418, 0x72, 0x0, 0x0, 0x2200c12) 2021-11-16_22:20:55.33520 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 2021-11-16_22:20:55.33522 internal/poll.(*pollDesc).waitRead(...) 2021-11-16_22:20:55.33524 /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 2021-11-16_22:20:55.33526 internal/poll.(*FD).Accept(0x400039d400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) 2021-11-16_22:20:55.33529 /usr/local/go/src/internal/poll/fd_unix.go:401 +0x1b0 2021-11-16_22:20:55.33535 net.(*netFD).accept(0x400039d400, 0x4000078ba0, 0x0, 0x1) 2021-11-16_22:20:55.33796 /usr/local/go/src/net/fd_unix.go:172 +0x2c 2021-11-16_22:20:55.33807 net.(*TCPListener).accept(0x4000acb7d0, 0x1000000000001, 0x4000abfd40, 0x4000078ba0) 2021-11-16_22:20:55.33812 /usr/local/go/src/net/tcpsock_posix.go:139 +0x2c 2021-11-16_22:20:55.33816 net.(*TCPListener).Accept(0x4000acb7d0, 0x1e201, 0x4000bb2630, 0x30, 0x30) 2021-11-16_22:20:55.33820 /usr/local/go/src/net/tcpsock.go:261 +0x5c 2021-11-16_22:20:55.33824 golang.org/x/net/netutil.(*limitListener).Accept(0x4000ad97a0, 0x30, 0x30, 0x7f8f28df18, 0x30) 2021-11-16_22:20:55.33828 /var/cache/omnibus/src/prometheus/pkg/mod/golang.org/x/net@v0.0.0-20210119194325-5f4716e94777/netutil/listen.go:48 +0x48 2021-11-16_22:20:55.33832 github.com/mwitkow/go-conntrack.(*connTrackListener).Accept(0x4000acb830, 0x4000bb2630, 0x2670f00, 0x4000bb4080, 0x1d49240) 2021-11-16_22:20:55.33845 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/mwitkow/go-conntrack@v0.0.0-20190716064945-2f068394615f/listener_wrapper.go:100 +0x74 2021-11-16_22:20:55.33849 net/http.(*Server).Serve(0x4000bb4000, 0x29b0aa0, 0x4000acb830, 0x0, 0x0) 2021-11-16_22:20:55.33853 /usr/local/go/src/net/http/server.go:2981 +0x234 2021-11-16_22:20:55.33857 github.com/prometheus/exporter-toolkit/web.Serve(0x29b0aa0, 0x4000acb830, 0x4000bb4000, 0x0, 0x0, 0x29788d8, 0x4000a57170, 0x0, 0x0) 2021-11-16_22:20:55.34037 /var/cache/omnibus/src/prometheus/pkg/mod/github.com/prometheus/exporter-toolkit@v0.5.1/web/tls_config.go:192 +0x168 2021-11-16_22:20:55.34068 github.com/prometheus/prometheus/web.(*Handler).Run.func2(0x4000bac3c0, 0x29b0aa0, 0x4000acb830, 0x4000bb4000, 0x0, 0x0, 0x40006679e0) 2021-11-16_22:20:55.34074 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/web/web.go:585 +0x5c 2021-11-16_22:20:55.34078 created by github.com/prometheus/prometheus/web.(*Handler).Run 2021-11-16_22:20:55.34082 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/web/web.go:584 +0x684 2021-11-16_22:20:55.34086 2021-11-16_22:20:55.34089 goroutine 368 [select]: 2021-11-16_22:20:55.34124 github.com/prometheus/prometheus/tsdb/wal.(*WAL).run(0x40001443f0) 2021-11-16_22:20:55.34129 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/wal/wal.go:332 +0x88 2021-11-16_22:20:55.34134 created by github.com/prometheus/prometheus/tsdb/wal.NewSize 2021-11-16_22:20:55.34137 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/wal/wal.go:301 +0x2b8 2021-11-16_22:20:55.34141 2021-11-16_22:20:55.34145 goroutine 25145 [select]: 2021-11-16_22:20:55.34149 net/http.(*persistConn).writeLoop(0x400318fc20) 2021-11-16_22:20:55.34244 /usr/local/go/src/net/http/transport.go:2382 +0xa8 2021-11-16_22:20:55.34252 created by net/http.(*Transport).dialConn 2021-11-16_22:20:55.34256 /usr/local/go/src/net/http/transport.go:1744 +0x9bc 2021-11-16_22:20:55.34260 2021-11-16_22:20:55.34472 goroutine 24742 [select]: 2021-11-16_22:20:55.34480 github.com/prometheus/prometheus/tsdb.(*DB).run(0x4000bb40e0) 2021-11-16_22:20:55.34484 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/db.go:748 +0x128 2021-11-16_22:20:55.34488 created by github.com/prometheus/prometheus/tsdb.open 2021-11-16_22:20:55.34492 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/tsdb/db.go:698 +0x66c 2021-11-16_22:20:55.34496 2021-11-16_22:20:55.34500 goroutine 25140 [select]: 2021-11-16_22:20:55.34504 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4002002210, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.34508 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8 2021-11-16_22:20:55.34512 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.34516 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 2021-11-16_22:20:55.34520 2021-11-16_22:20:55.34524 goroutine 24785 [select]: 2021-11-16_22:20:55.34528 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4002476580, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.34532 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8 2021-11-16_22:20:55.34547 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.34551 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 2021-11-16_22:20:55.34555 2021-11-16_22:20:55.34559 goroutine 24784 [select]: 2021-11-16_22:20:55.34563 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x40024764d0, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.34567 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8 2021-11-16_22:20:55.34571 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.34575 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 2021-11-16_22:20:55.34588 2021-11-16_22:20:55.34593 goroutine 24781 [select]: 2021-11-16_22:20:55.34597 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x40024762c0, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.34601 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8 2021-11-16_22:20:55.34604 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.34608 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 2021-11-16_22:20:55.34620 2021-11-16_22:20:55.34624 goroutine 25168 [select]: 2021-11-16_22:20:55.34628 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x40003116b0, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.34632 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8 2021-11-16_22:20:55.34636 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.34640 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 2021-11-16_22:20:55.34643 2021-11-16_22:20:55.34647 goroutine 24780 [select]: 2021-11-16_22:20:55.34650 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4002476210, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.34654 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8 2021-11-16_22:20:55.34658 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.34662 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 2021-11-16_22:20:55.34666 2021-11-16_22:20:55.34669 goroutine 25141 [select]: 2021-11-16_22:20:55.34673 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x40020022c0, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.34677 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8 2021-11-16_22:20:55.34688 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.34692 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 2021-11-16_22:20:55.34749 2021-11-16_22:20:55.34755 goroutine 25106 [select]: 2021-11-16_22:20:55.34759 github.com/prometheus/prometheus/scrape.(*Manager).reloader(0x400015cd20) 2021-11-16_22:20:55.34763 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/manager.go:162 +0xb4 2021-11-16_22:20:55.34767 created by github.com/prometheus/prometheus/scrape.(*Manager).Run 2021-11-16_22:20:55.34771 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/manager.go:140 +0x3c 2021-11-16_22:20:55.34775 2021-11-16_22:20:55.34779 goroutine 25085 [select]: 2021-11-16_22:20:55.34782 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f6c0, 0x29c28e8, 0x400018a000) 2021-11-16_22:20:55.34786 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110 2021-11-16_22:20:55.34790 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x400317a300, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f6c0) 2021-11-16_22:20:55.34795 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74 2021-11-16_22:20:55.34799 created by github.com/prometheus/prometheus/rules.(*Manager).Update 2021-11-16_22:20:55.34811 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460 2021-11-16_22:20:55.34815 2021-11-16_22:20:55.34819 goroutine 25086 [select]: 2021-11-16_22:20:55.34823 github.com/prometheus/prometheus/rules.(*Group).run(0x40016256c0, 0x29c28e8, 0x400018a000) 2021-11-16_22:20:55.34827 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:385 +0x6ec 2021-11-16_22:20:55.34831 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x4001ed4000, 0x0, 0x4003186510, 0x40000518b0, 0x40016256c0) 2021-11-16_22:20:55.34842 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74 2021-11-16_22:20:55.34846 created by github.com/prometheus/prometheus/rules.(*Manager).Update 2021-11-16_22:20:55.34850 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460 2021-11-16_22:20:55.34858 2021-11-16_22:20:55.34864 goroutine 25087 [select]: 2021-11-16_22:20:55.34867 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f180, 0x29c28e8, 0x400018a000) 2021-11-16_22:20:55.34871 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110 2021-11-16_22:20:55.34875 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x400317a300, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f180) 2021-11-16_22:20:55.34887 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74 2021-11-16_22:20:55.34892 created by github.com/prometheus/prometheus/rules.(*Manager).Update 2021-11-16_22:20:55.34895 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460 2021-11-16_22:20:55.34898 2021-11-16_22:20:55.34903 goroutine 25088 [select]: 2021-11-16_22:20:55.34906 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f260, 0x29c28e8, 0x400018a000) 2021-11-16_22:20:55.34908 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110 2021-11-16_22:20:55.34911 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x40000a9e00, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f260) 2021-11-16_22:20:55.34914 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74 2021-11-16_22:20:55.34917 created by github.com/prometheus/prometheus/rules.(*Manager).Update 2021-11-16_22:20:55.34920 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460 2021-11-16_22:20:55.34923 2021-11-16_22:20:55.34926 goroutine 25089 [select]: 2021-11-16_22:20:55.34929 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f340, 0x29c28e8, 0x400018a000) 2021-11-16_22:20:55.34937 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110 2021-11-16_22:20:55.34941 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x40000a9e00, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f340) 2021-11-16_22:20:55.34944 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74 2021-11-16_22:20:55.34947 created by github.com/prometheus/prometheus/rules.(*Manager).Update 2021-11-16_22:20:55.34949 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460 2021-11-16_22:20:55.34952 2021-11-16_22:20:55.34955 goroutine 25090 [select]: 2021-11-16_22:20:55.34958 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f420, 0x29c28e8, 0x400018a000) 2021-11-16_22:20:55.34961 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110 2021-11-16_22:20:55.34964 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x40000a9f00, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f420) 2021-11-16_22:20:55.34967 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74 2021-11-16_22:20:55.34970 created by github.com/prometheus/prometheus/rules.(*Manager).Update 2021-11-16_22:20:55.34978 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460 2021-11-16_22:20:55.34982 2021-11-16_22:20:55.34984 goroutine 25091 [select]: 2021-11-16_22:20:55.34987 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f500, 0x29c28e8, 0x400018a000) 2021-11-16_22:20:55.34990 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110 2021-11-16_22:20:55.34993 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x40000a9f00, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f500) 2021-11-16_22:20:55.35002 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74 2021-11-16_22:20:55.35005 created by github.com/prometheus/prometheus/rules.(*Manager).Update 2021-11-16_22:20:55.35008 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460 2021-11-16_22:20:55.35011 2021-11-16_22:20:55.35014 goroutine 25092 [select]: 2021-11-16_22:20:55.35017 github.com/prometheus/prometheus/rules.(*Group).run(0x4000a3f5e0, 0x29c28e8, 0x400018a000) 2021-11-16_22:20:55.35020 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:325 +0x110 2021-11-16_22:20:55.35023 github.com/prometheus/prometheus/rules.(*Manager).Update.func1(0x4001ede000, 0x0, 0x4003186510, 0x40000518b0, 0x4000a3f5e0) 2021-11-16_22:20:55.35032 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:978 +0x74 2021-11-16_22:20:55.35035 created by github.com/prometheus/prometheus/rules.(*Manager).Update 2021-11-16_22:20:55.35038 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/rules/manager.go:968 +0x460 2021-11-16_22:20:55.35041 2021-11-16_22:20:55.35044 goroutine 25169 [select]: 2021-11-16_22:20:55.35047 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4000311760, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.35050 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8 2021-11-16_22:20:55.35053 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.35056 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 2021-11-16_22:20:55.35059 2021-11-16_22:20:55.35061 goroutine 25144 [IO wait]: 2021-11-16_22:20:55.35064 internal/poll.runtime_pollWait(0x7f653c5fb0, 0x72, 0xffffffffffffffff) 2021-11-16_22:20:55.35067 /usr/local/go/src/runtime/netpoll.go:222 +0x44 2021-11-16_22:20:55.35070 internal/poll.(*pollDesc).wait(0x4003de7598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) 2021-11-16_22:20:55.35073 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 2021-11-16_22:20:55.35082 internal/poll.(*pollDesc).waitRead(...) 2021-11-16_22:20:55.35085 /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 2021-11-16_22:20:55.35088 internal/poll.(*FD).Read(0x4003de7580, 0x4002056000, 0x1000, 0x1000, 0x0, 0x0, 0x0) 2021-11-16_22:20:55.35091 /usr/local/go/src/internal/poll/fd_unix.go:166 +0x180 2021-11-16_22:20:55.35094 net.(*netFD).Read(0x4003de7580, 0x4002056000, 0x1000, 0x1000, 0x0, 0x17e94, 0x4002214478) 2021-11-16_22:20:55.35097 /usr/local/go/src/net/fd_posix.go:55 +0x44 2021-11-16_22:20:55.35100 net.(*conn).Read(0x4000164668, 0x4002056000, 0x1000, 0x1000, 0x0, 0x0, 0x0) 2021-11-16_22:20:55.35103 /usr/local/go/src/net/net.go:183 +0x74 2021-11-16_22:20:55.35106 net/http.(*persistConn).Read(0x400318fc20, 0x4002056000, 0x1000, 0x1000, 0x40005f5cf8, 0x2af0bc, 0x4000abee40) 2021-11-16_22:20:55.35109 /usr/local/go/src/net/http/transport.go:1922 +0x60 2021-11-16_22:20:55.35112 bufio.(*Reader).fill(0x40063ce900) 2021-11-16_22:20:55.35114 /usr/local/go/src/bufio/bufio.go:101 +0xf8 2021-11-16_22:20:55.35117 bufio.(*Reader).Peek(0x40063ce900, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) 2021-11-16_22:20:55.35120 /usr/local/go/src/bufio/bufio.go:139 +0x40 2021-11-16_22:20:55.35123 net/http.(*persistConn).readLoop(0x400318fc20) 2021-11-16_22:20:55.35125 /usr/local/go/src/net/http/transport.go:2083 +0x170 2021-11-16_22:20:55.35128 created by net/http.(*Transport).dialConn 2021-11-16_22:20:55.35131 /usr/local/go/src/net/http/transport.go:1743 +0x99c 2021-11-16_22:20:55.35140 2021-11-16_22:20:55.35143 goroutine 24782 [select]: 2021-11-16_22:20:55.35146 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x4002476370, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.35149 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8 2021-11-16_22:20:55.35152 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.35160 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 2021-11-16_22:20:55.35164 2021-11-16_22:20:55.35176 goroutine 25172 [select]: 2021-11-16_22:20:55.35180 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x400052ec60, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.35184 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8 2021-11-16_22:20:55.35187 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.35190 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 2021-11-16_22:20:55.35193 2021-11-16_22:20:55.35195 goroutine 25173 [select]: 2021-11-16_22:20:55.35198 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x400052edc0, 0x37e11d600, 0x37e11d600, 0x0) 2021-11-16_22:20:55.35201 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:998 +0xc8 2021-11-16_22:20:55.35211 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync 2021-11-16_22:20:55.35217 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/scrape/scrape.go:510 +0x790 [...]
The golang/snappy package pointed me directly to use golang/snappy v0.0.3 #8593, but there are other similar reports like prometheus: 2.22.0 failing on RPI4 #15236 and unexpected fault address on raspberry pi4 and openwrt #8589.
The solution
The fix in pull request #8593 was merged on 13th March and released a day later in version 2.25.1.
The solution
The solution is download at least Prometheus 2.25.1 and replace the binary file.
Download Prometheus release.
$ wget https://github.com/prometheus/prometheus/releases/download/v2.25.1/prometheus-2.25.1.linux-arm64.tar.gz
Inspect archive file.
$ tar tvfz prometheus-2.25.1.linux-arm64.tar.gz
drwxr-xr-x circleci/circleci 0 2021-03-14 14:23 prometheus-2.25.1.linux-arm64/ drwxr-xr-x circleci/circleci 0 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/ -rw-r--r-- circleci/circleci 616 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/index.html.example -rw-r--r-- circleci/circleci 2675 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/node-cpu.html -rw-r--r-- circleci/circleci 3513 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/node-disk.html -rw-r--r-- circleci/circleci 5794 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/node-overview.html -rw-r--r-- circleci/circleci 1444 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/node.html -rw-r--r-- circleci/circleci 4103 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/prometheus-overview.html -rw-r--r-- circleci/circleci 1334 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/consoles/prometheus.html drwxr-xr-x circleci/circleci 0 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/console_libraries/ -rw-r--r-- circleci/circleci 2888 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/console_libraries/menu.lib -rw-r--r-- circleci/circleci 6152 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/console_libraries/prom.lib -rw-r--r-- circleci/circleci 926 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/prometheus.yml -rw-r--r-- circleci/circleci 11357 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/LICENSE -rw-r--r-- circleci/circleci 3420 2021-03-14 14:21 prometheus-2.25.1.linux-arm64/NOTICE -rwxr-xr-x circleci/circleci 85921133 2021-03-14 13:32 prometheus-2.25.1.linux-arm64/prometheus -rwxr-xr-x circleci/circleci 76259866 2021-03-14 13:34 prometheus-2.25.1.linux-arm64/promtool
Extract binary.
$ tar xvfz prometheus-2.25.1.linux-arm64.tar.gz prometheus-2.25.1.linux-arm64/prometheus
Stop Prometheus service.
$ sudo gitlab-ctl stop prometheus
ok: down: prometheus: 1s, normally up
Move original binary file.
$ mv /opt/gitlab/embedded/bin/{prometheus,prometheus.orig}
Copy the replacement.
$ sudo cp prometheus-2.25.1.linux-arm64/prometheus /opt/gitlab/embedded/bin/
Start service.
$ sudo gitlab-ctl start prometheus
ok: run: prometheus: (pid 190553) 0s
Check service status.
$ sudo gitlab-ctl status prometheus
run: prometheus: (pid 190553) 31s; run: log: (pid 12953) 81946s
Inspect service logs.
$ sudo gitlab-ctl tail prometheus
[...] 2021-11-16_22:43:14.47308 level=info ts=2021-11-16T22:43:14.472Z caller=main.go:751 msg="Server is ready to receive web requests." [...]
Check out Grafana graphs.
GitLab bug report #345859.