Configure Grafana application within GitLab instance to accept self-signed certificate.
Add a self-signed certificate, reconfigure GitLab instance and update Grafana Callback URL.
You will see login.OAuthLogin(missing saved state) server error when you try to automatically login to the Grafana application.
Inspect logs to find out why.
$ sudo gitlab-ctl tail grafana
[...] 2021-12-07_21:55:59.24745 t=2021-12-07T21:55:59+0000 lvl=info msg="Request Completed" logger=context userId=3 orgId=1 uname=milosz method=GET path=/login/gitlab status=302 remote_addr=10.114.53.1 time_ms=3 size=338 referer= 2021-12-07_21:56:00.48841 t=2021-12-07T21:56:00+0000 lvl=info msg="state check" logger=oauth queryState=e1420a2b2365c7 cookieState=e1420a2b2365c7 2021-12-07_21:56:00.53961 t=2021-12-07T21:56:00+0000 lvl=eror msg=login.OAuthLogin(NewTransportWithCode) logger=context userId=3 orgId=1 uname=milosz error="Post \"https://git.octocat.lab/oauth/token\": x509: certificate signed by unknown authority" 2021-12-07_21:56:00.54121 t=2021-12-07T21:56:00+0000 lvl=eror msg="Request Completed" logger=context userId=3 orgId=1 uname=milosz method=GET path=/login/gitlab status=500 remote_addr=10.114.53.1 time_ms=57 size=1754 referer="https://git.octocat.lab/oauth/authorize?access_type=online&client_id=3d629a79621f0a&redirect_uri=https%3A%2F%2Fgit.octocat.lab%2F-%2Fgrafana%2Flogin%2Fgitlab&response_type=code&scope=read_user&state=CCYcXkPfHvo8d_SPl9XnekSHo4lXzJEXMq2zI1Kx-4s%3D" [...]
Notice, the error x509: certificate signed by unknown authority shows that self-signed certificate is simply not trusted.
Inspect created certificates.
$ ls /etc/gitlab/ssl/
git.octocat.lab.crt git.octocat.lab.key git.octocat.lab.key-staging
Copy GitLab certificate to the trusted ones.
$ cp /etc/gitlab/ssl/git.octocat.lab.crt /etc/gitlab/trusted-certs/
Reconfigure GitLab instance.
$ sudo gitlab-ctl reconfigure
[...] Recipe: gitlab::add_trusted_certs * directory[/etc/gitlab/trusted-certs] action create (up to date) * directory[/opt/gitlab/embedded/ssl/certs] action create (up to date) * file[/opt/gitlab/embedded/ssl/certs/README] action create (up to date) * ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] action run * Moving existing certificates found in /opt/gitlab/embedded/ssl/certs * Symlinking existing certificates found in /etc/gitlab/trusted-certs Linking c0a2d4cd.0 from /etc/gitlab/trusted-certs/git.octocat.lab.crt - execute the ruby block Move existing certs and link to /opt/gitlab/embedded/ssl/certs [...]
Try to use Grafana application and inspect logs.
$ sudo gitlab-ctl tail grafana
[...] 2021-12-07_22:02:08.80012 t=2021-12-07T22:02:08+0000 lvl=info msg="Request Completed" logger=context userId=3 orgId=1 uname=milosz method=GET path=/login/gitlab status=302 remote_addr=10.114.53.1 time_ms=25 size=338 referer= 2021-12-07_22:02:10.46190 t=2021-12-07T22:02:10+0000 lvl=info msg="state check" logger=oauth queryState=7ba07c13d1df9f cookieState=7ba07c13d1df9f 2021-12-07_22:02:10.57261 t=2021-12-07T22:02:10+0000 lvl=eror msg=login.OAuthLogin(NewTransportWithCode) logger=context userId=3 orgId=1 uname=milosz error="Post \"https://git.octocat.lab/oauth/token\": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0" 2021-12-07_22:02:10.57657 t=2021-12-07T22:02:10+0000 lvl=eror msg="Request Completed" logger=context userId=3 orgId=1 uname=milosz method=GET path=/login/gitlab status=500 remote_addr=10.114.53.1 time_ms=130 size=1754 referer="https://git.octocat.lab/oauth/authorize?access_type=online&client_id=3d629a79621f0a&redirect_uri=https%3A%2F%2Fgit.octocat.lab%2F-%2Fgrafana%2Flogin%2Fgitlab&response_type=code&scope=read_user&state=_csmntS8mFtmTTKBs5NBatKfFk_LvjGIQAl_jtWJdRg%3D" [...]
Now, you will likely see x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0 error when you use self-signed certificate generated automatically as it uses Common Name instead of SAN extension.
$ sudo openssl x509 -in /etc/gitlab/ssl/git.octocat.lab.crt --text --nocert
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 0 (0x0)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = git.octocat.lab
Validity
Not Before: Nov 28 23:35:38 2021 GMT
Not After : Dec 28 23:35:38 2021 GMT
Subject: CN = git.octocat.lab
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:af:12:1b:22:d6:c5:64:17:42:a5:30:2c:57:b9:
9d:32:60:dc:1e:cb:e5:81:ff:00:ac:39:f2:86:e2:
cc:57:9c:98:8d:0f:b1:71:a7:eb:a3:ee:ab:64:78:
74:8e:11:e1:ae:3b:ae:c4:02:c2:d5:98:16:28:0e:
1a:9a:5a:4d:ba:a0:16:ac:e4:11:0d:d6:1e:b9:22:
87:26:1c:af:ef:5c:f7:1e:05:71:27:d3:f3:01:e5:
80:52:aa:45:bd:7a:38:b7:c2:43:9a:b3:37:90:e2:
9f:c0:fb:1f:c9:ab:af:48:f8:b3:a3:44:1e:59:47:
a1:40:d5:4a:d7:e1:97:3b:c6:8e:49:dd:7e:81:4b:
01:fc:d6:fe:2f:5c:22:f3:55:7c:db:00:68:59:06:
79:93:46:41:66:83:2f:10:a8:86:b9:09:a5:e2:07:
ba:04:f2:c2:13:29:9c:27:5a:9f:8f:7d:b0:0b:59:
23:dc:ec:cd:1b:20:bf:96:fd:1d:fb:ab:47:ce:28:
1a:b2:20:52:57:24:16:a5:62:21:e5:b0:71:52:da:
1e:85:33:7f:d1:f6:f4:8e:6c:2e:c4:02:a2:aa:de:
c7:dc:27:bb:10:7e:2f:f6:ef:36:9f:47:96:f2:73:
e4:73:30:47:7b:2e:cf:1a:82:08:df:96:6a:10:23:
2e:77
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
72:B7:F7:5E:31:A0:E4:FC:C4:31:AF:B4:DB:7D:84:EA:23:51:A9:38
Signature Algorithm: sha256WithRSAEncryption
17:ce:37:b1:3a:b8:e6:12:5c:03:f5:46:88:eb:2c:0a:13:e3:
67:99:50:b9:f0:c9:19:02:09:27:fc:24:4a:6b:75:3e:ab:fa:
3e:20:58:9b:ec:ee:4a:8a:43:af:0a:16:f0:ca:aa:2f:50:cc:
6f:80:85:f3:2b:ae:7d:f8:d1:61:fc:ac:cb:8c:25:c1:ad:03:
6a:cb:72:37:3c:63:09:18:66:04:00:c5:4c:93:df:b2:9d:e0:
0c:79:e6:be:9e:71:23:70:d6:b2:0f:20:ec:d7:db:1e:2b:98:
58:5f:20:a6:96:db:fa:3c:02:47:79:16:8e:d7:19:9a:cd:f9:
bd:8a:87:fd:d0:98:c5:b2:a7:c8:53:b5:da:ff:d2:26:ea:6a:
28:54:a0:d8:02:65:e5:e2:f4:62:36:44:5e:5b:15:ff:79:b2:
59:57:5b:95:15:a1:4c:81:12:bb:c6:bb:9d:3f:04:ac:81:c3:
d1:62:0b:58:93:72:c6:eb:0a:ca:01:82:b5:61:d9:af:70:9c:
f3:27:f3:1a:da:d8:18:ad:3e:38:15:aa:ea:f3:a5:a5:26:4a:
4d:3b:b4:92:d6:04:4f:de:d5:5c:7a:16:2d:c9:69:e4:bb:5c:
3e:ec:5e:db:a5:2c:a5:c6:d3:cc:3f:a2:ec:2d:ef:2e:02:db:
5b:d6:ca:92
Update Grafana configuration to define option mentioned earlier.
$ vim /etc/gitlab/gitlab.rb
[...]
################################################################################
## Grafana Dashboards
##! Docs: https://docs.gitlab.com/ee/administration/monitoring/prometheus/#prometheus-as-a-grafana-data-source
################################################################################
# grafana['enable'] = true
# grafana['log_directory'] = '/var/log/gitlab/grafana'
# grafana['home'] = '/var/opt/gitlab/grafana'
# grafana['admin_password'] = 'admin'
# grafana['allow_user_sign_up'] = false
# grafana['basic_auth_enabled'] = false
# grafana['disable_login_form'] = true
# grafana['gitlab_application_id'] = 'GITLAB_APPLICATION_ID'
# grafana['gitlab_secret'] = 'GITLAB_SECRET'
# grafana['env_directory'] = '/opt/gitlab/etc/grafana/env'
# grafana['allowed_groups'] = []
# grafana['gitlab_auth_sign_up'] = true
grafana['env'] = {
'GODEBUG' => "x509ignoreCN=0"
}
# grafana['metrics_enabled'] = false
# grafana['metrics_basic_auth_username'] = 'grafana_metrics' # default: nil
# grafana['metrics_basic_auth_password'] = 'please_set_a_unique_password' # default: nil
# grafana['alerting_enabled'] = false
[...]
Reconfigure GitLab instance.
$ sudo gitlab-ctl reconfigure
[...] Recipe: monitoring::grafana * directory[/var/log/gitlab/grafana] action create (up to date) * directory[/var/opt/gitlab/grafana] action create (up to date) * directory[/var/opt/gitlab/grafana/provisioning] action create (up to date) * directory[/var/opt/gitlab/grafana/provisioning/dashboards] action create (up to date) * directory[/var/opt/gitlab/grafana/provisioning/datasources] action create (up to date) * directory[/var/opt/gitlab/grafana/provisioning/notifiers] action create (up to date) * file[/var/opt/gitlab/grafana/CVE_reset_status] action delete (up to date) * link[/var/opt/gitlab/grafana/conf] action create (up to date) * link[/var/opt/gitlab/grafana/public] action create (up to date) * directory[/opt/gitlab/etc/grafana/env] action create (up to date) * ruby_block[populate Grafana configuration options] action run - execute the ruby block populate Grafana configuration options * env_dir[/opt/gitlab/etc/grafana/env] action create * directory[/opt/gitlab/etc/grafana/env] action create (up to date) * file[/opt/gitlab/etc/grafana/env/SSL_CERT_DIR] action create (up to date) * file[/opt/gitlab/etc/grafana/env/GODEBUG] action create - create new file /opt/gitlab/etc/grafana/env/GODEBUG - update content in file /opt/gitlab/etc/grafana/env/GODEBUG from none to 5fc039 - suppressed sensitive resource * template[/var/opt/gitlab/grafana/grafana.ini] action create (up to date) * file[/var/opt/gitlab/grafana/provisioning/dashboards/gitlab_dashboards.yml] action create (up to date) * file[/var/opt/gitlab/grafana/provisioning/datasources/gitlab_datasources.yml] action create (up to date) * service[grafana] action nothing (skipped due to action :nothing) * runit_service[grafana] action enable * ruby_block[restart_service] action nothing (skipped due to action :nothing) * ruby_block[restart_log_service] action nothing (skipped due to action :nothing) * ruby_block[reload_log_service] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/sv/grafana] action create (up to date) * template[/opt/gitlab/sv/grafana/run] action create (up to date) * directory[/opt/gitlab/sv/grafana/log] action create (up to date) * directory[/opt/gitlab/sv/grafana/log/main] action create (up to date) * template[/opt/gitlab/sv/grafana/log/config] action create (up to date) * ruby_block[verify_chown_persisted_on_grafana] action nothing (skipped due to action :nothing) * link[/var/log/gitlab/grafana/config] action create (up to date) * template[/opt/gitlab/sv/grafana/log/run] action create (up to date) * directory[/opt/gitlab/sv/grafana/env] action create (up to date) * ruby_block[Delete unmanaged env files for grafana service] action run (skipped due to only_if) * template[/opt/gitlab/sv/grafana/check] action create (skipped due to only_if) * template[/opt/gitlab/sv/grafana/finish] action create (skipped due to only_if) * directory[/opt/gitlab/sv/grafana/control] action create (up to date) * link[/opt/gitlab/init/grafana] action create (up to date) * file[/opt/gitlab/sv/grafana/down] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/service] action create (up to date) * link[/opt/gitlab/service/grafana] action create (up to date) * ruby_block[wait for grafana service socket] action run (skipped due to not_if) (up to date) [...]
Alternatively replace certificate with a proper one using subject alternative names. See how to generate multi-domain SSL certificate inside how to generate self-signed SSL certificate blog post
At this moment Grafana should work as expected.
