How to manage systemd services remotely

Manage systemd services remotely over SSH protocol.

Create user for systemd manager

Create systemd-manager user on remote machine.

$ sudo useradd --create-home systemd-manager

Prepare the directory and file that will contain public keys for public key authentication.

$ sudo mkdir /home/systemd-manager/.ssh

$ sudo touch /home/systemd-manager/.ssh/authorized_keys

$ sudo chown systemd-manager:systemd-manager /home/systemd-manager/.ssh /home/systemd-manager/.ssh/authorized_keys

$ sudo chmod 700 /home/systemd-manager/.ssh

$ sudo chmod 600 /home/systemd-manager/.ssh/authorized_keys

Populate public keys.

Remember to limit public keys to the systemd-stdio-bridge application that will be used to connect standard input to the D-Bus transport.

command="systemd-stdio-bridge",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc ssh-rsa AAAAC6Mye[...]RhUZQC systemd@manager

Configure authorization framework

Install policykit-1 application-level toolkit for managing administrative policies and privileges to populate configuration files.

$ sudo apt install policykit-1

Create authorization policy to allow systemd-manager user to execute systemctl commands without sudo.

$ cat << EOF | sudo tee /etc/polkit-1/localauthority/50-local.d/systemd-manager.pkla
[Allow user systemd-manager to execute systemctl commands]
Identity=unix-user:systemd-manager
Action=org.freedesktop.systemd1.manage-units
ResultAny=yes
EOF

Manage systemd services remotely

Display service status.

$ systemctl --host systemd-manager@server.example.org status nginx

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system.control/nginx.service.d
             └─50-MemoryLimit.conf
     Active: active (running) since Mon 2020-09-21 00:00:38 CEST; 25min ago
       Docs: man:nginx(8)
    Process: 1571 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 1572 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 1573
      Tasks: 3 (limit: 1149)
     Memory: 3.8M (limit: 1.0G)
     CGroup: /system.slice/nginx.service
             ├─1573 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─1574 nginx: worker process
             └─1575 nginx: worker process

Restart service.

$ systemctl --host systemd-manager@server.example.org restart nginx

Set property.

$ systemctl --host systemd-manager@server.example.org set-property nginx  MemoryLimit=2G

Check if service is enabled.

$ systemctl --host systemd-manager@server.example.org is-enabled nginx

enabled

List timers.

$ systemctl --host systemd-manager@server.example.org list-timers

NEXT                         LEFT               LAST                         PASSED      UNIT                         ACTIVATES
Thu 2020-09-03 23:50:16 CEST 2 weeks 3 days ago Sun 2020-09-20 23:23:30 CEST 1h 4min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2020-09-21 00:47:04 CEST 18min left         Sun 2020-09-20 13:24:52 CEST 11h ago     apt-daily.timer              apt-daily.service
Mon 2020-09-21 02:00:00 CEST 1h 31min left      Sun 2020-09-20 13:24:52 CEST 11h ago     logrotate.timer              logrotate.service
Mon 2020-09-21 02:00:00 CEST 1h 31min left      Sun 2020-09-20 13:24:52 CEST 11h ago     man-db.timer                 man-db.service
Mon 2020-09-21 08:05:29 CEST 7h left            Sun 2020-09-20 13:24:52 CEST 11h ago     apt-daily-upgrade.timer      apt-daily-upgrade.service
5 timers listed.
Pass --all to see loaded but inactive timers, too.

List sockets.

$ systemctl --host systemd-manager@server.example.org list-sockets

LISTEN                          UNITS                           ACTIVATES
/run/initctl                    systemd-initctl.socket          systemd-initctl.service
/run/systemd/fsck.progress      systemd-fsckd.socket            systemd-fsckd.service
/run/systemd/journal/dev-log    systemd-journald-dev-log.socket systemd-journald.service
/run/systemd/journal/socket     systemd-journald.socket         systemd-journald.service
/run/systemd/journal/stdout     systemd-journald.socket         systemd-journald.service
/run/systemd/journal/syslog     syslog.socket                   rsyslog.service
/run/udev/control               systemd-udevd-control.socket    systemd-udevd.service
/var/run/dbus/system_bus_socket dbus.socket                     dbus.service
audit 1                         systemd-journald-audit.socket   systemd-journald.service
kobject-uevent 1                systemd-udevd-kernel.socket     systemd-udevd.service
10 sockets listed.
Pass --all to see loaded but inactive sockets, too.

Authorize additional operations

Sometimes you need to authorize additional operations, for example to reload the systemd manager configuration.

$ systemctl --host systemd-manager@server.example.org daemon-reload

Failed to reload daemon: Interactive authentication required.

You just need to add an additional action or use an asterisk (not recommended).

Action=org.freedesktop.systemd1.manage-units;org.freedesktop.systemd1.reload-daemon

Action=org.freedesktop.systemd1.*

Use pkaction utility to display every possible action.

$ pkaction

org.dpkg.pkexec.update-alternatives
org.freedesktop.hostname1.get-product-uuid
org.freedesktop.hostname1.set-hostname
org.freedesktop.hostname1.set-machine-info
org.freedesktop.hostname1.set-static-hostname
org.freedesktop.locale1.set-keyboard
org.freedesktop.locale1.set-locale
org.freedesktop.login1.attach-device
org.freedesktop.login1.flush-devices
org.freedesktop.login1.halt
org.freedesktop.login1.halt-ignore-inhibit
org.freedesktop.login1.halt-multiple-sessions
org.freedesktop.login1.hibernate
org.freedesktop.login1.hibernate-ignore-inhibit
org.freedesktop.login1.hibernate-multiple-sessions
org.freedesktop.login1.inhibit-block-idle
org.freedesktop.login1.inhibit-block-shutdown
org.freedesktop.login1.inhibit-block-sleep
org.freedesktop.login1.inhibit-delay-shutdown
org.freedesktop.login1.inhibit-delay-sleep
org.freedesktop.login1.inhibit-handle-hibernate-key
org.freedesktop.login1.inhibit-handle-lid-switch
org.freedesktop.login1.inhibit-handle-power-key
org.freedesktop.login1.inhibit-handle-suspend-key
org.freedesktop.login1.lock-sessions
org.freedesktop.login1.manage
org.freedesktop.login1.power-off
org.freedesktop.login1.power-off-ignore-inhibit
org.freedesktop.login1.power-off-multiple-sessions
org.freedesktop.login1.reboot
org.freedesktop.login1.reboot-ignore-inhibit
org.freedesktop.login1.reboot-multiple-sessions
org.freedesktop.login1.set-reboot-to-firmware-setup
org.freedesktop.login1.set-self-linger
org.freedesktop.login1.set-user-linger
org.freedesktop.login1.set-wall-message
org.freedesktop.login1.suspend
org.freedesktop.login1.suspend-ignore-inhibit
org.freedesktop.login1.suspend-multiple-sessions
org.freedesktop.policykit.exec
org.freedesktop.policykit.lockdown
org.freedesktop.resolve1.register-service
org.freedesktop.resolve1.unregister-service
org.freedesktop.systemd1.manage-unit-files
org.freedesktop.systemd1.manage-units
org.freedesktop.systemd1.reload-daemon
org.freedesktop.systemd1.reply-password
org.freedesktop.systemd1.set-environment
org.freedesktop.timedate1.set-local-rtc
org.freedesktop.timedate1.set-ntp
org.freedesktop.timedate1.set-time
org.freedesktop.timedate1.set-timezone

Inspect specific action from the above-mentioned list.

$ pkaction --verbose --action-id org.freedesktop.systemd1.manage-units

org.freedesktop.systemd1.manage-units:
  description:       Manage system services or other units
  message:           Authentication is required to manage system services or other units.
  vendor:            The systemd Project
  vendor_url:        http://www.freedesktop.org/wiki/Software/systemd
  icon:
  implicit any:      auth_admin
  implicit inactive: auth_admin
  implicit active:   auth_admin_keep

Alternatively, inspect systemd policy file to identify possible actions.

$ cat /usr/share/polkit-1/actions/org.freedesktop.systemd1.policy

<?xml version="1.0" encoding="UTF-8"?> <!--*-nxml-*-->
<!DOCTYPE policyconfig PUBLIC "-//freedesktop//DTD PolicyKit Policy Configuration 1.0//EN"
        "http://www.freedesktop.org/standards/PolicyKit/1/policyconfig.dtd">
<!--
  SPDX-License-Identifier: LGPL-2.1+
  This file is part of systemd.
  systemd is free software; you can redistribute it and/or modify it
  under the terms of the GNU Lesser General Public License as published by
  the Free Software Foundation; either version 2.1 of the License, or
  (at your option) any later version.
-->
<policyconfig>
        <vendor>The systemd Project</vendor>
        <vendor_url>http://www.freedesktop.org/wiki/Software/systemd</vendor_url>
        <action id="org.freedesktop.systemd1.reply-password">
                <description gettext-domain="systemd">Send passphrase back to system</description>
                <message gettext-domain="systemd">Authentication is required to send the entered passphrase back to the system.</message>
                <defaults>
                        <allow_any>no</allow_any>
                        <allow_inactive>no</allow_inactive>
                        <allow_active>auth_admin_keep</allow_active>
                </defaults>
                <annotate key="org.freedesktop.policykit.exec.path">/lib/systemd/systemd-reply-password</annotate>
        </action>
        <action id="org.freedesktop.systemd1.manage-units">
                <description gettext-domain="systemd">Manage system services or other units</description>
                <message gettext-domain="systemd">Authentication is required to manage system services or other units.</message>
                <defaults>
                        <allow_any>auth_admin</allow_any>
                        <allow_inactive>auth_admin</allow_inactive>
                        <allow_active>auth_admin_keep</allow_active>
                </defaults>
        </action>
        <action id="org.freedesktop.systemd1.manage-unit-files">
                <description gettext-domain="systemd">Manage system service or unit files</description>
                <message gettext-domain="systemd">Authentication is required to manage system service or unit files.</message>
                <defaults>
                        <allow_any>auth_admin</allow_any>
                        <allow_inactive>auth_admin</allow_inactive>
                        <allow_active>auth_admin_keep</allow_active>
                </defaults>
                <annotate key="org.freedesktop.policykit.imply">org.freedesktop.systemd1.reload-daemon org.freedesktop.systemd1.manage-units</annotate>
        </action>
        <action id="org.freedesktop.systemd1.set-environment">
                <description gettext-domain="systemd">Set or unset system and service manager environment variables</description>
                <message gettext-domain="systemd">Authentication is required to set or unset system and service manager environment variables.</message>
                <defaults>
                        <allow_any>auth_admin</allow_any>
                        <allow_inactive>auth_admin</allow_inactive>
                        <allow_active>auth_admin_keep</allow_active>
                </defaults>
        </action>
        <action id="org.freedesktop.systemd1.reload-daemon">
                <description gettext-domain="systemd">Reload the systemd state</description>
                <message gettext-domain="systemd">Authentication is required to reload the systemd state.</message>
                <defaults>
                        <allow_any>auth_admin</allow_any>
                        <allow_inactive>auth_admin</allow_inactive>
                        <allow_active>auth_admin_keep</allow_active>
                </defaults>
        </action>
</policyconfig>

Limitations

You cannot edit services.

$ systemctl --host systemd-manager@server.example.org edit nginx

Cannot edit units remotely.

You cannot inspect service logs which should be obvious.

Create user for systemd manager#

Populate public keys.#

Configure authorization framework#

Manage systemd services remotely#

Authorize additional operations#

Limitations#