Categories
SysOps

How to prevent systemd service start operation from timing out

Specify systemd startup timeout option to prevent service start operation from timing out.

Inspect default timeout for start operation.

$ sudo systemctl show elasticsearch | grep ^Timeout
TimeoutStartUSec=1min 30s
TimeoutStopUSec=infinity

Elasticsearch service will be terminated if it cannot start in 90 seconds (by default).

$ sudo systemctl start elasticsearch
Job for elasticsearch.service failed because a timeout was exceeded.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Inspect service runtime status information to confirm the findings.

$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─startup-timeout.conf
   Active: failed (Result: timeout) since Thu 2019-11-28 23:14:45 GMT; 23s ago
     Docs: http://www.elastic.co
  Process: 3267 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=TERM)
 Main PID: 3267 (code=killed, signal=TERM)

Nov 28 23:14:44 debian systemd[1]: Starting Elasticsearch...
Nov 28 23:14:45 debian elasticsearch[3267]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Start operation timed out. Terminating.
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Main process exited, code=killed, status=15/TERM
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Failed with result 'timeout'.
Nov 28 23:14:45 debian systemd[1]: Failed to start Elasticsearch.

Alternatively, inspect the most recent log data from the journal.

$ sudo journalctl -u elasticsearch
-- Logs begin at Thu 2019-11-28 23:14:07 GMT, end at Thu 2019-11-28 23:15:26 GMT. --
Nov 28 23:14:44 debian systemd[1]: Starting Elasticsearch...
Nov 28 23:14:45 debian elasticsearch[3267]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Start operation timed out. Terminating.
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Main process exited, code=killed, status=15/TERM
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Failed with result 'timeout'.
Nov 28 23:14:45 debian systemd[1]: Failed to start Elasticsearch.

Create a service drop-in configuration directory.

$ sudo mkdir /etc/systemd/system/elasticsearch.service.d

Define TimeoutStartSec option to increase startup timeout.

$ echo -e "[Service]\nTimeoutStartSec=180" | sudo tee /etc/systemd/system/elasticsearch.service.d/startup-timeout.conf
[Service]
TimeoutStartSec=180

Reload systemd manager configuration.

$ sudo systemctl daemon-reload

Inspect altered timeout for start operation.

$ sudo systemctl show elasticsearch | grep ^Timeout
TimeoutStartUSec=3min
TimeoutStopUSec=infinity

Start the Elasticsearch service.

$ sudo systemctl start elasticsearch

Inspect Elasticsearch service status.

$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─startup-timeout.conf
   Active: active (running) since Thu 2019-11-28 23:43:47 GMT; 1min 49s ago
     Docs: http://www.elastic.co
 Main PID: 3990 (java)
    Tasks: 57 (limit: 238)
   Memory: 142.5M
   CGroup: /system.slice/elasticsearch.service
           ├─3990 /usr/share/elasticsearch/jdk/bin/java -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-10017090371480218455 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Dio.netty.allocator.type=unpooled -XX:MaxDirectMemorySize=60817408 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
           └─4082 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Nov 28 23:42:59 debian systemd[1]: Starting Elasticsearch...
Nov 28 23:43:01 debian elasticsearch[3990]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 28 23:43:47 debian systemd[1]: Started Elasticsearch.

The service startup timeout was not triggered this time as it was increased from 1min 30s to 3min.