How to prevent systemd service start operation from timing out

Specify systemd startup timeout option to prevent service start operation from timing out.

Inspect default timeout for start operation.

$ sudo systemctl show elasticsearch | grep ^Timeout
TimeoutStartUSec=1min 30s
TimeoutStopUSec=infinity

Elasticsearch service will be terminated if it cannot start in 90 seconds (by default).

$ sudo systemctl start elasticsearch
Job for elasticsearch.service failed because a timeout was exceeded.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Inspect service runtime status information to confirm the findings.

$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─startup-timeout.conf
   Active: failed (Result: timeout) since Thu 2019-11-28 23:14:45 GMT; 23s ago
     Docs: http://www.elastic.co
  Process: 3267 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=TERM)
 Main PID: 3267 (code=killed, signal=TERM)

Nov 28 23:14:44 debian systemd[1]: Starting Elasticsearch...
Nov 28 23:14:45 debian elasticsearch[3267]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Start operation timed out. Terminating.
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Main process exited, code=killed, status=15/TERM
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Failed with result 'timeout'.
Nov 28 23:14:45 debian systemd[1]: Failed to start Elasticsearch.

Alternatively, inspect the most recent log data from the journal.

$ sudo journalctl -u elasticsearch
-- Logs begin at Thu 2019-11-28 23:14:07 GMT, end at Thu 2019-11-28 23:15:26 GMT. --
Nov 28 23:14:44 debian systemd[1]: Starting Elasticsearch...
Nov 28 23:14:45 debian elasticsearch[3267]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Start operation timed out. Terminating.
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Main process exited, code=killed, status=15/TERM
Nov 28 23:14:45 debian systemd[1]: elasticsearch.service: Failed with result 'timeout'.
Nov 28 23:14:45 debian systemd[1]: Failed to start Elasticsearch.

Create a service drop-in configuration directory.

$ sudo mkdir /etc/systemd/system/elasticsearch.service.d

Define TimeoutStartSec option to increase startup timeout.

$ echo -e "[Service]\nTimeoutStartSec=180" | sudo tee /etc/systemd/system/elasticsearch.service.d/startup-timeout.conf
[Service]
TimeoutStartSec=180

Reload systemd manager configuration.

$ sudo systemctl daemon-reload

Inspect altered timeout for start operation.

$ sudo systemctl show elasticsearch | grep ^Timeout
TimeoutStartUSec=3min
TimeoutStopUSec=infinity

Start the Elasticsearch service.

$ sudo systemctl start elasticsearch

Inspect Elasticsearch service status.

$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─startup-timeout.conf
   Active: active (running) since Thu 2019-11-28 23:43:47 GMT; 1min 49s ago
     Docs: http://www.elastic.co
 Main PID: 3990 (java)
    Tasks: 57 (limit: 238)
   Memory: 142.5M
   CGroup: /system.slice/elasticsearch.service
           ├─3990 /usr/share/elasticsearch/jdk/bin/java -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-10017090371480218455 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Dio.netty.allocator.type=unpooled -XX:MaxDirectMemorySize=60817408 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
           └─4082 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Nov 28 23:42:59 debian systemd[1]: Starting Elasticsearch...
Nov 28 23:43:01 debian elasticsearch[3990]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 28 23:43:47 debian systemd[1]: Started Elasticsearch.

The service startup timeout was not triggered this time as it was increased from 1min 30s to 3min.