Graylog will generate a warning to inform you that socket receive buffer size (SO_RCVBUF
) should be increased.
[milosz@graylog-server-5 ~]$ tail -f /var/log/graylog-server/server.log 2020-04-12T12:29:48.540Z INFO [NetworkListener] Started listener bound to [192.0.2.11:9000] 2020-04-12T12:29:48.542Z INFO [HttpServer] [HttpServer] Started. 2020-04-12T12:29:48.542Z INFO [JerseyService] Started REST API at <192.0.2.11:9000> 2020-04-12T12:29:48.543Z INFO [ServiceManagerListener] Services are healthy 2020-04-12T12:29:48.544Z INFO [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=3, EtagService [RUNNING]=73, ConfigurationEtagService [RUNNING]=73, OutputSetupService [RUNNING]=73, JobSchedulerService [RUNNING]=73, GracefulShutdownService [RUNNING]=94, JournalReader [RUNNING]=95, UrlWhitelistService [RUNNING]=127, KafkaJournal [RUNNING]=143, MongoDBProcessingStatusRecorderService [RUNNING]=147, PeriodicalsService [RUNNING]=147, BufferSynchronizerService [RUNNING]=160, LookupTableService [RUNNING]=184, StreamCacheService [RUNNING]=420, JerseyService [RUNNING]=27175} 2020-04-12T09:29:48.546Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE] 2020-04-12T12:29:48.548Z INFO [ServerBootstrap] Graylog server up and running. 2020-04-12T12:29:48.567Z INFO [InputStateListener] Input [Beats/5cfe0aeaec88901911304649] is now STARTING 2020-04-12T12:29:48.864Z INFO [InputStateListener] Input [Beats/5cfe0aeaec88901911304649] is now RUNNING 2020-04-12T12:29:48.890Z WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=Filebeat, type=org.graylog.plugins.beats.Beats2Input, nodeId=null} (channel [id: 0x0dd4a4b0, L:/0:0:0:0:0:0:0:0%0:5044]) should be 1048576 but is 425984. [...]
Inspect tcp
manual page.
The maximum sizes for socket buffers declared via the SO_SNDBUF and SO_RCVBUF mechanisms are limited by the values in the /proc/sys/net/core/rmem_max and /proc/sys/net/core/wmem_max files. Note that TCP actually allocates twice the size of the buffer requested in the setsockopt(2) call, and so a succeeding getsockopt(2) call will not return the same size of buffer as requested in the setsockopt(2) call. TCP uses the extra space for administrative purposes and internal kernel structures, and the /proc file values reflect the larger sizes compared to the actual TCP windows. On individual connections, the socket buffer size must be set prior to the listen(2) or connect(2) calls in order to have it take effect. See socket(7) for more information.
Inspect maximum socket buffer size for SO_RCVBUF
(receive, rmem_max
) and SO_SNDBUF
(send, wmem_max
).
[milosz@graylog-server-5 ~]$ sysctl --all --pattern "net.core.[rw]mem_max" net.core.rmem_max = 212992 net.core.wmem_max = 212992
Update maximum socket receive buffer size.
[milosz@graylog-server-5 ~]$ sudo sysctl --write net.core.rmem_max=524288 net.core.rmem_max = 524288
Restart graylog-server
service.
[milosz@graylog-server-5 ~]$ systemctl restart graylog-server
Inspect Graylog log file on master node.
[milosz@graylog-server-5 ~]$ tail -f /var/log/graylog-server/server.log 2020-04-12T14:38:55.087Z INFO [NetworkListener] Started listener bound to [192.0.2.11:9000] 2020-04-12T14:38:55.089Z INFO [HttpServer] [HttpServer] Started. 2020-04-12T14:38:55.089Z INFO [JerseyService] Started REST API at <192.0.2.11:9000> 2020-04-12T14:38:55.091Z INFO [ServiceManagerListener] Services are healthy 2020-04-12T14:38:55.094Z INFO [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=6, ConfigurationEtagService [RUNNING]=56, BufferSynchronizerService [RUNNING]=56, OutputSetupService [RUNNING]=56, EtagService [RUNNING]=56, JobSchedulerService [RUNNING]=56, KafkaJournal [RUNNING]=58, JournalReader [RUNNING]=58, UrlWhitelistService [RUNNING]=59, GracefulShutdownService [RUNNING]=60, PeriodicalsService [RUNNING]=61, MongoDBProcessingStatusRecorderService [RUNNING]=121, LookupTableService [RUNNING]=155, StreamCacheService [RUNNING]=416, JerseyService [RUNNING]=27308} 2020-04-12T14:38:55.098Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE] 2020-04-12T14:38:55.111Z INFO [ServerBootstrap] Graylog server up and running. 2020-04-12T14:38:55.149Z INFO [InputStateListener] Input [Beats/5cfe0aeaec88901911304649] is now STARTING 2020-04-12T14:38:55.352Z INFO [InputStateListener] Input [Beats/5cfe0aeaec88901911304649] is now RUNNING
Do not forget to create persistent sysctl configuration on every master node.