How to deal with many small log files using Filebeat

Use Filebeat to deal with many small log files.

For example, I have a directory that contains several thousand small log files. These are created by short-running cron jobs on daily basics.

Filebeat configuration

I do not want to harvest everything as most of these files are quite old. Besides, I do not want to put a burden on the operating system.

So, I will harvest logs from files that were modified at most 3 hours ago (which will decrease the number of log files to dozens of files) and remove these files’ state after 4 hours as there is no reason to keep track of these after that time. I will use at most five harvesters at a time. It will take time to perform the initial operation, but should be more than enough to keep up after that. I will also scan for new files every two minutes and close files after one minute of inactivity to keep the number of open files low.

filebeat.inputs:
- type: log
  fields.application: application-cron           # additional field
  tags: ["application-cron", "application-logs"] # define tags
  paths:                                         # logs path
    - /var/log/application/cron/search_*.log
  ignore_older:    3h        # ignore files that were modified 3 hours ago                        
  clean_inactive:  4h        # removes the state of a file after 4h (it must be > ignore_older + scan_frequency)
  close_inactive:  1m        # close file handle after 1 minute of inactivity (file can be reopened after scan_frequency)
  scan_frequency:  2m        # check for new files every 2 minutes
  harvester_limit: 5         # limit the number of harvesters to 5 at a time

setup.template.enabled: false

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 2
  permissions: 0644


output.elasticsearch:
  hosts: ["192.0.2.110:9200"]
  index: "%{[fields.application]}-%{+yyyy.MM.dd}"

Filebeat behavior

It will randomly start harvesting logs from five files.

2020-04-19T13:25:39.961Z	INFO	log/harvester.go:228	Harvester started for file: /var/log/application/cron/search_2020-04-11.log
2020-04-19T13:25:40.161Z	INFO	log/harvester.go:228	Harvester started for file: /var/log/application/cron/search_2020-04-17.log
2020-04-19T13:25:41.332Z	INFO	log/harvester.go:228	Harvester started for file: /var/log/application/cron/search_2020-04-12.log
2020-04-19T13:25:41.447Z	INFO	log/harvester.go:228	Harvester started for file: /var/log/application/cron/search_2020-04-15.log
2020-04-19T13:25:41.649Z	INFO	log/harvester.go:228	Harvester started for file: /var/log/application/cron/search_2020-04-19.log

It will generate an error, but you can ignore it as the limit was introduced for a particular reason.

2020-04-19T13:25:39.924Z	ERROR	log/input.go:460	Harvester could not be started on new file: /var/log/application/cron/search_2020-04-14.log, Err: Harvester limit reached
2020-04-19T13:25:39.927Z	ERROR	log/input.go:460	Harvester could not be started on new file: /var/log/application/cron/search_2020-04-18.log, Err: Harvester limit reached

Harvesters are freed as files are closed after one minute of inactivity. Harvester will pick up any of these files if it gets modified during the three-hour time window.

2020-04-19T13:26:39.896Z	INFO	log/harvester.go:253	File is inactive: /var/log/application/cron/search_2020-04-11.log. Closing because close_inactive of 1m0s reached.
2020-04-19T13:26:40.991Z	INFO	log/harvester.go:253	File is inactive: /var/log/application/cron/search_2020-04-17.log. Closing because close_inactive of 1m0s reached.
2020-04-19T13:26:41.566Z	INFO	log/harvester.go:253	File is inactive: /var/log/application/cron/search_2020-04-12.log. Closing because close_inactive of 1m0s reached.
2020-04-19T13:26:41.748Z	INFO	log/harvester.go:253	File is inactive: /var/log/application/cron/search_2020-04-15.log. Closing because close_inactive of 1m0s reached.
2020-04-19T13:26:41.818Z	INFO	log/harvester.go:253	File is inactive: /var/log/application/cron/search_2020-04-19.log. Closing because close_inactive of 1m0s reached.

It will randomly start harvesting logs from the next five files, and so on.

2020-04-19T13:26:42.171Z	INFO	log/harvester.go:228	Harvester started for file: /var/log/application/cron/search_2020-04-14.log
2020-04-19T13:26:24.334Z	INFO	log/harvester.go:228	Harvester started for file: /var/log/application/cron/search_2020-04-06.log
2020-04-19T13:26:42.465Z	INFO	log/harvester.go:228	Harvester started for file: /var/log/application/cron/search_2020-04-18.log
2020-04-19T13:26:42.627Z	INFO	log/harvester.go:228	Harvester started for file: /var/log/application/cron/search_2020-04-09.log
2020-04-19T13:26:42.913Z	INFO	log/harvester.go:228	Harvester started for file: /var/log/application/cron/search_2020-04-16.log