Delete old Elasticsearch indices using Elasticsearch Curator
utility.
Installation
Install required dependencies.
$ sudo apt install gnupg2 apt-transport-https curl
Add a repository key.
$ curl --silent https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Add an Elasticsearch Curator
repository.
$ echo "deb [arch=amd64] https://packages.elastic.co/curator/5/debian9 stable main" | sudo tee /etc/apt/sources.list.d/curator.list
Update package index.
$ sudo apt update
Install Elasticsearch Curator
utility.
$ sudo apt install elasticsearch-curator
Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: elasticsearch-curator 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 20.4 MB of archives. After this operation, 78.9 MB of additional disk space will be used. Get:1 https://packages.elastic.co/curator/5/debian9 stable/main amd64 elasticsearch-curator amd64 5.8.1 [20.4 MB] Fetched 20.4 MB in 3s (6,016 kB/s) Selecting previously unselected package elasticsearch-curator. (Reading database ... 43831 files and directories currently installed.) Preparing to unpack .../elasticsearch-curator_5.8.1_amd64.deb ... Unpacking elasticsearch-curator (5.8.1) ... Setting up elasticsearch-curator (5.8.1) ...
Configuration
Create a configuration directory.
$ mkdir .curator
Create .curator/client-development-cluster.yml
client configuration file to access the development cluster.
--- client: hosts: - 192.0.2.112 port: 9200 url_prefix: use_ssl: certificate: client_cert: client_key: ssl_no_validate: http_auth: timeout: 90 master_only: logging: loglevel: INFO logfile: logformat: default blacklist: ['elasticsearch', 'urllib3']
logfile
will result in logging to the standard output.Create .curator/action-delete-old-application-cron-indices.yml
action configuration file to delete old application-cron-* indices.
actions: 1: action: delete_indices options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 14 exclude: - filtertype: pattern kind: prefix value: application-cron-
Execution
Inspect indices.
$ curl "http://localhost:9200/_cat/indices?h=index&s=index"
application-cron-2020.04.01 application-cron-2020.04.02 application-cron-2020.04.03 application-cron-2020.04.04 application-cron-2020.04.05 application-cron-2020.04.06 application-cron-2020.04.07 application-cron-2020.04.08 application-cron-2020.04.09 application-cron-2020.04.10 application-cron-2020.04.11 application-cron-2020.04.12 application-cron-2020.04.13 application-cron-2020.04.14 application-cron-2020.04.15 application-cron-2020.04.17 application-cron-2020.04.18 application-cron-2020.04.19
Execute an action, but do not perform any changes.
$ curator --dry-run --config ~/.curator/client-development-cluster.yml ~/.curator/action-delete-old-application-cron-indices.yml
2020-04-19 21:46:07,693 INFO Preparing Action ID: 1, "delete_indices" 2020-04-19 21:46:07,693 INFO Creating client object and testing connection 2020-04-19 21:46:07,696 INFO Instantiating client object 2020-04-19 21:46:07,696 INFO Testing client connectivity 2020-04-19 21:46:07,707 INFO Successfully created Elasticsearch client object with provided settings 2020-04-19 21:46:07,710 INFO Trying Action ID: 1, "delete_indices": No description given 2020-04-19 21:46:07,746 INFO DRY-RUN MODE. No changes will be made. 2020-04-19 21:46:07,746 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices". 2020-04-19 21:46:07,746 INFO DRY-RUN: delete_indices: application-cron-2020.04.01 with arguments: {} 2020-04-19 21:46:07,747 INFO DRY-RUN: delete_indices: application-cron-2020.04.02 with arguments: {} 2020-04-19 21:46:07,747 INFO DRY-RUN: delete_indices: application-cron-2020.04.03 with arguments: {} 2020-04-19 21:46:07,747 INFO DRY-RUN: delete_indices: application-cron-2020.04.04 with arguments: {} 2020-04-19 21:46:07,747 INFO DRY-RUN: delete_indices: application-cron-2020.04.05 with arguments: {} 2020-04-19 21:46:07,747 INFO Action ID: 1, "delete_indices" completed. 2020-04-19 21:46:07,747 INFO Job completed.
Execute an action.
$ curator --config ~/.curator/client-development-cluster.yml ~/.curator/action-delete-old-application-cron-indices.yml
2020-04-19 21:46:29,340 INFO Preparing Action ID: 1, "delete_indices" 2020-04-19 21:46:29,340 INFO Creating client object and testing connection 2020-04-19 21:46:29,343 INFO Instantiating client object 2020-04-19 21:46:29,344 INFO Testing client connectivity 2020-04-19 21:46:29,350 INFO Successfully created Elasticsearch client object with provided settings 2020-04-19 21:46:29,354 INFO Trying Action ID: 1, "delete_indices": No description given 2020-04-19 21:46:29,383 INFO Deleting 5 selected indices: ['application-cron-2020.04.01', 'application-cron-2020.04.05', 'application-cron-2020.04.04', 'application-cron-2020.04.03', 'application-cron-2020.04.02'] 2020-04-19 21:46:29,383 INFO ---deleting index application-cron-2020.04.01 2020-04-19 21:46:29,383 INFO ---deleting index application-cron-2020.04.05 2020-04-19 21:46:29,383 INFO ---deleting index application-cron-2020.04.04 2020-04-19 21:46:29,383 INFO ---deleting index application-cron-2020.04.03 2020-04-19 21:46:29,383 INFO ---deleting index application-cron-2020.04.02 2020-04-19 21:46:29,527 INFO Action ID: 1, "delete_indices" completed. 2020-04-19 21:46:29,527 INFO Job completed.
Inspect indices.
$ curl "http://localhost:9200/_cat/indices?h=index&s=index"
application-cron-2020.04.06 application-cron-2020.04.07 application-cron-2020.04.08 application-cron-2020.04.09 application-cron-2020.04.10 application-cron-2020.04.11 application-cron-2020.04.12 application-cron-2020.04.13 application-cron-2020.04.14 application-cron-2020.04.15 application-cron-2020.04.17 application-cron-2020.04.18 application-cron-2020.04.19
Scheduled execution
Use cron
to schedule curator
execution.
$ crontab -l
15 07 * * MON /usr/bin/curator --dry-run --config /home/curator/.curator/client-development-cluster.yml /home/curator/.curator/action-delete-old-application-cron-indices.yml && /usr/bin/curator --config /home/curator/.curator/client-development-cluster.yml /home/curator/.curator/action-delete-old-application-cron-indices.yml
Command-line interface
You can use a command-line interface to perform tests ad-hoc.
$ curator_cli --dry-run --config ~/.curator/client-development-cluster.yml delete_indices --filter_list '[{"filtertype":"age","source":"name","direction":"older","timestring":"%Y.%m.%d","unit":"days","unit_count":14}, {"filtertype":"pattern","kind":"prefix","value":"application-cron-"}]'
2020-04-19 21:54:04,537 INFO Instantiating client object 2020-04-19 21:54:04,538 INFO Testing client connectivity 2020-04-19 21:54:04,542 INFO Successfully created Elasticsearch client object with provided settings 2020-04-19 21:54:04,577 INFO DRY-RUN MODE. No changes will be made. 2020-04-19 21:54:04,577 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices". 2020-04-19 21:54:04,577 INFO DRY-RUN: delete_indices: application-cron-2020.04.01 with arguments: {} 2020-04-19 21:54:04,578 INFO DRY-RUN: delete_indices: application-cron-2020.04.02 with arguments: {} 2020-04-19 21:54:04,578 INFO DRY-RUN: delete_indices: application-cron-2020.04.03 with arguments: {} 2020-04-19 21:54:04,578 INFO DRY-RUN: delete_indices: application-cron-2020.04.04 with arguments: {} 2020-04-19 21:54:04,578 INFO DRY-RUN: delete_indices: application-cron-2020.04.05 with arguments: {} 2020-04-19 21:54:04,579 INFO "delete_indices" action completed.
You can use a command-line interface to execute specific actions ad-hoc.
$ curator_cli --host 127.0.0.1 delete_indices --filter_list '[{"filtertype":"age","source":"name","direction":"older","timestring":"%Y.%m.%d","unit":"days","unit_count":14}, {"filtertype":"pattern","kind":"prefix","value":"application-cron-"}]'
2020-04-19 21:54:12,361 INFO Instantiating client object 2020-04-19 21:54:12,362 INFO Testing client connectivity 2020-04-19 21:54:12,366 INFO Successfully created Elasticsearch client object with provided settings 2020-04-19 21:54:12,390 INFO Deleting 5 selected indices: ['application-cron-2020.04.01', 'application-cron-2020.04.04', 'application-cron-2020.04.03', 'application-cron-2020.04.05', 'application-cron-2020.04.02'] 2020-04-19 21:54:12,391 INFO ---deleting index application-cron-2020.04.01 2020-04-19 21:54:12,391 INFO ---deleting index application-cron-2020.04.04 2020-04-19 21:54:12,391 INFO ---deleting index application-cron-2020.04.03 2020-04-19 21:54:12,391 INFO ---deleting index application-cron-2020.04.05 2020-04-19 21:54:12,392 INFO ---deleting index application-cron-2020.04.02 2020-04-19 21:54:12,510 INFO "delete_indices" action completed.