How to delete old Elasticsearch indices

Delete old Elasticsearch indices using Elasticsearch Curator utility.

Installation

Install required dependencies.

$ sudo apt install gnupg2 apt-transport-https curl

Add a repository key.

$ curl --silent https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Add an Elasticsearch Curator repository.

$ echo "deb [arch=amd64] https://packages.elastic.co/curator/5/debian9 stable main" | sudo tee /etc/apt/sources.list.d/curator.list

Update package index.

$ sudo apt update

Install Elasticsearch Curator utility.

$ sudo apt install elasticsearch-curator

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  elasticsearch-curator
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 20.4 MB of archives.
After this operation, 78.9 MB of additional disk space will be used.
Get:1 https://packages.elastic.co/curator/5/debian9 stable/main amd64 elasticsearch-curator amd64 5.8.1 [20.4 MB]
Fetched 20.4 MB in 3s (6,016 kB/s)
Selecting previously unselected package elasticsearch-curator.
(Reading database ... 43831 files and directories currently installed.)
Preparing to unpack .../elasticsearch-curator_5.8.1_amd64.deb ...
Unpacking elasticsearch-curator (5.8.1) ...
Setting up elasticsearch-curator (5.8.1) ...

Configuration

Create a configuration directory.

$ mkdir .curator

Create .curator/client-development-cluster.yml client configuration file to access the development cluster.

---
client:
  hosts:
    - 192.0.2.112
  port: 9200
  url_prefix:
  use_ssl:
  certificate:
  client_cert:
  client_key:
  ssl_no_validate:
  http_auth:
  timeout: 90
  master_only:
logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Empty logfile will result in logging to the standard output.

Create .curator/action-delete-old-application-cron-indices.yml action configuration file to delete old application-cron-* indices.

actions:
  1:
    action: delete_indices
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 14
      exclude:
    - filtertype: pattern
      kind: prefix
      value: application-cron-

Execution

Inspect indices.

$ curl "http://localhost:9200/_cat/indices?h=index&s=index"

application-cron-2020.04.01
application-cron-2020.04.02
application-cron-2020.04.03
application-cron-2020.04.04
application-cron-2020.04.05
application-cron-2020.04.06
application-cron-2020.04.07
application-cron-2020.04.08
application-cron-2020.04.09
application-cron-2020.04.10
application-cron-2020.04.11
application-cron-2020.04.12
application-cron-2020.04.13
application-cron-2020.04.14
application-cron-2020.04.15
application-cron-2020.04.17
application-cron-2020.04.18
application-cron-2020.04.19

Execute an action, but do not perform any changes.

$ curator --dry-run --config ~/.curator/client-development-cluster.yml ~/.curator/action-delete-old-application-cron-indices.yml

2020-04-19 21:46:07,693 INFO      Preparing Action ID: 1, "delete_indices"
2020-04-19 21:46:07,693 INFO      Creating client object and testing connection
2020-04-19 21:46:07,696 INFO      Instantiating client object
2020-04-19 21:46:07,696 INFO      Testing client connectivity
2020-04-19 21:46:07,707 INFO      Successfully created Elasticsearch client object with provided settings
2020-04-19 21:46:07,710 INFO      Trying Action ID: 1, "delete_indices": No description given
2020-04-19 21:46:07,746 INFO      DRY-RUN MODE.  No changes will be made.
2020-04-19 21:46:07,746 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2020-04-19 21:46:07,746 INFO      DRY-RUN: delete_indices: application-cron-2020.04.01 with arguments: {}
2020-04-19 21:46:07,747 INFO      DRY-RUN: delete_indices: application-cron-2020.04.02 with arguments: {}
2020-04-19 21:46:07,747 INFO      DRY-RUN: delete_indices: application-cron-2020.04.03 with arguments: {}
2020-04-19 21:46:07,747 INFO      DRY-RUN: delete_indices: application-cron-2020.04.04 with arguments: {}
2020-04-19 21:46:07,747 INFO      DRY-RUN: delete_indices: application-cron-2020.04.05 with arguments: {}
2020-04-19 21:46:07,747 INFO      Action ID: 1, "delete_indices" completed.
2020-04-19 21:46:07,747 INFO      Job completed.

Execute an action.

$ curator --config ~/.curator/client-development-cluster.yml ~/.curator/action-delete-old-application-cron-indices.yml

2020-04-19 21:46:29,340 INFO      Preparing Action ID: 1, "delete_indices"
2020-04-19 21:46:29,340 INFO      Creating client object and testing connection
2020-04-19 21:46:29,343 INFO      Instantiating client object
2020-04-19 21:46:29,344 INFO      Testing client connectivity
2020-04-19 21:46:29,350 INFO      Successfully created Elasticsearch client object with provided settings
2020-04-19 21:46:29,354 INFO      Trying Action ID: 1, "delete_indices": No description given
2020-04-19 21:46:29,383 INFO      Deleting 5 selected indices: ['application-cron-2020.04.01', 'application-cron-2020.04.05', 'application-cron-2020.04.04', 'application-cron-2020.04.03', 'application-cron-2020.04.02']
2020-04-19 21:46:29,383 INFO      ---deleting index application-cron-2020.04.01
2020-04-19 21:46:29,383 INFO      ---deleting index application-cron-2020.04.05
2020-04-19 21:46:29,383 INFO      ---deleting index application-cron-2020.04.04
2020-04-19 21:46:29,383 INFO      ---deleting index application-cron-2020.04.03
2020-04-19 21:46:29,383 INFO      ---deleting index application-cron-2020.04.02
2020-04-19 21:46:29,527 INFO      Action ID: 1, "delete_indices" completed.
2020-04-19 21:46:29,527 INFO      Job completed.

Inspect indices.

$ curl "http://localhost:9200/_cat/indices?h=index&s=index"

application-cron-2020.04.06
application-cron-2020.04.07
application-cron-2020.04.08
application-cron-2020.04.09
application-cron-2020.04.10
application-cron-2020.04.11
application-cron-2020.04.12
application-cron-2020.04.13
application-cron-2020.04.14
application-cron-2020.04.15
application-cron-2020.04.17
application-cron-2020.04.18
application-cron-2020.04.19

Scheduled execution

Use cron to schedule curator execution.

$ crontab -l

15 07 * * MON  /usr/bin/curator --dry-run --config /home/curator/.curator/client-development-cluster.yml /home/curator/.curator/action-delete-old-application-cron-indices.yml && /usr/bin/curator --config /home/curator/.curator/client-development-cluster.yml /home/curator/.curator/action-delete-old-application-cron-indices.yml

Please read about the master_only configuration option as it will help you during the automation process.

Command-line interface

You can use a command-line interface to perform tests ad-hoc.

$ curator_cli --dry-run --config ~/.curator/client-development-cluster.yml delete_indices --filter_list '[{"filtertype":"age","source":"name","direction":"older","timestring":"%Y.%m.%d","unit":"days","unit_count":14}, {"filtertype":"pattern","kind":"prefix","value":"application-cron-"}]'

2020-04-19 21:54:04,537 INFO      Instantiating client object
2020-04-19 21:54:04,538 INFO      Testing client connectivity
2020-04-19 21:54:04,542 INFO      Successfully created Elasticsearch client object with provided settings
2020-04-19 21:54:04,577 INFO      DRY-RUN MODE.  No changes will be made.
2020-04-19 21:54:04,577 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2020-04-19 21:54:04,577 INFO      DRY-RUN: delete_indices: application-cron-2020.04.01 with arguments: {}
2020-04-19 21:54:04,578 INFO      DRY-RUN: delete_indices: application-cron-2020.04.02 with arguments: {}
2020-04-19 21:54:04,578 INFO      DRY-RUN: delete_indices: application-cron-2020.04.03 with arguments: {}
2020-04-19 21:54:04,578 INFO      DRY-RUN: delete_indices: application-cron-2020.04.04 with arguments: {}
2020-04-19 21:54:04,578 INFO      DRY-RUN: delete_indices: application-cron-2020.04.05 with arguments: {}
2020-04-19 21:54:04,579 INFO      "delete_indices" action completed.

You can use a command-line interface to execute specific actions ad-hoc.

$ curator_cli --host 127.0.0.1  delete_indices --filter_list '[{"filtertype":"age","source":"name","direction":"older","timestring":"%Y.%m.%d","unit":"days","unit_count":14}, {"filtertype":"pattern","kind":"prefix","value":"application-cron-"}]'

2020-04-19 21:54:12,361 INFO      Instantiating client object
2020-04-19 21:54:12,362 INFO      Testing client connectivity
2020-04-19 21:54:12,366 INFO      Successfully created Elasticsearch client object with provided settings
2020-04-19 21:54:12,390 INFO      Deleting 5 selected indices: ['application-cron-2020.04.01', 'application-cron-2020.04.04', 'application-cron-2020.04.03', 'application-cron-2020.04.05', 'application-cron-2020.04.02']
2020-04-19 21:54:12,391 INFO      ---deleting index application-cron-2020.04.01
2020-04-19 21:54:12,391 INFO      ---deleting index application-cron-2020.04.04
2020-04-19 21:54:12,391 INFO      ---deleting index application-cron-2020.04.03
2020-04-19 21:54:12,391 INFO      ---deleting index application-cron-2020.04.05
2020-04-19 21:54:12,392 INFO      ---deleting index application-cron-2020.04.02
2020-04-19 21:54:12,510 INFO      "delete_indices" action completed.

Installation#

Configuration#

Execution#

Scheduled execution#

Command-line interface#

Installation

Configuration

Execution

Scheduled execution

Command-line interface