Tag Archives: ELK

Elasticsearch Curator

Posted on by
Reply

To secure HDD space somehow, Curator can help you. The installation of Curator on Ubuntu is very simple. Because, however, the configuration way was changed, this post is helpful for you.

Elasticsearch Curator Installation

# sudo apt-get -y install python-pip
# sudo pip install elasticsearch-curator

 

Make a schedule on Crontab

As I mentioned early, if you used Curator over 4.0 version, you should configure as below. It’s not mandatory of the configuration path, “/etc/curator/”.

30 0 * * * /usr/local/bin/curator --config=/etc/curator/curator.yml /etc/curator/del_elastic_indices.yml

# cat /etc/curator/curator.yml

client:
  hosts:
    - 127.0.0.1
  port: 9200
  use_ssl: False
  ssl_no_validate: False
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile: /var/log/curator.log
  logformat: default

# cat /etc/curator/del_elastic_indices.yml

actions:
  1:
    action: delete_indices
    description: "Delete selected indices"
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

To check the result of the curator execution just before, you can ask Elasticseach by following RESTFull URL.

# curl -XGET 'http://localhost:9200/_cat/indices/*' -s
yellow open filebeat-2016.07.13 5 1 32 0 472.4kb 472.4kb
yellow open filebeat-2016.07.12 5 1 4 0 90.5kb 90.5kb
yellow open filebeat-2016.07.15 5 1 2980542 0 2.6gb 2.6gb
yellow open filebeat-2016.07.14 5 1 2604353 0 2.1gb 2.1gb
yellow open .kibana 1 1 103 0 89.3kb 89.3kb
yellow open filebeat-2016.07.11 5 1 3 0 54.1kb 54.1kb

Additionally, you can also check the log of the execution into a log file we specified in a curator configuration file, curator.yml.

# tail -n 5 /var/log/curator.log
2016-07-15 14:56:59,285 INFO Deleting selected indices
2016-07-15 14:56:59,285 INFO ---deleting index filebeat-2016.07.08
2016-07-15 14:56:59,285 INFO ---deleting index filebeat-2016.07.07
2016-07-15 14:56:59,326 INFO DELETE http://127.0.0.1:9200/filebeat-2016.07.07,filebeat-2016.07.08?master_timeout=30s [status:200 request:0.041s]

For your information, here is a cron job for old curator version.

30 0 * * * /usr/local/bin/curator --host 127.0.0.1 delete indices --older-than 7 --timestring \%Y.\%m.\%d --time-unit days