Configure Elasticsearch, Logstash and Kibana to use X-Pack and SSL

X-Pack is included in free Basic version of Elasticsearch and you should use it. Here are my notes how i configured Elasticsearch, Logstash and Kibana to use X-Pack and SSL with Ubuntu.

Create Certificates for SSL

I used my own private CA to create certs.

Elasticsearch certs must be placed in Elasticsearch configuration folder (/etc/elasticsearch/). Elasticsearch certificate common name must match ELASTICSEARCHHOST.DOMAIN

Kibana and Logstash certs can be anywhere but i still placed them into their config folders (/etc/logstash/, /etc/kibana/)

Elasticsearch

mkdir /etc/elasticsearch/certs
mv elasticsearch.key /etc/elasticsearch/certs/
mv elasticsearch.crt /etc/elasticsearch/certs/
cp MYCA.ca /etc/elasticsearch/certs/
chmod 640 /etc/elasticsearch/certs/*
chown root:elasticsearch -R /etc/elasticsearch/certs

Kibana

mkdir /etc/kibana/certs
mv kibana.key /etc/kibana/certs/
mv kibana.crt /etc/kibana/certs/
cp MYCA.ca /etc/kibana/certs/
chmod 640 /etc/kibana/certs/*
chown root:kibana -R /etc/kibana/certs

Logstash

mkdir /etc/logstash/certs
cp MYCA.ca /etc/logstash/certs/
chmod 640 /etc/logstash/certs/*
chown root:logstash -R /etc/logstash/certs

Configure Elasticsearch to use X-Pack

vi /etc/elasticsearch/elasticsearch.yml

network.host: "ELASTICSEARCHHOST.DOMAIN"
discovery.seed_hosts: ["ELASTICSEARCHHOST.DOMAIN"]
cluster.initial_master_nodes: ["ELASTICSEARCHHOST.DOMAIN"]
xpack.monitoring.collection.enabled: true
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.key: certs/elasticsearch.key
xpack.security.http.ssl.certificate: certs/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: certs/MYCA.ca
xpack.security.transport.ssl.key: certs/elasticsearch.key
xpack.security.transport.ssl.certificate: certs/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: certs/MYCA.ca

systemctl start elasticsearch

Setup passwords

Setup passwords for default users. You need kibana and logstash_system passwords later. You will login to kibana with elastic user.

run /usr/share/elasticsearch/bin/elasticsearch-setup-passwords

https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-xpack.html

Configure Kibana to use X-Pack

vi /etc/kibana/kibana.yml

server.name: "KIBANA.DOMAIN"
elasticsearch.hosts: ["https://ELASTICSEARCHHOST.DOMAIN:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: PASSWORD
server.ssl.enabled: true
server.ssl.certificate: "/etc/kibana/certs/kibana.crt"
server.ssl.key: "/etc/kibana/certs/kibana.key"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/MYCA.ca" ]
elasticsearch.ssl.verificationMode: full

systemctl start kibana

Create users for Logstash output/indexes

Login to Kibana and select Management on the left panel. There should be Security section now.

Create logstash output role

• Role name: logstash_output
• Cluster privileges: manage_index_templates  monitor

Create role for index

• Role name: INDEXPURPOSE_index
• Indices: INDEXPURPOSE-*
  • Privileges: create_index write

Create user for Logstash INDEXPURPOSE index output

• Username: INDEXPURPOSEUSER
• Password: INDEXPURPOSEPASSWORD
• Roles: logstash_output INDEXPURPOSE_index

https://www.elastic.co/guide/en/kibana/current/using-kibana-with-security.html

Configure Logstash to use X-Pack

vi /etc/logstash/logstash.yml

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: LOGSTASH_SYSTEMPASSWORD
xpack.monitoring.elasticsearch.hosts: ["https://ELASTICSEARCHHOST.DOMAIN:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/etc/logstash/certs/MYCA.ca"
xpack.monitoring.elasticsearch.sniffing: true
xpack.monitoring.collection.interval: 10s
xpack.monitoring.collection.pipeline.details.enabled: true

Add https and cacert to output

output {
  elasticsearch {
    hosts => ["https://ELASTICSEARCHHOST.DOMAIN:9200"]
    cacert => '/etc/logstash/certs/MYCA.ca'
    index => "INDEXPURPOSE-%{+YYYY.MM}"
    user => "INDEXPURPOSEUSER"
    password => "INDEXPURPOSEPASSWORD"
  }
}

https://www.elastic.co/guide/en/logstash/current/setup-xpack.html