Metron - Installing Elasticsearch and Kibana on Ubuntu

[#1]

For this tutorial we will be using Ubuntu 14.04.5. This setup can further be leveraged with Apache Metron (Incubating). Additional installation instructions for Metron core will be provided in another article.

Install Elasticsearch

First we'll install Elasticsearch 2.4. You'll need the following prerequisites:

• wget
• apt-transport-https
• Java

You can install them by logging into your ES node and executing the following:

sudo apt-get update
sudo apt-get install -y wget apt-transport-https
# If using oracle jdk 8
sudo apt-get install -y software-properties-common
sudo apt-add-repository -y ppa:webupd8team/java
sudo apt-get update
echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | sudo
debconf-set-selections
sudo apt-get install -y oracle-java8-installer

Now let's install Elasticsearch. Run the following commands on the node where you want to install ES.

# Get the Elasticsearch packages
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# Add the Elasticsearch packages to apt
echo "deb https://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
# Install Elasticsearch
sudo apt-get update && sudo apt-get install elasticsearch
# Configure for automatic start on system boot for System V systems
sudo update-rc.d elasticsearch defaults 95 10
# Start Elasticsearch
sudo service elasticsearch start

If you're running this in Docker, you'll also want to run the following before starting the es service:

# Setup networking
echo 'network.host: 0.0.0.0' >> /etc/elasticsearch/elasticsearch.yml

Check that Elasticsearch is running. Go to http://$ELASTICSEARCH_HOST:9200 and verify you see something like the following:

{
  "name" : "Saturnyne",
  "cluster_name" : "metron",
  "cluster_uuid" : "F-m2WjlDSAu_0TTCqXki1w",
  "version" : {
    "number" : "2.4.4",
    "build_hash" : "fcbb46dfd45562a9cf00c604b30849a6dec6b017",
    "build_timestamp" : "2017-01-03T11:33:16Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.2"
  },
  "tagline" : "You Know, for Search"
}

Install Kibana

Now we'll install Kibana 4.5.3 on Ubuntu 14.04.5. First you should have the following prerequisites:

• wget

You can install them by logging into your Kibana node and executing the following:

sudo apt-get update
sudo apt-get install -y wget

Now let's install Kibana. Run the following commands on the node where you want to install Kibana.

# Get the Elasticsearch/Kibana packages
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# Add the Kibana packages to apt
echo "deb http://packages.elastic.co/kibana/4.5/debian stable main" | sudo tee -a /etc/apt/sources.list
# Install Kibana
sudo apt-get update && sudo apt-get install kibana
# Configure for automatic start on system boot
sudo update-rc.d kibana defaults 95 10
# Configure Kibana for Elasticsearch host:port
# Note: set the host and port accordingly to point to your Elasticsearch host from the installation above.
sed -ri "s;^(\#\s*)?(elasticsearch\.url:).*;\2 'http://elasticsearch:9200';" /opt/kibana/config/kibana.yml
# Start Kibana
export PATH=/opt/kibana/bin:$PATH
kibana

That should be it. Now you should be able to go to http://$KIBANA_HOST:5601 and see the Kibana dashboard.

Extras

Setting up Docker

If you're looking to get a quick demo environment running, you can follow these steps to run this example in Docker. For this part we'll be using Docker for Mac on Mac OSX 10.12.

1. Setup Docker for Mac - https://docs.docker.com/docker-for-mac/
2. Add Docker files that setup the images for Elasticsearch and Kibana. See the attached ubuntu-dockerfile.tar.gz tarball.ubuntu-dockerfile.tar.gz
3. Untar the bundle. You should have 2 directories: ubuntu-elasticsearch and ubuntu-kibana
4. Build the Docker images (you'll want this so you can reuse them). Replace $YOUR_TAG_NAME with whatever you like. Don't forget the period '.' at the end of the line.
   1. docker build -f ubuntu-elasticsearch/Dockerfile -t $YOUR_TAG_NAME/ubuntu-elasticsearch .
   2. docker build -f ubuntu-kibana/Dockerfile -t $YOUR_TAG_NAME/ubuntu-kibana .
5. Run the containers. The container names will be "es" and "kibana."
   1. docker run -p 9200:9200 -P --name es -dit $YOUR_TAG_NAME/ubuntu-elasticsearch
   2. docker run -p 5601:5601 -P --name kibana -dit $YOUR_TAG_NAME/ubuntu-kibana
   3. Note: if you need to re-run for any reason (failed startup, for instance), kill and remove the containers, e.g.
      1. docker kill es
      2. docker rm es
6. Now login to the ES container and follow the ES install steps from above.
   1. docker exec -it es /bin/bash
   2. logout
7. Login to the Kibana container and follow the Kibana install steps from above.
   1. docker exec -it kibana /bin/bash
   2. logout

You should now have two running Docker containers that you are able to connect to from your localhost.

• http://localhost:9200
• http://localhost:5601

Note: There are currently limitations with Docker for Mac networking. Alternatively, you could use docker-machine for a more robust example.

Reference

• https://www.elastic.co/guide/en/elasticsearch/reference/2.4/setup-repositories.html
• https://www.elastic.co/guide/en/kibana/4.5/setup-repositories.html
• https://docs.docker.com/docker-for-mac/
• https://docs.docker.com/machine/overview/

[#2]

Hi @Michael Miklavcic, thanks for the article.

On my Ubuntu 14.04 openstack cluster, I was unable to start elasticsearch service after following the steps. It was failing with and error saying NoSuchFileException: /usr/share/elasticsearch/config. I had to follow the workaround in this article in order to have the services started successfully.

I did not have any issues with the Kibana install. It worked fine.

[#3]

Thanks for checking this out!