Make sense of your logs with Elastic Search - part 5: be a data wizard with Kibana

Configuring Kibana

Like most of the Elastic Search modules, Kibana can be configured with a single file, called kibana.yml. In my example Git repo, you can find this file under kibana/config.

The most important goal of Kibana's configuration file is, in our example, telling Kibana where it can find Elastic Search.

server.name: kibana
server.host: "0"
elasticsearch.url: http://elasticsearch:9200

When you're not using Docker, replace http://elasticsearch:9200 with your own value (for example http://localhost:9200).

Kibana behind a reverse proxy

When running behind a reverse proxy, you'll need to tell Kibana under which context root it will run.

For example, if you want to run Kibana behind http://my-reverse-proxy/kibana, add

server.basePath: "/kibana"

Also, when experiencing problems, have a look at this post.

Tell Logstash which indexes to visualize

Open up Kibana in a browser. In my example, the url is http://localhost:5761/.

The first step is to tell Kibana which indexes to show. In the video below, we add all indexes that begin with log.

When we've added the Logstash indexes, we go back to the Explore screen and see... nothing! Why? Shouldn't we see data? Yes, but have a look at the time filter, on the top right. Only the logs of the past 15 minutes are shown. When we change that time filter to a bigger period, our logs pop up!

Search through the logs

Select fields

We can select the fields that we would like to see in our list.

Filter

At the top left of the screen, you have a button to add filters.

Interested what happened right before or after a specific log line?

What happened right before a certain exception? Click on the interesting log line and choose "Show surrounding documents".

Showing off with fancy graphs

It's showtime. In the video below, I show how to create a simple graph.

This example is not that impressing. With enough data, you can create really useful graphs, to prove slowdowns, detect use cases that introduce exceptions, determine between which times you're app is used most, how your app is used, ...

In the past, my team has found the cause of a hard to find memory leak, thanks to good line charts.

A good graph is great to prove a theory, or just have an idea what's going on.

You can also filter in these graphs.

Stay informed with a dashboard

You can create a dashboard of your visualisations. Ideal to get up at a big screen, for your whole team to see.

If your logs get sent to Elastic Search immediately, be sure to hit the auto-refresh button. This gives you a real-time monitoring view!

Migrating between Kibanas

Need to migrate your visualisations and dashboards to another Kibana? You can export and import the saved objects.

Do the logs make sense now?

It's been a wild ride, but we've made it: our logs get mapped into filterable, groupable, sortable fields and we have a great tool to query them. We can discover new insights with visualisations.

What we've done is the tip of the iceberg, though. What if we would combine the logs with data in the database? What if we would add server metrics? The world of Elastic Search is a vast and evergrowing world. Have fun discovering!