WSO2 API Manager logs Monitoring with EFK

https://www.google.com/url?sa=i&url=https%3A%2F%2Fwww.bitlyft.com%2Fwhat-is-security-logging-and-monitoring%2F&psig=AOvVaw1WfrZXvJDO7fkBnFdQWDMu&ust=1619441518425000&source=images&cd=vfe&ved=0CAIQjRxqFwoTCPCP1vm3mfACFQAAAAAdAAAAABAP

Hello Everyone! 👋🏻

Today, we gonna see how we can monitor different WSO2 log files in an external visualizing interface such that Kibana.

If we consider the log files available in a single pack of WSO2 API Manager, there are a few such that wso2carbon logs , http_access_ logs , http_access_<date> logs and audit logs etc. Also, if the WSO2 API Manager has been deployed as a distributed deployment with WSO2 Analytics, then you may face a challenge to monitor all the logs in each server because there will be many for each components across all your platforms.

So, what can we do to make it easy? 🤔

https://www.google.com/url?sa=i&url=https%3A%2F%2Fwww.noslangues-ourlanguages.gc.ca%2Fen%2Fblogue-blog%2Ftroublesome-verb-pairs-eng%3Fwbdisable%3Dtrue&psig=AOvVaw2IZReKt_iPLq0Iewuxt9c0&ust=1619442349815000&source=images&cd=vfe&ved=0CAIQjRxqFwoTCKDMkpq7mfACFQAAAAAdAAAAABAD

In this article I am going to explain you one possible approach for the above issue. 🤗

This approach is with EFK Stack implementation which is a popular centralized logging solution. As the name implies, the EFK stack has three components, Elasticsearch, Fluentd, and Kibana.

Elasticsearch is a real-time, distributed, and scalable search engine which allows for full-text and structured search, as well as analytics. It is commonly used to index and search through large volumes of log data, but can also be used to search many different kinds of documents.

Kibana is a powerful data visualization frontend and dashboard for Elasticsearch. Kibana allows you to explore your Elasticsearch log data through a web interface.

Fluentd is used to collect, transform, and ship log data to the Elasticsearch backend.

Here, I will share fluentd configurations for wso2carbon logs, http_access_. logs, http_access_<date> logs, audit logs and wso2-gw-error logs in WSO2 API Manager 2.6 version.

For the testing purposes, I used this repository to run EFK stack in my local machine. If anyone would like to use this EFK Stack, following is the sample docker-compose.yml that I used.

docker-compose.yml

Now, let’s move on to the sample fluentd configuration for each log files. You need to add the below to your fluent.conf file.

fluent.conf

When configuring a log file in fluent.conf file, it involves two parts such that source and the match. The source tag is where the all data come from. So you need to give a type first and in our case it is tail.

The in_tail Input plugin allows fluentd to read events from the tail of text files. Its behavior is similar to the tail -F command.

Then we need to specify the log file path and the path parameter is used to this. pos_file is for fluend agent to keep a record of the original file read. tag is the name event we are creating here.

The match which is the one that tell fluentd what to do. Each match must include a match pattern and a @type parameter. Only events with the tag matching the pattern will be sent to the output destination. Refer to this for more information about all these parameters.

Now we are good to run the EFK stack.

After the EFK stack started, go to the Kibana dashboard (Ex: http:localhost:5601/app/kibana ) and create an index pattern. To create an Index pattern, go to the Management -> Index patterns -> Create index pattern -> Search {tag name} -> create index. You will need to create separate indices for each logs.

Then you will be able to see the logs in the Discover according to the created index pattern. An example is as follows.

Audit.log

Thank you for reading!

Be safe!!

Reference:
[1] https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes
[2] https://docs.fluentd.org/
[3] https://kapilanishantha.medium.com/wso2-log-monitoring-with-fluend-8812ae189393

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store