What is log aggregation?
In software development, it is common practice for developers to integrate functionalities that allows the application to generate various logs when certain events are triggered. This enables debugging when errors occur and performance monitoring as the application is in use. The information is recorded in a log file on the host operating system.
Often, however, these log files require access to the command line and manual interrogation to get to the relevant information. Moreover, an increase in infrastructure complexity and in the number of applications generates the need for a centralised overview of events pertaining (but not limited to) performance and security.
This is achieved by simplifying the information flow from all the different components to an easy-to-use interface where various log files are used as inputs and various rules are enforced to parse those inputs in order to highlight the most relevant information like application and infrastructure errors, performance data and security-related events. This is known as Log aggregation and it is used by organisations for complex data analysis, log queries, and security monitoring.
Log aggregation in Bunnyshell
In Bunnyshell this feature makes use of the Kibana log aggregator and supports the input of custom log files by specifying their paths on the OS filesystem.
One example is the Apache error and access log which has the following path in Ubuntu: /var/log/apache2/error.log, respectively /var/log/apache2/access.log.
By adding these paths in the Log Aggregation dashboard, all error logs from the Apache webserver would be visible from the Bunnyshell interface as well as the access logs necessary for security audits.
In real-life situations, multiple logs are necessary to have a complete overview of one or more applications using complex infrastructures such that developers can quickly identify error sources and performance bottlenecks and for security specialists to detect possible vulnerabilities.
With Log Aggregation the complexity of having to parse multiple log sources is reduced by bringing them all together in a common web interface that offers multiple tools like indexing, collecting, alerts, etc.