Five reasons you need to be thinking about logs

Logs were previously the last-resort debugging tool that a developer or Ops pro turned to when a serious issue arose. They were seen as complex data streams that only the most sophisticated and knowledgeable of engineers could understand. However, today, thanks to log-management innovations like real-time analysis, parsing and event visualizations, logs are now used for a wide range of use cases. Logs are quickly becoming invaluable to a broader set of users.

So why exactly should you be thinking about logs? And why are so many organizations, which already have a slew of monitoring tools, turning to log data to help understand what is happening across systems, applications and end users?

1. Nobody ever got fired for keeping logs! Like the old adage “Nobody ever got fired for buying IBM,” the same can be said for collecting and archiving your logs.

Logs maintain a paper trail of activity on your systems and can capture information pertaining to security, system performance, resource usage, business transactions and, frankly, whatever data you decide to write into them. Thus, it is a smart business decision to keep log data around so you can look back historically over system trends in the case of system performance issues or specific user actions in the case of security-related data.

Whereas, in the past, managing large volumes of log data was expensive and complex, modern cloud-based solutions make keeping a historic record of your logs dead simple. Furthermore, with the advent of cheap storage services suitable for archiving data (such as Amazon S3 or Azure Block Storage), it can be extremely cost-effective and efficient to simply archive your log data for safekeeping. Even if keeping log data is not needed to meet compliance or security standards, it’s almost always a good idea to hang onto it, just in case you ever happen to need it.

2. Logs give you the full picture, front to back and top to bottom. Logs already contain a ton of useful information. Before you add any log events from your own applications, your system components (operating systems, Web servers, application servers, databases, load balancers, routers, firewalls, etc.) already produce log data that contain valuable information on performance, response time, and the time and identity of those accessing your system.

Adding your own log events into the mix from your software applications can give you the entire picture—capturing data at every layer of the software stack (top to bottom) and all the way from your front-end components (browsers and mobile apps) through your back-end architecture (Web servers, app servers and databases). It covers all of your bases.

3. All your data is in one place. You can use log data as a single data stream that provides views into different aspects of your system. Log data can capture error events and exceptions, resource usage information, application performance information, feature usage information, and more. Thus, you can consolidate your tooling and reduce the requirements for numerous point solutions to give you a view into what is happening across your systems.

4. Logs maintain the evidence. Metrics dashboards based on log data have an important property that does not exist when creating dashboards using most other approaches. Put simply: Your logs maintain the evidence! This means that if there is a spike in the number of signups or an increase in your customers using a particular feature, you can quickly validate what caused that change.

While validating your data is something that can be particularly painstaking when using application performance-monitoring (APM) software, Web analytics tools or homegrown metrics dashboards as they generally only record aggregations of your data, advanced log-management tools provide the heightened validation today’s data-driven organizations need. It can be very useful when there is a sudden change in a KPI where you need to quickly understand the root cause of the change such that you can react appropriately.

5. You can’t always instrument the cloud. Systems have fundamentally changed over the past few years. If you think back even five years ago, systems were mainly on-premise or in a data center where you had complete control of your environment.

However, today it is becoming more and more common for systems to be deployed entirely in the cloud, or to at least make use of numerous cloud components. For cloud-based systems, full instrumentation is often not an option because many parts of the stack may no longer be under your full control. The access required to apply APM tools may not be available.

For example, with IaaS you only have access from the operating system and up, i.e. the operating system, the middleware and application tier. The provider will control everything below the operating system such as the hypervisor layer, the hardware and the network.

For those using PaaS, the situation is even more constrained because PaaS vendors tend to manage the OS and middleware components on behalf of their users. This limits your access to the application tier from an instrumentation perspective.

Finally, with SaaS components, you generally do not have any ability to instrument, and you are required to rely on any instrumentation APIs or endpoints provided by the SaaS vendor.

As a result, it is common for traditional APM solutions, which have claimed in the past to give 100% end-to-end visibility for on-premise systems, to only provide a fraction of that visibility for cloud-based systems. Alternative approaches are required to give visibility into cloud-based components that otherwise can become black boxes from a performance or system monitoring perspective. The existing log data and API data streams provided by cloud vendors can be analyzed by log-management and analytics solutions to provide real-time KPI dashboards. Tapping log data, you can get deep visibility into otherwise black box components.

Now accessible to technical and non-technical users, logs are used across organizations for debugging and troubleshooting, for production monitoring of live environments, and for mobile, Web and business analytics. Modern cloud-based log-management solutions allow you to easily interrogate your log data, visualize it and aggregate it into meaningful metrics dashboards. This provides visibility into many different aspects of your systems and business.

Logs have become a critical component of every organization, and you can’t afford to ignore them any longer.