Whitepapers

Simplifying Event Log Management

Event Logs are the window into what’s happening on your servers.

View PDF

So you have servers. Probably lots of them – could be three, ten, a hundred or more. Each of them contains a wealth of information about the security, performance and reliability of your users, servers and the network they reside on.

Event Logs are the window into what’s happening on your servers. They are commonly used for:

  • Identifying behaviors
  • Ensuring security
  • Finding problems
  • Proving compliance
  • Quantifying performance

But when you think of Event Logs, you don’t think about how awesome they are to point out and help solve all the issues implied above. And we all know why – Event Logs are a pain.

Why are Event Logs So Painful?

Let’s start with the obvious answer: There’s an inordinate amount of data. According to Gartner, a medium - sized enterprise creates 20,000 messages per second of operational data in activity logs. In a single, 8-hour day this comes to 500 million messages, adding up to more than 150 GB of operational data.

Now, this may not accurately represent your environment, but it evokes the same emotions you already feel when you think about your servers and all the logs they contain (and your head begins to spin).

The second obvious answer: Finding the needle in the proverbial event log haystack. Make that haystacks – you’re responsible for monitoring multiple servers, multiple logs, and multiple events. And once you have a grasp on all the data you need to search through, you need to determine what are you supposed to look for - Is it the event ID, the description, the source? Which query will provide a meaningful result?

To put all of this in perspective, let’s look at five aspects of Event Log Management that need to be addressed.

Consolidation

Unless you like doing the same job repeatedly for each server you manage, you’re going to need to consolidate your logs into one location. This makes the remainder of the Event Log Management tasks far easier.

Questions addressed here usually include:

  • Which server logs should I (and which should I not) consolidate?
  • Do I want/need to consolidate every log?
  • Do I want/need to consolidate every event entry?

A medium - sized enterprise creates 20,000 messages per second of operational data in activity logs.

Management

This next aspect sounds a bit redundant (of course there’s management in Event Log Management), but what is meant here is the management of the data that is consolidated. Storage, retention, backups, further consolidation all need to be addressed from several standpoints, including security and compliance. Questions addressed here usually include:

  • Where will I store the actively used logs?
  • How long will I maintain my log data?
  • What is my archiving strategy (think both age and medium)?

Monitoring

Deciding what to monitor is always a challenge. What gets monitored usually depends on what is important to a business. If it’s security or compliance, the answer may be access to data, or account creation in Active Directory, or even logon failures. If server performance, it may be Exchange service errors, or operating system warnings. Again, it all depends.

Microsoft provides a number of fields to search on, so we’re not just talking about searching for one field. Table 1 shows a sample of the fields you can use to filter your data.

But wait, there’s more! Monitoring isn’t about a single event. The simple copying of a file will generate a myriad of entries. So it’s not always as easy as “show me the event where” but may involve defining how a number of entries correlate to represent the event you wish to monitor.

Alerting

Monitoring, by itself, is useless. Without telling someone the house is on fire, the house will simply burn to the ground. The same is true with Event Logs. It’s great to monitor for specific issues, events, actions, etc, but it is the alerting that puts IT into action. Traditionally, alerts take the form of an email, but can also be SMS texts, SMNP traps, Dialog Boxes, even sounds.

Remediation

IT folks are some of the most dedicated, hard-working folks, such that they don’t always have time to fix every issue exactly when it happens. Part of your Event Log Management strategy should be the automatic fixing (or at least the first attempt to fix) an issue. This can be reboots, restarts of services, running of scripts that perform actions and the like.

Event Log Management strategy should include all 5 aspects:

Consolidation, Management, Monitoring, Alerting and Remediation.

Microsoft Provides the Basics

Microsoft does give credence to the idea that log management isn’t easy, which is why their Event Viewer has undergone changes throughout the years to include not just the ability to Find and Filter events, but also to perform some basic remediation.

Windows Server allows you to select a specific event and perform one of three actions, should the event occur.

Event Forwarding

With Windows Server 2008, Microsoft introduced Event Forwarding.

With Event Forwarding, logged events on Windows Server 2003 and 2008 servers can be forwarded to a centralized server, based on specified criteria.

The setup of Event Forwarding is a bit of work. It involves a number of steps just to get the servers ready to forward events and then you need to configure what gets forwarded. Truly, it is a good attempt at helping with the challenge of managing multiple servers by consolidating events, but is intended to scale to a few servers at most and only addresses one of the five parts of Event Log Management – Consolidation.

Don’t forget your other sources!

To make log management even more complex, Microsoft Windowsbased servers are only one source of logs you need to manage. Your non-Windows servers, firewalls, printers, switches, etc. all have valuable information to provide about the security or performance (or both) of your networks. There are other sources you need to include:

  • Syslogs – This is the most common standard for logging outside of Microsoft. Syslogs utilize a push technology that require a service running somewhere to accept and consolidate the syslog data.
  • Text Logs – Additionally, some systems, including SQL Server, write to text-based log files. These should also be considered.

It’s Still Not Easy Enough. Now What?

Reducing the amount of work needed to manage Event Logs can only be accomplished by utilizing a third-party solution designed to do the work you’d be doing manually, or with limited automation with native tools. Let’s discuss how to make Event party solution and by introducing you to Veriato Server Manager.

Remember that because Server Manager is template-driven, an Event filter, can be reused for additional consolidation, views of logs, and reporting.

Meet Veriato Server Manager

Server Manager consolidates, monitors, alerts on and responds to critical events, providing centralized management and reporting of Event Log and Syslog data. Server security and performance is maintained, the health of server resources is monitored, and adhering to compliance standards can be proved.

To truly consider Event Log Management “simple’, the solution you use should meet the followingthree criteria:

  • Scalable
  • Centralized
  • Automated

Let’s look at each and how Server Manager meets each.

Scalable – Single Solution

We’ve already discussed scalability a bit in this whitepaper in the context of Event Forwarding. But your work encompasses multiple logs, multiple servers, and multiple types of logs.

Server Manager provides robust capabilities to address Microsoft Event Logs, Syslogs and Text Logs from within the same solution, allowing you to consolidate, manage, monitor, alert and remediate issues across your entire network.

Some of you are monitoring logs for security reasons, while others are monitoring to maintain performance levels of service. If uptime and performance are of concern, you need to be monitoring beyond just logs. Server Manager also monitors server resources, disks, applications, Windows services, databases, TCP ports, well-known web services, you name it – all under one roof so you get a comprehensive view into what’s going on from both the log and performance perspective. Figure 7 shows the various types of performance-related monitors Server Manager supports.

Scalable – Multitudes of Nodes

Server Manager was designed to support the monitoring needs of your network. It can simultaneously be monitoring your Windows servers, Unix boxes, workstations, SANs, NASs, routers, printers, hubs, switches, firewalls, appliances, websites and more.

Scalable – Template Driven

Given that Server Manager can monitor so much, it has been designed to simplify the aspects of monitoring so that you’re not repeating the same tasks over and over again. Server Manager utilizes templates, shown in Figure 8, to define the various aspects of monitoring and management of event logs, including:

If performance is a concern, having a single solution that monitors both a servers logs, as well as its resources, services, processes, etc. provides you with a comprehensive view into server performance.

Let’s use at a real-world example to see how this benefits you. If you monitor multiple Exchange or SQL servers, you can simply define the events that need to be monitored, the times of day to monitor and the actions to take when the monitors are triggered then quickly apply that same template to all of the servers, as is appropriate. Likewise, should you simply want to reuse one aspect of that definition – let’s say an action to be taken – and apply it to a completely different set of servers being monitored for a completely different set of events, you can take that action template and utilize it somewhere else.

Next, let’s take a look at how a centralized solution simplifies Event Log Management.

Centralized – Log Consolidation

To properly monitor and manage logs, they need to be in one place. With Server Manager’s Consolidation Template, shown in Figure 9, you can easily select the servers, logs to be consolidated and filter the consolidated events (Figure 10) to ensure you only collect the events you need. Post-consolidation actions can also be applied to the logs as they are pulled in, providing you with management and alerting the moment consolidation occurs.

To properly monitor and manage logs, they need to be in one place.

Monitoring, by itself, is useless. Without telling someone the house is on fire, the house will simply burn to the ground.

Centralized – Log Management

Once you have the data, you need to plan on how you will store it, back it up, and make it available (as is appropriate) for review, reporting and retention.

Server Manager supports storing consolidated Log data in 4 different mediums:

  • SQL Server
  • Oracle
  • MySQL
  • Server Manager’s own proprietary binary file format

Besides backing up any consolidation databases, you may need to archive Logs directly for security or compliance purposes, including being encrypted and digitally signed. Figure 11 provides an example of how Server Manager can be configured to automatically backup Logs, which can be scheduled using Server Manager’s Schedule templates

Centralized – Log Reporting

The beautiful part about event log management is you already use filters. And what’s the basis for log reporting? Filters of course! So building reports is as easy as creating a filter template (or reusing one that you’ve already created). Server Manager has 15 turnkey reports, but is designed to allow you the flexibility to quickly generate your own reports using the report templates.

Reports can easily be re-run against current data, scheduled, posted to websites for viewing and saved out to HTML, TXT or CSV formats

Automated – Response

If Log Management stopped here, you’d be completely up to your ears in properly consolidated and monitored logs and with an Inbox full of alerts awaiting your response. Server Manager provides you with the ability to respond using a wide variety of actions.

The actions fall into three categories:

  • Alerts – Utilized to make the appropriate staff aware of an issue. These include sending an email or text, displaying a message box and playing a sound.
  • Documentation – Used to record the occurrence of an event in a separate system, log, etc. These include writing the event to a database, another event log, a syslog server, a file and sending an SNMP trap.
  • Remediation – Used as “first response” to fix issues. These include managing Windows services (stop, start, restart) and launching a process (which opens up a wide variety of possible actions – running a script, launching a backup or restore, shutting down a server, etc. – the list is limitless).

Automated – Log Management

To make Event Log Management truly work, it needs to be “set it and forget it.” Server Manager, as a whole, meets this criterion. It was designed as a management platform that performs the actions you’d normally accomplish by manual means, and automates each aspect of Event Log Management. From consolidation, to management, to monitoring, to reporting, to responding, Server Manager does it all automatically.

Let’s bring back the grading we gave to Microsoft’s native tools and see how Server Manager stacks up

What Else Does Server Manager Offer?

There’s a reason the product wasn’t given a name that implied management of event logs only. Remember, in the area of Log Management, it also manages Syslogs and Text logs, giving Server Manager the ability to monitor just about any system that produces logs

Additionally, as was briefly mentioned in the Scalable – Single Solution section of this whitepaper, Server Manager also monitors, alerts and remediates issues for:

  • Windows Resources – includes memory use, CPU utilization, and network throughput, individual processes, services, Active Directory and clock synchronization
  • Network Resources – includes email and web services, SSL and domain expiration
  • Disk Resources – includes disk space, SMART status, directory sizes, and file counts
  • Database Resources – includes SQL Server, Oracle, MySQL and ODBC

Conclusion

Your efforts to manage Event Logs can be with put towards an arduous and tedious process of systematically going through logs, looking for issues, and taking the appropriate action each and every time, or you can make a one-time investment to set up an automated way to stay informed of and maintain the current state of your Windows environment. With Veriato Server Manager, Event Log Management becomes a simple task of establishing an be with put towards an arduous and tedious process of systematically going through logs, looking for issues, and taking the appropriate action each and every time, or you can make a one-time investment to set up an automated way to stay informed of and maintain the current state of your Windows environment. With Veriato Server Manager, Event Log Management becomes a simple task of establishing once what needs to be monitored (and what to do about it), reducing issue elevation, increasing server and service uptime and improving your productivity.

This article discusses solutions that involve the following products:

Veriato Server Manager

Learn More