Angus Macdonald

I wrote previously about performance monitoring in .NET, but haven’t touched on the monitoring of Linux machines. There are a huge number of monitoring systems for Linux, but as with .NET performance monitors there isn’t much discussion of how they operate. This article points to a few of the bigger monitoring systems and looks at how they obtain their results.

EXISTING SYSTEMS

The paper A Taxonomy of Grid Monitoring Systems presents a good overview of existing systems and their architectures – most of which are designed to be run over large clusters, rather than individual machines.

If you’re looking to use a monitoring system I’d advise reading this paper. Most systems look to monitor the same aspects of system performance, but differ in their methods of storing data and handling distribution. Consequently there isn’t any one system which provides an all-encompassing solution, but there should be one to fit your needs.

Another system which isn’t mentioned in the  survey paper is Collectd, which is perhaps a more basic tool, but still contains enough functionality for most people. The best thing about Collectd (from my perspective) is its documentation. Their website lists the aspects of the system that can be can be documented, and goes as far as stating what Linux functions (or files) are used to obtain the measurements. If, like me, you’re looking to do some basic low-level measurements on your own this is incredibly useful.

OBTAINING MEASUREMENTS

If you’ve looked at the links above and concluded that the frameworks are too complex for your needs then I’d recommend looking at how these systems obtain their measurements – they generally all do the same thing. The source code for Collectd is the most readable, with each aspect (Disk, CPU, etc.) being split into separate files, largely free from verbose middleware code.

In Linux most stats can be found in the /proc folder – a pseudo-file system residing in virtual memory. Because it acts as a file system you can use ls to browse the available information.

This Red Hat page goes through many of the interesting files in /proc. Of these the following are of most interest:

• /proc/cpuinfo – Identifies the type and speed of the processors in this machine.
• /proc/loadavg – provides load averages for 1, 5, and 15 minute periods. For many applications this may be all you need to monitor, as it provides a good idea of the systems load without using any intrusive performance metrics.
• /proc/meminfo – Provides information on the system’s memory usage, including the total amount of memory available and the total amount currently free.

CALCULATING CPU UTILIZATION

Sadly none of the files in /proc report CPU utilization directly – /proc/stat reports the amount of time the processor spends in user time, nice time, kernel time, and idle time respectively, but easy to parse. If these values aren’t sufficient you can implement a function to calculate the percentage CPU utilization for a given period, or use a program which already does this.

This CyberCiti post provides a good overview of the latter. I found these to be slightly excessive for my own needs, so wrote something myself. I’ll try to post that at a later date.

A few weeks ago I posted an article about performance counters in .NET, which discussed how to get measurements on things such as CPU and memory utilization. Perhaps the biggest limitation of this class is the inability to get static information about the machine on which the program is running. For example, what if you want to find out how many processors a machine has, and the speed of each? Or if you need to get the IP address associated with a network interface?

Windows Management Instrumentation comprises a set of classes which allow you to query the system on anything from processors, to printers, to keyboard drivers. Essentially everything installed on the system can be queried as long as you know the correct command. This post provides some examples of WMI being used and links to the relevant MSDN sources which state what can be obtained in full.

Note: For the following examples to work make sure you import System.Management in your Visual Studio project. It is already included in the example project.

MAKING A QUERY

In WMI all results are accesed through SQL-like queries. For example, to find how much phsyical memory a computer has a query object is created, as follows:

ObjectQuery objectQuery = new ObjectQuery("select * from Win32_PhysicalMemory");

For a list of the attributes that can be selected from this look here.

Now we have a query object we can execute a query using ManagementObjectSearcher:

ManagementObjectSearcher searcher = new ManagementObjectSearcher(objectQuery);
ManagementObjectCollection vals = searcher.Get();

You can iterate over this collection as you would any other. The following code just prints out the amount of phyiscal memory on each piece of memory.

foreach (ManagementObject val in vals) {
Console.WriteLine(System.Convert.ToInt64(val.GetPropertyValue("Capacity")));
}

The ManagementObject has a number of properties (listed on the MSDN page) which can be retrieved. In this example the conversion to an integer value is completely unnecessary, but I’ve left it in because it becomes important when storing the value, rather than just printing it out.

val.GetPropertyValue("Capacity") can also be written as val["Capacity"].

WHAT QUERIES ARE POSSIBLE?

To see what classes can be queried have a look at the left of this MSDN page. Any part of the local configuration can be queried, making WMI a very powerful tool. However none of this is particularly efficient – you’ll notice a delay when executing some queries, so make sure you only do what’s necessary and cache the results.

EXAMPLE PROGRAM

Click on the following link for an example Visual Studio 2008 project performing some basic WMI queries and iterating over the results.

Example Project Source

If you’re looking for more dynamic results (such as CPU Utilization) I’d recommend looking at my post on Performance Counters.

This post discusses .NET Performance counters and their use in monitoring resource utilization.

.NET provides various classes for monitoring the resource utilization of the CPU, memory, disk and network cards. Sadly, the documentation is particularly vague with respect to what can be monitored and how this can be done. This article describes basic use of performance counters and gives a sample class (see end of text) which displays some of these on the command line.

PERFORMANCE COUNTERS
The PerformanceCounter class is used as the basis for all monitoring. This can be set up as follows:
private PerformanceCounter pcounter = new PerformanceCounter(categoryName, counterName, instanceName);

This constructor takes three parameters:

• categoryName refers to the type of resource to be monitored. The following values are permissible on all machines (you can also create your own custom counters):
  • Processor
  • System
  • Thread
  • Memory
  • LogicalDisk / PhysicalDisk
  • Network Interface
• counterName is the name of a specific counter within that category. For example, if monitoring the processor % Processor Time specifies that we want to look at the percentage of time the processor was busy during the sample period. The following pages list counters for each of the resources provided by Microsoft:
  • Processor/System/Thread
  • Memory
  • LogicalDisk/PhysicalDisk Disk
  • Network Interface
• InstanceName specifies the instance of the resource being monitored. For example, when monitoring disk utilization you may want to specify which drive is to be monitored. Alternatively, to monitor all instances just enter _Total.

SOME EXAMPLES

private PerformanceCounter processorcounter = new PerformanceCounter("Processor", "% Processor Time", "_Total");

Measures the percentage of time the processor was busy during the sample period.

private PerformanceCounter memcounter = new PerformanceCounter("Memory", "Available MBytes");

Shows the amount of physical memory available to the processor (in megabytes).

private PerformanceCounter diskcounter = new PerformanceCounter("LogicalDisk", "% Free Space", "_Total");

Shows the percentage of free disk space (over all logical disks).

private PerformanceCounter bandwidthcounter = new PerformanceCounter("Network Interface", "Current Bandwidth", "Intel[R] 82562V-2 10_100 Network Connection");

An estimate of the current bandwidth of the network interface. The instanceName in this case must be the name of the network card being monitored.

If you don’t know the name of your network card the following code may be helpful. It returns a string array of all the network cards on your local machine:

PerformanceCounterCategory category = new PerformanceCounterCategory("Network Interface");
String[] instancename = category.GetInstanceNames();

The same method can be used to find the names of logical disks or other devices on the local machine – just replace the Network Interface string with the category of your choice.

OBTAINING MEASUREMENTS
Now that we’ve instantiated our monitors we need some method of obtaining samples. For example, the following code extracts a sample from our processor counter:

float sample = processorcounter.NextValue();

There are a number of other fields and methods that may be of interest. To get a human readable description of the counter being used use the CounterHelp field:

String counter_description = processorcounter.CounterHelp;

FURTHER READING
This article covers the basics of the PerformanceCounter class. To see it used in practise check out some of these examples:

• An MSDN tutorial creating a WPF C# system monitor
• An MSDN tutorial creating a C# performance monitor

EXAMPLE PROJECT

Performance Counter Examples