A Tool Based Approach to Integrated Network Management

Local time: Friday, 05-Dec-2008 12:15:25 EST
Last update: at /special/conference/apwww95 , Friday, 21-May-2004 09:47:03 EST

A Tool Based Approach to
Integrated Network Management

David J. Hughes
Senior Network Programmer
Bond University

Bambi@Bond.edu.au http://Bond.edu.au/People/bambi/

Abstract
The most common question ever posed about a network management system (NMS) is "Can I make it do this?". More often than not the answer to this simple question is either "Sorry, No you can't." or "Well, that wall of manuals over there explains the API so I guess you could write the additional code yourself." After spending time and resources selecting a network management system and deploying it within an organisation, either of the above answers isn't well received. The basic fact is that most NMS environments are either complete, stand-alone applications with a fixed level of functionality (at least till the next release that is), or they provide a framework approach with a comprehensive and equally complicated API to cater for user extensions.
The phrase "tool based approach to network management" relates to the methods used to acquire the management information. By allowing the NMS to manipulate and process management details provided by data acquisition tools, adding functionality to the environment is usually just a matter of adding a new tool.
This paper outlines the tool based methodology and how it has been implemented within the Minerva Network Management Environment. The protocol used for integration of external tools will be covered in detail. The framework designed to incorporate tool generated data will also be outlined.
Keywords
Network Management, tool based methodology,TCP/IP, SNMP

A Not So Perfect World

The first thing that a network manager will notice as soon as a new Network Management System (NMS) has been installed is that it doesn't do everything s/he wants it to do. This is a basic fact of life as no NMS vendor can foresee or justify supporting every possible network environment. Sometimes there's a piece of equipment that isn't supported and other times it's a more custom requirement that the network manager would like to incorporate into the central management environment. In the case of hardware devices, they can often be incorporated because they support SNMP and have their own private SNMP MIB that can be loaded into the NMS. Other times the device doesn't speak TCP/IP so SNMP isn't a solution to the problem.
Lets outline a simple example. A typical university environment will have a mixture of various types of machines. There is usually a TCP/IP backbone running throughout the campus to provide connectivity to the Internet. Such a backbone would allow the NMS to send SNMP requests (SNMP uses TCP/IP) to all areas of the network. If the network manager wanted to monitor the status of a Macintosh file server, TCP/IP and SNMP aren't going to help much as the Mac file server will be speaking AFP, the Apple Filing Protocol.
Another common situation is the need to monitor applications running on a UNIX machine. In an SNMP world, checking the status of a process would require the development of a custom SNMP agent. This isn't a job for the feint hearted. In a world without SNMP, the system administrator would probably just write a shell script to grep through the output of the ps command to see if the process was still running. Alas, a fixed functionality NMS can't be made to do a ps so you are once again on your own. Wasn't life simple before we got all these high tech' productivity enhancing tools?
As a result of installing a new NMS, we find ourselves with the following choices:

Ignore a large part of our network
Run our old, simple tools as well as the "powerful" new NMS
Employ a new staff member to learn the internals of SNMP and to start writing custom SNMP agents
Employ two new staff members to read the NMS's API specification and to add new data acquisition methods to the NMS.

And you thought you got a great deal on that Network Management System, didn't you?.

Wouldn't It Be Nice....

In the good ol' days before NMS's, operations staff still had a pretty good idea about status of the network and other computing resources. Things were done in a less centralised or convenient manner but the general idea still got across to the people who needed to know. Most times it was because a shell script ran from cron every half an hour and pinged a host and sent the operations staff an email message if it didn't answer. Wouldn't it be nice if adding to the NMS was as simple as it was to write another little shell script. Wouldn't it be nice if you could use the standard tools you have on your UNIX machine to acquire data for your NMS.
The concept of a tool based approach is that you can use existing tools with as little effort to extend the NMS. This is achieved by providing a mechanism via which the NMS can execute an external tool to acquire information and the acquired information can be passed back to the NMS in a format it can understand. There are other considerations, such as how the returned data is to be processed but as we'll see they are not as large a problem as you may initially think.

The Framework

Before the integration of external tools can be discussed, the framework into which they are being integrated needs to be outlined. The Minerva framework is an object based framework where a logical piece of information is encapsulated within an object. Various classes of objects are defined to represent the information that is to be acquired and monitored. Instances of those objects are then created and organised into trees of objects where each tree represents a node or device on the network. A top level view of a group of object trees (i.e. a map) is shown in Figure 1.
The definition of an object class includes all the information required to acquire, process and represent the management information. It also includes details to allow GUI based configuration of the object by the user.

(16K image)

Figure 1. Object Trees

Associated with each object class is an external tool programme that is used to acquire the data. In some instances, the tool is defined as "-- No Tool --" in which case there is no external code gathering the management data. This may appear to be a strange situation but you'll see that you can do a lot of work in the data post-processing setup of an object class.
Post-processing of the management data is performed via an embedded ESL script called the trigger script. ESL is a C like scripting language that can be embedded within applications. The management information and various other configuration details are passed to the ESL script as pre-declared global variables. The script then processes the data as required and analyses it. Based on what the script determines, it will return with 1 of 4 possible status codes. Each status code reflects a unique status of the object to the central management system software (i.e. OK, Warning, Alert or Unknown).
Because ESL is a fully feature programming language you can do a lot within this embedded script. ESL provides a comprehensive library of functions including general purpose file IO, comprehensive string handling, interprocess communication and signal handling. Within the trigger script you can execute standard UNIX tools, monitor their output or exit status and then return the appropriate state information to the central management code. Using an ESL script and the Columbia Appletalk Package for UNIX, appletalk monitoring support can be added to Minerva in about 10 minutes!
ESL also offers full access to the CMU SNMPv2 library. A custom interface library has been developed to simplify the process of writing SNMP based code. Acquiring information from a router or similar network device via SNMP from an ESL script is about as easy as reading data from a file. Naturally, the data acquired can be interpreted by the ESL script and the relevant information and status returned to the management kernel.
The definition of an object class also includes details about the data contained within the object. Data variables are defined by and encapsulated within the object. The content of these variables provides the data upon which computation and analysis is performed. The results returned by the testing tools are stored directly within the objects internal data variables. The internal variables are usually transient variables used only during analysis of the data within a trigger script. Certain variables may be flagged as persistent variables in which case they retain their content between invocations of the trigger script. This feature is useful when calculating changes in things like SNMP counter variables.

(11K image)

Figure 2. Object Class Definition

Other information, including where the class fits into the object class hierarchy, what dependencies it has on other objects, and how it should be graphically represented are also contained within the object definition. Class configuration is shown in Figure 2.

TIP - The Smoke & Mirrors

If an external tool is to be used to gather data for the NMS there are several things that need to be standardised. Firstly, the format of the information returned by the tool to the NMS must be defined. Secondly, the format of configuration information passed to the tool at run-time, such as a timout value for a ping test, must also be defined.
The Minerva Network Management Environment solves these problems via a simple protocol that is used to manage communications between the NMS and the tool. The Tool Integration Protocol (TIP) is a simple, text based protocol, that provides facilities including the bidirectional passing of formatted information and remote execution control of the external tool.
TIP is provided to developers as a simple library that offers and event driven interface to their testing software. As soon as the software is executed, control is passed to the main TIP routine which accepts and processes TIP commands sent by the management software. The TIP command set includes

Exit
Send Global Config Requirements
Send List of Supported Tests
Send Test Config Requirements
Run Test
Perform Automated Discover

Before calling the main TIP execution loop routine, the testing software registers a function to be called for each of the protocol commands. When a command is received from the management software, any parameters, such as configuration details, are processed and stored in a symbol table prior to the execution of the user defined function. The testing code can then retrieve any such data from the symbol table via a simple API call.
Data is passed to and received from a TIP tool using a simple ESL styled encoding. Data tuples such as $timeout="10"; are passed via TIP to the external tool at runtime. Similarly, the tool may return a result such as $packet_loss="7" to indicate a 7% packet loss.

GUI Based Configuration

A major problem with using external, user supplied programs from a graphical management environment is that the user will still want to configure the remote tool using the GUI. If the reason for using external tools is to simplify things, it makes no sense to force the external tools to provide X window based configuration GUIs.
Minerva gets around this problem using one of the encoding rules provided by TIP. You may remember that one of the protocol commands requested that the tool send its global configuration requirements to the management software. An example of a possible response is given below:
$pingcount=int("Number of Pings", 10, 200, 100);
$pktlen=int("Packet Length in Bytes", 64, 1400, 256);
$timeout=int("Max Timeout in seconds", 5, 60, 10);
On receipt of this information, Minerva constructs a Motif dialog box that depicts this request. The result of this request is shown in the top section of the screen shot shown in Figure 3. To explain the encoding, the tool has requested and integer value for the number of pings between the range of 10 and 200 and that the default value is 100. On decoding this string, Minerva would create a Motif scale widget with a minimum value of 10, a maximum value of 200 and an initial value of 100.
TIP GUI encoding is also used to configure an instance of an object class. A TIP GUI string is provided as part of the object class definition and Minerva uses that information when the user creates or reconfigures an object. Information that varies on a per node basis and that is not part of the actual data acquisition is configured in this way.

(7K image)

Figure 3. Object Configuration Via TIP

An example of the use of a TIP GUI specification is shown in the middle section of the screen shot shown in Figure 3. That part of the dialog allows the user to set thresholds for the interpretation of the data acquired by the tool within the trigger script. In this case it is setting the packet loss level at which the object is placed into an alert or warning state. Obviously, this is not part of the data acquisition stage, rather it is part of the data post-processing stage.

Pulling it all together

As a result of TIP and the encoding rules it provides, we can now configure an external data acquisition tool, execute it and retrieve the management information generated by the tool. We can also "configure" the object to which that information pertains. The only remaining question is "How does it all tie in together?"
It was mentioned that the data encoding used by TIP to return information to the management kernel was a tag and value format like:
$packet_loss="7";
For anyone that has ever used the ESL programming language, you may think it looks a lot like an ESL assignment statement. Well, that's no coincidence. Each tuple that is returned in the above described format is used to create a global variable within the TIP environment with an initial value set to the value returned. Similarly, the results of TIP GUI configuration, such as the packet loss level for an alert state are also transformed into global TIP variables. The configuration information that was sent to the tool when the data was acquired is also setup in the global ESL variable space.
Once all the configuration information and acquired data has been added to the ESL environment, the embedded ESL script is executed. The script can then do simple analysis of the information. The ESL script used to post-process the packet loss results is shown in Figure 4. As you can see, there's no complex API functions or even sophisticated programming involved. It's a simple matter of comparing the data returned by the tool ($loss) with the trigger configuration set by the user ($warnLoss and $alertLoss).
	/*
	** Object Trigger Script
	*/
	 
	main()
	{
		if ( (int)$loss >= (int)$alertLoss)
		{
			return($ALERT);
		}
		if ( (int)$loss >= (int)$warnLoss)
		{
			return($WARN);
		}
		return($OK);
	}
Figure 4. Trigger Script for Packet Loss Test

A more sophisticated trigger script is shown below in Figure 5. In this case, the script opens an SNMP connection to the target host and requests the contents of the first variable in the System group of the MIB. If a response is received, the host is flagged as reachable via SNMP and further SNMP test objects that depend on the status of the SNMP Reachability object can be undertaken. If no response is received, the object is marked as unreachable and further SNMP testing is postponed.
	/*
	** Object Trigger Script
	*/
	 
	main()
	{
        	array   $result,
                	$vars = [ "system" ];
        	int     $fd;
	 
        	snmpinit();
        	$fd = snmpopen($NODE,$community);
        	if ($fd < 0)
        	{
                	$reachable = "snmpopen() failed!";
                	return($UNKNOWN);
        	}
        	$result = snmpgetnext($fd, $vars);
        	if ( !defined(&result))
        	{
                	$reachable = "No Response";
                	return($ALERT);
        	}
        	else
        	{
                	$reachable = "OK";
                	return($OK);
        	}
	}
Figure 5. SNMP Reachability Trigger Script

Conclusion

It can be seen that if an appropriate framework and integration protocol are used, external tools and externally acquired information can be incorporated into a centralised management environment. The fact that complex programming and API knowledge are not requires should ease the integration of an NMS into an organisation and encourage the use of sophisticated management techniques

Availability

Minerva is still in Alpha test stage and is certainly not stable enough for deployment in mission critical applications. It is however in use on a daily basis at Bond University for the management of the campus network. The latest copy of the Alpha release is available via ftp at Bond.edu.au in /pub/Minerva. At this time it is available only in binary form for SunOS based SPARC systems.
ESL is available as a stand alone and embedded language for other purposes. The latest version of ESL is available via ftp from Bond.edu.au in /pub/Minerva/esl.
The database system used behind Minerva for the management of the objects and management information is Mini SQL. The latest version of mSQL is available via ftp from Bond.edu.au in /pub/Minerva/msql
Various papers and other documentation on Minerva, mSQL and ESL are available via the World-Wide Web on the Hughes Technologies pages at:
http://AusWeb.com.au/computer/Hughes/

Author Information

David Hughes has been employed as the Senior Network Programmer for Bond University since 1988. During his time at Bond, David has also lectured in Artificial Intelligence, and UNIX Systems programming and Kernel Internals. He is completing a Ph.D. in Computer Science (network management) this year.
Outside of the University, David is Managing Director of Hughes Technologies Pty Ltd , a UNIX and Internetworking consulting company. He is also the Technical Manager of Web Publishing Australia , an Internet promotion and publicity organisation. David can be reached via electronic mail at either:

Bambi@Bond.edu.au
or

Bambi@Hughes.com.au

[Return to Table of Contents]
COPYRIGHT © 1995 by AUUG95 and APWWW95 Charles Sturt University. ALL RIGHTS RESERVED. ISBN 1 875781 43 9