Return to Conference Proceedings
In one instance, I have two servers running side by side on a network, both acting as fairly basic file servers - one running Linux and the other running NT. I use these servers to benchmark the two operating systems and measure their suitability for various tasks.
Both machines have identical hardware, networking and memory, and are both connected to an identical RAID unit operating as the primary disk storage.
I have started with a fairly basic benchmarking excercise, which is only a small part of the task at hand. The real effort of actually benchmarking several such systems could turn into a research project of its own. In this paper, I examine some of the whys and wherefores of such a project.
If you choose a PC with more or less RAM then you will get different results. You will end up with probably widely varying results depending on the configuration of the disks that you use, whether you use one chipset or another on the motherboards, ethernet cards, etc.
Of course the two servers were, for the purpose of this excercise, running different operating systems. These were:
These are the things that occurred to me when I was doing these tests that I was able to take care of:
The difficulty is that it is very hard to set up a 'test' network that has a fixed (non-zero) load that is also reliable and measureable. I could have artificially flooded the network with 'ping' or other packets, either at random or fixed intervals, or provided another external stimulus to the network to simulate a fixed level of LAN activity, but it would have been very difficult (because of the nature of CSMA/CD networks including ethernet) to determine exactly what proportion of the network bandwidth was in use at any point in time.
With the right equipment (and a lot more time and resources), this would be achievable and possibly even worthwhile.
The two operating systems do, however, perform a basic set of tasks in common. These tasks include:
The results were as follows:
What I have chosen to measure is what the user sees on his/her end of the network - copying files to and from the server machine. Although this is not a true indication of network performance, it is one thing that is easy to measure and gives some (although not an exact) measure of the relative strengths of these operating systems with respect to user level applications.
For the purpose of this exercise, I loaded Samba version 1.9.15 patch level 3 on to the Linux PC. This was set up in a very basic configuration with almost no security. On the Windows NT end, all protocols other than TCP/IP were disabled (that is, no NetBEUI or IPX/SPX - after all it is the operating system and not the protocol I am analysing here).
The figures for a similar test to the one above performed across the network were:
| Linux | 95.47 sec |
| Windows NT | 112.75 sec |
I am aware that Andrew Tridgell and various others have performed some benchmarking on Samba, as well as having provided some useful information about performance improvements that can be made with the Samba server (see Speed.txt in the Samba doc/ directory, included with the source code of the Samba distribution). I deliberately made absolutely no attempt to implement any of the changes that were suggested in that file as I wanted to operate with, as closely as possible, 'vanilla' versions of the systems concerned.
These differences make certain things very hard to measure. In particular, certain utilities that are available on Linux are not available on Windows NT, and vice versa. I could use a C compiler to provide my own versions of dd or WINFILE.EXE but that was not the objective of this paper - I am attempting to benchmark the operating systems in their raw 'vanilla' state and I certainly don't want the efficiencies or otherwise of the C compilers to interfere with this (there are other benchmarking suites that will do that task).
The first benchmark allowed me to test the process of reading and writing a file. It did not allow me to individually test the components of such an operation.
The components of this operation that I would have liked to have measured are:
Similarly, in measuring network performance I would like to be able to measure all of the components of this performance, including all of the components listed above. In many cases, the lack of homogenous tools is a problem; in some cases, I just do not have the time or resources. However, the biggest single issue that stops me being able to do all of the measurements I want to do is the lack of the source code for the NT Server operating system!
I have my own idea of the sorts of things that this suite should be able to do, including:
I am continually reminded of an article that went into circulation in the Australian Unix community some years ago, into which a great deal of work had been placed. The author of that article had to all intents and purposes dismantled the kernels of several vendor supplied Unix System V variants and performed some radical and quite thorough benchmarking on both the operating systems and the machines that they were running on (low end minicomputers).
The final decision on what system to buy was later taken by management and was based entirely on price.
However, more importantly, the numbers collected above (and I could collect a large deal more on other operating systems such as Novell Netware, Solaris, SCO OpenServer, etc.) are to a large extent meaningless in the real world. There are many factors, other than the raw disk performance of the operating system, that affect the total throughput of a network. These include:
I could of course go on for several pages about items that affect network performance. However, it is relatively clear that a 5% speed difference in the raw throughput of the operating system (or even a 15% difference in the raw network performance at zero load) would not translate into anything more than 1% in real performance at the user level - probably much less.
The answer to all of these questions is 'It doesn't matter'.
Examine a typical WWW server situation: Company X has put their server onto the Internet. They have purchased a 64K ISDN link to a well-known major Internet service provider.
Just looking at the latency, a typical ISDN router has a latency of between 20ms and 45ms. A LAN may add approximately 1ms to this, something under 5% of the total latency anyway. A typical user somewhere out on the Internet in Australia is likely to have approximately 200ms (modem) plus anywhere between 75 and 500ms (depending on their ISP) latency to Company X's ISDN gateway. That means that the gateway itself has something in the order of 10% impact on the total server - user latency. 1% of that 10% is 0.1% - the net effect of any LAN on the total user - server ratio.
Similarly, examining the bandwidth, a 10MBit/sec LAN will transfer a 50MB file (raw performance) in about 40 seconds assuming the highest possible theoretical bandwidth of 1.25MB/sec. The overhead of TCP/IP and the interference of the operating systems added approximately 150% to this based on the above figures, taking the total transfer time up to 100 seconds or thereabouts.
Assume that the user is experiencing a download time for the file of around 3.5KB/sec (about the maximum you'd expect from a 28.8K modem running 'flat-chat'). That means that the download time for this file over the modem will be about 14500 seconds. The time taken to transmit the file's packets to the router is approximately 0.6% of this time, while the operating system's component is around 0.4%.
These two issues mean that doubling the performance of the operating system could mean anything up to half a percent improvement in the performance of your web server! Impressive? No, I don't think so.
Return to Conference Proceedings