Imaging in Bulk for the Internet

1 Introduction: History and problems of the site

I have not changed my view over the intervening two-and-a-half years, because the growth of the Web has not changed its basic complexion. The Web now has some excellent sites, but the majority are trivial, poorly maintained, often self-indulgent and low in nourishment value. I have aimed to make ArtServe interesting both to the student of Art History, and to the general viewer interested in art and architecture in the Mediterranean.

ArtServe is accessed from all over the world some 36,000 times per day on average (that is, well over 3 gigabytes of material per week). Access statistics are available at http://rubens/current.html .

The server has some notable deficiencies, many of which are to do with a dearth of staff to help in its upkeep, others of which have to do with the nature of Web servers. Staffing problems underline a basic difficulty with serious servers - namely that, the bigger they get, the more complicated they get, in a progression which is far from linear. This problem is highlighted by the manifest deficiencies of current server technology. Briefly, any organisation (and its concomitant, searching) must be arranged using external programs: it is not imposed - as it should be - by the server. Maintenance of large sites - and Artserve currently holds some 19,000 images - therefore becomes more difficult than it need be and users do not get the full benefit from them because they cannot necessarily find what they seek.

'Content' for an Art Historian means images, mapped at the very least by a datafile, and preferably with a searchable forms interface to make life easier for the user. This paper discusses the equipment used to take the server from the initial 2,800 images to the current 19,000 - well over five gigabytes of data - and the programs used to regiment the images to make them smoothly accessible to the server. It is a straightforward account, with illustrations to pretty things up a bit, and an Appendix which details the various programs written to process the images and make them available in a couth fashion to the server.

1.1 The Processes Involved

1. Digitising the 'real world' photograph, object, slide or negative;
2. Conversion to suitable formats (JPEG and GIF);
3. Sizing the images appropriately;
4. Organising the images into directories; and
5. Grouping the images into HTML pages, or suites of pages;

There are various possibilities along the way. The laserdisk provides a convenient half-way house, storing analog frames at video resolution, which the Sony PHV-A7E Photo Video camera presents to it as a video or S-VHS stream for capture on disk. The DEC Alpha computer has a digital capture card in it, but direct presentation of the frames to it would be very tedious. Using the two devices described here, a 'hit' rate of well over 200 images per hour can be attained, and these are then digitised in batch (into JPEG images) whilst the operator does something else.

Direct digital devices such as the Nikon E2 digital camera and the Nikon scanner present opportunities in that they offer four times video resolution (in the case of the camera) and much higher resolutions in the case of the scanner. Such resolutions present problems at the same time because the speed of the Web, and of people's connections, means that such large images cannot conveniently be viewed without custom-built software, called zoom, which we are currently developing. As for speed, the Nikon E2 (in my cheaper configuration) manages an image every couple of seconds (a sports variety, with more memory, can do continuous frames); written to PCMCIA cards (typically 30 images per 10Mb card, depending on quality desired), the images are of excellent quality. The camera is certainly capable of bulk-processing, but I do not use it on a copystand, preferring the Sony device detailed below. As for the scanner, the hopper holds about 50 (thin) mounted slides, and the speed of processing depends on the memory the machine has available.

A very flexible device is the Sony video camera with framebuffer, which can be used not only for photographs and prints (for example), but also for negatives and slides, with light transmitted through a lightbox. The lens (a Fujinon S16x6.7 BERM-18) allows such close-focusing that one-third the area of a 35mm slide can be captured as a full frame.

2 Hardware

Imaging hardware has evolved over the past three years. Beginning with a S-VHS video camera feeding into a 24-bit board on an Amiga, the suite is now as follows:

• Sony PHV-A7E Photo Video camera

  
  

  
  This is for imaging 35mm slides. The device pumps out composite or S-VHS. This can be fed directly into the video card on the Alpha but is generally used with a laserdisk; the majority of the early images on ArtServe (for example, those dealing with Classical Sites in Turkey have been digitised with this device.

• Writeable laserdisk

  
  
  This device holds 36,250 frames per side, at video resolution, and routines which came with the Alpha allow the digitising of any quantity of frames into JPEGs or GIFs. This device has been used for all the student images (which they typically access using machines which have small screens) and for over half of the images on rubens. The procedure is exactly the same when using a commercial or privately pressed laserdisk: for example, Dr Clive Ruggles' Survey of Prehistoric Ritual Monuments in the British Isles was prepared directly from the laserdisk he himself prepared.
  

• Nikon E2 (Fujix) digital camera (which writes about 30 1.2 megapixel JPEG images to a PCMCIA card)
  

  
  

  Laptops often come equipped with PCMCIA readers, but separate devices are available:

  
  

  This device is not cheap (circa $20,000 with a lens, flash, etc), but the results are excellent; a sample image (1.2 megapixels) is available at http://rubens.anu.edu.au/laserdisk/0214/21485.JPG.

• Sony DXC-930P 3-CCD video camera (which, by a cunning half-pixel shift, can write a 1.6 megapixel image (after interpolation) to a computer, via a framebuffer)

  My setup is the 'old' model; the current one, called the CatsEye, writes the same size images without interpolation, and costs about $25,000. With a close-up lens, this is an excellent copystand camera. It can talk RGB (or S-VHS or composite) straight into the laserdisk recorder as well, and has been used not only for copying photographs (most recently the Borobudur images), but also 35mm slides. A great advantage is its speed, especially now we have it talking to a Linux machine, for it writes JPEGs in under 20 seconds; a sample image (1.6 megapixels) is available at http://rubens.anu.edu.au/boro.project/basement/2.JPG .

  The same camera is extremely versatile and was used in closeup to provide detailed image of computer cables and connectors;

• Nikon Super-Coolscan 35mm scanner

  This new acquisition is to experiment with much larger images (up to about six megapixels), given that a principle of digitising is to do it at the best resolution one can achieve and then scale down for current use. Equipped with a hopper, this manages about ten slides per hour, and can be left working overnight; it is attached to a W95 box with a mere 16Mb of memory - and no doubt throughput speed could be increased were more memory to be added; a sample image (about 5 megapixels) is available http://rubens.anu.edu.au/imageserve/sample.JPG.

• Sony writeable CDROM
  

  

  Although we naturally use tape to backup our systems, writing to CDROM now costs under $1,500 for the hardware and software, and under $10 for a 650Mb blank. The process is very useful for 'bell and braces' backups, and for providing demonstration material from the server when a network is not available.

3 Software

The basic image-processing utilities available under flavours of Unix - xv, xli, ImageMagick, and the pbmtools - all proved most useful. The software written in-house is described in an Appendix to this paper.

4 Conclusion
Imaging: the next five years

Three years ago, any digital image appeared a marvel. Today, when a reasonable-quality digital camera (offering video resolution) costs $1,000, expectations have risen so that bigger and better images are making their appearance. Three years ago, my site was devoted to images exclusively of video resolution. Now, however, with the purchase of devices such as the Nikon E2 digital camera, and the Sony DXC-930P (with framebuffer), images of four-times and five-times video resolution respectively can be generated.

Storing such images in JPEG format is little problem given the cost of hard disks, but serving them presents problems because their size is usually still too large for most people. Users like big images, but they need ways of sampling sections and of having the image delivered to them (not necessarily involving a Web browser). Hence the development of our zoom program - see below, in the Appendix.

But 'bigger and better', whilst representing the progression in imaging, has not been reflected in Web technology, and herein lies the rub for the development of large sites. The current generation of servers and clients is easy to install and generally works first time. But the technology was never designed to deal with large quantities of material, perhaps in different media (text, sound, images, postscript files, video) so that we have seen the development of a host of add-ons which attempt to mitigate perceived deficiencies. Programs like wwwstat and pwebstats look after the reporting of log files, whilst Harvest/Glimpse allows us to pretend that the heaps of material underneath the Home Page really are in something approaching order, rather than chaos. CGI scripts help to impose various layers of order as well.

Professor Maurer and his team at the University of Graz have developed HyperG (now to be called HyperWave) as a 'second generation' server and set of clients (for various platforms.

Apart from the obvious problem with URLs (which should obviously become Universal resource Names), they point out problems in the current generation with scalability, with searching and with a basic lack of structure. HyperWave deals with these matters and also offers bidirectional links (hence link consistency - no more dead links), a read-write database (their description of the server), a full-text server with inverted indexes, a document dache server, and our old friend from databases - namely, referential integrity.

Appendix:
Software for Image-Processing in Bulk

Throughout the work with ArtServe, we have kept in mind the need to provide users in Art History (and any other discipline which uses a lot of images backed by a database) with effective, robust and reasonably simple software - although it should be made clear that, as the Web changes, so the features and type of the programs needed to populate it will also change.

Software

The guiding principles adopted for our software were as follows:

• All programs have been written in perl, which is the lingua franca of the Web. This is free software and is available for PC, Mac and Unix platforms.

• We have avoided commercial software, preferring to use solutions which are not dependent upon commercial imperatives. Hence there has been no use of 'database handlers': all our datafiles are plain ASCII text files and these are handled by the standard Unix utilities, controlled by perl code.

• All editing work is predicated upon the use of the Unix 'vi' editor. This is programmable (essential for some of our applications: see below). More importantly, it works from the command-line on Unix machines, and across the network (via Telnet) from PCs and Macs.

The programs divide into several groups:

Writing Image Datafiles

When dealing with large numbers of images, writing the datafile which describes them can be a bugbear. All the datafiles used on the ArtServe server are in plain ASCII format - there is no traditional 'database handler'.

The simplest method for writing such files is to do it directly into a text editor (normally vi, and cut and paste when (as is often the case) the bulk of records needs to be repeated - as, for example, when doing a series of records of the West facade of Chartres. The disadvantage of this method is that it is not particularly fast, and other arrangements need to be made for viewing the images to be catalogued, in their correct sequence.

This matter has now been addressed with a program using a Web interface, and with the (temporary) name of Bus-Queue which soon got changed to Image-Queue. First, the images are captured, and processed to provide thumbnail GIFs to sit alongside the larger JPEGs. Under my system, all images are filed in numerical directories, which makes various kinds of processing much easier.

The database records are entered using a forms interface to the Web browser, and this can be tailored using setup screens. Two types of entry are possible - namely, type-in boxes and pull-down menus. The labels and lengths of the former are set up in advance, as are the pull down menus, the latter being particularly useful when a restricted set of values is required, rather than any whimsy from the person doing the entering. No parameters are fixed in the rock, and all may be changed when (as is most likely) one realises the need to increase a type-in length here or make an addition to a menu there. An additional setup feature allows the user to specify how to treat the next record, obvious possibilities being to retain the parameters of the previous record and, in the case of some numerical fields, up the number by one.

But the program is called Image-Queue because of the way it presents the thumbnail images so the user can see what is going on. The image to be databased appears in the middle of the window, with the next four across the bottom of the window. (To examine the image in detail, click on it to bring up the large JPEG.) Once databased, the tally augments by one, new record appears, and the first thumbnail goes top left on the window. The next one joins it, and so on. Thus, after completing four new records, there are nine thumbnail images visible - the four already completed, the one currently being done, and the next four in the sequence. They shuffle from bottom right to top left - hence the analogy with a bus queue.

Datafiles and the Display of Databased Images

• salami is a program for slicing up a datafile, with images, into Web-sized 'bites'. Originally created in 1994, this has been completely re-written. Salami has two main uses: slicing an image database, or preparing a group of images for a lecture, seminar or tutorial. In either case, run the program with no arguments. All the options are set in the configuration file. For slicing, all the images in the selected database are collected in display pages called display*.html and (since there could be a lot of them) files called index.html, index1.html, etc. are created to point at the display files. The configuration file can be set up to sort on any field from the database, and the output format can be varied somewhat. For , a file must be created containing accession numbers for all the images needed. Typically, the file might be called numbers and there might be about 50 images to a lecture. The images will appear in the order of the numbers in this file (that is, no sorting is performed by salami). A single display page called lecture.html is produced. Salami is configured in a file called salami.ph . This is read from the current directory, if possible. If there is no salami.ph in the current directory the file /usr/local/bin/salami.ph is used. This default file is set up to do slicing.

  salami is highly configurable, and this is necessary not just to suit individual taste, but also different machine and (especially) monitor sizes. The user can select headers, the number of thumbnail images across the page, the particular fields from the datafile to print underneath them, and the field to hotspot. For use in 'slow network' setups, the program can display simply records, with no inline thumbnails (although this somewhat defeats the point of 'visual' displays).

• checkdb goes through all the named files and makes a listing to standard output of every distinct value found for each field (and the program can specify the field separator - default: a comma). This can be useful for detecting data entry errors, minor spelling variations, etc. Values for each field are sorted so minor variations on the same phrase for the same field are likely to be near each other in the output. Each value is displayed inside square brackets to make it obvious if there are leading or trailing blanks. A header is written at the start of each field. These headers contain 5 equals signs in a row, which make it easy to skip to the next field when viewing the output; for example, in vi or less, type '/====='.

• checkfields is a simple awk script which prints out the number of fields in each record, so the user can attain regularity in the datafile, without which any interrogation and results would be garbled.

• casedb repairs upper- and lower-case defects in ASCII datafiles. The construction of datafiles is exacting work, often done by several people in dribs and drabs; with the best will in the world, consistency is difficult to achieve. If one field reads 'London' and the same field in a different record reads 'london', then both entities need to be interrogated to achieve the proper catchment. Much better to improve the consistency of the datafile. This perl program allows such 'post-processing' of the datafile: command-line switches allow the upper- or lower-casing of the first letters of words, or of whole words, in a specified field or fields. Before operating, the program makes a backup copy of the datafile(s) in question.

• fixnumbers looks at the number of the current directory, figures out what the numbers of the JPG and GIF files in this directory should be, and sets about changing all the file names so the numbers are right. So, if you get the files into the correct directory, fixing their names is easy.