Return to Conference Proceedings
There is no compelling reason to change the structure of the refereeing process to adapt to an electronic medium, but there are some more highly interactive, and more open, forms of certification that might be tried (O'Donnell).
It is our hypothesis that such experimentation as O'Donnell speaks of so dismissively is critical to the future of electronic scholarly publishing. But the proof will be in the testing, so we are presenting a suggestion for one such interactive, more open form of certification. We believe that in time, in an evolutionary process, experimentation with such systems will prove that there are, in fact, compelling reasons for taking advantage of the new technology represented by the Web in the process of certifying the quality of scholarly contributions. To understand why we say this, it is important to know what scholarly journals are, and what role they play in the lives of scholars.
Scholarly journals are, themselves, a technology for delivering results of scholarly activity to the world. As such, they incorporate a wide variety of other technologies: language, paper, print, computers, graphic representation modes, colour representation and application, and many more. From one view, the history of scholarly journals can be seen as the history of an ongoing attempt to automate the delivery of information to as wide an audience as possible. From the first scholarly print-based journal published three centuries ago to the latest, electronic-based journal published on the Web, scholarly journal publishers have routinely looked to adopt the latest technology advances in order to lessen the costs while enhancing the value of their publications and increasing their audiences. In practice, this has meant the elimination of manual effort at every feasible point in the publication process.
During the last forty years, the application of technology to scholarly journals has focused principally on the physical production process, though the distribution aspects have not been ignored. Indeed, our organisation, OCLC Online Computer Library Center, Inc., along with its wholly owned subsidiary, Information Dimensions, Inc., has developed a product targeted towards publishers, OCLC STEPS System for Total Electronic Publishing Services (STEPS) using the latest in automation technologies applicable to the production and distribution processes. Other such systems from other vendors are also coming online.
But what has been left out of the reinventing of the fabrication of scholarly journals has been the process that actually procures the contents of journals: the scholarly articles. Generally, the procurement process, usually known in the trade as the refereeing process, is a highly manual affair, involving the interaction of numerous individuals, in a variety of ways, including the transfer of documents, reviews of documents, comments on documents and revised documents via a myriad of forms and means. Doing this manually has high costs involved, both in terms of time and resources expended and increased opportunities for error per article finally accepted for publication. Simply making a straight conversion or adaptation of the process from manual to automated will eliminate a significant amount of costs. But with time and experimentation, there is more to be gained by making changes that go beyond simple adaptation.
Before the Internet, before the World Wide Web, there was no reasonable way to automate the refereeing process. Now that they do exist, change can occur, but not only in order 'to adapt to an electronic medium' but because the electronic medium makes it possible to effect meaningful change. What the medium makes possible in our view is firstly, a more cost-effective process for the publisher. This will be done the old fashioned way by eliminating as much as possible the actual human effort expended in selecting deserving articles via automation. Secondly, the medium affords an opportunity to measure the value of individual scholars' contributions for promotion and tenure decisions via automation. These possibilities are actually intertwined, as will be shown later.
Before addressing the specifics of our proposal, we will first briefly review the present state of automation in publishing, as well as the current electronic journal environment.
Once the typesetting/photocomposition portion of the process was automated, and the copyediting was automated via word processors, the next step was to automate the production workflow environment. Software systems for keeping track of, and scheduling events for, articles and issues of journals have been around for a decade or more. These packages do not, of themselves, cause the events to happen, such as starting the photocomposition engine at the appropriate time with the given articles for a issue. But they do allow for an online, interactive accounting of all the various statuses of all the components being worked on at any given instance of time. They save a great deal of manual effort. Once all these subsystems were automated, the next step in the automation evolution of publishing was the integration of all these subsystems into one system.
Jolanda L. von Hagen of Springer-Verlag, at a seminar at Bond University in May 1992, laid a picture of the traditional publishing workflow alongside a picture of that same workflow when Xyvision's version of an integrated publishing system is used. The traditional workflow, which included using various automated tools such as word processors, graphics packages and photocomposition/typesetting engines, contained twenty-five processes. Under the Xyvision system, this reduces to a mere six (von Hagen). A similar reduction can be realised with OCLC STEPS, a fully integrated electronic publishing environment that allows for the importing, authoring, editing, composition, typesetting and distribution of texts via multiple mediums based on an SGML-aware database engine and any of several applications packages integrated as a shell around the database. Such systems represent the state-of-the-art in automation of the physical production of journals, as well as their distribution, in multiple formats.
With a STEPS-like system, it is possible to enter articles into a database, either through capture and conversion from formats other than SGML, or through importing them directly from SGML authoring tools, copyedit them, then schedule them for a given issue of a journal through an automated workflow manager and, at the appointed time, out will come the fully composed and typeset journal ready for printing on paper, or placing on a CDROM, or mounting on the Web or other online environment. Depending on the complexity of the documents involved and the distribution medium, it is possible to have no human intervention beyond the scheduling for publication step. Very complex documents such as those with heavy mathematics or intricate tables would probably need some intervention in the photocomposition stage, but this is only on an as needed basis. Online journals can be loaded automatically from STEPS-like systems after they have been composed, while for CDROM or paper journals, some human intervention is necessary to create and distribute the individual copies, though automated tools can certainly make this task very simple.
STEPS-like systems are aimed at publishers of many journals and monographs, such as academic presses, professional societies and commercial scholarly publishers, and are still not deployed even amongst this market in any appreciable way. The technology that has impacted scholarly communication more forcefully are the desktop publishing systems noted briefly above. These are systems designed to run on an individual workstation with an attached laser printer and/or World Wide Web Server connection. More powerful than their cousins, the What-You-See-Is-What-You-Get (WYSIWYG) word processors and desktop publishing programs give individuals the capability of fully composing and typesetting whole documents and either printing them or mounting them on online systems with all the photocomposition/typesetting quality that the larger publishers have traditionally had at their disposal.
Thus, individual scholars can for instance, layout their own texts in a high-quality manner, then publish (that is, distribute) them through a WWW or Gopher Server to the world of the Web/Internet. Desktop publishing tied to online delivery systems like the Web can make any person capable of publishing their own work. The question becomes, why would a scholar do such a thing? The answer lies in what motivates scholars to publish. This factor will be explored more fully later in the paper.
International technology standards, both de jure and de facto, are the basis for making such highly integrated, fully automated systems not just possible, but practical. Without standards such as ANSI Standard Character Information Interchange (ASCII), Standard Generalised Markup Language (SGML), Hypertext Markup Language (HTML), Adobe Postscript and Portable Document Format(PDF), Hypertext Transfer Protocol (HTTP), Rich Text Format (RTF), Transmission Control Protocol/Internet Protocol (TCP/IP) and dozens, if not hundreds, of others, then the automation we have today in the publishing world could not exist. The question that needs to be asked is what standards do we need that will make further automation possible and practical? Indeed, what is there left to be automated? These questions, and some possible answers, will be further developed later.
Lancaster isolates four primary stages in the evolutionary process of electronic publications:
As Lancaster points out, examples of all four co-exist in the here and now but logically they are manifestations of separate evolutionary stages. What is clear from Lancaster's presentation is that the electronic publication world did not start out with electronic only, totally new journals that from day one took full advantage of all the capabilities afforded by an electronic medium of creation and distribution. The movement has been gradual and deliberate (Lancaster).
Hickey, taking a slightly different tack, gives a detailed examination of the capabilities of three electronic forms of publication in terms of their relative advantages and disadvantages. Hickey's classification of forms, simple text, page image and structured text do not exactly match Lancaster's stages; still he clearly shows an evolutionary movement from straight-forward presentation, to more complex, to still more complex. As with Lancaster's stages, Hickey's three forms co-exist in time, with examples of each currently residing on the Internet/Web (Hickey).
We believe concurrent existence of separate stages and forms of electronic publications will continue for some time, probably into the next century. However, it is the stage four journals, most likely utilising Hickey's structured text form, that will be best suited to evolve the kinds of capability we are presenting here - an automated peer review process integrated into the presentation of the journal articles.
Additionally, there is an economic value associated with each scholar's reputation amongst the scholar's peers for doing high quality scholarship. Like the outcomes of the committees on raises, promotions and tenure, scholarly reputations are largely based on the quality and number of publications by a given scholar. Quality, in this sense, is measured by the number of peers who actually read and appreciate the publications of a given scholar and do them the honour of citing their work in a positive manner in their own publications. The higher the reputation of the citing scholar, the more weight the cited scholar receives in terms of their own reputation. This is a very informal economic exchange with no central accounting office keeping score, unlike with the promotion and tenure committees. But it is important and is effective because it can lead to leadership roles in the larger community such as being asked to serve as a reviewer or editor for the important journals in the field. This in turn can lead to positive evaluations by the university committees that control raises, promotions, and tenure grants.
Thus, scholars have a vested interest in there existing as many possible high-quality publication venues - that is, journals for most scholars, monograph publishers for the rest, as can be viably supported. It does not hurt if there are also a good number of less prestigious, but definitely not disreputable, venues available as well, since no scholar always hits a home run every time. Therefore, scholars have a vested interest in keeping the costs of journal production as low as possible so that more journals can be produced with existing funds. As we all know, the piece of the scholarly funding pie available for publishing is not getting any bigger, so the goal is to cut it into as many smaller pieces as possible. This is one possible reason for the experimentation by small groups and individuals with desktop publishing systems and Internet/Web publishing. This approach has many advantages such as timeliness of publication, direct control of the presentation values and opening up the potential audience by large factors.
All of which is to say that a journal, in order to survive, must keep production costs down while keeping quality high, both in terms of content and presentation of content. As discussed earlier, current automation systems go a long way towards keeping down significantly the cost of the physical production of journals, whether paper or electronic, and their distribution. They also increase the presentation quality. However, the cost of acquiring the content has not yet been addressed by automation in any significant way. This is not an insignificant cost to publishers who have to shoulder all the work of forming and maintaining the network of peer reviewers and editors that keep the flow of high quality papers into the publisher's journals at a sufficiently high level in order to maintain market share. Automating this process could further lower the costs of publishers, freeing more capital to fund more journals, or raise the quality of existing ones. It could also lower the costs of entry into the publishing field, bringing more competition. This is a possible result that the following player group in this triad of players can only be very happy about.
Libraries are being hard pressed to keep up with the rising costs, which means that library patrons are being shorted in terms of what they want - large numbers of high quality scholarly journals readily available to them. It also means that in the long run, publishers will be publishing fewer journals, leading to fewer venues for scholars to publish their work in, leading to more difficulty in attaining their career objectives in terms of raises, promotions and acquiring tenure. The good news is that the Web offers an opportunity to change the parameters of this currently zero sum game. By automating the entire scholarly communication process via the World Wide Web model, the possibility exists for lowering the costs.
We believe the next phase of automation will address the need readers and authors have for peer reviewed scholarship and to further automate the publisher tasks. Tools developed in this phase will enable the mediation of the evaluation function which is so important to the progress of scholarship. This mediation will consist of reviewers providing feedback to authors so that work can be improved, authors receiving formal recognition of the quality of their work, and readers having automated filtering aids for identifying high quality scholarship. One consequence of these tools will be to automate some of the management of the collaborative process carried out by publishers, thus lowering their costs.
These technologies will certainly, at a minimum, enable scholars to receive timely comments from peers and readers to benefit from that evaluation of the material. But they can do more. Among other things, collaborative software technologies can
The metadata record will also contain status information about the article. This status will consist of whether or not the article has been reviewed, usage statistics (how many readers have accessed it), citation statistics, reviewer's ranking and reader evaluations.
In addition to the formal review, readers will be able to filter the database by highly referenced papers. That is, the reader can make sure that they only look at papers which have been referenced in other works in the archive. This provides another measure of quality that a reader can use to filter the archive during a search.
Perhaps the most exciting possibility is special interest communities to form within the population of all readers of an archive in order to evaluate articles based on the group's own set of criteria. In other words, small groups with specialised interests can provide different ratings of quality. These groups might be formally organised, like special interest groups within the scholarly and professional societies, or they might be informal, consisting of anonymous readers who share similar interest profiles. Thus, the ratings given to an article by its reader would be maintained anonymously by the archive, grouping the ratings for retrieval purposes by reader profiles. Thus, if a reader wanted to read articles that had been evaluated highly by other readers with a similar interest profile, the archive could supply all such articles by first selecting those readers with similar interest profiles and then retrieving those articles that were ranked highly by that group of similar readers.
Return to Conference Proceedings