Helsinki University Library
Centre for Microfilming and Conservation
Majken Bremer-Laamanen
Email: bremerla@helsinki.fi
					22.02.2002

A Nordic Digital Newspaper Library

Newspapers on microfilm will in a short time be a part of the overall digitisation programme in our libraries. With the turn of this millennium we are experiencing a totally new way of using newspapers, of using our historical collections.   

During the next few years it is possible for us all, with large newspaper collections on microfilm, to digitise our collections, to build ”Digital Newspaper Libraries” for our users. Periodicals, journals and newspapers, have turned out to be a target that has a great demand on the Internet from researchers as well as the public in large.  For curators in libraries this is a great challenge. Libraries should take the responsibility of digitising these collections, to build up digital newspaper libraries, to ensure that the cultural heritage is available for all.  

The Project

The aim of the Nordic TIDEN -project was to test criterias for microfilm as a platform for digitisation and full text searching. As a result a Nordic Digital Newspaper Library has been established in October, 2001, with Nordic newspapers from 1640-1860 on the Internet, http://tiden.kb.se. We have launched a digital newspaper library with several hundred thousand pages, for the public and for researchers interested in the Nordic Countries. 

The libraries participating in TIDEN were the Royal Library of Stockholm, the National Library of Norway and the State and University Library of Århus. The co-ordinator for the project was Helsinki University Library, the Centre of Microfilming and Conservation. This project, was supported by, the Nordic Council of Scientific Information, the NORDINFO, and the libraries during 1998-2001. 

To support the Nordic project the Library in Finland started an additional project AURORA in 1999, cofinanced by the Finnish Ministry of Education. 

The project TIDEN did stretch both geographically and in a time perspective. In 2000 financing was granted by NORDINFO also to Iceland, Greenland and the Faroe Islands to start digitising their newspapers, under the project name VÄSTNORD.

Some newspaper publishers did show an interest in the project, which in Norway has resulted in including a local newspaper Nordlands Avis from the years 1893-1978 into the project and making it available in the Internet. In Finland the largest newspaper publisher Sanoma Oy will make indexes of their newspaper collections to the 1990’s available at Helsinki University Library.

The Objectives of the Nordic Project

The main objective of the project has been to use microfilm as an intermediate for future digitisation. 

Microfilm enables large-scale production of digital images. In Norway and in Finland we have built production lines for digitisation of newspapers from film. The objective has been to start with a part of the newspapers and when this project is finished, integrate the digitisation of the newspapers to the libraries ordinary functions. 

In Denmark and Norway the objective has been to get as much of the collections on the web as possible – with minimal search possibilities. That is search on title, date and place.

In Finland and Sweden we have included full text search, which enables search on each word in the textual content. In addition in Finland an Index of articles with hierarchical search terms from 1771-1890 will be available.

The Newspaper Contents

The contents in the Digital Newspaper Library of TIDEN have been chosen according to the importance of the newspapers and copyright possibilities in the member countries.

Kungliga Biblioteket chose to digitise Post- och Inrikes from 1640-1721 in Sweden. Some predecessors to this actual first newspaper have been digitised from 1620-1630’s. This means that information starting from the 30-year war in Europe is included and available today for full text searching until 1721. 

In Finland we chose to build a digital platform for the day to day life in the 18th and 19th centuries. In this project every newspaper, containing 44 titles, is digitised for full text searching from 1771-1860, starting with our first newspaper Tidningar utgifne af et Sällskap I Åbo in 1771 containing 90.000 pages. This project will be continued by a second project, which will include all newspapers until 1890, with 800.000 pages more.

Denmark continues in the TIDEN project, almost where Sweden stopped, with Adresseavisen from 1751 – 1890. Shorter periods of other titles will also be included, like Berlinske Tidende 1863-1865, Dannevirke 1862-1864 and Faedrelandet 1863-1865. 

Norway is covering almost two centuries, the 19th and 20th century with Den norske rigstiden 1815-1882 Adresseavisa 1802-1900 and Norlands Avis 1893-1978.

The number of pages we have on the net at this point is actually of minor importance. More value lies in the development and innovations we have made to build the production line, for digitisation and full text searching, for our large library collections.

As I know the Finnish project best, and as it is the most complete I will introduce it to you.

The Finnish part of the project

Helsinki University Library

Helsinki University Library – the Finnish National Library was established in 1640 and has had legal deposit right since 1707.

Our Library is carrying out most of its preservation services in the Centre for Microfilming and Conservation. It was established in 1990 in Mikkeli, 250 kilometers from Helsinki. Digitisation and microfilming, supported by the Centre’s conservation and preservation expertise, will constitute the Library’s reformatting activities. As the responsibility of library´s digitisation services are here, the work for both the TIDEN and AURORA projects are situated in the Centre. website: http://digi.lib.helsinki.fi. 

Helsinki University Library had in 1997 chosen digitisation of all newspapers  - due to their high demand - as one of the main targets of its reformatting programme. The digitisation of newspapers was also ranked as first priority by the research libraries in Finland. And as practically all newspapers in Finland have been microfilmed, it was possible to use microfilm as intermediary for digitisation needs. Due to copyright restrictions the digitisation had to be focused on pre 1890-material.

The work flow

The production has to be as fully automated as possible. However, human intervention is needed at some stages of digitisation, indexing and OCR treatment.

The individual steps of the process are as follows:

Microfilming		refilming the newspapers if the quality of the present microfilms is not good enough

Digitisation		scanning of the microfilms; the process is fast but the scanner had problems with the varying sizes of the pages on the film. A special software has been made to identify the page images and to clean the black margins. This has been proved successful and today films are scanned automatically.

OCR			conversion of the images to text files; requires many adjustments and training of the software especially for text in Gothic letters.

Identification		identification of title, issue, date, pages and attachments of the newspaper takes place in principle automatically but requires some human treatment.

Database import		Putting the prepared material to the database system

Microfilming

When using microfilm as intermediary the quality of the microfilms is the key to the success. Of course, the quality is crucial also for the preservation in general because we have to accept the simple fact that in the long run newspapers will survive only on microfilm. It is a stable medium with a life length in centuries. It is readable by the human eye and offers a very high resolution, good enough for conversions onto new carriers in the future.

Digitisation
	
The preservation microfilms in our collections have advantages and disadvantages relating to digitisation. High-contrast microfilms have a limited dynamic range, which in many cases is a good thing. This is a good feature especially if you compare the scanning result of the is quite sufficient, favouring 16x or lower reduction originals and the microfilms of grey text on brittle and brownish paper. Smaller shades, spots or wrinkles in the paper are eliminated in the film. The text is clearly extinguishable from the base. It makes the OCR- process easier, using bitonal images. 

In an economic perspective scanning from film is much faster than using the originals when oversized items like newspapers are concerned.

On the other hand photographic material is best when digitising in greyscale from the original. But there is also a clear difference between the different kinds of microfilm copies used for digitisation, because the negative direct duplicate gives the poorest results for the photographs in the newspapers.

If we want to extend the quality of our microfilming for future OCR and full text searching we could extend our microfilming process beyond using international and national standards and guidelines. We could also improve the identification of the newspaper numbers and pages by using barcodes. This would have two positive features.
-	checking of content could be atomised in the microfilming process
-	identification of digitised newspapers and pages would be available in the microfilm
This improvement is still at a testing stage at Helsinki University Library. 

OCR

In addition to making the newspapers available on the Internet digitisation has also to create a new powerful search tool because an improved access was the main reason of the whole exercise in Finland. 

The basis for full text searching is that the words in the text files are searchable which again requires that the images have to be converted into text files with the help of an Optical Character Recognition process.

Because of the Gothic letters, much used in Northern Euorpe and the poor quality in most of the original newspapers, the OCR conversion has required much effort. Much attention has been devoted to it both in the project TIDEN, where the Library in this question co-operated with the Royal Library of Stockholm, and in the co-operative project with the publishing house Sanoma Oy.

After investigating the available OCR softwares we chose Fine Reader 5.0. It is able to handle the columns of the newspapers and is not too sensitive to react too much on the unevenness of the quality of the original pages. In addition to these features the software also can handle the text in batches when it first has been trained. For example in Sanoma Oy 23 computers are continuously running, converting the images into text files.

The results on testing the text for OCR on eight different newspapers have shown that there are several factors influencing the quality of the conversion, the most important of them being the text font, language and reduction rate. Roman style, Swedish language and a low reduction rate gave the best results, in our newspapers. At the moment we have OCR read 44 titles that are available since March, 2002 at the Digital newspaper library in three main languages used in our newspapers during the 18th and 19th century: Finnish, Swedish and German.

Identification

The last point in this process, the entrying of each image and its textual version should aim at automation when dealing with large quantities of material.  Usually, every image has to be addressed on behalf of the title, date, number and page.

We have built a software for the identification of large quantities of images. By indicating the title and ISSN-number of a publication this can automatically be used for all chosen newspaper numbers. Each newspaper number, when chosen, get in turn automatically correct page numbers. 

When the identification step has been completed the software creates an XML file that is copied to the server along with images and text files. 

Database import

The server picks up new files from server and process them automatically. At this point surrograte images in various formats is created. Database importing is done by custom java software

The Engine Room

Much effort has been put into developing a proper functional architecture for the digital newspaper library, as a part of an overall digital architecture. This will include all kinds of materials and the production of the digital versions as the digital original, the digital master and the digital image on the web and the textual versions for full text searching. URN identificators are generated automatically.

We have also information on articles and the authors of them together with search words. The database structure is combined with permanent archiving and retrieval. Solutions have to be applicaple for other types of material such as maps, periodicals and books.

Text search software

From the very beginning it was obvious that a hundred per cent OCR conversion is impossible. It was equally obvious that it was unthinkable to start proofreading the great mass of the converted text manually. That is why two decisions were made. 

The first one was that the basic tool for the users of the digitised newspapers will be the digital facsimile of the original pages. The ASCII version will be used for searching purposes only. 

The other decision was to find a retrieval software which could manage a limited percentage of errors. This is also needed for the retrieval of the old fashioned language used in the newspapers. 

The Royal Library of Stockholm had tested an advanced retrieval software Convera Retrievalware by Excalibur. It has among other features a so called fuzzy search function which is able to identify the searched words even if one to three letters would differ from the word sought for – because of misspelling or old fashioned language. It process’ the words as bit-strings and uses pattern recognition to find matches. Therefore Retrievalware was chosen. Finnish and Swedish language support has been added to the normal search functions.

In action

The Nordic Digital Newspaper Libraries were opened to the general public on 25th of October 2001 in a press conference in Mikkeli along with the joint project website. The Nordic and the Finnish Historical Newspaper Library was prominately featured on the largest Finnish newspaper Helsingin Sanomat and later on in Finnish television. On first day there were 90 000 hits and at the time of writing (30th of January)  the total hitcount is about 800 000. Systems have been running without problems since the launch. We have received plenty of user feedback on the system. Most of it has been positive along with some error reports and improvement suggestions. 

The website of Finnish researchers of history AGRICOLA has noticed the TIDEN- project and been most enthusiastic about it. It will be an important cornerstone of the virtual library of the future. Based on the experience and the infrastructure developed with the help of these projects the National Library of Finland will create a national digitisation service, which will be strong enough to assume the responsibility for large national projects. 

Conclusions
 
There is a great demand for Digital Historical Newspaper Libraries. There have been newspapers published for almost half a millennium now. The TIDEN project covers almost four centuries of which one and a half with full text search. For the Nordic project the interfaces of the participating libraries will be searchable via the TIDEN web page http://tiden.kb.se. We are still working together to enhance search possibilities between the Nordic newspaper databases. Perhaps cross language search will be the next step after that. 

With microfilm, researchers have used a limited number of titles for historical research. By fulltext search the sources will be widely extended. It might be that research will focus in other new directions. The history on everyday life is now searchable as never before. Boarders are abandoned.