w/c 12 November – Discovery news roundup

One particularly interesting thing I noticed this past month was that tweets about open data, linked data and metadata were starting to come thick and fast from people within my network who sit well outside the library and cultural data domains. In particular the tweets from attendees of the Lasa’s Charity Digital Summit and the ‘Nesta in Manchester’ event about innovation seemed to include a rich vein of tweets about all things open. Perhaps an indicator that open data’s tipping point is approaching?

Some highlights from the world of resource discovery and open data in recent weeks:

• Clay Shirky participated in some high profile advocacy for openness when he delivered his keynote at the recent Educause conference. He argues that the business case for opening up data comes as a result of opening up data – which is not a new assertion but it’s a useful reminder that waiting until there’s a clear business case before taking action could be a riskier strategy than it seems.
• The Programmable Web’s API directory contains 49 APIs and, as well as listing them all, their blogpost highlights some of the mashups that have been created using library APIs.
• There was a great deal of interest in the Rijksmuseum’s recent launch of Rijksstudio which allows anyone to freely download from a choice of 125,000 high-res images from their collection and use the images as they like. They also provide access to their collection’s metadata via an API and their website already lists an impressive number of apps built using their API. You can read our recently published case study on the Rijksmuseum.
• IOP Publishing announced that they have moved to a Creative Commons licence for their open access articles and basic bibliographic metadata, in order to increase the visibility of the research they publish.

Updates from a couple of large-scale projects in Europe and the US:

• As the Europeana Libraries project draws to a close they have taken the opportunity to highlight the achievements of the project and also introduce the next phase of the project: Europeana Cloud, which is set to make a further 2.4 million metadata records available. Europeana also shared a preview of their new search interface which provides an API console that developers can use to build and test search requests.
• The Digital Public Library of America reported on how the initiative has been moving forward and shared their Technical Principles which include a strong commitment to open metadata. Last week they held their first ‘Appfest Hackathon‘ which saw participants envisaging new ways to utilise cultural heritage metadata and content from the DPLA’s digital hubs.

News from Discovery partner organisations:

• Research Libraries UK (RLUK) published their ‘Hidden Collections’ report which focuses on the problem of uncatalogued items held in UK research libraries. From a discovery point of view these items (which are estimated to represent nearly 20% of the libraries book stocks) are simply invisible so it will be interesting to watch the collective initiatives which look set to be launched in order to address the issue at an above-campus level.
• The M25 Consortium of Academic Libraries launched Search25, a ‘regional resource discovery tool’, which enables users to search, locate and obtain resources from any library within the consortium. The tool is just one outcome of the JISC-funded Search25 project which has paved the way for the possible adoption of the Discovery Open Metadata Principles in the future.
• The JISC-funded SPRUCE project (Sustainable PReservation Using Community Engagement) have announced a rough schedule of mashups and hackathons for museum, library and archive professionals which will be taking place next year. The main focus of the events is digital sustainability and previous events have included a strong focus on extracting, aggregating and mapping metadata.

Please note that places are still available for the second of our free Discovery Licensing Clinics on 30 November in London. It is an opportunity for managers and decision makers from libraries, archives and museums to get practical advice on open data licensing from our assembled team of experts.

Some preliminary highlights from the Discovery programme

The Discovery programme is nearing its completion date of December 2012. Most of the projects have finished or are wrapping up. Our efforts are now directed towards gathering together all that we have learned and produced in the programme.

The programme has covered a lot of ground so pulling everything together will take us some time. While that happens I thought it might be worth listing a selection of preliminary highlights of the programme. This blog is based on a talk I gave at the RLUK conference so the focus is on libraries and archives rather than museums.

Future approaches to Discovery

It is not clear what the future is for resource discovery. It is unlikely that there will just be one approach to resource discovery for libraries, museums and archives. The future is likely to be plural. While discovery has not developed firm answers on what the future is. We have experimented with a range of approaches and have identified those that are promising.

These approaches are recorded in the Discovery case studies and guidance site. They can be used to inform future plans in libraries, museums and archives. Or if the approaches seem promising enough they can be emulated or the tools that have been developed can be used. We are planning to produce a toolkit so that all these tools are in one place.

What is clear is that we are not alone in experimenting with these kind of approaches. This is a global movement with many and diverse institutions exploring similar approaches. The case studies and guidance recognise this by including explorations of the approaches of the Wellcome Trust, the Rijksmuseum and the Victoria and Albert Museum.

Innovative cataloguing

Resource discovery starts with cataloguing. The focus of the programme was not on cataloguing but a couple of interesting innovative approaches have emerged from the project.

The Institute of Education decided to explore new ways of cataloguing their collection. This involves the creation of basic records in Drupal, enriching of these records using professional cataloguer input then exporting of these records into the LMS. This may sound a roundabout way of doing things as I have written it but it was 3.5 times quicker and therefore cheaper than the current approaches. This allows the catalloguer to concentrate on the record enrichment by adding index terms. Full figures are available on their blog. They also developed lightweight ways to catalogue uncatalogued material which offers a significant saving in researcher time when using the material. More detail on this on the blog.

The second exploration of catalogues focused on the collection as a whole. The Copac collections management project used the copac data to create tool to allow librarians to analyse their collections and make decisions on which items can be removed from the collection and which are rare and need to be retained. This tool has been trialled by a number of libraries. During their trial, the University of Manchester found that the tool was 86% more effective than manual checking of the collection. Details on how this figure was arrived at can be found in the case study.

Greater impact through linking

Linking items in collections with relevant items in other collections offers the possibility of enabling richer resource discovery services and supports new and emerging research interests. Linked data is an intriguing option for enabling this. I don’t think the discovery programme has come up with a definitive answer on whether linked data is the future for libraries, museums and archives. But I think that the evidence is fairly strong that it will be a part of the future.

The programme included a number of projects experimenting with linked data for libraries and archives and there is work to be done to gather all of these together. However there are some headlines that we can report now:

• The use case in archives seems to be strong as linking resources by place and person is something that should be useful to researchers and students
• The step change project worked with Axiell to update CALM so that archives can create linked data records from within CALM. This functionality will be included in the next update and has the potential to benefit the large number of archives that use CALM. This linked data creation functionality is also available as a stand alone tool called Alicat.
• Cambridge were able to create linked data records for 2.3 million books for their project which cost just under £40,000.
• The ArchivesHub project Linking Lives has worked to use people as hooks to explore archive collections. This uses linked data and the model they have developed is being reused internationally. 
• The Pelagios project has created a way to use linked data to identify ancient places in archive collections and there is a vibrant community growing around their approach.

Of course the Discovery programme is not alone in investigating linked data. The Library of Congress, OCLC, The British Library, Europeana and the DPLA are all using or investigating some form of linked data technology in pursuing their aims.

Linked data is not the only option for bringing different collections together and allowing people to use them in new ways. This can also be done with APIs and there are two discovery exemplar projects doing just this for Shakespeare and for WW1. Work on these is still underway but both are looking promising and offer some very interesting lessons for how to aggregate collections to enable new forms of resource discovery and research.

Enhanced shared services

We already have many shared services that help people discover those resources. Throughout the programme we have worked with those services to enhance them to help realise the resource discovery taskforce vision. It s worth a separate post on all of the ways the services have been developed so for now, I will just list the services that have been developed in the programme:

• Aim25
• ArchivesHub
• COPAC
• Culture Grid
• GoGeo
• MediaHub
• OpenURL Router
• Search25
• SUNCAT
• UK Archival Thesaurus

Business case

These are challenging economic times so it was important to address the business case for libraries, museums and archives to invest effort in improving resource discovery. The results of this work can be seen in the business case section of the discovery guidance. We worked with senior managers from libraries, museums and archives throughout the programme to ensure what we were doing address their needs. As part of this work we produced a series of videos where a selection of senior managers talk about their needs, challenges and predictions and they make for interesting viewing.

What’s next?

We are in the process of reviewing the Discovery programme and the resource discovery taskforce vision that kicked it all off. This review will produce a set of recommendations on what we should do next. These will be available in January. We will be looking to pull all of the outputs from the programme into a form that makes it easy for people to learn from the programme and to use what has been produced. We are also in the process of putting together an event for 2013 that brings together people from around the world that are working on addressing resource discovery challenges and seeing what we can learn from each other. More information on all of these things to follow soon.