Several of us at The British Library took part in the CURATEcamp file id hackathon on Friday.

We decided that one issue we could make a useful impact on was identification of various ebook formats. eBooks are an important content type for the British Library, especially with the expected implementation of non-print legal deposit legislation next year. For a long list of formats look here: http://wiki.mobileread.com/wiki/E-book_formats

Having gathered a test corpus of various ebook formats, we set about looking at how the files were recognised by file, Tika and Droid.  Initial testing showed that Droid did not recognise formats other than AZW/Mobi, PDF & TXT, whilst Tika misidentified many of the formats, particularly as chemical/x-pdb.  File was much better at recognising more of the formats, but it did not identify all files.  Whilst file had the best coverage of all the tools it did not associate a useful mime type.  As, perhaps, a sensible default position over misidentification, it identified the mime type of many files as application/octet-stream.

Table showing results before our work on file signatures was carried out:

*indicates no mimetype specified by tool.  Results in italics indicate incorrect identification.

We then spent the day developing signatures that Tika could use to identify the ebooks in our test corpus by using fidget.  By my count we added thirteen signatures to a custom-mimetypes.xml which enabled Tika to correctly identify all of our test files by the end of the day.

Members of the team who were not creating signatures were busy researching existing mime types for formats, identifying where the identification was failing, identifying discrepancies in the output from the three tools and investigating the effect of rights management/encryption on epubs.

We still have some work to do, particularly around determining how correct and robust our new signatures are. We need more test files, with ground truth, to check the new signatures.  One concern is that we may be recognising the same file format, created by multiple sources, in multiple ways.  For example – are PDB files all different or compatible?  i.e. are we recognising the producer of the file and not the format?  An analogy could be PDFs created by different tools; they are one output format, and not different formats depending on producer.

We also need to consider more closely the implications of creating mime types, which we were forced to do for some formats.  What impact will this have on different tools’ abilities to, in theory, provide the same information about the same files, if the new mime types are not shared?

In a more general sense how does false-positive identification of files affect any assessment of the contents of a repository?  What are the digital preservation risks of a false or missed identification?  What should be done when identification tools don’t agree?