By Michael Shallcross
At SAA 2015, Courtney Mumma (formerly of Artefactual Systems) and I participated in a panel discussion at the Electronic Records Section meeting on “implementing digital preservation tools and systems,” with a focus on “the lessons learned through the planning, development, testing, and production of digital preservation applications.”
The University of Michigan’s Bentley Historical Library is in the midst of a two-year project (2014-2016) funded by the Andrew W. Mellon Foundation to integrate ArchivesSpace, Archivematica, and DSpace in an end-to-end digital archives workflow (for more information on the project itself, see our blog).
Artefactual Systems is responsible for the development work on the project, which has involved adding new functionality to the Archivematica digital preservation system to permit the appraisal and arrangement of digital archives as well as the integration of ArchivesSpace functionality within Archivematica so that users can create and edit archival description in addition to associating digital objects with that information.
Working closely with a team of software developers has been highly revelatory for us here at the Bentley, as we’ve never had a project of a similar scale and scope. Among the many lessons we’ve learned from the experience, I’d like to call particular attention to the following:
- Balancing the feasibility of a project with expectations can be difficult! The poet Rainer Maria Rilke wrote, “You see, I want a lot. Perhaps I want everything”–and that’s a pretty fair characterization of where we were at the beginning of the project. We were like kids in a candy store, and as a result we had some (perhaps) unreasonably high expectations of what could be accomplished over the course of our grant project.
It’s really easy for non-developers to take the ease and simplicity of well-designed apps for granted and so the staff of Artefactual Systems have been instrumental in helping us temper our expectations. In particular, their developers have been great at alerting us when ideas would take an inordinate amount of time or effort–resources that would be much better spent on refining core functionality instead of adding on non-critical bells and whistles.
- Advocating for unique local needs while creating something for the community. From the get-go, we knew we wanted the outcomes of our project to be relevant and usable by the archives and digital preservation communities at large. This responsibility was made explicit in our award agreement with the Mellon Foundation, which specifies that all code and documentation will be made freely and publicly available. Of course, this very straightforward goal is somewhat complicated by the fact that we have been employing a homegrown system and procedures for our digital archives ingest process. Our use of DSpace as a preservation repository and access portal for our born-digital archives has also forced us to adopt some conventions (such as concatenating archival description into a single title to compensate for DSpace’s flat structure) that might be unnecessary in another environment. At the same time, we learned earlier this year that the University of Michigan will be moving to the Hydra repository platform in the coming years and so we needed to be sure that this move away from DSpace was also accommodated by our project outcomes. We’ve sought to address these considerations with two main strategies.
First, Artefactual Systems is committed to open source software and has implemented recognized standards throughout the design and development process. This will ensure that the end products are not locked into any proprietary systems and may be modified and configured by other institutions. Using established standards also facilitates interoperability among different platforms (including ArchivesSpace, Archivematica, and DSpace) and local infrastructure.
Second, we’ve made every attempt to leave the workflows and procedures flexible and scalable to avoid being overly prescriptive. We understand that a wide diversity of practices are in place among the archives and digital preservation communities: Some institutions work at an item level while others take an MPLP, aggregate approach. Likewise, local policies or regulations may require certain procedures (such as the identification of sensitive data) or restrictions for some classes of content. Through the configuration of settings within Archivematica or customization of microservices, a very broad swath of potential users should be able to have key needs met. The final products of the grant will also be modular, so that institutions may elect to adopt some, all, or none of the features we are developing (for instance, an institution may want to use the appraisal functionality but may have no need for ArchivesSpace integration). In cases where the Bentley does have specific and possibly unique needs (such as packaging content in .zip files to simplify the management and downloading of content), users will be able to turn features on/off in Archivematica’s administration pane. In trying to accommodate varieties of practice, the Bentley has relied upon input from Artefactual Systems as well as peer institutions that are also using ArchivesSpace, Archivematica, and/or DSpace.
- Embracing agile development methodologies. This project has also marked our first experience with an “agile” development methodology, an iterative, incremental approach to software development which incorporates repeated, relatively rapid rounds of requirements analysis, coding and client acceptance testing. This stands in contrast to more traditional approaches, which involve long periods of development followed by deploying completed or nearly-completed features ready for production use. The agile method allows the software developers at Artefactual and the archivists at the Bentley to work together to iteratively refine requirements, introducing new requirements and re-prioritizing existing ones as the need arises. Initially, this process gave us a better appreciation for how our functional requirements and user stories could be translated into specific development tasks. We have also appreciated how the agile approach allows us to break down this very complex project into “sprints”: specific goals or tasks that are much more manageable in scope and which permit us to change priorities or plans as needed and also build upon lessons from earlier sprints (both successes and failures).
Progress on the project continues (and took us to iPRES 2015 for both a poster presentation and a joint workshop called “Using Open-Source Tools to Fulfill Digital Preservation Requirements”); you can get updates via our project blog or by following us on twitter (@umbhlcuration). Thanks for your time and feel free to drop me a line (shallcro[at]umich[dot]edu) if you have any questions or comments.
Michael Shallcross is an Assistant Director for Curation at the Bentley Historical Library and Adjunct Lecturer in Information at the University of Michigan School of Information.