Embedded Archives at the Institute for Social Research

by Kelly Chatain

This is the fourth post in the BloggERS Embedded Series.

As any archivist will tell you, the closer you can work with creators of digital content, the better. I work for the Institute for Social Research (ISR) at the University of Michigan. To be more precise, I am a part of the Survey Research Center (SRC), one of five centers that comprise the Institute and the largest academic social science research center in the United States. But really, I was hired by the Survey Research Operations (SRO) group, the operational arm of SRC, that conducts surveys all over the world collecting vast and varied amounts of data. In short, I am very close to the content creators. They move fast, they produce an extraordinary amount of content, and they needed help.

Being an ‘embedded’ archivist in this context is not just about the end of the line; it’s about understanding and supporting the entire lifecycle. It’s archives, records management, knowledge management, and more, all rolled into one big job description. I’m a functional group of one interacting with every other functional group within SRO to help manage research records in an increasingly fragmented and prolific digital world. I help to build good practices, relationships, and infrastructure among ourselves and other institutions working towards common scientific goals.

Lofty. Let’s break it down a bit.

Find it, back it up, secure it

When I arrived in 2012, SRO had a physical archive of master study files that had been tended to by survey research staff over the years. These records provide important reference points for sampling and contacting respondents, designing questionnaires, training interviewers, monitoring data collection activities, coding data, and more. After the advent of the digital age, a few building moves, and some server upgrades, they also had an extensive shared drive network and an untold number of removable media containing the history of more recent SRO work. My first task was to centralize the older network files, locate and back up the removable media, and make sure sensitive data was out of reach. Treesize Professional is a great tool for this type of work because it creates detailed reports and clear visualizations of disk space usage. This process also produced SRO’s first retention schedule and an updated collection policy for the archive.

Charts produced by Treesize Professional used for the initial records survey and collection.
A small selection of removable media containing earlier digital content.

Welcome, GSuite

Despite its academic home, SRO operates more like a business. It serves University of Michigan researchers as well as external researchers (national and international), meeting the unique requirements for increasingly complex studies. It maintains a national field staff of interviewers as well as a centralized telephone call center. The University of Michigan moved to Google Apps for Education (now GSuite) shortly after I arrived, which brought new challenges, particularly in security and organization. GSuite is not the only documentation environment in which SRO operates, but training in the Googleverse coincided nicely with establishing guidance on best practices for email, file management, and organization in general. For instance, we try to label important emails by project (increasingly decisions are documented only in email) which can then be archived with the other documentation at the end of the study (IMAP to Thunderbird and export to pdf; or Google export to .mbox, then into Thunderbird). Google Drive files are downloaded to our main projects file server in .zip format at the end of the study.

Metadata, metadata, metadata

A marvelous video on YouTube perfectly captures the struggle of data sharing and reuse when documentation isn’t available. The survey data that SRO collects is delivered to the principal investigator, but SRO also collects and requires documentation for data about the survey process to use for our own analysis and design purposes. Think study-level descriptions, methodologies, statistics, and more. I’m still working on finding that delicate balance of collecting enough metadata to facilitate discovery and understanding while not putting undue burden on study staff. The answer (in progress) is a SQL database that will extract targeted structured data from as many of our administrative and survey systems as possible, which can then be augmented with manually entered descriptive metadata as needed. In addition, I’m looking to the Data Documentation Initiative, a robust metadata standard for documenting a wide variety of data types and formats, to promote sharing and reuse in the future.

DDI is an international standard for describing data.

Preserve it

The original plan for digital preservation was to implement and maintain our own repository using an existing open-source or proprietary system. Then I found my new family in the International Association for Social Science Information Services & Technology (IASSIST) and realized I don’t have to do this alone. In fact, just across the hall from SRO is the Inter-University Consortium for Political and Social Research (ICPSR), who recently launched a new platform called Archonnex for their data archive(s). Out of the box, Archonnex already delivers much of the basic functionality SRO is looking for, including support for the ever-evolving  preservation needs of digital content, but it can also be customized to serve the particular needs of a university, journal, research center, or individual department like SRO.

Searching for data in OpenICPSR, built on the new Archonnex platform.

 

The embedded archivist incorporates a big picture perspective with the specific daily challenges of managing records in ways that not many positions allow. And you never know what you might be working on next…


Kelly Chatain is Associate Archivist at the Institute for Social Research, University of Michigan in Ann Arbor. She holds an MS from Pratt Institute in Brooklyn, NY.

Advertisements

Processing Digital Research Data

By Elise Dunham

This is the sixth post in our Spring 2016 series on processing digital materials.

———

The University of Illinois at Urbana-Champaign’s (Illinois) library-based Research Data Service (RDS) will be launching an institutional data repository, the Illinois Data Bank (IDB), in May 2016. The IDB will provide University of Illinois researchers with a repository for research data that will facilitate data sharing and ensure reliable stewardship of published data. The IDB is a web application that transfers deposited datasets into Medusa, the University Library’s digital preservation service for the long-term retention and accessibility of its digital collections. Content is ingested into Medusa via the IDB’s unmediated self-deposit process.

As we conceived of and developed our dataset curation workflow for digital datasets ingested in the IDB, we turned to archivists in the University Archives to gain an understanding of their approach to processing digital materials. [Note: I am not specifying whether data deposited in the IDB is “born digital” or “digitized” because, from an implementation perspective, both types of material can be deposited via the self-deposit system in the IDB. We are not currently offering research data digitization services in the RDS.] There were a few reasons for consulting with the archivists: 1) Archivists have deep, real-world curation expertise and we anticipate that many of the challenges we face with data will have solutions whose foundations were developed by archivists and 2) If, through discussing processes, we found areas where the RDS and Archives have converging preservation or curation needs, we could communicate these to the Preservation Services Unit, who develops and manages Medusa, and 3) I’m an archivist by training and I jump on any opportunity to talk with archivists about archives!

Even though the RDS and the University Archives share a central goal–to preserve and make accessible the digital objects that we steward–we learned that there are some operational and policy differences between our approaches to digital stewardship that necessitate points of variance in our processing/curation workflow:

Appraisal and Selection

In my view, appraisal and selection are fundamental to the archives practice. The archives field has developed a rich theoretical foundation when it comes to appraisal and selection, and without these functions the archives endeavor would be wholly unsustainable. Appraisal and selection ideally tend to occur in the very early stages of the archival processing workflow. The IDB curation workflow will differ significantly–by and large, appraisal and selection procedures will not take place until at least five years after a dataset is published in the IDB–making our appraisal process more akin to that of an archives that chooses to appraise records after accessioning or even during the processing of materials for long-term storage. Our different approaches to appraisal and selection speak to the different functions the RDS and the University Archives fulfill within the Library and the University.

The University Archives is mandated to preserve University records in perpetuity by the General Rules of the University, the Illinois State Records Act. The RDS’s initiating goal, in contrast, is to provide a mechanism for Illinois researchers to be compliant with funder and/or journal requirements to make results of research publicly available. Here, there is no mandate for the IDB to accept solely what data is deemed to have “enduring value” and, in fact, the research data curation field is so new that we do not yet have a community-endorsed sense of what “enduring value” means for research data. Standards regarding the enduring value of research data may evolve over the long-term in response to discipline-specific circumstances.

To support researchers’ needs and/or desires to share their data in a simple and straightforward way, the IDB ingest process is largely unmediated. Depositing privileges are open to all campus affiliates who have the appropriate University log-in credentials (e.g., faculty, graduate students, and staff), and deposited files are ingested into Medusa immediately upon deposit. RDS curators will do a cursory check of deposits, as doing so remains scalable (see workflow chart below), and the IDB reserves the right to suppress access to deposits for a “compelling reason” (e.g., failure to meet criteria for depositing as outlined in the IDB Accession Policy, violations of publisher policy, etc.). Aside from cases that we assume will be rare, the files as deposited into the IDB, unappraised, are the files that are preserved and made accessible in the IDB.

Preservation Commitment

A striking policy difference between the RDS and the University Archives is that the RDS makes a commitment to preserving and facilitating access to datasets for a minimum of five years after the date of publication in the Illinois Data Bank.

The University Archives, of course, makes a long-term commitment to preserving and making accessible records of the University. I have to say, when I learned that the five-year minimum commitment was the plan for the IDB, I was shocked and a bit dismayed! But after reflecting on the fact that files deposited in the IDB undergo no formal appraisal process at ingest, the concept began to feel more comfortable and reasonable. At a time when terabytes of data are created, oftentimes for single projects, and budgets are a universal concern, there are logistical storage issues to contend with. Now, I fully believe that for us to ensure that we are able to 1) meet current, short-term data sharing needs on our campus and 2) fulfill our commitment to stewarding research data in an effective and scalable manner over time, we have to make a circumspect minimum commitment and establish policies and procedures that enable us to assess the long-term viability of a dataset deposited into the IDB after five years.

The RDS has collaborated with archives and preservation experts at Illinois and, basing our work in archival appraisal theory, have developed guidelines and processes for reviewing published datasets after their five-year commitment ends to determine whether to retain, deaccession, or dedicate more stewardship resources to datasets. Enacting a systematic approach to appraising the long-term value of research data will enable us to allot resources to datasets in a way that is proportional to the datasets’ value to research communities and its preservation viability.

Convergences

To show that we’re not all that different after all, I’ll briefly mention a few areas where the University Archives and the RDS are taking similar approaches or facing similar challenges:

  • We are both taking an MPLP-style approach to file conversion. In order to get preservation control of digital content, at minimum, checksums are established for all accessioned files. As a general rule, if the file can be opened using modern technology, file conversion will not be pursued as an immediate preservation action. Establishing strategies and policies for managing a variety of file formats at scale is an area that will be evolving at Illinois through collaboration of the University Archives, the RDS, and the Preservation Services Unit.
  • Accruals present metadata challenges. How do we establish clear accrual relationships in our metadata when a dataset or a records series is updated annually? Are there ways to automate processes to support management of accruals?
  • Both units do as much as they can to get contextual information about the material being accessioned from the creator, and metadata is enhanced as possible throughout curation/processing.
  • The University Archives and the RDS control materials in aggregation, with the University Archives managing at the archival collection level and the RDS managing digital objects at the dataset level.
  • More? Certainly! For both the research data curation community and the archives community, continually adopting pragmatic strategies to manage the information created by humans (and machines!) is paramount, and we will continue to learn from one another.

Research Data Alliance Interest Group

If you’re interested in further exploring the areas where the principles and practices in archives and research data curation overlap and where they diverge, join the Research Data Alliance (RDA) Archives and Records Professionals for Research Data Interest Group. You’ll need to register with the RDA, (which is free!), and subscribe to the group. If you have any questions, feel free to get in touch!

IDB Curation Workflow

The following represents our planned functional workflow for handling dataset deposits in the Illinois Data Bank:

Dunham_ProcessingDigitalReserachData_PublishedDepositScan_ERSblog_1
Workflow graphic created by Elizabeth Wickes. Click on the image to view it in greater detail.

Learn More

To learn more about the IDB policies and procedures discussed in this post, keep an eye on the Illinois Data Bank website after it launches next month. Of particular interest on the Policies page will be the Accession Policy and the Preservation Review, Retention, Deaccession, Revision, and Withdrawal Procedure document.

Acknowledgements

Bethany Anderson and Chris Prom of the University of Illinois Archives

The rest of the Research Data Preservation Review Policy/Procedures team: Bethany Anderson, Susan Braxton, Heidi Imker, and Kyle Rimkus

The rest of the RDS team: Qian Zhang, Elizabeth Wickes, Colleen Fallaw, and Heidi Imker

———

Dunham_ProcessingDigitalReserachData_PublishedDepositScan_ERSblog_2Elise Dunham is a Data Curation Specialist for the Research Data Service at the University of Illinois at Urbana-Champaign. She holds an MLS from the Simmons College Graduate School of Library and Information Science where she specialized in archives and metadata. She contributes to the development of the Illinois Data Bank in areas of metadata management, repository policy, and workflow development. Currently she co-chairs the Research Data Alliance Archives and Records Professionals for Research Data Interest Group and is leading the DACS workshop revision working group of the Society of American Archivists Technical Subcommittee for Describing Archives: A Content Standard.