Embedded Archives at the Institute for Social Research

by Kelly Chatain

This is the fourth post in the BloggERS Embedded Series.

As any archivist will tell you, the closer you can work with creators of digital content, the better. I work for the Institute for Social Research (ISR) at the University of Michigan. To be more precise, I am a part of the Survey Research Center (SRC), one of five centers that comprise the Institute and the largest academic social science research center in the United States. But really, I was hired by the Survey Research Operations (SRO) group, the operational arm of SRC, that conducts surveys all over the world collecting vast and varied amounts of data. In short, I am very close to the content creators. They move fast, they produce an extraordinary amount of content, and they needed help.

Being an ‘embedded’ archivist in this context is not just about the end of the line; it’s about understanding and supporting the entire lifecycle. It’s archives, records management, knowledge management, and more, all rolled into one big job description. I’m a functional group of one interacting with every other functional group within SRO to help manage research records in an increasingly fragmented and prolific digital world. I help to build good practices, relationships, and infrastructure among ourselves and other institutions working towards common scientific goals.

Lofty. Let’s break it down a bit.

Find it, back it up, secure it

When I arrived in 2012, SRO had a physical archive of master study files that had been tended to by survey research staff over the years. These records provide important reference points for sampling and contacting respondents, designing questionnaires, training interviewers, monitoring data collection activities, coding data, and more. After the advent of the digital age, a few building moves, and some server upgrades, they also had an extensive shared drive network and an untold number of removable media containing the history of more recent SRO work. My first task was to centralize the older network files, locate and back up the removable media, and make sure sensitive data was out of reach. Treesize Professional is a great tool for this type of work because it creates detailed reports and clear visualizations of disk space usage. This process also produced SRO’s first retention schedule and an updated collection policy for the archive.

Charts produced by Treesize Professional used for the initial records survey and collection.
A small selection of removable media containing earlier digital content.

Welcome, GSuite

Despite its academic home, SRO operates more like a business. It serves University of Michigan researchers as well as external researchers (national and international), meeting the unique requirements for increasingly complex studies. It maintains a national field staff of interviewers as well as a centralized telephone call center. The University of Michigan moved to Google Apps for Education (now GSuite) shortly after I arrived, which brought new challenges, particularly in security and organization. GSuite is not the only documentation environment in which SRO operates, but training in the Googleverse coincided nicely with establishing guidance on best practices for email, file management, and organization in general. For instance, we try to label important emails by project (increasingly decisions are documented only in email) which can then be archived with the other documentation at the end of the study (IMAP to Thunderbird and export to pdf; or Google export to .mbox, then into Thunderbird). Google Drive files are downloaded to our main projects file server in .zip format at the end of the study.

Metadata, metadata, metadata

A marvelous video on YouTube perfectly captures the struggle of data sharing and reuse when documentation isn’t available. The survey data that SRO collects is delivered to the principal investigator, but SRO also collects and requires documentation for data about the survey process to use for our own analysis and design purposes. Think study-level descriptions, methodologies, statistics, and more. I’m still working on finding that delicate balance of collecting enough metadata to facilitate discovery and understanding while not putting undue burden on study staff. The answer (in progress) is a SQL database that will extract targeted structured data from as many of our administrative and survey systems as possible, which can then be augmented with manually entered descriptive metadata as needed. In addition, I’m looking to the Data Documentation Initiative, a robust metadata standard for documenting a wide variety of data types and formats, to promote sharing and reuse in the future.

DDI is an international standard for describing data.

Preserve it

The original plan for digital preservation was to implement and maintain our own repository using an existing open-source or proprietary system. Then I found my new family in the International Association for Social Science Information Services & Technology (IASSIST) and realized I don’t have to do this alone. In fact, just across the hall from SRO is the Inter-University Consortium for Political and Social Research (ICPSR), who recently launched a new platform called Archonnex for their data archive(s). Out of the box, Archonnex already delivers much of the basic functionality SRO is looking for, including support for the ever-evolving  preservation needs of digital content, but it can also be customized to serve the particular needs of a university, journal, research center, or individual department like SRO.

Searching for data in OpenICPSR, built on the new Archonnex platform.

 

The embedded archivist incorporates a big picture perspective with the specific daily challenges of managing records in ways that not many positions allow. And you never know what you might be working on next…


Kelly Chatain is Associate Archivist at the Institute for Social Research, University of Michigan in Ann Arbor. She holds an MS from Pratt Institute in Brooklyn, NY.

Embedded Memory

by Michelle Ganz

This is the third post in the BloggERS Embedded Series.

As an archivist in a small corporate repository for an architecture, design, and consulting firm I have a unique set of challenges and advantages. By being embedded with the creators as they are creating, I have the opportunity to ensure that archival standards are applied at the point of creation rather than after the collection has been transferred to a repository.

My environment is especially unique in the types of digital files I’m collecting. From the architecture side, William McDonough + Partners, I acquire architectural project files (CAD files), sketches, renderings, photographs and video of project phases, presentations, press, media files surrounding the project, and other associated materials. Many of these files are created on specialized, and expensive, software.

From McDonough Innovation, William McDonough’s sustainable development advising firm, I collect the world-changing ideas that Mr. McDonough is developing as they evolve. Mr. McDonough is a global thought leader who provides targeted ideas, product concepts, and solutions to a wide range of sustainable growth issues faced by corporate officers, senior executives, product designers, and project managers. He often works with CEOs to set the vision, and then with the management team to set goals and execute projects. His values-driven approach helps companies to embed sustainable growth principles into their corporate culture and to advance progress toward their positive vision. Archiving an idea is a multi-faceted endeavor. Materials can take the form of audio notes, sketches in a variety of programs, call or meeting recordings, and physical whiteboards. Since my role is embedded within the heart of Mr. McDonough’s enterprises, I ensure that all the right events are captured the right way, as they happen. I gather all the important contextual information and metadata about the event and the file. I can obtain permissions at the point of creation and coordinate directly with the people doing the original capture to ensure I get archival quality files.

Challenges in this environment are very different than what my academic counterparts face. In the academic world there was a chain of leadership that I could advocate to as needed. In my small corporate world there is no one to appeal to once my boss makes up their mind. Corporate interests are all focused on ROI (return on investment), and an archival department is a financial black hole; money is invested, but they will never see a financial return. This means that every new project must show ROI in more creative ways. I focus on how a project will free up other people to do more specialized tasks. But even this is often not enough, and I find myself advocating for things like file standards and server space. Many of the archival records are videos of speeches, events, meetings, or other activities and take up a huge amount of server space. A single month’s worth of archival files can be as large as 169 GB. In the academic setting where the archives is often a part of the library, the IT department is more prepared for the huge amounts of data that come with modern libraries; folding the archival storage needs into this existing digital preservation framework is often just a matter of resource allocation or funds.

Also, nearly every archival function that interacts with outside entities requires permissions these firms are not used to giving. Meetings can include people from 3 or 4 companies in 4 or 5 countries with a variety of NDAs in place with some, but not all, of the parties. In order to record a meeting I must obtain permission from every participant; this can be rather complicated and can create a lot of legal and privacy issues. A procedure was put in place to request permission to record when meetings are set up, as well as when meetings are confirmed. A spreadsheet was created to track all of the responses. For regular meeting participants annual permissions are obtained. This procedure, while effective, is time-consuming. For many meeting participants they are unfamiliar with what an archive is. There are many questions about how the information will be used, stored, disseminated, and accessed.  There are also a lot of questions around the final destination of the archive and what that means for their permissions. To help answer these questions I created fact sheets that explain what the archives are, how archival records are collected and used, deposit timelines, copyright basics, and links to more information. To further reassure participants, we give them the option of asking for a meeting to be deleted after the fact.

This is the server stack for the archives and the two firms. The archive uses 2 blades.
This hub connects the archive to workstations and handles the transfer of TBs of data.

Preservation and access are unique challenges, especially with the architecture files. Many of the project-related files are non-traditional file formats like .dwg, .skb, .indd, .bak, et al., and are created in programs like AutoCAD and SketchUp Pro. I work with the IT department to ensure that the proper backups are completed. We back up to a local server as well as one in the city, but offsite, and a third dark archive in California. I also perform regular checks to confirm the files can open. Due to the fact that projects are often reopened years later, it is impractical to convert the files to a more standardized format. To ensure some level of access without specialized terminals, final elements of the project are saved in a .pdf format. This includes final drawings/renderings and presentations.

Furthermore, I often find myself in the awkward position of arguing with records creators in favor of keeping files that they don’t want but I know have archival value. Without the benefit of patrons, and their potential needs, I am arguing for the future needs of the very people I am arguing with! Without a higher level of administration to appeal to, I am often left with no recourse but to do things that are not in the best interests of the collection. This leads to the unfortunate loss of materials but may not be as bad as it first appears. When considering how traditional archival collections are created and deposited, it is well within reason that these items would never have made it into the collection. I like to think that by being embedded in the creation process, I am able to save far more than would otherwise be deposited if the creators were left to make appraisal decisions on their own.


Michelle Ganz is the Archives Director at McDonough Innovation and the former archivist at Lincoln Memorial University. She received her MILS from the University of Arizona and a BA from the Ohio State University. In addition to her passion for all things archival Michelle loves to cook, read, and watch movies.

Modernization of the A.D. Hopkins collection at the Smithsonian Institution Department of Entomology

by Teresa Boyd

This is the second post in the BloggERS Embedded Series.

The Smithsonian Institution’s Department of Entomology has recently finished phase one of their multiyear project to digitize their portion of the A.D. Hopkins notes and records system, which includes about 100 years of observations, both in the field and in the lab. A.D. Hopkins developed the system in order to collect biological and natural history notes about species, the environment they were in, as well as the collectors and locations of collection. This collection was adopted by the United States Department of Agriculture (USDA) when Hopkins was named Chief of Forest Insect Investigations, though Hopkins is known to have developed and used the system while working at West Virginia University in the late 1800s. The Smithsonian Institution’s Department of Entomology has historically worked very closely with the USDA and therefore obtained the largest portion of the Hopkins card file over the years.

 

It was important to Hopkins to collect as much information as possible about specimens because he felt it was the quickest way to understand the situation of potential pests and to find solutions to harmfully invasive species. Part of Hopkins’ methodology was to encourage average citizens to send in specimens and observations to the USDA, the Smithsonian, or one of the forest experiment stations that were located throughout the United States, which were then incorporated into the Hopkins note system. Some of these notes are also documentation about lab research such as specimen rearing, specimen transfers, and communications between lab and field. A few of these notes are also cross-referenced, so often a lab note can be traced back to a field note, making it easier for researchers to quickly see the correlation between field and lab (work that was often done by different individuals.) The numbers on each individual card within the A.D. Hopkins system correlates to specimens that are housed in various locations. Traditionally a researcher or scientist would ask for the notes that were associated with a a specimen number. By creating an online repository of the notes, the Smithsonian hopes to further enrich researchers with new tools to expand their work and perhaps find new ways to use the data which has been collected by past researchers and scientists.

I have been working on this project as a lone archivist for the past 5 years, scanning the card file portion of the collection, and am now working on preparing these scans for a website that will be built specifically for this type of collection. The Smithsonian Institution’s Department of Entomology hopes to begin sending the scans of the cards to the Smithsonian Transcription Center soon to crowdsource the transcriptions. This cuts down on the time it takes to transcribe the older material which is all handwritten. I will be adding the transcribed notes to the digitized card on the website so that researchers will be able to go to the website, look up a specific card, and see both the original scan and the transcribed notes, making it easy for anyone to be able to use the information contained in the Hopkins collection. Additionally these scans will be incorporated into the Department of Entomology’s collections database by matching specimens to their unique card numbers;  thereby giving researchers the complete picture.

The Smithsonian Institution’s work to digitize and make their A.D. Hopkins collection publicly available is not the first of its kind; the USDA had previously accomplished this in the 1980s, and has made their documents available on the USDA website, HUSSI. There is hope that in the future other institutions that have their own portions of the A.D. Hopkins notes and records system will also begin to digitize and make them available online, supplementing the Smithsonian and USDA efforts to make this invaluable data available to researchers. 


Teresa Boyd is an archivist for the Department of State and a volunteer archivist for the Smithsonian Institute’s Department of Entomology. She holds a degree in Library and Information Science from the University of Arizona.

Capturing Common Ground

by Leslie Matthaei

This is the first post in the BloggERS Embedded Series.

Every Tuesday I am asked the same question: “T Coast today?” T Coast, or Tortilla Coast, is the preferred lunch location for some of the photographers that occupy the Photography Branch, in the Curator Division, of the federal agency Architect of the Capitol. The agency oversees the maintenance of building and landscapes on Capitol Hill to include the Library of Congress buildings, Supreme Court, United States Botanic Gardens, House and Senate Office Buildings, Capitol Power Plant, and, of course, the United States Capitol. I have joined the professional photographers at T Coast for more than a dozen lunches now. I am here for the Taco Salad and comradery but mostly I am here to listen. And to ask questions. I am an embedded archivist.

461700
The Library of Congress Thomas Jefferson Building
467882
United States Capital Building
469327
United States Botanic Garden – Bartholdi Park

I use my time at the T Coast lunch table to get to know the photographers and for them to get to know me. I discovered very quickly that the photographers and I have a lot in common. For example, the photographers are often assigned to shoot a long-term project (Collection) which may have multiple phases (Series), and for each phase, they go out on specific days to shoot (File Units). They cull excess and/or duplicate photographs. And they generally have a tried workflow for ingesting their born-digital objects to edit in Adobe Lightroom then upload them to an in-house Digital Asset Management system known as PhotoLightbox. Within PhotoLightbox, they are responsible for defining the security status of an individual image or group of images and providing the descriptive metadata. Tapping into parallel duties has allowed me to bridge potential knowledge gaps in explaining what roles and functions I can provide the branch as a whole.

One rather large knowledge gap is descriptive metadata. To be sure, the photographers in our agency are incredibly busy and in high demand. And they are professionally trained photographers. They see the world through aesthetics. It is not necessarily their job to use PhotoLightbox to help a researcher find images of the East Front extension that occurred in the 1950s, for example. That is my role, and when I query PhotoLightbox, the East Front extension project is represented in multiple ways: EFX, East Front (Plaza) Extension, East Extension, Capitol East Front Extension. You may see where this is going: there is no controlled vocabulary. When, in a staff meeting, I pitched the idea of utilizing controlled vocabularies, they immediately understood the need. Following their lead, the conversation turned to having me develop a data entry template for each of their shoots.

Matthaei_DataEntryTemplate
An example of the data entry template.

I admit now that my first spin through PhotoLightbox revealed a pressing need for controlled vocabularies, among other concerns the database presented. I am the type of person that when I see a problem, I want to fix it immediately. Yet I knew that if my first professional introduction to the photographers was a critique of how unworkable their data entry was and had been over time, I might turn them off immediately. Instead, I went to lunch. I credit the results produced by this particular staff meeting to the time that I put in getting to know the photographers, getting to understand each of their respective workflows, and understanding a little bit about the historic function and purpose of the office within the agency.

I have another half dozen lunches to go before I begin to talk to the photographers about the need for digital preservation of born-digital images over the long-term and both of our roles in the surrounding concepts and responsibilities. I have a few more lunches after that to get their assistance in codifying the decision we are making together into branch policies. I feel confident, however, that I have their complete buy-in for the work that I have been tasked to do in the branch. Instead of seeing me as another staff member making them do something they do not want, I am seen as someone who can help them gain control of and manage their assets in a way that has yet to be done in the branch. I cannot do it alone, I need their help. And some chips and salsa every once in a while.


Matthaei
Leslie Matthaei

Leslie Matthaei is an Archivist in the Photography Branch, Curator Division, Architect of the Capitol. She holds an MLIS from the University of Arizona, and an MA and BA in Media Arts from the University of Arizona.