Preserve This Podcast!

by Molly Schwartz

Mary Kidd (MLIS ’14) and Dana Gerber-Margie (MLS ’13) first met at a Radio Preservation Task Force meeting in 2016. They bonded over experiences of conference fatigue, but quickly moved onto topics near and dear to both of their hearts: podcasts and audio archiving. Dana Gerber-Margie has been a long-time podcast super-listener. She is subscribed to over 1400 podcasts, and she regularly listens to 40-50 of them. She launched a podcast recommendation newsletter when she was getting her MLS, called “The Audio Signal,” which has grown into a popular podcast publication called The Bello Collective. Mary was a National Digital Stewardship Resident at WNYC, where she was creating a born-digital preservation strategy for their archives. She had worked on analog archives projects in the past — scanning and transferring collections of tapes — but she’s embraced the madness and importance of preserving born-digital audio. Mary and Dana stayed in touch and continued to brainstorm ideas, which blossomed into a workshop about podcast preservation that they taught at the Personal Digital Archives conference at Stanford in 2017, along with Anne Wootton (co-founder of Popup Archive, now at Apple Podcasts).

Then Mary and I connected at the National Digital Stewardship Residency symposium in Washington, DC in 2017. I got my MLS back in 2013, but since then I’ve been working more at the intersection of media, storytelling, and archives. I had started a podcast and was really interested, for selfish reasons, in learning the most up-to-date best practices for born-digital audio preservation. I marched straight up to Mary and said something like, “hey, let’s work together on an audio preservation project.” Mary set up a three-way Skype call with Dana on the line, and pretty soon we were talking about podcasts. How we love them. How they are at risk because most podcasters host their files on commercial third-party platforms. And how we would love to do a massive outreach and education program where we teach podcasters that their digital files are at risk and give them techniques for preserving them. We wrote these ideas into a grant proposal, with a few numbers and a budget attached, and the Andrew W. Mellon Foundation gave us $142,000 to make it happen. We started working on this grant project, called “Preserve This Podcast,” back in February 2018. We’ve been able to hire people who are just as excited about the idea to help us make it happen. Like Sarah Nguyen, a current MLIS student at the University of Washington and our amazing Project Coordinator.

Behaviors chart from the Preserve This Podcast! survey.

One moral of this story is that digital archives conferences really can bring people together and inspire them to advance the field. The other moral of the story is that, after months of consulting audio preservation experts and interviewing podcasters and getting 556 podcasters to take a survey and reading about the history of podcasting, we can confirm that podcasts are disappearing and podcast producers are not adequately equipped to preserve their work against the onslaught of forces working against the long-term endurance of digital information rendering devices. There is more information on our website about the project (preservethispodcast.org) and in the report about the survey findings. Please reach out to mschwartz@metro.org or snguyen@metro.org if you have any thoughts or ideas.

---

Molly Schwartz is the Studio Manager at the Metropolitan New York Library Council (METRO). She is the host and producer of two podcasts about libraries and archives — Library Bytegeist and Preserve This Podcast. Molly did a Fulbright grant at the Aalto University Media Lab in Helsinki, was part of the inaugural cohort of National Digital Stewardship Residents in Washington, D.C., and worked at the U.S. State Department as a data analyst. She holds an MLS with a specialization in Archives, Records and Information Management from the University of Maryland at College Park and a BA/MA in History from the Johns Hopkins University.

Advertisements

Using Python, FFMPEG, and the ArchivesSpace API to Create a Lightweight Clip Library

by Bonnie Gordon

This is the twelfth post in the bloggERS Script It! Series.

Context

Over the past couple of years at the Rockefeller Archive Center, we’ve digitized a substantial portion of our audiovisual collection. Our colleagues in our Research and Education department wanted to create a clip library using this digitized content, so that they could easily find clips to use in presentations and on social media. Since the scale would be somewhat small and we wanted to spin up a solution quickly, we decided to store A/V clips in a folder with an accompanying spreadsheet containing metadata.

All of our (processed) A/V materials are described at the item level in ArchivesSpace. Since this description existed already, we wanted a way to get information into the spreadsheet without a lot of copying-and-pasting or rekeying. Fortunately, the access copies of our digitized A/V have ArchivesSpace refIDs as their filenames, so we’re able to easily link each .mp4 file to its description via the ArchivesSpace API. To do so, I wrote a Python script that uses the ArchivesSpace API to gather descriptive metadata and output it to a spreadsheet, and also uses the command line tool ffmpeg to automate clip creation.

The script asks for user input on the command line. This is how it works:

Step 1: Log into ArchivesSpace

First, the script asks the user for their ArchivesSpace username and password. (The script requires a config file with the IP address of the ArchivesSpace instance.) It then starts an ArchivesSpace session using methods from ArchivesSnake, an open-source Python library for working with the ArchivesSpace API.

Step 2: Get refID and number to start appending to file

The script then starts a while loop, and asks if the user would like to input a new refID. If the user types back “yes” or “y,” the script then asks for the the ArchivesSpace refID, followed by the number to start appending to the end of each clip. This is because the filename for each clip is the original refID, followed by an underscore, followed by a number, and to allow for more clips to be made from the same original file when the script is run again later.

Step 3: Get clip length and create clip

The script then calculates the duration of the original file, in order to determine whether to ask the user to input the number of hours for the start time of the clip, or to skip that prompt. The user is then asked for the number of minutes and seconds of the start time of the clip, then the number of minutes and seconds for the duration of the clip. Then the clip is created. In order to calculate the duration of the original file and create the clip, I used the os Python module to run ffmpeg commands. Ffmpeg is a powerful command line tool for manipulating A/V files; I find ffmprovisr to be an extremely helpful resource.

Clip from Rockefeller Family at Pocantico – Part I , circa 1920, FA1303, Rockefeller Family Home Movies. Rockefeller Archive Center.

Step 4: Get information about clip from ArchivesSpace

Now that the clip is made, the script uses the ArchivesSnake library again and the find_by_id endpoint of the ArchivesSpace API to get descriptive metadata. This includes the original item’s title, date, identifier, and scope and contents note, and the collection title and identifier.

Step 5: Format data and write to csv

The script then takes the data it’s gathered, formats it as needed—such as by removing line breaks in notes from ArchivesSpace, or formatting duration length—and writes it to the csv file.

Step 6: Decide how to continue

The loop starts again, and the user is asked “New refID? y/n/q.” If the user inputs “n” or “no,” the script skips asking for a refID and goes straight to asking for information about how to create the clip. If the user inputs “q” or “quit,” the script ends.

The script is available on GitHub. Issues and pull requests welcome!

---

Bonnie Gordon is a Digital Archivist at the Rockefeller Archive Center, where she focuses on digital preservation, born digital records, and training around technology.

Introduction to a Find Loop in Bash

by Andrew Weaver

This is the fourth post in the bloggERS Script It! Series.

---

As the Digital Infrastructure and Preservation Librarian at Washington State University Libraries, I am responsible for helping to ensure the integrity and provenance of all digital files maintained in our archives. Recently a lot of my work has involved processing legacy materials in order to generate or expand upon their associated metadata. As much of this work is specific to certain file formats, I often find myself trying to hunt down all examples of a certain file type spread across thousands of directories representing many terabytes worth of data. To do this I have been using the Bash find command. This allows me to generate lists of file paths which I can then loop through to perform actions as needed, such as file validation and metadata extraction. As I imagine that I am not the only one with vast amounts of legacy files to process, I wanted to write up a quick explanation of this method and hopefully save others some time!

For this example, I will be targeting .wav files, but this process would work on any type of file. The main caveat to this method is that it involves locating files solely by file extension; this means that any files that have non-conforming extensions will not be discovered, and files that have been erroneously changed to the target extension will show up as false positives. If a higher level of certainty is desired for target files, an intermediate step could be to first run a loop verifying all results through a tool such as Seigfried or MediaConch.

Example of Find

A basic example of the find command is:

find "My-Folder" -iname "*.wav"

This command can be broken down as follows:

• • find initiates the command.
  • My-Folder is the file path, either “absolute” or “relative,” to the target directory that will be searched. The “absolute” path is the entire path to your directory from the base of your system and can be quickly found by dragging your directory into your terminal. An example of what this might look like is /home/username/Desktop/myfolder. The “relative” path is a path from where you already are, so for example, if your terminal was open to your Desktop, the relative path to the directory in the preceding example would just be myfolder. The search is recursive, meaning that any sub-folders will also be searched. If . is used instead of a specific directory, then the current directory will be searched. Note that if there are any spaces in your file path, you will have to put it in quotes.
  • -iname tells the find command that you will be searching for case insensitive terms in the filenames. This helps account for people mixing case, such as .tif and .TIF, over the years.
  • "*.wav" gives the find command the term that you will be searching for. In this instance, the * is a wildcard, so the results will include any files that end in .wav.

Running this command recursively finds all WAV files (or again, more accurately, all files that have the .wav extension) in the target directory and will spit out their file paths into your terminal. By itself this can be informative, but it really becomes useful when paired with a loop to process those results.

Loop Example

In this example I will show how to use the results of the previous command to batch perform the sample preservation action of embedding MD5 checksums into the discovered WAV files using BWF Metaedit (Download available here).

The entire command is:

find "/Path/To/My-Folder" -iname "*.wav" | while read targetfile ; do bwfmetaedit --MD5-Embed --reject-overwrite "$targetfile"; done

which can be broken down as follows:

find "/Path/To/My-Folder" -iname "*.wav" is the find command demonstrated previously. This provides the file paths that will be used by the following commands.

| This is called a ‘pipe’ and is what Bash uses to send the output of one command to another. In this case, we are sending the output of the find command into a loop. (This is the vertical bar symbol on the keyboard, not a capital I).

while read targetfile ; do This is what starts the loop. What this does is “read” each chunk of information that is coming from the pipe, in this case a file path to each matching file, and then assign each one to the variable “targetfile.” The name “targetfile” is an arbitrary name I chose for this example; it could be replaced with anything, such as “x” or “myfile,” as long as you are consistent with using the same variable name in the following steps.

bwfmetaedit --MD5-Embed --reject-overwrite "$targetfile" This is the actual preservation action that will be run on every file discovered by the find command. The first part bwfmetaedit --MD5-Embed is the BWF Metaedit command that embeds a checksum into each target WAV file. --reject-overwrite is a safeguard that tells BWF Metaedit not to overwrite any existing information that might already be in the file. The second part "$targetfile" makes the command run on the variable that was created in the previous step, with the result that BWF Metaedit will run on all the WAV files the find command located. It is important not to forget the quotation marks around this variable; otherwise it won’t work on any filenames that have spaces in them. Since this variable represents file paths in the loop, it can be used in combination with any command you would want to run on your files!

; done ends the loop once it is done reading the output of the find command.

Conclusion

Since all that needs to be done to use this technique in different contexts is to change the file extension in the initial find command and then change the preservation action that targets the variable in the loop, I find that I am using similar constructions on an almost daily basis! By using find loops I am able to save a lot of time (and pain) when dealing with legacy data. I hope this example will help others to do the same!

---

Andrew Weaver is the Digital Infrastructure and Preservation Librarian at Washington State University. He is from Seattle, and graduated with an MLIS from the University of Washington in 2015. Previously he has worked in the UW Libraries and at the CUNY TV Archive as part of the NDSR program.