Code from "Measuring data rot: an analysis of the continued availability of shared data from a single university"

Description

R scripts and files from the article "Measuring Data Rot: An Analysis of the Continued Availability of Shared Data from a Single University" by Kristin Briney.

This research looked at supplemental data links from publications in CaltechAUTHORS and tested them for their availability on the web using web scraping and hand testing in the Chrome browser.

Project file:

• SupplementaryData.Rproj: R project covering all of the R scripts.

Data input:

• supp-data.csv: raw data from CaltechAUTHORS, downloaded in May 2023.

Scripts to be done in order:

• 1-DataParsing.R: Cleans up the input data for processing and calculates some summary statistics.
• 2-DOIScraping.R: Scrapes and processes DOI prefixes.
• 3-DOIAnalysis.R: Adds DOI prefix information back into larger dataset.
• 4-LinkDecay.R: Scrapes URLs and DOIs to see if the data is still available and outputs data.
• 5-dataRot.R: Measures error rate (based on sampling), creates figures, and fits Poisson regression to data loss over time.

Data outputs:

• 4-linkDecay.csv: Full data set of supplemental data links including outcomes of web scraping.
• 4-linksToCheck.csv: Information on all supplemental data links that failed to scrape or could not be web scraped, basically everything that needed to be checked by hand.
• 5-resolves.csv: Final dataset denoting if links resolve, which was used for generating figures and fits.
• 5-sampling.csv: Results of comparing a sample from CaltechAUTHORS versus journal articles to check for errors in recording supplemental data links in CaltechAUTHORS.

DOI files:

• doi-data.zip: contains information harvested from DataCite and CrossRef looking up various DOI prefixes.
• 2-DOI_APIs.txt: not used for computation but contains information on DataCite and CrossRef's API's.

Other files:

• Other files are labelled 1-4 at the beginning of the file name to note which part of the analysis workflow they were created during. Most capture all or parts of the data at mid-points in the analysis.
• Files labelled with 5 at the beginning of the file name are used for final calculations and figures.

Files

Additional details