The Mouse Action Recognition System (MARS): pose annotation data

Description

The study of naturalistic social behavior requires quantification of animals' interactions. This is generally done through manual annotation—a highly time consuming and tedious process. Recent advances in computer vision enable tracking the pose (posture) of freely-behaving animals. However, automatically and accurately classifying complex social behaviors remains technically challenging. We recently introduced the Mouse Action Recognition System (MARS), an automated pipeline for pose estimation and behavior quantification in pairs of freely interacting mice (Segalin et al, 2020). This Dataset includes the training, test, and validation sets used to train MARS's supervised classifiers for three social behaviors of interest: close investigation, mounting, and attack. Included in the dataset are 15000 pairs of top- and front-view frames from videos of pairs of interacting mice. Each frame has been manually annotated by five individuals for a total of nine (top-view) or thirteen (front-view) keypoints: the nose, ears, base of neck, hips, base of tail, middle of tail, and end of tail, and additionally the four paws (front-view only.)

Files

Methods

Data Acquisition Experimental mice ("residents") were transported in their homecage (with cagemates removed) to a behavioral testing room, and acclimatized for 5-15 minutes. Homecages were then inserted into a custom-built hardware setup (Hong et al, 2015) with infrared video captured at 30 fps from top- and front-view cameras (Point Grey Grasshopper3) recorded at 1024x570 (top) and 1280x500 (front) pixel resolution using StreamPix video software (NorPix). Following two further minutes of acclimatization, an unfamiliar group-housed male or female BALB/c mouse ("intruder") was introduced to the cage, and animals were allowed to freely interact for a period of approximately 10 minutes. BALB/c mice are used as intruders for their white coat color (simplifying identity tracking), as well as their relatively submissive behavior, which reduces the likelihood of intruder-initiated aggression. In some videos, mice are implanted with a cranial cannula, or with a head-mounted miniaturized microscope (nVista, Inscopix) or optical fiber for optogenetics or fiber photometry, attached to a cable of varying color and thickness. Surgical procedures for these implantations can be found in (Karigo et al, 2020). To create a data set of video frames for labeling, we sampled 64 videos from several years of experimental projects. We extracted a set of 15,000 individual frames each from the top- and front-view cameras, giving a total of 2,700,000 individual keypoint annotations (15,000 frames x (7 top-view + 11 front-view keypoints per mouse) x 2 mice x 5 annotators). 5,000 of the extracted frames included resident mice with a fiberoptic cable, cannula, or head-mounted microendoscope with cable. Pose Annotation We defined nine anatomical keypoints in the top-view video (the nose, ears, base of neck, hips, and tail base, midpoint, and endpoint), and 13 keypoints in the front-view video (top-view keypoints plus the four paws). The tail mid- and endpoint annotations were subsequently discarded for training of MARS, however these annotations are still included in the "raw_keypoints" files in this Dataset. We used the crowdsourcing platform Amazon Mechanical Turk (AMT) to obtain manual annotations of pose keypoints on a set of video frames. AMT workers were provided with written instructions and illustrated examples of each keypoint, and instructed to infer the location of occluded keypoints. To compensate for annotation noise, each keypoint was annotated by five AMT workers, and a "ground truth" location for that keypoint was defined as the median across annotators. The median was computed separately in the x and y dimensions. Annotations of individual workers were also post-processed to correct for common mistakes, such as confusing the left and right sides of the animals. Another common worker error was to mistake the top of the head-mounted microendoscope for the resident animal's nose; we visually screened for these errors and corrected them manually.

Other

This dataset contains images and pose annotations ("keypoints") for top- and front-view cameras on 15,000 pairs of movie frames. Keypoints are provided for each camera, in two formats: MARS_raw_keypoints_(top/front).manifest contains annotations from each worker for each keypoint/image, in the raw ".manifest" format expected by MARS_Developer. It still includes tail mid- and end-point annotations, and has not been corrected for common annotator mistakes such as left/right flipping of body parts. For each frame, it contains: the corresponding image filename annotatedResult-metadata, some metadata about the GroundTruth labeling job (may be ignored). annotatedResult, a dictionary containing annotations from each worker who labeled the image. Each worker's annotations are encoded in a string json in ['annotationsFromAllWorkers]['content'] MARS_keypoints_(top/front).json contains the processed annotations used to train MARS: for each body part in each image, we take the median of keypoint coordinates in the x and y dimensions, after correcting for annotator errors. Keypoint locations are provided in pixels rounded to the nearest tenth of a pixel. The width and height of each image are also provided, so that keypoints can be scaled to fractional values if needed.

Other

Related Publication: The Mouse Action Recognition System (MARS): a software pipeline for automated analysis of social behaviors in mice Segalin, Cristina Caltech Williams, Jalani Caltech Karigo, Tomomi Caltech Hui, May Caltech Zelikowsky, Moriel University of Utah Sun, Jennifer J. Caltech Perona, Pietro Caltech Anderson, David J. Caltech Kennedy, Ann Northwestern bioRxiv 2020-07-27 https://doi.org/10.1101/2020.07.26.222299 eng

Additional details