Published June 7, 2021 | Version 1.0
Dataset Open

Caltech Mouse Social Interactions (CalMS21) Dataset

  • 1. ROR icon California Institute of Technology
  • 2. ROR icon Northwestern University


Multi-agent behavior modeling aims to understand the interactions that occur between agents. We present a multi-agent dataset from behavioral neuroscience, the Caltech Mouse Social Interactions (CalMS21) Dataset. Our dataset consists of trajectory data of social interactions, recorded from videos of freely behaving mice in a standard resident-intruder assay. To help accelerate behavioral studies, the CalMS21 dataset provides benchmarks to evaluate the performance of automated behavior classification methods in three settings: (1) for training on large behavioral datasets all annotated by a single annotator, (2) for style transfer to learn inter-annotator differences in behavior definitions, and (3) for learning of new behaviors of interest given limited training data. The dataset consists of 6 million frames of unlabeled tracked poses of interacting mice, as well as over 1 million frames with tracked poses and corresponding frame-level behavior annotations. The challenge of our dataset is to be able to classify behaviors accurately using both labeled and unlabeled tracking data, as well as being able to generalize to new settings.

Files (109.8 GB)
Name Size
28.3 GB Preview Download
74.2 GB Preview Download
912.1 MB Preview Download
4.4 kB Download
10.9 kB Preview Download
556.2 MB Preview Download
9.2 kB Preview Download
1.8 GB Preview Download
3.5 GB Preview Download
457.4 MB Preview Download


Our trajectory dataset is available in json format. Please download and unzip the zip files for each task to access the data. Our video datasets are available as mp4 format ( or seq format ( Additionally, we release the pre-computed MARS features on our dataset in The readme included for download with the dataset provides a more detailed description of the data format, and a script to read json files and convert them to npy files (Python) is included. See our website for more info: See our paper for more details on the dataset and benchmarks:


Please cite the following paper if you find the dataset useful: BibTeX: @article{calms21, title={The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions}, author={Sun, Jennifer J and Karigo, Tomomi and Chakraborty, Dipam and Mohanty, Sharada P and Wild, Benjamin and Sun, Quan and Chen, Chen and Anderson, David J and Perona, Pietro and Yue, Yisong and Kennedy, Ann}, journal={arXiv preprint arXiv:2104.02710}, year={2021} }

Additional details

September 13, 2022
November 28, 2022