Interactive Platform for Annotating and Classifying Mouse Behavior
Shivani Ramesh
Mentor: Dr. Carlos Campos, Diabetes Institute, University of Washington Department of Medicine.
Date/Time: August 22nd, 2025 at 11:00 AM.
Abstract: Animal behavior analysis is foundational to neuroscience research, providing translational measures of learning and memory, affect and mood, and neuropsychiatric disease. While analysis of behavior is ubiquitous, existing methods require manual annotation and are often time-consuming and prone to bias. Here, we are developing an interactive web platform as an efficient, generalizable, and user-friendly answer to these challenges. The platform integrates a Next.js-based React frontend with a FastAPI backend developed with Python to provide a complete, end-to-end workflow for behavioral analysis.
The behavioral classification pipeline begins with raw video input, uploaded through the user interface, which is processed to isolate cropped images of the mouse. Each frame is passed through a mesh inference model that projects 3D vertices onto the 2D image, generating a 2D vertex array representation for each detected object. A posture inference module transforms these vertex arrays into compact posture embeddings that encode semantic information about the animal’s pose. Consecutive posture embeddings are then processed by a sequence model to produce temporally contextualized sequence embeddings, capturing the dynamics of behavior over time. The user, through the interface’s behavior coding platform, will annotate the uploaded video and assign class labels. They can then choose how they want to sample the annotations into a training and validation set for further analysis. In the supervised setting, these sequence embeddings are paired with user-annotated behavior start and end points to train classification models capable of predicting behavior labels at scale. The app is currently equipped with two distinct classification models: a K-Nearest Neighbors (KNN) classifier and a more advanced Neural Network (NN) with LSTM layers. These classifiers process sequence embeddings to predict behavior labels, providing a robust tool for automated analysis. Following initial classifier training, the platform generates comprehensive reports that include key metrics such as accuracy, precision, and recall. To enhance interpretability and provide transparent feedback, the platform visualizes classifier outputs through predicted video snippets, confidence labels, and timeline visualizations. This allows users to review model performance, identify misclassified frames, and refine their annotations for iterative improvement. This foundational prototype not only automates the analysis of mouse behavior but also empowers researchers to validate and interpret their findings through an intuitive and interactive interface, laying the groundwork for a future production-ready application.
Future work will focus on expanding the platform’s capabilities to create a fully integrated and deployed behavioral analysis ecosystem. This includes linking the interface directly to fine tuned tracking and posture extraction models, enabling seamless end-to-end processing from raw video to behavior classification without requiring external preprocessing steps. Support for multi-animal tracking and interaction modeling will allow researchers to study complex social behaviors. Enhancements to temporal modeling architectures could improve the sensitivity and generalizability of behavior classification across experimental conditions. Optimizing model efficiency and scalability will enable deployment on larger datasets and real-time annotation support, further bridging the gap between manual observation and fully automated, high-throughput behavioral analysis.