We thank Talmo Pereira, a Salk Fellow at the Salk Institute, for taking part in this interview and sharing his story and several insights about developing AI-based methods for studying complex biological systems. His recent work focuses on quantifying behaviors through markerless motion capture using the SLEAP tool. It finds applications in neuroscience, cancer research, and plant biology. SLEAP tracks the movements of animals, plants, and humans, aiding disease prediction and climate change mitigation. Pereira envisions using AI to diagnose diseases early by analyzing behavioral patterns in videos.
Research Background, Contributions to Quantitative Phenotyping, and Challenges in Understanding Behavior
Where did you work before joining the Salk Institute?
Prior to joining Salk, I was doing my Ph.D. in Neuroscience at Princeton University with Mala Murthy and Josh Shaevitz and worked at Google AI as a Ph.D. Research Intern. My focus was on developing computer vision methods to capture motion and quantify behavior from videos.
How has your recent work contributed to the field of quantitative phenotyping of complex behaviors?
My recent work focused on the problem of markerless motion capture, also known as pose tracking. Just like how motion capture suits are used in Hollywood to track the movements of the body parts of actors to create CGI sequences, the idea with markerless motion capture is to do the same thing — but without the suit!
This is a complex task for computers to do but has become much more tractable with the advent of deep neural networks, the backbone of modern AI. One of the reasons that these networks work so well is that they can learn from massive amounts of data, in this case, millions of images of people in a variety of settings that were manually annotated via crowd workers that clicked on the “ground truth” location of each body part.
Once we have a neural network trained to track body parts across video sequences, we can use the patterns of how they move to infer quantitative phenotypes of complex behaviors, for example, to detect when an individual is running versus scratching their head — and a statistical description of exactly how they move their body parts to do so.
What is the main challenge in understanding animal and human behavior in neuroscience?
There are many challenges to understanding behavior, but a key one that these AI approaches solve is how we can quantitatively measure it. By capturing the movements that underlie behavior, we can devise ways to automatically detect occurrences of known behaviors such as grooming or jumping. We can then correlate the timing or frequency of known behaviors with experimental variables, such as whether the individual has a disease or not, in order to establish a link between biology and observable behavior. In animals, this is more readily verifiable because we can control the environmental conditions that the animal is in, record them for longer, and do further experiments to measure the internal states of their brain and body to validate their associations with the behavior patterns.
A harder problem is to define what behavior is in the first place. For example, where do we draw the line between a “fast walk” and a “slow jog” if both allow you to move forward at the same velocity? (If you’re interested, Calhoun et al explore even harder examples, such as, ‘A person sweats in response to hot air. Is this a behavior?’) Some methods, such as Keypoint-MoSeq (Weinreb et al, 2023), have emerged that allow you to leverage pose tracking to define behaviors purely based on statistical patterns in the data. These have the promise of being more objective but still lack the ability to incorporate knowledge about clear and known distinctions about what is a behavior.
In either case, by having the pose tracking as a starting and unambiguous representation of movements underlying the behaviors, we are at a much better starting point than trying to identify “behaviors” from the pixel data alone since it gives us direct information about how body parts are moving, which ultimately is how the brain produces (many forms of) behavior.
SLEAP: Leveraging Artificial Intelligence for Innovative Motion Tracking Technology
How does SLEAP use artificial intelligence?
SLEAP is an open-source multi-animal pose tracking software that uses deep neural networks, a core algorithm in modern AI, to learn how to track poses automatically in images. It does this by learning from user examples of where each body part should be located in a training set of images.
One of the key innovations in SLEAP is the ability to train these neural network models with very small training sets, on the order of tens to hundreds of images, demonstrating that it’s possible to use this form of AI without requiring the resources of an industry lab to generate the training set. This makes it possible to practically use markerless motion capture across a diversity of datasets that we see across animal behavior studies.
What is the main innovation of the SLEAP tool?
SLEAP is a framework rather than a single model or method. This is motivated by the observation that there are many complex datasets with unique properties. Rather than trying to find one algorithm that works for everything, SLEAP implements a variety that each has its trade-offs and suitability to each use case. For example, if you have several subjects visible in the frame, you may need to use an algorithm that can reason about how detected body parts should be associated with each other to form distinct poses, whereas for data with single subjects, it suffices to just detect the location of each body part and you’ll know that they belong to the individual.
We put a lot of work into making SLEAP user-friendly and accessible, both by making it easy to set up and to use with no technical background. As a result, we’ve had all sorts of folks use SLEAP, from high schoolers that use it for their research and science fair projects, to professional biological research scientists without a background in AI.
To make this possible, we combine industry-grade software engineering practices with a cutting of general open-source and scientific computing software such as TensorFlow, attrs, and PySide2.
Can you provide examples of how the SLEAP tool is used in animal behavior studies?
SLEAP has been used in numerous animal studies to extract quantitative metrics of behaviors of interest.
- In Mills et al. (2023), SLEAP was used to characterize mouse social behavior to study how neural circuits determine whether sensory stimuli should be interpreted as “good” or “bad”.
- In Mabuchi et al. (2023), SLEAP was used to quantify courtship behavior to understand how fruit flies use visual information to direct their movements in a social context.
- In Legan et al. (2022), SLEAP was used to study how paper wasps shift into a more “vigilant” state following aggressive encounters.
- In Leonardis et al. (2022), SLEAP was used to measure how rats interacted with a “robot rat” in a study into the basis of social interactions.
- In Gutierrez-Castellanos et al. (2023), SLEAP was used to quantify “rejection” behaviors in female mice in the context of a study on the neural circuits that are influenced by reproductive cycle hormones.
These are just some recent examples to highlight the diversity of application areas, scientific questions, and species that SLEAP has been used with.
How does the SLEAP tool contribute to disease prediction and climate mitigation efforts?
Since joining Salk, one of the areas that my lab has focused on is how to translate motion capture data into insights about biology.
For disease prediction, we’ve been working with several collaborators to develop analytics pipelines that can be used to automatically extract phenotypes that may be predictive of the onset and progression of disease. For example, in a pilot study that measured the behavioral changes associated with amyotrophic lateral sclerosis (ALS), a neurodegenerative disease that leads to loss of motor coordination and eventually death, we find that we can detect clear quantitative markers of the disease in untrained mice living in their standard home cage environment in a fully automated fashion. We are now working to build a complete characterization of all behaviors associated with different stages of this and other diseases, including pancreatic cancer and Alzheimer’s Disease.
Another application area that we’ve been focusing on that was made possible by Salk’s unique interdisciplinary environment is plant biology for climate change. Salk is the home of the Harnessing Plants Initiative, a multi-group effort to curb climate change by developing crop plants that have deeper and more massive roots, resulting in enhanced carbon sequestration as carbon dioxide is converted into plant matter in the soil. While there is a lot known about the biology of how plant roots develop, there was a need for more automated and reliable methods for quantitatively characterizing the root system architecture. We applied SLEAP to these data and are now working to develop tools for extracting meaningful traits such as the branching angles and root depths that can be used to characterize plants with enhanced carbon sequestration capabilities.
Looking ahead…
How do you envision the future use of AI in diagnosing cancer and ALS in humans?
As we develop a better understanding of how subtle or statistically-defined changes in your body language relate to disease, we hope to apply the same type of AI we use in SLEAP to detect diseases such as cancer and ALS in humans. This will require significant clinical validation before it can be used as a reliable diagnostic tool, but given how easy and cheap it is to record videos, we hope that one day we will be able to extract the behavioral signatures of disease from videos you can record on your smartphone. If done correctly, this could drastically reduce the burden on our healthcare system and allow for much earlier screening and prevention of diseases that are easier to treat at earlier stages.
