The Red Hen Lab is an international, interdisciplinary research network investigating multimodal communication: how language, gestures, images, sound, and other forms of communication interact. For this, Red Hen collects and processes enormous amounts of data, ranging from television broadcasts, radio recordings and subtitles to historical texts, ancient writing systems, and even representations of gestures in Greek vase paintings.
A major part of the project is developing the tools needed to work with these datasets, including automated data acquisition, distributed storage, multimodal analysis, machine learning, and search and visualization tools. Much of this software is developed as open source.
In 2020, I worked as a student research assistant on one of Red Hen’s video-processing projects at the University of Osnabrück, through Peter Uhrig, who was interim professor of Computational Linguistics and Cognitive Science at the time. My work focused on developing and maintaining parts of a pipeline for automatic gesture analysis in large video corpora, particularly the identification and tracking of people throughout videos.
Gesture Recognition in the Ellen DeGeneres Show
One of the datasets used by Red Hen consists of 30 episodes of the Ellen DeGeneres Show, featuring interviews with various celebrities. The interesting part for the researchers is the relationship between spoken language and hand gestures: People frequently accompany their speech with gestures, which can carry additional information, emphasize certain points, or otherwise relate to what is being said. [1]https://sites.google.com/case.edu/techne-public-site/datasets/gsoc21-ellen-gestures
The videos were manually annotated by gesture researcher Yao Tong using ELAN, a tool for annotating audio and video recordings. These annotations provide a basis for developing and evaluating methods to automatically identify gestures and relate them to speech.
Automating this analysis requires quite a few processing steps. First, OpenPose is used to estimate the poses of people in the videos, detecting body, hand, and facial keypoints. However, simply detecting a skeleton in each frame is not enough. People move around, disappear behind others, and are shown from different camera angles. Especially in a talk show, where the camera frequently switches between speakers, the pipeline needs to figure out which detections belong to the same person, even across different shots.
My Contribution
I worked on several parts of this processing pipeline, particularly person identification and tracking across shots. This involved using face recognition neural networks to identify whether people detected at different points in a video were actually the same person, and assigning the corresponding pose and gesture data to the correct identities.
I also worked on shot segmentation, detecting transitions between camera shots to divide videos into meaningful segments. Together with person detection, tracking, and pose estimation, these steps make it possible to extract continuous information about a person’s movements, rather than just a collection of unrelated detections in individual frames.
The pipeline was based in part on SkelShop, an open-source toolkit for processing large collections of videos. Besides OpenPose, it includes functionality for tracking people, extracting and comparing facial embeddings, identifying faces, and processing video data.
Another challenge was running all of this on very large datasets. Processing thousands of hours of video, particularly with computationally expensive neural networks, is not something you want to do on a single machine. We therefore deployed the pipeline on a High-Performance Computing (HPC) cluster, using Snakemake to orchestrate the different processing steps and Singularity to containerize the software and its dependencies.
The cluster used SLURM to distribute jobs across different computing nodes, including both CPU and GPU nodes. This way, different stages of the pipeline could run on appropriate hardware, and large video collections could be processed in parallel. Snakemake also made it possible to manage dependencies between processing steps and avoid having to repeat expensive computations unnecessarily.
My own contributions to the project are not publicly available, but the open-source SkelShop repository and its documentation provide a good overview of the underlying tools and processing architecture.
Further Development
The Ellen DeGeneres dataset was subsequently used in a separate gesture recognition benchmark, developed as part of Google Summer of Code 2021.
In that project, OpenPose keypoints were converted into time-series representations of the detected people and their movements. These were then fed into a neural network combining convolutional LSTM layers and 3D convolutions to classify hand gestures. The model achieved an accuracy of approximately 72%, with a precision of 76% and recall of 67%.
While this benchmark was not part of my own work, it illustrates what the processing pipeline was intended to enable: turning large amounts of raw audiovisual material into structured, automatically annotated datasets that can be used to investigate the relationship between language and gestures.
Links
- Red Hen Lab - The international research network.
- Video Processing Pipeline - Overview of Red Hen’s video-processing tools.
- Gesture Detection Thesis - Earlier work on automatic gesture detection at Red Hen.
- SkelShop and its documentation - Open-source video-processing toolkit, including person tracking, face recognition, and HPC workflows.
- Ellen DeGeneres Gesture Dataset and Benchmark - Dataset description and subsequent gesture recognition results (Red Hens latest work on the Ellen PipelineA benchmark deep learning model was developed on this dataset to identify hand gestures.
In this project, the model has a 2 step process. Firstly, OpenPose keypoints are generated for the
persons in the video. Then they are parsed and divided into 3D time-series data of the form
(no-keypoints X no-of-persons X window-size). This data is fed into a model consisting of
2D Convolutional-LSTM layers (ConvLSTM2D in Keras) and 3D Convolutional layers […]
The code is deployed in a Singularity Container available on the Case HPC Cluster.
The best performance obtained was Accuracy: 0.7207, Precision: 0.7581, and Recall: 0.6684. https://sites.google.com/case.edu/techne-public-site/datasets/gsoc21-ellen-gestures ) - ELAN and annotation instructions - Tools for manual video annotation.
- Singularity at Case Western Reserve University and HPC introduction - Background on the computing infrastructure.