Predict future action labels instead of predicting pixel-level information — Summarization of research paper by Facebook AI

Motivation

Anticipating actions before they are executed are necessary in day to day lives for pedestrians crossing the road or for an athlete going for a high jump or playing volleyball . We have also seen that it serves wide range of practical applications including autonomous driving, robotics and secured system monitoring. Prior work done in this field has included partial observation of executed actions, without visualizing the near future. In contrast, this blog concentrates around anticipating actions seconds before they start. The proposed approach discussed here, is the fusion of a purely anticipatory model with a complementary model constrained to reason about the present. The complementary model predicts present action and scene attributes, and reasons about how they evolve over time.

Action anticipation to anticipate future actions in videos seconds before they are performed. Source — https://research.fb.com/wp-content/uploads/2019/05/Leveraging-the-Present-to-Anticipate-the-Future-in-Videos.pdf?

Automatic video understanding has improved significantly over the last few years, involving action recognition, temporal action localization, video search, video summarization and video categorization. The goal of action recognition is to recognize what action is being performed in a given video. For instance, an autonomous car should be able to recognize the intent of a pedestrian to cross the road much before the action is actually initiated in order to avoid an accident.

Other examples of future model prediction involves predicting entire sequence of locations based on a human activity using recurrent neural networks (LSTMs). This is more frequent in the domain of sports analytics such as basketball, water polo, tennis and soccer, as they aim to anticipate intention and future trajectories of a ball and individual players, by combining on-wrist motion accelerometer, heart-rate monitor and camera.

Generating video of possible future actions requires correct anticipation of long-term future. In practical industrial applications like delay tolerant industrial and safety-critical systems is definitely a critical task. This becomes challenging as the future is not deterministic. This could result in several outcomes from the current observation.

Limitations of Previous Work

• Requires smart restricted setup.
• An updated well defined system with a strong “action grammar”. The grammar should be customised and redefined specific to the context. For example videos specific to cooking, need pre-learnt and pre-defined recipes.
• Application of prediction is restricted to videos with annotation sequences.

This blog is structured as follows:

• Bringing into light a new framework, conceptualised by Facebook AI for the task of anticipating human actions. The conceptualised model is designed to produce future outcomes, seconds before they are performed. Further the scope and extensibility of the model stretches well beyond everyday actions to safety-critical systems when no partial observation of the action is available.
• The model is decomposed into two complementary models. The first component of the system is the predictive model, anticipates action directly from visual inputs and video datasets of any kind. The second one, the transitional model, is first constrained to predict what is happening in the observed time interval and then leverages this prediction to anticipate the future actions.

Modeling Process

Amidst future pixel, motion, semantic mask prediction, future frame prediction has recently attracted many research efforts. It offers different mechanisms to generate future video by:

• Predict with highest accuracy future video frames.
• Leverage a multi-scale network architecture, configurable and open to dynamic updates.
• Adversarial training with Generator and Discriminator to minimize an image’s gradient difference loss during Stochastic Gradient Ascent (SGD).
• Extrapolate and generating future video frames by applying transformations of pixels from the past.
• Using probabilistic modeling to generate multiple future frames from a single image.

Action Anticipation Model

Action Anticipation Models works on the principle of anticipating an action T seconds before it starts. If V denotes a video, then Va:b, denotes the segment of V starting at time a and ending at time b, and with Yc the label of the action that starts at time c .

“The objective is find a function f such that f(V0:t) predicts Yt+T” .

where Yt is the video with an action starting at a time t and the objective is to predict the frame after T seconds. The main idea behind our model f is decomposed as a weighted average of two functions, a predictive model fpred and a transitional model ftrans:

where α is a dataset dependent hyper-parameter chosen by validation. The first function fpred is trained to predict the future action directly from the observed segment. On the other hand, ftrans is first constrainedto compute high-level properties of the observed segment (e.g., attributes or the action performed in the present). Then, in a second stage, ftrans uses this information to anticipate the future action. In the next subsections, the prediction strategies to learn fpred and ftrans are explained.

The objective of the task is to predict an action T seconds before it starts to be performed. While the predictive model directly anticipates the future action, the transitional model is first constrained to output what is currently happening. Then, it uses this information to anticipate future actions.

As you can see the above figure above, it illustrates the adversarial network predicting a child’s action based on future video frame generation way before it has actually occurred. This kind of prediction of future actions would help to add extra additional security measures during monitoring live video.

This is accomplished through the use of transitional models. Upper: Action Recognition (AR) based transitional model learns to prediction future actions based on the predictions of an action recognition classifier applied on current/present frames (clips). Lower: Visual Attributes (VA) based transitional model learns to predict future actions based on visual attributes of the current/present frames (clips).

Predictive model (fpred)

The goal of the predictive model fpred is to directly anticipate future action from the visual input. As opposed to ftrans, fpred is not subject to any specific constraint. In order to train a “video V with action labels Yt0+T , . . . , Ttn+T, for each label Yti+T”, the objective is to minimize the loss:

where s(ti) = max(0, ti −tpred),

l is the cross entropy loss, tpred is a dataset dependent hyper-parameter, also chosen by validation, that represents the maximum temporal interval of a video fpred has access to. This hyper-parameter is essential because looking too much in the past may add irrelevant information that degrades prediction performance. This loss is then summed up over all videos from the training dataset. fpred is a linear model which takes as input a video descriptor.

Transitional model (ftrans)

The transitional model ftrans splits the prediction into two stages: gs and gt. The first stage gs aims at recognizing a current state s, describing the observed video segment. The state s can represent an action or a latent action-attribute. The second stage gt takes as input the current state s, and anticipates the next action given the current state s. gs is a complex function extracting high-level information from the observed video segment, while gt is a simple linear function operating on the state s and modeling the correlation between the present state and the future action. There are two different approaches for transitional model:

• Based on Action Recognition (AR) and
• Based on Visual Attributes (VR)

Transitional Model based on Visual Attributes leverage visual attributes to anticipate the future. Visual attributes have been previously used for action recognition. The idea is to first predefine a set of visual attributes describing the presence or absence of objects, scenes or atomic actions in a video. Facebook Research team trains the model based on these visual attributes for action recognition that express the transitional model. The current state :

predicted by gs, is then a vector of visual attributes probabilities, where a is the number of visual attributes. Given the presently observed visual attribute s, gt predicts the future action, with gt being modeled as a low-rank linear model:

These parameters are learned, in the same manner as the predictive model, by minimizing the cross entropy loss between the predicted action given by gt(s) and the future action ground-truth. Implementing gt through a low-rank model reduces the number of parameters to estimate and leads to better accuracy.

Transitional Model based on Action Recognition

Real-world videos often consist of a sequence of elementary actions performed by a person in order to reach a final goal such as Preparing coffee, Changing car tire or Assembling a chair. Many datasets come with a training set where each video has been annotated with action labels and segment boundaries for all occurring actions. When this is available action labels can be used, instead of predefined visual attributes for state s. The anticipation of the next action significantly depends on the present action being performed. With Markov assumption on the sequence of performed actions an ordered sequence of action annotations can be represented as:

where an defines the action class performed in video segment Vn. The model

can be represented as:

∀n∈{0,…,N−1}, i∈{1,…,K}.

It denotes a conditional probability event, based on previous sequential actions.

is decomposed in terms of two factors: An action recognition model gs(Vn) that predicts P(an = j|Vn) using conditional probability based on the current action.

A transition matrix T that captures the statistical correlation between the present and the future action, i.e. such that :

gt takes as input the probability scores of each action given by gs to anticipate the next action in a probabilistic manner:

The transition matrix T can be computedby estimating the conditional probabilities between present and future actions from the the sequences of action annotations in the training set.

Prediction Explainability.

The transitional model ftrans provides interpretable predictions that can be easily analyzed as:

• The function gt of the transitional model can be expressed as a simple linear model. This can be applied to the state s, which when used jointly in conjunction with visual attributes plays an important role to predict future actions.
• The linear weights of gt serves to convey the importance of each element in s. These weights contribute either to amplify or to suppress the anticipated movement of the action class. In order to anticipate future actions, given an action class k, the linear weights of gt are decomposed and analyzed to infer the most important visual attributes. The most dominant of the visual attributes and the actions of the said class are foremostly most responsible for the prediction of action class k.
• It also provides an easy way to diagnose the source of mis-predictions. For example, in any case when the transitional model anticipates wrongly an action k , then the reason behind such mis-prediction can be analyzed as recognition problem (i.e. wrong detection score for the visual attribute/action class) or due erroneous updations of the learned transition weights.

Conclusion

• Facebook AI research has opened a new domain of machine learning solutions where future actions are predicted by leveraging ML models built for future action anticipation.
• The main motivating factor for this method is to model action anticipation as a fusion of two complementary modules. The predictive approach is a purely anticipatory model, which aims at directly predicting future action given the present. The transitional model is first constrained to recognize “what is currently seen in the present actions and then uses this output to anticipate future actions”.
• The transitional model ftrans, is assigned the task is to analyze and discover the dominant visual attributes and its role and contribution for anticipation of each action class.
• As quoted by FB AI Research “The Action Recognition (AR) transitional model performs better than the Visual Attributes (VA) transitional model”. However, both are outperformed by the purely-predictive model (fpred).
• As ensembling model prediction works better in ML modeling, here we see that combining the predictive model with either of the two transitional models (Predictive + Transitional (VA) / Predictive + Transitional (AR)) boosts the performance, thus yielding further accuracy gains.
• The observed video segment V is represented, analysed and segmented to perform action prediction. The overall strategy is to split the video into clips, extract clips representation and perform pooling through a ConvNet over each of these clips. The uniformly splitted small clips V = [V1,…,VN] with 8 or 16 frames can be fed into a pre-trained video CNN C. From the penultimate layer of the CNN, an L2-normalized one-dimensional representation C(Vi) is extracted for each clip Vi.
• In order to visualize the resultant video a temporal aggregation is carried out. In other words, Aggregate([C(V1), . . . , C(VN )]) of the extracted features get a one-dimensional video representation for V.

References:

1. https://research.fb.com/wp-content/uploads/2019/05/Leveraging-the-Present-to-Anticipate-the-Future-in-Videos.pdf?
2. http://activity-net.org/download.html
3. https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Heilbron_ActivityNet_A_Large-Scale_2015_CVPR_paper.pdf