Expression Measurement

Vendor.com's state of the art expression measurement models for the voice, face, and language.

Intro

Vendor.com's state of the art expression measurement models for the voice, face, and language are built on 10+ years of research and advances in computational approaches to emotion science (semantic space theory) pioneered by our team. Our expression measurement models are able to capture hundreds of dimensions of human expression in audio, video, and images.

em_diagram

Measurements

  • Facial Expression, including subtle facial movements often seen as expressing love or admiration, awe, disappointment, or cringes of empathic pain, along 48 distinct dimensions of emotional meaning. Our Facial Expression model will also optionally output FACS 2.0 measurements, our model of facial movements including traditional Action Units (AUs such as “Inner brow raise”, “Nose crinkle”) and facial descriptions (“Smile”, “Wink”, “Hand over mouth”, “Hand over eyes”)
  • Speech Prosody, or the non-linguistic tone, rhythm, and timbre of speech, spanning 48 distinct dimensions of emotional meaning.
  • Vocal Burst, including laughs, sighs, huhs, hmms, cries and shrieks (to name a few), along 48 distinct dimensions of emotional meaning.
  • Emotional Language, or the emotional tone of transcribed text, along 53 dimensions.

Expressions are complex and multifaceted; they should not be treated as direct inferences of emotional experience.

Model training

The models were trained on human intensity ratings of large-scale, experimentally controlled emotional expression data gathered using the methods described in these papers: Deep learning reveals what vocal bursts express in different cultures and Deep learning reveals what facial expressions mean to people in different cultures.

While our models measure nuanced expressions that people most typically describe with emotion labels, it's important to remember that they are not a direct readout of what someone is experiencing. Sometimes, the outputs from facial and vocal models will show different emotional meanings, which is completely normal. Generally speaking, emotional experience is subjective and its expression is multimodal and context-dependent.

Specific expressions by modality

Our models measure 53 expressions identified through the subtleties of emotional language and 48 expressions discerned from facial cues, vocal bursts, and speech prosody.

EXPRESSION

LANGUAGE

FACE/BURST/PROSODY

Admiration

Adoration

Aesthetic Appreciation

Amusement

Anger

Annoyance

Anxiety

Awe

Awkwardness

Boredom

Calmness

Concentration

Concentration

Contempt

Contentment

Craving

Desire

Determination

Disappointment

Disapproval

Disgust

Distress

Doubt

Ecstasy

Embarrassment

Empathic Pain

Enthusiasm

Entrancement

Envy

Excitement

Fear

Gratitude

Guilt

Guilt

Interest

Joy

Love

Nostalgia

Pain

Pride

Realization

Relief

Romance

Sadness

Sarcasm

Satisfaction

Shame

Surprise (negative)

Surprise (positive)

Sympathy

Tiredness

Triumph