Understanding emotion scores

Emotion can be described two ways: a categorical distribution across named emotions, and a dimensional projection onto a small number of continuous measures. They do not disagree — one is a projection of the other.

The categorical scores below are on every result, from either surface. The dimensional measures are on the realtime frames only; a REST analysis response carries the label scores without them.

Categorical: scores per label

scores maps each label to a value in 0…1, and dominant names the highest-scoring one.

{
  "dominant": "happy",
  "scores": {
    "angry": 0.01, "contempt": 0.005, "disgust": 0.005, "fear": 0.005,
    "happy": 0.65, "neutral": 0.20, "sad": 0.03, "surprise": 0.10
  }
}

The three label sets are different

Video (8) — angry · contempt · disgust · fear · happy · neutral · sad · surprise

Audio (8) — angry · calm · disgust · fear · happy · neutral · sad · surprise

Text (28)

Group	Labels
Positive	admiration amusement approval caring desire excitement gratitude joy love optimism pride relief
Negative	anger annoyance disappointment disapproval disgust embarrassment fear grief nervousness remorse sadness
Other	confusion curiosity realization surprise neutral

Video and audio are not interchangeable

Audio has calm where video has contempt. Plotting them on one chart without accounting for that silently compares two different things.

Read the keys; do not hardcode the set

The text set may be extended — up to four further labels (sarcasm, boredom, contempt, envy) can appear, for 32 in total. Code that iterates the object's keys keeps working. Code with 28 fixed slots breaks. The same applies to video and audio.

Dimensional: the circumplex projection

On a realtime emotion frame, alongside scores:

Field	Range	Meaning
valence	−1 … 1	Unpleasant → pleasant
arousal	−1 … 1	Calm → activated
intensity	0 … 1.414	Distance from the neutral centre — the magnitude of (valence, arousal)
expressiveness	0 … 1	1 − p(neutral). How much expression is present at all, which is not the same as how strong it is
emotion_entropy	0 … 1	Certainty. 0 is one clear emotion; 1 is uniformly unsure
circumplex_version	string	Which coordinate table produced the five above

intensity and expressiveness answer different questions. A face can be unmistakably, calmly neutral — low intensity, low expressiveness. It can also be clearly expressing something mild — low intensity, high expressiveness. If you are picking one number to drive a UI, expressiveness is usually the one that matches intuition.

emotion_entropy is a confidence signal. A high value means the model saw something it could not commit to. Surfacing a dominant label from a high-entropy frame as though it were certain overstates what the analysis found; 1 - emotion_entropy reads naturally as certainty.

Three rules for storing these

All six fields are nullable. null means "not computed for this frame" — never treat it as zero.

Do not compare values across circumplex_version values. The coordinate tables are not on a common scale. If you persist these numbers, persist the version beside them.

A frame with the five measures populated but circumplex_version: null is v1. That combination means the producer predates the field, and v1 was the only table that existed then. Read it as "v1" rather than as unknown.

Text emotion arrives late, on purpose

In a realtime session, an audio segment's text_emotions is zero-filled while the session runs, and dominant_text_emotion is null. The fields exist so the frame shape stays constant; the real scores follow in a separate text_emotion frame, matched on start_time_ms.

The exception is the final segment, finalised while the session is being torn down: it carries real values inline and gets no follow-up frame.

Code that joins on start_time_ms and falls back to whatever is on the frame itself handles both cases. Code that reads text_emotions directly from a live audio frame renders a wall of zeros.

The follow-up is best-effort. If the analysis fails or times out for a segment, no text_emotion frame arrives for it at all. Treat a missing one as "no update available" — not as an error, and not as a zero result. Don't block a segment's display waiting for one.