Text alone cannot hear sarcasm. Four channels read the same moment, and attention decides which to trust.
01
Fusion x attention
The problem with words
People do not put their feelings in one place. The words can be flat while the voice is not, or the face can contradict both, so a model reading text alone is working from the least reliable channel available to it.
This system reads four at once, in Python and PyTorch: the words through RoBERTa, the voice through Wav2Vec2, the video through a CNN and Transformer, and the face through FacialNet. Rather than averaging them, attention weights each channel by how much emotion it is actually carrying in that moment, so a shouted word counts for more than a still face.
Fusing them beat every single channel on its own: 71% on MELD, against a 75% human baseline, and 85% on three-class sentiment.
02
Cross-lingual evaluation
The language finding
Then I ran it on Persian. Text accuracy fell by 47%.
The model had not learned emotion. It had learned English. Nothing in a score reported on its training language tells you which of the two you have, which is why a model that has never been tested outside its own language is a model nobody has really tested.
03
Research tooling
Seeing inside the run
Training four models and a fusion layer is slow, and when a run goes wrong you usually find out at the end. I built a desktop application that showed the metrics as they moved and used an LLM to summarise what each run had done, so a bad configuration could be killed in the first few minutes instead of the last.
Result
A model fluent in one language is illiterate in another.
- 47%
- text accuracy lost on Persian
- 71%
- on MELD, against a 75% human baseline
- 4
- modalities, beating every single one