Vector Institute MOZAIC AIXPERT

MOZAIC: An Audio-Video Deepfake Benchmark for Telling What Is Fake

Shaina Raza1, Mohamed Hafez1, Ahmed Radwan1, Khalid Elgazzar2, Kathryn Hume1

1Vector Institute, Canada    2Ontario Tech University, Canada

MOZAIC gives every clip a four-way label (real or fake video × real or fake audio), so a detector's score can be traced to the picture or the sound, and asks four questions: is it fake, what kind, whose identity, and why.

137,745 clips169.9 hours1,000 identities15,014 source pairs289,587 questions4 tasks
MOZAIC overview: paired identities, video and voice generation, quality control, the six released conditions per pair, identity-disjoint splits, and the four evaluation tasks
Figure 1. MOZAIC. Top: a real clip of person A is paired with a same-gender person B; three video generators (FaceFusion, Wan2.2-S2V, Wan2.2-Animate) and one voice converter (Seed-VC) create the fakes, pairs that fail quality control are dropped, and each pair is released in six conditions (real or fake video with real, vocoded or fake audio). Bottom: the four tasks: which modality is fake (T1), how the video was made (T2), whether a query clip is real and shows the reference person (T3), and a written explanation (T4).

Abstract

Audio-video (AV) deepfakes can fake the face, the voice, or both. Most benchmarks only ask whether a clip is real or fake, so they cannot show whether a detector relies on the picture, the voice, or encoding traces.

We present MOZAIC, a paired benchmark with a four-way label (real/fake video × real/fake audio) on every clip, so each modality's contribution can be measured directly. Fakes cover inner-face swapping, whole-frame video diffusion and voice conversion. The train, validation and test splits have disjoint identities. MOZAIC has 137,745 clips from 15,014 source pairs and four tasks: T1 audio-video authenticity, T2 manipulation type, T3 identity verification and T4 explanation.

We evaluate 12 omni-modal LLMs (MLLMs) zero-shot and five trained detectors. Zero-shot, the best MLLMs reach 35.6% on T1, 41.0% on T2, 47.5% on T3 and 5.15 out of 10 on T4, and catch fake video far more often than fake audio (43.6% vs 7.5%). With training, the best detectors reach 98.4% on T1 and 99.9% on T2, and LoRA fine-tuning lifts Qwen2.5-Omni to 83.4%, 73.6% and 75.8% on T1–T3; its explanation score rises from 4.42 to 6.98. The best T1 detector falls to 78.3% when the face-swap generator is held out of training, so generalization and explanation remain open. We release the dataset, a compact MOZAIC-Small split, code, training recipes, and prompts.

Overview

What MOZAIC measures

Sunburst of clips by label (RVRA 30,028, RVFA 15,014, FVRA, FVFA) and by generator (FaceFusion, Wan2.2-S2V, Wan2.2-Animate)

(a) Clips by label and generator.

Bar chart: median similarity of each fake to source A (low, 0.01 to 0.14) and to target B (high, 0.50 to 0.87) for FaceFusion, Wan2.2-S2V, Wan2.2-Animate and Seed-VC

(b) Identity transfer.

MOZAIC statistics. (a) Clips by label (inner ring) and generator (outer ring). (b) Median similarity of each fake to the source A and the target B, face for video generators and voice for Seed-VC: every generator moves the identity to B.

Dataset

  • PairsA source clip of one person and a target person of the same gender, from VoxCeleb2: 15,014 pairs, at 3.08 s (77 frames) or 6.12 s (153 frames).
  • FakesThe inner face is swapped (FaceFusion), or the whole frame is generated, from the sound (Wan2.2-S2V) or from the original's motion (Wan2.2-Animate). The voice is converted (Seed-VC).
  • Vocoded copiesReal voices are also released through BigVGAN, the vocoder Seed-VC uses, so a vocoder trace alone does not mean fake.
  • Same processing, split by personEvery clip gets the same final processing, and nobody in the test split appears in training.
Two pairs (a woman and a man), each shown as the real recording, FaceFusion, Wan2.2-S2V and Wan2.2-Animate: frames at 0 s and 3 s, a close-up, and the changed pixels in red
What each generator changes. One 3 s test pair per generator (rows), from the same real frames of A. FaceFusion replaces only the inner face; Wan2.2-Animate and Wan2.2-S2V redraw the whole frame, driven by A's motion and voice, respectively. Last column: target B (top) and changed pixels (red).

Clips are 3.08 s (77 frames) or 6.12 s (153 frames), 224×224 at 25 fps, with 16 kHz sound. The splits are by person: 700, 150 and 150 identities in train, val and test (96,957, 20,553 and 20,235 clips), half men and half women in each. MOZAIC-Small has 16,995 clips (21.0 hours) from 1,858 whole pairs of the same 1,000 identities, with 47,087 questions for T1 to T4.

Watch and Listen

The same test pair in all four versions. Pick the generator for the fake face; one clip plays at a time.

Animate shows another test pair, in all four versions.

RVRAreal face, real voice
RVFAreal face, fake voice (Seed-VC)
FVRAfake face (FaceFusion), real voice
FVFAfake face (FaceFusion), fake voice (Seed-VC)

Web copies for viewing; the released clips are unchanged.

Evaluation Tasks

Each card shows the right answer to one real test question.

T1

Audio-video authenticity

Which part of this audio-visual clip, if any, is AI-generated or manipulated?

  • the video
  • the audio
  • both
  • neither

91,830 questions · frames and sound

T2

Manipulation type

Which part of this video, if any, was generated or altered by AI?

  • real
  • face swap
  • full-body swap
  • fully generated

45,915 questions · frames

T3

Identity verification

Is the face (or voice) in clip 1 real or fake, and is it the same person as in clip 2?

  • real, same person
  • real, different person
  • fake of this person
  • fake of a different person

151,842 questions · two clips, face or voice

T4

Explanation

Which parts of this audio-visual clip, if any, were generated or altered by AI, where, and what in the clip shows it?

“Only the inner face of the video is AI-generated; the hair, head, body and background are untouched. … While she speaks, her facial features look somewhat soft and blurry …”

11,330 questions · MOZAIC-Small · judge score out of 10

Results

T1 to T3: accuracy on the full test split. T4: judge score out of 10 on the MOZAIC-Small test split. The fine-tuned model is a single training run.

ModelT1T2T3T4
Qwen3-Omni-30B-A3B18.52%34.13%28.22%4.22
Qwen2.5-Omni-7B19.63%33.60%32.21%4.42
MiniCPM-o 4.524.00%33.48%32.26%4.39
Nemotron 3 Nano Omni18.70%33.94%32.92%4.50
Gemma 4 12B18.18%33.27%29.65%4.12
OmniVinci16.63%30.05%30.04%4.17
Uni-MoE-2.0-Omni29.81%31.21%26.08%4.33
Ola-7B32.50%32.71%22.45%4.02
VITA-1.517.35%31.99%22.51%4.52
VideoLLaMA2-AV30.88%29.62%29.19%4.41
Gemini 3.1 Pro35.58%36.68%47.51%5.15
Gemini 3.8 Flash31.56%40.98%40.92%5.09
Qwen2.5-Omni-7B + LoRA, trained on MOZAIC-Small (single run)83.8%78.3%75.6%6.98
Most common answer33.39%33.22%30.04%–

Generalization

Per generator, zero-shot models barely beat chance; trained detectors do well in distribution but lose much of it on a generator held out of training.

Heatmap of T1 balanced accuracy per generator (FaceFusion, Wan2.2-S2V, Wan2.2-Animate, Seed-VC) for the twelve zero-shot models: open models stay within a few points of 50%, the Gemini models reach 59 to 69% on the video generators and at most 55% on Seed-VC
T1 balanced accuracy (%) per generator for every off-the-shelf model on the MOZAIC test set. Each cell averages the fake-part recall for that generator with the real-part recall of the same modality (video; audio for Seed-VC), so a constant answer scores 50% (chance). Red is below chance, blue above, saturating at ±10 points; bold marks the best model per generator.
Dumbbell chart of macro-F1 for AV-HuBERT, HAVIC, PE-AV and Qwen2.5-Omni when each generator is seen in training and when it is held out: the largest drops are on FaceFusion
Generalization to unseen generators. Macro-F1 (four classes) of each detector when a generator is seen in training (blue) and when it is held out from training (pink); each group holds out one generator and is scored on its clips. The line length shows the drop.

Key findings

  • Near chance, zero-shot. For the open models, 36 of 40 model–generator cells lie within four points of chance; the converted voice is the blind spot for every model.
  • A new face swap breaks the detectors. With FaceFusion held out of training, T1 accuracy on its clips falls from 98.6% to 78.3% for PE-AV and from 78.9% to 56.4% for Qwen2.5-Omni; holding out a whole-frame generator costs PE-AV less than a point.

What the Explanations Say

T4 asks a model to say which part of a clip is fake, where, and what shows it. Four test clips, two models.

Four MOZAIC-Small test clips (real; face swap with real voice; real video with converted voice; fully generated) with the verdicts of Gemini 3.1 Pro zero-shot and the fine-tuned Qwen2.5-Omni, each marked against the reference with the judge score
T4 on four MOZAIC-Small test clips: real, face swap with real voice, real video with converted voice, and fully generated. Borders, envelopes and pills: green real, red generated; dashed box: swapped face. Each chip gives what the model says about the video (V) or audio (A), a mark (check: matches the reference; wave: right verdict, wrong region; cross: contradicts it) and the judge's 0 to 10 score, followed by a verbatim excerpt.

Key findings

  • Open zero-shot models say "real". for 96 to 100% of videos, whatever they are. On the judge, none beats a fixed always-real answer.
  • Gemini 3.1 Pro over-claims the inner face. on about half of all clips, real ones included, and never names a generated whole frame.
  • Fine-tuning fixes the voice, not the region. Qwen2.5-Omni gets the voice right (97% of real, 78% of converted) but calls most videos whole-frame generated.

Conclusion

MOZAIC pairs every source clip with all four real/fake combinations of video and audio, plus a vocoded control, so a score can be traced to the video or the audio a model actually uses. Measured this way, today's omni-modal LLMs are near chance zero-shot: the best reaches 35.6% on the four-way task, fake voices mostly go unnoticed, and the open models' free-text explanations score below a baseline that calls everything real. Trained detectors reach up to 98.4% in distribution, but the best one loses 20 points on a held-out face-swap generator.

Quick Start

The repository has the code to generate the clips, build the questions, train the baselines, and run and score every model; each folder has its own README.

git clone https://github.com/VectorInstitute/mozaic.git
cd mozaic && uv sync

# run a model on T1, then score its answers (no model needed to score)
python -m mozaic.evaluation.run_inference --config gemini_3_8_flash --task t1
python -m mozaic.evaluation.run_eval --config gemini_3_8_flash --task t1

# T4 on MOZAIC-Small: answers, the judge, then the scores
python -m mozaic.evaluation.run_inference --config gemini_3_8_flash --task t4 --subset small
python -m mozaic.evaluation.judge.run_judge --config gemini_3_8_flash --task t4 --subset small
python -m mozaic.evaluation.run_eval --config gemini_3_8_flash --task t4 --subset small

# train a baseline on MOZAIC-Small
torchrun --nproc_per_node=4 -m mozaic.training.run_train --config avhubert/t1_ln --subset small

Access and Licence

The code is under the Apache License 2.0. The dataset is on Hugging Face under CC BY-NC 4.0. Anyone shown in a clip can ask for it to be removed.

BibTeX

To cite MOZAIC:

@misc{mozaic2026,
  title  = {{MOZAIC}: An Audio-Video Deepfake Benchmark for Telling What Is Fake},
  author = {Raza, Shaina and Hafez, Mohamed and Radwan, Ahmed and Elgazzar, Khalid and Hume, Kathryn},
  year   = {2026},
  note   = {Project page}
}

Acknowledgements

Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute (vectorinstitute.ai/#partners).

This research was funded by the European Union's Horizon Europe research and innovation programme under the AIXPERT project (Grant Agreement No. 101214389), which aims to develop an agentic, multi-layered, GenAI-powered framework for creating explainable, accountable, and transparent AI systems.

Vector Institute AIXPERT