EgoCaptioner

Towards Comprehensive and Fine-Grained Egocentric Video Captioning

Yuqi Tang1,2, Yang Shi3,2,†, Yuran Wang3, Tengfei Liu3, Rundong Wang4, Zongyang Qiu1, Qixun Wang3, Zhuoran Zhang3, Xuanyu Zhu3, Yifan Xu5, Yiyan Ji5, Shujie Li6, Shixuan Liu7, Haotian Wang7, Yuanxing Zhang2,♠, Pengfei Wan2, Chengwei Qin1,†

1HKUST(GZ), 2KlingAI Research, 3PKU, 4USTC, 5NJU, 6HKU, 7THU.

† Corresponding Author ♠ Project Lead

Example Captions

Pick a task type, play the video, and select a step to jump to that moment.

Question

Overview

Detailed Steps

    Abstract

    Egocentric video captioning requires fine-grained detail and accurate timing, yet current multimodal large language models (MLLMs) often miss brief actions, confuse their order, or misinterpret the wearer’s motion. Existing caption formats also offer limited support for combining detailed narration with precise timing.

    We present EgoCaptioner, which pairs a video-level overview of the setting, participants, and the wearer’s overall activity with timed steps whose narration adapts to varying perceptual demands and marks key moments at specific timestamps. We develop a coarse-to-fine annotation pipeline that establishes event structure before expanding step descriptions, yielding 50,000 training examples from eight egocentric video sources.

    Complementary rewards assess factual accuracy and coverage, measure step-level temporal alignment, and penalize fragmented or redundant narration, guiding caption refinement through Group Relative Policy Optimization (GRPO). To improve the reliability of factual supervision, we supplement caption references with pose-derived motion facts when sensor data are available.

    Across five egocentric benchmarks, EgoCaptioner achieves leading performance among evaluated open-source models in caption quality, caption-conditioned question answering, and temporal alignment. It also surpasses Gemini 3.1 Pro on PhysCapBench and all evaluated proprietary baselines on Minerva-Ego under the caption-to-QA protocol. Ablations support the complementary roles of factual and temporal rewards and show that coarse-to-fine annotation provides more effective supervision than single-pass annotation.

    • 50,000training examples
    • 8egocentric video sources
    • 5evaluation benchmarks

    What a Caption Covers

    EgoCaptioner overview: an egocentric video is described by an overview and timed detailed steps, with structured understanding, accurate motion validated by pose, and task-focused details for manipulation, locomotion, social interaction and attention.
    Each caption pairs an overview of the scene, people and overall activity with timed steps. Step narration adapts to what the moment needs: hand–object interaction, spatial movement, social interaction, or visual attention. Motion descriptions are checked against body and camera poses when sensor data exist.

    Method

    Data Construction

    Gemini 3.1 Pro and Seed 2.1 Pro annotate each video from a global overview to event outlines and then timestamped steps. The draft is reviewed against the full video and corrected. Body and camera poses are converted into timed motion facts.

    Data construction pipeline with four stages: coarse-to-fine annotation, review and correction, instruction construction, and pose-derived motion facts.
    Data construction. Each validated caption yields full-video, event-conditioned, and overview instructions.

    Rewards for GRPO

    Step-level captions combine fact, time, and style rewards. The fact reward is the F1 of judge-scored precision and recall over extracted facts. The time reward matches predicted and reference steps by temporal IoU. The style reward penalizes fragmented or redundant steps. Overview captions pair the content reward with a format reward.

    Reward design: fact, time and style rewards for step-level captions, and content plus format rewards for overview captions.
    Reward design for egocentric captioning.

    Benchmark Results

    Table 1. Caption quality, video understanding, and temporal alignment across five egocentric benchmarks.

    Scroll horizontally to see all metrics.

    Model EgoMe-Overview EgoMe-Steps EgoTempo Minerva-Ego EgoHome PhysCapBench
    B@4 METEOR R-L CIDEr METEOR SODA-c SODA-d Acc Score Accopen PSJ Overall
    Proprietary MLLMs
    Gemini 3.1 Pro 3.3218.6224.455.613.987.4146.8023.601.4444.2952.8675.06
    Gemini 3.8 Flash 2.6219.2722.120.796.2610.3448.3141.202.3107.1471.0982.77
    GPT-6-Astra 1.2816.1215.950.005.916.2543.1252.602.84215.7179.4284.87
    Open-source MLLMs and Captioners
    Qwen3-VL-8B-Instruct 1.5915.7216.500.055.736.3443.3314.000.9661.4355.8868.94
    Qwen3-VL-8B-Thinking 1.7315.9318.180.454.685.4838.0517.201.1001.4354.0268.98
    Qwen3.5-9B 2.6517.5822.594.015.718.0250.2521.601.3880.0046.9051.24
    Qwen3.8-27B 1.3715.7114.820.027.247.0945.6730.801.84211.4369.2869.55
    Molmo2-8B 1.2014.6415.020.114.764.3038.1919.201.2047.1454.2053.20
    Tarsier2-7B 2.7217.3621.051.010.000.000.0018.001.2122.860.0062.92
    CapRL-Video-4B 1.3414.2312.500.004.103.1730.2531.801.97810.0062.8775.94
    TimeLens-8B 1.5915.3516.230.106.377.1046.9317.801.1241.4360.1560.25
    AVoCaDO 1.1413.5312.640.000.000.000.0013.600.9184.290.0060.84
    Egocentric MLLMs and Captioners
    EgoThinker† 2.9615.8623.034.851.861.7914.9324.401.77015.719.4839.88
    EgoGPT 2.0114.1719.982.500.000.000.009.200.6900.000.0032.92
    EgoCaptioner 4.0320.4023.590.6512.268.0456.3838.402.06618.5775.5178.84

    Higher is better for every metric. All scores are scaled by 100 except EgoTempo Score (0–5). B@4 denotes BLEU-4 and R-L denotes ROUGE-L. PhysCapBench Overall averages seven dimensions. † indicates direct video QA. Best and second-best mark rankings among open-source models.