CASTER: A Multimodal Dataset and Benchmark for Observation-Grounded StarCraft Commentary Generation

Commentary Alignment and State Tracking for Esports Research: utterance-aligned observer-view clips, observer-visible structured state, and quality-controlled professional commentary.

239 matches 28,734 aligned instances 2 benchmark tasks
GitHub Hugging Face

Abstract

Many multimedia language-generation tasks require grounding in both visible content and evolving, partially observable structured states. Professional esports commentary exemplifies this: in StarCraft: Brood War, meaningful commentary relies on short-term temporal context, observer camera motion, and observer-visible unit configurations. We introduce CASTER (Commentary Alignment and State Tracking for Esports Research), a multimodal dataset for observation-grounded commentary in partially observable environments. It aligns utterance-level observer-view clips, structured states from replay logs and viewport traces, and ASR+LLM-refined, manually verified commentary.

CASTER comprises 239 matches and 28,734 aligned clip–state–commentary instances with strict game-level splits. We define two tasks for reproducible evaluation: (1) Clip-to-Observation, predicting structured observer-state events from clips, and (2) Clip+Observation-to-Commentary, generating commentary conditioned on synchronized visual and structured inputs. We detail the synchronization pipeline, report a 6,000-sample manual audit of alignment quality, and provide zero-shot baselines. Results indicate that recovering structured state from clips alone remains challenging, while Task 2 establishes reference points for commentary generation under various conditionings. CASTER offers a novel testbed for state-grounded narration, multimodal game understanding, and commentary assistance.

Pipeline

Overview of the alignment and observation-generation pipeline.
Overview of the alignment and observation-generation pipeline. For each replay, we render a full-match observer-view video, align commentary segments to replay frames, extract utterance-level clips, aggregate observer-visible structured state with viewport traces, compress chunk-level events into scene-level observations, and perform sampled manual quality control.

Alignment Example

Replay-rendered observer-view frame used in the CASTER alignment example.

Replay-rendered observer view at frame 15,393

Frame Player Unit Name (ID) X, Y HP ...
15393 2 Terran_Marine(223) 3181, 1536 40
15393 2 Terran_Siege_Tank_Siege(227) 3440, 1456 36
15393 2 Terran_SCV(232) 3339, 1487 60
15393 2 Terran_Siege_Tank_Siege(293) 3368, 1460 70
15393 2 Terran_Siege_Tank_Tank(321) 3266, 1490 60
15393 2 Terran_Missile_Turret(323) 3456, 1600 200
15393 2 Terran_Missile_Turret(350) 3424, 1504 200
15393 1 Protoss_Dragoon(329) 3424, 1280 100
15393 1 Protoss_Dragoon(330) 3274, 1320 100
... ... ... ... ... ...

Synchronized structured state (state.csv) at frame 15,393

{
  "seg_index": 86,
  "clip_path": "match_124/clip/match_124_086",
  "speech": "The shuttle gets taken down instantly! Meanwhile, the Dragoons are trying to find an angle, firing from the low ground up into [PLAYER_1]'s position.",
  "speech_tag": "ACTION_CALL",
  "time": "15325~15421",
  "events": [
    {
      "frames": "15325~15421",
      "viewport": "[3002,1301]",
      "units": {
        "[PLAYER_1]": [
          "Marine*1 [3181,1536]",
          "Missile_Turret*3 [3306,1621]",
          "SCV*2 [3311,1474]]->[3315,1508]",
          "Siege_Tank_Siege_Mode*2 [3404,1458]",
          "Siege_Tank_Tank_Mode*2 [3114,1533]]->[3129,1611]",
          "Vulture*2 [3357,1548]]->[3335,1568]",
          "Vulture_Spider_Mine*2 [3094,1382]"
        ],
        "[PLAYER_2]": [
          "Dragoon*4 [3390,1275]]->[3347,1295]",
          "Observer*1 [3161,1256]",
          "Zealot*2 [3313,1481]]->[3316,1483]"
        ]
      }
    }
  ]
}

Aligned clip-level record (context.json)

Example of multimodal alignment in CASTER. The replay-rendered visual observation is synchronized with its underlying structured state, which tracks observer-visible unit information. These frame-level details are then compressed into a clip-level JSON schema containing summarized gameplay events and utterance-aligned commentary.

Game-Level Diversity Statistics

Race picks

Matchup distribution

Game-level diversity statistics. T, P, and Z denote the three playable races in StarCraft: Terran, Protoss, and Zerg, respectively.

Dataset Summary and Sampled Quality Control

Dataset statistics
Matches239
Clips / utterances28,734
Avg. clip duration7.07 s
Median clip duration6.30 s
Total clip hours56.43
Avg. utterance length (tokens)29.5
Avg. utterance length (words)23.5
Video resolution640 × 480
Frame rate (FPS)23.81
Train / Validation / Test matches205 / 17 / 17
Quality control (sampled audit N=6000)
Temporal alignment within ± 5 frames95% (5700/6000)
Viewport acceptable (≤ 30 px error)95% (5700/6000)
Refined commentary judged meaning-preserving99% (5940/6000)
Not retained after final QC2% (120/6000)

Label Distribution

Distribution of discourse-function labels in CASTER. Four segments without a final valid tag are excluded from this analysis, yielding 28,730 tagged instances.

Verifying Label Integrity in Commentary Classification

Confusion matrix of the auxiliary ModernBERT classifier evaluated on the annotated dataset.
Confusion matrix of the auxiliary ModernBERT classifier evaluated on the annotated dataset. The strong diagonal dominance with negligible offdiagonal overlap indicates that the LLM-induced annotation logic is systematic and highly consistent.
t-SNE projection of the contextualized representations from the classifier's final hidden layer.
t-SNE projection of the contextualized representations from the classifier's final hidden layer. The highly cohesive and strictly separated clusters confirm that the chosen discourse categories capture genuinely distinct semantic intents.