The Video Shows You Something. The Answer Is Somewhere Else.

Research paper: Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agents (arXiv:2608.23329)

An 8B model went from 9.6% to 57.7%. The difference wasn’t a bigger model. It learned how to research.

VideoRover shows how temporal grounding can anchor an adaptive research agent, and how training turns tool access into evidence-seeking behaviour.

Some video questions require two linked forms of evidence.

A coin appears in close-up for two seconds of a twelve-minute clip; the question asks about the ancestry of the Roman family whose name is stamped on it. A footballer appears in a post-match interview; the question asks which year he scored his third World Cup goal.

The video identifies the subject. The web supplies the requested fact.

Solving the problem means finding the decisive visual evidence, researching beyond the video, and keeping those two processes connected. What the agent sees should determine what it searches for. What it finds should be able to send it back into the video for another look.

That is the idea behind VideoRover, a framework from researchers at Shandong University, Kuaishou Technology and collaborators across several institutions. It brings together active video reasoning, where a model decides which parts of a video deserve closer inspection, and deep research, where an agent searches, opens sources and gathers evidence over multiple steps.

VideoRover trains the coordination between those capabilities: use the current evidence state to choose the next move.

Start with a better view

VideoRover begins from a compressed orientation view of the video: one frame per second, capped at 256 frames, each reduced to 224 x 224 pixels.

The training trajectories then impose a clear first action. The agent identifies a candidate interval and crops it. That segment is sampled more densely, at two frames per second, up to 32 frames, at the video’s original resolution.

From there, the research process becomes adaptive. The agent can crop another segment, image-search a keyframe, run a text search, open a source, revisit the video, or answer. Each observation updates the shared research state and changes the next choice.

Temporal grounding therefore becomes the anchor for the entire research loop.

Anyone who has worked on long-video annotation will recognize the underlying quality problem. A long recording is reduced to a small trace before serious reasoning begins. The quality of the later reasoning depends on whether the decisive moment survives that reduction.

VideoRover encodes the response directly: look closer before you go find out.

Task design isolates the capability

The paper’s data construction is worth studying in its own right.

Every question carries two dependencies. The video identifies a target that the question refers to indirectly, such as “a book cover displayed in the video.” External knowledge then supplies the fact being asked about. Together, the two sources form the task.

The pipeline starts with visually identifiable people, places, organizations, works, products and events drawn from Wikipedia. Candidate YouTube videos are retrieved. One model localizes likely intervals across densely sampled frames and subtitles; another independently checks that the selected keyframe matches the target.

Questions are written with the target’s identity withheld. Retrieval is then disabled and a smaller video model attempts each question using the video and prompt alone. Questions it can already solve are discarded.

That filter removes shortcut-solvable examples and concentrates the dataset on the intended capability: identify from video, research beyond it, and connect the evidence.

The same discipline continues into trajectory generation. A text-only research planner decides what to do next; a vision model reports what is actually visible in the sampled frames and crops. The pipeline keeps trajectories that reach the correct final answer.

This produces 26,000 supervised trajectories of ten tool calls or fewer. Longer questions become candidates for reinforcement learning. For those, the authors sample five complete rollouts, filter out questions the model already solves in more than three attempts, and retain 3,000 harder instances balanced around grounding difficulty, search frequency and interaction complexity.

That construction gives reinforcement learning useful variation inside each group: the same policy sometimes succeeds and sometimes fails, creating a signal from the difference between those behaviours.

The four-number result

The clearest result in the paper is the progression of the same 8B backbone as more of the behaviour is taught.

9.57% - Qwen3-VL-8B answering directly.

31.85% - the same backbone with the shared agentic tool suite.

39.28% - VideoRover-8B after supervised fine-tuning on verified research trajectories.

57.71% - VideoRover-8B after reinforcement learning on the harder long-horizon cases.

The sequence separates four sources of capability. Tool access lifts the backbone from 9.57% to 31.85%. Supervised trajectories add a research prior: how to ground the video, choose a search mode, inspect sources and keep evidence connected. Reinforcement learning then pushes the agent to adapt that process under live tool feedback across longer research chains.

At 57.71%, VideoRover-8B finishes 0.71 points behind Gemini-3-Flash’s 58.42% average in the proprietary direct-answer setting, and ahead of the larger Qwen3.5-27B using the identical agentic tool suite at 55.42%. The operating conditions differ: VideoRover uses external tools, while the proprietary models answer directly. The proximity shows how far an 8B backbone can travel when evidence acquisition is trained as part of the capability.

The ablations reinforce the role of each evidence route. The full model scores 57.71%, compared with 49.57% when dense video cropping is removed, 49.00% without image search, 23.71% without text search, and 13.85% with the web tools removed. Text search supplies the external facts. Cropping and image search help establish what the agent is looking at. The policy has to diagnose which evidence gap is active and select the tool that closes it.

From frame selection to evidence-seeking trajectories

VideoRover fits into a broader shift in video reasoning.

VideoITG made frame selection instruction-conditioned, selecting evidence according to the question being asked. Temporal Chain of Thought pushed toward iterative retrieval of the frames a model needs during reasoning. VideoRover extends that pattern beyond the boundary of the video: acquire the right visual evidence, use it to guide research, then allow the research to reshape what you inspect next.

For human-data work, this shifts the supervision unit toward the evidence-seeking process itself.

• Which interval contains the anchor?

• Which keyframe identifies the target clearly enough?

• What uncertainty remains after inspecting it?

• Which query follows from that uncertainty?

• Which source resolves it?

• Does the new evidence confirm the interpretation, or trigger another inspection of the video?

That work sits closer to research and adjudication than conventional labeling.

What would be valuable to build next

VideoRover-Bench finishes with human review of every surviving item for question and answer correctness and for genuine dependence on both video and external knowledge. The final benchmark contains 300 examples, balanced across short and long videos and easy, medium and hard research difficulty.

That human layer suggests several useful extensions.

One is a human-authored hard set designed independently of the generation pipeline, giving researchers a complementary test of whether the same capability profile survives a different source of task construction.

Another is explicit evidence-chain annotation: the decisive interval, identifying keyframe, useful query, supporting source and any necessary return to the video. That would allow an evaluator to score the research path alongside the final answer and test whether the process is grounded and reproducible.

A third is ambiguity adjudication. When multiple people, objects or moments could plausibly anchor a question, a reviewer can establish the defensible interpretation before the agent trajectory is scored.

Together, these extensions make evidence acquisition itself a quality object. The human role moves upstream into shaping, verifying and adjudicating the path a model takes through multimodal evidence.

That is the human-data opportunity VideoRover makes visible. Frontier multimodal systems increasingly need people who can evaluate how models find and connect evidence across sources.

VideoRover is an early, unusually concrete map of that transition.

We teach temporal evidence grounding as part of Data Gradient’s Frontier Multimodal Data series. If you’re building video agents or the data systems behind them, we’d love to hear what you’re seeing.