Gemini Video API: Timestamps and Reviewable Clip Evidence

A timestamp is useful only when another person can open the same recording, jump to the indicated moment, and see what supports the claim. A video summary saying “the operator checks the seal at 02:14” leaves several questions unanswered: which file, which seal, whether the action was visible, and whether 02:14 refers to the original recording or an extracted clip. This workshop builds a small evidence package around those questions, using Gemini video understanding to propose moments and deterministic checks to reject unusable references.

Use a short, non-sensitive demonstration recording for the first run. The working example is a hypothetical equipment inspection video containing three actions: a label check, a seal check, and a final indicator check. No result below represents an observed model test. The deliverable is an evidence record for each requested action, including a source identifier, a bounded interval, an observation, and a review decision. General media setup is covered in the existing Gemini multimodal integration guide; here the work begins where a successful upload ends.

Prepare the source before asking for timestamps

Keep the original media file unchanged and assign an application identifier before uploading it. Record its duration from a media inspection tool, its SHA-256 digest, the upload identifier, and the requested processing interval. Python's hashlib documentation describes the digest operations needed for this manifest. A digest links a result to particular bytes; it does not establish that the recording itself is authentic or that an observation is correct. That narrower claim is enough to prevent an edited replacement from silently inheriting an earlier review.

Use the Gemini Files API to upload the recording and inspect its state before analysis. Continue only after the file is ready; a returned URI alone is insufficient evidence that processing succeeded. Preserve the local source independently of the provider upload because an application may need to replay a review after the uploaded object becomes unavailable. The reference request below assumes an already active file URI, deliberately separating upload readiness from the evidence task.

Choose one clock for the whole package: seconds from the start of the original file. Represent intervals as half-open ranges, meaning the start is included and the end marks the boundary after the observation. If a local export starts at original second 120, its player may display zero while the evidence manifest still records 120. Store that mapping explicitly. Never infer the clock from a timestamp's size, because a ten-second value could describe either the beginning of a recording or ten seconds into a later excerpt.

Define what would count as visible evidence

Write the inspection question before writing the prompt. “Was the product checked?” is too broad for useful review. “Find the interval where the operator rotates the package so the printed lot label faces the camera” identifies a visible action without asserting that the printed lot is legible. Divide compound claims into separate observations. A visible rotation, readable label text, and a successful quality decision require different evidence, even when they occur close together.

Give each requested action a stable key, such as seal_check. Allow an empty result when the action cannot be located, and distinguish visual evidence from narration. A speaker saying “the seal is intact” supports a statement about speech; it does not establish that the seal is visible or intact. This distinction applies the general principles discussed in multimodal output validation to a very specific failure: accepting plausible narration as a visual inspection record.

Keep the first response small. Request only the target action, a short interval, the visible observation, and an uncertainty note. Do not ask for an entire transcript, a chapter index, and operational recommendations in the same call. A narrower response makes it easier to discover whether the timestamp is wrong or the observation is unsupported. Google documents timestamp questions and clipping controls in its video understanding guide; the acceptance rules in this workshop remain application decisions.

Make one bounded native request

This example uses the current native Interactions API. Google's Interactions overview recommends that interface for new development while describing generateContent as supported legacy. It is an upstream reference implementation, using a Google credential against Google's domain. It does not establish that a gateway supports the same media controls. Set GEMINI_MODEL to an available model that supports video understanding and the chosen structured output configuration, then use an approved active file URI.

The request uses static processing because the live video documentation restricts explicit clipping and frame-rate controls to that mode. Current offset examples use duration strings such as "120s", rather than bare integers. A five-frames-per-second sample is a workshop choice for a short inspection segment, not a precision guarantee. Increasing sampling can expose a brief movement, but cannot recover details absent from the source or ensure that the model describes every sampled frame correctly.

# Native Google Interactions API; one paid request when executed.
import json
import math
import os
import urllib.request

start = float(os.environ.get("CLIP_START_SECONDS", "0"))
end = float(os.environ["CLIP_END_SECONDS"])
duration = float(os.environ["SOURCE_DURATION_SECONDS"])
if not all(math.isfinite(x) for x in (start, end, duration)):
    raise ValueError("Times must be finite")
if not 0 <= start < end <= duration:
    raise ValueError("Invalid source interval")

schema = {
    "type": "object",
    "properties": {
        "found": {"type": "boolean"},
        "start_s": {"type": ["number", "null"]},
        "end_s": {"type": ["number", "null"]},
        "observation": {"type": "string"},
        "uncertainty": {"type": "string"}
    },
    "required": ["found", "start_s", "end_s", "observation", "uncertainty"],
    "additionalProperties": False
}
payload = {
    "model": os.environ["GEMINI_MODEL"],
    "store": False,
    "input": [
        {"type": "video", "uri": os.environ["GEMINI_VIDEO_FILE_URI"],
         "mime_type": "video/mp4",
         "processing": {"type": "static", "start_offset": f"{start}s",
                        "end_offset": f"{end}s", "fps": 5}},
        {"type": "text", "text": (
            "Locate the visible seal-check action. Use seconds from the start "
            "of the ORIGINAL full recording, not from the selected interval. "
            "Describe only visible evidence. If absent or uncertain, set found "
            "to false and both times to null. Do not treat narration as proof."
        )}
    ],
    "response_format": {"type": "text", "mime_type": "application/json",
                        "schema": schema}
}
request = urllib.request.Request(
    "https://generativelanguage.googleapis.com/v1beta/interactions",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json",
             "x-goog-api-key": os.environ["GEMINI_API_KEY"]},
    method="POST"
)
with urllib.request.urlopen(request, timeout=120) as response:
    interaction = json.load(response)
if interaction.get("status") != "completed":
    raise RuntimeError("Interaction did not complete")
texts = [part["text"] for step in interaction.get("steps", [])
         if step.get("type") == "model_output"
         for part in step.get("content", []) if part.get("type") == "text"]
if len(texts) != 1:
    raise ValueError("Expected one final JSON text part")
result = json.loads(texts[0])
if type(result.get("found")) is not bool:
    raise ValueError("Missing or invalid found flag")
for name in ("observation", "uncertainty"):
    if not isinstance(result.get(name), str):
        raise ValueError(f"Invalid {name}")
a, b = result.get("start_s"), result.get("end_s")
if result["found"]:
    if any(type(v) not in (int, float) or not math.isfinite(v) for v in (a, b)):
        raise ValueError("Invalid evidence times")
    if not start <= a < b <= end or not result["observation"].strip():
        raise ValueError("Evidence falls outside requested interval")
elif a is not None or b is not None:
    raise ValueError("Absent evidence must not have timestamps")
print(json.dumps({"interaction_id": interaction.get("id"),
                  "review_status": "pending", "evidence": result}, indent=2))

The script checks completion before reading final output, and reads only the model output text rather than intermediate steps. Its timeout bounds the client wait; a timeout does not prove that upstream computation never occurred. Failed requests should remain distinct from an empty but completed search. The structured output guide explains the response-format envelope and supported schema subset. The local checks enforce interval rules that a JSON shape alone cannot express adequately for this workflow.

Do not place the source identifier or checksum under model control. Attach those values after parsing, from the manifest associated with the request. Otherwise an answer could point to the right time in the wrong asset while still passing type validation. Likewise, keep the requested action key outside generated text. The model's job is to propose an observation; the application already knows which source and question produced that proposal.

Try to disprove the proposed interval

Open the original recording at the returned start time and watch beyond the returned end. Review a little context on both sides, while retaining the narrower claimed interval in the record. For the hypothetical seal check, ask whether the seal appears, whether the operator's action is visible, and whether any relevant detail is obstructed. A correct timestamp pointing to an obstructed view is still insufficient support for a visual claim. Record that as an evidence problem rather than silently moving the interval.

Perform a second, tightly scoped Gemini request only when it answers a concrete uncertainty. For example, re-inspect the proposed window and ask whether the hand obscures the seal throughout the action. Keep that response as another observation with its own request identifier. Agreement between two calls can help prioritize review, but both calls see the same imperfect source and may repeat the same error. The final acceptance decision should reflect what a reviewer can actually verify in the clip.

Test the clock convention deliberately. Create a fixture where the only target action occurs well after the recording begins, then process a window around it. If the response returns a plausible clip-relative time, the bounds check should reject it instead of adding the clip offset automatically. An automatic correction would conceal an ambiguous contract. Clarify the prompt or normalize a known transport convention in an explicitly versioned adapter, then rerun the fixture.

For repeated actions, declare the selection rule before scoring. If the question asks for the first inspection but the model returns the clearest inspection, both timestamp and observation may look convincing while the task is wrong. Include examples with repeated motions and a near miss, such as touching the package without inspecting it. These fixtures measure the evidence contract more directly than checking whether a generic summary sounds accurate.

When the first window is empty, decide whether the search was exhaustive enough to support absence. A negative result from seconds 120 through 150 says nothing about an action at second 90. Store the inspected ranges with the result and make the interface distinguish “not found in this range” from “not present in the recording.” Expanding the search is a new request with a larger evidence scope, not a correction to an already completed result.

Use a stable fixture to compare sampling settings. Hold the source, target action, and requested window constant while changing only the frame rate. Review whether the candidate interval became more useful and whether the observation changed. This produces an interpretable experiment without implying that more frames always improve the final decision.

Package the clip so the reviewer can use it

A review card needs a playable source, its original timeline range, the proposed observation, and the reviewer outcome. It also needs enough provenance to recover the request: model identifier, processing mode, frame sampling, prompt version, and schema version. Persist proposed and approved intervals separately. The patterns in session event sourcing are relevant here because a correction should append a decision rather than erase the model's earlier claim.

If you export a shorter media file, retain both clocks and verify the exported duration. The FFmpeg documentation distinguishes seeking and transcoding behavior, so avoid promising frame-exact boundaries from a convenient stream-copy command. For review, a slightly wider context clip can be appropriate, provided its manifest states the export range and highlights the actual claimed range. A screenshot can supplement the clip, but a single still cannot establish the order of a movement.

Give unavailable sources a visible state. A reviewer should never confuse “clip cannot be loaded” with “claim rejected” or “action absent.” The same separation between retrieval success and answer quality appears in the Gemini URL-context retrieval investigation. Here, retrieval means loading the exact video bytes or their approved derivative. Missing media is an infrastructure issue that prevents a decision; it is not a negative inspection result.

Measure the workflow with reviewable outcomes: accepted intervals, unsupported claims, wrong occurrence selections, and clock errors. Track reviewer adjustment distance separately from whether the observation was true. A system can locate the correct action approximately yet require expensive manual trimming. Conversely, precise timestamps can accompany incorrect descriptions. Keeping these measurements separate helps decide whether to improve temporal prompting, source recording quality, or the wording of the inspection criterion.

Take the first small test into your application

Connect the evidence card to your media application only after the fixtures expose no unresolved clock ambiguity. The broader multimodal production architecture discussion provides adjacent ideas for asset handling, but the acceptance boundary here remains concrete: a proposed observation with a reproducible source interval. Never equate a successful API response with an approved inspection, and avoid expanding a pilot into unattended operational decisions before understanding its failures.

For a 4SAPI evaluation, open 4SAPI, inspect the models available to your account, and confirm the supported media route in the current API documentation. The existing native versus aggregated Gemini integration guide explains why endpoint choice deserves an explicit decision. Start with one approved demonstration clip and one action. Confirm video input, clipping, structured response handling, and credential pairing for your selected route before adapting the native request.

Primary references

• Google: video understanding
• Google: Files API
• Google: Interactions API
• Google: structured outputs
• Python: hashlib
• FFmpeg documentation