Gemini Audio Diarization for Support-Call Handoff QA

The escalation note says that the agent promised a replacement by Friday. In the recording, the customer asked for Friday delivery and the agent said that dispatch would need confirmation. A fluent summary has moved a deadline across the conversation and assigned it to the wrong person. The operational problem is not whether the transcript reads naturally. It is whether the next support agent can distinguish a customer request, an agent commitment, and an unresolved dependency before continuing the case.

The following is a hypothetical service-design case, not a customer account or a reported benchmark. A small support operation wants a draft handoff after a recorded call about a replacement accessory. The call has two speakers, a short hold, and one ambiguous exchange near the end. Gemini produces diarized transcription evidence; the application assembles a draft; a reviewer checks consequential statements. Success means the handoff preserves who said what and what remains unresolved, with playable evidence for each claim.

The handoff has a smaller job than the transcript

The handoff should contain the issue, the customer's requested outcome, any explicit agent commitments, unresolved questions, and the next assigned action. Each statement references one or more source turns. It should not reproduce the entire call or decide whether the customer deserves a particular remedy. The earlier Gemini transcription overview covers the speech-to-text topic. This design begins with the downstream risk that a recognizable voice label can still be assigned the wrong operational role.

Use neutral speaker identifiers until role attribution is verified. A label such as spk_1 identifies a voice cluster within the result; it is not an employee identifier, a customer identifier, or proof of authority. The first voice could be an automated greeting, a transferred specialist, or the customer speaking before the agent introduction. Map speaker labels to roles using the approved recording context and clear conversational evidence, then record who approved the mapping. Do not identify a person from their voice.

Keep statements at the level their evidence supports. “Customer requested a replacement by Friday” is different from “Agent committed to Friday delivery,” and both differ from “Warehouse confirmed Friday dispatch.” The hypothetical recording supports the first statement and leaves the second unconfirmed. The handoff should expose that unresolved boundary. A downstream agent can then check fulfillment rather than having to repair a promise that the earlier agent never made.

Choose an evidence-preserving transcription configuration

The current Gemini transcription guide documents verbatim transcription with speaker diarization and word-level timestamps. It also documents compatibility limits: smart transcription cannot be combined with those annotations, and custom vocabulary cannot be combined with diarization or word timestamps. For this evidence task, use the documented verbatim configuration. Cleanup can happen later in a separate display layer; it should not replace the evidence representation used to establish a commitment or a refusal.

The same guide currently limits annotated processing to thirty minutes and describes attribution for three or more speakers as experimental. Those limits matter when a call transfers or brings in a supervisor. Route calls outside the pilot's supported conditions to review instead of silently treating them as ordinary two-speaker cases. If long calls must be divided, preserve original offsets and treat every chunk's speaker labels as local until continuity is established. Reusing spk_1 across chunks does not establish that the voice stayed the same.

Use an approved recording with a known duration and preserve its original identifier. Upload it through the Files API, check readiness, and pass the returned reference into analysis. The example below starts with that prepared reference. Its use of store false opts out of stored interaction state; it does not delete the uploaded file or establish a complete retention policy. Asset access and deletion should follow the application's existing handling rules for support recordings.

This is post-call review. It can wait for a completed response and should not interrupt the live conversation. Real-time voice-agent behavior has different timing and turn-taking requirements, discussed separately in the Gemini Live voice-agent article. Keeping the first pilot asynchronous lets the team inspect role errors and missing evidence before introducing any dependency between model output and what an agent says to a customer.

Retrieve annotations without flattening away their meaning

The native reference below calls Gemini Interactions with a Google credential. The model name is selected from the environment and must be a transcription model supporting the documented configuration. The current REST transcription examples use a file input reference, while annotations arrive inside final content. Google's Interactions overview identifies this as the recommended interface for new projects; legacy generateContent remains supported. This request is separate from any gateway endpoint or key.

# Native Google transcription request. Requires an active uploaded file.
import json
import os
import urllib.request
from decimal import Decimal

source_duration = Decimal(os.environ["CALL_DURATION_SECONDS"])
if not source_duration.is_finite() or source_duration <= 0:
    raise ValueError("Invalid call duration")
if source_duration > 1800:
    raise ValueError("This annotated pilot accepts calls up to 30 minutes")

body = {
    "model": os.environ["GEMINI_TRANSCRIBE_MODEL"],
    "store": False,
    "input": [{"type": "file", "file_uri": os.environ["GEMINI_AUDIO_FILE_URI"]}],
    "generation_config": {"transcription_config": {"mode": {
        "type": "verbatim", "diarization_mode": "speaker",
        "timestamp_granularities": ["word"]
    }}}
}
request = urllib.request.Request(
    "https://generativelanguage.googleapis.com/v1beta/interactions",
    data=json.dumps(body).encode(), method="POST",
    headers={"Content-Type": "application/json",
             "x-goog-api-key": os.environ["GEMINI_API_KEY"]}
)
with urllib.request.urlopen(request, timeout=180) as response:
    interaction = json.load(response)
if interaction.get("status") != "completed":
    raise RuntimeError("Incomplete transcription")

def seconds(raw):
    if not isinstance(raw, str) or not raw.endswith("s"):
        raise ValueError("Expected a duration string ending in s")
    value = Decimal(raw[:-1])
    if not value.is_finite() or value < 0:
        raise ValueError("Invalid annotation offset")
    return value

words = []
for step in interaction.get("steps", []):
    if step.get("type") != "model_output":
        continue
    for content in step.get("content", []):
        for annotation in content.get("annotations", []):
            if annotation.get("type") != "word_info":
                continue
            start = seconds(annotation.get("start_offset"))
            end = seconds(annotation.get("end_offset"))
            if not start <= end <= source_duration:
                raise ValueError("Annotation outside source recording")
            words.append({
                "word_id": f"w{len(words):06d}",
                "speaker": annotation.get("speaker"),
                "start_s": str(start), "end_s": str(end),
                "text": annotation.get("text", "")
            })
if not words:
    raise ValueError("No word annotations; handoff evidence is unavailable")
labels = {w["speaker"] for w in words if w["speaker"]}
needs_review = any(not w["speaker"] or not w["text"] for w in words)
print(json.dumps({"interaction_id": interaction.get("id"),
                  "role_mapping_status": "unverified",
                  "review_required": True,
                  "annotation_issues": needs_review or len(labels) != 2,
                  "words": words}, indent=2))

The code intentionally fails if annotations are missing, even when readable transcript text exists. A text-only fallback might be adequate for search, but it cannot satisfy this handoff's speaker-and-time evidence contract. The parser retains duration values as decimal strings; Python's decimal documentation explains the exact decimal arithmetic used for bounds checks. It does not sort or merge annotations automatically, so the application can inspect the returned order and diagnose overlaps before constructing turns.

A successful response leaves role_mapping_status unverified. That is the correct starting state. Group adjacent words into turns only under an explicit rule, such as the same speaker with a bounded gap, and preserve the constituent word identifiers. A hold, overlap, or interruption can make a simple grouping rule misleading. Retain raw annotations alongside normalized turns so a reviewer can correct a boundary without losing the original model output or the reason the grouping occurred.

Design the review around three risky statements

For the hypothetical accessory call, concentrate the review on requested delivery, promised action, and the remaining dependency. The customer's deadline belongs in requested_outcome. The agent's promise to contact dispatch belongs in commitments. Dispatch confirmation belongs in unresolved_items until the call explicitly establishes it. This separation gives the next agent useful context without compressing every future-looking sentence into a single “resolution” paragraph that blurs who owns which action.

The handoff generator can receive normalized turns with immutable identifiers and verified role labels. Ask it to propose claims with a claim type, actor, text, and evidence-turn identifiers. The Gemini audio understanding guide covers broader reasoning over audio; for this design, use that capability or a text-capable Gemini model only after defining the evidence boundary. Structured output can stabilize the draft shape, as covered in the existing JSON extraction guide, but does not establish attribution truth.

Make the generator preserve modality in ordinary language. “Will check” differs from “has checked,” and “could arrive” differs from “will arrive.” Require direct supporting words for an operational commitment, while allowing a separate paraphrase for readability. If a commitment depends on another team's confirmation, preserve that dependency in the draft. The reviewer should see both the proposed statement and the evidence turn with surrounding context, rather than an isolated quote that may omit a qualification.

# Standalone local QA fixture. These turns and claims are hypothetical.
turns = {
    "t01": {"speaker": "spk_1", "text": "I need it by Friday."},
    "t02": {"speaker": "spk_2", "text": "I will check with dispatch."},
    "t03": {"speaker": "spk_2", "text": "The date is not confirmed yet."}
}
verified_roles = {"spk_1": "customer", "spk_2": "agent"}

def check_claim(claim):
    evidence = [turns[i] for i in claim["evidence_turn_ids"] if i in turns]
    if len(evidence) != len(claim["evidence_turn_ids"]) or not evidence:
        return "review: missing evidence"
    if claim["kind"] == "agent_commitment":
        if any(verified_roles.get(t["speaker"]) != "agent" for t in evidence):
            return "review: commitment attributed to a different role"
    quote = claim.get("supporting_quote", "")
    if not quote or not any(quote in t["text"] for t in evidence):
        return "review: supporting quote not found"
    return "review: attribution checks passed; verify meaning in audio"

bad_claim = {
    "kind": "agent_commitment", "text": "Delivery promised by Friday",
    "evidence_turn_ids": ["t01"], "supporting_quote": "I need it by Friday."
}
assert check_claim(bad_claim) == "review: commitment attributed to a different role"
print(check_claim(bad_claim))

The deterministic check catches one specific category error: an agent commitment backed only by a customer's turn. It cannot decide whether “I will check with dispatch” supports a delivery guarantee, because word matching is not semantic verification. Its passing outcome therefore still requires review. This layered treatment follows the distinction between output plausibility and evidence discussed in Gemini multimodal validation, while making the failure condition operationally concrete.

Handle uncertainty where the next agent will see it

Put unresolved attribution beside the affected statement, not in an obscure metadata panel. If a short acknowledgment overlaps the customer's request, label the interpretation as unresolved and link the relevant interval. Do not transform an uncertain “yes” into approval of everything in the preceding turn. The next agent needs a clear reason to listen again. A general confidence score does not explain whether the uncertainty concerns the speaker, the words, or the business meaning.

Maintain separate records for source evidence, proposed handoff, and reviewer corrections. If a reviewer changes the role mapping, regenerate or revalidate claims that depend on that mapping. If only a typo is corrected, preserve the original annotation and record the correction without rewriting earlier request logs. The session event-sourcing article provides related recovery patterns. Here, the practical benefit is that a disputed promise can be traced to its recording, draft, and review decision.

Recording access should follow the user's existing support permissions. A handoff reader may need a short authorized playback interval without access to unrelated cases. Keep credentials out of transcripts, application logs, and shareable draft exports. The API data-access article offers broader implementation context. For the pilot, the useful operational rule is simple: create the smallest review artifact that still lets the assigned reviewer verify the statement, and retain it through the established case lifecycle.

Evaluate the handoff, not just recognition quality

Build a small, consented or synthetic evaluation set around known operational distinctions. Include a customer deadline with no agent promise, a qualified promise, a correction of an earlier statement, a hold followed by the same agent, and a transfer to another agent. Add a recording where one speaker quotes the other: “You said Friday” should not become a fresh Friday commitment. These fixtures probe role continuity and commitment meaning in ways a transcript readability score cannot.

Measure false commitments, missing dependencies, incorrect actor assignments, and reviewer time to locate evidence. Track transcription errors separately from handoff errors. A transcript may contain a minor product-name error while preserving the commitment correctly, or recognize every word while assigning the decisive sentence to the wrong speaker. The distinction between summarization and generation discussed in this LLM summarization guide matters because a handoff must preserve constraints rather than improve the story by filling gaps.

Choose acceptance criteria before looking at pilot results. For example, require human approval of every proposed commitment and reject drafts with unresolved speaker mapping. These are design choices for this hypothetical operation, not universal performance thresholds. When a failure occurs, classify whether it began in recognition, diarization, role mapping, turn grouping, or claim generation. That classification directs the next experiment and prevents a prompt revision from becoming the default response to a bad recording or a missing annotation.

Start with one call and one reviewable handoff

The first useful deployment is a draft beside the recording, with no automatic promise, refund, or case closure. Let a support reviewer approve the role mapping and inspect the few statements that affect the next action. Preserve the difference between a completed transcript and an accepted handoff. Once that distinction is visible in the product, the team can improve turnaround time without confusing faster drafting with reliable operational decisions.

To try the workflow with 4SAPI, open 4SAPI, inspect your available models, and verify the native transcription or audio route in the current documentation. The Gemini endpoint guide explains the integration distinction; availability of a Gemini model name alone does not establish support for word annotations or diarization settings. Run one approved call through a confirmed route, inspect its actual annotations, and make a reviewer-supported handoff your first acceptance artifact.