DeepSeek vs GPT API: Read-Only SQL Evaluation

DeepSeek vs GPT API for SQL: Compare Results, Not Query Strings

Two generated queries can look unrelated and return exactly the same answer. Two almost identical queries can disagree about revenue because one includes cancelled orders. That makes SQL a particularly poor workload for judging an API by how convincing its response looks. A useful DeepSeek versus GPT comparison starts with business questions, frozen data, and independently checked answers. The model supplies a candidate query; your evaluator decides whether its result satisfies the question. This article builds that evaluation boundary for a small, read-only analytics workload before either model can influence a production report.

The deliverable is an evidence package you can hand to an analytics owner: questions, reference queries, database fixtures, generated candidates, execution outcomes, and adjudicated failures. It is not a published contest result, and no live model comparison is claimed here. The examples show a reproducible method using a synthetic SQLite snapshot. An organization using PostgreSQL or another warehouse should repeat the evaluation in that dialect with equivalent isolation. A SQLite success does not establish compatibility with another database, its date arithmetic, or its permissions.

Write the answer contract before the prompt

Consider this question: “Which customers spent at least 5,000 cents on completed orders during October?” The answer contract must specify the timezone, inclusive start and exclusive end dates, treatment of refunds, and whether customer names or stable identifiers are required. A reference query should embody those choices, and an analyst should inspect its output. Without that contract, a disagreement may expose an ambiguous question rather than a model error. Keep ambiguous questions in a separate clarification set instead of letting an evaluator quietly choose whichever interpretation matches its preferred model.

For an initial suite, include filtering, grouping, joins, absence conditions, and ranking with ties. Give every question a permanent identifier and a declared output shape. An unordered customer list requires multiset comparison; a leaderboard requires ordered comparison and an explicit tie rule. Store monetary amounts as integer minor units in this teaching fixture, so rounding does not obscure a join mistake. Define how empty results differ from a single aggregate row containing NULL. These are product requirements, not details that a generic similarity score can recover afterward.

Separate result equivalence from textual equivalence. Aliases, join order, and subqueries can vary without changing the answer. Conversely, removing DISTINCT or using an inner join can silently alter row multiplicity. Compare complete rows with their duplicates preserved, then check column position against the declared contract. Do not sort each row's values or convert every value to a string. Those shortcuts can make a swapped identifier and amount look acceptable. A business owner may authorize a numeric tolerance, but it should be column-specific and agreed before model outputs are inspected.

Use several small worlds, not one convenient database

A wrong query can accidentally pass on one snapshot. If every customer has exactly one order, an incorrect counting query may look correct. Build a base fixture and several deliberately changed fixtures: a customer with two orders, an order at the date boundary, a cancelled high-value order, a customer without orders, and two customers sharing a display name. Keep the question and reference query fixed across these worlds. The model sees the schema and definitions, while the local evaluator checks whether its candidate remains correct under each relevant variation.

These fixtures are counterexamples to plausible mistakes, not synthetic evidence of model superiority. Record which business rule each fixture exercises. For a top-three request, add a tie at third place and require a deterministic secondary ordering when the product expects exactly three rows. For a missing-activity question, add NULL values that distinguish NOT IN from safer alternatives. Review the fixture design with someone who owns the real metric. An impressive aggregate pass rate is weak evidence if the suite never tests the cases that previously broke the team's reports.

Keep development questions separate from a final holdout set, and split by business template rather than wording. “October revenue by region” and “November revenue by region” are usually the same template. If one tunes the prompt using the former, the latter is not a demanding independent test. Record prompt revisions, schema versions, and snapshot hashes together. The operational ownership practices in the multi-model gateway governance guide are useful here because the comparison must remain reproducible after a model alias or business definition changes.

Generate candidates through a controlled API profile

The DeepSeek Chat Completions reference and OpenAI Chat reference describe their respective upstream request and response contracts. They do not establish that every optional upstream parameter is available through a gateway. For this experiment, use the shared chat endpoint with an exact model identifier verified in your 4SAPI account. Run the same script with DEEPSEEK_MODEL and GPT_MODEL values selected from that account, preserving the identifier and returned model field for each attempt. Do not infer identity from the assistant's prose.

The following standalone Python example makes one gateway request and writes a candidate record; it never connects to a database. Install no third-party package. Set FOUR_SAPI_KEY and LLM_MODEL before running it, using a 4SAPI-issued credential only. The Python DeepSeek integration guide covers the initial connection, while the 4SAPI integration walkthrough explains account, group, and model selection. Keep API credentials out of the later execution worker.

# generate_candidate.py: one paid call when you explicitly run it.
import json
import os
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen

model = os.environ["LLM_MODEL"]
prompt = """SQLite schema:
orders(customer_id INTEGER, total_cents INTEGER, status TEXT,
       ordered_at TEXT)
ordered_at uses UTC ISO dates. Each row is one order.
Return exactly one read-only SELECT statement, without Markdown.
Find customer_id and summed total_cents for completed orders from
2026-10-01 inclusive to 2026-11-01 exclusive, totals at least 5000.
Output those two columns in that order. Result order is irrelevant.
No other tables, functions with side effects, or external access."""
payload = {"model": model, "stream": False, "messages": [
    {"role": "user", "content": prompt}
]}
request = Request("https://4sapi.com/v1/chat/completions",
    data=json.dumps(payload).encode(), method="POST",
    headers={"Authorization": "Bearer " + os.environ["FOUR_SAPI_KEY"],
             "Content-Type": "application/json"})
try:
    with urlopen(request, timeout=60) as response:
        data = json.load(response)
except HTTPError as exc:
    raise SystemExit(f"HTTP {exc.code}; record as transport failure")
except (URLError, TimeoutError) as exc:
    raise SystemExit(f"Request failed: {type(exc).__name__}")
choice = data["choices"][0]
sql = choice.get("message", {}).get("content")
if choice.get("finish_reason") != "stop" or not isinstance(sql, str):
    raise SystemExit("Incomplete or unsupported response; do not execute")
if not sql.strip() or len(sql.encode()) > 2000:
    raise SystemExit("Candidate is empty or exceeds fixture limits")
Path("candidate.json").write_text(json.dumps({
    "question_id": "completed_october_spend", "requested_model": model,
    "returned_model": data.get("model"), "response_id": data.get("id"),
    "sql": sql.strip()
}, indent=2), encoding="utf-8")

Reject incomplete, empty, or explanatory output as a format failure in the first-pass comparison. Automatically extracting a plausible SELECT from surrounding prose changes the intervention being measured. If repair matters to your application, define a separate repaired-result track with a fixed retry budget and identical feedback rules for both routes. Likewise, a transport failure belongs in the denominator of end-to-end completion but is not a SQL semantic error. Keeping these labels separate lets the next engineering action address the actual failure.

Execute only inside the fixture boundary

Never run generated SQL against the application's normal database connection. Use a small, trusted synthetic snapshot containing no secrets, personal data, or production credentials. The Python sqlite3 reference documents read-only URI connections and connection callbacks; SQLite separately documents query_only, its authorization callback, and runtime limits. Combine those controls with an isolated operating-system process. A prefix check for the word SELECT does not provide an execution boundary.

The next standalone file is a deliberately restricted Unix teaching runner, requiring Python 3.11 or later. It expects a trusted, frozen fixture.sqlite containing only the small orders table described above. Its child process inherits no API-key environment, permits only selected reads and aggregate functions, and has CPU, address-space, statement-size, result-size, and wall-time limits. It denies unsupported syntax instead of widening permissions. For hostile inputs outside this controlled fixture exercise, place the worker in a separate container with no network and only the snapshot mounted read-only.

# evaluate_sql.py: local only; no model request is made.
import json
import os
from pathlib import Path
import sqlite3
import subprocess
import sys
import time

REFERENCE = """SELECT customer_id, SUM(total_cents) FROM orders
WHERE status = 'completed' AND ordered_at >= '2026-10-01'
AND ordered_at < '2026-11-01' GROUP BY customer_id
HAVING SUM(total_cents) >= 5000"""

def worker(snapshot, sql):
    import resource
    resource.setrlimit(resource.RLIMIT_CPU, (2, 2))
    resource.setrlimit(resource.RLIMIT_AS, (256 * 1024**2,) * 2)
    resource.setrlimit(resource.RLIMIT_FSIZE, (0, 0))
    if len(sql.encode()) > 2000:
        raise ValueError("SQL limit")
    con = sqlite3.connect(Path(snapshot).resolve().as_uri()
                          + "?mode=ro&immutable=1", uri=True)
    try:
        con.execute("PRAGMA query_only=ON")
        con.execute("PRAGMA trusted_schema=OFF")
        con.setlimit(sqlite3.SQLITE_LIMIT_LENGTH, 100_000)
        con.setlimit(sqlite3.SQLITE_LIMIT_SQL_LENGTH, 2000)
        con.setlimit(sqlite3.SQLITE_LIMIT_COLUMN, 20)
        con.setlimit(sqlite3.SQLITE_LIMIT_COMPOUND_SELECT, 4)
        columns = {"", "customer_id", "total_cents", "status", "ordered_at"}
        def authorize(action, arg1, arg2, database, trigger):
            if action == sqlite3.SQLITE_SELECT:
                return sqlite3.SQLITE_OK
            if (action == sqlite3.SQLITE_READ and database == "main"
                and arg1 == "orders" and arg2 in columns):
                return sqlite3.SQLITE_OK
            if (action == sqlite3.SQLITE_FUNCTION
                and (arg2 or "").lower() in {"sum", "count", "min", "max"}):
                return sqlite3.SQLITE_OK
            return sqlite3.SQLITE_DENY
        con.set_authorizer(authorize)
        deadline = time.monotonic() + 1.0
        con.set_progress_handler(lambda: int(time.monotonic() > deadline), 1000)
        rows = con.execute(sql).fetchmany(501)  # execute permits one statement
        if len(rows) > 500 or any(len(row) != 2 for row in rows):
            raise ValueError("Result contract exceeded")
        return rows
    finally:
        con.close()

def run(snapshot, sql):
    completed = subprocess.run(
        [sys.executable, "-I", str(Path(__file__).resolve()), "--worker", snapshot],
        input=json.dumps(sql), capture_output=True, text=True, timeout=3,
        env={"LANG": "C.UTF-8"}, close_fds=True, check=True)
    return json.loads(completed.stdout)

def bag(rows):
    from collections import Counter
    return Counter(tuple((type(value).__name__, value) for value in row)
                   for row in rows)

if __name__ == "__main__":
    if len(sys.argv) == 3 and sys.argv[1] == "--worker":
        print(json.dumps(worker(sys.argv[2], json.load(sys.stdin))))
    else:
        candidate = json.loads(Path("candidate.json").read_text())["sql"]
        snapshot = str(Path("fixture.sqlite").resolve())
        print("equivalent_on_this_fixture:",
              bag(run(snapshot, candidate)) == bag(run(snapshot, REFERENCE)))

This runner intentionally supports a narrower language than general SQLite. An otherwise correct query using a disallowed function is a policy rejection, which should appear separately from a wrong answer. Do not loosen the allowlist just to improve a favorite model's score. Expand the contract, review the added capabilities, and rerun both models if the workload genuinely needs them. The same distinction between structurally acceptable input and permitted application behavior appears in the DeepSeek strict tool-calling guide; schema compliance alone cannot authorize database operations.

Read the comparison as an investigation

For each model, count attempted questions, transport failures, invalid outputs, policy rejections, execution failures, and answers equivalent across all required fixtures. Retain per-question results so a strong aggregate cannot conceal systematic failure on absence queries. Inspect disagreements between models before discussing a winner. If both fail the same question, inspect the business definition and reference query too. The reference is an executable claim maintained by your team, not an infallible oracle simply because a human wrote it first.

Pay special attention to empty answers. A candidate that filters on an impossible status can match a genuinely empty reference result by accident. Add a companion fixture containing one qualifying row and confirm that the reference changes as expected. Then add a second nonqualifying row that differs in only one business condition. This pair establishes that the candidate responds to the intended condition rather than simply producing no rows. Label the original and companion fixtures as one question family so they do not inflate the number of independent tasks in the report.

Adjudication should preserve the original candidate, even when its mistake is obvious. Suppose the model groups by customer name and merges two different customer identifiers. Store the failing rows, the violated rule, and the smallest fixture that demonstrates the problem. A reviewer can explain the fix, but the repaired SQL must not replace the original in the first-pass record. Otherwise, the evaluation gradually becomes a measure of reviewer skill. Keep human-assisted completion as a separate workflow with its own time and review requirements.

Also distinguish a correct answer from a correct implementation outside the tested data. Passing every fixture supplies evidence within that suite; it is not a formal proof for every possible database. Expand fixtures when a new counterexample is found, version the suite, and rerun previously accepted routes. For sensitive reporting, retain analyst review until the organization has enough representative evidence to change that policy. This creates a useful feedback loop in which each discovered error strengthens the evaluator instead of disappearing into an informal prompt adjustment.

Run repeated attempts only under a declared policy. Repetition can reveal instability, but choosing the most favorable response afterward inflates apparent reliability. Report first-attempt outcomes and repeated-attempt consistency separately. Alternate the order in which the two routes receive requests to reduce time-of-day effects, and retain response identifiers for investigation. If latency becomes a deployment constraint, measure it in its own track using the first-visible-token comparison method; the SQL pass rate cannot tell you how long an analyst waited.

A release decision should name the accepted workload, not declare a universally superior family. For example, a team might approve one exact route for reviewed monthly aggregation questions while keeping ranking and missing-activity questions in a human queue. That is a proposed decision shape, not a result reported here. Once correctness gates are met, the DeepSeek versus GPT cost analysis can inform a separate operating-budget review. Keeping those decisions distinct prevents a cheap malformed answer from being counted as a successful business task.

Start with one clearly defined metric and a handful of counterexample fixtures. Create or inspect your account at 4SAPI, check the current API documentation, and select the two exact accessible model identifiers. Generate candidates without connecting the model process to a database, then evaluate locally under the fixed boundary. The useful first milestone is a traceable explanation of one disagreement. That artifact gives your team a sound basis for expanding the suite and deciding where either API can be trusted.

Primary references checked November 7, 2026

• DeepSeek Chat Completions API
• OpenAI Chat API reference
• Python sqlite3 documentation
• SQLite query_only
• SQLite authorization callback
• SQLite implementation limits