API Documentation

The complete, client-facing reference for the Ultron Live backend. It documents both API versions and every REST and WebSocket surface, with request and response shapes.

The backend exposes two parallel workflows. Pick one version per client and use it consistently.

All protected endpoints accept any one of the following on every request:

Priority order: x-api-key → Authorization: Bearer ulk_... → Authorization: Bearer <jwt>

Success envelope (REST)

{ "success": true, "message": "…", "data": { /* optional */ } }

Some high-frequency endpoints (e.g. /vision) return fields at the top level rather than under data.

Error envelope (REST)

{ "success": false, "message": "Human-readable error", "error": "optional detail" }

Credits

Metered endpoints deduct credits as inputTokens / 3000 + outputTokens / 1000 (a metered call requires a balance of at least 1 credit). New accounts start with 200 credits; Pro grants 5,000. A 402 is returned when the balance is below the minimum.

Available under both /api/v1/auth/* and /api/v2/auth/* with identical behavior. Flow: sign up or log in to receive a 6-digit OTP, then verify it to receive the JWT, apiKey, and profile.

POST /auth/signup — no auth

Creates a new account and emails an OTP.

{ "email": "user@example.com", "firstName": "Ada", "lastName": "Lovelace", "dateOfBirth": "1990-12-10" }

201 → account created, OTP sent. Errors: 400 invalid email/dob · 409 email already registered (use login).

POST /auth/login — no auth

Sends an OTP to an existing user.

{ "email": "user@example.com" }

200 → OTP sent. Errors: 404 account not found (sign up first).

POST /auth/verify-otp — no auth

Verifies the OTP and logs in. The optional timezone is an IANA string.

{ "email": "user@example.com", "otp": "847291", "timezone": "America/New_York" }

{
  "success": true,
  "message": "Login successful",
  "data": {
    "token": "<jwt — valid 7 days>",
    "user": {
      "id": "…", "email": "…", "firstName": "Ada", "lastName": "Lovelace",
      "apiKey": "ulk_…", "credits": 200, "isVerified": true,
      "timezone": "America/New_York", "plan": "free", "createdAt": "…"
    }
  }
}

GET /auth/me — auth

Returns the live profile + current credits. Call on page load for an accurate balance.

POST /auth/session-token — API key only (v2)

Call this from your backend with the API key (x-api-key). It is rejected (403) for JWT / session-token auth, so a leaked token can't mint more.

// request (all optional)
{ "endUserId": "user_789", "ttlSeconds": 600 }

// response
{ "success": true, "data": { "token": "<jwt>", "jti": "…", "expiresIn": 600, "expiresAt": "…Z" } }

The token bills the owning account (shared credit pool) and is session-scoped: it can run inference and read templates/tools, but returns 403 on account-management endpoints (billing, profile, MCP writes, template writes, minting). It works for REST and as the WebSocket token. Default lifetime 10 min, max 1 hr. The jti identifies the token for revocation.

Built-in abuse guards (automatic, no client config):

POST /auth/session-token/revoke — API key only (v2)

Revoke session tokens for your account. Send exactly one of:

{ "jti": "…" }             // kill one leaked token
{ "endUserId": "user_789" }  // block an end-user (rejects tokens + refuses new mints)
{ "all": true }            // "log everyone out": reject all tokens issued before now

Takes effect immediately across REST and WebSocket. A revoked token returns 401; with a tokenProvider the SDK simply mints a fresh one, so revoking a leakedtoken doesn't disrupt legitimate users. Only session tokens are affected — your API key and full logins keep working. Companion endpoints: POST /auth/session-token/unblock ({ endUserId }) and GET /auth/session-token/revocations.

POST /auth/timezone — auth

Validates and saves a real IANA zone. Safe to call on every load.

{ "timezone": "America/New_York" }

PUT /auth/profile — auth

Send only fields to change. dateOfBirth: null clears it.

{ "firstName": "Ada", "lastName": "Lovelace", "dateOfBirth": "1990-12-10" }

GET /auth/transactions — auth

Paginated billing history (limit ≤ 100).

{ "data": { "transactions": [ { "amount", "currency", "status", "description", "paidAt", "createdAt" } ],
            "pagination": { "page", "limit", "total", "totalPages" } } }

GET /auth/subscription — auth

{ "data": { "plan": "free", "subscriptionStatus": null, "subscriptionCurrentPeriodEnd": null,
            "cancelAtPeriodEnd": false, "planStartedAt": null } }

POST /auth/subscription/cancel — auth

Cancels the Stripe subscription at period end. Errors: 400 no active subscription.

The REST request/response shape is identical across versions. Under v2 the same endpoints additionally accept the dynamic-assessment config. For high-throughput live use, prefer the WebRTC transport.

POST /vision/session/start — auth

Creates a session that links frames and enables summarization + chat. No body.

{ "success": true, "sessionId": "…" }

POST /vision — auth (core, metered)

One frame in → AI commentary out. Call in a loop (~every 1.5–2s).

{
  "image": "data:image/jpeg;base64,/9j/4AAQ…",
  "timestamp": 1711948800000,
  "model": "gemini-2.5-flash",
  "history": ["Player picked up a health pack.", "Moving to objective."],
  "sessionId": "…"
}

{ "success": true, "receivedAt": "…", "response": "Player secured the flag.",
  "latency": 342, "model": "gemini-2.5-flash", "provider": "gemini",
  "usage": { "inputTokens": 1024, "outputTokens": 18 },
  "creditsUsed": 0.36, "creditsRemaining": 199.64 }

Errors: 400 no image · 401 auth · 402 insufficient credits · 500 upstream.

POST /vision/chat — auth (metered)

Chat over session summaries + recent frame commentary + up to 2 live frames.

{ "sessionId": "…", "message": "What was I working on?",
  "frames": ["data:image/jpeg;base64,…"], "enableAudio": true, "voiceName": "Zephyr" }

The recommended real-time transport: a persistent peer connection with three data channels (frames, chat, control). Much lower latency than per-frame REST.

Connection flow

1. GET /api/{v}/webrtc/config            → ICE servers
2. open WS  /ws/{v}                       → signaling
3. send { type:"auth", apiKey|token }     → wait for auth:ok
4. create RTCPeerConnection(iceServers)
5. create data channels: frames, chat, control
6. createOffer → send {type:"offer",sdp}; receive {type:"answer",sdp}
7. exchange ICE candidates ({type:"ice", candidate})
8. channels open → startSession → stream frames

GET /webrtc/config — auth

{ "success": true,
  "iceServers": [ { "urls": "stun:stun.l.google.com:19302" }, … ],
  "wsUrl": "/ws" }

Signaling messages

→ { "type": "auth", "apiKey": "ulk_…" }          // first msg, within 10s
← { "type": "auth:ok", "userId": "…", "email": "…" }

→ { "type": "offer", "sdp": "…" }
← { "type": "answer", "sdp": "…" }

→ { "type": "ice", "candidate": { "candidate": "…", "sdpMid": "…", "sdpMLineIndex": 0 } }
← { "type": "ice", "candidate": { … } }

Data channels

frames — send raw JPEG ArrayBuffer (preferred) or JSON. For v2, attach ordering metadata and capture faster (~5–10 fps):

→ { "base64": "…", "seq": 42, "captureTs": 1711948800000, "model": "gemini-2.5-flash" }
← { "type": "frame:result", "response": "…", "seq": 42, "golden": true,
    "usage": { "inputTokens": 800, "outputTokens": 15 },
    "creditsUsed": 0.28, "creditsRemaining": 199.72, "sessionId": "…" }
← { "type": "flow:result", "response": "…higher-level narrative…", "seqs": [33,…,42] }  // v2 only
← { "type": "frame:error", "error": "Insufficient credits", "credits": 0.5 }

chat

→ { "action": "chat", "message": "What was I working on?", "sessionId": "…",
    "enableAudio": true, "voiceName": "Zephyr" }
← { "type": "chat:result", "response": "…", "creditsRemaining": 199.40, "latency": 891,
    "audio": "…", "audioMimeType": "audio/mpeg" }

control

// v1: simple session
→ { "action": "startSession" }
← { "action": "sessionStarted", "sessionId": "…" }

// v2: golden + assessment config (all optional)
→ { "action": "startSession", "config": { "golden": {…}, "outputType": "json", "templateId": "…" } }
← { "action": "sessionStarted", "sessionId": "…", "ingestFps": 8, "golden": { "enabled": true } }

→ { "action": "stopSession" }     ← { "action": "sessionStopped" }
→ { "action": "ping" }            ← { "action": "pong", "ts": … }

A relay that streams two-way audio between the user's mic and OpenAI Realtime or Gemini Live, normalized to the OpenAI event format so the client uses one protocol regardless of provider.

Lifecycle

open → send auth (inside onopen) → auth:ok → relay:ready → stream audio

Wait for relay:ready before sending audio. Do not send on auth:ok.

Messages

// 1) auth (first message, inside onopen). provider: "openai" (default) | "gemini"
→ { "type": "auth", "token": "Bearer <jwt>", "provider": "gemini" }
← { "type": "auth:ok", "userId": "…", "email": "…", "provider": "gemini" }
← { "type": "relay:error", "error": "Invalid API key" }   // then closes with code 4003

// 2) provider ready
← { "type": "relay:ready", "provider": "gemini" }

// 3) stream mic audio (PCM16 mono, base64)
→ { "type": "input_audio_buffer.append", "audio": "<base64 pcm16>" }

// 4) assistant audio + events (OpenAI-normalized)
← { "type": "response.audio.delta", "delta": "<base64 pcm16 24kHz>" }
← { "type": "input_audio_buffer.speech_started" }   // barge-in: stop local playback
← { "type": "relay:credits_exhausted" }             // then closes with code 4004
← { "type": "relay:upstream_closed" }

Audio format (relay does NOT resample)

Server VAD is on — no manual silence detection needed. Close codes: 4003 auth failed, 4004 credits exhausted.

Available under both versions.

POST /stripe/create-session — auth

Starts a Stripe Checkout for a plan and returns the hosted checkout URL.

{ "planID": "price_1NXkAb…" }   // a plan priceID, or a plan _id

{ "success": true, "message": "Checkout session created",
  "data": { "url": "https://checkout.stripe.com/c/pay/…" } }

Redirect the user to data.url. On return, refresh state with GET /auth/subscription and GET /auth/me. Entitlement activation is handled server-side via Stripe webhooks. Errors: 404 plan not found.

One config-driven vision engine for assessments (PASS/FAIL checklists, structured JSON verdicts, document review). Prefix /api/v2. Auth required on all.

The engine runs behind three transports that share one config:

Shared config fields: outputType (commentary|text|json), prompt, promptMode (append|replace), responseSchema (required for json), templateId, fields, model.

Templates (presets)

GET    /templates           list
GET    /templates/:id       get
POST   /templates           create  (admin)
PUT    /templates/:id       update  (admin)
DELETE /templates/:id       delete  (admin)

Upload + poll

POST /api/v2/assess        multipart/form-data, async → 202 { jobId }
GET  /api/v2/assess/:jobId  poll → { status: "processing" } then
                                   { status: "done", assessment, outputType, usage, creditsUsed }
GET  /api/v2/assessments    list the caller's assessment jobs/sessions

/assess form fields: media (1–30 files: a video, frames, or documents; ≤18 MB each), id_image (optional), the shared config fields, plus the optional per-run extraChecks / skipChecks (JSON). outputType defaults to text. 413 if a file exceeds 18 MB.

curl -X POST https://live-api.ultronai.me/api/v2/assess \
  -H "x-api-key: ulk_xxx" \
  -F "outputType=json" -F "templateId=665a…d0e" \
  -F 'fields={"collectionType":"Oral swab"}' \
  -F "media=@frame1.jpg" -F "media=@frame2.jpg" -F "id_image=@id.jpg"

Verdict & confidence (human-in-the-loop)

For outputType: json, the engine guarantees a calibrated verdict and adds the fields verdict, confidence, and needs_review to your schema. An element that cannot be directly verified is capped low and marked UNCERTAIN (absence of evidence is never a confident PASS/FAIL). When confidence is below the template's confidenceThreshold (default 70), the verdict is forced to UNCERTAIN and needs_review: true, so a human is engaged automatically.

{
  "verdict": "UNCERTAIN",          // PASS | FAIL | UNCERTAIN
  "confidence": 38,                // 0–100, the lowest per-check confidence
  "needs_review": true,            // confidence < threshold or genuinely uncertain
  "original_verdict": "FAIL",      // model's lean before the policy (only if overridden)
  "reasoning": "...",
  "uncertainty_factors": ["sealing step not shown"],
  "checks": [
    { "name": "Proper Collection", "result": "UNCERTAIN", "confidence": 38, "observed": "vial sealing off-frame" }
  ]
}

Add / skip questions per run

Adjust a template's questions for one run without editing it: append extraChecks ([{ title, instruction }]) or drop existing ones with skipChecks (1-based index or title). Both also work on a live startSession config. To change the questions permanently, edit the template's checks via PUT /templates/:id.

-F 'extraChecks=[{"title":"Gloves worn","instruction":"Were gloves worn the whole time?"}]' \
-F 'skipChecks=[2,"Identity Match"]'

Off-camera collections

collectionMode: "off_camera" (per request or as the template default) — for privately-collected types (urine, vaginal, rectal) where only the post-collection sealing/packaging is on camera. It auto-skips every check marked onCameraOnly: true and tells the model not to penalize the unseen collection. Default on_camera; the result echoes collectionMode.

-F 'collectionMode=off_camera'

Pass any value in the model field of /vision, /vision/chat, assessment config, etc.

Realtime voice providers are chosen via the provider field (openai/gemini) in the WS auth message, not model.

Pricing: inputTokens / 3000 + outputTokens / 1000. A metered call requires a balance of at least 1 credit.

Rough estimate: a Gemini-Flash frame ≈ 800 in + 15 out ≈ 0.28 credits; at 1 frame / 1.5s ≈ ~11 credits/min. The v2 Golden Frame pipeline reduces cost further by filtering frames before model calls.

All REST errors use { "success": false, "message": "…", "error": "…" }.

WebSocket close codes: 4003 auth failed · 4004 credits exhausted.