Jay Brown
395 lines
21 KiB
Markdown
395 lines
21 KiB
Markdown
# Chatbot Backend Guide
|
|
|
|
This guide covers the chatbot backend that queryAPI exposes for the Aaria
|
|
Agent Service integration. It is the operator entry point; design
|
|
rationale lives in `plans/chatbot_plan_codex.v10.md`.
|
|
|
|
## Overview
|
|
|
|
queryAPI exposes a synchronous HTTP facade over the Aaria FastAPI agent
|
|
service. Client UIs talk to queryAPI; queryAPI calls FastAPI and
|
|
persists the conversation in Postgres so future turns inherit the
|
|
prior context.
|
|
|
|
Who calls it:
|
|
- Client UIs as `client_user`: create sessions, post turns, list turns,
|
|
patch document/folder scope.
|
|
- Super-admins: list, get, soft-delete sessions across all users for a
|
|
given client.
|
|
|
|
What it does NOT do (out of scope per plan v10):
|
|
- Full-text search.
|
|
- GIN indexes.
|
|
- Costing / billing.
|
|
- Persisted structured data or result rows from the agent.
|
|
- User-initiated session deletes (only super-admin DELETE).
|
|
|
|
## Architecture
|
|
|
|
### Data model (`bot_*` tables, migration 131)
|
|
|
|
| Table | Purpose |
|
|
| ----------------------- | ----------------------------------------------------------------------------------------------- |
|
|
| `bot_sessions` | Owner-scoped chatbot conversations. Soft-deletable via `is_deleted`. Derives `last_terminal_ordinal` and `last_seen_ordinal` from `bot_turns`. |
|
|
| `bot_session_documents` | Per-session document scope. Empty set means "all documents for the client" (plan §4 fallback). |
|
|
| `bot_session_folders` | Per-session folder scope. Folder ids preserved in the FastAPI payload; expansion to documents happens in M3 payload assembly. |
|
|
| `bot_turns` | Append-only turn log. Status is one of `in_flight`, `completed`, `errored`, `abandoned`, `session_deleted`. |
|
|
|
|
Two CHECK constraints govern `bot_turns`:
|
|
|
|
1. `bot_turns_status_values` enforces the enum set.
|
|
2. `bot_turns_status_fields_consistent` enforces the (status, completion,
|
|
error) consistency matrix from plan §4 plus a guard clause for
|
|
unknown statuses. The guard clause `OR status NOT IN (...)` lets
|
|
`bot_turns_status_values` own the rejection of unknown values
|
|
without `_status_fields_consistent` firing first under Postgres's
|
|
alphabetical-by-name CHECK evaluation order. The plan §4 literal
|
|
three-clause matrix is preserved verbatim for documented statuses.
|
|
|
|
The partial unique index `bot_turns_one_inflight_per_session` enforces
|
|
"at most one in-flight turn per session". This is the primary
|
|
mechanism for serializing concurrent submissions.
|
|
|
|
### AddTurn flow (plan §6)
|
|
|
|
```text
|
|
AddTurn(input):
|
|
validate prompt (1 <= rune count <= MaxTurnChars)
|
|
attemptID = uuid.New()
|
|
|
|
tx1:
|
|
LockBotSessionForOwner(...)
|
|
AbandonExpiredInflightForOwner(grace = max(2 * RequestTimeoutSeconds, 60s))
|
|
re-derive last_seen_ordinal / last_terminal_ordinal
|
|
GetBotTurn(input.Ordinal)
|
|
if existing exists:
|
|
classify by status; reset on errored/abandoned (prompt must match,
|
|
ordinal must equal last_terminal_ordinal); reject on others.
|
|
else:
|
|
check ordinal matches natural target (last_seen+1 if has_inflight,
|
|
else last_terminal+1); InsertBotTurnPrompt; partial unique index
|
|
may reject -> ErrPriorTurnInFlight.
|
|
if input.Ordinal == 1:
|
|
SetBotSessionTitle(first 120 runes of prompt) where title=''
|
|
end tx1
|
|
|
|
Build conversation_history (last RecentTurns completed turns x 2 messages, asc).
|
|
Build chat_scope (stored docs + folder-expanded docs, dedup; folder ids preserved).
|
|
Call FastAPI /agent/chat OUTSIDE any DB tx (network round-trip).
|
|
|
|
tx2:
|
|
LockBotSessionForOwner(...)
|
|
if no row -> session was soft-deleted mid-call:
|
|
MarkBotTurnSessionDeleted(...) with attempt_id compare-and-set
|
|
return ErrSessionDeletedDuringCall (HTTP 410)
|
|
if FastAPI returned a typed error:
|
|
FailBotTurn(...) with error JSON containing attempt count
|
|
return mapped agent_* sentinel (HTTP 502)
|
|
if FastAPI succeeded but answer.text is empty/oversize:
|
|
FailBotTurn(...) with agent_invalid_response error JSON
|
|
return ErrAgentInvalidResponse (HTTP 502)
|
|
success path:
|
|
CompleteBotTurn(...) with attempt_id compare-and-set
|
|
summarize metadata (latency, tokens)
|
|
apply state update: null -> preserve prior; non-null -> strip
|
|
cached_results, write if under 64KB cap, else log warn + preserve
|
|
end tx2
|
|
```
|
|
|
|
The two-transaction model is deliberate: no DB connection is held
|
|
while calling FastAPI (which can take tens of seconds), and tx2's
|
|
attempt_id compare-and-set rejects stale callbacks (`zero rows
|
|
affected -> ErrTurnSuperseded`). Plan §6 + §11.
|
|
|
|
### State management
|
|
|
|
`bot_sessions.state jsonb` mirrors the agent's session state across
|
|
turns. On a successful turn:
|
|
|
|
- FastAPI `updated_session_state == null` -> preserve prior state. The
|
|
service does NOT call `UpdateBotSessionState`.
|
|
- `updated_session_state` non-null -> `StripCachedResults` removes the
|
|
top-level `cached_results` key (a hash-keyed MAP, often the largest
|
|
field). Then `SessionStateOversize` checks the result against
|
|
`chatbot.StateMaxBytes = 64 * 1024`. Under cap -> persist; over cap
|
|
-> log warn and preserve prior state.
|
|
|
|
The canonical example of a `session_state` payload is
|
|
`plans/chatbot.stuff/sample.response.json`. It has 10 top-level keys:
|
|
`turn`, `resolved_entities`, `last_scope`, `last_intent`,
|
|
`last_query_sql`, `last_query_plan`, `last_result_hash`,
|
|
`cached_results`, `last_scope_signature`, `validator_retry_count`.
|
|
After strip, the persisted payload has the remaining 9 top-level keys.
|
|
|
|
### Retry policy (FastAPI client, plan §7)
|
|
|
|
- Up to 3 attempts per HTTP call.
|
|
- Retry on HTTP 429, 500, 502, 503, 504, and transport timeouts.
|
|
- Do NOT retry other 4xx, malformed JSON, HTTP 200 with `status:"error"`,
|
|
or HTTP 200 with empty/null `answer.text`.
|
|
- Reuse the same `request_id` (`{sessionID}:{ordinal}`) on every
|
|
attempt. The caller mints it once; the client must not regenerate
|
|
per-attempt.
|
|
- One parent context enforces total wall time. Per-attempt timeout =
|
|
`cfg.RequestTimeoutSeconds` via `context.WithTimeout` derived from
|
|
the parent.
|
|
- Header literal: `X-API-Key: <cfg.APIKey verbatim>`. Production
|
|
values are JSON-encoded strings (`{"AGENT_SERVICE_API_KEY":"value"}`)
|
|
per plan §11.
|
|
|
|
### Error JSON shape
|
|
|
|
All `bot_turns.error` writes use the plan §6 canonical shape:
|
|
|
|
```json
|
|
{
|
|
"code": "<sentinel-string>",
|
|
"message": "<human-readable>",
|
|
"attempt": <int>
|
|
}
|
|
```
|
|
|
|
The `attempt` field carries the FastAPI client's `AttemptCount` so
|
|
operators can distinguish single-failure from exhausted-retry scenarios.
|
|
|
|
## Configuration reference
|
|
|
|
Five env vars, plan §8. Validated at startup via
|
|
`chatbot.ChatbotConfig.Validate` (called between
|
|
`serviceconfig.InitializeConfig` and `cfg.InitializeAuthConfig`).
|
|
|
|
| Variable | Default | Validation | Notes |
|
|
| --------------------------------- | ---------------------- | ----------------------------------- | ---------------------------------------------------- |
|
|
| `CHATBOT_SERVICE_URL` | (required, notEmpty) | env-loader rejects empty | FastAPI base URL. |
|
|
| `CHATBOT_API_KEY` | (required, notEmpty) | env-loader rejects empty | Sent verbatim as `X-API-Key` header. |
|
|
| `CHATBOT_REQUEST_TIMEOUT_SECONDS` | 60 | `>= 1` | Per-attempt FastAPI timeout. |
|
|
| `CHATBOT_MAX_TURN_CHARS` | 2000 | `1 <= x <= 65536` | Caps prompt and `answer.text` rune count. Upper bound (`chatbot.MaxTurnCharsCeiling`) mirrors the OpenAPI `prompt.maxLength`. |
|
|
| `CHATBOT_RECENT_TURNS` | 5 | `1 <= x <= 5` | Conversation-history length and session-list preview cap. |
|
|
|
|
Derived values (constants/helpers, not env vars):
|
|
|
|
- `chatbot.StateMaxBytes = 64 * 1024` — session-state cap (plan §8).
|
|
- `chatbot.MaxTurnCharsCeiling = 65536` — upper bound for `CHATBOT_MAX_TURN_CHARS`, locked to the OpenAPI `BotTurnAddRequest.prompt.maxLength`. Validate() rejects any value above this.
|
|
- `chatbot.ChatbotConfig.InflightGraceSeconds() = 2 * RequestTimeoutSeconds`
|
|
per plan §8 literal. The service uses
|
|
`effectiveGraceSeconds(cfg) = max(2 * RequestTimeoutSeconds, 60)` as
|
|
the actual abandonment threshold to keep small `RequestTimeoutSeconds`
|
|
in test fixtures from prematurely abandoning legitimately in-flight
|
|
turns. Production `RequestTimeoutSeconds=60` already exceeds the
|
|
60-second floor.
|
|
|
|
## API reference
|
|
|
|
### User routes (`/client/{clientId}/...`)
|
|
|
|
All routes are singular `/client/...` per plan §3 so Permit.io's
|
|
existing `client_user` policy applies.
|
|
|
|
| Method | Path | OperationId | Description |
|
|
| ------ | ----------------------------------------------------------------- | ------------------------- | -------------------------------------------------------- |
|
|
| POST | `/client/{clientId}/bot/sessions` | `createBotSession` | Create a new owner-scoped session. |
|
|
| GET | `/client/{clientId}/bot/sessions` | `listBotSessions` | List the caller's sessions; bounds: `limit in [1,200]`, `offset >= 0`. |
|
|
| GET | `/client/{clientId}/bot/sessions/{sessionId}` | `getBotSession` | Read one owner-scoped session. |
|
|
| PATCH | `/client/{clientId}/bot/sessions/{sessionId}/scope` | `patchBotSessionScope` | Add/remove documents and folders from session scope. |
|
|
| POST | `/client/{clientId}/bot/sessions/{sessionId}/turns` | `createBotTurn` | Submit a new turn. Plan §6 algorithm. |
|
|
| GET | `/client/{clientId}/bot/sessions/{sessionId}/turns` | `listBotTurns` | List all turns for an owner-scoped session, ASC. |
|
|
| GET | `/client/{clientId}/documents/{documentId}/schema` | `getDocumentSchema` | Return the document's bound JSON Schema or `null` (omitted in JSON when unbound). |
|
|
| GET | `/client/{clientId}/documents/{documentId}/all-metadata` | `getDocumentAllMetadata` | Bundle of (document, labels, customMetadata) for one doc. |
|
|
|
|
### Super-admin routes
|
|
|
|
| Method | Path | OperationId | Description |
|
|
| ------ | ----------------------------------------------------- | --------------------------------- | -------------------------------------------- |
|
|
| GET | `/super-admin/bot/sessions?clientId=...` | `listBotSessionsAsSuperAdmin` | List all sessions for a client (cross-user). |
|
|
| GET | `/super-admin/bot/sessions/{sessionId}?clientId=...` | `getBotSessionAsSuperAdmin` | Read one session by id (cross-user). |
|
|
| DELETE | `/super-admin/bot/sessions/{sessionId}?clientId=...` | `deleteBotSessionAsSuperAdmin` | Soft-delete; transitions any in-flight turns to `session_deleted`. |
|
|
|
|
### `BotSessionResponse.lastTurn`
|
|
|
|
`lastTurn` returns `last_terminal_ordinal` (plan §4 derived note), NOT
|
|
`last_seen_ordinal`. The latter is computed from `bot_turns` but used
|
|
only internally by `AddTurn` for ordinal classification. Clients should
|
|
display `lastTurn` as "the last completed turn this session has".
|
|
|
|
### `DocumentSchemaResponse.schema`
|
|
|
|
When the document has no `custom_schema_id` binding, the response is
|
|
HTTP 200 with the `schema` field omitted (the field is `*map[...]` with
|
|
`omitempty`, so JSON does NOT emit `"schema": null`; the key is
|
|
absent). Clients should treat both "key absent" and "key present and
|
|
null" as the unbound case.
|
|
|
|
### `BotLabelRecord` vs `LabelRecord`
|
|
|
|
The all-metadata endpoint uses `BotLabelRecord` (plain-string
|
|
`appliedBy`) instead of the existing `LabelRecord` (email-validated
|
|
`appliedBy`). Both shapes wrap the same `documentLabels.appliedBy
|
|
varchar` column; the divergence exists because the email-regex Marshal
|
|
validator on `LabelRecord` rejects non-email seed values that pre-date
|
|
the email-typing convention. Long-term cleanup is to relax
|
|
`LabelRecord.AppliedBy` to plain string. Until then, the chatbot
|
|
endpoints emit `BotLabelRecord`.
|
|
|
|
### HTTP status mapping
|
|
|
|
| Status | Sentinels / cases |
|
|
| ------ | -------------------------------------------------------------------------------------------- |
|
|
| 400 | `prompt_empty`, `prompt_too_long`, `add_remove_conflict`, `document_cross_client`, `folder_cross_client`, `clientId is required` (super-admin), invalid `limit`/`offset` |
|
|
| 401 | Missing Cognito subject |
|
|
| 404 | Cross-client / cross-user / soft-deleted session; document not found in client scope |
|
|
| 409 | `turn_in_flight`, `prior_turn_in_flight`, `already_complete`, `turn_session_deleted`, `ordinal_out_of_range`, `turn_superseded` (legacy: `prompt_mismatch` — service-layer assertion only; not reachable from the wire as of 2026-05-07) |
|
|
| 410 | `session_deleted_during_call` |
|
|
| 502 | `agent_application_error`, `agent_missing_answer`, `agent_invalid_response`, `agent_transport_error` |
|
|
| 500 | Unexpected backend errors |
|
|
|
|
`mapBotTurnError` dispatch order matters because `ErrAlreadyComplete`
|
|
chains to `ErrOrdinalOutOfRange` via a custom `Is()` method. The
|
|
sentinels are matched in declaration order; `ErrAlreadyComplete` must
|
|
be checked before `ErrOrdinalOutOfRange` so the message string
|
|
"already_complete" is returned in the HTTP retry-of-completed scenario.
|
|
The chain exists so plan §11's concurrency-test loss set
|
|
(`ErrTurnInFlight | ErrPriorTurnInFlight | ErrOrdinalOutOfRange`) covers
|
|
the case where a concurrent submitter's ordinal lands on the row the
|
|
winner just completed.
|
|
|
|
### Test-pattern note
|
|
|
|
Test assertions on HTTP error message bodies use substring matches.
|
|
`turn_in_flight` matches both `turn_in_flight` and `prior_turn_in_flight`
|
|
(the latter contains the former). M4's `TestAddTurn_SameOrdinalStillInflight_409`
|
|
exploits this: the test name says "same ordinal", but the algorithm
|
|
returns `prior_turn_in_flight` because the natural target is `last_seen+1`
|
|
when has_inflight; the substring assertion still passes. Both 409s
|
|
have the same operator semantics ("can't submit while a turn is in
|
|
flight").
|
|
|
|
## Operator runbook
|
|
|
|
### Failed-turn classifications
|
|
|
|
When a `bot_turns` row has a non-`completed` status, operators consult
|
|
`bot_turns.error` for the canonical error JSON and pick the recovery
|
|
path below.
|
|
|
|
#### `errored`
|
|
|
|
The FastAPI agent returned a typed error or the response failed
|
|
validation. The error JSON includes `code`, `message`, and `attempt`.
|
|
Common codes:
|
|
|
|
| Code | Meaning | Recovery |
|
|
| -------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
|
|
| `agent_application_error` | FastAPI returned 200 with `status:"error"`. Code/message in the inner `error` field. | Client may resubmit the same prompt at the same ordinal; the algorithm reset path runs. |
|
|
| `agent_missing_answer` | FastAPI returned 200 with empty/null/whitespace `answer.text`. | Same as above. May indicate a prompt the agent cannot answer; client UX should suggest rephrasing. |
|
|
| `agent_invalid_response` | FastAPI returned 200 but body failed structural validation (malformed JSON or oversize completion). | Same as above. May indicate an upstream agent change; check upstream logs. |
|
|
| `agent_transport_error` | FastAPI exhausted its 3-attempt retry budget on transport-level failures (429/5xx/timeouts). | Same as above. May indicate FastAPI overload; check upstream health and rate-limit. |
|
|
|
|
To retry: client POSTs to the same `/turns` route with the same prompt.
|
|
The handler picks the `errored` row's ordinal via prompt-match;
|
|
`AddTurn` runs the existing-row reset path; a new `attempt_id` is
|
|
minted; tx2 records the new outcome.
|
|
|
|
If the client posts a *different* prompt instead of retrying, the
|
|
handler advances to `last_terminal_ordinal+1` (a fresh `in_flight` at
|
|
ordinal N+1) rather than rejecting with 409 `prompt_mismatch`. The
|
|
errored row at ordinal N stays in history. See
|
|
`plans/chatbot.stuff/fix.chat.bugs.1.md` §3 Option A for the rationale.
|
|
|
|
#### `abandoned`
|
|
|
|
The grace window expired without a terminal status. Most often this
|
|
means the queryAPI process died mid-call and the next caller's
|
|
`AbandonExpiredInflightForOwner` swept the stale row. The error JSON
|
|
carries:
|
|
|
|
```json
|
|
{"code":"turn_abandoned_grace_expired","message":"no terminal status reached within grace window"}
|
|
```
|
|
|
|
Recovery is identical to `errored`: client resubmits the same prompt.
|
|
The reset path applies.
|
|
|
|
#### `session_deleted`
|
|
|
|
A super-admin DELETEd the session while the turn was in flight, OR an
|
|
older retry path hit the session-deleted-during-call branch. The error
|
|
JSON carries:
|
|
|
|
```json
|
|
{"code":"session_deleted_by_admin","message":"session deleted while turn in flight","attempt":<int>}
|
|
```
|
|
|
|
OR (from the AddTurn tx2 path):
|
|
|
|
```json
|
|
{"code":"session_deleted_during_call","message":"session was soft-deleted while FastAPI call was in flight","attempt":<int>}
|
|
```
|
|
|
|
Recovery: NONE. The session is permanent. The client must create a new
|
|
session and resubmit. The historical `session_deleted` row remains for
|
|
audit; LIST turns surfaces it with the original prompt and the error
|
|
JSON.
|
|
|
|
### Performance notes
|
|
|
|
`decorateSessionList` (the M2 list-sessions handler path) does N+1
|
|
scope reads: one `loadScope` round-trip per session in the list.
|
|
Default `limit=20`, hard cap `limit=200`. Under default load the
|
|
overhead is negligible; if a deployment surfaces latency on the list
|
|
endpoint, consider a single batch read of all scope rows for the
|
|
listed session ids. Plan v10 marked this as acceptable for v1.
|
|
|
|
### Rate-limit overrides
|
|
|
|
`RATE_LIMIT_ENDPOINT_OVERRIDES` is a JSON object keyed by
|
|
`METHOD /path` using **Echo's path template form** (`:clientId`,
|
|
`:sessionId`), NOT the OpenAPI `{clientId}` form. The middleware uses
|
|
`c.Path()` which returns Echo's registered path string; an override
|
|
keyed by `{clientId}` will silently fail to match.
|
|
|
|
Example for the chatbot turn endpoint:
|
|
|
|
```bash
|
|
RATE_LIMIT_ENDPOINT_OVERRIDES='{"POST /client/:clientId/bot/sessions/:sessionId/turns":{"global_rate":50,"global_burst":100,"rate":2,"burst":4}}'
|
|
```
|
|
|
|
**M6 operator-facing behavior change**: setting an override here NOW
|
|
throttles the actual request rate (per-(IP, route) budget). Before M6,
|
|
overrides only affected the `RateLimit` and `Retry-After` headers; the
|
|
Allow() decision used the default rate/burst. Plan §10 M6 + plan §11.
|
|
|
|
## References
|
|
|
|
- Plan: `plans/chatbot_plan_codex.v10.md` (design rationale and
|
|
authoritative spec).
|
|
- FastAPI swagger: `plans/chatbot.stuff/openapi.chatbot.swagger.json`.
|
|
- Sample `session_state` fixture: `plans/chatbot.stuff/sample.response.json`.
|
|
- Cloud FastAPI deployment (dev): see plan §11 for the URL and the
|
|
literal `X-API-Key` value.
|
|
|
|
## Known warts
|
|
|
|
These are not bugs; they are conscious tradeoffs documented for
|
|
maintainers.
|
|
|
|
1. **CHECK guard clause** in `bot_turns_status_fields_consistent`. The
|
|
plan §4 literal three-clause matrix is preserved verbatim; the
|
|
trailing `OR status NOT IN (...)` is an evaluation-order guard so
|
|
`bot_turns_status_values` owns the rejection of unknown enum values.
|
|
2. **`ErrAlreadyComplete` chain to `ErrOrdinalOutOfRange`** via custom
|
|
`Is()` so plan §11's concurrency-test loss set covers the case
|
|
where a loser's ordinal lands on the winner's just-completed row.
|
|
3. **`BotLabelRecord` vs `LabelRecord`** divergence (above).
|
|
4. **`schema=null`** uses field omission (omitempty) rather than
|
|
explicit JSON null. Clients should accept both.
|
|
5. **`M5DocumentID` alias** is cosmetically distinct from `DocumentID`
|
|
in the generated types. Both are `openapi_types.UUID`. Cosmetic only.
|
|
6. **Rate-limit override `:param` template** form, not OpenAPI
|
|
`{param}` form (above).
|
|
7. **N+1 scope reads** in session list (above).
|
|
8. **Grace-window floor of 60s** in `effectiveGraceSeconds`; the plan
|
|
§8 literal `2 * RequestTimeoutSeconds` is preserved on the chatbot
|
|
config method, but the service applies the floor at the call site
|
|
so test fixtures with `RequestTimeoutSeconds=1` (2s grace) do not
|
|
prematurely abandon legitimately in-flight turns.
|