query-orchestration/docs/ai.generated/software-architecture.md

QueryAPI Software Architecture Guide

This document provides a comprehensive guide to the QueryAPI software architecture for new engineers joining the project. It explains the purpose of each architectural layer, how components interact, and what logic belongs where when adding new features.

Table of Contents

• Architecture Overview
• Component Layers
• Request Flow
• Adding a New API Endpoint
• Key Patterns
• Component Diagrams

Architecture Overview

QueryAPI follows a layered architecture pattern with clear separation of concerns:

flowchart TB
    subgraph External["External Layer"]
        Client[HTTP Client]
        OpenAPI[OpenAPI Spec<br/>serviceAPIs/queryAPI.yaml]
    end

    subgraph Entry["Entry Point"]
        Main[cmd/queryAPI/main.go]
    end

    subgraph Server["Server Layer"]
        Listener[internal/server/api/listener.go]
        Echo[Echo HTTP Router]
    end

    subgraph Middleware["Middleware Layer"]
        RateLimit[Rate Limiting]
        JWT[JWT Auth Middleware]
        TokenValidation[Token Validation]
        PermitIO[Permit.io Authorization]
        OAPIValidation[OpenAPI Request Validation]
    end

    subgraph Generated["Generated Layer"]
        APIGen[api/queryAPI/api.gen.go<br/>ServerInterface + Types]
        RegisterHandlers[RegisterHandlers]
    end

    subgraph Controllers["Controller Layer"]
        Controllers_impl[api/queryAPI/*.go<br/>Controllers struct]
    end

    subgraph Services["Service Layer"]
        ServiceImpl[internal/&lt;domain&gt;/service.go<br/>Business Logic]
    end

    subgraph Data["Data Access Layer"]
        Repository[internal/database/repository/*.sql.go<br/>SQLC Generated]
        Queries[internal/database/queries/*.sql<br/>SQL Definitions]
        Migrations[internal/database/migrations/*.sql<br/>Schema Changes]
    end

    subgraph Infrastructure["Infrastructure"]
        DB[(PostgreSQL)]
        S3[(AWS S3)]
        SQS[(AWS SQS)]
    end

    Client --> Main
    OpenAPI -.->|generates| APIGen
    Main --> Listener
    Listener --> Echo
    Echo --> RateLimit --> JWT --> TokenValidation --> PermitIO --> OAPIValidation
    OAPIValidation --> RegisterHandlers
    RegisterHandlers --> Controllers_impl
    Controllers_impl --> ServiceImpl
    ServiceImpl --> Repository
    Queries -.->|generates| Repository
    Repository --> DB
    ServiceImpl --> S3
    ServiceImpl --> SQS

Component Layers

1. Entry Point (cmd/queryAPI/main.go)

Purpose: Application bootstrap and dependency wiring.

Responsibilities:

• Print version information
• Initialize configuration from environment variables
• Create and wire all service dependencies
• Register handlers with the router
• Start the HTTP server

Key Code Pattern:

// Create services
cli := client.New(cfg)
doc := document.New(cfg)

// Wire services into Controllers
services := &queryapi.Services{
    Client:   cli,
    Document: doc,
}
cons := queryapi.NewControllers(services, cfg)

// Register routes
queryapi.RegisterHandlers(cfg.Router, cons)

When to modify: When adding a new domain service that needs to be injected into the API layer.

---

2. Server Layer (internal/server/api/listener.go)

Purpose: HTTP server configuration, middleware setup, and lifecycle management.

Responsibilities:

• Create and configure the Echo HTTP router
• Register middleware in the correct order
• Set up health and metrics endpoints
• Initialize background task runners
• Handle graceful shutdown

Key Components:

Component	Purpose
Server struct	Holds router, port, and cleanup functions
New() function	Creates server with all middleware configured
Listen() method	Starts the HTTP server
Middleware chain	Rate limiting -> Auth -> Validation

Middleware Order (applied in sequence):

1. Config injection middleware
2. Prometheus metrics
3. Request logging (slog)
4. Recovery (panic handling)
5. CORS
6. Debug logging (when DEBUG=true)
7. Rate limiting
8. Authentication routes registration
9. Test user injection (when DISABLE_AUTH=true)
10. Handler registration
11. OpenAPI request validation

---

3. Middleware Layer (internal/cognitoauth/)

Purpose: Authentication and authorization enforcement.

Key Files:

File	Purpose
middleware.go	Main middleware functions (TokenValidationMiddleware, JWTAuthMiddleware)
handler.go	OAuth callback and login flow handlers
permitio.go	Permit.io authorization client
token.go	JWT token operations (refresh, validation)
validation.go	Token signature and claims validation

Authentication Flow:

sequenceDiagram
    participant C as Client
    participant M as Middleware
    participant J as JWKS
    participant P as Permit.io

    C->>M: Request with JWT
    M->>M: Check if public path
    alt Public Path
        M->>C: Allow (skip auth)
    else Protected Path
        M->>M: Extract token from header/cookie
        M->>J: Verify token signature
        J-->>M: Token valid
        M->>M: Extract user claims
        M->>P: Check permission(user, action, resource)
        P-->>M: Permitted/Denied
        alt Permitted
            M->>C: Allow request
        else Denied
            M->>C: 403 Forbidden
        end
    end

Path Types:

• Public paths: /health, /swagger/*, /metrics, /eula (GET only)
• Auth-only paths: /identity, /eula/status, /eula/agree (authenticated but no specific permissions)
• Protected paths: All other routes (require both authentication and Permit.io authorization)

---

4. Generated API Layer (api/queryAPI/api.gen.go)

Purpose: Type-safe API interface generated from OpenAPI specification.

Generated by: oapi-codegen from serviceAPIs/queryAPI.yaml

Key Generated Components:

Component	Purpose
ServerInterface	Interface that controllers must implement
Request/Response types	Strongly-typed DTOs (e.g., ClientCreate, DocClient)
RegisterHandlers()	Connects routes to handler methods
ServerInterfaceWrapper	Wraps handlers with parameter extraction
Parameter types	Path/query parameter types (e.g., ClientID)

Example Interface Method:

// From ServerInterface
CreateClient(ctx echo.Context) error
GetClient(ctx echo.Context, id ClientID) error

Important: Never edit api.gen.go directly. Modify serviceAPIs/queryAPI.yaml and run task generate.

---

5. Controller Layer (api/queryAPI/*.go)

Purpose: HTTP request handling and response formatting.

Key Files:

File	Domain
controllers.go	Controllers struct and constructor
client.go	Client endpoints
documents.go	Document endpoints
folders.go	Folder endpoints
labels.go	Label endpoints
collector.go	Collector endpoints
export.go	Export endpoints
authHandlers.go	Authentication-related endpoints
adminHandlers.go	User administration endpoints

Responsibilities:

• Bind and validate request bodies
• Call appropriate service methods
• Transform service results to API response types
• Return appropriate HTTP status codes
• Handle errors with proper HTTP error responses

Controller Pattern:

func (s *Controllers) GetClient(ctx echo.Context, id ClientID) error {
    // 1. Call service layer
    client, err := s.svc.Client.Get(ctx.Request().Context(), id)
    if err != nil {
        // 2. Handle error
        return echo.NewHTTPError(http.StatusBadRequest,
            fmt.Sprintf("Unable to get client: %s", err))
    }

    // 3. Transform to API type and return
    return ctx.JSON(http.StatusOK, DocClient{
        Id:      client.ID,
        Name:    client.Name,
        CanSync: client.CanSync,
    })
}

What belongs here:

• Request parsing and binding
• Input validation (beyond OpenAPI validation)
• Calling service methods
• Response transformation
• HTTP status code decisions

What does NOT belong here:

• Business logic
• Database operations
• External service calls
• Complex data transformations

---

6. Service Layer (internal/<domain>/)

Purpose: Business logic and domain operations.

Structure per domain:

internal/client/
├── service.go      # Service struct and constructor
├── create.go       # Create operation
├── get.go          # Read operations
├── list.go         # List operations
├── status.go       # Status checks
└── *_test.go       # Unit tests

Service Pattern:

// Service struct with config dependency
type Service struct {
    cfg serviceconfig.ConfigProvider
}

// Constructor
func New(cfg serviceconfig.ConfigProvider) *Service {
    return &Service{cfg}
}

// Business operation
func (s *Service) Get(ctx context.Context, id string) (*Client, error) {
    // Access database through config
    client, err := s.cfg.GetDBQueries().GetClient(ctx, id)
    if err != nil {
        return nil, err
    }
    return parseFullClient(client), nil
}

What belongs here:

• Business logic and rules
• Input normalization and validation
• Orchestration of multiple repository calls
• Transaction management
• Domain-specific transformations
• Business error handling

Accessing Infrastructure:

// Database queries
s.cfg.GetDBQueries().GetClient(ctx, id)

// Database transactions
s.cfg.ExecuteDBTransaction(ctx, func(ctx context.Context, q *repository.Queries) error {
    // Multiple operations in a transaction
    return nil
})

// Object storage
s.cfg.GetStoreClient()   // S3 client
s.cfg.GetBucket()        // Bucket name

// Queue operations
s.cfg.GetQueueClient()

---

7. Repository Layer (internal/database/repository/)

Purpose: Type-safe database access generated from SQL.

Generated by: SQLC from internal/database/queries/*.sql

Key Files:

File	Purpose
db.go	Queries struct and DBTX interface
models.go	Generated Go types from database schema
client.sql.go	Generated client query methods
document.sql.go	Generated document query methods
etc.	One file per SQL source file

SQL Query Definition (internal/database/queries/client.sql):

-- name: GetClient :one
SELECT * FROM fullClients WHERE clientId = $1;

-- name: CreateClient :exec
INSERT INTO clients (clientId, name) VALUES ($1, $2);

-- name: ListClients :many
SELECT * FROM fullClients ORDER BY name;

Generated Go Code:

func (q *Queries) GetClient(ctx context.Context, clientid string) (*Fullclient, error)
func (q *Queries) CreateClient(ctx context.Context, arg *CreateClientParams) error
func (q *Queries) ListClients(ctx context.Context) ([]*Fullclient, error)

Transaction Support:

// Create queries with transaction
q.WithTx(tx)

---

8. Database Migrations (internal/database/migrations/)

Purpose: Version-controlled schema evolution.

Naming Convention: YYYYMMDDHHMMSS_description.up.sql / .down.sql

Examples:

• 00000000000003_clients.up.sql - Create clients table
• 00000000000109_create_folders_table.up.sql - Add folders feature

Migration Tool: golang-migrate

---

9. Configuration Layer (internal/serviceconfig/)

Purpose: Environment-based configuration with dependency injection.

Key Interface (ConfigProvider):

type ConfigProvider interface {
    // Database access
    GetDBQueries() *repository.Queries
    GetDBPool() *pgxpool.Pool
    ExecuteDBTransaction(ctx, func) error

    // Logging
    GetLogger() *slog.Logger

    // AWS services
    GetStoreClient() objectstore.Client  // S3
    GetQueueClient() queue.Client        // SQS
    GetBucket() string

    // Auth
    GetAuthConfig() auth.ConfigProvider
}

Configuration Composition:

type QueryAPIConfig struct {
    api.BaseConfig                    // Server config
    clientsync.ClientSyncConfig       // Queue config
    objectstore.ObjectStoreConfig     // S3 config
    BackgroundRunner *backgroundtask.Runner
}

---

Request Flow

Here's how a request flows through the system:

sequenceDiagram
    participant C as HTTP Client
    participant E as Echo Router
    participant MW as Middleware Chain
    participant V as OpenAPI Validator
    participant W as ServerInterfaceWrapper
    participant H as Controller Handler
    participant S as Service
    participant R as Repository
    participant DB as PostgreSQL

    C->>E: POST /client
    E->>MW: Apply middleware
    MW->>MW: Rate limit check
    MW->>MW: JWT validation
    MW->>MW: Permit.io authorization
    MW->>V: Validate request body
    V->>W: Route to wrapper
    W->>W: Extract path/query params
    W->>H: CreateClient(ctx)
    H->>H: ctx.Bind(&req)
    H->>S: client.Create(ctx, params)
    S->>S: Normalize input
    S->>R: CreateClient(ctx, dbParams)
    R->>DB: INSERT INTO clients...
    DB-->>R: Success
    R-->>S: nil error
    S-->>H: clientID, nil
    H->>H: Build response
    H-->>C: 201 Created + JSON body

---

Adding a New API Endpoint

Follow these steps to add a new endpoint (e.g., GET /client/{id}/summary):

Step 1: Define the API Contract

Edit serviceAPIs/queryAPI.yaml:

paths:
  /client/{id}/summary:
    parameters:
      - $ref: "#/components/parameters/ClientID"
    get:
      operationId: getClientSummary
      tags:
        - ClientService
      summary: Get client summary
      security:
        - jwtAuth: []
      responses:
        "200":
          description: Client summary
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/ClientSummary"

components:
  schemas:
    ClientSummary:
      type: object
      required:
        - id
        - documentCount
      properties:
        id:
          type: string
        documentCount:
          type: integer

Step 2: Generate Code

task generate

This updates:

• api/queryAPI/api.gen.go - Adds GetClientSummary to ServerInterface and ClientSummary type

Step 3: Add Database Query (if needed)

Edit internal/database/queries/client.sql:

-- name: GetClientDocumentCount :one
SELECT COUNT(*) as count FROM documents WHERE clientId = $1;

Run: task generate

Step 4: Add Service Method

Create or edit internal/client/summary.go:

package client

import "context"

type Summary struct {
    ID            string
    DocumentCount int64
}

func (s *Service) GetSummary(ctx context.Context, id string) (*Summary, error) {
    // Validate client exists
    client, err := s.Get(ctx, id)
    if err != nil {
        return nil, err
    }

    // Get document count
    count, err := s.cfg.GetDBQueries().GetClientDocumentCount(ctx, id)
    if err != nil {
        return nil, fmt.Errorf("failed to get document count: %w", err)
    }

    return &Summary{
        ID:            client.ID,
        DocumentCount: count,
    }, nil
}

Step 5: Implement Handler

Edit api/queryAPI/client.go:

func (s *Controllers) GetClientSummary(ctx echo.Context, id ClientID) error {
    summary, err := s.svc.Client.GetSummary(ctx.Request().Context(), id)
    if err != nil {
        return echo.NewHTTPError(http.StatusBadRequest,
            fmt.Sprintf("Unable to get client summary: %s", err))
    }

    return ctx.JSON(http.StatusOK, ClientSummary{
        Id:            summary.ID,
        DocumentCount: int32(summary.DocumentCount),
    })
}

Step 6: Add Tests

Create internal/client/summary_test.go:

func TestService_GetSummary(t *testing.T) {
    // Use testcontainers for real database testing
    cfg := test.NewTestConfig(t)
    svc := client.New(cfg)

    // Create test client
    clientID, err := svc.Create(ctx, client.CreateParams{...})
    require.NoError(t, err)

    // Test summary
    summary, err := svc.GetSummary(ctx, clientID)
    require.NoError(t, err)
    assert.Equal(t, clientID, summary.ID)
    assert.Equal(t, int64(0), summary.DocumentCount)
}

Step 7: Verify

task lint          # Check for issues
task test:functional  # Run test suite
task fullsuite:ci  # Run full test suite

---

Key Patterns

Dependency Injection via ConfigProvider

Services receive all dependencies through the ConfigProvider interface:

type Service struct {
    cfg serviceconfig.ConfigProvider
}

func (s *Service) DoSomething(ctx context.Context) error {
    // Database
    s.cfg.GetDBQueries().SomeQuery(ctx, ...)

    // Logging
    s.cfg.GetLogger().Info("Operation completed")

    // S3
    s.cfg.GetStoreClient().PutObject(...)

    return nil
}

Transaction Handling

Use ExecuteDBTransaction for operations requiring atomicity:

err := s.cfg.ExecuteDBTransaction(ctx, func(ctx context.Context, q *repository.Queries) error {
    if err := q.CreateClient(ctx, params); err != nil {
        return err
    }
    if err := q.CreateRootFolder(ctx, folderParams); err != nil {
        return err
    }
    return nil
})

Error Handling

Services return errors, controllers convert to HTTP errors:

// Service - return domain errors
if client == nil {
    return nil, errors.New("client not found")
}

// Controller - convert to HTTP error
if err != nil {
    return echo.NewHTTPError(http.StatusNotFound, err.Error())
}

Input Normalization

Normalize and validate inputs in the service layer:

func (s *Service) normalizeCreate(params *CreateParams) error {
    // Trim whitespace
    params.Name = strings.TrimSpace(params.Name)

    // Validate format
    if !isValidID(params.ID) {
        return errors.New("invalid ID format")
    }
    return nil
}

---

Component Diagrams

Domain Package Structure

flowchart LR
    subgraph "internal/client/"
        S[service.go<br/>Service struct]
        C[create.go<br/>Create operation]
        G[get.go<br/>Read operations]
        L[list.go<br/>List operations]
        ST[status.go<br/>Status checks]
    end

    subgraph "Repository"
        R[repository/client.sql.go]
    end

    S --> C
    S --> G
    S --> L
    S --> ST
    C --> R
    G --> R
    L --> R
    ST --> R

Configuration Composition

classDiagram
    class ConfigProvider {
        <<interface>>
        GetDBQueries()
        GetLogger()
        GetStoreClient()
    }

    class BaseConfig {
        Port int
        BaseURL string
        CognitoConfig
        DBConfig
        AWSConfig
    }

    class QueryAPIConfig {
        BaseConfig
        ClientSyncConfig
        ObjectStoreConfig
    }

    ConfigProvider <|.. BaseConfig
    BaseConfig <|-- QueryAPIConfig

Service Layer Interactions

flowchart TB
    subgraph Controllers
        CH[Client Handler]
        DH[Document Handler]
    end

    subgraph Services
        CS[Client Service]
        DS[Document Service]
        US[Upload Service]
    end

    subgraph Repository
        CR[Client Queries]
        DR[Document Queries]
    end

    subgraph Infrastructure
        DB[(PostgreSQL)]
        S3[(S3)]
    end

    CH --> CS
    DH --> DS
    DH --> US
    CS --> CR
    DS --> DR
    US --> DR
    US --> S3
    CR --> DB
    DR --> DB

---

Quick Reference

Layer	Location	Responsibility
Entry Point	cmd/queryAPI/main.go	Bootstrap, wire dependencies
Server	internal/server/api/listener.go	HTTP server, middleware setup
Middleware	internal/cognitoauth/	Auth, authorization
Generated	api/queryAPI/api.gen.go	Interface, types (don't edit)
Controllers	api/queryAPI/*.go	Request handling, response formatting
Services	internal/<domain>/	Business logic
Repository	internal/database/repository/	Database access (generated)
Queries	internal/database/queries/*.sql	SQL definitions
Migrations	internal/database/migrations/	Schema changes
Config	internal/serviceconfig/	Environment config, DI

---

See Also

• 02-service-architecture.md - Overall system architecture
• 06-code-organization.md - Project structure details
• 03-api-documentation.md - API reference
• 04-data-architecture.md - Database schema