query-orchestration/docs/ai.generated/06-code-organization.md

# Code Organization

## Project Structure

The project follows Go Standard Project Layout with enhanced organization for batch processing, observability, and background task management.

The codebase follows the [Standard Go Project Layout](https://github.com/golang-standards/project-layout) with additional conventions specific to the domain requirements.

### Directory Overview

```
query-orchestration.2/
├── cmd/                    # Main applications and entry points
├── internal/               # Private application code  
├── api/                    # Generated API server implementations
├── pkg/                    # Public library code
├── deployments/            # Deployment configurations
├── docs/                   # Documentation
├── scripts/                # Build and automation scripts
├── test/                   # Integration and e2e tests
├── assets/                 # Test data and sample files
└── vendor/                 # Vendored dependencies
```

## Package Architecture

### Command Layer (`cmd/`)

Entry points for all deployable services, following the pattern:
```
cmd/
├── [serviceName]/
│   └── main.go            # Service bootstrap and configuration
```

**Naming Conventions**:
- **APIs**: `[functionality]API` (e.g., `queryAPI`)
- **Runners**: `[functionality]Runner` (e.g., `docInitRunner`)
- **Utilities**: Descriptive names (e.g., `healthcheck`)

### Internal Packages (`internal/`)

Core business logic and infrastructure components, organized by domain:

#### New Components (Since June 2025):
- **`internal/backgroundtask/`**: Background task execution framework with periodic scheduling
- **`internal/document/batch/`**: Batch upload processing and ZIP archive handling  
- **`internal/serviceconfig/observability/prometheus/`**: Comprehensive Prometheus metrics framework

### Internal Packages (`internal/`)

Core business logic organized by domain boundaries:

#### Domain Services
```
internal/
├── client/                # Client management domain
├── document/              # Document processing domain
├── fieldextraction/       # Field extraction domain
├── collector/             # Client-specific configurations
└── export/                # Data export operations
```

#### Infrastructure Services  
```
internal/
├── database/              # Data persistence layer
├── server/                # HTTP and queue server infrastructure
├── serviceconfig/         # Configuration management
├── cognitoauth/          # Authentication middleware
├── test/                  # Testing utilities
└── validation/            # Input validation
```

### API Layer (`api/`)

Generated server implementations from OpenAPI specifications:
```
api/
├── queryAPI/              # REST API implementation
│   ├── api.gen.go         # Generated types and interfaces
│   ├── controllers.go     # Business logic implementation
│   ├── authHandlers.go    # Authentication endpoints
│   └── *.go              # Domain-specific handlers
└── [runnerName]/          # Queue-based service implementations
    └── runner.go          # Queue message processing
```

## Design Patterns and Principles

### Service Layer Pattern

Each domain follows a consistent service layer structure:

```go
// Service struct with injected dependencies
type Service struct {
    cfg ConfigProvider  // Configuration interface
}

// Constructor with interface-based dependency injection
func New(cfg ConfigProvider) *Service {
    return &Service{cfg: cfg}
}

// Business operations
func (s *Service) CreateEntity(ctx context.Context, req CreateRequest) (*Entity, error) {
    // Business logic implementation
}
```

### Repository Pattern

Database access is abstracted through repositories:

```go
// Generated by SQLC from SQL queries
type Repository interface {
    CreateClient(ctx context.Context, arg CreateClientParams) (Client, error)
    GetClient(ctx context.Context, clientID string) (Client, error)
    // ... other operations
}

// Service uses repository interface
type Service struct {
    repo Repository
}
```

### Configuration Provider Pattern

Services receive configuration through composed interfaces:

```go
// Domain-specific configuration interface
type ConfigProvider interface {
    serviceconfig.ConfigProvider      // Base configuration
    database.ConfigProvider          // Database access
    objectstore.ConfigProvider       // S3 access
    // ... other provider interfaces
}

// Concrete configuration embeds all providers
type Config struct {
    serviceconfig.BaseConfig
    database.Config
    objectstore.Config
}
```

### Dependency Injection

Services are composed through constructor injection:

```go
// In main.go
cfg := &Config{
    BaseConfig:     baseConfig,
    DatabaseConfig: dbConfig,
    // ... other configs
}

// Create service with injected dependencies
svc := service.New(cfg)

// Create controller with service dependency
controller := api.NewController(svc)
```

## Naming Conventions

### Package Names
- **Domain packages**: Singular nouns (`client`, `document`, `fieldextraction`)
- **Utility packages**: Descriptive (`serviceconfig`, `cognitoauth`)
- **Generated packages**: Match source (`queryAPI` from `queryAPI.yaml`)

### File Names
- **Service files**: `service.go` (main business logic)
- **Test files**: `*_test.go` (alongside source)
- **Private tests**: `*private_test.go` (package-private testing)
- **Interface files**: Match domain (`client.go`, `document.go`)

### Function and Method Names
- **Public functions**: PascalCase (`CreateClient`, `GetDocument`)
- **Private functions**: camelCase (`validateInput`, `processResult`)
- **Constructors**: `New` or `New[Type]` (`New`, `NewService`)
- **Getters**: No `Get` prefix for simple accessors (`ID()`, not `GetID()`)

### Variable Names
- **Short scope**: Single letters (`i`, `j`, `c` for client)
- **Medium scope**: Abbreviations (`ctx`, `req`, `resp`)
- **Long scope**: Descriptive names (`clientRepository`, `configProvider`)
- **Constants**: PascalCase (`MaxRetryAttempts`, `DefaultTimeout`)

### Type Names
- **Structs**: PascalCase nouns (`Client`, `DocumentProcessor`)
- **Interfaces**: PascalCase with optional "er" suffix (`ConfigProvider`, `Processor`)
- **Enums**: PascalCase with type suffix (`QueryType`, `ProcessingStatus`)

## Code Style Guidelines

### Import Organization
```go
package main

import (
    // Standard library
    "context"
    "fmt"
    
    // Third-party packages  
    "github.com/labstack/echo/v4"
    "github.com/google/uuid"
    
    // Local packages
    "queryorchestration/internal/client"
    "queryorchestration/internal/database"
)
```

### Error Handling
```go
// Wrap errors with context
func (s *Service) ProcessDocument(ctx context.Context, id uuid.UUID) error {
    doc, err := s.repo.GetDocument(ctx, id)
    if err != nil {
        return fmt.Errorf("failed to get document %s: %w", id, err)
    }
    
    if err := s.processContent(doc); err != nil {
        return fmt.Errorf("failed to process document content: %w", err)
    }
    
    return nil
}
```

### Interface Design
```go
// Prefer small, focused interfaces
type ClientReader interface {
    GetClient(ctx context.Context, id string) (*Client, error)
}

type ClientWriter interface {
    CreateClient(ctx context.Context, client *Client) error
    UpdateClient(ctx context.Context, client *Client) error
}

// Compose when needed
type ClientRepository interface {
    ClientReader
    ClientWriter
}
```

### Struct Design
```go
// Group related fields
type Client struct {
    // Identity
    ID   string
    Name string
    
    // Configuration
    CanSync bool
    
    // Metadata
    CreatedAt time.Time
    UpdatedAt time.Time
}

// Use embedding for composition
type APIClient struct {
    Client
    AuthToken string
}
```

## Testing Patterns

### Table-Driven Tests
```go
func TestService_CreateClient(t *testing.T) {
    tests := []struct {
        name    string
        input   CreateClientRequest
        want    *Client
        wantErr bool
    }{
        {
            name: "valid client",
            input: CreateClientRequest{
                ID:   "test-client",
                Name: "Test Client",
            },
            want: &Client{
                ID:   "test-client", 
                Name: "Test Client",
            },
            wantErr: false,
        },
        {
            name: "empty ID",
            input: CreateClientRequest{
                Name: "Test Client",
            },
            want:    nil,
            wantErr: true,
        },
    }
    
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            // Test implementation
        })
    }
}
```

### Mock Usage
```go
// Generate mocks for interfaces
//go:generate mockery --name=ClientRepository --output=mocks

func TestService_WithMocks(t *testing.T) {
    // Create mock
    mockRepo := &mocks.ClientRepository{}
    
    // Set expectations
    mockRepo.On("GetClient", mock.Anything, "test-id").
        Return(&Client{ID: "test-id"}, nil)
    
    // Create service with mock
    svc := &Service{repo: mockRepo}
    
    // Test and verify
    result, err := svc.GetClient(context.Background(), "test-id")
    assert.NoError(t, err)
    assert.Equal(t, "test-id", result.ID)
    
    mockRepo.AssertExpectations(t)
}
```

### Integration Test Structure
```go
func TestIntegration_DocumentProcessing(t *testing.T) {
    // Setup test environment
    ctx := context.Background()
    testDB := test.NewDatabase(t)
    testS3 := test.NewObjectStore(t)
    
    // Create service with real dependencies
    cfg := &Config{
        Database:    testDB,
        ObjectStore: testS3,
    }
    svc := New(cfg)
    
    // Run test scenario
    client := test.CreateTestClient(t, svc)
    document := test.UploadTestDocument(t, svc, client.ID)
    
    // Verify results
    result := test.ProcessDocument(t, svc, document.ID)
    assert.NotNil(t, result)
    
    // Cleanup handled by test utilities
}
```

## Architecture Decisions

### Configuration Management
- **Environment-first**: All configuration from environment variables
- **Type safety**: Strong typing with validation
- **Composition**: Interface-based configuration providers
- **Testing**: Override-friendly for test scenarios

### Error Handling
- **Wrap errors**: Add context at each layer
- **Structured logging**: Use slog for consistent error logging
- **Graceful degradation**: Continue operation when possible
- **Client-friendly**: Translate internal errors to appropriate HTTP status

### Database Access
- **Type safety**: SQLC generates type-safe database code
- **Migrations**: Version-controlled schema evolution
- **Connection pooling**: Shared connections across services
- **Transaction management**: Explicit transaction boundaries

### API Design
- **Contract-first**: OpenAPI specification drives implementation
- **Version compatibility**: Backward-compatible changes
- **Validation**: Automatic request/response validation
- **Documentation**: Auto-generated from specification

This organization enables maintainable, testable, and scalable code while following Go best practices and domain-driven design principles.