Merged in feature/aws-config-fix3 (pull request #184)

Make AWS variables optional * comment * aws vars optional
2025-09-17 18:12:51 +00:00
parent 6c209e5a81
commit 8a13b6d3b1
4 changed files with 101 additions and 20 deletions
@@ -1,10 +1,13 @@
 # CLAUDE.md

 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-## General rules  DO NOT VIOLATE
-NEVER code additional features that I did not explicitly ask for without asking permission. 
+
+## General rules DO NOT VIOLATE
+
+NEVER code additional features that I did not explicitly ask for without asking permission.
 NEVER create mock data or simplified components unless explicitly told to do so
 NEVER replace existing complex components with simplified versions - always fix the actual problem or the feature that was requested.
+Never just bypass tests that have regressed. Stop and find out why they are failing and fix them.
 ALWAYS work with the existing codebase - do not create new simplified alternatives
 ALWAYS find and fix the root cause of issues instead of creating workarounds
 When debugging issues, focus on fixing the existing implementation, not replacing it
@@ -13,11 +16,12 @@ Don’t build things I am not specifically asking for unless its required to bui
 When asked to make unit tests avoid mocks and use the actual thing that is being tested when reasonable.
 For commands that read such as grep or cat there is no need to ask for permission.

-## IDE related 
+## IDE related
+
 Always use ide diagnostics to validate code changes when running with ide integration (goland, vscode)

-
 ## linting all edits after every change
+
 Always run `task lint` once you think you are done with edits to verify your changes have not caused other issues.
 When running `task lint` output should always be clean. 0 errors.  
 Warnings about variable not set are ok.  
@@ -28,12 +32,14 @@ Use `task go:lint -- --fix` to fix go (file not formmatted) types of errors from
 ## Essential Commands

 ### Development Setup
- `devbox shell` - Enter development environment (Nix-based)  I will start you already in the devbox shell for the project if there is one. 
+
+- `devbox shell` - Enter development environment (Nix-based) I will start you already in the devbox shell for the project if there is one.
 - `touch .env` - Create environment variables file
- `task fullsuite:ci` - Complete development workflow (generate, build, lint, test) This must pass clean or its not working. If its output has the word `FAIL` then investigate what the issue is before calling the change complete.  
- when running fullsuite:ci make sure that you have stopped the containers that run from a previous `task compose:refresh' if this applies. (use task compose:down to clean those up first) 
+- `task fullsuite:ci` - Complete development workflow (generate, build, lint, test) This must pass clean or its not working. If its output has the word `FAIL` then investigate what the issue is before calling the change complete.
+- when running fullsuite:ci make sure that you have stopped the containers that run from a previous `task compose:refresh' if this applies. (use task compose:down to clean those up first)

 ### Core Development Tasks
+
 - `task generate` - Generate all code (DB queries via SQLC, OpenAPI clients/servers, mocks, docs)
 - `task build` - Build Docker image
  - `task lint` - Run all linting (Go, YAML, JSON, Docker, OpenAPI)
@@ -41,9 +47,8 @@ Use `task go:lint -- --fix` to fix go (file not formmatted) types of errors from
 - `task test:functional` - Run full test suite with 80% coverage threshold
 - `task precommit` - Pre-commit checks for CI/CD readiness

-
-
 ### Testing Commands
+
 - `task test:unit:short` - Quick unit tests
 - `task test:race` - Race condition detection
 - `task test:perf` - Performance analysis with slowest test reporting
@@ -51,11 +56,12 @@ Use `task go:lint -- --fix` to fix go (file not formmatted) types of errors from
 - `task test:bench` - Benchmark execution
 - full suite of tests is `task fullsuit:ci` and it must have this statement at the end of its output 'All coverage checks passed!'

-
 ## Architecture Overview

 ### System Design
+
 This is a **document processing and query orchestration platform** with microservices architecture using:
+
 - **Event-driven processing** via AWS SQS queues
 - **RESTful API** (queryAPI on port 8080) for external interactions
 - **Runner services** (ports 8081-8090) as queue consumers
@@ -63,6 +69,7 @@ This is a **document processing and query orchestration platform** with microser
 - **AWS services** (S3, SQS, Textract, Cognito) for cloud functionality

 ### Document Processing Pipeline
+
 1. **Upload** → S3 storage → storeEventRunner
 2. **Init** → Validation/deduplication → docInitRunner
 3. **Sync** → Client eligibility → docSyncRunner
@@ -72,6 +79,7 @@ This is a **document processing and query orchestration platform** with microser
 7. **Export** → Result generation

 ### Core Entities
+
 - **Client**: Customer configurations with document sets and permissions
 - **Collector**: Query collection specifications per client
 - **Query**: Data processing blocks with dependency chains
@@ -79,6 +87,7 @@ This is a **document processing and query orchestration platform** with microser
 - **Export**: Manual result output processes

 ### Service Architecture
+
 - **queryAPI** (8080): Main REST API with OpenAPI-generated handlers
 - **Runners** (8081-8090): Queue-based processing services implementing `Controller` interface
  - Each runner consumes from specific SQS queues
@@ -88,20 +97,25 @@ This is a **document processing and query orchestration platform** with microser
 ## Development Patterns

 ### Code Generation
+
 All code generation happens via `task generate`:
+
 - **SQLC**: Database queries from `internal/database/queries/*.sql`
 - **OpenAPI**: API clients/servers from `serviceAPIs/*.yaml`
 - **Mockery**: Test mocks for interfaces
 - **gomarkdoc**: Documentation generation

 ### Testing Strategy
+
 - **Testcontainers**: Docker-based integration testing for database/queue interactions
 - **80% coverage threshold** (60% function coverage) enforced
 - **Private test files**: Use `*private_test.go` for internal testing
 - **Mock interfaces**: Auto-generated in `mocks/` directory

 ### Project Structure
+
 Follows Go Standard Project Layout:
+
 - `cmd/`: Application entry points (main.go files)
 - `internal/`: Private application code
 - `api/`: Generated API server implementations
@@ -109,18 +123,22 @@ Follows Go Standard Project Layout:
 - `mocks/`: Auto-generated test mocks

 ### Naming Conventions
+
 - **Service**: Commands deployed as APIs (e.g., queryAPI)
 - **Runner**: Commands deployed as queue consumers (e.g., docTextRunner)
 - **API naming**: `<functionality>API` format
 - **Runner naming**: `<functionality>Runner` format

 ### Environment Configuration
+
 Environment variable hierarchy (first overrides subsequent):
+
 1. `devbox.json` "env" section
 2. `devbox.json` "init_hook" exports
 3. `.env` file variables

 ### Authentication & Authorization
+
 - **AWS Cognito**: JWT-based authentication
 - **Permit.io**: RBAC integration for fine-grained permissions
 - Middleware handles token validation and permission checking
@@ -128,6 +146,7 @@ Environment variable hierarchy (first overrides subsequent):
 ## Key Dependencies

 ### Core Technologies
+
 - **Go 1.24**: Primary language
 - **PostgreSQL**: Database with pgx driver
 - **Echo v4**: HTTP framework with middleware
@@ -135,6 +154,7 @@ Environment variable hierarchy (first overrides subsequent):
 - **AWS SDK v2**: Cloud service integration

 ### Code Quality Tools
+
 - **golangci-lint**: Go linting
 - **yamllint**: YAML validation
 - **Task**: Build automation
@@ -143,18 +163,21 @@ Environment variable hierarchy (first overrides subsequent):
 ## Quick Start for New Features

 ### Adding New API Endpoint
+
 1. Modify `serviceAPIs/queryAPI.yaml` OpenAPI specification
 2. Run `task generate` to regenerate API code
 3. Implement handlers in `api/queryAPI/`
 4. Add tests following existing patterns

 ### Adding New Queue Consumer
+
 1. Create `api/<runnerName>/runner.go` implementing `Controller` interface
 2. Create `cmd/<runnerName>/main.go` entry point
 3. Add queue configuration in `internal/serviceconfig/queue/`
 4. Add to docker-compose and deployment configurations

 ### Database Changes
+
 1. Add migration files in `internal/database/migrations/`
 2. Add queries in `internal/database/queries/`
 3. Run `task db:generate` to regenerate Go code
@@ -1,5 +1,6 @@
 # Query Orchestration

+AWS ambient version
 This repository contains the DoczyAI code.

 Using the following project as a baseline: `https://github.com/golang-standards/project-layout`.
@@ -28,12 +29,14 @@ For regular usage, please use `task` scripts in order run the most appropriate c
 ## Available Task Commands

 ### Core Development Workflow
+
 - **`task fullsuite:ci`** - Complete CI workflow: generate, build, lint, test with coverage
 - **`task fullsuite`** - Full development suite (similar to CI but for local development)
 - **`task fullsuite:graph`** - Generate dependency graph visualization
 - **`task precommit`** - Pre-commit checks for CI/CD readiness

 ### Code Generation
+
 - **`task generate`** - Generate all code (DB queries via SQLC, OpenAPI clients/servers, mocks, docs)
 - **`task go:generate`** - Generate Go-specific code
 - **`task openapi:generate`** - Generate OpenAPI clients and servers from specs
@@ -43,38 +46,46 @@ For regular usage, please use `task` scripts in order run the most appropriate c
 - **`task test:mocks:generate`** - Generate test mocks

 ### Build Commands
+
 - **`task build`** - Build the project
 - **`task docker:build`** - Build Docker images
 - **`task docker:build:debug`** - Build Docker images with debug support

 ### Docker Compose Management
+
 #### Build Commands
+
 - **`task compose:build`** - Build Docker images using local compose file
 - **`task compose:build:test`** - Build Docker images using test compose file
 - **`task compose:build:generate`** - Build Docker images using generate compose file

 #### Start/Up Commands
+
 - **`task compose:up`** - Full local development startup with AWS resource initialization
 - **`task compose:up:test`** - Start test environment containers
 - **`task compose:up:generate`** - Start containers for code generation
 - **`task compose:up:aws`** - Start AWS-specific environment

 #### Stop/Down Commands
+
 - **`task compose:down`** - Stop and remove local containers
 - **`task compose:down:test`** - Stop and remove test containers
 - **`task compose:down:aws`** - Stop and remove AWS containers

 #### Cleanup Commands
+
 - **`task compose:clean`** - Stop containers and remove volumes (local)
 - **`task compose:clean:test`** - Stop containers and remove volumes (test)
 - **`task compose:clean:aws`** - Stop containers and remove volumes (AWS)

 #### Other Compose Commands
+
 - **`task compose:refresh`** - Full restart: rebuild everything and start fresh
 - **`task compose:rebuild`** - Force complete rebuild of all containers (use when build cache causes issues)
 - **`task compose:lint`** - Validate syntax of all compose files

 ### Testing Commands
+
 - **`task test:functional`** - Run functional tests with 80% coverage threshold
 - **`task test:functional:full`** - Run all tests and reset coverage tracking
 - **`task test:unit:short`** - Quick unit tests
@@ -89,6 +100,7 @@ For regular usage, please use `task` scripts in order run the most appropriate c
 - **`task test:wait`** - Wait for test dependencies

 ### Linting Commands
+
 - **`task lint`** - Run all linting (Go, YAML, JSON, Docker, OpenAPI)
 - **`task go:lint`** - Run Go-specific linting
 - **`task go:lint -- --fix`** - Fix automatically fixable Go linting issues
@@ -99,14 +111,17 @@ For regular usage, please use `task` scripts in order run the most appropriate c
 - **`task db:lint`** - Lint database-related files

 ### Database Commands
+
 - **`task db:mig:create`** - Create new database migration
 - **`task db:generate`** - Generate Go code from SQL queries

 ### AWS Commands
+
 - **`task aws:login`** - Login to AWS services
 - **`task aws:textract:credentials`** - Set up AWS Textract credentials

 ### Utility Commands
+
 - **`task deps:tidy`** - Tidy Go module dependencies

 ## Environment Variables
@@ -124,20 +139,23 @@ When using environment variables, there is a hierarchy that will be respected. (
 The following environment variables are required for the application to function properly:

 #### Core Application
+
 ```bash
 LOG_LEVEL=INFO                          # Logging level: DEBUG|INFO|WARN|ERROR
 ```

 #### Database (Required)
+
 ```bash
 PGHOST=localhost                        # Database host
-PGPORT=5432                            # Database port  
+PGPORT=5432                            # Database port
 PGDATABASE=query_orchestration         # Database name
 PGUSER=postgres                        # Database username
 PGPASSWORD=your_password               # Database password
 ```

 #### AWS Configuration (Required)
+
 ```bash
 AWS_REGION=us-east-1                   # AWS region
 AWS_ACCESS_KEY_ID=your_access_key      # AWS access key ID
@@ -146,11 +164,13 @@ AWS_SESSION_TOKEN=your_session_token   # AWS session token (if using temporary c
 ```

 #### S3 Storage (Required)
+
 ```bash
 BUCKET=your-s3-bucket-name             # S3 bucket for document storage
 ```

 #### SQS Queue URLs (All Required)
+
 ```bash
 QUEUE_URL=https://sqs.region.amazonaws.com/account/queue-name          # Main query processing queue
 DOCUMENT_INIT_URL=https://sqs.region.amazonaws.com/account/doc-init    # Document initialization queue
@@ -164,6 +184,7 @@ QUERY_VERSION_SYNC_URL=https://sqs.region.amazonaws.com/account/query-version-sy
 ```

 #### Authentication (Required for Production)
+
 ```bash
 COGNITO_USER_POOL_ID=us-east-1_xxxxxxx # AWS Cognito User Pool ID
 COGNITO_CLIENT_ID=your_client_id       # AWS Cognito App Client ID
@@ -172,14 +193,16 @@ COGNITO_DOMAIN=your-cognito-domain     # AWS Cognito domain name
 ```

 #### Authorization (Required for RBAC)
+
 ```bash
 PERMIT_IO_API_KEY=permit_key_xxxxxxx   # Permit.io API key for RBAC
-PERMIT_IO_PDP_URL=http://localhost:7766 # Permit.io Policy Decision Point URL - can use cloud version optionally 
+PERMIT_IO_PDP_URL=http://localhost:7766 # Permit.io Policy Decision Point URL - can use cloud version optionally
 ```

 ### Optional Environment Variables

 #### Development & Testing
+
 ```bash
 DEBUG=true                             # Enable debug mode (development only)
 DISABLE_AUTH=true                      # Bypass authentication (development/testing only)
@@ -187,18 +210,21 @@ DB_NOSSL=true                         # Disable SSL for database connections (de
 ```

 #### AWS LocalStack (Development)
+
 ```bash
 AWS_ENDPOINT_URL=http://localhost:4566 # Override AWS endpoints for LocalStack
 AWS_S3_USE_PATH_STYLE=true            # Use path-style S3 URLs for LocalStack
 ```

 #### Server Configuration
+
 ```bash
 PORT=8080                             # HTTP server port (default: 8080)
 HTTP_HOST=0.0.0.0                     # HTTP server host (default: 0.0.0.0)
 ```

 #### Observability
+
 ```bash
 ENABLE_OTEL=false                     # Enable OpenTelemetry (default: false)
 ```
@@ -273,8 +299,8 @@ This will:

 1. Add a file named `<api_name>.yml`. The api name is important, as it will be used as a reference throughout the codebase. The api name must be in the format `<functionality>API`. E.g. queryAPI, ClientAPI
 2. Run `task openapi:generate`. This will generate the following:
-  a. A location to implement the server-side functions in `./api/<api_name>/`, as well as generated functions, models and swagger docs.
-  b. A location with the client-side code in `./pkg/<api_name>/`
+   a. A location to implement the server-side functions in `./api/<api_name>/`, as well as generated functions, models and swagger docs.
+   b. A location with the client-side code in `./pkg/<api_name>/`
 3. Implement the controllers in `./api/<api_name>/`
 4. Create a command in `./cmd/<api_name>/main.go` which creates a new instance from `./internal/api`

@@ -295,6 +321,7 @@ The project includes two build tasks for different use cases:
 - `task docker:build:debug` - Build the debug image with Delve

 The debug image is built using `build/Dockerfile.debug`, which:
+
 - Installs Delve debugger with static linking for Alpine compatibility
 - Builds Go binaries with debug symbols (`-gcflags="all=-N -l"`)
 - Exposes port 2345 for debugger connections
@@ -305,17 +332,28 @@ To enable debugging for a service in `deployments/compose.local.yaml`, modify th

 ```yaml
 query_api:
-  image: queryorchestration:debug  # Use debug image instead of latest
-  command: ["/dlv", "exec", "./queryAPI", "--listen=:2345", "--headless", "--api-version=2", "--accept-multiclient", "--continue"]
+  image: queryorchestration:debug # Use debug image instead of latest
+  command:
+    [
+      "/dlv",
+      "exec",
+      "./queryAPI",
+      "--listen=:2345",
+      "--headless",
+      "--api-version=2",
+      "--accept-multiclient",
+      "--continue",
+    ]
  ports:
-    - "8080:8080"  # Application port
-    - "2345:2345"  # Debugger port
+    - "8080:8080" # Application port
+    - "2345:2345" # Debugger port
  expose:
    - 8080
    - 2345
 ```

 Key changes from production configuration:
+
 1. **Image**: Use `queryorchestration:debug` instead of `queryorchestration:latest`
 2. **Command**: Replace direct binary execution with Delve wrapper
 3. **Ports**: Expose port 2345 for debugger connections
@@ -328,6 +366,7 @@ Key changes from production configuration:
 ### Connecting a Debugger

 #### GoLand/IntelliJ IDEA
+
 1. Go to **Run → Edit Configurations...**
 2. Click **"+"** → **"Go Remote"**
 3. Configure:
@@ -339,7 +378,9 @@ Key changes from production configuration:
 6. Set breakpoints in your code and make HTTP requests to trigger them

 #### VS Code
+
 1. Add to `.vscode/launch.json`:
+
 ```json
 {
  "name": "Connect to Docker",
@@ -351,6 +392,7 @@ Key changes from production configuration:
  "host": "localhost"
 }
 ```
+
 2. Set breakpoints and start debugging

 The debugger will connect to the running container and pause execution when breakpoints are hit, allowing you to inspect variables, step through code, and debug issues in the containerized environment.
@@ -362,11 +404,13 @@ The system uses a structured naming convention for S3 objects that embeds metada
 ### Object Key Structure

 S3 object keys follow this format:
+
 ```
 {client_id}/{location}/{date}/{part}/{filename}
 ```

 #### Example
+
 ```
 test_client_1757026403/import/20250904/0/2025-09-04T225324Z~test_client_1757026403~import~56df8aa9-873c-48ea-a0e4-bc4f6b8ad69e~019916ef-46ab-73f7-b97b-5709df5c3985
 ```
@@ -374,13 +418,16 @@ test_client_1757026403/import/20250904/0/2025-09-04T225324Z~test_client_17570264
 ### Components

 #### Directory Path (Prefix)
+
 - **Client ID**: Unique identifier for the client (e.g., `test_client_1757026403`)
 - **Location**: Processing stage - `import`, `text`, or `export`
 - **Date**: Date in `YYYYMMDD` format (e.g., `20250904`)
 - **Part**: Partition number for storage distribution (e.g., `0`)

 #### Filename
+
 The filename uses tilde (`~`) as a delimiter and contains:
+
 - **Timestamp**: Full ISO timestamp (e.g., `2025-09-04T225324Z`)
 - **Client ID**: Repeated for self-contained metadata
 - **Location**: Repeated for self-contained metadata
@@ -389,15 +436,19 @@ The filename uses tilde (`~`) as a delimiter and contains:
 - **File Extension**: Optional file type extension (e.g., `.pdf`)

 ### Location Types
+
 - **import**: Initial document upload location
 - **text**: Text extraction results storage
 - **export**: Export results storage

 ### Partitioning Strategy
+
 The part number (`0-n`) is used to distribute files across multiple directories, preventing performance issues with too many objects in a single S3 prefix. Export location does not use partitioning.

 ### Metadata Encoding
+
 The filename contains all necessary metadata for document recovery and processing:
+
 - Self-contained recovery capability from filename alone
 - Support for both path-based and filename-based queries
 - Human-readable format for debugging and troubleshooting
@@ -11,8 +11,8 @@ import (
 type Profile string

 type AWSConfig struct {
-	AWSKeyID         string  `env:"AWS_ACCESS_KEY_ID,required,notEmpty"`
-	AWSSecretKey     string  `env:"AWS_SECRET_ACCESS_KEY,required,notEmpty"`
+	AWSKeyID         string  `env:"AWS_ACCESS_KEY_ID"`
+	AWSSecretKey     string  `env:"AWS_SECRET_ACCESS_KEY"`
 	AWSRegion        string  `env:"AWS_REGION,required,notEmpty"`
 	AWSDefaultRegion string  `env:"AWS_DEFAULT_REGION"`
 	AWSSessionToken  string  `env:"AWS_SESSION_TOKEN"`
@@ -164,6 +164,13 @@ func initializeConfigPrivate(cfg ConfigProvider) error {
 	// Reinitialize with environment level
 	cfg.SetDefaultLogger()

+	// Log AWS credential method being used
+	if cfg.GetAWSKeyID() != "" {
+		slog.Info("Using explicit AWS credentials from environment variables")
+	} else {
+		slog.Info("Using AWS credential chain (IAM role, instance profile, etc.)")
+	}
+
 	return initErr
 }