Merged in feature/permit-policy-cleanup (pull request #210)

cleanup permit policies and correct documentation

* docs

* policy cleanup

* refactor

* Merge remote-tracking branch 'origin/feature/permit-policy-cleanup' into feature/permit-policy-cleanup

* docs and edits

docs/ai.generated/01-system-overview.md

@@ -3,12 +3,11 @@
 ## DoczyAI Document Processing Platform
 
 ### Purpose and Domain
-DoczyAI is a cloud-native document processing platform designed to extract structured information from documents (primarily PDFs). The system serves as a multi-tenant platform where organizations can upload documents, perform text extraction, and organize documents with folders and labels for business intelligence and analysis.
+DoczyAI is a cloud-native document processing platform for multi-tenant PDF ingestion and processing. Organizations can upload and organize documents with folders and labels, and store structured field extraction results for downstream analysis.
 
 ### Core Business Capabilities
 - **Multi-tenant Document Processing**: Isolated processing environments per client
 - **Batch Document Processing**: ZIP archive uploads for processing multiple documents simultaneously
-- **Text Extraction Pipeline**: Automated OCR and text extraction from documents
 - **Automated Processing Pipeline**: Seamless flow from document upload to structured output
 - **Version Management**: Complete versioning for configurations
 - **Export and Integration**: Structured data export capabilities for downstream systems
@@ -20,7 +19,6 @@ DoczyAI is a cloud-native document processing platform designed to extract struc
 - **Database**: PostgreSQL 17.2 with SQLC for type-safe queries
 - **Messaging**: AWS SQS for event-driven microservices
 - **Storage**: AWS S3 for document storage
-- **Text Processing**: AWS Textract for OCR and text extraction
 - **Authentication**: AWS Cognito with JWT-based authorization
 - **Monitoring**: Prometheus metrics with 12 standard metric types and OpenTelemetry tracing
 - **Authorization**: Permit.io RBAC integration for fine-grained permissions
@@ -30,7 +28,8 @@ DoczyAI is a cloud-native document processing platform designed to extract struc
 
 ### System Architecture Pattern
 The system follows an **event-driven microservices architecture** with the following characteristics:
-- **8 distinct services**: 1 HTTP API + 6 queue-based runners + 1 utility service
+- **Active services**: 1 HTTP API + 5 active queue runners
+- **Compatibility services**: Additional deprecated runners may remain in some deployments
 - **Asynchronous communication**: SQS queues between services with batch processing capabilities
 - **Background Task Framework**: Comprehensive background task execution with periodic scheduling
 - **State management**: PostgreSQL as central data store with enhanced schema for batch operations
@@ -55,7 +54,6 @@ graph TB
         SE[storeEventRunner<br/>8081] --> DI[docInitRunner<br/>8082]
         DI --> DS[docSyncRunner<br/>8083]
         DS --> DC[docCleanRunner<br/>8084]
-        DC --> DT[docTextRunner<br/>8085]
     end
 
     subgraph "Synchronization Services"
@@ -66,7 +64,6 @@ graph TB
         DB[(PostgreSQL<br/>Database)]
         S3[(AWS S3<br/>Storage)]
         SQS[AWS SQS<br/>Queues]
-        TEXTRACT[AWS Textract<br/>Text Extraction]
         COGNITO[AWS Cognito<br/>Authentication]
         PROM[Prometheus<br/>Monitoring]
     end
@@ -83,7 +80,6 @@ graph TB
     DI --> DB
     DS --> DB
     DC --> S3
-    DT --> TEXTRACT
 
     CS --> SQS
     
@@ -95,7 +91,7 @@ graph TB
 ### Infrastructure Requirements
 - **Compute**: Container orchestration platform (Docker/Kubernetes)
 - **Database**: PostgreSQL 17.2+ with connection pooling
-- **AWS Services**: S3, SQS, Textract, Cognito services
+- **AWS Services**: S3, SQS, Cognito services
 - **Monitoring**: Prometheus-compatible metrics collection
 - **Networking**: Load balancer for API endpoints
 
@@ -117,18 +113,17 @@ graph TB
 ### External System Dependencies
 1. **AWS S3**: Document storage and retrieval
 2. **AWS SQS**: Inter-service messaging and event processing
-3. **AWS Textract**: OCR and text extraction from documents
-4. **AWS Cognito**: User authentication and session management
-5. **PostgreSQL**: Central data persistence
+3. **AWS Cognito**: User authentication and session management
+4. **PostgreSQL**: Central data persistence
 
 ### API Integration
 - **REST API**: OpenAPI 3.0.3 specification with Swagger UI
 - **Authentication**: Bearer token or OAuth2 flows
-- **Rate Limiting**: 100 requests per 60-second window
+- **Rate Limiting**: Dual-layer global and per-IP rate limiting (configurable via environment variables)
 - **Content Types**: JSON for API, multipart/form-data for uploads
 
 ### Performance Characteristics
 - **Throughput**: Designed for high-volume document processing
 - **Latency**: Sub-second API response times for most operations
 - **Availability**: Distributed architecture supports high availability
-- **Durability**: S3 and RDS provide 99.999999999% (11 9's) and 99.95% durability respectively
+- **Durability**: S3 and RDS provide 99.999999999% (11 9's) and 99.95% durability respectively