Merged in feature/track-filesize (pull request #200)

track file sizes for all documents in system * feature complete needs dev testing
2026-01-12 17:46:07 +00:00
parent 4ad8168f35
commit ebf47c6013
37 changed files with 1514 additions and 374 deletions
@@ -472,6 +472,120 @@ Queue Processing: < 1 second per message
 - Resource utilization trends
 ```

+## AWS IAM Requirements
+
+### Required S3 Permissions
+
+The document processing pipeline requires specific S3 permissions for file operations. Below are the minimum IAM permissions required for each service.
+
+#### docInitRunner Service
+
+The `docInitRunner` service requires `s3:GetObject` permission to measure file sizes during document initialization. This is used by the `HeadObject` operation to retrieve file metadata.
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "DocumentInitS3Permissions",
+      "Effect": "Allow",
+      "Action": [
+        "s3:GetObject",
+        "s3:PutObject",
+        "s3:HeadObject"
+      ],
+      "Resource": [
+        "arn:aws:s3:::${BUCKET_NAME}/*"
+      ]
+    },
+    {
+      "Sid": "DocumentInitBucketAccess",
+      "Effect": "Allow",
+      "Action": [
+        "s3:ListBucket"
+      ],
+      "Resource": [
+        "arn:aws:s3:::${BUCKET_NAME}"
+      ]
+    }
+  ]
+}
+```
+
+**Note**: The `s3:GetObject` permission is required for the `HeadObject` API call that measures file sizes. Without this permission, document initialization will fail with an access denied error.
+
+#### File Size Tracking (January 2026)
+
+Documents now track file size at initialization time. The `docInitRunner` service measures file size using S3 `HeadObject` before storing documents. This requires:
+
+- **Permission**: `s3:GetObject` on the document bucket
+- **Environment**: `BUCKET` environment variable must be set
+- **Error Handling**: If HeadObject fails, document creation will fail (not silently ignored)
+
+#### Compose File Configuration
+
+For local development with localstack, the bucket is configured via environment variables:
+
+```yaml
+# deployments/compose.local.yaml
+docInitRunner:
+  environment:
+    BUCKET: ${BUCKET_IN}  # Maps to the "documentin" bucket
+```
+
+For production deployments, ensure the IAM role attached to the service has the required S3 permissions above.
+
+### SQS Permissions
+
+Queue consumer services require SQS permissions:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "SQSConsumerPermissions",
+      "Effect": "Allow",
+      "Action": [
+        "sqs:ReceiveMessage",
+        "sqs:DeleteMessage",
+        "sqs:GetQueueAttributes",
+        "sqs:ChangeMessageVisibility"
+      ],
+      "Resource": [
+        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_init",
+        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_sync",
+        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_clean",
+        "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:document_text"
+      ]
+    }
+  ]
+}
+```
+
+### Textract Permissions
+
+For OCR text extraction, additional Textract permissions are required:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "TextractPermissions",
+      "Effect": "Allow",
+      "Action": [
+        "textract:StartDocumentTextDetection",
+        "textract:GetDocumentTextDetection",
+        "textract:StartDocumentAnalysis",
+        "textract:GetDocumentAnalysis"
+      ],
+      "Resource": "*"
+    }
+  ]
+}
+```
+
 ## Security Operations

 ### Security Monitoring