jacob-mathison/query-orchestration
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

batch upload impl

Browse Source
This commit is contained in:
jay brown
2025-08-06 15:06:18 -07:00
parent fac5615c15
commit 7014fa790b
17 changed files with 1272 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files
api/queryAPI/README.md
+119
View File
@@ -0,0 +1,119 @@
# queryAPI Package
This directory contains the REST API implementation for the Query Orchestration platform. The code here serves as the HTTP interface layer between external clients and the internal business logic.
## Architecture Overview
### Controllers Struct (controllers.go)
The `Controllers` struct is the central architectural component that implements the **Service Layer/Facade pattern** for the entire REST API.
#### Purpose
- **OpenAPI Contract Implementation**: Implements the `ServerInterface` interface generated from `serviceAPIs/queryAPI.yaml`
- **Service Aggregation**: Consolidates 11 different business services into a single, cohesive API layer
- **Dependency Injection Container**: Provides clean separation between HTTP handlers and business logic
#### Structure
```go
type Controllers struct {
svc *Services // Aggregated business services
cfg ConfigProvider // Combined auth & object storage configuration
}
```
#### Pattern Benefits
1. **Single Responsibility**: Each handler method focuses solely on HTTP↔Service translation
2. **Testability**: Services can be easily mocked for unit testing
3. **Modularity**: New services can be added without changing the controller pattern
4. **Type Safety**: Ensures compile-time checking against OpenAPI specification
## Key Architectural Components
### 1. Generated Code (api.gen.go)
- **DO NOT EDIT**: This file is auto-generated from `serviceAPIs/queryAPI.yaml`
- Contains all request/response types and the `ServerInterface` interface
- Regenerated via `task generate` command
- Ensures API contract consistency
### 2. Handler Files
Each file implements specific domain endpoints:
- `client.go` - Client management operations
- `collector.go` - Query collector configuration
- `documents.go` - Document upload and management
- `export.go` - Export operations
- `query.go` - Query execution and management
- `status.go` - Health check and status endpoints
- `authHandlers.go` - Authentication flow handlers (login, logout, callbacks)
### 3. Service Dependencies
The Controllers struct aggregates these internal services:
- **Export Service** - Manages export processes
- **Collector Service** - Handles query collections
- **Query Service** - Query execution logic
- **Client Service** - Client entity management
- **Document Service** - Document processing
- **DocumentUpload Service** - S3 upload handling
- **DocumentBatch Service** - Batch document operations
### 4. Authentication Integration
- Uses AWS Cognito for JWT-based authentication
- Permit.io integration for RBAC (Role-Based Access Control)
- Authentication middleware validates tokens before reaching handlers
- `ConfigProvider` interface provides auth configuration access
## Handler Pattern
All handlers follow a consistent pattern:
```go
func (s *Controllers) HandlerName(ctx echo.Context, params...) error {
// 1. Bind/validate request data
req := RequestType{}
if err := ctx.Bind(&req); err != nil {
return echo.NewHTTPError(http.StatusBadRequest, err)
}
// 2. Call appropriate service method
result, err := s.svc.ServiceName.Method(ctx.Request().Context(), ...)
if err != nil {
return echo.NewHTTPError(http.StatusBadRequest, "Error message")
}
// 3. Return formatted response
return ctx.JSON(http.StatusOK, ResponseType{...})
}
```
## Testing Structure
- `*_test.go` files provide unit tests for each handler group
- `test_helpers.go` contains shared test utilities and fixtures
- Tests use mocked services to validate handler logic in isolation
- Integration tests exist at the service layer level
## Important Notes for Maintenance
1. **OpenAPI-First Development**: Always update `serviceAPIs/queryAPI.yaml` first, then regenerate code
2. **Service Layer Separation**: Business logic belongs in `internal/` services, not in handlers
3. **Error Handling**: Use Echo's `HTTPError` for consistent error responses
4. **Context Propagation**: Always pass `ctx.Request().Context()` to service methods for proper cancellation
5. **No Direct Database Access**: All data operations must go through service layer
6. **Consistent Status Codes**: Follow REST conventions (201 for creation, 200 for success, 400 for client errors)
## Development Workflow
1. Modify OpenAPI spec in `serviceAPIs/queryAPI.yaml`
2. Run `task generate` to update generated code
3. Implement/update handler methods in Controllers
4. Add/update tests for new functionality
5. Run `task lint` to ensure code quality
6. Run `task test:functional` to validate changes
## Configuration
The `ConfigProvider` interface combines:
- Authentication configuration (Cognito settings, JWT validation)
- Object storage configuration (S3 bucket settings)
- Base service configuration (ports, logging, database)
This unified configuration is injected into Controllers at startup, providing all necessary runtime settings.
api/queryAPI/authHandlers.go
+6 -6
View File
@@ -28,20 +28,20 @@ func (s *Controllers) Logout(ctx echo.Context) error {
// the middleware will have already removed the auth cookie
// Construct the Cognito logout URL with proper parameters
logoutURL := s.authConfig.GetAuthLogoutURL()
clientID := s.authConfig.GetAuthClientID()
logoutURL := s.cfg.GetAuthLogoutURL()
clientID := s.cfg.GetAuthClientID()
// The logout_uri should redirect back to our home page
// Use the same base URL that was used for redirect URI setup
baseURL := ctx.Scheme() + "://" + ctx.Request().Host
logoutRedirectURI := baseURL + s.authConfig.GetAuthHomePath()
logoutRedirectURI := baseURL + s.cfg.GetAuthHomePath()
// Build the full Cognito logout URL with query parameters
parsedURL, err := url.Parse(logoutURL)
if err != nil {
s.authConfig.GetAuthLogger().Error("Failed to parse logout URL", "error", err)
s.cfg.GetAuthLogger().Error("Failed to parse logout URL", "error", err)
// Fallback to simple redirect to home
return ctx.Redirect(http.StatusFound, s.authConfig.GetAuthHomePath())
return ctx.Redirect(http.StatusFound, s.cfg.GetAuthHomePath())
}
query := parsedURL.Query()
@@ -51,7 +51,7 @@ func (s *Controllers) Logout(ctx echo.Context) error {
cognitoLogoutURL := parsedURL.String()
s.authConfig.GetAuthLogger().Debug("Redirecting to Cognito logout",
s.cfg.GetAuthLogger().Debug("Redirecting to Cognito logout",
"logout_url", cognitoLogoutURL,
"client_id", clientID,
"logout_uri", logoutRedirectURI)
api/queryAPI/controllers.go
+13 -6
View File
@@ -13,6 +13,7 @@ import (
querytest "queryorchestration/internal/query/test"
queryupdate "queryorchestration/internal/query/update"
"queryorchestration/internal/serviceconfig/auth"
"queryorchestration/internal/serviceconfig/objectstore"
)
const Name = "queryAPI"
@@ -31,14 +32,20 @@ type Services struct {
DocumentBatch *documentbatch.Service
}
type Controllers struct {
svc *Services
authConfig auth.ConfigProvider
// ConfigProvider combines auth and objectstore interfaces for the Controllers
type ConfigProvider interface {
auth.ConfigProvider
objectstore.ConfigProvider
}
func NewControllers(services *Services, authConfig auth.ConfigProvider) *Controllers {
type Controllers struct {
svc *Services
cfg ConfigProvider
}
func NewControllers(services *Services, cfg ConfigProvider) *Controllers {
return &Controllers{
svc: services,
authConfig: authConfig,
svc: services,
cfg: cfg,
}
}
api/queryAPI/documents.go
+55 -32
View File
@@ -5,6 +5,7 @@ import (
"log/slog"
"net/http"
"queryorchestration/internal/document/batch/storage"
documentupload "queryorchestration/internal/document/upload"
"github.com/google/uuid"
@@ -128,48 +129,70 @@ func (s *Controllers) ListDocumentBatches(ctx echo.Context, clientId ClientID, p
// UploadDocumentBatch handles ZIP archive upload containing multiple PDFs
func (s *Controllers) UploadDocumentBatch(ctx echo.Context, clientId ClientID) error {
// Get the multipart form
form, err := ctx.MultipartForm()
// Parse multipart form to get ZIP file
file, err := ctx.FormFile("archive")
if err != nil {
slog.Error("unable to get form data", "error", err)
return echo.NewHTTPError(http.StatusBadRequest, "unable to get form data")
slog.Error("failed to get uploaded file", "error", err, "client_id", clientId)
return echo.NewHTTPError(http.StatusBadRequest, "file upload required")
}
// Get the archive file
files := form.File["archive"]
if len(files) == 0 {
return echo.NewHTTPError(http.StatusBadRequest, "missing archive file")
} else if len(files) > 1 {
return echo.NewHTTPError(http.StatusBadRequest, "too many files")
}
file := files[0]
// Skip ZIP validation for tests - in production this would check file content
// if file.Header.Get("Content-Type") != "application/zip" {
// return echo.NewHTTPError(http.StatusBadRequest, "file must be a ZIP archive")
// }
// For now, we'll create a batch record with placeholder total documents
// In the real implementation, we would:
// 1. Store the ZIP file to S3
// 2. Start an async process to extract and process PDFs
// 3. Return immediately with 202 Accepted
// Create batch record
batchID, err := s.svc.DocumentBatch.Create(ctx.Request().Context(), clientId, file.Filename, 0)
// Open uploaded file
src, err := file.Open()
if err != nil {
slog.Error("unable to create batch", "error", err)
return echo.NewHTTPError(http.StatusInternalServerError, "unable to create batch")
slog.Error("failed to open uploaded file", "error", err, "client_id", clientId)
return echo.NewHTTPError(http.StatusInternalServerError, "failed to process uploaded file")
}
defer src.Close()
// Initialize storage service with objectstore config
storageService := storage.New(s.cfg)
// Validate ZIP file
err = storageService.ValidateZIPFile(file.Filename, file.Size)
if err != nil {
slog.Error("ZIP validation failed", "filename", file.Filename, "size", file.Size, "error", err, "client_id", clientId)
return echo.NewHTTPError(http.StatusBadRequest, fmt.Sprintf("invalid ZIP file: %v", err))
}
// Build response
// Store ZIP file to S3
storageResult, err := storageService.StoreZIPFile(ctx.Request().Context(), string(clientId), file.Filename, src)
if err != nil {
slog.Error("failed to store ZIP file", "error", err, "client_id", clientId)
return echo.NewHTTPError(http.StatusInternalServerError, "failed to store ZIP file")
}
// Create batch record with storage metadata
// Note: totalDocs is unknown at this point, will be determined during extraction in later parts
batchID, err := s.svc.DocumentBatch.CreateWithStorage(
ctx.Request().Context(),
string(clientId),
file.Filename,
0, // totalDocs unknown in Part 1
storageResult.Bucket,
storageResult.Key,
storageResult.SizeBytes,
)
if err != nil {
slog.Error("failed to create batch record", "error", err, "client_id", clientId)
return echo.NewHTTPError(http.StatusInternalServerError, "failed to create batch record")
}
slog.Info("ZIP batch upload completed",
"batch_id", batchID,
"filename", file.Filename,
"size_bytes", storageResult.SizeBytes,
"s3_key", storageResult.Key,
"client_id", clientId,
)
// Return 202 Accepted with batch ID and status URL
statusUrl := fmt.Sprintf("/clients/%s/documents/batches/%s", clientId, batchID)
response := BatchUploadResponse{
BatchId: BatchID(batchID),
Status: BatchStatusProcessing,
StatusUrl: fmt.Sprintf("/client/%s/document/batch/%s", clientId, batchID.String()),
Status: "processing",
StatusUrl: statusUrl,
}
// Return 202 Accepted for async processing
return ctx.JSON(http.StatusAccepted, response)
}
api/queryAPI/documents_test.go
+238 -5
View File
@@ -1,19 +1,27 @@
package queryapi_test
import (
"archive/zip"
"bytes"
"encoding/json"
"fmt"
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"path/filepath"
"testing"
"time"
"queryorchestration/internal/database/repository"
"queryorchestration/internal/test"
queryapi "queryorchestration/api/queryAPI"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/google/uuid"
"github.com/labstack/echo/v4"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
queryapi "queryorchestration/api/queryAPI"
"queryorchestration/internal/database/repository"
"queryorchestration/internal/test"
)
func TestUploadDocument(t *testing.T) {
@@ -175,6 +183,7 @@ func TestUploadDocumentBatch(t *testing.T) {
cfg := &ControllerConfig{}
test.CreateDB(t, cfg)
initializeTestConfig(t, cfg)
test.CreateAWSResources(t, cfg)
svc := createControllerServices(cfg)
cons := queryapi.NewControllers(svc, cfg)
@@ -283,3 +292,227 @@ func TestCancelDocumentBatch(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, repository.BatchStatusCancelled, batch.Status)
}
func TestUploadDocumentBatch_Part1_Success(t *testing.T) {
// Setup: Create testcontainers (database + localstack)
cfg := &ControllerConfig{}
test.CreateDB(t, cfg)
initializeTestConfig(t, cfg)
test.CreateAWSResources(t, cfg)
svc := createControllerServices(cfg)
cons := queryapi.NewControllers(svc, cfg)
// Create test client with unique ID
clientID := fmt.Sprintf("test_client_part1_%d", time.Now().UnixNano())
err := cfg.GetDBQueries().CreateClient(t.Context(), &repository.CreateClientParams{
Clientid: clientID,
Name: "Test Client Part1",
})
require.NoError(t, err)
// Create test ZIP file with 3 PDFs (test1.pdf, test2.pdf, test3.pdf)
zipContent := createTestZIPFile(t, 3)
// Create multipart form request
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("archive", "test-batch.zip")
require.NoError(t, err)
_, err = io.Copy(part, zipContent)
require.NoError(t, err)
err = writer.Close()
require.NoError(t, err)
// Test: Upload ZIP batch
e := echo.New()
rec := httptest.NewRecorder()
ctx := e.NewContext(
httptest.NewRequest(http.MethodPost, "/", body),
rec,
)
ctx.Request().Header.Set("Content-Type", writer.FormDataContentType())
err = cons.UploadDocumentBatch(ctx, clientID)
require.NoError(t, err)
// Verify: HTTP response
assert.Equal(t, http.StatusAccepted, rec.Code)
var response queryapi.BatchUploadResponse
err = json.Unmarshal(rec.Body.Bytes(), &response)
require.NoError(t, err)
assert.NotEmpty(t, response.BatchId)
assert.Equal(t, queryapi.BatchStatusProcessing, response.Status)
assert.NotEmpty(t, response.StatusUrl)
// Verify: Batch exists in database with storage metadata
batch, err := cfg.GetDBQueries().GetBatchUploadWithStorage(t.Context(), &repository.GetBatchUploadWithStorageParams{
ID: uuid.UUID(response.BatchId),
ClientID: clientID,
})
require.NoError(t, err)
assert.Equal(t, "test-batch.zip", batch.OriginalFilename)
assert.Equal(t, repository.BatchStatusProcessing, batch.Status)
assert.NotNil(t, batch.ArchiveBucket)
assert.NotEmpty(t, *batch.ArchiveBucket)
assert.NotNil(t, batch.ArchiveKey)
assert.NotEmpty(t, *batch.ArchiveKey)
assert.Contains(t, *batch.ArchiveKey, clientID)
assert.Contains(t, *batch.ArchiveKey, "test-batch.zip")
assert.NotNil(t, batch.FileSizeBytes)
assert.Greater(t, *batch.FileSizeBytes, int64(0))
// Verify: ZIP file exists in S3
s3Client := cfg.GetStoreClient()
_, err = s3Client.HeadObject(t.Context(), &s3.HeadObjectInput{
Bucket: batch.ArchiveBucket,
Key: batch.ArchiveKey,
})
require.NoError(t, err, "ZIP file should exist in S3")
}
func TestUploadDocumentBatch_Part1_InvalidFile(t *testing.T) {
cfg := &ControllerConfig{}
test.CreateDB(t, cfg)
initializeTestConfig(t, cfg)
svc := createControllerServices(cfg)
cons := queryapi.NewControllers(svc, cfg)
// Create test client with unique ID
clientID := fmt.Sprintf("test_client_invalid_%d", time.Now().UnixNano())
err := cfg.GetDBQueries().CreateClient(t.Context(), &repository.CreateClientParams{
Clientid: clientID,
Name: "Test Client Invalid",
})
require.NoError(t, err)
// Create invalid file (not ZIP)
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("archive", "test.pdf")
require.NoError(t, err)
_, err = part.Write([]byte("not a zip file"))
require.NoError(t, err)
err = writer.Close()
require.NoError(t, err)
// Test: Upload invalid file
ctx := echo.New().NewContext(
httptest.NewRequest(http.MethodPost, "/", body),
httptest.NewRecorder(),
)
ctx.Request().Header.Set("Content-Type", writer.FormDataContentType())
err = cons.UploadDocumentBatch(ctx, clientID)
// Verify: Error response
require.Error(t, err)
httpErr, ok := err.(*echo.HTTPError)
require.True(t, ok)
assert.Equal(t, http.StatusBadRequest, httpErr.Code)
assert.Contains(t, httpErr.Message, "invalid file type")
}
func TestUploadDocumentBatch_Part1_TooLarge(t *testing.T) {
cfg := &ControllerConfig{}
test.CreateDB(t, cfg)
initializeTestConfig(t, cfg)
svc := createControllerServices(cfg)
cons := queryapi.NewControllers(svc, cfg)
// Create test client with unique ID
clientID := fmt.Sprintf("test_client_toolarge_%d", time.Now().UnixNano())
err := cfg.GetDBQueries().CreateClient(t.Context(), &repository.CreateClientParams{
Clientid: clientID,
Name: "Test Client TooLarge",
})
require.NoError(t, err)
// Create oversized ZIP file
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
// Set a fake Content-Length header to simulate large file
part, err := writer.CreateFormFile("archive", "huge.zip")
require.NoError(t, err)
// Write minimal content but set file size in header
_, err = part.Write([]byte("PK")) // ZIP file header
require.NoError(t, err)
err = writer.Close()
require.NoError(t, err)
// Test: Upload with modified request to simulate large file
req := httptest.NewRequest(http.MethodPost, "/", body)
req.Header.Set("Content-Type", writer.FormDataContentType())
// Override the file size in the form data
ctx := echo.New().NewContext(req, httptest.NewRecorder())
// Manually create form with large file size
form, _ := ctx.MultipartForm()
if len(form.File["archive"]) > 0 {
// Simulate 101MB file
form.File["archive"][0].Size = 101 * 1024 * 1024
}
err = cons.UploadDocumentBatch(ctx, clientID)
// Verify: Error response
require.Error(t, err)
httpErr, ok := err.(*echo.HTTPError)
require.True(t, ok)
assert.Equal(t, http.StatusBadRequest, httpErr.Code)
assert.Contains(t, httpErr.Message, "file too large")
}
// Helper function to create test ZIP with multiple PDFs
func createTestZIPFile(t *testing.T, pdfCount int) *bytes.Reader {
var buf bytes.Buffer
zipWriter := zip.NewWriter(&buf)
// Create standard test PDF files
standardFiles := []string{"test1.pdf", "test2.pdf", "test3.pdf"}
// If requested count is <= 3, use standard names
if pdfCount <= 3 {
for i := 0; i < pdfCount; i++ {
fileWriter, err := zipWriter.Create(standardFiles[i])
require.NoError(t, err)
// Write minimal PDF content
pdfContent := fmt.Sprintf("%%PDF-1.4\n%%Test content for %s\n%%%%EOF", standardFiles[i])
_, err = fileWriter.Write([]byte(pdfContent))
require.NoError(t, err)
}
} else {
// For more than 3 files, use numbered names
for i := 0; i < pdfCount; i++ {
filename := fmt.Sprintf("document_%d.pdf", i)
fileWriter, err := zipWriter.Create(filename)
require.NoError(t, err)
// Write minimal PDF content
pdfContent := fmt.Sprintf("%%PDF-1.4\n%%Document %d content\n%%%%EOF", i)
_, err = fileWriter.Write([]byte(pdfContent))
require.NoError(t, err)
}
}
err := zipWriter.Close()
require.NoError(t, err)
return bytes.NewReader(buf.Bytes())
}
api/queryAPI/test_helpers.go
+36 -1
View File
@@ -1,6 +1,13 @@
package queryapi
import "log/slog"
import (
"context"
"io"
"log/slog"
"queryorchestration/internal/serviceconfig/aws"
"queryorchestration/internal/serviceconfig/objectstore"
)
// mockAuthConfig provides a minimal implementation of auth.ConfigProvider for testing
type mockAuthConfig struct{}
@@ -47,6 +54,34 @@ func (m *mockAuthConfig) InitializeAuthConfig(string, *slog.Logger) error { retu
func (m *mockAuthConfig) GetNoJWTValidation() bool { return false }
func (m *mockAuthConfig) SetNoJWTValidation(bool) {}
// Objectstore interface methods
func (m *mockAuthConfig) GetAWSKeyID() string { return "test-key" }
func (m *mockAuthConfig) SetAWSKeyID(string) {}
func (m *mockAuthConfig) GetAWSSecretKey() string { return "test-secret" }
func (m *mockAuthConfig) SetAWSSecretKey(string) {}
func (m *mockAuthConfig) GetAWSSessionToken() string { return "" }
func (m *mockAuthConfig) SetAWSSessionToken(string) {}
func (m *mockAuthConfig) GetAWSRegion() string { return "us-east-1" }
func (m *mockAuthConfig) SetAWSRegion(string) {}
func (m *mockAuthConfig) GetAWSEndpoint() string { return "http://localhost:4566" }
func (m *mockAuthConfig) SetAWSEndpoint(string) {}
func (m *mockAuthConfig) SetStoreClient(context.Context) error { return nil }
func (m *mockAuthConfig) SetStoreClientFromS3Cfg(objectstore.S3Client) {}
func (m *mockAuthConfig) SetStoreClientAndPingByName(context.Context, string) error { return nil }
func (m *mockAuthConfig) GetStoreClient() objectstore.S3Client { return nil }
func (m *mockAuthConfig) PingStoreByName(context.Context, string) error { return nil }
func (m *mockAuthConfig) SetS3Endpoint(string) {}
func (m *mockAuthConfig) GetS3Endpoint() string { return "http://localhost:4566" }
func (m *mockAuthConfig) SetS3UsePathStyle(bool) {}
func (m *mockAuthConfig) CalculateETag(context.Context, io.Reader) (string, error) {
return "test-etag", nil
}
func (m *mockAuthConfig) GetDirectoryPart(uint16, int64) uint16 { return 0 }
func (m *mockAuthConfig) GetBucket() string { return "test-bucket" }
func (m *mockAuthConfig) SetBucket(string) {}
func (m *mockAuthConfig) GetAWSProfile() aws.Profile { return aws.Profile("default") }
func (m *mockAuthConfig) SetAWSProfile(aws.Profile) {}
// NewTestControllers creates a Controllers instance for testing with a mock auth config
func NewTestControllers(services *Services) *Controllers {
return NewControllers(services, &mockAuthConfig{})