Merged in feature/support-more-types (pull request #219)
support new types for import * tests pass * missing file * bug fixes
This commit is contained in:
Jay Brown
@@ -3,7 +3,7 @@
|
||||
## DoczyAI Document Processing Platform
|
||||
|
||||
### Purpose and Domain
|
||||
DoczyAI is a cloud-native document processing platform for multi-tenant PDF ingestion and processing. Organizations can upload and organize documents with folders and labels, and store structured field extraction results for downstream analysis.
|
||||
DoczyAI is a cloud-native document processing platform for multi-tenant document ingestion and processing. The system supports a wide range of file types including PDF, images (TIFF, JPEG, PNG, BMP), Office documents (DOCX, XLSX, PPTX), plain text, and email (EML). Organizations can upload and organize documents with folders and labels, and store structured field extraction results for downstream analysis.
|
||||
|
||||
### Core Business Capabilities
|
||||
- **Multi-tenant Document Processing**: Isolated processing environments per client
|
||||
|
||||
@@ -327,7 +327,7 @@ Uploads a document for a specific client.
|
||||
|
||||
**Form Parameters**:
|
||||
|
||||
- `file` (required): Document file (PDF)
|
||||
- `file` (required): Document file. Accepted types: PDF, TIFF, JPEG, PNG, BMP, DOCX, XLSX, PPTX, TXT, EML
|
||||
- `filename` (optional): Custom filename (max 4096 chars)
|
||||
|
||||
**Response**: `200` Document uploaded
|
||||
@@ -463,16 +463,20 @@ Permanently deletes a document and all dependent data: document entries, cleans
|
||||
|
||||
### `POST /client/{id}/document/batch`
|
||||
|
||||
Uploads a ZIP archive containing multiple PDF documents for asynchronous batch processing.
|
||||
Uploads a ZIP archive containing multiple documents for asynchronous batch processing.
|
||||
|
||||
**Security**: Requires JWT authentication
|
||||
**Content Type**: `multipart/form-data`
|
||||
**Max Archive Size**: 1GB
|
||||
**Supported Archive Type**: ZIP only
|
||||
|
||||
**Accepted File Extensions**: `.pdf`, `.tiff`, `.tif`, `.jpeg`, `.jpg`, `.png`, `.bmp`, `.docx`, `.xlsx`, `.pptx`, `.txt`, `.eml`
|
||||
|
||||
Note: Macro-enabled Office formats (`.docm`, `.xlsm`, `.pptm`) are intentionally excluded because documents containing macros are rejected during validation.
|
||||
|
||||
**Form Parameters**:
|
||||
|
||||
- `archive` (required): ZIP file containing PDF documents
|
||||
- `archive` (required): ZIP file containing documents
|
||||
|
||||
**Response** (`202 Accepted`):
|
||||
|
||||
@@ -578,7 +582,7 @@ Retrieves detailed status information for a specific batch upload, including per
|
||||
"updated_at": "2024-01-01T00:04:55Z"
|
||||
},
|
||||
{
|
||||
"filename": "spreadsheet.xlsx",
|
||||
"filename": "macro_enabled.xlsm",
|
||||
"outcome": "invalid_type",
|
||||
"error_detail": null,
|
||||
"document_id": null,
|
||||
@@ -601,7 +605,7 @@ The `document_outcomes` array provides per-file tracking through the entire proc
|
||||
| `failed_read` | Could not read file content from ZIP archive | Yes |
|
||||
| `failed_s3_upload` | Could not upload extracted file to S3 | Yes |
|
||||
| `failed_upload` | Upload handler returned an error | Yes |
|
||||
| `invalid_type` | File is not a PDF | Yes |
|
||||
| `invalid_type` | File type is not in the accepted extensions list | Yes |
|
||||
| `init_complete` | Document record created, forwarded to sync | No |
|
||||
| `init_duplicate` | Duplicate detected during init processing | Yes |
|
||||
| `sync_complete` | Sync check passed, forwarded to clean | No |
|
||||
|
||||
@@ -1049,7 +1049,7 @@ CREATE TABLE documentEntries (
|
||||
- Multiple entries possible per document for version tracking
|
||||
|
||||
#### Document Cleans (`documentCleans`)
|
||||
PDF validation and format processing records.
|
||||
Document validation and format processing records.
|
||||
|
||||
```sql
|
||||
CREATE TYPE cleanFailType AS ENUM (
|
||||
@@ -1061,10 +1061,27 @@ CREATE TYPE cleanFailType AS ENUM (
|
||||
'small_dimensions',
|
||||
'large_dimensions',
|
||||
'small_dpi',
|
||||
'large_dpi'
|
||||
'large_dpi',
|
||||
'password_protected',
|
||||
'contains_macros',
|
||||
'empty_content',
|
||||
'binary_content',
|
||||
'unsupported_encoding',
|
||||
'contains_attachments'
|
||||
);
|
||||
|
||||
CREATE TYPE cleanMimeType AS ENUM ('application/pdf');
|
||||
CREATE TYPE cleanMimeType AS ENUM (
|
||||
'application/pdf',
|
||||
'image/tiff',
|
||||
'image/jpeg',
|
||||
'image/png',
|
||||
'image/bmp',
|
||||
'application/docx',
|
||||
'application/xlsx',
|
||||
'application/pptx',
|
||||
'text/plain',
|
||||
'message/rfc822'
|
||||
);
|
||||
|
||||
CREATE TABLE documentCleans (
|
||||
id uuid PRIMARY KEY DEFAULT uuid_generate_v7(),
|
||||
@@ -1091,6 +1108,33 @@ CREATE TABLE documentCleans (
|
||||
- Cannot have both success and failure indicators
|
||||
- Tracks specific failure reasons for troubleshooting
|
||||
|
||||
**Supported MIME Types**:
|
||||
- `application/pdf` -- PDF documents
|
||||
- `image/tiff` -- TIFF images (multi-frame supported)
|
||||
- `image/jpeg` -- JPEG images
|
||||
- `image/png` -- PNG images
|
||||
- `image/bmp` -- BMP images
|
||||
- `application/docx` -- Word documents
|
||||
- `application/xlsx` -- Excel spreadsheets
|
||||
- `application/pptx` -- PowerPoint presentations
|
||||
- `text/plain` -- Plain text files
|
||||
- `message/rfc822` -- Email messages (EML)
|
||||
|
||||
**Failure Reasons**:
|
||||
- `invalid_mimetype` -- Unrecognized or unsupported content type
|
||||
- `invalid_read` -- Could not read document content
|
||||
- `invalid_read_pages` -- Could not read individual pages
|
||||
- `zero_page_count` -- Document has no pages/sheets/slides
|
||||
- `large_file` -- File exceeds size limit (500MB for images/Office, 100MB for text/email)
|
||||
- `small_dimensions` / `large_dimensions` -- Image dimensions outside acceptable range
|
||||
- `small_dpi` / `large_dpi` -- Image resolution outside acceptable range (min 72 DPI)
|
||||
- `password_protected` -- Document is encrypted or password-protected (DOCX, XLSX, PPTX, OLE2)
|
||||
- `contains_macros` -- Document contains VBA macros (DOCX, XLSX, PPTX)
|
||||
- `empty_content` -- Document has no meaningful content (blank images, empty text)
|
||||
- `binary_content` -- Text file contains binary data (null bytes)
|
||||
- `unsupported_encoding` -- Text file uses an unsupported character encoding
|
||||
- `contains_attachments` -- Email contains MIME attachments (only plain text emails accepted)
|
||||
|
||||
#### Document Clean Entries (`documentCleanEntries`)
|
||||
Version tracking for clean operations.
|
||||
|
||||
|
||||
@@ -106,7 +106,7 @@ This documentation provides comprehensive coverage of the DoczyAI Document Proce
|
||||
- **Monitoring**: Prometheus + OpenTelemetry
|
||||
|
||||
### 📊 Key Capabilities
|
||||
- **Document Processing**: PDF upload, download, initialization, sync-gating, and cleaning
|
||||
- **Document Processing**: Multi-format document upload (PDF, TIFF, JPEG, PNG, BMP, DOCX, XLSX, PPTX, TXT, EML), download, initialization, sync-gating, and cleaning
|
||||
- **Multi-tenant Support**: Isolated client environments
|
||||
- **Document Organization**: Folders and labels for document management
|
||||
- **Export Operations**: Structured data export in multiple formats
|
||||
|
||||
@@ -6,7 +6,7 @@ For API reference details, see [API Documentation - Batch Document Operations](.
|
||||
|
||||
## 1. Upload a Batch
|
||||
|
||||
Submit a ZIP archive containing PDF documents for batch processing.
|
||||
Submit a ZIP archive containing documents for batch processing. Accepted file types: PDF, TIFF, JPEG, PNG, BMP, DOCX, XLSX, PPTX, TXT, EML.
|
||||
|
||||
```
|
||||
POST /client/AAA/document/batch
|
||||
@@ -86,7 +86,7 @@ Once ZIP extraction finishes, `status` changes to `completed` and all files appe
|
||||
"document_id": "...0003"
|
||||
},
|
||||
{
|
||||
"filename": "spreadsheet.xlsx",
|
||||
"filename": "macro_report.xlsm",
|
||||
"outcome": "invalid_type",
|
||||
"document_id": null
|
||||
}
|
||||
@@ -126,7 +126,7 @@ When all documents have reached a terminal outcome, the batch is fully processed
|
||||
"clean_fail_reason": "invalid PDF structure: missing xref table"
|
||||
},
|
||||
{
|
||||
"filename": "spreadsheet.xlsx",
|
||||
"filename": "macro_report.xlsm",
|
||||
"outcome": "invalid_type",
|
||||
"document_id": null
|
||||
}
|
||||
@@ -142,9 +142,9 @@ Each terminal outcome indicates a specific action for the caller:
|
||||
|
||||
- **`duplicate`** -- The file hash matches a document already in the system for this client. Use the returned `document_id` to reference the existing document. No resubmission needed.
|
||||
|
||||
- **`clean_failed`** -- The PDF was corrupt or failed validation. Check `clean_fail_reason` for the specific failure detail. Fix the source file and resubmit.
|
||||
- **`clean_failed`** -- The document was corrupt or failed validation. Check `clean_fail_reason` for the specific failure detail. Possible reasons include `invalid_mimetype`, `password_protected`, `contains_macros`, `empty_content`, `binary_content`, `unsupported_encoding`, `contains_attachments`, dimension/DPI issues, and more. Fix the source file and resubmit.
|
||||
|
||||
- **`invalid_type`** -- The file is not a PDF. Only PDF files are accepted.
|
||||
- **`invalid_type`** -- The file extension is not in the accepted list. Accepted extensions: `.pdf`, `.tiff`, `.tif`, `.jpeg`, `.jpg`, `.png`, `.bmp`, `.docx`, `.xlsx`, `.pptx`, `.txt`, `.eml`. Macro-enabled formats (`.docm`, `.xlsm`, `.pptm`) are not accepted.
|
||||
|
||||
- **`failed_open`** / **`failed_read`** -- The file inside the ZIP was damaged or unreadable. Re-create the ZIP archive and resubmit.
|
||||
|
||||
@@ -161,7 +161,7 @@ Each file in a batch starts at extraction and progresses through the document pr
|
||||
```
|
||||
ZIP extracted
|
||||
|
|
||||
+-- invalid_type (not PDF) .............. TERMINAL
|
||||
+-- invalid_type (unsupported ext) ....... TERMINAL
|
||||
+-- failed_open / failed_read ........... TERMINAL
|
||||
+-- failed_s3_upload / failed_upload .... TERMINAL
|
||||
+-- duplicate (hash match) .............. TERMINAL
|
||||
|
||||
Reference in New Issue
Block a user