doczyai-pipelines/readme.md

Field Extraction Pipeline

Contract field extraction using LLMs.

Setup

Install uv (if not already installed)

# Ubuntu/WSL (recommended)
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc  # or restart terminal

# macOS
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows (PowerShell - native, not WSL)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Verify installation: uv --version

Install dependencies

uv sync

Running the Code

Use uv run with Python's module flag from the project root:

# Run the default SaaS pipeline
uv run python -m src.pipelines.saas.main

# Run with unified runner (supports client selection)
uv run python -m src.pipelines.runner --client saas --input-dir /path/to/input

# Run with specific client
uv run python -m src.pipelines.runner --client clover --input-dir /path/to/input

Note: Always use uv run to ensure correct virtual environment. Always use -m flag for proper imports.

Development

Before pushing, always run these checks to avoid breaking the CI pipeline:

# Format code
uv run black src/

# Type checking
uv run mypy src/

# Run tests
uv run pytest

All checks must pass before merging to main.

Project Structure

├── src/
│   ├── pipelines/              # Pipeline implementations
│   │   ├── runner.py           # Unified CLI entry point with client routing
│   │   ├── saas/               # Default SaaS pipeline
│   │   │   └── main.py         # Main entry point for SaaS
│   │   ├── shared/             # Shared pipeline components
│   │   │   ├── preprocessing/  # Document preprocessing
│   │   │   ├── extraction/     # Field extraction logic
│   │   │   └── postprocessing/ # Result postprocessing
│   │   └── clients/            # Client-specific overrides
│   │       └── clover/         # Clover client customizations
│   ├── core/                   # Core utilities (registry, fieldset)
│   ├── constants/              # Constants, mappings, and field definitions
│   │   ├── mappings/           # Crosswalk JSON files
│   │   └── lists/              # Lookup lists
│   ├── prompts/                # LLM prompt templates
│   ├── utils/                  # Shared utilities (IO, string, logging, etc.)
│   ├── codes/                  # Medical code extraction utilities
│   ├── crosswalk/              # Crosswalk mapping logic
│   ├── embeddings/             # Pre-computed embeddings for code matching
│   ├── qc_qa/                  # QC/QA validation pipeline
│   ├── parent_child/           # Parent-child relationship mapping
│   ├── document_classification/ # Document type classification (DTC)
│   └── tests/                  # Unit tests
├── documentation/              # Project documentation
├── outputs/                    # Pipeline output files (gitignored)
├── logs/                       # Log files (gitignored)
└── pyproject.toml              # Project dependencies (uv/pip)

Branching

Naming Conventions

Type	Pattern	Use Case	Example
Feature	feature/<ticket>-<description>	New functionality	feature/PROJ-123-add-export-csv
Bugfix	bugfix/<ticket>-<description>	Bug fixes	bugfix/PROJ-456-fix-null-handling
Hotfix	hotfix/<ticket>-<description>	Urgent production fixes	hotfix/PROJ-789-critical-parse-error
Test	test/<description>	Testing/experimentation	test/lesser-table-caching-refactor
Release	release/<version>	Release preparation	release/v1.2.0

Branch Guidelines

• Use lowercase with hyphens (kebab-case) for descriptions
• Include ticket number when applicable (e.g., JIRA, GitHub issue)
• Keep branch names concise but descriptive
• Delete branches after merging

Workflow

1. Create branch from main
2. Make changes and commit with clear messages
3. Run checks before pushing: uv run black src/ && uv run mypy src/ && uv run pytest
4. Create PR to main
5. Ensure all CI checks pass
6. Get code review approval
7. Squash and merge