doczyai-pipelines/readme.md

# Field Extraction Pipeline

Contract field extraction using LLMs.

## Setup

### Install uv (if not already installed)

```bash
# Ubuntu/WSL (recommended)
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc  # or restart terminal

# macOS
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows (PowerShell - native, not WSL)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
```

Verify installation: `uv --version`

### Install dependencies

```bash
uv sync
```

## Running the Code

Use `uv run` with Python's module flag from the project root:

```bash
# Run the default SaaS pipeline
uv run python -m src.pipelines.saas.main

# Run with unified runner (supports client selection)
uv run python -m src.pipelines.runner --client saas --input-dir /path/to/input

# Run with specific client
uv run python -m src.pipelines.runner --client clover --input-dir /path/to/input
```

**Note:** Always use `uv run` to ensure correct virtual environment. Always use `-m` flag for proper imports.

## Development

**Before pushing, always run these checks to avoid breaking the CI pipeline:**

```bash
# Format code
uv run black src/

# Type checking
uv run mypy src/

# Run tests
uv run pytest
```

All checks must pass before merging to main.

## Project Structure

```
├── src/
│   ├── pipelines/              # Pipeline implementations
│   │   ├── runner.py           # Unified CLI entry point with client routing
│   │   ├── saas/               # Default SaaS pipeline
│   │   │   └── main.py         # Main entry point for SaaS
│   │   ├── shared/             # Shared pipeline components
│   │   │   ├── preprocessing/  # Document preprocessing
│   │   │   ├── extraction/     # Field extraction logic
│   │   │   └── postprocessing/ # Result postprocessing
│   │   └── clients/            # Client-specific overrides
│   │       └── clover/         # Clover client customizations
│   ├── core/                   # Core utilities (registry, fieldset)
│   ├── constants/              # Constants, mappings, and field definitions
│   │   ├── mappings/           # Crosswalk JSON files
│   │   └── lists/              # Lookup lists
│   ├── prompts/                # LLM prompt templates
│   ├── utils/                  # Shared utilities (IO, string, logging, etc.)
│   ├── codes/                  # Medical code extraction utilities
│   ├── crosswalk/              # Crosswalk mapping logic
│   ├── embeddings/             # Pre-computed embeddings for code matching
│   ├── qc_qa/                  # QC/QA validation pipeline
│   ├── parent_child/           # Parent-child relationship mapping
│   ├── document_classification/ # Document type classification (DTC)
│   └── tests/                  # Unit tests
├── documentation/              # Project documentation
├── outputs/                    # Pipeline output files (gitignored)
├── logs/                       # Log files (gitignored)
└── pyproject.toml              # Project dependencies (uv/pip)
```

## Branching

### Naming Conventions

| Type | Pattern | Use Case | Example |
|------|---------|----------|---------|
| Feature | `feature/<ticket>-<description>` | New functionality | `feature/PROJ-123-add-export-csv` |
| Bugfix | `bugfix/<ticket>-<description>` | Bug fixes | `bugfix/PROJ-456-fix-null-handling` |
| Hotfix | `hotfix/<ticket>-<description>` | Urgent production fixes | `hotfix/PROJ-789-critical-parse-error` |
| Test | `test/<description>` | Testing/experimentation | `test/lesser-table-caching-refactor` |
| Release | `release/<version>` | Release preparation | `release/v1.2.0` |

### Branch Guidelines

- Use lowercase with hyphens (kebab-case) for descriptions
- Include ticket number when applicable (e.g., JIRA, GitHub issue)
- Keep branch names concise but descriptive
- Delete branches after merging

### Workflow

1. Create branch from `main`
2. Make changes and commit with clear messages
3. Run checks before pushing: `uv run black src/ && uv run mypy src/ && uv run pytest`
4. Create PR to `main`
5. Ensure all CI checks pass
6. Get code review approval
7. Squash and merge