Faizan Mohiuddin
8c9060e425
Feature/multithreading * fixes * Merge main into feature/multithreading Resolved conflicts: - Kept timing instrumentation in file_processing.py - Kept new 3-step Exhibit-based approach for one-to-n processing - Maintained parallelization improvements (20 workers) Changes include: - Timing utils integration for performance monitoring - Increased max_workers from 5 to 20 across all components - Parallelized code_breakout and grouper_breakout - Fixed tin_npi_funcs function call parameters * Fix max_workers error for empty documents - Add check to skip parallel processing when no pages exist - Use min(len(all_page_tasks), 20) to prevent max_workers=0 - Handles edge case of documents with no exhibits or pages * Fix max_workers=0 errors in one_to_n_funcs - Add checks before all ThreadPoolExecutor creations - Prevents errors when processing empty lists: - carveout_and_special_case - breakout - special_case_breakout - filter_services_without_reimbursements - run_lob_relationship - Ensures executor only created when there are items to process * Reduce code processing parallelism to prevent API throttling Lower max_workers from 20 to 10 for code_breakout and grouper_breakout to prevent overwhelming Bedrock API with concurrent requests * fixed * Merged main into feature/multithreading * Move documentation into folder * Fix logging statements * Merge branch 'main' into feature/multithreading * Refactor for clarity * Update previous exhibit passing logic * properly simplify exhibits * Merged main into feature/multithreading * update conditional for None * exhibit multithreading changed * exhibit multithreading changed * Merge remote-tracking branch 'origin/main' into feature/multithreading * synced with main * Parallelize dynamic assignment, refactor HSC field worker, add timing/exhibit unit tests, tidy imports/ignore helpers * analyze_regression.py edited online with Bitbucket * count_pages.py edited online with Bitbucket * compare_regressed_with_baseline.py edited online with Bitbucket * simple_testbed_compare.py edited online with Bitbucket * run_testbed_metrics_regressed.py edited online with Bitbucket Approved-by: Katon Minhas
201 lines
8.8 KiB
Markdown
201 lines
8.8 KiB
Markdown
# Performance Analysis Guide
|
||
|
||
## Overview
|
||
This guide explains the timing instrumentation added to the pipeline and how to interpret the results to identify bottlenecks and parallelization opportunities.
|
||
|
||
## Instrumented Components
|
||
|
||
### 1. Main Pipeline (`main.py`)
|
||
- **`read_constants`**: Loading configuration and constants
|
||
- **`read_input`**: Reading input files from S3 or local storage
|
||
- **`warm_prompt_caches`**: Pre-warming Claude prompt caches
|
||
- **`parallel_file_processing`**: Main parallel processing of all files (CRITICAL METRIC)
|
||
- **`concat_results`**: Concatenating all file results into final dataframes
|
||
- **`qc_qa_validation`**: Quality control and QA validation
|
||
- **`write_outputs`**: Writing final outputs to S3/local storage
|
||
- **`parent_child_mapping`**: Parent-child relationship mapping (if enabled)
|
||
|
||
### 2. Per-File Processing (`file_processing.py`)
|
||
Each file goes through these stages (tracked with `context=filename`):
|
||
- **`{filename}.preprocess`**: Text cleaning, splitting, header/footer detection
|
||
- **`{filename}.one_to_n_exhibit_chunking`**: Chunking document into exhibits
|
||
- **`{filename}.one_to_n_extraction`**: Extracting one-to-many relationships (parallel exhibits)
|
||
- **`{filename}.one_to_n_generate_reimb_ids`**: Generating reimbursement IDs
|
||
- **`{filename}.one_to_one_extraction`**: Extracting one-to-one fields (see below)
|
||
- **`{filename}.merge_one_to_one_into_one_to_n`**: Merging results
|
||
- **`{filename}.code_processing`**: Code breakout and grouper processing
|
||
- **`{filename}.postprocess`**: Final postprocessing and validation
|
||
- **`{filename}.write_individual`**: Writing individual file results
|
||
|
||
### 3. One-to-One Processing (`file_processing.py` -> `run_one_to_one_prompts`)
|
||
- **`{filename}.one_to_one.provider_info`**: Extracting TIN/NPI provider information
|
||
- **`{filename}.one_to_one.hybrid_smart_chunking`**: RAG-based field extraction (CRITICAL)
|
||
- **`{filename}.one_to_one.full_context`**: Full-context field extraction (fallback)
|
||
|
||
### 4. Hybrid Smart Chunking (RAG) (`hybrid_smart_chunking_funcs.py`)
|
||
This is often the bottleneck. Detailed timing:
|
||
- **`{filename}.hsc.setup`**: Setting up embeddings, reranker, and text splitter
|
||
- **`{filename}.hsc.preprocessing`**: Document preprocessing and chunking
|
||
- **`{filename}.hsc.vectorstore_creation`**: Creating FAISS vectorstore + hybrid retriever
|
||
- **`{filename}.hsc.parallel_field_processing`**: Processing all fields in parallel (max 5 workers)
|
||
- **Per-field timing** (inside parallel processing):
|
||
- **`{filename}.hsc.field.{field_name}.retrieval`**: BM25 + semantic retrieval + reranking
|
||
- **`{filename}.hsc.field.{field_name}.llm_call`**: Claude API call for field extraction
|
||
|
||
## How to Analyze Results
|
||
|
||
### Step 1: Run Your Pipeline
|
||
```bash
|
||
poetry run python -m src.investment.main run_mode=local read_mode=local input_dir=test-multi write_to_s3=False state=MS-IL-NY-CA-TX-WI-AR-KY-TN-ME-OH-OR-FL-AZ-IN-NJ-MI max_workers=50 client=Centene-Parkland-Molina-Moda-Healthnet-Trillium
|
||
```
|
||
|
||
### Step 2: Review Timing Summary
|
||
At the end of execution, you'll see:
|
||
1. **Total pipeline duration**
|
||
2. **Hierarchical timing summary** organized by component
|
||
|
||
Look for the output sections:
|
||
```
|
||
================================================================================
|
||
PIPELINE COMPLETE - Total time: XXX.XXs
|
||
================================================================================
|
||
HIERARCHICAL TIMING SUMMARY
|
||
================================================================================
|
||
```
|
||
|
||
### Step 3: Identify Bottlenecks
|
||
|
||
#### Key Metrics to Check:
|
||
|
||
1. **`parallel_file_processing` total time**
|
||
- This should be the largest component
|
||
- Compare to total pipeline time to see overhead
|
||
- **If this is much shorter than pipeline time**: Look for overhead in setup, I/O, or postprocessing
|
||
|
||
2. **Per-file timings** (look at individual file contexts)
|
||
- **Find the slowest files**: Look for files with longest total time
|
||
- **Identify slow components within files**: Check which stage takes longest
|
||
- Is it `one_to_one.hybrid_smart_chunking`?
|
||
- Is it `one_to_n_extraction`?
|
||
- Is it `postprocess`?
|
||
|
||
3. **Hybrid Smart Chunking (HSC) breakdown**
|
||
- **`hsc.vectorstore_creation`**: Should be relatively quick (<5s per file)
|
||
- **`hsc.parallel_field_processing`**: Often the bottleneck
|
||
- Check if fields are processing in parallel (should see ~5 fields at a time)
|
||
- Look at per-field retrieval vs llm_call times
|
||
- **`hsc.field.{field_name}.llm_call`**: LLM latency (network + Claude processing)
|
||
- High variance? Could be prompt cache issues or rate limiting
|
||
- Consistently slow? Could be context size or model load
|
||
|
||
4. **One-to-N parallel processing**
|
||
- Check if exhibits and pages are processing in parallel
|
||
- Look for imbalanced workload (some exhibits much slower than others)
|
||
|
||
### Step 4: Optimization Opportunities
|
||
|
||
#### Scenario 1: HSC Vectorstore Creation is Slow
|
||
- **Problem**: Document preprocessing/chunking is slow
|
||
- **Solution**:
|
||
- Consider caching vectorstores per document
|
||
- Reduce chunk_size or chunk_overlap in HSC_CONFIG
|
||
- Optimize preprocessing pipeline
|
||
|
||
#### Scenario 2: HSC LLM Calls are Slow
|
||
- **Problem**: Too many sequential LLM calls or high latency
|
||
- **Current**: max_workers=5 for field processing
|
||
- **Solution**:
|
||
- Increase `max_workers` in `hybrid_smart_chunking_funcs.py:304` (currently 5)
|
||
- Check Bedrock connection pool (now set to 60, should be sufficient)
|
||
- Verify prompt caching is working (check cache hits in logs)
|
||
|
||
#### Scenario 3: HSC Retrieval/Reranking is Slow
|
||
- **Problem**: RAG retrieval taking too long per field
|
||
- **Solution**:
|
||
- Reduce `ENSEMBLE_TOP_K` or `RERANKER_TOP_K` in RAG config
|
||
- Consider using faster reranker model
|
||
- Optimize BM25 index size
|
||
|
||
#### Scenario 4: Parallel File Processing Not Saturated
|
||
- **Problem**: Not all `max_workers` are being used
|
||
- **Check**:
|
||
- Connection pool exhaustion (should be fixed with `max_pool_connections=60`)
|
||
- Memory constraints (monitor memory usage)
|
||
- I/O bottlenecks (local disk or S3 bandwidth)
|
||
- **Solution**:
|
||
- Increase `max_workers` carefully (currently 50)
|
||
- Profile memory usage to find leaks
|
||
|
||
#### Scenario 5: One-to-N Exhibits Not Parallel
|
||
- **Current**: max_workers=5 for exhibit processing, max_workers=5 for page processing
|
||
- **Solution**:
|
||
- Increase these limits in `file_processing.py:244` (exhibits) and `:192` (pages)
|
||
- Balance against total file-level parallelism to avoid oversubscription
|
||
|
||
#### Scenario 6: Postprocessing is Slow
|
||
- **Problem**: Code processing or postprocessing taking too long
|
||
- **Solution**:
|
||
- Profile specific postprocessing functions
|
||
- Look for opportunities to batch operations
|
||
- Consider caching expensive lookups
|
||
|
||
## Current Parallelization Strategy
|
||
|
||
### Three Levels of Parallelism:
|
||
|
||
1. **File Level** (main.py)
|
||
- `max_workers=50` (from command line arg)
|
||
- 50 files processed simultaneously
|
||
|
||
2. **Exhibit/Page Level** (within each file)
|
||
- Exhibits: `max_workers=5` per file
|
||
- Pages: `max_workers=5` per exhibit
|
||
- This means potentially 50 files × 5 exhibits × 5 pages = 1,250 threads (too many!)
|
||
- **Currently limited by nested executors**
|
||
|
||
3. **Field Level** (within HSC per file)
|
||
- `max_workers=5` for field extraction
|
||
- 5 fields extracted in parallel per file
|
||
|
||
### Total Theoretical Concurrency:
|
||
- **Bedrock API calls**: Up to 60 concurrent (connection pool limit)
|
||
- **Thread pools**: Nested executors create complex concurrency
|
||
- **Bottleneck**: Likely the 60-connection Bedrock pool with 50 file workers
|
||
|
||
## Recommendations After First Run
|
||
|
||
1. **Run with timing enabled** and review the hierarchical summary
|
||
2. **Identify the top 3 slowest components** by total time
|
||
3. **Check if parallelism is effective**:
|
||
- Are all max_workers being utilized?
|
||
- Is there thread contention or blocking?
|
||
4. **Look for outliers**: Files that take much longer than others
|
||
5. **Profile memory usage**: Ensure no memory leaks with many threads
|
||
6. **Monitor Bedrock throttling**: Check for rate limit errors in logs
|
||
|
||
## Next Steps
|
||
|
||
Based on timing results, we can:
|
||
1. **Increase parallelism** where bottlenecks are found
|
||
2. **Add caching** for repeated operations (vectorstores, embeddings)
|
||
3. **Optimize prompts** to reduce token count and latency
|
||
4. **Batch operations** where sequential processing is required
|
||
5. **Add more granular timing** to specific slow functions
|
||
|
||
## Timing Log Format
|
||
|
||
Timing logs appear as:
|
||
```
|
||
INFO - ⏱️ read_constants: 0.50s
|
||
INFO - ⏱️ parallel_file_processing: 450.23s
|
||
DEBUG - ⏱️ file1.pdf.preprocess: 2.34s
|
||
DEBUG - ⏱️ file1.pdf.one_to_one.hybrid_smart_chunking: 45.67s
|
||
DEBUG - ⏱️ file1.pdf.hsc.setup: 0.12s
|
||
DEBUG - ⏱️ file1.pdf.hsc.vectorstore_creation: 3.45s
|
||
DEBUG - ⏱️ file1.pdf.hsc.parallel_field_processing: 40.12s
|
||
DEBUG - ⏱️ file1.pdf.hsc.field.EFFECTIVE_DT.retrieval: 0.89s
|
||
DEBUG - ⏱️ file1.pdf.hsc.field.EFFECTIVE_DT.llm_call: 2.34s
|
||
```
|
||
|
||
Use these to drill down into specific bottlenecks!
|