Faizan Mohiuddin
8c9060e425
Feature/multithreading * fixes * Merge main into feature/multithreading Resolved conflicts: - Kept timing instrumentation in file_processing.py - Kept new 3-step Exhibit-based approach for one-to-n processing - Maintained parallelization improvements (20 workers) Changes include: - Timing utils integration for performance monitoring - Increased max_workers from 5 to 20 across all components - Parallelized code_breakout and grouper_breakout - Fixed tin_npi_funcs function call parameters * Fix max_workers error for empty documents - Add check to skip parallel processing when no pages exist - Use min(len(all_page_tasks), 20) to prevent max_workers=0 - Handles edge case of documents with no exhibits or pages * Fix max_workers=0 errors in one_to_n_funcs - Add checks before all ThreadPoolExecutor creations - Prevents errors when processing empty lists: - carveout_and_special_case - breakout - special_case_breakout - filter_services_without_reimbursements - run_lob_relationship - Ensures executor only created when there are items to process * Reduce code processing parallelism to prevent API throttling Lower max_workers from 20 to 10 for code_breakout and grouper_breakout to prevent overwhelming Bedrock API with concurrent requests * fixed * Merged main into feature/multithreading * Move documentation into folder * Fix logging statements * Merge branch 'main' into feature/multithreading * Refactor for clarity * Update previous exhibit passing logic * properly simplify exhibits * Merged main into feature/multithreading * update conditional for None * exhibit multithreading changed * exhibit multithreading changed * Merge remote-tracking branch 'origin/main' into feature/multithreading * synced with main * Parallelize dynamic assignment, refactor HSC field worker, add timing/exhibit unit tests, tidy imports/ignore helpers * analyze_regression.py edited online with Bitbucket * count_pages.py edited online with Bitbucket * compare_regressed_with_baseline.py edited online with Bitbucket * simple_testbed_compare.py edited online with Bitbucket * run_testbed_metrics_regressed.py edited online with Bitbucket Approved-by: Katon Minhas
36 lines
977 B
Python
36 lines
977 B
Python
import time
|
|
|
|
from src.utils import timing_utils
|
|
|
|
|
|
def test_timing_stats_add_and_avg():
|
|
stats = timing_utils.TimingStats(name="unit")
|
|
stats.add(0.1)
|
|
stats.add(0.3)
|
|
assert stats.count == 2
|
|
assert stats.min_time == 0.1
|
|
assert stats.max_time == 0.3
|
|
assert abs(stats.avg_time - 0.2) < 1e-6
|
|
|
|
|
|
def test_timing_tracker_record_and_reset():
|
|
tracker = timing_utils.TimingTracker()
|
|
tracker.record("step", 0.05, context="fileA")
|
|
tracker.record("step", 0.10, context="fileA")
|
|
stats = tracker.get_stats()
|
|
assert "fileA.step" in stats
|
|
assert stats["fileA.step"].count == 2
|
|
tracker.reset()
|
|
assert tracker.get_stats() == {}
|
|
|
|
|
|
def test_timed_block_records_duration():
|
|
tracker = timing_utils.get_tracker()
|
|
tracker.reset()
|
|
with timing_utils.timed_block("sleep_test", context="unit"):
|
|
time.sleep(0.01)
|
|
stats = tracker.get_stats()
|
|
key = "unit.sleep_test"
|
|
assert key in stats
|
|
assert stats[key].count == 1
|