Katon Minhas
afb6d5185d
Feature/lesser table caching refactor hybrid * chore: Remove unused duplicate main.py from shared pipeline * fix: Correct crosswalk paths in aarete_derived.py * chore: Remove unused documentation files from fieldExtraction * docs: Add documentation files to documentation folder * docs: Update README with uv setup, expanded project structure, and branching conventions * docs: Add uv installation steps with Ubuntu/WSL emphasis * Enable prompt caching for all remaining LLM calls - Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE, EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH, SPECIAL_CASE_ASSIGNMENT - Update all invoke_claude() calls in saas and clover pipelines to use cache=True with corresponding _INSTRUCTION() functions - Add new instructions to get_cacheable_instructions() for cache warming - Update tests for new instruction functions Functions now using caching: - prompt_exhibit_level - prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF) - prompt_fee_schedule_breakout - prompt_grouper_breakout - prompt_special_case_assignment - prompt_exhibit_linkage - prompt_exhibit_header - prompt_smart_chunked (ONE_TO_ONE templates) - prompt_date_fix - prompt_derived_term_date - prompt_exhibit_title_match - provider_name_match_check 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Reorder * feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction - Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction - Extract full paragraph text of offset/recoupment provisions from contracts - Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence - Fix reorder_columns to preserve extra columns not in COLUMN_ORDER - Update QC/QA output path to outputs/qc_qa/ * fix: Update dev deps and test assertions for QC/QA output path - Add pytest/pytest-mock to dev dependencies for mypy type checking - Update test assertions to expect outputs/qc_qa instead of qa_qc_output * style: Apply black formatting to prompt_templates.py * Merge main, move scripts * Archive some scripts * update py version * remove .py version file * Remove ASCII characters * Restore testbed code * restore tracking * Update testbed metrics * Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT - Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification - Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination - Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification - Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT - Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK - Add new instructions to get_cacheable_instructions() for cache warming - Add unit tests for new instruction functions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix postprocessing_funcs to remove invalid columns * Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid * Revert prompt caching changes from aed1b73c * update formatting * Update imports Approved-by: Sha Brown Approved-by: Praneel Panchigar
65 lines
2.7 KiB
Python
65 lines
2.7 KiB
Python
import boto3
|
|
import pandas as pd
|
|
from concurrent.futures import ThreadPoolExecutor, as_completed
|
|
|
|
|
|
"""
|
|
This script is used to copy complex contract text files to a different folder in the same S3 bucket.
|
|
The script reads a CSV file that contains the filenames and their table counts. It filters the rows where the table count is greater than or equal to 1.
|
|
|
|
This script is NOT CLIENT SPECIFIC. It is a generic script that can be used for any client.
|
|
|
|
"""
|
|
|
|
|
|
# Function to copy a single file
|
|
def copy_file(bucket_name, source_prefix, dest_prefix, file_name, profile_name):
|
|
try:
|
|
# Initialize S3 client
|
|
session = boto3.Session(profile_name=profile_name)
|
|
s3 = session.client('s3')
|
|
copy_source = {'Bucket': bucket_name, 'Key': f'{source_prefix}/{file_name}'}
|
|
dest_key = f'{dest_prefix}/{file_name}'
|
|
s3.copy_object(CopySource=copy_source, Bucket=bucket_name, Key=dest_key)
|
|
print(f'Successfully copied from {copy_source} to {dest_key}')
|
|
except Exception as e:
|
|
# print(f'Failed to copy {file_name}: {e}')
|
|
print('File not in bucket. Skipping')
|
|
pass
|
|
|
|
# Function to process copying in parallel
|
|
def copy_files_in_parallel(bucket_name, source_prefix, dest_prefix, file_list,profile_name ,max_workers=10):
|
|
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
|
futures = [
|
|
executor.submit(copy_file, bucket_name, source_prefix, dest_prefix, file_name, profile_name=profile_name)
|
|
for file_name in file_list
|
|
]
|
|
for future in as_completed(futures):
|
|
future.result() # This will raise any exceptions that occurred during execution
|
|
|
|
# Main function
|
|
def main():
|
|
# Client bucket where the text files are stored
|
|
bucket_name = 'centene-national-contracting-files'
|
|
|
|
# This is used to copy complex contract text files to a different folder
|
|
source_prefix = 'batch6_16_file/txt_files'
|
|
dest_prefix_3a = 'batch6_16_file/complex_contract_files/txt_files'
|
|
max_workers = 50
|
|
profile_name = 'default' # Set this as per your AWS profile
|
|
|
|
# Read the table analysis file (from DS code) and read the filenames and their table counts
|
|
csv_file = 'C:\\Doczy\\National contracting\\Somefolder\\CNC-6to16-Table-Analysis.csv' # Replace with your CSV file path
|
|
file_df = pd.read_csv(csv_file)
|
|
|
|
# Filter rows where 'tables' value is greater than or equal to 1
|
|
complex_table_files = file_df[file_df['Table Count'] >= 1]
|
|
|
|
# Assuming the CSV has a column 'file_name' with the list of file names
|
|
file_list = complex_table_files['Filename_pdf'].tolist()
|
|
|
|
copy_files_in_parallel(bucket_name, source_prefix, dest_prefix_3a, file_list, profile_name = profile_name, max_workers=max_workers)
|
|
|
|
if __name__ == '__main__':
|
|
main()
|