Praneel Panchigar
637d2dea1f
Feature/new output format
* bugfix
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* requested postprocessing changes
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full_outputs/dashboard_results/ for consolidated dashboard results
- full_outputs/ for error files
- automation_qa-qc/ for QC/QA validated results and statistics
- parent-child/ for parent-child mapping outputs
- tracking/ for usage and cost tracking data
- individual/ for per-file CC results (dashboard individual files removed)
- Update file naming conventions to match new structure
- Remove QC/QA processing for error files (error files are saved without validation)
Post-Processing Changes:
- Add standard N/A value cleaning: remove placeholder values (N/A, UNKNOWN, etc.)
when they are the only value in a cell (applies before CC/dashboard split)
- Normalize all _IND fields to contain only 'Y' or 'N' values (no blanks)
- Ensure standard cleaning runs before splitting into CC …
* Ran Black for CI
* Refactor file splitting logic and add comprehensive tests
- Refactored splitting logic in io_utils.py:
- Consolidated repeated splitting code into two focused helper functions:
- _write_local_split_files() for local file writing with splitting
- _write_s3_split_files() for S3 file writing with splitting
- Both helpers use shared split_dataframe_by_filename() function
- Added MAX_ROWS_PER_SPLIT configuration (default: 70000) in config.py
- Added comprehensive test coverage for splitting logic:
- Tests for split_dataframe_by_filename() with various scenarios
- Tests for local and S3 write operations with single and multiple splits
- Tests for cc_results_full, dashboard_results_full, and qc_qa_cc_full output types
- Fixed existing test failures (write_s3 error handling, path assertions)
- Improved code maintainability and readability
* Fix failing tests in test_postprocess.py
- Updated standard_postprocess tests to use actual columns from FIELD_FORMAT_MAPPING
(PAYER_NAME, CONTRACT_TITLE) instead of custom test columns that get dropped
- Added FILE_NAME column to all file structure test DataFrames (required for splitting logic)
- Added MAX_ROWS_PER_SPLIT mock configuration for splitting tests
- Fixed patch decorators for S3 tests to properly mock logging
All 41 tests now passing.
* Black for CI
* Blank [] and ['[]'] in output instead of displaying them
- Add placeholder patterns in clean_na_values for [], ['[]'], ["[]"]
- Update format_as_json_list to return blank for empty lists instead of []
- Filter out empty-list placeholder items from list values in format_as_json_list
- Add tests for clean_na_values empty list handling and format_as_json_list
Co-authored-by: Cursor <cursoragent@cursor.com>
* Merged DEV into feature/new_output_format
* Pipeline config, parent-child, dashboard, and runner fixes
- Parent-child: enable by default, write to run directory, S3 upload via io_utils
- Dashboard: optional (CC only by default), run_dashboard=True to enable
- Add io_utils.upload_local_file_to_s3 for centralized file uploads
- Parent-child returns (row_count, local_path); pipeline handles S3 upload
- Add TODO in config for WRITE_PC_TO_S3 removal after approval
- Fix indentation errors in runner.py
Co-authored-by: Cursor <cursoragent@cursor.com>
Approved-by: Katon Minhas
112 lines
1.2 KiB
Plaintext
112 lines
1.2 KiB
Plaintext
# Python
|
|
*.py[cod]
|
|
*.pyo
|
|
__pycache__/
|
|
.mypy_cache/
|
|
.pytest_cache/
|
|
myenv/
|
|
*.egg-info/
|
|
|
|
# IDE
|
|
.idea/
|
|
.cursor/
|
|
.vscode/
|
|
|
|
# OS
|
|
.DS_Store
|
|
Thumbs.db
|
|
*:Zone.Identifier
|
|
|
|
# Logs
|
|
*.log
|
|
|
|
# Environment
|
|
.env
|
|
*.env
|
|
|
|
# Data files (general exclusion)
|
|
*.csv
|
|
*.xlsx
|
|
*.xlsb
|
|
*.zip
|
|
*.json
|
|
*.txt
|
|
*.pdf
|
|
*.PDF
|
|
*.TXT
|
|
|
|
# Allow tracked data files in src/
|
|
!src/constants/**/*.csv
|
|
!src/constants/**/*.json
|
|
!src/prompts/*.json
|
|
!requirements.txt
|
|
|
|
# Build artifacts
|
|
dist/
|
|
build/
|
|
target/
|
|
*.class
|
|
*.jar
|
|
*.war
|
|
*.app
|
|
*.exe
|
|
|
|
# Large media files
|
|
*.mp4
|
|
*.tiff
|
|
*.avi
|
|
*.flv
|
|
*.mov
|
|
*.wmv
|
|
|
|
# Terraform
|
|
.terraform/
|
|
terraform.tfstate*
|
|
.terraform.lock.hcl
|
|
*.tfplan
|
|
*.pem
|
|
textract-pipeline/terraform/builds/
|
|
|
|
# Outputs
|
|
output/
|
|
outputs/
|
|
|
|
# Local/temporary
|
|
subset/
|
|
docs/
|
|
local_scripts/
|
|
untracked_tests/
|
|
new/
|
|
textfiles/
|
|
texts/
|
|
|
|
# Streamlit (legacy)
|
|
streamlit/history.csv
|
|
streamlit/RESULTS
|
|
streamlit/DB/
|
|
streamlit/RAW_DOCUMENTS/
|
|
streamlit/SOURCE_DOCUMENTS/
|
|
streamlit/contract_field_values.csv
|
|
streamlit/contract_fields.csv
|
|
streamlit/sample.csv
|
|
streamlit/temp*.csv
|
|
streamlit/results.csv
|
|
streamlit/venv
|
|
|
|
# Legacy paths
|
|
fieldExtraction/embeddings/
|
|
|
|
# Node (if any)
|
|
node_modules/
|
|
|
|
# Test reports
|
|
TEST*.xml
|
|
|
|
# Shell scripts (debugging/utility)
|
|
*.sh
|
|
|
|
# PRD Documents
|
|
*.prd
|
|
*prd.md
|
|
.git.instructions.md
|