Alex Galarce
384b3c5d31
Refactor CPT postprocessing, create notebook to postprocess test bed for MCS team * Move comment * Implement testbed postprocessing functions * Refactor normalize CPT postprocessing * remove old notebook, add new postprocess notebook * Enhance normalize_cpt_fields to handle comma-separated values in lists * Update normalize_cpt_fields to return an empty string for None values * Add testbed postprocessing script, removed notebook Approved-by: Katon Minhas
25 lines
990 B
Python
25 lines
990 B
Python
# This script postprocesses the testbed data from an Excel file. It uses a custom
|
|
# function `testbed_postprocess` from the `testbed_utils` module (similar to
|
|
# postprocess.postprocess() but skipping some steps)
|
|
|
|
# Run this from the fieldExtraction directory:
|
|
# poetry run python -m src.testbed.testbed_postprocess
|
|
|
|
#### CONFIG ####
|
|
testbed_filename = "src/testbed/v6_DoczyAI_all_test_bed_LIVE_5.6.25.xlsx"
|
|
testbed_sheetname = 'all_test_bed_labels'
|
|
output_filename = "src/testbed/v6_DoczyAI_all_test_bed_LIVE_5.6.25_POSTPROCESSED.xlsx"
|
|
|
|
#### IMPORTS ####
|
|
import pandas as pd
|
|
from src.testbed.testbed_utils import testbed_postprocess
|
|
|
|
#### EXECUTION ####
|
|
# Load the testbed
|
|
df = pd.read_excel(testbed_filename, sheet_name=testbed_sheetname)
|
|
print(f"Loaded testbed from {testbed_filename} with {df.shape[0]} rows and {df.shape[1]} columns")
|
|
|
|
# Postprocess the testbed
|
|
testbed_postprocess(df).to_excel(output_filename, index=False)
|
|
|
|
print(f"Postprocessed testbed saved to {output_filename}") |