Merged in feature/self-repair-main (pull request #313)

Feature/self repair main

* fixes

* fixes

* fixes

* fixes

* fixes

* fixes

* fixes

* fixes

* new testing

* new testing

* added unit tests under fieldExtraction/tests/main-unit-tests

* unit test fixes

* unit test fixes

* merge into main

* modified consolidate_output and slight changes to qa_qc_helpers to accomodate qaqc functionality w faizan and michael's changes

* merging self repair into main

* synced with main now

* fixes

* provider billing removed

* provider billing removed

* provider billing fixed

* commented out load dotenv in config

* Merged main into feature/self-repair-main

* modified check_s3_bucket_exists function by returning False if credentials not found for pipeline

* Merge branch 'feature/self-repair-main' of https://bitbucket.org/aarete/doczy.ai into feature/self-repair-main


Approved-by: Michael McGuinness

fieldExtraction/scripts/qa_qc.py

@@ -15,17 +15,18 @@ from qa_qc_helpers import (
                            perform_qc_qa,
                            save_qcqa_wb
                            )
-from consolidate_output import consolidate_adhoc
+from consolidate_output import consolidate_output
 
 
 file_name_column = config.FILE_NAME_COLUMN
 
 """python scripts/qa_qc.py input_dir=______ output_dir=_______ ac_df=________ b_df=______ read_mode=____ df_read_mode=________
-You should have both input and output (ouput is just an empty folder) and then one of ac_df or b_df or both
+You should have both input and output (input is local or s3, ouput is just an empty folder) and then one of ac_df or b_df or both
 The final report will be saved in the parent directory of when you are running under reports/
 Run this from the base folder of the repo."""
 
 def main():
+    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
     script_dir = os.path.dirname(os.path.abspath(__file__))
     parent_dir = os.path.dirname(script_dir)
 
@@ -84,7 +85,7 @@ def main():
         print(f"Unmatched B filenames saved to '{unmatched_b_output_path}'")
 
         # Perform an inner merge to keep only matching rows for the main output
-        ac_df, b_df, merged_df = consolidate_adhoc(ac_df, b_df)
+        ac_df, b_df, merged_df = consolidate_output(timestamp, ac_df, b_df)
         merged_df = utils.remove_unnamed_columns(merged_df)
         ac_rows_before = len(ac_df)
         b_rows_before = len(b_df)
@@ -102,7 +103,6 @@ def main():
 
 
     wb = perform_qc_qa(ac_df, b_df, merged_df, input_dict)
-    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
     save_qcqa_wb(wb, timestamp)