Merged in feature/proc-desc (pull request #400)

feature/proc desc * Update crosswalks - individual code level * add embeddings * Update embeddings * Proc mappings * code description mapping for PROC codes * Code map bug fixes * docstrings * Initialize Rev and Diag descriptions * Add 0 to front for rev codes * Comments * Merged main into feature/proc-desc * Comments * Separate grouper and diag codes * add grouper version * Bugfix" * Add admit type code * add proc mod description * Add bill type pos mapping * Merged main into feature/proc-desc * Uncomment test code * Bugfix * Remove test.py * list bugfix * Merged main into feature/proc-desc * concurrency Approved-by: Alex Galarce
2025-02-20 04:10:33 +00:00
parent a18d562d34
commit 664397d713
47 changed files with 96262 additions and 56 deletions
@@ -0,0 +1,34 @@
+
+import src.utils.embedding_utils as embedding_utils
+from src.investment.code_funcs import get_proc_crosswalk, get_clean_value
+from crosswalk.crosswalk_utils import CrosswalkBuilder
+
+import faiss
+import pickle
+
+import os
+import pandas as pd
+import numpy as np
+
+from sentence_transformers import SentenceTransformer
+
+# Load model
+roberta_model = SentenceTransformer("all-roberta-large-v1")
+
+# If running for the first time
+mapping_dir = "crosswalk/mapping_csvs/proc_cd"
+pkl_dir = "embeddings"
+for filename in os.listdir(mapping_dir):
+    if filename.endswith(".csv"):
+        print(f"Creating embeddings for {filename}")
+        stripped_filename = filename.strip(".csv")
+        mapping_df = pd.read_csv(os.path.join(mapping_dir, filename))
+        proc_crosswalk = CrosswalkBuilder().from_df(mapping_df, from_col="Code", to_col="Description")
+        proc_choices = [get_clean_value(x) for x in proc_crosswalk.mapping.values()]
+        os.makedirs(os.path.join(pkl_dir, stripped_filename), exist_ok=True)
+        proc_index = embedding_utils.create_faiss_index(choices=proc_choices, 
+                                                        model=roberta_model, 
+                                                        save_path=os.path.join(pkl_dir, stripped_filename, "index.bin"),
+                                                        embedding_path=os.path.join(pkl_dir, stripped_filename, "embeddings.npy"),
+                                                        choices_path=os.path.join(pkl_dir, stripped_filename, "choices.pkl")
+                                                        )