jacob-mathison/doczyai-pipelines
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

Merged in hotfix/roll-back-effective-date-prompt (pull request #332)

Browse Source 
roll back to old prompt

* roll back to old prompt


Approved-by: Katon Minhas
This commit is contained in:
Alex Galarce
2024-12-13 21:23:28 +00:00
committed by Katon Minhas
parent 26c12469bc
commit 365527d3d0
2 changed files with 93 additions and 238 deletions
Download Patch File Download Diff File Expand all files Collapse all files
fieldExtraction/src/cnc_hotfix_effective_date_utils.py
+51 -119
View File
@@ -1,125 +1,49 @@
import re
import ac_smart_chunking
regex_backticks = r"\|([^|]+)\|"
#TODO: leave this one in here; don't move it to prompts until we get confirmation that we want to handle meridian differently.
# TODO: leave this one in here; don't move it to prompts until we get confirmation that we want to handle meridian differently.
def get_effective_date_meridian_prompt(context):
prompt_template = f""""
Please analyze the following contract and extract ONLY the Signature Date. FOLLOW these precise rules in order:
Please analyze the following contract and extract ONLY the Signature Date. Follow these precise rules in order:
1. LOOK FOR SIGNATURE DATE:
- FIND dates near signature sections, signature lines, or signature blocks
- CHECK for dates beside, above, or below signature lines
2. IF MULTIPLE matches are found:
- CHECK that each date is CLEARLY associated with signatures
- PRIORITIZE the LATEST FULL DATE
- DO NOT include dates from non-signature sections
3. DATE FORMAT HANDLING RULES:
[A] ANALYZE THE DATE:
- IF already in YYYY-MM-DD format → KEEP IT
- IF in another format → PROCEED TO [B]
[B] CHECK FOR FORMAT CERTAINTY:
ONLY proceed with conversion if either:
a) Format is UNAMBIGIOUS and COMPREHENDABLE in the document
b) Date has WRITTEN MONTHS making format unambiguous ("January 1, 2024")
c) Date format is SELF-EVIDENT and LOGICALLY UNAMBIGUOUS meaning:
- Numbers MUST make it IMPOSSIBLE to interpret in any other way
- There can be NO OTHER POSSIBLE valid date interpretation
- Examples:
* "13/05/2024" where 13 can ONLY be a day as it's impossible for it to be a month
* "15 June 12" where given modern context, this MUST be June 15, 2012
[C] YEAR HANDLING RULES:
PRIMARY STEP - ANALYZE YEAR DIGITS:
• WHEN ENCOUNTERING SHORT/INCOMPLETE YEARS:
1. Look for Signature Date
DETECTION PHASE:
▸ CHECK NEARBY DATES:
- LOOK for ALL dates near the target date
- IF ALL nearby dates show SAME year (e.g., all 2020):
* USE that year for completion
* Example: "July 1ST 20" near "June 15 2020" and "May 20 2020" → complete as 2020
- IF ALL nearby years are NOT SAME:
* KEEP original format
* DO NOT attempt completion
2. If multiple matches are found:
- Prioritize the latest full date
ACTION PHASE:
▸ FOR CONFIRMED LAST TWO DIGITS:
- Single digit (0-9): ADD "200" prefix (e.g., "9" → "2009")
- Two digits (00-49): ADD "20" prefix (e.g., "12" → "2012")
- Two digits (50-99): ADD "19" prefix (e.g., "95" → "1995")
3. If only partial dates are found, return N/A.
4. Return the date in YYYY-MM-DD format or N/A if date not found. Enclose just your final answer in |pipes|.
Here is the contract to analyze:
## START CONTRACT TEXT ##
{context}
▸ FOR CONFIRMED FIRST TWO DIGITS:
COMPLETE ONLY IF:
- Nearby dates CONSISTENTLY show same year
- Example: "15 June, 20__" near several 2020 dates → complete as "2020"
▸ IF POSITION UNCLEAR or NO CONSISTENT YEAR PATTERN:
- KEEP original format
- NO completion attempt
## END CONTRACT TEXT ##
VALIDATION:
• ONLY apply these rules when:
- Format certainty criteria met
- Evidence is logically undeniable
- No other interpretation possible
Before returning the output, VERIFY the date format is YYYY-MM-DD. If the date is not found, return N/A.
[D] MAKE FORMAT DECISION:
IF ANY condition from [B] is met AND there is NO POSSIBLE ambiguity:
- CONVERT to YYYY-MM-DD format
- APPLY year rules from [C] if needed
IN ALL OTHER CASES:
- KEEP the date in its ORIGINAL format exactly as found
- DO NOT attempt conversion
- DO NOT infer format from patterns or other dates
4. IF NO valid signature date found:
- The answer MUST BE N/A
5. Before providing the final answer, SHOW your reasoning:
- LIST ALL signature dates found that MATCH Rule #1
- For EACH date found, EXPLAIN:
* WHERE it was found (specific signature context)
* WHY it is being CONSIDERED or EXCLUDED
- IF converting to YYYY-MM-DD format:
* EXPLAIN why you are EXTREMELY CONFIDENT about the format
* SHOW your conversion work
* JUSTIFY any year prefix assumptions
- IF keeping original format:
* EXPLAIN why conversion wasn't possible
* CONFIRM the date is complete and valid
- EXPLAIN your final date selection or why N/A is being returned
6. ENCLOSE just your final answer in |pipes|.
Here is the contract to analyze:
## START CONTRACT TEXT ##
{context}
## END CONTRACT TEXT ##
Finally state "Answer: |YYYY-MM-DD| or |Original Date Format| or |N/A|"
Finally state "Answer: |YYYY-MM-DD| or |N/A|"
"""
return prompt_template
def chunk_with_include_exclude_keywords(
text_dict: dict[str, str],
text_dict: dict[str, str],
include_keywords: list[str],
exclude_keywords: list[str],
case_sensitive: bool = False,
apply_date_filtering: bool = True
):
apply_date_filtering: bool = True,
):
"""Look for pages that include any of the include_keywords and exclude any pages that contain any of the exclude_keywords.
Args:
text_dict (dict[str, str]): A dictionary keyed by string page number and valued
by string page contents
@@ -138,53 +62,61 @@ def chunk_with_include_exclude_keywords(
exclude_keywords = [keyword.lower() for keyword in exclude_keywords]
text_dict = {key: value.lower() for key, value in text_dict.items()}
if apply_date_filtering:
if apply_date_filtering:
pages_containing_date = set()
for page_num, page_content in text_dict.items():
dates_list = re.findall(ac_smart_chunking.DATE_PATTERN, page_content)
if len(dates_list) > 0:
pages_containing_date.add(page_num)
pages_containing_date = sorted(pages_containing_date,key=int)
pages_containing_date = sorted(pages_containing_date, key=int)
# debug lines for inclusion and exclusion keywords found on each page
# print(f"Pages containing date: {pages_containing_date}")
d = list() # to record details
d = list() # to record details
for page_num, page_content in text_dict.items():
if page_num in pages_containing_date:
inclusion_keywords = set()
exclusion_keywords = set()
contains_inclusion_keyword = False
for keyword in include_keywords:
contains_inclusion_keyword = True
if keyword in page_content:
inclusion_keywords.add(keyword)
for keyword in exclude_keywords:
if keyword in page_content:
exclusion_keywords.add(keyword)
# print(f"Page #: {page_num} contains inclusion keywords: {inclusion_keywords} and contain exclusion keywords: {exclusion_keywords} | Inclusion Flag: {contains_inclusion_keyword}")
if contains_inclusion_keyword:
d.append({"Page Number": page_num, "Inclusion Keywords": inclusion_keywords, "Exclusion Keywords": exclusion_keywords})
d.append(
{
"Page Number": page_num,
"Inclusion Keywords": inclusion_keywords,
"Exclusion Keywords": exclusion_keywords,
}
)
# print(f"Page #: {page_num} contains inclusion keywords: {inclusion_keywords} and contain exclusion keywords: {exclusion_keywords}")
page_list = []
for page in text_dict.keys():
if not apply_date_filtering or (apply_date_filtering and page in pages_containing_date):
if any(keyword in text_dict[page] for keyword in include_keywords) and not any(
keyword in text_dict[page] for keyword in exclude_keywords
):
if not apply_date_filtering or (
apply_date_filtering and page in pages_containing_date
):
if any(
keyword in text_dict[page] for keyword in include_keywords
) and not any(keyword in text_dict[page] for keyword in exclude_keywords):
# if str(int(page) - 1) in text_dict.keys():
# page_list.append(str(int(page) - 1))
# print(f"Taking page #: {str(int(page) - 1)} (as buffer for following page)")
@@ -200,7 +132,7 @@ def chunk_with_include_exclude_keywords(
page_list_sorted_str = list(map(str, page_list_sorted))
# print(", ".join([page for page in page_list_sorted_str]))
# print(f"Obtained total pages: {len(page_list_sorted_str)}")
return page_list_sorted_str, d
return page_list_sorted_str, d
fieldExtraction/src/prompts.py
+42 -119
View File
@@ -1,5 +1,5 @@
import valid
import config
import valid
#####################################################################################
##################################### BOTTOM UP #####################################
@@ -397,142 +397,64 @@ If for any question, no correct answer is found, return 'N/A'.
"""
def get_effective_date_prompt(context):
batch_4_keywords = """
- "dated this" FOLLOWED BY a date
- "made and entered into" FOLLOWED BY a date
- "made this" FOLLOWED BY a date
- "dated and effective" FOLLOWED BY a date
def get_effective_date_prompt(context: str) -> str:
"""
Generate a prompt template to extract the effective date from a contract.
Args:
context (str): The contract text to analyze.
Returns:
str: The prompt template for extracting the effective date.
"""
batch_4_keywords = batch_4_keywords if any([cand in config.BATCH_ID.lower() for cand in ["cnc-4","cnc-4a","cnc-4b"]]) else ""
prompt_template = f"""
Please analyze the following contract and extract ONLY the effective date of the contract or amendment. THE DATE MAY OR MAY NOT BE EXPLICITLY LABELED AS AN EFFECTIVE DATE. FOLLOW these precise rules in order:
Please analyze the following contract and extract ONLY the effective date of the contract or amendment. Follow these precise rules in order:
1. LOOK FOR dates using the following guidelines:
a) LOOK FOR dates that appear after these common phrases but not limited to (case-insensitive matching):
- "Effective Date:" FOLLOWED BY a date
- "FOR HEALTH PLAN USE ONLY" section CONTAINING "Effective Date:"
- "To be completed by Health Plan only:" section CONTAINING "Effective Date:"
- "is entered into as of" FOLLOWED BY a date
- "effective as of" FOLLOWED BY a date
- "shall begin on" FOLLOWED BY a date
- "Effective Date of Agreement:" FOLLOWED BY a date
- "Effective Date of Amendment" FOLLOWED BY a date
{batch_4_keywords}
b) ALSO LOOK FOR dates in phrases that SEMANTICALLY INDICATE an effective date, even if they DO NOT EXACTLY MATCH the phrases above. Examples include but ARE NOT LIMITED TO:
- Phrases indicating when an contract or amendment begins or takes effect
- Phrases specifying the start date of the contract or amendment
- Phrases defining when the contract or amendment becomes operational
- ANY other phrasing that CLEARLY INDICATES the date when the contract or amendment becomes effective
1. Look for dates that appear after phrases matching these exact patterns:
- "Effective Date:" followed by a date
- "FOR HEALTH PLAN USE ONLY" section containing "Effective Date:"
- "To be completed by Health Plan only:" section containing "Effective Date:"
- "is entered into as of" followed by a date
2. DO NOT INCLUDE the date found in the Signature Date section:
- IF AND ONLY IF there is NO MENTION of that date being the effective date
2. If multiple matches are found:
- Prioritize dates within "FOR HEALTH PLAN USE ONLY" or "To be completed by Health Plan only" sections
- Then prioritize dates that directly follow "Effective Date:"
3. IF MULTIPLE matches are found:
- VERIFY that you have PRECISELY followed RULE #1 and RULE #2
- EVERY SINGLE DATE in the MULTIPLE matches MUST STRICTLY follow RULE #1 and RULE #2
- PRIORITIZE the LATEST date
3. Do not use the date found in the "Signature Date" section.
- If there is nothing next to "Effective Date:", return N/A.
- If there is only one date in the document, don't extract this date unless it is explicitly labeled "effective date". It is okay to return N/A.
4. DATE FORMAT HANDLING RULES:
PRIMARY GOAL: Convert date to YYYY-MM-DD format in specific cases only
DECISION FLOW:
[A] ANALYZE THE DATE:
- IF already in YYYY-MM-DD format → KEEP IT
- IF in another format → PROCEED TO [B]
[B] CHECK FOR FORMAT CERTAINTY:
ONLY proceed with conversion if either:
a) Format is UNAMBIGIOUS and COMPREHENDABLE in the document
b) Date has WRITTEN MONTHS making format unambiguous ("January 1, 2024")
c) Date format is SELF-EVIDENT and LOGICALLY UNAMBIGUOUS meaning:
- Numbers MUST make it IMPOSSIBLE to interpret in any other way
- There can be NO OTHER POSSIBLE valid date interpretation
- Examples:
* "13/05/2024" where 13 can ONLY be a day as it's impossible for it to be a month
* "15 June 12" where given modern context, this MUST be June 15, 2012
4. If partial dates are found, return N/A.
[C] YEAR HANDLING RULES:
PRIMARY STEP - ANALYZE YEAR DIGITS:
• WHEN ENCOUNTERING SHORT/INCOMPLETE YEARS:
5. Return the date in YYYY-MM-DD format or N/A if date not found. Enclose just your final answer in |pipes|.
Here is the contract to analyze:
## START CONTRACT TEXT ##
{context}
DETECTION PHASE:
▸ CHECK NEARBY DATES:
- LOOK for ALL dates near the target date
- IF ALL nearby dates show SAME year (e.g., all 2020):
* USE that year for completion
* Example: "July 1ST 20" near "June 15 2020" and "May 20 2020" → complete as 2020
- IF ALL nearby years are NOT SAME:
* KEEP original format
* DO NOT attempt completion
## END CONTRACT TEXT ##
ACTION PHASE:
▸ FOR CONFIRMED LAST TWO DIGITS:
- Single digit (0-9): ADD "200" prefix (e.g., "9" → "2009")
- Two digits (00-49): ADD "20" prefix (e.g., "12" → "2012")
- Two digits (50-99): ADD "19" prefix (e.g., "95" → "1995")
▸ FOR CONFIRMED FIRST TWO DIGITS:
COMPLETE ONLY IF:
- Nearby dates CONSISTENTLY show same year
- Example: "15 June, 20__" near several 2020 dates → complete as "2020"
▸ IF POSITION UNCLEAR or NO CONSISTENT YEAR PATTERN:
- KEEP original format
- NO completion attempt
Ensure that you do not use the date found in the "Signature Date" section.
VALIDATION:
• ONLY apply these rules when:
- Format certainty criteria met
- Evidence is logically undeniable
- No other interpretation possible
[D] MAKE FORMAT DECISION:
IF ANY condition from [B] is met AND there is NO POSSIBLE ambiguity:
- CONVERT to YYYY-MM-DD format
- APPLY year rules from [C] if needed
IN ALL OTHER CASES:
- KEEP the date in its ORIGINAL format exactly as found
- DO NOT attempt conversion
- DO NOT infer format from patterns or other dates in document
Before returning the output, VERIFY the date format is YYYY-MM-DD. If the date is not found, return N/A.
5. IF NO valid date is found:
- The answer MUST BE N/A
6. Before providing the final answer, SHOW your reasoning:
- LIST ALL dates found that MATCH patterns in RULE #1
- For EACH date found:
* EXPLAIN why it's considered for effective date or not
* IF dealing with partial year, SHOW nearby dates found
* EXPLAIN if nearby dates show consistent year pattern
- IF converting to YYYY-MM-DD format, EXPLAIN your confidence in the conversion
- JUSTIFY your final date selection or why N/A is being returned
7. ENCLOSE just your final answer in |pipes|.
Here is the contract to analyze:
## START CONTRACT TEXT ##
{context}
## END CONTRACT TEXT ##
Finally state "Answer: |YYYY-MM-DD| or |Original Date Format| or |N/A|"
Finally state "Answer: |YYYY-MM-DD| or |N/A|"
"""
return prompt_template
AC_DICT = {
"CONTRACT_TITLE": "What is the complete title of the contract or the document? It is generally written in CAPITAL LETTERS and is generally found at the beginning of the document before a paragraph containing provider and payer name. It typically contains the words AGREEMENT, AMENDMENT, CONTRACT, ADDENDUM, MEMORANDUM, LETTER, PROVIDER or REQUEST. In case of agreement, typical structure is the provider name, followed by the phrase with agreement, followed potentially by a number signifying a version of the contract. In case of amendment, typical structure is word amendment followed by number, followed by the phrase with agreement. Extract the complete title of the contract or document as accurately as possible. Ensure: All leading characters are included for each word (e.g. 'NETWORK' instead of 'ETWORK'). Spaces between words are preserved in phrases (e.g., 'PROVIDER COLLABORATION' instead of 'PROVIDERCOLLABORATION'). If text appears incomplete or lacks spacing, use logical patterns like capitalization or common formatting rules to infer and correct the output. Replace new lines and line breaks with space.",
"PAYER_NAME": "What is the name of the payer that is a party to the contract as stated in the Preamble?",
"PROV_GROUP_NAME": "What is the name of the Group provider that is a party to the contract? This is generally found near keyword provider. Only return the name of the Group provider along with dba name.",
#"PROV_GROUP_NPI": "What is the group provider's national provider identifier number mentioned in the contract? This may be found on the signature page or on the roster. It consists of 10 digits. Only return these 10 digits.",
#"PROV_GROUP_TIN_SIGNATORY": "What is the Group provider's taxpayer identification number stated near signature in the contract? This is a 9 digit long number typically with a hyphen after the first 2 digits. Return only this 9 digit number with a hyphen after the first 2 digits.",
#"PROV_GROUP_TIN": "What is the Group provider's taxpayer identification number stated in the contract? This is a 9 digit long number typically with a hyphen after the first 2 digits in it. Return only this 9 digit number with a hyphen after the first 2 digits. As an example, it has the format of XX-XXXXXXX Do not elaborate or add context. Do not pull any dates, do not pull any numbers that deviate from earlier described pattern. It must always be 9 digits. Strictly follow this instructions. Do not return any other information.",
#"PROV_NPI_OTHER": 'What are all the other provider national provider identifier numbers (NPI) associated with this agreement? They consist of 10 digits and do not contain "-". Provide the list of all the other provider identifier numbers including those that are present in the table or attachment.',
#"PROV_TIN_OTHER": "What are the provider taxpayer identification numbers (TIN) for each of the other providers associated with this agreement? They consist of 9 digits and may contain a hyphen after the first 2 digits. This is usually listed in the preample of the contract or on a roster. Provide only the 9 digit TINs.",
# "PROV_GROUP_NPI": "What is the group provider's national provider identifier number mentioned in the contract? This may be found on the signature page or on the roster. It consists of 10 digits. Only return these 10 digits.",
# "PROV_GROUP_TIN_SIGNATORY": "What is the Group provider's taxpayer identification number stated near signature in the contract? This is a 9 digit long number typically with a hyphen after the first 2 digits. Return only this 9 digit number with a hyphen after the first 2 digits.",
# "PROV_GROUP_TIN": "What is the Group provider's taxpayer identification number stated in the contract? This is a 9 digit long number typically with a hyphen after the first 2 digits in it. Return only this 9 digit number with a hyphen after the first 2 digits. As an example, it has the format of XX-XXXXXXX Do not elaborate or add context. Do not pull any dates, do not pull any numbers that deviate from earlier described pattern. It must always be 9 digits. Strictly follow this instructions. Do not return any other information.",
# "PROV_NPI_OTHER": 'What are all the other provider national provider identifier numbers (NPI) associated with this agreement? They consist of 10 digits and do not contain "-". Provide the list of all the other provider identifier numbers including those that are present in the table or attachment.',
# "PROV_TIN_OTHER": "What are the provider taxpayer identification numbers (TIN) for each of the other providers associated with this agreement? They consist of 9 digits and may contain a hyphen after the first 2 digits. This is usually listed in the preample of the contract or on a roster. Provide only the 9 digit TINs.",
"TERMINATION_UPON_NOTICE": 'Extract the number of days can the contract be terminated by either party by giving notice. The answer may be found near the words "written notice of such termination" or "prior to the expiration". Only extract a value if one is present. Only return the number of days and nothing else.',
"TERMINATION_UPON_CAUSE": 'After how many days can the contract be terminated by either party upon breach of material term? The answer is usually found in section 7.2.2 or 10.2 but might be present in other sections as well. The answer is present after the Term language section. The answer may be found near the words "is in breach of any material term or condition". Here is an example of possible input text and the correct response. EXAMPLE INPUT TEXT: "By either party upon ninety (90) days prior written notice if the other party is in material breach of this Agreement, except that such termination shall not take place if the breach is cured within sixty (60) days following the written notice". EXAMPLE RESPONSE: 90.',
"TIME_TO_OBJECT": "How many days does the provider have to object to the amendment? The answer may be present in the Amendment section.",
@@ -614,6 +536,7 @@ AC_DICT = {
"RELATIONSHIP_OF_PARTIES_LANGUAGE": "Extract the entire Relationship of Parties passage present in the contract.",
}
def AC_EFFECTIVE_DATE_CLEANUP(contract_text):
return f"""### Contract Start ### {contract_text} ### Contract End ###
Above is a contract. What is the contract effective date mentioned in any of the following locations: the signatory section, the preamble of the agreement, or the start of the amendment? Look for phrases such as /'This amendment is effective/'.