141 lines
6.9 KiB
Plaintext
141 lines
6.9 KiB
Plaintext
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"The first part of the process is to take a list of file names and their respective page counts and perform a simple de-duplication process that takes care of renaming files which are not actually duplicates and discarding those which are \"true\" duplicates. \n",
|
||
|
|
"* We decide this based on the page counts. If the same file names have different page counts, they are not actually duplicates. However if the file name is same and the page count is same - they are classified as \"true\" duplicates.\n",
|
||
|
|
"\n",
|
||
|
|
"The Input CSV should essentially have two important columns - All the File Names, File Paths, and their Page Counts.\n",
|
||
|
|
"\n",
|
||
|
|
"Our output for duplicate file names should rename them as (1), (2), .... and so on.\n",
|
||
|
|
"\n",
|
||
|
|
"Follow the comments to understand the step by step procedure taken to handle de-duplication and conduct renaming."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"# Importing pandas to deal with the csv as a dataframe\n",
|
||
|
|
"import pandas as pd\n",
|
||
|
|
"# Importing concurrent futures to support multithreading\n",
|
||
|
|
"import concurrent.futures\n",
|
||
|
|
"\n",
|
||
|
|
"# Helper Function that takes in a group of files with the exact same file name and performs the core logic\n",
|
||
|
|
"def process_file_group(group):\n",
|
||
|
|
" # This first drops rows with same file name and same page count and then sorts the page counts in order\n",
|
||
|
|
" group = group.drop_duplicates(subset=['File Name', 'Page Count']).sort_values('Page Count').reset_index(drop=True)\n",
|
||
|
|
" # This is just a simple function to add the index as (n) in the correct place of the file name\n",
|
||
|
|
" def add_suffix(file_name, idx):\n",
|
||
|
|
" if file_name.endswith('.Pdf'):\n",
|
||
|
|
" return file_name[:-4] + f'({idx + 1}).Pdf'\n",
|
||
|
|
" elif file_name.endswith('.pdf'):\n",
|
||
|
|
" return file_name[:-4] + f'({idx + 1}).pdf'\n",
|
||
|
|
" else:\n",
|
||
|
|
" return file_name + f'({idx + 1})'\n",
|
||
|
|
" # This run the above function on all of the file names in the group and populates those in a new column\n",
|
||
|
|
" group['New File Name'] = [add_suffix(file_name, idx) for idx, file_name in enumerate(group['File Name'])]\n",
|
||
|
|
" return group\n",
|
||
|
|
"\n",
|
||
|
|
"input_csv = 'icertis/icertis_files.csv'\n",
|
||
|
|
"df = pd.read_csv(input_csv)\n",
|
||
|
|
"# This is a good trick to identify rows which have duplicate file names and keep only those in the dataframe\n",
|
||
|
|
"duplicates_only = df[df.duplicated(subset=['File Name'], keep=False)]\n",
|
||
|
|
"# The group by function is used to bring all the same file names in the same group so that we can process and relabel those\n",
|
||
|
|
"grouped = duplicates_only.groupby(['File Name'])\n",
|
||
|
|
"print(grouped.head())\n",
|
||
|
|
"# This is a typical multithreading technique to run the process function on every group and store the results\n",
|
||
|
|
"with concurrent.futures.ThreadPoolExecutor() as executor:\n",
|
||
|
|
" results = executor.map(process_file_group, [group for _, group in grouped])\n",
|
||
|
|
"\n",
|
||
|
|
"# Now we simply put back the results into an output csv\n",
|
||
|
|
"optimized_df = pd.concat(results)\n",
|
||
|
|
"optimized_df.to_csv('icertis/icertis_duplicate_files.csv', index=False)\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"This the second step where we do two things - \n",
|
||
|
|
"* figure out which files we are concerned about in the current processing and get a csv of those original file names.\n",
|
||
|
|
"* Apply a custom 5-pager logic that does the following \n",
|
||
|
|
" * For a specific file name, if we have a file that has greater than 5 additional pages compared to all other duplicates of that file, we only take that max page count file\n",
|
||
|
|
" * If there are files within the 5 page range of the max page count file, we are not sure which file to consider and hence we add all those files\n",
|
||
|
|
"\n",
|
||
|
|
"We need to inputs here, one is the list of file names to consider and second is the previously generated csv of all duplicate files handled.\n",
|
||
|
|
"\n",
|
||
|
|
"Follow the comments to understand the step by step procedure taken"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"# Importing pandas to deal with the csv as a dataframe\n",
|
||
|
|
"import pandas as pd\n",
|
||
|
|
"\n",
|
||
|
|
"# Here we load the two input csvs and set up the output dataframe\n",
|
||
|
|
"df_dup = pd.read_csv('duplicate_files_list.csv')\n",
|
||
|
|
"# df_dup['File Name'] = df_dup['File Name'][:-4]\n",
|
||
|
|
"df_priority = pd.read_csv('batch6_16_comparison_output.csv')\n",
|
||
|
|
"df_priority = df_priority[df_priority['Only in Master Tracker'].str.contains(r'\\(\\d+\\)', na=False)]\n",
|
||
|
|
"# df_priority['File Name'] = df_priority['File Name'][:-4]\n",
|
||
|
|
"output_df = pd.DataFrame(columns=df_dup.columns)\n",
|
||
|
|
"\n",
|
||
|
|
"# Here we iterate over all the file names in the concerned file name list\n",
|
||
|
|
"print(len(df_priority['Only in Master Tracker']))\n",
|
||
|
|
"for file in df_priority['Only in Master Tracker']:\n",
|
||
|
|
" # We extract the relevant rows from the duplicate files that match the name\n",
|
||
|
|
" relevant_rows = df_dup[df_dup['New File Name'] == (file+\".Pdf\")]\n",
|
||
|
|
" # If there is only one matching row, we have no logic needed and can just use that\n",
|
||
|
|
" if len(relevant_rows) == 1:\n",
|
||
|
|
" output_df = pd.concat([output_df, relevant_rows])\n",
|
||
|
|
" # Otherwise we apply the 5 pager logic\n",
|
||
|
|
" elif len(relevant_rows) > 1:\n",
|
||
|
|
" # We use this max page count of all the matching files to compare against others\n",
|
||
|
|
" max_page_count = max(relevant_rows['Page Count'])\n",
|
||
|
|
" flag = False\n",
|
||
|
|
" for pgcnt in relevant_rows['Page Count'].unique():\n",
|
||
|
|
" # If we find another matching file within 5 pages of the max count, we can add that to our output\n",
|
||
|
|
" if max_page_count - pgcnt <= 5:\n",
|
||
|
|
" output_df = pd.concat([output_df, relevant_rows[relevant_rows['Page Count'] == pgcnt]])\n",
|
||
|
|
" else:\n",
|
||
|
|
" print(file)\n",
|
||
|
|
"\n",
|
||
|
|
"# here we do some post processing to ensure we are not keeping full duplicate rows and then convert the ouput df to a csv\n",
|
||
|
|
"output_df = output_df.drop_duplicates()\n",
|
||
|
|
"output_df.reset_index(drop=True, inplace=True)\n",
|
||
|
|
"output_df.to_csv('batch6_16_output_file_2.csv')\n"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"metadata": {
|
||
|
|
"kernelspec": {
|
||
|
|
"display_name": ".venv",
|
||
|
|
"language": "python",
|
||
|
|
"name": "python3"
|
||
|
|
},
|
||
|
|
"language_info": {
|
||
|
|
"codemirror_mode": {
|
||
|
|
"name": "ipython",
|
||
|
|
"version": 3
|
||
|
|
},
|
||
|
|
"file_extension": ".py",
|
||
|
|
"mimetype": "text/x-python",
|
||
|
|
"name": "python",
|
||
|
|
"nbconvert_exporter": "python",
|
||
|
|
"pygments_lexer": "ipython3",
|
||
|
|
"version": "3.12.3"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"nbformat": 4,
|
||
|
|
"nbformat_minor": 2
|
||
|
|
}
|