query-orchestration/cmd/cognito_test/user.creation.tool/readme.md

Cognito + Permit.io User Lifecycle Sync Tool

A CLI tool that creates, updates, deletes, disables, and enables users in AWS Cognito and automatically syncs them to Permit.io with their Cognito Subject ID, maintaining role assignments. The tool is idempotent and supports dry-run mode with comprehensive audit logging.

Overview

This tool:

1. Reads user data from a CSV file (supports comments with #)
2. Idempotently creates or updates users in AWS Cognito
3. Idempotently creates or updates corresponding users in Permit.io
4. Intelligently manages roles - ensures users have exactly the roles specified in the CSV (removes old roles, adds new ones)
5. Provides comprehensive audit logging for compliance
6. Supports dry-run mode for testing without side effects
7. Reports success/failure for each step

Key Features

• 🔄 Idempotent Operations: Safe to run multiple times with the same data
• 🎯 Smart Role Management: Automatically adds/removes roles to match CSV specification
• 🔍 Dry-Run Mode: Test operations without making actual changes
• 📋 Audit Logging: Complete audit trail for compliance requirements
• 💬 CSV Comments: Support for comment lines starting with #
• ⚡ Incremental Updates: Only processes changes, skips existing data
• 🔐 User Access Control: Disable/enable users in Cognito without deletion

Operation Modes

Delete Mode (--delete-users)

Runs the inverse of the normal create operation, deleting any users listed in the CSV:

1. Reads user data from a CSV file
2. Finds each user in Permit.io by email (skips if not found)
3. Deletes the user from Permit.io (if exists)
4. Deletes the user from Cognito (if exists)
5. Reports success/failure for each step

Note: For delete mode, only email addresses are required in the CSV file (names and roles are optional).

email,first_name,last_name,admin,viewer
"foo1@gmail.com",,,,
"foo2@gmail.com",,,,

Disable Mode (--disable-users)

Disables users in AWS Cognito without deleting them. This prevents users from logging in while preserving their account and settings:

1. Reads user data from a CSV file
2. Checks if each user exists in Cognito
3. Disables the user account in Cognito (if exists and enabled)
4. Reports success/failure for each step
5. Idempotent: Reports "already disabled" for users who are already disabled

Important Notes:

• Disabled users cannot authenticate or access applications
• User data, roles, and settings are preserved
• Users can be re-enabled later using --enable-users
• Only affects Cognito - does not modify Permit.io roles or user data
• For disable mode, only email addresses are required in the CSV file (names and roles are optional)

Enable Mode (--enable-users)

Re-enables previously disabled users in AWS Cognito:

1. Reads user data from a CSV file
2. Checks if each user exists in Cognito
3. Enables the user account in Cognito (if exists and disabled)
4. Reports success/failure for each step
5. Idempotent: Reports "already enabled" for users who are already enabled

Important Notes:

• Only works on existing Cognito users
• Does not create new users - use default mode for user creation
• Only affects Cognito enable/disable status - does not modify roles
• For enable mode, only email addresses are required in the CSV file (names and roles are optional)

Prerequisites

1. AWS Account with:

   • An existing Cognito User Pool
   • Appropriate IAM permissions for Cognito operations
   • AWS credentials configured

   If you are running this from the devbox environment you can use the task aws:login to perform the login for you and authenticate your cli to be the same as your broswer aws credentials.
   If you are running this in the aws cloud console it should just work.

2. Permit.io Account with:

   • A project and environment set up
   • API key with user creation permissions (organization, project, or environment-level key)

Configuration

Permit.io API Key Types

Permit.io supports three levels of API keys, and this tool works with all of them:

Key Type	Scope	Usage
Organization	All projects and environments	Use with explicit -project and -env flags
Project	Single project, all its environments	Use with -project use-key and explicit -env flag
Environment	Single environment only	Use with -project use-key (auto-detects both project and env)

Recommendation: Use environment-scoped keys for production deployments as they provide the principle of least privilege.

Environment Variables

Create a .env file (see .env.example):

# Permit.io Configuration
PERMIT_KEY=your-permit-api-key-here

# AWS Cognito Configuration
COGNITO_USER_POOL_ID=us-east-1_xxxxxxxxx

# AWS Configuration
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your-access-key        # Or use IAM role
AWS_SECRET_ACCESS_KEY=your-secret-key
# or use profile
AWS_PROFILE=name_of_sso_profile_that_is_already_logged_in

CSV Format

The CSV file should have the following format (no user_id needed):

email,first_name,last_name,role1,role2,...,roleN
# This is a comment line - will be ignored
john.doe@example.com,John,Doe,admin,viewer
jane.smith@example.com,Jane,Smith,editor,viewer
# Another comment
bob.wilson@example.com,Bob,Wilson,admin

• email: User's email address (required for all operations, used as Cognito username)
• first_name: User's first name (required for create operations only, optional for delete/disable/enable)
• last_name: User's last name (required for create operations only, optional for delete/disable/enable)
• role columns: Up to 99 role assignments (empty cells are ignored)
• Comments: Lines starting with # are ignored

Important: When creating users, both first_name and last_name are required. For delete, disable, and enable operations, only the email field is necessary.

Usage

Basic Usage

# Load environment variables
export $(cat .env | xargs)

# Create/update users (idempotent)
./cognito-permit-sync -project myproject -env production -csv users.csv

# Dry run to see what would happen
./cognito-permit-sync -project myproject -env production -csv users.csv --dry-run

# Delete users
./cognito-permit-sync -project myproject -env production -csv users.csv --delete-users

# Disable users (prevents login, preserves account)
./cognito-permit-sync -project myproject -env production -csv users.csv --disable-users

# Enable users (re-enables disabled accounts)
./cognito-permit-sync -project myproject -env production -csv users.csv --enable-users

# Dry run operations (works with all modes)
./cognito-permit-sync -project myproject -env production -csv users.csv --delete-users --dry-run
./cognito-permit-sync -project myproject -env production -csv users.csv --disable-users --dry-run
./cognito-permit-sync -project myproject -env production -csv users.csv --enable-users --dry-run

List Available Projects and Environments

./cognito-permit-sync -project list

Using Project/Environment-Scoped API Keys

If you have a project-scoped or environment-scoped Permit.io API key (rather than an organization-level key), you can use -project use-key to automatically detect the project and environment from the API key:

# With an environment-scoped key (auto-detects both project and environment)
./cognito-permit-sync -project use-key -csv users.csv

# With a project-scoped key (auto-detects project, requires -env flag)
./cognito-permit-sync -project use-key -env production -csv users.csv

This is useful when:

• You have a scoped API key and don't know the project/environment IDs
• You want to avoid hardcoding project IDs in scripts
• You're working with environment-specific deployments

To check what project/environment your API key is scoped to:

./get.scope.from.key.sh your_permit_api_key

Command Line Options

• -project: Permit.io project ID/key, or special values:
  • list - Show available projects and environments
  • use-key - Auto-detect project from API key scope
• -env: Permit.io environment ID or key (required unless using use-key with an environment-scoped API key)
• -csv: Path to CSV file containing users (required)
• -api-url: Permit.io API base URL (default: https://api.permit.io)
• --delete-users: Delete users instead of creating/updating them
• --disable-users: Disable users in Cognito instead of creating/updating them
• --enable-users: Enable users in Cognito instead of creating/updating them
• --dry-run: Perform a dry run without making actual changes

Important: Only one operation mode can be specified at a time (--delete-users, --disable-users, or --enable-users). If none are specified, the tool defaults to create/update mode.

How It Works

Create/Update Mode (Default)

For each user in the CSV:

1. Checks if user exists in Cognito:

   • If exists: Retrieves existing user details (including Subject ID)
   • If not exists: Creates new user with email as username, sets email_verified to true

2. Checks if user exists in Permit.io:

   • If exists: Uses existing user
   • If not exists: Creates user with Cognito Subject ID as key

3. Manages roles intelligently:

   • Gets current roles assigned to user
   • Removes roles that are no longer needed
   • Adds new roles specified in CSV
   • Skips roles that are already assigned correctly

Syncing Existing Cognito Users to a New Permit.io Project

The tool handles the scenario where users already exist in Cognito but need to be synced to a different Permit.io project:

• Cognito: User is detected as existing, details are retrieved (no changes made)
• Permit.io: User is created in the target project using their existing Cognito Subject ID
• Roles: Assigned according to the CSV specification

This is useful when:

• Migrating users to a new Permit.io project
• Setting up a new environment with existing Cognito users
• Syncing users across multiple Permit.io projects

Delete Mode (--delete-users)

For each user in the CSV:

1. Finds user in Permit.io by email:

   • If found: Proceeds with deletion
   • If not found: Skips Permit.io deletion

2. Deletes from Permit.io (if user exists)

3. Checks if user exists in Cognito:

   • If found: Deletes user
   • If not found: Skips Cognito deletion

Disable Mode (--disable-users)

For each user in the CSV:

1. Checks if user exists in Cognito:

   • If not found: Reports error and skips
   • If found: Proceeds with disable check

2. Checks current user status:

   • If already disabled: Reports "already disabled" and skips
   • If enabled: Proceeds with disable operation

3. Disables user in Cognito:

   • Sets user's enabled status to false
   • User cannot authenticate until re-enabled
   • All user data and settings are preserved

Enable Mode (--enable-users)

For each user in the CSV:

1. Checks if user exists in Cognito:

   • If not found: Reports error and skips
   • If found: Proceeds with enable check

2. Checks current user status:

   • If already enabled: Reports "already enabled" and skips
   • If disabled: Proceeds with enable operation

3. Enables user in Cognito:

   • Sets user's enabled status to true
   • User can authenticate normally
   • All previous settings and roles are restored

Dry-Run Mode (--dry-run)

Works with all operation modes:

• Simulates all operations without making actual API calls
• Shows exactly what would be done
• Logs all actions to dry-run-audit.log
• Perfect for testing and validation
• Recommended: Always test with --dry-run first, especially for disable/enable operations

Audit Logging

Audit Files

• Normal operations: Logged to audit.log
• Dry-run operations: Logged to dry-run-audit.log

Audit Log Format

Each audit entry includes:

• Timestamp (YYYY-MM-DD HH:MM:SS)
• Action type (CREATE, DELETE, DISABLE, ENABLE, ASSIGN_ROLE, UNASSIGN_ROLE, SKIP)
• System (COGNITO, PERMIT)
• User identifier (email and/or subject ID)
• Result (SUCCESS, FAILURE, SKIPPED)
• Additional details

Sample Audit Entries

# Create/Update operations
2024-06-04 10:30:15 [CREATE] [COGNITO] [john.doe@example.com|abc123-def456] SUCCESS: User created successfully
2024-06-04 10:30:16 [SKIP] [PERMIT] [john.doe@example.com|abc123-def456] SKIPPED: User already exists
2024-06-04 10:30:17 [UNASSIGN_ROLE] [PERMIT] [john.doe@example.com] SUCCESS: role='old_role', Role removed successfully
2024-06-04 10:30:18 [ASSIGN_ROLE] [PERMIT] [john.doe@example.com] SUCCESS: role='admin', Role assigned successfully

# Disable/Enable operations
2024-06-04 11:15:20 [DISABLE] [COGNITO] [john.doe@example.com] SUCCESS: User disabled successfully
2024-06-04 11:15:21 [SKIP] [COGNITO] [jane.smith@example.com] SKIPPED: User already disabled
2024-06-04 11:20:30 [ENABLE] [COGNITO] [john.doe@example.com] SUCCESS: User enabled successfully

# Delete operations
2024-06-04 12:00:45 [DELETE] [PERMIT] [john.doe@example.com|abc123-def456] SUCCESS: User deleted successfully
2024-06-04 12:00:46 [DELETE] [COGNITO] [john.doe@example.com] SUCCESS: User deleted successfully

Compliance Benefits

• Complete audit trail of all user lifecycle operations
• Immutable log entries with precise timestamps
• Action tracking for both successful and failed operations
• Role change tracking for access control compliance
• Dry-run logging for change planning and approval processes

Output Example

🔍 DRY RUN MODE - No actual changes will be made
============================================================
Found 2 users to import
Will create users in Cognito User Pool: us-east-2_21upuTkkT
Then sync to Permit.io project: ebde1e1e9623491cab6f8112e67bd61c
============================================================

Processing user 1/2: john.doe@example.com
  ℹ️  User already exists in Cognito
  ℹ️  User already exists in Permit.io with key: abc123-def456-789
  ✓ Removed role: old_admin
  🔍 [DRY RUN] Would assign role: editor
  ℹ️  Role already assigned: viewer
  ✅ [DRY RUN] User would be successfully processed

Processing user 2/2: jane.smith@example.com
  🔍 [DRY RUN] Would create user in Cognito
  🔍 [DRY RUN] Would create user in Permit.io
  🔍 [DRY RUN] Would assign role: admin
  ✅ [DRY RUN] User would be successfully processed

============================================================
IMPORT SUMMARY
============================================================
Total users processed: 2
Successfully created: 2
Failed: 0
Time taken: 1.234s
Average time per user: 617ms

🔍 DRY RUN COMPLETED - Check dry-run-audit.log for details

Error Handling

The tool handles various error scenarios gracefully:

• Duplicate users: Safely skips creation, proceeds with role management
• Missing users: Safely skips deletion if user doesn't exist
• Partial failures: Continues processing other users
• API errors: Detailed error messages with context
• Network errors: Proper error reporting and logging

Idempotent Behavior

Safe to Run Multiple Times

• User Creation: Won't fail if user already exists
• Role Management: Only makes necessary changes (add missing roles, remove extra roles)
• Partial Recovery: Can complete partially failed runs
• No Duplicate Data: Smart detection prevents duplicate entries

Role Management Examples

If a user currently has roles [admin, viewer, old_role] and CSV specifies [admin, editor]:

• ✅ Keeps: admin (already assigned)
• ➕ Adds: editor (new role)
• ➖ Removes: viewer, old_role (no longer needed)

IAM Permissions Required

Your AWS credentials need these Cognito permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cognito-idp:AdminCreateUser",
        "cognito-idp:AdminGetUser",
        "cognito-idp:AdminDeleteUser",
        "cognito-idp:AdminDisableUser",
        "cognito-idp:AdminEnableUser"
      ],
      "Resource": "arn:aws:cognito-idp:REGION:ACCOUNT:userpool/USER_POOL_ID"
    }
  ]
}

New Permissions Required:

• cognito-idp:AdminDisableUser: Required for --disable-users functionality
• cognito-idp:AdminEnableUser: Required for --enable-users functionality

Best Practices

1. Always Dry-Run First: Use --dry-run to validate changes before applying
2. Review Audit Logs: Check audit logs for compliance and troubleshooting
3. Incremental Updates: The tool handles incremental changes efficiently
4. Backup Strategy: Keep CSV files versioned for rollback capability
5. Monitor Performance: Use built-in timing information for optimization
6. Test with Small Batches: Validate with a few users before bulk operations
7. Disable vs Delete:
   • Use --disable-users to temporarily prevent access while preserving user data
   • Use --delete-users only when permanently removing users
   • Disabled users can be re-enabled with --enable-users
8. Staged Deployments: Test disable/enable operations in development environments first

Troubleshooting

"User already exists" (No Longer an Error)

• ✅ Fixed: Tool now handles existing users gracefully
• ✅ Action: Continues with role management for existing users

Audit Log Analysis

# Check recent audit activity
tail -f audit.log

# Filter for failures
grep "FAILURE" audit.log

# Check role changes
grep "ROLE" audit.log

# Check disable/enable operations
grep "DISABLE\|ENABLE" audit.log

# Analyze dry-run results
cat dry-run-audit.log

Disable/Enable Issues

• Permission Errors: Ensure your AWS credentials have AdminDisableUser and AdminEnableUser permissions
• User Not Found: Disable/enable operations only work on existing Cognito users
• Already Disabled/Enabled: The tool reports current state - this is normal idempotent behavior
• Verification: Check audit logs to confirm operations completed successfully

Role Synchronization Issues

• Check audit log for role assignment/removal details
• Verify CSV format for role columns
• Use dry-run mode to preview role changes

API Rate Limiting

• Built-in 100ms delays between operations
• Monitor API response times in console output
• Consider breaking large CSV files into smaller batches

Development

Running Tests

Set your .env with the required variables or make them part of your environment:

PERMIT_KEY="permit_key_redacted..."
COGNITO_USER_POOL_ID="us-east-2_21upuTkkT"
AWS_REGION=us-east-2
AWS_PROFILE=aarete

go test -tags=aws -v ./...

Building for Different Platforms

# Linux
GOOS=linux GOARCH=amd64 go build -o cognito-permit-sync-linux

# macOS
GOOS=darwin GOARCH=amd64 go build -o cognito-permit-sync-darwin

# Windows
GOOS=windows GOARCH=amd64 go build -o cognito-permit-sync.exe

Recent Improvements ✨

• ✅ Idempotent Operations: Safe to run multiple times
• ✅ Smart Role Management: Automatically sync roles to match CSV
• ✅ Audit Logging: Complete compliance trail with timestamps
• ✅ Dry-Run Mode: Test changes without side effects
• ✅ CSV Comments: Support for comment lines with #
• ✅ Better Error Handling: Graceful handling of missing/existing users
• ✅ Performance Insights: Detailed timing and success metrics
• ✅ User Disable/Enable: Temporarily disable users without deletion
• ✅ Enhanced Access Control: Fine-grained user access management
• ✅ Extended Audit Trail: Track all disable/enable operations
• ✅ Idempotent Disable/Enable: Smart state checking prevents unnecessary operations
• 🆕 Scoped API Key Support: Use -project use-key to auto-detect project/environment from API key
• 🆕 Cross-Project User Sync: Sync existing Cognito users to new Permit.io projects

Future Enhancements

• Batch operations for even better performance
• Progress bar for large imports
• Export failed users to CSV for retry
• Support for custom Cognito attributes
• Multi-tenant support in Permit.io
• Configuration file support
• Integration with CI/CD pipelines

License

[Your License Here]