query-orchestration/test/integration/README.md

# Admin User Management Integration Tests

This directory contains integration tests for admin user management functionality that use **real AWS Cognito and Permit.io APIs** (no mocks).

## Overview

These tests validate the complete user management workflow including:
- User creation in AWS Cognito and Permit.io
- User retrieval and attribute updates
- User enable/disable operations
- User deletion from both systems
- Role assignment and unassignment in Permit.io
- User listing with pagination
- Idempotent operations
- Environment-based user filtering (`custom:environment_id`)
- Environment access checks (matching, mismatched, untagged, nonexistent users)
- EULA service pipeline simulation (list + filter)

## Why Integration Tests Are Separate

The admin user management code is **excluded from standard CI/CD coverage requirements** because:
1. Free tier LocalStack doesn't support AWS Cognito
2. Integration tests require real AWS credentials
3. Integration tests require real Permit.io API access
4. Tests create and modify real user accounts
5. Tests should not run automatically in CI/CD pipelines

These tests are designed for **manual execution** during development and pre-production validation.

## Prerequisites

### AWS Requirements

1. **AWS SSO Session** - You must have an active SSO session for the `aarete` profile:
   ```bash
   task aws:login
   ```

2. **IAM Role** with the following Cognito permissions:
   ```json
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "cognito-idp:AdminCreateUser",
           "cognito-idp:AdminDeleteUser",
           "cognito-idp:AdminGetUser",
           "cognito-idp:AdminUpdateUserAttributes",
           "cognito-idp:AdminDisableUser",
           "cognito-idp:AdminEnableUser",
           "cognito-idp:ListUsers",
           "cognito-idp:DescribeUserPool",
           "cognito-idp:AddCustomAttributes"
         ],
         "Resource": "arn:aws:cognito-idp:REGION:ACCOUNT_ID:userpool/POOL_ID"
       }
     ]
   }
   ```

3. **Cognito User Pool ID** - `us-east-2_21upuTkkT` (from devbox.json)

### Permit.io Requirements

1. **Permit.io Account** with API access
2. **API Key** with permissions to:
   - Create users
   - Delete users
   - Assign/unassign roles
   - Read user roles
3. **Project ID** and **Environment ID** from Permit.io dashboard
4. **Roles configured** in Permit.io matching your `permit_policies.yaml`:
   - `super_admin`
   - `user_admin`
   - `auditor`
   - `client_user`

## Setup Instructions

### Step 1: Create Environment File

```bash
cd test/integration
cp .env.example .env
```

### Step 2: Configure Credentials

Edit `test/integration/.env` and fill in your actual credentials:

```bash
# AWS Configuration
AWS_REGION=us-east-2
COGNITO_USER_POOL_ID=us-east-2_21upuTkkT

# Suppress welcome emails to avoid AWS Cognito's 50 emails/day limit
COGNITO_SUPPRESS_EMAILS=true

# Permit.io Configuration
PERMIT_IO_API_KEY=permit_key_YOUR_KEY_HERE
PERMIT_IO_PROJECT_ID=default
PERMIT_IO_ENV_ID=dev
PERMIT_IO_TENANT=default
```

**IMPORTANT:** Never commit the `.env` file to git! It contains sensitive credentials.

### Step 3: Verify Permit.io Roles

Ensure the following roles exist in your Permit.io environment:
- `super_admin`
- `user_admin`
- `auditor`
- `client_user`

You can create roles through the Permit.io dashboard, API, or use the setup tool at `cmd/auth_related/permit.setup/`.

## Running the Tests

### CRITICAL: Devbox/LocalStack Environment Override

When running inside `devbox shell`, the following environment variables are set for LocalStack
and **will hijack all AWS SDK calls** (including Cognito) if not overridden:

| Variable | Devbox Value | Problem |
|---|---|---|
| `AWS_ENDPOINT_URL` | `http://localhost:4566` | Routes all calls to LocalStack |
| `AWS_ACCESS_KEY_ID` | `test` | Fake LocalStack credential |
| `AWS_SECRET_ACCESS_KEY` | `test` | Fake LocalStack credential |
| `AWS_SESSION_TOKEN` | (empty) | May conflict with SSO tokens |
| `AWS_REGION` | `us-east-1` | Cognito pool is in `us-east-2` |

**You MUST unset these variables** to reach real AWS Cognito. The recommended command prefix is:

```bash
env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN -u AWS_ENDPOINT_URL \
  AWS_REGION=us-east-2 AWS_PROFILE=aarete
```

### Quick Start (Full Suite)

1. Authenticate with AWS SSO:
   ```bash
   task aws:login
   ```

2. Verify credentials work (from devbox shell):
   ```bash
   env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN -u AWS_ENDPOINT_URL \
     aws sts get-caller-identity --profile aarete
   ```

3. Run all 29 integration tests:
   ```bash
   env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN -u AWS_ENDPOINT_URL \
     AWS_REGION=us-east-2 AWS_PROFILE=aarete \
     go test -tags=integration -v -count=1 -timeout=300s ./test/integration/...
   ```

### Run Only EnvFilter Tests (16 tests - Cognito only)

```bash
env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN -u AWS_ENDPOINT_URL \
  AWS_REGION=us-east-2 AWS_PROFILE=aarete \
  go test -tags=integration -v -count=1 -timeout=300s ./test/integration/... -run TestEnvFilterSuite
```

### Run Only Admin Tests (13 tests - Cognito + Permit.io)

```bash
env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN -u AWS_ENDPOINT_URL \
  AWS_REGION=us-east-2 AWS_PROFILE=aarete \
  go test -tags=integration -v -count=1 -timeout=300s ./test/integration/... -run TestSuite
```

### Run Only Permit.io Tests (No Cognito)

```bash
env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN -u AWS_ENDPOINT_URL \
  AWS_REGION=us-east-2 AWS_PROFILE=aarete \
  go test -tags=integration -v -count=1 -timeout=300s ./test/integration/... -run 'TestSuite/TestPermitIO'
```

### Run a Specific Test

```bash
env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN -u AWS_ENDPOINT_URL \
  AWS_REGION=us-east-2 AWS_PROFILE=aarete \
  go test -tags=integration -v -count=1 -timeout=300s ./test/integration/... -run TestEnvFilterSuite/TestCreateUser_SetsEnvironmentID
```

## Test Suites

The test suite includes **29 integration tests** organized into two suites:

### EnvFilter Suite (16 tests - `TestEnvFilterSuite`)

Tests environment-based user filtering with `custom:environment_id` in real Cognito:

| Test | Description |
|------|-------------|
| `TestCreateUser_SetsEnvironmentID` | Creates user with environment_id, verifies attribute persisted |
| `TestCreateUser_NoEnvironmentID` | Creates user without environment_id (legacy/untagged) |
| `TestGetUser_ExtractsEnvironmentID` | Fetches user and extracts custom:environment_id |
| `TestListUsers_FiltersByEnvironment` | Filters user list by environment, includes untagged |
| `TestListUsers_NoFilterWhenEnvEmpty` | Empty filter returns all users (backward compat) |
| `TestUserBelongsToEnvironment_Integration` | Tests ownership logic with real Cognito data |
| `TestUpdateUser_PreservesEnvironmentID` | Updating other attrs doesn't lose environment_id |
| `TestDeleteUser_EnvScoped` | Delete works for same-env, blocked for cross-env |
| `TestDisableUser_EnvScoped` | Disable works for same-env, blocked for cross-env |
| `TestEnableUser_EnvScoped` | Enable works for same-env, blocked for cross-env |
| `TestEnsureEnvironmentIDAttribute_Idempotent` | Schema registration is idempotent |
| `TestCheckEnvAccess_MatchingUser_RealCognito` | Matching env user passes access check |
| `TestCheckEnvAccess_UntaggedUser_RealCognito` | Untagged user passes (backward compat) |
| `TestCheckEnvAccess_MismatchedUser_RealCognito` | Mismatched env user is denied |
| `TestCheckEnvAccess_NonexistentUser_RealCognito` | Nonexistent user sub is denied |
| `TestEULA_FetchUsersFilteredByEnv_RealCognito` | EULA pipeline: list + filter by env |

### Admin Suite (13 tests - `TestSuite`)

Tests complete user management workflow across Cognito and Permit.io:

| Test | Description |
|------|-------------|
| `TestCreateUser` | Creates user in both Cognito and Permit.io |
| `TestCreateUserIdempotent` | Verifies idempotent user creation behavior |
| `TestGetUser` | Retrieves user details from Cognito |
| `TestUpdateUserAttributes` | Updates user's first and last name |
| `TestDisableEnableUser` | Disables and re-enables a user account |
| `TestDeleteUser` | Deletes user from both Cognito and Permit.io |
| `TestListUsers` | Lists users with pagination support |
| `TestRoleAssignment` | Assigns and unassigns roles in Permit.io |
| `TestPermitIO_CreateAndGetUser` | Creates user in Permit.io, verifies exists |
| `TestPermitIO_AssignMultipleRoles` | Assigns/unassigns multiple roles |
| `TestPermitIO_DeleteUser` | Deletes user, verifies with 404 |
| `TestPermitIO_UserLifecycle` | Complete 9-step lifecycle test |
| `TestPermitIO_IdempotentOperations` | Tests duplicate operations are idempotent |

## Test User Management

### Automatic Cleanup

The test suites implement automatic cleanup in `TearDownSuite`:
- All test users are tracked during creation
- After all tests complete, users are automatically deleted from both Cognito and Permit.io
- If cleanup fails, warnings are logged but tests still pass
- **Optional:** Set `SKIP_PERMITIO_CLEANUP=true` to preserve Permit.io users for manual inspection

### Manual Cleanup

If tests are interrupted and cleanup doesn't run:

```bash
# List test users in Cognito (from devbox shell)
env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN -u AWS_ENDPOINT_URL \
  aws cognito-idp list-users --user-pool-id us-east-2_21upuTkkT --profile aarete --region us-east-2

# Delete a specific user
env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN -u AWS_ENDPOINT_URL \
  aws cognito-idp admin-delete-user \
  --user-pool-id us-east-2_21upuTkkT \
  --username user@example.com \
  --profile aarete --region us-east-2
```

### Test Email Format

Test users are created with emails in these formats:
- Admin suite: `int-test-{name}-{timestamp}@example.com`
- EnvFilter suite: `envtest-{name}-{nanotimestamp}@example.com`

## AWS Cognito Email Limits

AWS Cognito has a **daily limit of 50 emails** when using the default email service. The test suite sets `COGNITO_SUPPRESS_EMAILS=true` to avoid this limit. Users are still created with `email_verified: true`.

## Troubleshooting

### Error: "Attributes did not conform to the schema: custom:environment_id"

**Cause:** The `custom:environment_id` attribute hasn't been registered on the Cognito user pool.
**Solution:** The EnvFilter suite's `SetupSuite` automatically calls `EnsureEnvironmentIDAttribute` to register it. If running individual tests, run the full `TestEnvFilterSuite` first.

### Error: "InvalidSecurityToken" or "UnrecognizedClientException"

**Cause:** Devbox/LocalStack credentials (`AWS_ACCESS_KEY_ID=test`) are being used instead of SSO.
**Solution:** Use the `env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN -u AWS_ENDPOINT_URL` prefix.

### Error: "Dial tcp localhost:4566: connection refused"

**Cause:** `AWS_ENDPOINT_URL=http://localhost:4566` is routing to LocalStack which isn't running.
**Solution:** Unset `AWS_ENDPOINT_URL` with the `env -u AWS_ENDPOINT_URL` prefix.

### Error: "ExpiredToken" or SSO session expired

**Solution:** Re-authenticate with `task aws:login` then re-run the tests.

### Error: Required environment variable not set

**Solution:** Ensure all required variables are set in `test/integration/.env`.

### Error: Role not found in Permit.io

**Solution:** Create the required roles in your Permit.io environment using `cmd/auth_related/permit.setup/`.

### Tests hang or timeout

**Solution:**
- Check network connectivity to AWS and Permit.io
- Verify AWS credentials are not expired (`task aws:login`)
- Try a longer timeout: `-timeout=600s`

## CI/CD Exclusion

The admin handler code is **intentionally excluded** from CI/CD coverage checks.
See `scripts/tests.yml` for the exclusion list. This allows `task fullsuite:ci` to pass
without requiring cloud credentials.

## Security Considerations

1. **Never commit credentials** - The `.env` file is gitignored
2. **Use test environments** - Don't run against production Cognito/Permit.io
3. **Limit IAM permissions** - Use least privilege for test credentials
4. **Rotate credentials** - Regularly rotate API keys and access keys
5. **Monitor test accounts** - Review test user creation in CloudWatch/Permit.io logs
6. **Clean up regularly** - Ensure test users are deleted after runs