Merged in feature/rate-limiting (pull request #190)
Implement global and per ip rate limiting * all tests passing * test fix * Enhances test environment for rate limiting Updates the test environment to better support rate limiting tests. - Increases rate limits in test container to avoid interference with legitimate test traffic. - Increases the polling interval for client status checks to reduce load on the rate limiter. - Adds logic to retry status checks if rate limiting is encountered. * Merge branch 'main' of bitbucket.org:aarete/query-orchestration into feature/rate-limiting * build fix * test fix
This commit is contained in:
Jay Brown
@@ -48,6 +48,216 @@ Terminates user session and clears authentication cookies.
|
||||
#### `GET /home`
|
||||
Protected dashboard endpoint showing user information.
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
The queryAPI service implements dual-layer rate limiting to protect against DDoS attacks and ensure fair resource allocation.
|
||||
|
||||
### Rate Limit Architecture
|
||||
|
||||
**Dual-Layer Protection:**
|
||||
- **Layer 1: Global/Aggregate Rate Limiting** - Protects total system capacity across all IP addresses
|
||||
- **Layer 2: Per-IP Rate Limiting** - Prevents individual client abuse
|
||||
|
||||
Both layers must pass for a request to proceed. The global limiter checks first for faster failure paths under DDoS conditions.
|
||||
|
||||
### Default Rate Limits
|
||||
|
||||
By default, all endpoints are limited to:
|
||||
- **Global**: 500 requests per second total (1000 burst)
|
||||
- **Per-IP**: 5 requests per second per IP (10 burst)
|
||||
- **Cleanup Interval**: 3 minutes for inactive IP limiters
|
||||
|
||||
**Important:** No endpoints are exempt from rate limiting. This is a security requirement to prevent DDoS attacks.
|
||||
|
||||
### Configuration
|
||||
|
||||
Rate limits are configured via environment variables:
|
||||
|
||||
```bash
|
||||
# Global/Aggregate rate limiting (protects total system capacity)
|
||||
RATE_LIMIT_GLOBAL_RATE=500 # total req/s across ALL IPs
|
||||
RATE_LIMIT_GLOBAL_BURST=1000 # total burst capacity across ALL IPs
|
||||
|
||||
# Per-IP rate limiting (protects against individual abusers)
|
||||
RATE_LIMIT_DEFAULT_RATE=5 # req/s per IP
|
||||
RATE_LIMIT_DEFAULT_BURST=10 # burst per IP
|
||||
RATE_LIMIT_EXPIRES_IN=180 # seconds (3 minutes)
|
||||
|
||||
# Per-endpoint overrides (JSON format)
|
||||
RATE_LIMIT_ENDPOINT_OVERRIDES={}
|
||||
```
|
||||
|
||||
### Configuring Endpoint-Specific Overrides
|
||||
|
||||
Endpoints can have custom rate limits based on their resource requirements. Configure using the `RATE_LIMIT_ENDPOINT_OVERRIDES` environment variable.
|
||||
|
||||
#### Endpoint Route Format
|
||||
|
||||
Routes use the format: `{HTTP_METHOD} {PATH}`
|
||||
|
||||
**Examples:**
|
||||
- `GET /health`
|
||||
- `POST /documents`
|
||||
- `GET /documents/{id}` (use route template, not actual IDs)
|
||||
- `POST /client/{id}/document/batch`
|
||||
|
||||
#### Override Configuration Structure
|
||||
|
||||
Each endpoint requires 4 values:
|
||||
|
||||
```json
|
||||
{
|
||||
"ENDPOINT_ROUTE": {
|
||||
"global_rate": 100, // Total req/s across ALL IPs
|
||||
"global_burst": 200, // Total burst across ALL IPs
|
||||
"rate": 5, // Req/s per individual IP
|
||||
"burst": 10 // Burst per individual IP
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Setting Environment Variable
|
||||
|
||||
**Single endpoint:**
|
||||
```bash
|
||||
export RATE_LIMIT_ENDPOINT_OVERRIDES='{"POST /documents": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10}}'
|
||||
```
|
||||
|
||||
**Multiple endpoints:**
|
||||
```bash
|
||||
export RATE_LIMIT_ENDPOINT_OVERRIDES='{
|
||||
"GET /health": {"global_rate": 500, "global_burst": 1000, "rate": 100, "burst": 200},
|
||||
"GET /metrics": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
|
||||
"GET /swagger/*": {"global_rate": 200, "global_burst": 400, "rate": 50, "burst": 100},
|
||||
"GET /auth/login": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
|
||||
"GET /auth/callback": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
|
||||
"GET /auth/logout": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
|
||||
"GET /auth/home": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
|
||||
"POST /documents": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
|
||||
"POST /documents/batch": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2},
|
||||
"GET /documents/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
|
||||
"GET /client/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
|
||||
"POST /query/test": {"global_rate": 20, "global_burst": 40, "rate": 2, "burst": 5},
|
||||
"POST /export": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2}
|
||||
}'
|
||||
```
|
||||
|
||||
**In .env file (single line, valid JSON):**
|
||||
```bash
|
||||
RATE_LIMIT_ENDPOINT_OVERRIDES={"GET /health":{"global_rate":500,"global_burst":1000,"rate":100,"burst":200},"POST /documents":{"global_rate":50,"global_burst":100,"rate":5,"burst":10}}
|
||||
```
|
||||
|
||||
#### Recommended Endpoint Tiers
|
||||
|
||||
**Tier 1: Monitoring/Infrastructure (High Limits)**
|
||||
```json
|
||||
{
|
||||
"GET /health": {"global_rate": 500, "global_burst": 1000, "rate": 100, "burst": 200},
|
||||
"GET /metrics": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
|
||||
"GET /swagger/*": {"global_rate": 200, "global_burst": 400, "rate": 50, "burst": 100}
|
||||
}
|
||||
```
|
||||
|
||||
**Tier 2: Authentication (Moderate - Prevent Brute Force)**
|
||||
```json
|
||||
{
|
||||
"GET /auth/login": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
|
||||
"GET /auth/callback": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20},
|
||||
"GET /auth/logout": {"global_rate": 100, "global_burst": 200, "rate": 10, "burst": 20}
|
||||
}
|
||||
```
|
||||
|
||||
**Tier 3: Read Operations (Moderate)**
|
||||
```json
|
||||
{
|
||||
"GET /client/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
|
||||
"GET /documents/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40},
|
||||
"GET /collector/{id}": {"global_rate": 200, "global_burst": 400, "rate": 20, "burst": 40}
|
||||
}
|
||||
```
|
||||
|
||||
**Tier 4: Write/Processing (Low - Resource Intensive)**
|
||||
```json
|
||||
{
|
||||
"POST /documents": {"global_rate": 50, "global_burst": 100, "rate": 5, "burst": 10},
|
||||
"POST /documents/batch": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2},
|
||||
"POST /query/test": {"global_rate": 20, "global_burst": 40, "rate": 2, "burst": 5},
|
||||
"POST /export": {"global_rate": 10, "global_burst": 20, "rate": 1, "burst": 2}
|
||||
}
|
||||
```
|
||||
|
||||
#### Verification
|
||||
|
||||
After restarting the service, check logs for confirmation:
|
||||
```
|
||||
level=INFO msg="Rate limiting enabled" global_rate=1000 global_burst=2000
|
||||
default_rate=10 default_burst=20 overrides_count=12
|
||||
```
|
||||
|
||||
The `overrides_count` should match your configured endpoints.
|
||||
|
||||
#### Troubleshooting
|
||||
|
||||
**Invalid JSON:** Validate with `echo $RATE_LIMIT_ENDPOINT_OVERRIDES | jq .`
|
||||
|
||||
**Override not applied:**
|
||||
- Verify HTTP method matches exactly (e.g., `POST` not `post`)
|
||||
- Use route template for paths with parameters (`/documents/{id}` not `/documents/123`)
|
||||
- Check for extra/missing slashes in path
|
||||
- Enable debug logging: `DEBUG=true` to see route matching
|
||||
|
||||
### Rate Limit Headers
|
||||
|
||||
All API responses include a `RateLimit` header indicating the current limit:
|
||||
|
||||
```
|
||||
RateLimit: 20;window=2
|
||||
```
|
||||
|
||||
**Format:** `{burst};window={seconds}`
|
||||
- `burst`: Maximum requests allowed in a burst
|
||||
- `window`: Time window in seconds (calculated as burst/rate)
|
||||
|
||||
### Rate Limit Exceeded Response
|
||||
|
||||
When the rate limit is exceeded, the API returns:
|
||||
|
||||
**HTTP Status:** `429 Too Many Requests`
|
||||
|
||||
**Headers:**
|
||||
- `Retry-After: {seconds}` - Number of seconds before retrying
|
||||
- `RateLimit: {burst};window={seconds}` - Current rate limit
|
||||
|
||||
**Response Body:**
|
||||
```json
|
||||
{
|
||||
"message": "rate limit exceeded"
|
||||
}
|
||||
```
|
||||
|
||||
### Testing Rate Limits
|
||||
|
||||
Test rate limiting behavior with curl:
|
||||
|
||||
```bash
|
||||
# Exceed rate limit by making rapid requests
|
||||
for i in {1..25}; do
|
||||
curl -i http://localhost:8080/client/AAA \
|
||||
-H "Authorization: Bearer $TOKEN"
|
||||
done
|
||||
|
||||
# Observe 429 responses after burst is exhausted
|
||||
```
|
||||
|
||||
### Technical Implementation
|
||||
|
||||
- **Algorithm**: Dual-layer token bucket (golang.org/x/time/rate)
|
||||
- **Layer 1**: Global rate limiter (shared across all IPs)
|
||||
- **Layer 2**: Per-IP rate limiter (separate bucket per IP via c.RealIP())
|
||||
- **Storage**: In-memory with automatic cleanup
|
||||
- **Middleware Position**: Applied BEFORE authentication to protect auth endpoints
|
||||
- **Protection**: Defends against both distributed DDoS and single-source attacks
|
||||
|
||||
**Security**: Requires authentication
|
||||
**Response**: HTML dashboard page
|
||||
|
||||
|
||||
Reference in New Issue
Block a user