Introduction
API idempotency key conflict errors occur when an API rejects or mishandles requests due to idempotency key collisions, causing duplicate transactions, lost updates, or rejected legitimate requests. Idempotency ensures that making the same API request multiple times produces the same result as making it once—a critical requirement for payment processing, order creation, and state-changing operations where network retries are common. Common causes include idempotency key reuse after TTL expiration, key generation collision (non-unique keys), race conditions during key registration, storage backend failures losing key history, idempotency key scope mismatch (different users/endpoints sharing keys), cache eviction purging key history prematurely, distributed system clock skew affecting TTL calculation, and incorrect implementation returning cached error responses instead of reprocessing. The fix requires implementing robust idempotency key generation, proper storage with appropriate TTL, atomic key registration, and correct handling of idempotent vs non-idempotent requests. This guide provides production-proven patterns for API idempotency across REST APIs, payment processors (Stripe, PayPal), order management systems, and distributed architectures.
Symptoms
409 Conflict: Idempotency key already used422 Unprocessable Entity: Duplicate request detected- Duplicate charges or orders despite idempotency key
- Request rejected but idempotency key still consumed
- Different response returned on retry (not idempotent)
- Idempotency key "expired" but request never completed
- Race condition creates duplicate resources with same key
- Cached error response returned on legitimate retry
- Idempotency key works once but fails on second request
- Distributed nodes have inconsistent idempotency state
Common Causes
- Idempotency key reused after TTL expired (key purged from storage)
- Non-unique key generation (collision between different requests)
- Race condition: two requests with same key processed simultaneously
- Storage backend (Redis, database) lost idempotency key record
- Key scope too narrow (keys shared across users or contexts)
- Cache eviction (LRU) purged key before TTL expiration
- Clock skew between servers affecting TTL calculation
- Error responses cached and returned on retry (should reprocess)
- Idempotency not implemented for all HTTP methods correctly
- Distributed system without shared idempotency storage
Step-by-Step Fix
### 1. Implement correct idempotency key handling
Idempotency key validation:
```python # Python/Flask example with correct idempotency from flask import Flask, request, jsonify import redis import json import hashlib from datetime import timedelta
app = Flask(__name__) redis_client = redis.Redis(host='localhost', port=6379)
# Idempotency TTL: 24 hours (adjust based on use case) IDEMPOTENCY_TTL = timedelta(hours=24)
def get_idempotency_key(): """Extract idempotency key from request headers.""" return request.headers.get('Idempotency-Key')
def check_idempotency(key: str): """ Check if request with this key was already processed. Returns (is_duplicate, stored_response) """ stored = redis_client.get(f"idempotency:{key}")
if stored: data = json.loads(stored) return True, data.get('response')
return False, None
def store_idempotency_result(key: str, response: dict, status_code: int): """Store result for future idempotent requests.""" data = { 'response': response, 'status_code': status_code, 'timestamp': datetime.utcnow().isoformat() }
# Use SETNX for atomic operation + EXPIRE for TTL pipe = redis_client.pipeline() pipe.setnx(f"idempotency:{key}", json.dumps(data)) pipe.expire(f"idempotency:{key}", int(IDEMPOTENCY_TTL.total_seconds())) pipe.execute()
@app.route('/api/payments', methods=['POST']) def create_payment(): idempotency_key = get_idempotency_key()
if not idempotency_key: return jsonify({'error': 'Idempotency-Key header required'}), 400
# Check if already processed is_duplicate, cached_response = check_idempotency(idempotency_key)
if is_duplicate: # Return cached response (idempotent behavior) return jsonify(cached_response), 200
# Process the payment try: payment_data = request.get_json()
# Validate payment if not validate_payment(payment_data): error_response = {'error': 'Invalid payment data'} # Don't store error responses for idempotency # (allow retry on validation failure) return jsonify(error_response), 400
# Create payment (idempotent operation) payment = process_payment(payment_data)
response = { 'id': payment.id, 'status': payment.status, 'amount': payment.amount }
# Store successful response for future idempotent requests store_idempotency_result(idempotency_key, response, 200)
return jsonify(response), 200
except PaymentError as e: # Store error response too (so retry returns same error) error_response = {'error': str(e)} store_idempotency_result(idempotency_key, error_response, 500) return jsonify(error_response), 500
def validate_payment(data: dict) -> bool: """Validate payment data.""" return all(k in data for k in ['amount', 'currency', 'source'])
def process_payment(data: dict) -> Payment: """Process actual payment.""" # Payment processing logic pass
class Payment: def __init__(self, id, status, amount): self.id = id self.status = status self.amount = amount
class PaymentError(Exception): pass ```
### 2. Fix idempotency key generation
Generate unique idempotency keys:
```python # BAD: Non-unique key generation def bad_key_generation(user_id, amount): # Collision likely if same user sends same amount return f"{user_id}:{amount}"
# GOOD: Include timestamp and random component import uuid from datetime import datetime
def good_key_generation(user_id, request_data): """Generate unique idempotency key client-side.""" # Client should generate this, not server # But if server must: include request hash + timestamp + random request_hash = hashlib.sha256( json.dumps(request_data, sort_keys=True).encode() ).hexdigest()[:16]
timestamp = datetime.utcnow().strftime('%Y%m%d%H%M%S') random_part = uuid.uuid4().hex[:8]
return f"{user_id}:{timestamp}:{request_hash}:{random_part}"
# BEST: Client generates and sends key # Client-side key generation (JavaScript) def generateIdempotencyKey(requestData) { // Include: user ID, endpoint, request body hash, timestamp const bodyHash = crypto .createHash('sha256') .update(JSON.stringify(requestData)) .digest('hex') .substring(0, 16);
const timestamp = Date.now(); const random = crypto.randomBytes(4).toString('hex');
return usr_${userId}_${timestamp}_${bodyHash}_${random};
}
// Use key in request fetch('/api/payments', { method: 'POST', headers: { 'Content-Type': 'application/json', 'Idempotency-Key': generateIdempotencyKey(paymentData) }, body: JSON.stringify(paymentData) }); ```
Key scope and isolation:
```python # Idempotency keys should be scoped appropriately
# BAD: Global key scope (keys can collide across users) def check_idempotency_bad(key): return redis_client.get(f"idempotency:{key}") # Global namespace
# GOOD: Scope by user/tenant def check_idempotency_scoped(key: str, user_id: str): # Keys are unique per user namespaced_key = f"idempotency:user:{user_id}:{key}" return redis_client.get(namespaced_key)
# BETTER: Include endpoint in scope def check_idempotency_full(key: str, user_id: str, endpoint: str): # Same key can be used for different endpoints namespaced_key = f"idempotency:user:{user_id}:endpoint:{endpoint}:{key}" return redis_client.get(namespaced_key)
# Store with full scoping def store_idempotency(key: str, user_id: str, endpoint: str, response: dict): namespaced_key = f"idempotency:user:{user_id}:endpoint:{endpoint}:{key}" data = { 'response': response, 'timestamp': datetime.utcnow().isoformat() } redis_client.setex( namespaced_key, int(timedelta(hours=24).total_seconds()), json.dumps(data) ) ```
### 3. Fix race condition issues
Atomic idempotency check-and-store:
```python # Race condition: Two requests with same key arrive simultaneously # Both check idempotency (not found), both process, both store # Result: Duplicate charge!
# Solution: Use atomic operations
# Redis Lua script for atomic check-and-store LOCK_SCRIPT = """ local key = KEYS[1] local response_key = KEYS[2] local ttl = tonumber(ARGV[1]) local lock_value = ARGV[2] local response_data = ARGV[3]
-- Try to acquire lock (NX = only if not exists) local lock_acquired = redis.call('SET', key, lock_value, 'NX', 'EX', ttl)
if lock_acquired then -- Lock acquired, process request -- Store response redis.call('SET', response_key, response_data, 'EX', ttl) -- Release lock redis.call('DEL', key) return {1, response_data} else -- Lock not acquired, another request is processing -- Wait and return cached response local cached = redis.call('GET', response_key) if cached then return {0, cached} else -- Still processing, return retry indicator return {-1, nil} end end """
class IdempotencyManager: def __init__(self, redis_client): self.redis = redis_client self.lock_script = self.redis.register_script(LOCK_SCRIPT)
def process_idempotent(self, key: str, user_id: str, process_func, ttl=3600): """ Process request with idempotency guarantee.
Args: key: Idempotency key from client user_id: User/tenant identifier process_func: Function to call if this is a new request ttl: Time-to-live for idempotency record (seconds) """ lock_key = f"idempotency:lock:user:{user_id}:{key}" response_key = f"idempotency:response:user:{user_id}:{key}" lock_value = str(uuid.uuid4())
while True: # Retry loop for race conditions result = self.lock_script( keys=[lock_key, response_key], args=[ttl, lock_value, 'processing'] )
status, data = result[0], result[1]
if status == 1: # Lock acquired, this is the winning request # Execute the actual business logic response = process_func() return response
elif status == 0: # Another request is processing, return cached response return json.loads(data)
elif status == -1: # Race condition, retry time.sleep(0.1 * random.random()) # Jitter continue
# Usage idempotency = IdempotencyManager(redis_client)
@app.route('/api/payments', methods=['POST']) def create_payment(): idempotency_key = request.headers.get('Idempotency-Key') user_id = get_current_user_id()
if not idempotency_key: return jsonify({'error': 'Idempotency-Key required'}), 400
def process_payment_logic(): # This only runs once per unique key payment_data = request.get_json() payment = create_payment_in_db(payment_data) return { 'id': payment.id, 'status': payment.status, 'amount': payment.amount }
response = idempotency.process_idempotent( key=idempotency_key, user_id=user_id, process_func=process_payment_logic, ttl=86400 # 24 hours )
return jsonify(response) ```
### 4. Fix TTL and storage issues
Appropriate TTL configuration:
```python # Idempotency TTL trade-offs: # - Too short: Legitimate retries treated as new requests (duplicates) # - Too long: Storage bloat, keys never expire
# Recommended TTL by use case: TTL_CONFIG = { 'payment': timedelta(hours=24), # Payment retries common 'order': timedelta(hours=12), # Order creation 'subscription': timedelta(hours=24), # Subscription signup 'webhook': timedelta(minutes=30), # Webhook delivery 'sync': timedelta(hours=1), # Data sync }
def get_ttl_for_endpoint(endpoint: str) -> timedelta: """Get appropriate TTL for endpoint type.""" if 'payment' in endpoint: return TTL_CONFIG['payment'] elif 'order' in endpoint: return TTL_CONFIG['order'] elif 'webhook' in endpoint: return TTL_CONFIG['webhook'] else: return timedelta(hours=12) # Default
# Storage backend options
# Option 1: Redis (recommended for performance) class RedisIdempotencyStore: def __init__(self, redis_client, default_ttl=timedelta(hours=24)): self.redis = redis_client self.default_ttl = default_ttl
def get(self, key: str) -> Optional[dict]: data = self.redis.get(f"idempotency:{key}") return json.loads(data) if data else None
def set(self, key: str, response: dict, ttl: timedelta = None): ttl = ttl or self.default_ttl data = { 'response': response, 'timestamp': datetime.utcnow().isoformat() } self.redis.setex( f"idempotency:{key}", int(ttl.total_seconds()), json.dumps(data) )
def delete(self, key: str): self.redis.delete(f"idempotency:{key}")
# Option 2: Database (for durability requirements) class DatabaseIdempotencyStore: def __init__(self, db_session): self.db = db_session
def get(self, key: str, user_id: str) -> Optional[dict]: record = self.db.query(IdempotencyRecord).filter_by( key=key, user_id=user_id, expires_at > datetime.utcnow() ).first()
if record: return record.response_data return None
def set(self, key: str, user_id: str, response: dict, ttl: timedelta): record = IdempotencyRecord( key=key, user_id=user_id, response_data=response, created_at=datetime.utcnow(), expires_at=datetime.utcnow() + ttl )
# Use INSERT ... ON CONFLICT for atomic upsert self.db.merge(record) self.db.commit()
# Option 3: Database table schema # CREATE TABLE idempotency_records ( # id SERIAL PRIMARY KEY, # key VARCHAR(255) NOT NULL, # user_id VARCHAR(64) NOT NULL, # endpoint VARCHAR(255) NOT NULL, # response_data JSONB NOT NULL, # status_code INTEGER NOT NULL, # created_at TIMESTAMP NOT NULL DEFAULT NOW(), # expires_at TIMESTAMP NOT NULL, # UNIQUE(key, user_id, endpoint) # ); # # CREATE INDEX idx_idempotency_expires ON idempotency_records(expires_at); # CREATE INDEX idx_idempotency_user ON idempotency_records(user_id); ```
### 5. Handle error responses correctly
Error response idempotency:
```python # Should error responses be cached for idempotency?
# Option 1: Cache all responses (including errors) # Pro: True idempotency - same error returned on retry # Con: Validation errors cached (user can't fix and retry)
def store_all_responses(key, response, status_code): """Store all responses including errors.""" store_idempotency_result(key, response, status_code) # All responses cached
# Option 2: Only cache successful responses # Pro: Allows retry on validation errors # Con: Not truly idempotent for error cases
def store_success_only(key, response, status_code): """Only store successful responses.""" if 200 <= status_code < 300: store_idempotency_result(key, response, status_code) # Error responses not cached
# Option 3: Cache server errors, not client errors (recommended) def store_server_errors_only(key, response, status_code): """Store server errors (5xx) but not client errors (4xx).""" if 200 <= status_code < 300: # Success - cache store_idempotency_result(key, response, status_code) elif 500 <= status_code < 600: # Server error - cache (allow safe retry) store_idempotency_result(key, response, status_code) # Client errors (4xx) not cached - user should fix and retry
# Implementation example @app.route('/api/payments', methods=['POST']) def create_payment(): idempotency_key = request.headers.get('Idempotency-Key')
if not idempotency_key: return jsonify({'error': 'Idempotency-Key required'}), 400
# Check for cached response is_duplicate, cached_response = check_idempotency(idempotency_key)
if is_duplicate: return jsonify(cached_response['response']), cached_response['status_code']
try: # Process payment payment = process_payment(request.get_json()) response = {'id': payment.id, 'status': payment.status} store_idempotency_result(idempotency_key, response, 200) return jsonify(response), 200
except ValidationError as e: # Client error - don't cache, allow retry with fixed data return jsonify({'error': str(e)}), 400
except PaymentError as e: # Server error - cache, safe to retry error_response = {'error': str(e), 'code': 'PAYMENT_ERROR'} store_idempotency_result(idempotency_key, error_response, 500) return jsonify(error_response), 500 ```
### 6. Test idempotency implementation
Idempotency test cases:
```python import pytest from unittest.mock import patch
class TestPaymentIdempotency:
@pytest.fixture def client(self): app.config['TESTING'] = True with app.test_client() as client: yield client
def test_idempotent_request_returns_same_response(self, client): """Same idempotency key returns cached response.""" idempotency_key = 'test-key-123' payment_data = {'amount': 100, 'currency': 'USD', 'source': 'tok_visa'}
# First request response1 = client.post('/api/payments', json=payment_data, headers={'Idempotency-Key': idempotency_key})
# Second request with same key response2 = client.post('/api/payments', json=payment_data, headers={'Idempotency-Key': idempotency_key})
# Both should return same response assert response1.status_code == 200 assert response2.status_code == 200 assert response1.json == response2.json
# Only one payment should be created payments = Payment.query.all() assert len(payments) == 1
def test_different_keys_create_different_payments(self, client): """Different idempotency keys create separate payments.""" payment_data = {'amount': 100, 'currency': 'USD', 'source': 'tok_visa'}
response1 = client.post('/api/payments', json=payment_data, headers={'Idempotency-Key': 'key-1'})
response2 = client.post('/api/payments', json=payment_data, headers={'Idempotency-Key': 'key-2'})
assert response1.status_code == 200 assert response2.status_code == 200 assert response1.json['id'] != response2.json['id']
def test_missing_idempotency_key_allowed(self, client): """Requests without idempotency key are processed (but not idempotent).""" payment_data = {'amount': 100, 'currency': 'USD', 'source': 'tok_visa'}
# Without idempotency key response = client.post('/api/payments', json=payment_data)
assert response.status_code == 200
# Retry without key creates new payment (not idempotent) response2 = client.post('/api/payments', json=payment_data) assert response2.json['id'] != response.json['id']
def test_idempotency_key_expires(self, client): """Idempotency key expires after TTL.""" idempotency_key = 'test-key-expiry' payment_data = {'amount': 100, 'currency': 'USD', 'source': 'tok_visa'}
# First request response1 = client.post('/api/payments', json=payment_data, headers={'Idempotency-Key': idempotency_key})
# Simulate TTL expiration (mock time or wait) with patch('datetime.datetime') as mock_datetime: mock_datetime.utcnow.return_value = datetime.utcnow() + timedelta(hours=25)
# Request after TTL should be treated as new response2 = client.post('/api/payments', json=payment_data, headers={'Idempotency-Key': idempotency_key})
# Different payment created assert response2.json['id'] != response1.json['id']
def test_concurrent_requests_same_key(self, client): """Concurrent requests with same key don't create duplicates.""" import threading
idempotency_key = 'test-key-concurrent' payment_data = {'amount': 100, 'currency': 'USD', 'source': 'tok_visa'} results = []
def make_request(): response = client.post('/api/payments', json=payment_data, headers={'Idempotency-Key': idempotency_key}) results.append(response.json)
# Send concurrent requests threads = [threading.Thread(target=make_request) for _ in range(5)] for t in threads: t.start() for t in threads: t.join()
# All should return same payment assert len(set(r['id'] for r in results)) == 1 assert len(Payment.query.all()) == 1 ```
Prevention
- Require Idempotency-Key header for all POST/PUT/PATCH requests
- Use UUID v4 or cryptographically random keys (not predictable)
- Scope keys by user/tenant to prevent collisions
- Set appropriate TTL based on use case (24h for payments typical)
- Use atomic operations (Lua scripts, database constraints) for check-and-store
- Cache server errors (5xx) for safe retry, not client errors (4xx)
- Monitor idempotency cache hit rate and storage size
- Document idempotency behavior in API documentation
- Test concurrent requests with same key in load tests
- Implement idempotency at API gateway layer for consistency
Related Errors
- **409 Conflict**: Resource conflict (may be idempotency-related)
- **422 Unprocessable Entity**: Validation or duplicate request
- **408 Request Timeout**: Request took too long (retry may cause duplicate)
- **503 Service Unavailable**: Service unavailable (safe to retry with idempotency)
- **Rate limit exceeded**: Too many requests (different from idempotency)