Part 3: Idempotency

Weight: 20%

Make your service safe to retry by implementing idempotent operations.

---

Objectives

• Understand why some operations are unsafe to retry
• Implement request IDs for deduplication
• Design idempotent stateful operations

---

Background

Your calculator from Parts 1-2 is safe to retry because operations like Add(10, 5) always return 15, regardless of how many times you call it.

But what about operations that modify state?

# NOT idempotent - calling twice increments twice!
increment_counter()  # counter = 1
increment_counter()  # counter = 2 (different result!)

# Idempotent - calling twice sets to same value
set_counter(10)      # counter = 10
set_counter(10)      # counter = 10 (same result!)

In this part, you'll add stateful operations and make them safe to retry.

---

Requirements

1. Implement Stateful Operations

Already Defined in Proto

The counter operations are already defined in the provided calculator.proto file. You do NOT need to modify the proto file - just implement these operations in your server.

The provided proto file includes these stateful operations:

service Calculator {
  // ... existing operations ...

  // Stateful operations (already defined in provided proto)
  rpc IncrementCounter(CounterRequest) returns (CounterResult);
  rpc GetCounter(Empty) returns (CounterResult);
  rpc ResetCounter(Empty) returns (CounterResult);
}

message CounterRequest {
  string request_id = 1;  // Unique ID for deduplication
  int32 increment = 2;     // Amount to increment (default: 1)
}

message CounterResult {
  int32 value = 1;
  bool was_duplicate = 2;  // True if request was already processed
}

message Empty {}

2. Implement Request Deduplication

Track processed requests to avoid duplicate execution:

import uuid
from collections import OrderedDict
import time

class CalculatorServicer(calculator_pb2_grpc.CalculatorServicer):
    def __init__(self, cache_size=1000, cache_ttl=300):
        """
        Args:
            cache_size: Maximum number of cached requests
            cache_ttl: Time-to-live for cached requests (seconds)
        """
        self.counter = 0
        self.request_cache = OrderedDict()  # request_id -> (result, timestamp)
        self.cache_size = cache_size
        self.cache_ttl = cache_ttl

    def IncrementCounter(self, request, context):
        # Check if request was already processed
        if request.request_id in self.request_cache:
            cached_result, timestamp = self.request_cache[request.request_id]

            # Check if cache entry is still valid
            if time.time() - timestamp < self.cache_ttl:
                print(f"Duplicate request detected: {request.request_id}")
                return calculator_pb2.CounterResult(
                    value=cached_result,
                    was_duplicate=True
                )

        # Process new request
        self.counter += request.increment if request.increment else 1
        result = self.counter

        # Cache the result
        self._cache_request(request.request_id, result)

        return calculator_pb2.CounterResult(
            value=result,
            was_duplicate=False
        )

    def _cache_request(self, request_id, result):
        """Cache a processed request."""
        # Remove oldest entry if cache is full
        if len(self.request_cache) >= self.cache_size:
            self.request_cache.popitem(last=False)

        self.request_cache[request_id] = (result, time.time())

3. Client-Side Request IDs

Client must generate unique IDs for each logical request:

import uuid

def increment_with_retry(stub, increment=1, max_retries=3):
    """
    Increment counter with idempotent retries.

    Args:
        stub: gRPC stub
        increment: Amount to increment
        max_retries: Maximum retry attempts

    Returns:
        CounterResult with final counter value
    """
    # Generate request ID once (same for all retries)
    request_id = str(uuid.uuid4())

    for attempt in range(max_retries):
        try:
            response = stub.IncrementCounter(
                calculator_pb2.CounterRequest(
                    request_id=request_id,
                    increment=increment
                ),
                timeout=2.0
            )

            if response.was_duplicate:
                print(f"Request {request_id} was a duplicate (retry succeeded)")

            return response

        except grpc.RpcError as e:
            if e.code() == grpc.StatusCode.DEADLINE_EXCEEDED:
                if attempt < max_retries - 1:
                    wait_time = (2 ** attempt) + random.uniform(0, 1)
                    print(f"Timeout, retrying with same request_id: {request_id}")
                    time.sleep(wait_time)
                else:
                    raise
            else:
                raise

Key Points:

• ✅ Generate request_id once before first attempt
• ✅ Use same request_id for all retry attempts
• ✅ Use UUIDs to ensure uniqueness

---

Testing Requirements

Test Scenarios

Demonstrate idempotency working correctly:

Test	Scenario	Expected Behavior
Normal Operation	Single request, no retries	Counter increments once
Duplicate Detection	Send same request_id twice manually	Second request detected as duplicate
Retry with Idempotency	Request times out, retry succeeds	Counter increments only once
Multiple Clients	Two clients with different request_ids	Counter increments twice

Required Test Script

def test_idempotency():
    """Test that retries don't cause duplicate increments."""
    stub = get_stub()

    # Reset counter
    stub.ResetCounter(calculator_pb2.Empty())

    # Test 1: Normal increment
    result = stub.IncrementCounter(
        calculator_pb2.CounterRequest(
            request_id=str(uuid.uuid4()),
            increment=1
        )
    )
    assert result.value == 1
    assert not result.was_duplicate

    # Test 2: Duplicate request (same request_id)
    request_id = str(uuid.uuid4())
    stub.IncrementCounter(
        calculator_pb2.CounterRequest(request_id=request_id, increment=1)
    )
    result = stub.IncrementCounter(
        calculator_pb2.CounterRequest(request_id=request_id, increment=1)
    )
    assert result.value == 2  # Counter is at 2
    assert result.was_duplicate  # But second request was duplicate

    # Test 3: Simulate retry scenario
    # (Use server with delay to trigger timeout and retry)

---

Deliverables

📦 Updated Code:

• calculator.proto - Add stateful operations
• server.py - Implement request deduplication
• client.py - Generate and use request IDs
• test_idempotency.py - Test suite

📊 Test Report:

Document test results showing:

• Counter value after normal operations
• Duplicate detection working
• Retry scenario not causing double-increment

📹 Demo Video (2 minutes):

Show:

1. Normal increment operations
2. Manually sending duplicate request - detected!
3. Simulated retry scenario - counter only increments once
4. Multiple clients incrementing independently

---

Grading Rubric

Criterion	Points	Description
Request ID Generation	4	Client generates UUIDs correctly
Deduplication Logic	8	Server detects and handles duplicates
Cache Management	3	Cache has size limit and TTL
Testing	3	Comprehensive test scenarios
Demo	2	Clear demonstration of idempotency
Total	20

---

Analysis Questions

Answer these in your report:

Question 1

Why is it important to generate the request_id before the first attempt, not on each retry?

Question 2

What happens if the server's request cache fills up and evicts an entry while a client is still retrying?

Question 3

Which of these operations are naturally idempotent? Why?

• set_value(key, value)
• increment_value(key)
• delete_key(key)
• append_to_list(key, value)

Question 4

How would you make a withdraw(account_id, amount) operation idempotent?

---

Tips

UUID Best Practice

Use uuid.uuid4() for request IDs - they're globally unique and don't require coordination.

Cache Considerations

• Memory: Large caches consume memory. Use LRU eviction.
• TTL: Old entries should expire to prevent unbounded growth.
• Persistence: In production, cache might need to persist across server restarts.

Real-World Example

Payment APIs like Stripe use idempotency keys:

stripe.Charge.create(
    amount=1000,
    currency="usd",
    idempotency_key="order_12345"  # Safe to retry!
)

---

Common Pitfalls

❌ Mistake: Generating new request_id on each retry

# WRONG - each retry gets different ID!
for attempt in range(3):
    request_id = str(uuid.uuid4())  # New ID each time
    stub.IncrementCounter(...)

✅ Correct: Generate once, reuse on retries

# CORRECT - same ID for all retries
request_id = str(uuid.uuid4())
for attempt in range(3):
    stub.IncrementCounter(
        CounterRequest(request_id=request_id)
    )

---

Next Steps

Your service can now safely retry stateful operations! Next, protect against cascading failures in Part 4: Circuit Breaker.

---

Resources

• Idempotency in APIs
• UUID Documentation