Token-Saving Architecture

This is what actually reduces API consumption.

The Three Mechanisms

1. Semantic Cache (Biggest Win)

Before: Every question hits the API After: Similar questions return cached responses

# First ask (miss - hits API)
curl -X POST http://helm:8675/cache/semantic-lookup \
  -H "Content-Type: application/json" \
  -d '{"prompt": "How do I setup Traefik?", "model": "claude-3-opus"}'

# Response: {"hit": false}
# -> Call LLM, get response
# -> Store response:
curl -X POST http://helm:8675/cache/semantic-store \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "How do I setup Traefik?",
    "response": "...",
    "model": "claude-3-opus",
    "tokens_in": 500,
    "tokens_out": 800
  }'

# Second ask, slightly different (HIT - no API call)
curl -X POST http://helm:8675/cache/semantic-lookup \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Traefik setup help", "model": "claude-3-opus"}'

# Response: {"hit": true, "similarity": 0.92, "response": "...", "tokens_saved": 1300}

Savings: 80-90% on repeated questions

2. RAG Context Selection (Moderate Win)

Before: Inject ALL skills/conventions (2000+ tokens) After: Inject only top 3 relevant (400-600 tokens)

# Legacy endpoint - returns EVERYTHING
curl "http://localhost:8080/context?project=/opt/home-server"
# Returns: 50 skills, 10 conventions = ~3000 tokens

# RAG endpoint - returns only relevant
curl "http://helm:8675/context/rag?query=How+do+I+setup+Docker+Compose&project=/opt/home-server"
# Returns: 3 skills about Docker, 2 conventions = ~600 tokens

Savings: 60-80% on context injection

3. Conversation Compression (Moderate Win)

Before: Full conversation history sent every request After: Old turns summarized, only recent kept full

# Compress a long conversation
curl -X POST http://helm:8675/compress \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [...],  # Your conversation history
    "keep_last_n": 3,
    "max_tokens": 2000
  }'

# Response:
{
  "messages": [...],  # Compressed version
  "original_tokens": 8000,
  "compressed_tokens": 2000,
  "tokens_saved": 6000,
  "reduction_percent": 75.0
}

Savings: 50-75% on conversation history

Integration Flow

# Your agent wrapper
async def query_llm(prompt, conversation_history, project=None):
    # 1. Check semantic cache FIRST
    cache_result = await httpx.post(
        "http://helm:8675/cache/semantic-lookup",
        json={"prompt": prompt, "model": "claude-3-opus"}
    )
    
    if cache_result.json()["hit"]:
        # No API call needed!
        return cache_result.json()["response"]
    
    # 2. Get ONLY relevant context (not everything)
    context = await httpx.get(
         "http://helm:8675/context/rag",
        params={"query": prompt, "project": project}
    )
    
    # 3. Compress conversation history
    compressed = await httpx.post(
         "http://helm:8675/compress",
        json={"messages": conversation_history, "keep_last_n": 3}
    )
    
    # 4. Build final prompt with compressed history + relevant context
    final_prompt = f"""
    {context.json()['skills']}
    {context.json()['conventions']}
    
    {compressed.json()['messages']}
    
    User: {prompt}
    """
    
    # 5. Call LLM
    response = await call_llm_api(final_prompt)
    
    # 6. Store in semantic cache
    await httpx.post(
         "http://helm:8675/cache/semantic-store",
        json={
            "prompt": prompt,
            "response": response,
            "tokens_in": len(final_prompt.split()),
            "tokens_out": len(response.split())
        }
    )
    
    return response

Expected Savings

Scenario	Before	After	Savings
Repeated question	1500 tokens	0 tokens (cache hit)	100%
Similar question	1500 tokens	0 tokens (semantic match)	100%
New question, known project	3500 tokens	1200 tokens	65%
Long conversation (10+ turns)	12000 tokens	4000 tokens	67%

Real-world average: 50-70% reduction in token consumption

Why No Vector DB?

For your scale (single user, <1000 items):

Approach	Query Time	Setup	Overhead
In-memory cosine sim	~5ms	None	None
SQLite + embeddings	~10ms	None	None
Qdrant/Chroma	~2ms	Docker container	500MB+ RAM

Verdict: Vector DB adds complexity without meaningful benefit at your scale.

New Endpoints

Endpoint	Purpose
`POST /cache/semantic-lookup`	Find similar cached responses
`POST /cache/semantic-store`	Store with embedding for matching
`GET /context/rag?query=...`	RAG-based context selection
`POST /compress`	Summarize conversation history
`GET /tokens/count?text=...`	Count tokens in text
`GET /cache/stats`	Cache statistics
`POST /cache/clear-old`	Cleanup old cache entries

System Prompt for Agents

## Token Efficiency Protocol

You have access to local infrastructure that reduces API usage:

**Before responding to any request:**
1. Call `POST /cache/semantic-lookup` with the user's prompt
2. If hit (similarity >= 0.85), return cached response directly
3. If miss, call `GET /context/rag?query={prompt}` for relevant context only

**For long conversations:**
1. Call `POST /compress` every 5+ turns
2. Use compressed history for subsequent requests

**After providing valuable responses:**
1. Call `POST /cache/semantic-store` to cache for future
2. Call `skills/create_skill` if it's a reusable pattern

**Token budget awareness:**
- Keep responses concise
- Don't repeat injected context
- Reference skills by ID when possible

This infrastructure saves 50-70% on token consumption.

5.8 KiB Raw Blame History