Troubleshooting

Solutions for common Headroom issues.

Proxy Server Issues

Proxy will not start

Symptom: headroom proxy fails or hangs.

# Check if port is already in use
lsof -i :8787

# Try a different port
headroom proxy --port 8788

# Check for missing dependencies
pip install "headroom-ai[proxy]"

# Run with debug logging
headroom proxy --log-file ~/.headroom/logs/proxy.jsonl --log-messages

Connection refused when calling proxy

Symptom: curl: (7) Failed to connect to localhost port 8787

# Verify proxy is running
curl http://localhost:8787/health

# Check if proxy started on a different port
ps aux | grep headroom

Proxy returns errors for some requests

Symptom: Some requests work, others fail with 502/503.

# Check proxy logs for the actual error
headroom proxy --log-file ~/.headroom/logs/proxy.jsonl --log-messages

# Verify API key is set
echo $OPENAI_API_KEY   # or ANTHROPIC_API_KEY

# Test the underlying API directly
curl https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Windows: ML content detection hangs or silently falls back

Symptom: On Windows 11 24H2+, every proxied request stalls (historically Optimization failed: TimeoutError), or the first detection in each process burns ~5 seconds and compression quality drops because detection runs on the non-ML fallback tiers. The proxy log may show magika ONNX session init timed out.

Cause: The Rust core loads ONNX Runtime dynamically. Without ORT_DYLIB_PATH, the bare Windows DLL search resolves onnxruntime.dll to C:\Windows\System32\onnxruntime.dll — the Windows ML OS component (1.17.x), which deadlocks ort session initialization instead of returning an error.

Fix: Headroom pins ORT_DYLIB_PATH automatically at import time to the DLL inside the onnxruntime pip package (included in headroom-ai[proxy]). Confirm in the startup log:

Pinned ORT_DYLIB_PATH to bundled ONNX Runtime: ...\onnxruntime\capi\onnxruntime.dll

If the pin is skipped (library install without onnxruntime), either install it or point the variable at any modern ONNX Runtime yourself:

pip install onnxruntime
# or
$env:ORT_DYLIB_PATH = "C:\path\to\onnxruntime.dll"

HEADROOM_MAGIKA_INIT_TIMEOUT_SECS (default 5) bounds the init as a safety net; on timeout detection degrades to non-ML tiers for the process lifetime.

No Token Savings

Symptom: stats['session']['tokens_saved_total'] is 0.

Diagnosis:

stats = client.get_stats()
print(f"Mode: {stats['config']['mode']}")          # Should be "optimize"
print(f"SmartCrusher: {stats['transforms']['smart_crusher_enabled']}")

Common causes:

• Mode is audit (observation only, no modifications)
• Messages do not contain tool outputs
• Tool outputs are below the 200-token threshold
• Data is not compressible (high uniqueness, code, grep results)

Solutions:

import { compress } from 'headroom-ai';

// Ensure the proxy is running in optimize mode
// (default, unless --no-optimize was passed)
const result = await compress(messages, { model: 'gpt-4o' });
console.log(`Saved: ${result.tokensSaved} tokens`);
console.log(`Compressed: ${result.compressed}`);

# 1. Ensure mode is "optimize"
client = HeadroomClient(
    original_client=OpenAI(),
    provider=OpenAIProvider(),
    default_mode="optimize",  # NOT "audit"
)

# 2. Or override per-request
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    headroom_mode="optimize",
)

# 3. Lower the compression threshold
config = HeadroomConfig()
config.smart_crusher.min_tokens_to_crush = 100  # Default is 200

Dashboard shows 0 compressed/saved tokens after upgrading to 0.31.0

Symptom: After upgrading from 0.27.0 to 0.31.0, the dashboard's compression / "Tokens Saved" figures read ~0, even though total token spend is the same or lower than before.

Cause: This is a default-mode change, not a regression. 0.31.0 ships the coding savings profile as the out-of-box default, and coding runs the proxy in cache mode (see Savings profiles). Cache mode freezes the provider prefix and compresses only the newest turn delta — this deliberately avoids busting the provider's prompt cache. So the compression figure is small, and the savings shift to cheaper prefix-cache reads (cached input tokens are billed at a fraction of list price). On a short prompt there is little delta to compress, so the compression tile can read ~0 while your actual cost still drops.

Where the savings show up: Look at the Prefix Cache Impact panel and the Compression vs Cache tile on the dashboard — these reflect cache-read savings rather than per-request compression. The headline "Tokens Saved" tile only counts compression, so in cache mode it understates the real benefit.

To get 0.27.0-style compression numbers back: run the proxy in token mode, which prioritizes visible compression:

# Per run
headroom proxy --mode token

# Or pick a token-mode profile
HEADROOM_SAVINGS_PROFILE=balanced headroom proxy   # ~70% target
HEADROOM_SAVINGS_PROFILE=agent-90 headroom proxy   # ~90% target

Trade-off: token mode maximizes visible compression but rewrites prior turns, which can reduce provider prefix-cache hits — so raw compression goes up while cache-read savings go down. Cache mode is the default because, for long coding sessions, preserving the prefix cache usually wins overall.

Claude Code context window is larger through the proxy

Symptom: After pointing Claude Code at Headroom (ANTHROPIC_BASE_URL), /context all shows more tokens used than a direct session — the "System tools" and "MCP tools" lines grow by tens of thousands of tokens, before you send any message.

Cause: Claude Code normally defers most tool schemas behind its server-side Tool Search Tool (it sends only tool names and loads full schemas on demand). It enables this only when it believes it is talking directly to api.anthropic.com. The moment ANTHROPIC_BASE_URL is a custom host, Claude Code can't assume the endpoint supports the feature, so it falls back to eagerly materializing every tool schema into the local context window. This is a Claude Code client-side decision made before the request reaches the proxy — no proxy header can reverse it.

Solution: set ENABLE_TOOL_SEARCH so Claude Code keeps deferring tools through the proxy. The proxy forwards the tool_reference blocks correctly, so deferral works end-to-end (both streaming and non-streaming, subscription and API-key auth).

# Easiest: `headroom wrap claude` sets ENABLE_TOOL_SEARCH=true automatically.
headroom wrap claude

# Choose the mode (true = always defer, the default; auto / auto:N = defer only
# when tool definitions exceed N% of the budget; false = off):
headroom wrap claude --tool-search auto

# Running `claude` manually instead of via wrap? Set it yourself:
ENABLE_TOOL_SEARCH=true ANTHROPIC_BASE_URL=http://localhost:8787 claude

Verify (before / after) with /context all in a fresh session, no messages sent:

When deferral is off, the proxy log also prints a one-time hint naming the fix.

See issue #746 for the full analysis.

Remote Control unavailable through custom ANTHROPIC_BASE_URL

Symptom: When Claude Code runs with ANTHROPIC_BASE_URL set to a custom host (for example, Headroom), the Remote Control menu is absent.

Cause: This is a Claude-side gate. Headroom only receives normal API traffic and can still compress it, but Claude evaluates Remote Control availability before proxy traffic reaches the server.

Fix: Use Headroom for normal proxied API sessions, and launch Claude directly (without ANTHROPIC_BASE_URL) when you need Claude Remote Control.

ENABLE_TOOL_SEARCH is unaffected and can stay enabled for context-window savings while routing through Headroom.

Server-managed settings unavailable through custom ANTHROPIC_BASE_URL

Symptom: Settings pushed from Admin Settings > Claude Code > Managed settings in the claude.ai console (server-managed settings) don't apply to sessions running through Headroom, even though they apply fine without the proxy.

Cause: This is a Claude-side gate, not a Headroom limitation. Per Anthropic's docs, server-managed settings require a direct connection to api.anthropic.com; if ANTHROPIC_BASE_URL is set to any non-default host — which is exactly what wrapping via Headroom does — Claude Code skips the settings fetch entirely for that session. The request never reaches Headroom, so there is no endpoint for Headroom to implement or proxy.

This is separate from the OS-level managed-settings.json file (macOS /Library/Application Support/ClaudeCode/, Linux /etc/claude-code/, Windows C:\Program Files\ClaudeCode\): that file is read straight from local disk at startup and is unaffected by ANTHROPIC_BASE_URL or Headroom. If that file isn't taking effect, the cause is unrelated to proxying (path, permissions, or JSON syntax) — check claude --debug-file <path> and search the log for Remote settings.

Fix: None available on the Headroom side — this is an intentional Anthropic security boundary (a proxy in the path could otherwise forge org policy). If your org relies on server-managed settings, deploy the same policy as endpoint-managed settings (MDM profile, Windows registry, or a local managed-settings.json) instead, since those are read locally and unaffected by proxying.

See Server-managed settings platform availability and issue #3074.

Compression Too Aggressive

Symptom: LLM responses are missing information that was in tool outputs.

# 1. Keep more items
config = HeadroomConfig()
config.smart_crusher.max_items_after_crush = 50  # Default: 15

# 2. Skip compression for specific tools
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    headroom_tool_profiles={
        "important_tool": {"skip_compression": True},
    },
)

# 3. Disable SmartCrusher entirely
config.smart_crusher.enabled = False

High Latency

Symptom: Requests take longer than expected.

Diagnosis:

import time
import logging

logging.basicConfig(level=logging.DEBUG)

start = time.time()
response = client.chat.completions.create(...)
print(f"Total time: {time.time() - start:.2f}s")

Solutions:

# 1. Use BM25 instead of embeddings (faster)
config = HeadroomConfig()
config.smart_crusher.relevance.tier = "bm25"

# 2. Increase threshold to skip small payloads
config.smart_crusher.min_tokens_to_crush = 500

# 3. Disable transforms you don't need
config.cache_aligner.enabled = False
config.rolling_window.enabled = False

Installation Issues

pipx installs an older Headroom version

Symptom: PyPI shows a newer headroom-ai release, but pipx install or pipx upgrade keeps an older version. A pinned install can also fail with No matching distribution found.

Cause: pipx resolves packages inside its app virtual environment. If that environment uses a Python version that Headroom does not publish wheels for yet, pip may skip newer releases and choose the newest compatible build it can use.

Check the interpreter:

pipx list

Install with a supported Python explicitly:

pipx install --python python3.13 "headroom-ai[all]"

For a pinned release:

pipx install --python python3.13 "headroom-ai[all]==0.21.4"

If you already have Headroom installed under pipx, uninstall it first or reinstall it with the supported interpreter.

pip install fails with C++ compilation error

Symptom: RuntimeError: Unsupported compiler -- at least C++11 support is needed!

# Linux / Debian-based (including Docker)
apt-get install -y build-essential && pip install headroom-ai

# macOS (Xcode command line tools)
xcode-select --install && pip install headroom-ai

For Docker, install and remove build tools in one layer:

FROM python:3.11-slim
RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
    && pip install "headroom-ai[proxy]" \
    && apt-get purge -y build-essential && apt-get autoremove -y \
    && rm -rf /var/lib/apt/lists/*

ModuleNotFoundError: No module named 'headroom'

# Check it is installed in the right environment
pip show headroom-ai

# If using virtual environment, ensure it is activated
source venv/bin/activate

# Reinstall
pip install --upgrade headroom-ai

Missing optional dependency

# For proxy server
pip install "headroom-ai[proxy]"

# For embedding-based relevance scoring
pip install "headroom-ai[relevance]"

# For code compression (tree-sitter)
pip install "headroom-ai[code]"

# For everything
pip install "headroom-ai[all]"

Windows: Defender blocks ast-grep-cli (sg.exe) during install

Symptom: On Windows, uv tool install "headroom-ai[all]" (or pip install) fails while installing the ast-grep-cli wheel, and Windows Defender flags sg.exe:

error: Failed to install: ast_grep_cli-0.44.1-py3-none-win_amd64.whl (ast-grep-cli==0.44.1)
Caused by: failed to open file ...\ast_grep_cli-0.44.1.data\scripts\sg.exe:
The operation did not complete successfully because the file contains a virus
or potentially unwanted software. (os error 225)

Threat: Trojan:Win64/Lazy!MTB

Cause: A known false positive in the upstream ast-grep-cli wheel's bundled sg.exe (ast-grep/ast-grep#2799), not a Headroom issue. ast-grep is a base dependency, so the block also affects the [proxy] extra. Headroom uses ast-grep only for optional AST-based Read-output outlining and runs normally without it — the only impact is the install-time quarantine.

Workarounds (safest first):

1. Run the proxy in Docker — no local wheel is installed, so Defender is never triggered. See Docker install; the image is ghcr.io/headroomlabs-ai/headroom.

2. Restore the file from quarantine and retry — open Windows Security → Virus & threat protection → Protection history, select the sg.exe detection, choose Restore, then re-run the install command. This changes no persistent settings.

3. Add a temporary, scoped Defender exclusion during install (last resort; requires an elevated PowerShell). Only do this if you accept excluding a known false positive, and remove the exclusion afterward:

   # Scope the exclusion to uv's tools directory, install, then remove it
   $uvTools = (uv tool dir)
   Add-MpPreference -ExclusionPath $uvTools
   uv tool install "headroom-ai[all]"
   Remove-MpPreference -ExclusionPath $uvTools

   Do not disable Defender wholesale — keep the exclusion narrow and temporary.

4. Report the false positive to Microsoft so a corrected signature ships for everyone: submit sg.exe at the Microsoft Security Intelligence sample submission page.

uv build errors: "src does not appear to be a Python project"

Symptom: uv tool install "headroom-ai[all]" fails to build a dependency (commonly litellm or cryptography) with:

error: Failed to build: <package>==<version>
Caused by: `src does not appear to be a Python project, as neither `pyproject.toml`
nor `setup.py` are present`

or a wheel install fails with Unknown wheel data type: .DS_Store.

Cause: Corrupted or stale entries in uv's local build/wheel cache, not a Headroom dependency-pin problem. Headroom does not pin exact versions of litellm or cryptography that would trigger this.

Fix: clear uv's cache and reinstall:

uv cache clean
uv tool install "headroom-ai[all]"

To clear the cache for just one package instead:

uv cache clean litellm

If the failure is specifically for ast-grep-cli==0.44.1, that release is already excluded by Headroom's dependency pin (ast-grep-cli>=0.30.0,!=0.44.1) due to a compromised supply-chain build (ast-grep/ast-grep#2799) — uv cache clean and a plain reinstall should pick up a safe version automatically.

Provider-Specific Issues

OpenAI: Invalid API key

import os
from openai import OpenAI

api_key = os.environ.get("OPENAI_API_KEY")
if not api_key:
    raise ValueError("OPENAI_API_KEY not set")

client = HeadroomClient(
    original_client=OpenAI(api_key=api_key),
    provider=OpenAIProvider(),
)

Anthropic: Authentication error

import os
from anthropic import Anthropic

api_key = os.environ.get("ANTHROPIC_API_KEY")
client = HeadroomClient(
    original_client=Anthropic(api_key=api_key),
    provider=AnthropicProvider(),
)

Unknown model warnings

# For custom/fine-tuned models, specify context limit
client = HeadroomClient(
    original_client=OpenAI(),
    provider=OpenAIProvider(),
    model_context_limits={
        "ft:gpt-4o-2024-08-06:my-org::abc123": 128000,
        "my-custom-model": 32000,
    },
)

ValidationError on Setup

result = client.validate_setup()
print(result)

# Common issues:
# {"provider": {"ok": False, "error": "No API key"}}
#   -> Set OPENAI_API_KEY or pass api_key to OpenAI()
#
# {"storage": {"ok": False, "error": "unable to open database"}}
#   -> Check path permissions, use :memory: for testing
#
# {"config": {"ok": False, "error": "Invalid mode"}}
#   -> Use "audit" or "optimize" only

For testing, use in-memory storage:

client = HeadroomClient(
    original_client=OpenAI(),
    provider=OpenAIProvider(),
    store_url="sqlite:///:memory:",
)

Debugging Techniques

Enable Full Logging

import logging

# See everything
logging.basicConfig(
    level=logging.DEBUG,
    format="%(asctime)s %(name)s %(levelname)s %(message)s",
)

# Or just Headroom logs
logging.getLogger("headroom").setLevel(logging.DEBUG)

Use Simulation to Inspect Transforms

plan = client.chat.completions.simulate(
    model="gpt-4o",
    messages=messages,
)

print(f"Tokens: {plan.tokens_before} -> {plan.tokens_after}")
print(f"Transforms: {plan.transforms_applied}")
print(f"Waste signals: {plan.waste_signals}")

import json
print(json.dumps(plan.messages_optimized, indent=2))

Test Transforms Directly

from headroom import SmartCrusher, Tokenizer
from headroom.config import SmartCrusherConfig
import json

config = SmartCrusherConfig()
crusher = SmartCrusher(config)
tokenizer = Tokenizer()

messages = [
    {
        "role": "tool",
        "content": json.dumps({"items": list(range(100))}),
        "tool_call_id": "1",
    }
]

result = crusher.apply(messages, tokenizer)
print(f"Tokens: {result.tokens_before} -> {result.tokens_after}")

Getting Help

1. Enable debug logging and check the output
2. Use simulate() to see what transforms would apply
3. Run validate_setup() for configuration issues
4. File an issue at github.com/headroomlabs-ai/headroom with your Headroom version, Python version, provider, debug log output, and minimal reproduction code