Case study
Research Agent
Plan a coverage map, gather evidence as atomic claims, audit for gaps, write the report once, then verify every citation. No orchestration framework; every design decision backed by a benchmark, not a vibe check.
43.13
GPT-5.5 RACE score on DeepResearch Bench — inside the published top-10 band
0.08 points above the published rank-9 entry (Perplexity Research, 43.05) and 1.91 above the rank-10 cutoff (Grok Deeper Search, 41.22) — on the benchmark's 50 English tasks, judged by the same GPT-5.5 RACE judge as the live leaderboard, running on claude-haiku-4-5, a small/cheap model.
04 —
Stack
AI & LLM
litellm (provider-agnostic: Anthropic, OpenAI, OpenRouter, Ollama)Claude Haiku 4.5 (benchmark model)Native function/tool calling for structured extractionPrompt engineeringAgentic AI — iterative research loop
Retrieval & grounding
Tavily Search APIDuckDuckGo Searchhttpx + lxml page fetchingCitation verification & source-quality grounding
Evaluation
DeepResearch BenchLLM-as-judge (GPT-5.5 RACE)127 tests, network-mocked, pytest
Backend
PythonFastAPI (SSE streaming)UvicornTyper + Rich CLI
Frontend
ReactViteTailwind CSSFramer Motionreact-markdown
Dev & deploy
uvDockerGitHub Actions CI
05 —
How the score was earned
- A coverage plan with breadth classification, so broad questions can't stop after a single supported claim.
- Atomic evidence storage instead of a repeatedly-rewritten running summary — detail survives many search rounds instead of degrading through re-compression.
- One write-once synthesis pass after research finishes, not a summary-of-summaries.
- A deterministic citation-compliance postcondition: unsupported or unknown-URL citations get one bounded repair pass, then a fallback — never another unchecked model guess.
06 —
Architecture
- No framework — a plain, readable Python loop, with iterative and deep research modes behind the same model and tool interfaces.
- Provider-agnostic via one model string (litellm) — runs on hosted models or fully local through Ollama, no API key required.
- A minimal Tool abstraction over search and page fetching.
- Defensive and hardened: bounded retries, fail-loud, skip paths.
- Every architecture change measured against DeepResearch Bench, an external benchmark, not graded by vibes.
07 —
Engineering proof
- 127 tests passing, all network-mocked (~2s to run).
- 50/50 English benchmark tasks completed, zero failed agent reports.
- 2× the valid citations per report versus the pre-planner baseline (32.76 vs. 16.38), with citation validity held within 0.3 points (90.80% vs. 91.1%).
- Dockerized single-process deploy; bring-your-own-key web UI — keys live only in localStorage, never server-side.
08 —
Screenshots
