About

About the project

The benchmark becomes a stable lab where techniques can change and be compared without rewriting the runner or binding the system to one backend.

Main idea

We want more evidence about how scaling inference-time compute changes the performance of LLMs. The target is a relationship comparable to OpenAI's o1 and o3 results, where they see a log-linear relationship between inference compute and performance.

So the project pushes for the strongest possible agentic performance using test-time techniques — Budget Tracker, Structured Notes, Best-of-N, and others — so that the resulting scaling laws are measured against strong agents rather than naive baselines.

Measuring this cleanly means making LLM benchmark experiments faster, more modular, and reproducible. Instead of modifying the benchmark runner for every technique, the project introduces an intermediate layer:

Benchmark runner -> Proxy with techniques -> Base model backend

This layer makes it possible to experiment with model behavior during benchmark execution:

  • change the prompt;
  • change the request;
  • analyze the response;
  • run additional model calls;
  • aggregate answers;
  • enable Best-of-N;
  • save intermediate state;
  • compact context;
  • track tool calls;
  • collect usage;
  • combine several techniques;
  • preserve run configuration.

Short version

The project builds infrastructure for experimenting with language-model behavior inside a benchmark loop. The main practical target is GAIA through InspectAI, but the architecture is not tightly coupled to GAIA.

The project has two major parts:

1. Test-Time Scaling Proxy
2. GAIA Experiment Launcher

The proxy adds experimental techniques on top of an OpenAI-compatible model. The launcher makes the full pipeline reproducible across different inference backends and compute environments.

Why not change the benchmark

The benchmark runner should stay clean. Embedding techniques directly into benchmark code makes it harder to maintain, to compare and toggle techniques, to reproduce results, and to reuse the approach on another benchmark.

So experimental logic lives in the proxy layer: the proxy owns the techniques, the benchmark runs and scores tasks, and the backend serves the model.

Practical stack

GAIA Benchmark
        ↓
InspectAI GAIA runner
        ↓
Test-Time Scaling Proxy
        ↓
OpenAI-compatible model backend
        ↓
Local / Colab / Mac / GPU inference

For InspectAI, the proxy looks like a normal model. For the proxy, the base model looks like a normal OpenAI-compatible endpoint.

Both sides remain simple:

  • the benchmark does not know about techniques;
  • the base model backend does not know about the benchmark;
  • the proxy connects them and adds experimental logic.

Research directions

The project studies test-time techniques that change model behavior through inference organization rather than training.

Example directions:

  • Best-of-N;
  • self-consistency;
  • adaptive sampling;
  • ranked voting;
  • context compaction;
  • structured notes;
  • budget-aware reasoning;
  • tool-result summarization;
  • branch isolation;
  • final-answer selection;
  • intermediate state tracking.

The general idea is not to change model weights, but to improve how the model is called, how context is stored, how intermediate results are processed, and how the final answer is selected.

Test time scaling

Test-time scaling tries to improve answer quality by spending more compute at inference time.

The simplest example is to make several attempts and choose the best one.

More advanced variants:

  • run several independent reasoning branches;
  • ask the model to verify its own answer;
  • use voting;
  • estimate task difficulty and choose the compute budget adaptively;
  • save intermediate conclusions;
  • use extra model calls only for difficult tasks.

The project is not limited to a strict definition of test-time scaling. Context-saving techniques also matter because real compute is limited and models can be small.

Context optimization

In GAIA-like tasks the history grows fast — prompts, reasoning, tool calls and results, retries, branch histories. For a small context window that becomes a bottleneck: the model can lose information, overflow the context, or degrade on a long history.

So the project supports techniques that compress tool results, extract facts, save structured notes, drop irrelevant history, and isolate branches — helping small models work effectively on benchmark tasks.

Why small models matter

The project reflects a practical situation: access to large closed models or a large GPU cluster is not always available.

Commonly available resources are:

  • small open-source models;
  • quantized models;
  • local CPU;
  • Mac / Apple Silicon;
  • Google Colab;
  • one local GPU;
  • limited context;
  • limited memory;
  • slow inference.

In this setting, techniques that improve the usefulness of the model without training and without heavy infrastructure are especially important.

Heterogeneous compute

llama.cpp       — CPU inference
llama.cpp CUDA  — Google Colab
vLLM            — local GPU machine
MLX             — Mac / Apple Silicon

This affects the whole design:

  • the backend is selected through configuration;
  • the benchmark runner does not depend on the backend;
  • the proxy does not depend on a specific model;
  • the launcher knows how to start the selected pipeline;
  • results and logs are saved separately;
  • the run can be reproduced.

Project modularity

The project is split into two logical parts. The proxy / micro-framework handles experimental techniques. The runner / launcher handles reproducible experiment execution.

Proxy micro framework

This part provides an OpenAI-compatible proxy, hook points, task-local state, middleware chains, feature flags, technique modules, extra model calls, and answer aggregation.

Runner launcher

This part starts the base model backend, proxy, and InspectAI / GAIA; performs readiness checks; saves logs; performs cleanup; handles backend-specific env; and supports multiple inference environments.

The project currently lives in one repository because the proxy, launcher, benchmark runner, and backend are tightly connected during active development. After stabilization, the infrastructure can be split into reusable components: tts-proxy-framework and gaia-experiment-launcher.

Project output

The output is not a single benchmark score. It is infrastructure that makes such scores quick to produce and compare.

The project contribution includes:

  • a unified benchmark pipeline;
  • a proxy layer for test-time techniques;
  • a hook-based micro-framework;
  • task-local state;
  • feature flags;
  • support for Best-of-N and related techniques;
  • context-saving capabilities;
  • a launcher for different backends;
  • Colab support;
  • Mac / Apple Silicon support;
  • CPU / CUDA scenarios;
  • reproducible .env configuration;
  • centralized logs;
  • a path toward reusable components.

Typical experiment

1. Choose a backend
2. Choose a model
3. Configure GAIA task and split
4. Enable feature flags
5. Start the launcher
6. Launcher starts the backend
7. Launcher starts the proxy
8. Launcher starts InspectAI / GAIA
9. Proxy applies selected techniques
10. Benchmark receives final answers
11. Logs and results are saved
12. Configuration can be reproduced later

Configuration example

BASE_MODEL_RUNNER_TYPE=vllm16GB
HF_TOKEN=hf_...
GAIA_TASK=inspect_evals/gaia_level1
GAIA_SPLIT=validation
FEATURE_SELF_CONSISTENCY=1
FEATURE_CONTEXT_COMPACTION=1
FEATURE_EXAMPLE=0
BASE_MODEL_API_BASE_URL=http://127.0.0.1:18081/v1
BASE_MODEL_API_KEY=EMPTY
BASE_MODEL_NAME=qwen3.5-9b-q4

This structure makes the run understandable: one file describes the experiment, and another describes the concrete backend.

Audience

The project is aimed at LLM-behavior researchers and benchmark teams — anyone developing or comparing inference-time techniques, running GAIA on local open-source models under limited compute, and wanting benchmark code kept separate from experimental code.

Limitations

The project is aimed at research benchmark runs, not production serving.

This means:

  • the main focus is research convenience;
  • additional model calls are acceptable;
  • experimental logic is acceptable;
  • reproducibility matters;
  • logs matter;
  • flexibility matters;
  • latency is not the only primary criterion.

Compute is still the main limit: the strongest combination of features has not been run end-to-end yet, so the best achievable agentic performance is likely higher than the current results show.

The project also does not promise support for every possible scaffold. It is best suited for techniques that can be expressed through lifecycle hooks and a middleware chain.

Summary

CAPA is practical infrastructure for pushing and measuring agentic performance inside a benchmark loop. An OpenAI-compatible proxy with a hook-based micro-framework lets techniques — context saving, test-time scaling, feature-based activation with task-local state — be switched on and combined, while a GAIA / InspectAI launcher keeps runs reproducible across inference backends.

The goal is to make new model-behavior techniques easy to add, run, and compare — so scaling-law estimates for agents come from the strongest setup we can build rather than a naive baseline, without rewriting the benchmark or binding the system to one backend.