Proxy micro framework and runner infrastructure

Short version

Test-Time Scaling Proxy is an OpenAI-compatible proxy over a base language model. It is designed for experiments with test-time scaling, Best-of-N, self-consistency, context-saving, and other lightweight techniques that can be inserted into a benchmark loop without changing the benchmark code.

InspectAI / GAIA -> Test-Time Scaling Proxy -> OpenAI-compatible model backend

The benchmark calls the proxy as a normal OpenAI-compatible model. The proxy can modify the request, call the base model, process the response, run additional model calls, aggregate results, and return one final response in the expected format.

Why the proxy exists

The goal is to make model-behavior experiments fast, reproducible, and extensible. Instead of embedding every new technique into the benchmark runner, a technique is implemented as a separate module and connected to task, model-call, and tool-call lifecycle hooks.

This is especially useful when many techniques need to be compared and change often, some require several internal model calls with state preserved between steps, the benchmark must stay unchanged, the base model may run on different backends, and the run is a research loop rather than production inference.

What the proxy does

The proxy accepts OpenAI-compatible requests and itself looks like a normal model to the benchmark runner.

In practice the proxy can accept a /v1/chat/completions request; read its messages, tools, and parameters; adjust the prompt before forwarding to the base model's OpenAI-compatible endpoint; then modify the response, run several internal calls, aggregate them, select the best answer, save usage and internal state, and return one final OpenAI-compatible response to the benchmark runner.

All of this logic is hidden from the benchmark. It continues to behave as if it were working with one model.

Technique classes

The proxy is not meant for every possible inference architecture. It is best suited for techniques that naturally fit a hook-based model.

Good fits include Best-of-N, self-consistency, ranked voting, adaptive Best-of-N, budget tracking, context compaction, structured notes, and ReSum-like methods — more generally, any technique that records intermediate state, modifies the prompt before a call, analyzes the response after it, compresses tool results, tracks usage and context length, or adds lightweight test-time scaling without rewriting the execution loop.

Heavy tree-search scaffolds, complex external planners, and production serving architectures are less natural fits, although parts of them can still be expressed through hooks.

Hook based micro framework

The proxy contains a small micro-framework with hook points grouped by lifecycle.

Task lifecycle

These hooks are related to the start and end of a benchmark task.

• before_task_call
• after_task_call

They can initialize technique state, save the original task payload, start a new branch, collect the final answer, select the best result, or clear state after the task finishes.

Model call lifecycle

These hooks are related to each call to the base model.

• before_chat_message
• after_chat_message

They can modify the request before sending it to the model, add instructions, manage message history, collect usage, log responses, save intermediate variants, and prepare data for later aggregation.

Tool call lifecycle

These hooks are related to tool calls and tool results.

• before_tool_call
• after_tool_call

They can track tool calls, modify payloads, compress tool results, save observations, manage context after tool use, and implement techniques tied to tool-augmented reasoning.

Interface and state

class Technique:
    feature_name = "FEATURE_MY_TECHNIQUE"

    def before_task_call(self, payload): ...
    def before_chat_message(self, payload): ...
    def after_chat_message(self, response): ...
    def after_task_call(self, response): ...
    def before_tool_call(self, response): ...
    def after_tool_call(self, payload): ...

A technique does not need complex infrastructure. In the simple case, it is one file and one class that implements only the required hooks. The remaining hooks can stay empty or return data unchanged.

Task local state

Task-local state is available between hooks. This is an important part of the architecture.

A technique can store counters, intermediate answers, usage, model- and tool-call counts, the original and current-branch payloads, voting results, compacted context, structured notes, task-complexity signals, and any data a later hook needs.

This enables multi-step techniques such as Best-of-N, which needs to remember branches, collected answers, remaining attempts, and the selected final answer.

Composition and flags

Multiple techniques can be enabled at once. Active techniques form a linear middleware chain:

request/response -> Technique A hook -> Technique B hook -> Technique C hook

For example, one technique can track budget, another can compact context, a third can run Best-of-N, a fourth can collect structured notes, and a fifth can select the final answer.

This composition does not require rewriting the benchmark runner and does not require merging all techniques into one monolithic module.

Feature flags

FEATURE_SELF_CONSISTENCY=1
FEATURE_CONTEXT_COMPACTION=1
FEATURE_EXAMPLE=0

This makes it quick to enable or disable techniques, save an experiment configuration, reproduce a run later, and compare several technique sets on the same benchmark.

Technique examples

Benchmark request
        ↓
TTS Proxy
        ↓
Branch 1 -> base model
Branch 2 -> base model
Branch 3 -> base model
        ↓
Aggregation / voting / selection
        ↓
Final response to benchmark

The benchmark receives one final answer and does not need to know that several internal attempts were made.

This technique needs to save the original task payload, run several isolated branches, collect their answers, select the final one, and return a single OpenAI-compatible response.

Context saving example

In GAIA-like tasks, the model can use tools, receive long tool results, and gradually accumulate context. For small models or models with limited context, this becomes a problem.

The proxy can implement context-saving techniques that compress long tool results, extract structured notes, drop irrelevant history, store compact task state, track context growth, and add a short state summary to the prompt instead of the full history.

This is especially important under limited compute, when using small local models, quantized models, or models with a small context window.

Proxy boundaries

A normal model wrapper usually only forwards requests. Test-Time Scaling Proxy adds a research-oriented lifecycle layer.

It does not simply pass the request to the backend; it creates intervention points in the benchmark loop:

• before the task;
• before the model call;
• after the model call;
• before the tool call;
• after the tool result;
• after the task finishes.

This turns the proxy into a micro-framework for experiments, not just an HTTP proxy.

Responsibility boundary

Test-Time Scaling Proxy does not replace the benchmark runner and does not replace the model backend.

The benchmark runner is responsible for:

• choosing the benchmark;
• loading tasks;
• scoring;
• the execution loop;
• tools;
• evaluating the result.

The base model backend is responsible for:

• loading the model;
• inference;
• the OpenAI-compatible endpoint;
• backend-specific behavior: llama.cpp, vLLM, MLX, and so on.

Test-Time Scaling Proxy is responsible for:

• experimental intervention points;
• connecting techniques;
• task-local state;
• additional model calls;
• aggregation;
• context-saving;
• preserving the OpenAI-compatible interface for the benchmark.

Place and audience

Today the micro-framework is part of a unified GAIA/InspectAI experiment project. Later it can be extracted into a reusable repository such as tts-proxy-framework.

Such a repository could be used not only for GAIA, but for any benchmark or evaluation loop that can call an OpenAI-compatible model endpoint.

Audience

The component is useful for test-time-scaling researchers, benchmark-infrastructure developers, teams comparing inference-time techniques, people working with local or small LLMs, and anyone who wants to separate benchmark code from experimental techniques or prototype new scaffolds on OpenAI-compatible models quickly.

Summary

Test-Time Scaling Proxy makes benchmark experiments simpler:

• the benchmark remains unchanged;
• the model remains a normal OpenAI-compatible backend;
• techniques are added as separate modules;
• hooks provide intervention points in the task lifecycle;
• task-local state enables multi-step techniques;
• feature flags make experiments reproducible;
• different techniques can be combined in one run.

The main idea is to separate research techniques from the benchmark runner and model backend by turning them into a pluggable layer between the two.

Short version

GAIA Experiment Launcher starts the full experimental loop:

base model backend -> Test-Time Scaling Proxy -> InspectAI / GAIA

It allows the same benchmark to run reproducibly on different inference backends and compute environments: CPU, CUDA, Google Colab, Mac / Apple Silicon, and local GPU machines.

Problem and pipeline

Local LLM experiments depend heavily on the runtime environment. One model may run through llama.cpp on CPU, another through llama.cpp CUDA in Colab, another through MLX on Apple Silicon, and another through vLLM on a GPU server. The benchmark runner and proxy still need the same OpenAI-compatible endpoint.

Without a launcher, every run becomes a manual sequence: start the backend and confirm the model loaded, start the proxy and confirm it is reachable, start InspectAI / GAIA, pass the endpoint and API key correctly, save logs, clean up processes after failures, and repeat all of it on the next machine.

The launcher automates this process.

Launch pipeline

1. Base model backend
2. Test-Time Scaling Proxy
3. InspectAI / GAIA benchmark runner

The backend serves an OpenAI-compatible API. The proxy connects to the backend and exposes another OpenAI-compatible API to the benchmark. InspectAI / GAIA calls the proxy as the model.

Launcher and backends

The project needs repeated runs across different configurations — models, backends, techniques, feature flags, GAIA levels and splits, runtime environments, Docker / no-Docker modes, and local machines or Colab.

The launcher turns this into a reproducible configuration-driven process.

Supported inference modes

llama.cpp       — CPU inference
llama.cpp CUDA  — Google Colab
vLLM            — local GPU machine
MLX             — Mac / Apple Silicon

These modes cover several practical scenarios.

llama.cpp CPU inference

This is used to run models on CPU. It is useful when no GPU is available or when the pipeline needs to be checked on a simple machine. CPU inference is slow, but convenient as a baseline and fallback.

llama.cpp CUDA Google Colab

This is used for CUDA inference in Google Colab. Colab is important because it provides GPU access without setting up a dedicated machine. The no-Docker mode is especially important in Colab because Docker is not always available or convenient in a notebook environment.

vLLM local GPU machine

vLLM is used as a local GPU backend. It is useful for OpenAI-compatible serving, even if in some experiments it may be less convenient or perform worse than other options.

The key point is that the launcher separates the benchmark pipeline from the concrete backend: if one backend is a poor fit, it can be replaced without changing benchmark or proxy logic.

MLX Mac Apple Silicon

MLX is used on Mac / Apple Silicon. This matters for local development and experiments on Apple hardware. It allows a Mac to act as a full pipeline test environment without requiring CUDA.

What the launcher does

The launcher starts pipeline components in order:

base model backend -> TTS Proxy -> InspectAI / GAIA

Then it controls component readiness, saves logs, and helps reproduce the run.

Main functions:

• starts pipeline components sequentially;
• checks endpoint readiness;
• cleans processes during restarts;
• saves stdout/stderr logs for all components in one place;
• can save Colab results and logs to Google Drive;
• uses .env for configuration;
• allows backend switching without rewriting scripts;
• separates common benchmark parameters from backend-specific parameters.

Endpoint readiness

Pipeline components depend on each other. InspectAI / GAIA should not start before the proxy is available. The proxy should not start work without a base model backend. The backend must first load the model and expose an endpoint.

1. Start base model backend
2. Start Test-Time Scaling Proxy
3. Start InspectAI / GAIA benchmark

A launched process is not necessarily ready to receive requests. The launcher performs readiness checks before moving to the next stage, which reduces manual errors and is especially important for heavy models and unstable environments such as Colab.

Logs and process cleanup

Logs are essential for debugging. If a run fails, it should be clear where: the backend failed to load the model, the proxy could not reach the backend, the benchmark runner errored, the endpoint never started, memory ran out, configuration broke, or a technique returned an invalid response.

The launcher stores stdout/stderr logs for all parts in one place. This makes it possible to reconstruct the run and understand which component failed.

Process cleanup

Repeated experiments often involve failures, hanging processes, or restarts. Without cleanup, stale backends, stale proxies, port conflicts, or competing processes can appear.

The launcher cleans processes during restarts and shutdowns.

This is useful during iterative work:

• change .env;
• enable a new feature;
• restart the benchmark;
• get new logs;
• repeat with another model.

Colab and Docker

Google Colab is an accessible CUDA environment, but a constrained one: it is temporary, processes can fail, the runtime can restart, files can disappear, Docker may be unavailable, and results must be saved explicitly.

The launcher therefore supports a workflow where benchmark results and logs can be saved to Google Drive. This makes Colab useful not only as a one-off environment, but as a compute backend for a series of experiments.

Docker and no Docker modes

The project accounts for the fact that different environments require different launch methods. Docker is convenient for local reproducibility, but it is not always available in Colab. Therefore the launcher supports a no-Docker scenario.

The split matters because:

• local runs can use Docker;
• Colab can run without Docker;
• scripts and .env remain the common control surface;
• the benchmark pipeline keeps the same structure.

Env configuration

BASE_MODEL_RUNNER_TYPE=vllm16GB
HF_TOKEN=hf_...
GAIA_TASK=inspect_evals/gaia_level1
GAIA_SPLIT=validation
FEATURE_SELF_CONSISTENCY=1
FEATURE_CONTEXT_COMPACTION=1
FEATURE_EXAMPLE=0

This file controls:

• backend runner selection;
• GAIA dataset access;
• benchmark task;
• split;
• enabled experimental techniques.

Backend specific env

BASE_MODEL_API_BASE_URL=http://127.0.0.1:18081/v1
BASE_MODEL_API_KEY=EMPTY
BASE_MODEL_NAME=qwen3.5-9b-q4

This allows switching the backend without mixing model, endpoint, and benchmark parameters in one place.

The common .env describes the experiment. The backend-specific .env describes the concrete way the base model is served.

Reproducibility

A saved .env effectively describes the run.

From it, one can see:

• which backend was used;
• which model was connected;
• which benchmark task was run;
• which split was used;
• which techniques were enabled;
• which environment parameters mattered.

This is important in a research workflow, where results need to be compared and differences between runs need to be explained.

Small compute

The project was created under limited-compute conditions, and that affects the architecture.

When a large GPU cluster is not available, heterogeneous resources become necessary: local CPU, Mac / Apple Silicon, Google Colab, local GPU, and quantized or small models across different backends.

The launcher avoids binding the whole project to one ideal inference setup. Instead, the pipeline can move between available environments.

Place and audience

The launcher currently lives in the same repository as the proxy and GAIA/InspectAI runner. Later it can be extracted into gaia-experiment-launcher.

Such a repository could be used independently of a specific set of test-time techniques and serve as an infrastructure layer for running GAIA / InspectAI on different compute backends.

Audience

The launcher is useful for local GAIA runs, benchmarks on small models, backend comparisons, Colab and Mac / Apple Silicon runs, reproducible evaluation, and research teams that need to switch models and backends quickly under limited compute.

Summary

GAIA Experiment Launcher handles the infrastructure side of the project:

• starts the full benchmark pipeline;
• supports different inference backends;
• works with heterogeneous compute;
• is controlled through .env;
• saves logs;
• checks endpoint readiness;
• cleans processes;
• helps reproduce experiments.

The main idea is that launching a benchmark should be as reproducible and switchable as the experiment configuration.