Cmpnd Releases dspy-wsp: Weight-Space Programming Comes to DSPy

Cmpnd has released dspy-wsp, a prototype optimizer that brings weight-space programming to DSPy. Weight-space programming is a technique in which a frontier LLM reads another model’s weights and edits them directly. There are no gradients, no training loop and no fine-tuning dataset. dspy-wsp packages this as WeightSpace, an optimizer that plugs into any DSPy program and metric. In Cmpnd’s tests, about $5 to $10 of GPT-6-Astra calls improved every decision model they tried. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension.

TL;DR

• What it is: A DSPy optimizer (WeightSpace) where a writer LLM edits a student model’s weights. MIT license, pip install dspy-wsp, needs DSPy 3.5.
• Size: Tested on open decision models from 0.6B to 9B parameters. One 25-pass run edited a single layer or head.
• Runs on: One Mac for all experiments. 2B models used about 6 GB of memory. d1-omni-600M needs 4 GB or more (Mac or GPU).
• Performance: WeightSpace alone added 1.9 to 16.1 points on Typed Decisions. With ReAnchor, gains were 6.5 to 20.0 points.
• Best: d1-omni-600M rose from 50.9% to 69.1% (+18.3) with WeightSpace + ReAnchor.
• Worst: Clef-Flash (9B) gained only +1.9 from WeightSpace alone. With the original paper’s stricter rule, d1-omni-600M gained just +2.8.
• Bottom line: Best: a cheap, gradient-free edit that ships as a small, verifiable patch. Worst: very experimental, slow (78 to 167 minutes per run) and weakest on larger models.

What is weight-space programming?

Weight-space programming (WSP) treats a neural network’s weights as a program that can be read and rewritten. The idea comes from MIT research: Sparks of Weight-Space Programming.

A frontier model (the research team used GPT-6-Astra) looks at weight tensors, reasons about what each part computes, and changes them to match a goal written in plain English. The research team stresses that WSP names an emerging capability, not a fixed method. Gradients were banned in their experiments, but the concept does not rule them out.

How is it different from fine-tuning or prompt optimization?

It helps to compare the 3 levers developers usually pull:

• Prompt optimization (DSPy’s MIPROv2, GEPA) edits text: instructions and examples. The model’s weights never change.
• Fine-tuning (DSPy’s BootstrapFinetune, used inside BetterTogether) changes weights with gradient descent. It needs training data, a training loop and usually a GPU.
• Weight-space programming changes weights by reasoning. The editor forms a hypothesis (“these neurons over-score option B”), writes code to patch them, and checks the result with a few forward passes.

The research post frames WSP as the next step after using LLMs to optimize text artifacts. Earlier work let LLMs hill-climb on prompts, harnesses and training code. Weights were left to Adam or to trained weight-generating networks. WSP argues LLM reasoning can now reach the weights directly.

What did the original research show?

The MIT team ran 9 experiments. Several are striking:

• Reading weights: Astra named the correct game (Breakout, Pong, Pac-Man, Freeway) for 4 PPO policy networks from weights alone.
• Fixing a broken policy: A Kinetix car-on-ramp policy failed all 256 episodes. One edit to 3.5% of its weights made it succeed in all 1,024 test episodes.
• Writing from zero: On MinAtar Breakout, a zero-initialized network written in 1 shot scored 517.4. PPO reached 25.3 after 500M steps.
• Fixing an LLM: Qwen3.5-0.8B looped on math problems. Astra traced the issue to MLP layers 15 and 19 and changed 131,072 weights (0.016% of the model). With 50 forward passes, MATH-500 rose from 7.9% to 48.1% and GSM8K from 13.8% to 67.1%.

The research also shows limits. On Gemma 3 270M, edits moved MATH-500 from 6.8% to between 7.0% and 7.7%. The research team did not count that as a real gain. Editing can remove an obstacle, but it could not add a missing ability.

How does dspy-wsp work?

dspy-wsp turns that research loop into a standard DSPy optimizer. The GitHub Repo describes 3 actors:

1. The student: the small model your DSPy program runs on. Its weights load into a local Python process.
2. The writer: a frontier LLM, recommended as gpt-6-astra at maximum reasoning effort.
3. The sandbox: a kernel where the writer runs code. macOS uses built-in sandbox-exec; Linux uses bubblewrap.

The writer works in turns. Each turn has a short explanation, then 1 code block. That block is either:

• a Python block that reads or edits weights inside the sandbox, or
• a forward block that runs your DSPy program on practice examples.

Forward passes come from a fixed budget (max_passes=25 in the tests). When the budget runs out, the writer’s best edit becomes the patch. A hidden validation set then compares original and edited weights on the same examples and seeds.

What data does the writer see?

The repo is explicit about data separation:

• trainset is the writer’s practice set.
• holdout moves a fraction (0.25 in the example) of training data into a check set. The writer can score edits on it but never sees the examples.
• valset stays hidden and runs once at the end.
• Metrics can return dspy.Prediction(score=..., feedback=...). The feedback text is visible to the writer.

What did the writer actually change?

The results write-up lists every edit. The edits are small and targeted:

d1-2B has no decision head, so the writer edited its backbone at layer 23. On d1-omni-600M, the writer rewrote part of the decision head’s scorer.

What does the code look like?

The Repo example triages support tickets. The signature uses only decision-type outputs:

import dspy
from typing import Literal
from dspy_wsp import WeightSpace, load_decision_backend

class Triage(dspy.Signature):
    """Triage a support ticket."""
    ticket: str = dspy.InputField()
    urgent: bool = dspy.OutputField(desc="Does this ticket need a reply today?")
    team: Literal["billing", "technical", "account"] = dspy.OutputField(desc="Which team should handle it?")

backend = load_decision_backend("d1")   # d1-omni-600M
optimizer = WeightSpace(
    metric=metric,
    writer_lm=dspy.LM("openai/gpt-6-astra", model_type="responses", reasoning_effort="max"),
    backend=backend,
    max_passes=25,
    holdout=0.25,
    run_dir="runs/triage",
)
compiled = optimizer.compile(dspy.Predict(Triage), trainset=train, valset=test)
compiled(ticket="I was charged twice this month.")   # runs on the edited weights

After compile, the program calls the edited local model instead of a hosted API. compiled.wsp_patch.summary() shows what changed.

‘)});
var G=[[0,9.3,16.1,18.3],[0,11.6,13.3,17.3],[0,9.5,13.4,17.3],[0,1.5,1.9,6.5]];
var rc=0;
function drawBars(k){rc=k;qa(‘#rsb .btn’).forEach(function(b){b.classList.toggle(‘on’,+b.dataset.r===k)});var r=qa(‘#bars .br’);
r.forEach(function(row,i){var v=D[k][i];var f=row.querySelector(‘.fill’);f.style.width=”0″;var g=i?’ +’+G[k][i].toFixed(1)+’‘:”;row.querySelector(‘.v’).innerHTML=v.toFixed(1)+’%’+g;setTimeout(function(){f.style.width=(v/85*100)+’%’},60+i*120)})}
qa(‘#rsb .btn’).forEach(function(b){b.onclick=function(){drawBars(+b.dataset.r)}});drawBars(0);
window.addEventListener(‘load’,post);window.addEventListener(‘resize’,post);setTimeout(post,300);
})();