Program-as-Weights: A Programming Paradigm for Fuzzy Functions

ChapterPal

Wentao Zhang$^{1}$ $^{*}$
$^{1}$University of Waterloo, {w564zhan, lhotsko, pynie, yuntian}@uwaterloo.ca

Liliana Hotsko$^{1}$ $^{*}$
$^{1}$University of Waterloo, {w564zhan, lhotsko, pynie, yuntian}@uwaterloo.ca

Woojeong Kim$^{2}$ $^{*}$
$^{2}$Cornell University, [email protected]

Pengyu Nie$^{1}$
$^{1}$University of Waterloo, {w564zhan, lhotsko, pynie, yuntian}@uwaterloo.ca

Stuart Shieber$^{3}$
$^{3}$Harvard University, [email protected]

Yuntian Deng$^{1}$
$^{1}$University of Waterloo, {w564zhan, lhotsko, pynie, yuntian}@uwaterloo.ca

$^{*}$ Equal contribution

Abstract

Many everyday programming tasks resist clean rule-based implementation, such as alerting on important log lines, repairing malformed JSON, or ranking search results by intent, and are increasingly outsourced to large language model APIs at the cost of locality, reproducibility, and price. We propose fuzzy-function programming: compiling such a function from a natural-language specification into a compact, locally-executable neural artifact. We instantiate this paradigm with Program-as-Weights (PAW), in which a 4B compiler trained on FuzzyBench, a 10M-example dataset we release, emits parameter-efficient adapters for a frozen, lightweight interpreter. A 0.5B Qwen2 interpreter executing PAW programs matches the performance of direct prompting of Qwen2-72B, while using roughly one fiftieth of the inference memory and running at 30 tokens/s on a MacBook M3. PAW reframes the foundation model from a per-input problem solver into a tool builder: invoked once per function definition, it produces a small reusable artifact whose subsequent calls per function application are cheap and offline.

Executive Summary: The document introduces Program-as-Weights (PAW), a new approach to handling “fuzzy” programming tasks—functions such as log monitoring, intent-based search ranking, JSON repair, and tool selection—that resist precise symbolic rules. These tasks are currently handled by repeated calls to large language-model APIs, which create ongoing costs, reproducibility risks, and a lack of offline or self-contained operation.

The work set out to test whether a fuzzy function could be compiled once from a natural-language description into a small, reusable neural artifact that runs locally on a fixed, lightweight model. The authors built a 10-million-example training set called FuzzyBench, trained a 4-billion-parameter compiler to emit task-specific LoRA adapters, and paired it with a frozen 0.6-billion-parameter interpreter. They evaluated the system on held-out test specifications, compared it against direct prompting of much larger models and against conventional fine-tuning, and tested five practical case studies plus a multimodal extension.

A 0.6 B interpreter running PAW programs reached 73.8 % exact match on FuzzyBench, surpassing direct prompting of a 32 B model (68.7 %) while using roughly fifty times less inference memory. The same system runs at 30 tokens per second on a MacBook M3 after quantization, with each program occupying about 23 MB on disk. The approach proved robust to noisy or ambiguous specifications because the compiler first rewrites them into a clean “pseudo-program.” It also generalized to image-conditioned tasks without changing the interpreter.

These results indicate that large models can be used once, at compile time, to produce compact, versionable, and locally executable programs. The shift reduces API dependence, improves reproducibility and privacy, and makes previously impractical fuzzy functions economical to run continuously on-device or in browsers.

Organizations facing repeated fuzzy-logic workloads should pilot PAW on a narrow set of high-volume tasks, such as log triage or internal search reranking, using the released compiler and interpreter. Further work needed before broad adoption includes cross-interpreter portability tests, expanded validation on non-synthetic tasks, and exploration of additional adapter formats.

The main limitations are that each compiler is tied to a specific interpreter family, training data is entirely synthetic, and most evaluations cover single-step functions. The quantitative claims rest on a large, controlled benchmark and consistent ablation results, giving reasonable confidence in the reported performance gains, while real-world deployment outcomes remain to be measured.

1. Introduction

Section Summary: Traditional programming excels at precise, rule-based tasks but struggles with fuzzy real-world problems that resist exact specification, such as spotting important log messages or parsing messy text. Developers currently handle these by calling large language-model APIs, which the authors find costly, inconsistent, and dependent on remote services. They introduce Program-as-Weights, a system that converts a plain-language description into a tiny neural adapter run locally by a small frozen interpreter, delivering fast and private results without repeated API calls.

Programming has historically been about writing explicit rules in a formal language designed for the purpose, a programming language. A function is defined by code, and the computer executes it deterministically. For many tasks, this paradigm works beautifully: sorting numbers, processing structured data, computing matrix products. Yet a large class of real-world functions resists precise specification. Consider, for instance, filtering a computer log to alert someone only on the log lines or messages that matter, repairing malformed JSON, or ranking search results by intent. Even apparently "simple" tasks, such as writing a regular expression to parse text with many edge cases, prove brittle. Beyond underspecification, real-world inputs are noisy: typos and format drift routinely break hand-written rules and regexes. These are fuzzy functions ([1]): problems that humans find intuitive but that cannot be fully captured by crisp symbolic rules.

Today, developers frequently outsource such fuzziness to LLM APIs. It is increasingly common to find codebases where a remote LLM is called (e.g., $\texttt{gpt(``extract answer'', text)}$) to implement functions that are otherwise intractable to program. This approach is undeniably convenient, but it is costly, fragile, undermines reproducibility because providers may silently update their models ([2]), and prevents software from being self-contained.

We propose a different paradigm with three steps: the developer describes the function in natural language; a neural compiler turns that description into a small neural binary; and a frozen, lightweight neural interpreter, installed once on the user's device, runs that binary just like a user-defined function (Figure 1). We call this paradigm Program-as-Weights (PAW). Any sufficiently expressive parameter-efficient module (PEFT) emitted by a hypernetwork can serve as the program form; we instantiate two, prefix-tuning and text-to-LoRA, and find LoRA better, with future PEFTs possibly better still.

A PAW program has two halves. The first is a pseudo-program in natural language, a restatement of the user's specification. The second is a PEFT module that re-tunes the frozen interpreter for this one task: in our precursor system this was a prefix-tuning KV cache; in our current system it is a LoRA generated by the compiler from its own hidden states and injected into the interpreter. The discrete half shields the interpreter from typos and ambiguity in the original specification; the continuous half supplies the fine-grained behavioral control that text alone cannot.

The compile pipeline has two stages, both running 4B Qwen3 models. The first stage is a pseudo compiler, an off-the-shelf model we never train: prompted with a small task-rewriting template, it turns the user's spec into a clean pseudo-program of a paraphrased description plus a handful of input-output examples. The second stage is a LoRA compiler that we train: it reads the spec and the pseudo-program and emits the LoRA. We train the LoRA compiler on $\textsc{FuzzyBench}$, a 10M-example dataset we release with this paper, built incrementally across 29 thematic versions covering more than 800 categories of fuzzy text tasks such as classification, format conversion, parsing, fuzzy matching, natural-language commands, agentic tool use, and many more.

The result is a small, fast, and accurate system. A Qwen3-0.6B interpreter executing PAW programs outperforms direct prompting of Qwen3-32B (73.78% vs. 68.70% exact match) at roughly one fiftieth the inference memory. Quantized, the same system runs at 30 tokens per second on a MacBook M3 from a $\sim$ 430 MB GGUF base shared across functions plus a 23 MB per-program LoRA adapter; a smaller GPT-2 path runs entirely client-side in the browser via WebAssembly.

We see Program-as-Weights as a concrete step toward a small-model future ([3]), in which the heavy lifting happens once at compile time and the day-to-day work of running software happens locally. We illustrate its applications in five case studies: output triage (event-driven log monitoring), custom classification (intent-based site navigation), fuzzy search (semantic search reranking), agent preprocessing (a tool-calling pipeline that scores 93% on $\textsc{ToolCall-15}$), and creative generation (a multilingual word-guessing game). Each is the kind of fuzzy task that resists symbolic implementation but does not need an API call to a 30B-parameter model on every input. We additionally show the abstraction's modality generality: replacing only the compiler with a vision-language model while keeping the same interpreter runs PAW programs on image-conditioned fuzzy tasks. Our code can be found at https://github.com/programasweights and a public demo is available at https://programasweights.com.

2. Programs as Weights

Section Summary: The section introduces "neural programs" as a way to capture fuzzy functions—those more easily described in natural language or examples than in code—by compiling a user specification once into a reusable artifact rather than querying a large model repeatedly. This artifact consists of a hybrid program with both discrete tokens and continuous weights such as LoRA adapters; a small fixed interpreter then executes the program on new inputs to produce the desired outputs. The result behaves like ordinary software—it can be saved, versioned, and imported—while the heavy specialization work happens at compile time and the interpreter remains unchanged.

Let $f : X \to Y$ denote a function whose behavior is more naturally specified through natural language, examples, or constraints than through symbolic code, a fuzzy function. Instead of repeatedly invoking an LLM to approximate $f$, we propose to compile a neural program that specializes a fixed model to implement $f$.

Formally, let $s$ denote a user specification, expressed in natural language and optionally accompanied by example input-output pairs $(x, y)$. A neural Compiler maps $s$ to a program $p$. A small fixed neural Interpreter executes $p$ on inputs $x \in X$ to produce outputs $\hat{y} \in Y$:

$ p ;=; \texttt{Compiler}(s), \qquad \hat{y} ;=; \texttt{Interpreter}(p, x) ;\approx; f(x).\tag{1} $

This division mirrors classical programming, where a compiler translates source code into an executable that is later run by a runtime. The crucial difference is that the executable here is a learned parameter blob, and the runtime is a neural network. The interpreter does not need to be retrained: introducing a new fuzzy function only requires compiling a new program $p$.

Hybrid programs.

For conceptual simplicity, $p$ may be viewed as a single continuous object. In our concrete instantiation, however, $p$ is a hybrid of a discrete and a continuous component:

$ p ;=; \bigl(p_{\text{discrete}}, ; p_{\text{continuous}} \bigr).\tag{2} $

The discrete component $p_{\text{discrete}}$ is a variable-length sequence of tokens that acts as a self-contained "pseudo-program" presented to the interpreter as part of its input. The continuous component $p_{\text{continuous}}$ can be implemented using any PEFT method, such as a LoRA injected into the interpreter.

Why "program"?

This framing matters because it determines how the artifact is used. A compiled PAW program is a single file ($\sim$ 23 MB at $Q4_0$ for a 0.6B interpreter, plus a one-time shared base) that can be saved, version-controlled, distributed via package managers, and called from Python or JavaScript with a two-line API. PAW programs are objects of the same kind as Python modules: they have a name and a version, but their behavior is encoded in weights rather than in source code. The compiler is the part that does the heavy lifting; the interpreter is a fixed runtime, comparable to a CPU or a byte-code interpreter in conventional software stacks.

3. The Compiler–Interpreter System

Section Summary: The PAW system turns a user's task description into a working program through three fixed parts that stay independent of any particular fine-tuning method. An off-the-shelf model first rewrites the description into a short pseudo-program of restated instructions and examples, after which a second trained model converts that text into a tiny, task-specific module such as a LoRA adapter by mapping its hidden states onto shared low-rank weights. At runtime a single frozen language model simply attaches the generated module and runs the user's input to produce the answer, letting one interpreter serve many different programs without retraining.

3.1 Compiler–interpreter abstraction

The PAW pipeline has three components, none of which depend on the specific PEFT chosen for the program form. A pseudo compiler $C_p$ reads the spec $s$ and produces a discrete pseudo-program $p_{\text{discrete}}$. A PEFT compiler $C_{\text{PEFT}}$ reads the spec together with $p_{\text{discrete}}$ and emits a small parameter-efficient module $p_{\text{continuous}}$ from its hidden states. The frozen interpreter ingests $p_{\text{continuous}}$ at runtime — by attaching it to the appropriate target modules and running the user's input $x$ through it — to produce the output $\hat{y}$. We instantiate the PEFT module in two ways: a prefix-tuning KV cache (Section 3.3) and a LoRA (Section 3.2), with the latter being our current best.

Pseudo compiler.

The pseudo compiler $C_p$ is an off-the-shelf Qwen3-4B-Instruct-2507 model that we never train. Given a specification $s$, we prompt $C_p$ with a small task-rewriting template (full text in Appendix C) that asks for a clean restatement of the task plus a handful of representative input-output examples. The output is the discrete component $p_{\text{discrete}}$ of the program.[^1] The pseudo compiler is shared by both PEFT instantiations below.

[^1]: In an early prototype we trained a single compiler to generate this discrete component via reinforcement learning, but observed that, regardless of seed prompt, the compiler converged on this same paraphrase-plus-examples format; we therefore eliminate the RL stage by directly hand-crafting a prompt that produces this format from an off-the-shelf model.

3.2 Text-to-LoRA: our current best

LoRA compiler.

The LoRA compiler $C_L$ is a second 4B Qwen3 model, initialized from the same checkpoint as $C_p$ but trained (Section 4). Given the spec $s$ and the pseudo-program $p_{\text{discrete}}$ produced by $C_p$, $C_L$ runs a single forward pass on the concatenation $[s \mid p_{\text{discrete}} \mid \texttt{EOS} \mid \tau_1, \dots, \tau_T]$, where $\tau_{1{:}T}$ is a fixed sequence of $T = 64$ learned "prefix" tokens. We extract prefix-position hidden states from $L$ compiler layers spaced uniformly by depth ratio (one per interpreter layer), and stack them into the tensor $H \in \mathbb{R}^{L \times T \times d_{\text{teacher}}}$ that is fed to the LoRA mapper (Figure 2).

LoRA mapper.

The LoRA compiler's hidden states $H$ are converted into per-example LoRA weights by a small parameter-efficient module, the LoRA mapper. For each interpreter target-module type $m$ (attention $q!/!k!/!v!/!o$ and MLP $\text{gate}/\text{up}/\text{down}$), the mapper maintains shared learnable bases

$ A^{(m)}{1{:}N} ;\in; \mathbb{R}^{N \times r \times d{\text{in}}^{(m)}}, \qquad B^{(m)}{1{:}N} ;\in; \mathbb{R}^{N \times d{\text{out}}^{(m)} \times r}. $

These hidden states are mean-pooled over both the $L$ depth-aligned layers and the $T$ prefix positions, $\bar{h} = \tfrac{1}{LT}\sum_{l, t} H_{l, t}$, passed through a shallow MLP trunk $\phi$, and projected into mixing coefficients $\alpha^{A, B}_{l, m, n} \in \mathbb{R}$ for each layer $l$, module type $m$, and basis index $n$, via a single linear head. The LoRA at layer $l$ and module $m$ is

$ A^{\text{ex}}{l, m} ;=; \sum{n=1}^{N} \alpha^{A}{l, m, n} , A^{(m)}n, \qquad B^{\text{ex}}{l, m} ;=; \sum{n=1}^{N} \alpha^{B}_{l, m, n} , B^{(m)}_n.\tag{3} $

We use rank $r = 64$ and $N = 64$ shared bases per module type, applied to all layers and module types. Per fuzzy function, this injects approximately 38.5M LoRA parameters into the interpreter.[^2]

[^2]: We compare this design to several more-expressive alternatives (per-position aggregation, per-layer bases, per-position with per-layer bases, LoRA with prefix-tuning) in Section 7; none improve over the simple shared-basis design.

Interpreter.

The interpreter is a frozen language model. To execute a PAW program on input $x$, we (i) attach the LoRA in Equation 3 to the appropriate target modules, (ii) prepend $p_{\text{discrete}}$ to the input $x$, and (iii) generate the output autoregressively. Because the interpreter is frozen and the LoRA hot-swappable, a single device-resident interpreter can serve unboundedly many PAW programs; Figure 19 illustrates this "one runtime, many programs" picture for three example specifications.

$**Figure 2:** **Text-to-LoRA instantiation of PAW (Section 3.2).** *Left.* The trained LoRA compiler reads the function specification, the pseudo-program produced by an off-the-shelf prompted pseudo compiler $C_p$ (not depicted), and a fixed sequence of learned prefix tokens; it emits prefix-position hidden states $H$. *Middle.* The LoRA mapper mean-pools $H$, passes it through an MLP, and projects into mixing coefficients that compose LoRA matrices $(A^{\text{ex}}, B^{\text{ex}})$ over shared learnable bases Equation (3). *Right.* The frozen interpreter ingests $p_{\text{discrete}}$ prepended to the user input $x$, with the LoRA hot-attached, and generates the output autoregressively. The same pipeline holds for the prefix-tuning precursor (Section 3.3, with architecture in Figure 18); only the mapping from compiler hidden states to PEFT module changes (LoRA $\to$ KV-cache mapper).$

3.3 Prefix-tuning: a precursor instantiation

Prefix compiler.

Our precursor system replaced the LoRA mapper with a prefix-tuning mapper. The prefix compiler $C_P$ is a second 4B Qwen3 model trained the same way as $C_L$, with the only difference being how its prefix-position hidden states are consumed. Given $[s \mid p_{\text{discrete}} \mid \texttt{EOS} \mid \tau_{1:T}]$, $C_P$ produces hidden states $H \in \mathbb{R}^{L \times T \times d_{\text{teacher}}}$ at the same $L$ depth-aligned layers as in Section 3.2. Instead of pooling and projecting into LoRA weights, a small linear mapper $\psi$ projects these hidden states position-wise into KV pairs $(K^{\text{ex}}{l, t}, V^{\text{ex}}{l, t}) \in \mathbb{R}^{2 \times d_{\text{int}}}$ that are prepended to the interpreter's attention KV cache at every layer, in the manner of standard prefix-tuning ([4]) (see Figure 18 in the appendix for an architecture diagram). The interpreter then runs $x$ through its frozen attention with the additional $T$ prefix-position keys and values visible to every query.

Both methods solve the task.

At a controlled comparison scale (same amount of training compute), the prefix-tuning instantiation reaches 50.4% exact match on FuzzyBench, while the LoRA instantiation reaches 56.5% at $r{=}18$ ($r{=}18$ matches the prefix-tuning program size) and 65.7% at $r{=}64$ (Table 1). Both outperform the no-compiler prompting baseline (9.8%). LoRA is the stronger PEFT and is the instantiation we scale to the full training data in subsequent experiments.

\begin{tabular}{@lc@}
\toprule
Method & Accuracy \\
\midrule
Prompting & 0.098 \\
Prefix Tuning & 0.504 \\
Text-to-LoRA, $r{=}18$ & 0.565 \\
{Text-to-LoRA, $r{=}64$ (default)} & \textbf{0.657} \\
\bottomrule
\end{tabular}

4. Training

Section Summary: Only the PEFT compiler is trained while the pseudo-compiler and interpreter remain frozen, so the system learns to generate an adapter that steers the fixed interpreter toward correct outputs when paired with a precomputed pseudo-program. Training proceeds by feeding each input and its pseudo-program through the compiler to produce hidden states that are mapped into LoRA parameters; these parameters are injected into the interpreter, and the sole objective is to maximize the likelihood of the desired target sequence. Gradients from this supervised loss flow back through the frozen interpreter to update only the compiler and mapper parameters.

Only the PEFT compiler is trained. The pseudo compiler $C_p$ is held off-the-shelf and frozen; the interpreter is also frozen. The PEFT compiler is trained to produce a PEFT adapter that, when injected into the frozen interpreter alongside a fixed pseudo-program, maximizes the likelihood of the target output. With both endpoints frozen, this reduces to a single supervised objective. We concentrate below on the LoRA instantiation; the prefix-tuning precursor (Section 3.3) was trained with the same SFT recipe described below, substituting the prefix-tuning mapper for the LoRA mapper.

Objective.

For each training triple $(s, x, y)$, we look up a pre-generated pseudo-program $p_{\text{discrete}} = C_p(s)$, run a forward pass through $C_L$ on $[s \mid p_{\text{discrete}} \mid \texttt{EOS} \mid \tau_{1{:}T}]$ to obtain prefix-position hidden states, pass those through the LoRA mapper to obtain $p_{\text{LoRA}}$, and inject the result into the interpreter. The loss is the negative mean-token log-likelihood of the target $y$ under the frozen interpreter:

$ \mathcal{L}(\theta) ;=; \mathbb{E}{(s, x, y)}!\left[- \log P{\phi}!\left(y , \big|, p_{\text{discrete}}, , p_{\text{LoRA}}(\theta;, s, p_{\text{discrete}}), , x \right) \right],\tag{4} $

where $\theta$ is the parameters of $C_L$ and the LoRA mapper, and $\phi$ are the interpreter parameters. The gradient flows back through the frozen interpreter into the LoRA mapper and from there into $C_L$ 's hidden states. Full hyperparameters and compute setup are in Appendix G.

5. FuzzyBench: A 10M-Example Dataset of Fuzzy Functions

Section Summary: FuzzyBench is a dataset of 10 million examples, each consisting of a natural-language specification together with matching input-output pairs, created to train models that turn vague instructions into working functions. The data were produced in two stages with gpt-5.2: first generating a wide range of realistic specifications, then creating input-output examples for each, and the collection was built incrementally across 29 versions to cover everyday fuzzy tasks such as text parsing, web reasoning, code generation, and safety checks. It supplies clean train-validation-test splits on entirely new specifications plus noisy test variants, while the strongest models reach at most about 96 percent accuracy on the verified test set.

A central obstacle to training PAW-style methods is the lack of a public dataset for "compile a fuzzy function from a specification." We construct FuzzyBench, a 10M-example dataset in which every example is a triple $(s, x, y)$ of (specification, input, target output), generated using gpt-5.2.

Construction.

We use a two-stage pipeline. In the first stage, we prompt gpt-5.2 to generate natural-language specifications of fuzzy functions. Each prompting call produces eight specifications, and we run repeated calls under different category constraints to cover the breadth of fuzzy tasks developers actually encounter. In the second stage, for each specification, we prompt gpt-5.2 again to generate eight input/output pairs. Specifications are split 80/10/10 by spec into train/validation/test, so that test specifications are entirely unseen at training time. For evaluation, we construct a verified test set on which an independent strong model (gpt-5-mini) and gpt-5.2 agree on the output, removing examples where the target itself is ambiguous. Full prompts are in Appendix B.

Thematic coverage.

FuzzyBench is built incrementally over 29 versions, each adding 100K-500K examples covering a new family of fuzzy tasks. Figure 3 groups the resulting 10M examples into seven high-level task families that span what developers actually encounter in fuzzy logic: from raw text processing and parsing to agentic tool use, web intelligence, code-and-command generation, and safety/verification. The full per-version timeline (29 entries; the first version alone establishes 277 base categories, and the final dataset covers more than 800 sub-categories) is in Appendix F.

**Figure 3:** **FuzzyBench-10M task-family distribution.** 29 incremental thematic versions are mapped to 7 high-level families. "Core text processing & NLP" is the largest family because the v1 base layer (2.5M examples; 277 base categories) covers parsing, classification, NER, coreference, and sentiment; the remaining 7.5M examples spread across the other six families.

Noise variants.

For robustness evaluation, we additionally release noise-perturbed versions of the test set along eight axes: typos, grammar errors, ambiguity, formatting drift, "all noise" (combined), terse phrasing, casual phrasing, and paraphrase. Each axis comes in three intensity levels (light, medium, heavy). Section 8 reports robustness numbers.

Empirical ceiling.

The data-generating model itself, gpt-5.2, achieves 96.09%; gpt-5-mini achieves 91.87%. These bound how high any compiled function trained on this data can reach.

6. Main Results

Section Summary: PAW enables a tiny 0.6-billion-parameter language model to serve as an interpreter that runs compiler-generated programs, delivering 73.78 percent accuracy on FuzzyBench and beating direct prompting of a 32-billion-parameter model while using roughly fifty times less memory. The same small interpreter also handles image-conditioned tasks after a vision-language compiler encodes the necessary adaptations into a compact module, outperforming several larger vision-language models on diagram and structured-output benchmarks. Results further show that even much weaker base models can become effective interpreters once equipped with these compiler-produced adaptations, although performance drops on long-form generation where context limits become binding.

We compare PAW against three families of baselines, all evaluated on the same test sets as PAW so that any compute or data-generation differences are absorbed in the comparison.

Baselines.

(i) Direct prompting of open-weight LMs (Qwen3 0.6B, 4B, 8B, 14B, 32B; OLMo3-7B; gpt-oss-20B), and of two API models that bound the empirical ceiling (gpt-5-mini and gpt-5.2). (ii) Symbolic code generation: ALCHEmist's LM-to-code pipeline ([5]), where a strong LM writes Python code to solve the fuzzy task and the code is executed at inference. (iii) Standard adaptation of the same 0.6B base: full fine-tuning across 1-4 epochs, and fixed (non-compiler-generated) LoRAs at ranks $r \in {18, 64, 128}$.

Main result.

Table 2 summarizes the main numbers. A 0.6B-parameter interpreter executing PAW programs achieves 73.78% exact match on FuzzyBench, outperforming prompting Qwen3-32B (68.70%) while using approximately $50\times$ less inference memory ($\sim$ 1.2 GB at bf16 vs. $\sim$ 60 GB).

::: {caption="Table 2: Main results. FuzzyBench uses exact match accuracy on the verified test set. Following the WRENCH benchmark setup of ALCHEmist ([5]), SMS uses F1 and the rest use Acc. Contained indicates the program is self-contained and executable without internet access. PS is per-program shipping size; for prompting baselines this is the prompt/spec size, and for PAW it is the deployed PEFT adapter (Q4_0 quantized for Qwen3 0.6B and Qwen3.5 0.8B; fp32 for GPT-2). †: numbers taken from [5], which uses 10-sample majority voting; the reimplementation row uses single-sample for fairness. ^*: zero F1 due to zero recall."}

:::

Cross-interpreter scaling.

Among three interpreters GPT-2 124M, Qwen3 0.6B, and Qwen3.5 0.8B, Qwen3 0.6B is the strongest interpreter; the hybrid 0.8B is slightly weaker. GPT-2 124M, despite having only 1/5 the parameters of Qwen3 0.6B and no instruction tuning, still achieves 54%, suggesting that the compiler-generated LoRA can encode usable task adaptations even into very small, weakly-capable bases.

Multimodal generalization without changing the interpreter.

The compiler-interpreter abstraction extends to image-conditioned fuzzy functions without changing the interpreter. We swap the text-only Qwen3-4B-Instruct compiler for the same-family Qwen3-VL-4B compiler ([6]), keep the same Qwen3 0.6B interpreter, and reuse the same LoRA mapper. Image conditioning is fully encoded in the PEFT module emitted by the VL compiler, so the small text interpreter never sees pixels. Table 3 reports six image-conditioned tasks: three CoSyn-400K diagram-understanding tasks (Chemical, Circuit, Music) ([7, 8]), the structured-output Im2LaTeX-100K ([9]) and Im2SMILES-20K ([10]) tasks, and the open-ended visual question answering TextVQA ([11]); full prompts are in Appendix D.

PAW (LoRA) outperforms all VLM baselines (up to 4B parameters) on the three CoSyn diagram tasks (Circuit 0.274 vs. 0.196 best baseline; Chemical 0.414 vs. 0.258; Music 0.552 vs. 0.470) at $\sim$ 0.6B interpreter size. On the long-form structured generation task Im2LaTeX, PAW (LoRA) is weaker than its prefix-tuning precursor (0.181 vs. 0.391); a discrete-pseudo-only ablation in Appendix D.1 shows the gap arises because the long input/output examples in the pseudo-program crowd the small interpreter's context budget on long-form tasks.

::: {caption="Table 3: Image-conditioned fuzzy functions. The PAW rows use the same Qwen3 0.6B and Qwen 3.5 0.8B interpreters as in Table 2 (only the compiler is swapped, from Qwen3-4B-Instruct to Qwen3-VL-4B)."}

:::

7. Ablations

Section Summary: Experiments on the LoRA mapper showed that several more elaborate designs underperformed the simplest version, which pools prefix tokens, passes them through a shallow network, and produces mixing weights over a shared basis. Removing the compiler entirely and comparing against fixed LoRAs or full fine-tuning of the same base model caused accuracy to drop sharply, confirming that the performance gain stems specifically from the compiler-generated adapters. Additional architecture checks appear in an appendix.

Architectural variants of the LoRA mapper.

We tried several variants of the LoRA mapper that, on paper, are strictly more expressive than the default (mean-pool over prefix tokens, shallow trunk, shared bases). Each made things worse. Table 4 reports accuracy across these variants. The simplest design of mean-pooling the prefix-token hidden states into one vector, running a single residual MLP, and projecting to mixing coefficients over a shared basis set, is the strongest. We do not have a clean theoretical explanation for this; we report the finding so that future work need not rediscover it.

\begin{tabular}{@lc@}
\toprule
Mapper variant & Accuracy \\
\midrule
\textbf{Default} ($r{=}64$, $N{=}64$, shared bases) & \textbf{0.6223} \\
Per-position aggregation & 0.5598 \\
Per-position $+$ per-layer bases & 0.5559 \\
Per-layer bases (only) & 0.6028 \\
LoRA $+$ prefix-tuning (both pathways) & 0.6033 \\
\bottomrule
\end{tabular}

\begin{tabular}{@lc@}
\toprule
Method (0.6B base) & Accuracy \\
\midrule
Fixed LoRA $r{=}18$ & 0.4236 \\
Fixed LoRA $r{=}64$ & 0.5210 \\
Fixed LoRA $r{=}128$ & 0.5159 \\
Full fine-tuning & 0.5840 \\
\textbf{PAW (Qwen3 0.6B)} & \textbf{0.7378} \\
\bottomrule
\end{tabular}

Compiler vs. no compiler.

Table 5 compares PAW with three same-base "no compiler" baselines on FuzzyBench: full fine-tuning of the 0.6B Qwen3 interpreter, and per-task fixed LoRAs at three ranks. The same data, the same base model, the same training budget; only the compiler is removed. PAW exceeds full fine-tuning by 15.4 percentage points and the strongest fixed LoRA by 21.7 points, showing that the gain comes specifically from compiler-generated LoRA.

Other ablations.

Additional ablations on model architecture decisions can be found at Appendix H.

8. Robustness to Noisy Specifications

Section Summary: PAW remains fairly robust when given realistic but noisy specifications containing typos, grammar mistakes, ambiguity, or other flaws, degrading only a few percentage points even under heavy combined noise. The system owes this tolerance to its two-stage design: a larger compiler model first converts the imperfect specification into a clean, structured pseudo-program, shielding the smaller interpreter from the original errors. Experiments confirm the effect, showing substantially larger accuracy drops on noisy inputs when the interpreter is forced to work directly from the raw specification instead.

Real specifications written by developers are noisy: they contain typos, ambiguity, and grammar errors. We evaluate PAW on noise-perturbed versions of the test_clean specifications across seven axes (typos, grammar, ambiguity, formatting, all-noise combined, terse, paraphrase).

Table 6 reports the robustness results. PAW degrades only slightly under heavy noise — but why? We hypothesise that this robustness is mediated by the discrete pseudo-program: the 4B compiler converts the noisy spec into a clean restatement before the small interpreter ever sees it. To test the hypothesis, we trained a variant that bypasses the pseudo-program and feeds the raw spec $s$ directly to the interpreter. The result (Table 7) confirms the hypothesis. On clean inputs, feeding the pseudo-program rather than the raw spec is only 1.6 points better. On heavy-typo specifications, however, the gap widens to 4.5 points. The compiler, a 4B LM whose entire job is to read fuzzy specifications and emit a clean restatement, effectively denoises the input that the small interpreter sees, which is why PAW degrades little when the original specification is corrupted.

: Table 6: Robustness to noise. The 8-axis variants modify the spec but leave the input unchanged. PAW degrades only slightly even under combined heavy noise.

	clean	typos	grammar	ambiguity	formatting	all-noise	terse	paraphrase
PAW Qwen3 0.6B (e2)	0.6692	0.6621	0.6731	0.6511	0.6526	0.6326	0.6499	0.6614
drop from clean	–	-0.7%	+0.4%	-1.8%	-1.7%	-3.7%	-1.9%	-0.8%

: Table 7: The pseudo-program protects the interpreter from noisy specifications. On heavy-typo specifications, feeding raw spec is 4.5 points worse than feeding the pseudo-program to the interpreter.

Interpreter input	Accuracy (clean)	Accuracy (heavy typos)
Pseudo-program (default)	0.6443	0.6108
Raw spec	0.6285	0.5662

9. Local Execution

Section Summary: PAW programs are packaged as single downloadable files that a small Python or JavaScript library can load and run entirely on a user’s device after the initial download, eliminating any further calls to external servers. To keep the footprint small, the shared model and each program’s adapter are quantized to compact GGUF formats that retain nearly the same accuracy while achieving usable speeds, such as roughly 32 tokens per second on a MacBook M3. The approach has already been demonstrated in practical settings including local log monitoring, intent-driven web navigation, semantic search, tool routing, and lightweight multilingual games.

Beyond benchmarks, to make PAW practical to use, we built a developer interface.

Developer interface.

A PAW program is a single file that can be downloaded, cached, and called via a small Python or JavaScript API. Figure 4 shows a complete minimal Python example: paw.compile(prompt) sends a specification to a compiler service and returns a serializable program object; paw.function(id_or_path) loads a compiled program and exposes it as a Python callable. After the first download, all execution happens locally with no external API calls.

**Figure 4:** **Developer interface.** *Left*: the compiler translates a natural-language specification into a neural program. *Right*: the interpreter loads this program and exposes it as a local function.

Quantization without measurable accuracy loss.

On-device execution requires a small footprint. We quantize both the shared interpreter base and each per-program LoRA adapter to GGUF formats compatible with llama.cpp ([13]). Table 8 reports our quantization findings on the 0.6B Qwen3 interpreter, validated at a 4096-example test subset: a 4-bit base ($Q4_K_M$, $\sim$ 484 MB) plus a $Q4_0$ LoRA adapter ($\sim$ 23 MB per program) loses only 1.3 points relative to bf16, and a $Q6_K$ base plus $Q4_0$ adapter is statistically indistinguishable from bf16.

\begin{tabular}{@lccc@}
\toprule
Configuration & Base size & Adapter size & Accuracy \\
\midrule
PyTorch bf16 (no quantization) & 1515 MB & -- & 0.6580 \\
fp16 base + fp32 LoRA & 1509 MB & 162 MB & 0.6594 \\
$Q8\_0$ base + $Q4\_0$ LoRA & 805 MB & 23 MB & 0.6567 \\
$Q6\_K$ base + $Q4\_0$ LoRA & 623 MB & 23 MB & 0.6575 \\
$Q5\_K\_M$ base + $Q4\_0$ LoRA & 551 MB & 23 MB & 0.6477 \\
$Q4\_K\_M$ base + $Q4\_0$ LoRA & 484 MB & 23 MB & 0.6453 \\
$\textsc{IQ4\_XS}$ base + $Q4\_0$ LoRA & 430 MB & 23 MB & 0.6462 \\
\bottomrule
\end{tabular}

Latency on a MacBook M3.

On a MacBook M3 with Metal acceleration, the $Q5_K_M$ base + $Q4_0$ adapter runs at 31.6 tokens/s with a 0.48 s cold load. Full per-quant tables for GPT-2 124M and Qwen3.5 0.8B are in Appendix K.

Case studies.

We applied PAW to five use cases. Event-driven log monitoring replaces the naive wait-based terminal watching in Cursor with a local classifier that fires only on the lines that matter. Intent-based site navigation provides a natural-language quick-find for a website without an LLM API call per request. Semantic search reranking adds intent-aware fuzzy search to an existing keyword index, again without putting an LLM in the request path. For tool calling, a 10-PAW-function pipeline scores 93% on $\textsc{ToolCall-15}$, capturing tool-routing behavior usually reserved for much larger models. The multilingual word-guessing game (Alien-Taboo) is a fuzzy interactive game in which each player turn is served by a 0.6B PAW interpreter on a small server, with one PAW program per language; the LLM is invoked only at compile time, which is what makes a game of this kind economical to host. Full details are in Appendix M.

10. Related Work

Section Summary: This section surveys prior research on hypernetworks that generate model parameters or adapters from textual descriptions or examples, along with methods for parameter-efficient fine-tuning such as LoRA and adapters, the creation of large synthetic instruction datasets, distillation of LLM behavior into programs or smaller models, and neural representations of programs. It contrasts these approaches with PAW, noting differences in how PAW produces hybrid discrete-continuous programs from fuzzy task specifications via a dedicated compiler rather than through per-task optimization or purely textual outputs. The review also highlights distinctions in training data sources and the resulting compiled artifacts being versioned and shareable.

Hypernetworks.

Hypernetworks ([14]) generate the weights of a target network from a small embedding, originally for vision and language modeling; subsequent work used them for continual learning ([15]), multi-task NLP via shared hypernetworks across tasks and layers ([16]), and as parameter-efficient adapters in their own right ([17]). The text-conditioned subfamily relevant to PAW maps a natural-language task description to PEFT parameters in a single forward pass: Hypter ([18]) generates BART-Large adapters from descriptions; HINT ([19]) generates prefix-and-adapter modules from instructions to amortize per-query encoding cost; HyperTuning ([20]) introduces a T5-based hypermodel that emits soft prefixes or LoRA parameters from few-shot examples; Text-to-LoRA ([21]) maps textual task descriptions to LoRAs distilled from pre-trained adapters; Generative Adapter ([22]) produces task-specific adapters from a single forward pass over context; HyperSteer ([23]) extends the idea to activation steering; Gist ([24]) compresses prompts into a few prefix tokens via attention-mask training; and MEND ([25]) distills demonstrations into vectors via two-stage meta-distillation. The most recent work closest to PAW maps natural-language contexts to LoRA in a single forward pass: SHINE ([26]) as a scalable in-context hypernetwork; HypeLoRA ([27]) for calibrated PEFT generation with structural coupling across layers; Doc-to-LoRA ([28]), which meta-learns to internalise a document into a LoRA adapter that the base model can then query without re-consuming the context; and Latent Context Compilation ([29]), which explicitly frames a LoRA module as a compiler that distills long context into compact portable buffer tokens. LoRA composition methods such as LoraHub ([30]) share basis sets across tasks, parallel to our shared-basis LoRA mapper. Compared with these, PAW (a) emits a hybrid (discrete pseudo-program $+$ continuous PEFT) program rather than a continuous-only adapter; (b) is trained on programmer-style fuzzy-function specifications (FuzzyBench-10M's 800+ task families) rather than on QA contexts or distilled per-task adapters; and (c) targets a developer-facing API where the compiled program is a versioned, distributable software artifact.

Parameter-efficient fine-tuning.

The PEFT building blocks our compiler emits are well-studied. Adapters ([31, 32]) insert small trainable bottlenecks into a frozen backbone; prefix-tuning ([4]) prepends learned key–value pairs to attention; prompt tuning ([33, 34]) learns soft input embeddings; LoRA ([35]) learns low-rank decomposed updates to the linear projections of attention and MLP layers; AdaLoRA ([36]) dynamically allocates rank budgets across layers; DoRA ([37]) decomposes pre-trained weights into magnitude and direction and applies LoRA only to the direction component, closing the gap to full fine-tuning; QLoRA ([38]) combines quantization with LoRA for memory-efficient fine-tuning. PAW differs in that the PEFT module is generated per example by a separate compiler from a textual specification, rather than learned per task by gradient descent on the target task. T-Few ([39]) argues that PEFT can outperform in-context learning at lower deployment cost — a related framing, with the difference that T-Few learns its PEFT per task while we generate it from a description.

Synthetic instruction-data generation.

FuzzyBench-10M is generated by an LLM (gpt-5.2) and follows the methodological precedent of LLM-generated instruction datasets. Self-Instruct ([40]) prompts a strong LLM to generate diverse instruction-input-output triples that are then used to fine-tune a smaller model. Unnatural Instructions ([41]) similarly generates instructions automatically. Textbooks Are All You Need ([42]) argues for synthetic-textbook-style training data for small-model pre-training. Magpie ([43]) self-synthesises 4M alignment instances from an aligned LLM with no seed prompts. FuzzyBench-10M differs in that the data-generating pipeline is task-class-specific (29 thematic versions covering categories developers actually encounter, rather than open-ended prompts), and we explicitly construct a verified test split (Section 5) where two strong LLMs must agree on the output to filter ambiguous targets. Recent small-model technical reports ([44, 45]) similarly emphasise high-quality synthetic data as the primary lever for closing capability gaps with frontier models at small scales.

Model distillation.

ALCHEmist ([5]) distills labelling logic from LLMs into Python programs that run on a standard interpreter. PAW shares the motivation of amortizing LLM usage but compiles directly into neural weights instead of textual code, which lets it implement fuzzy functions that resist symbolic encoding. Binder ([46]) translates a task input into a SQL/Python program with embedded LM API calls; PAW differs in that the program is the weights themselves, not a piece of text containing API calls. Distilling Step-by-Step ([47]) distills LLM reasoning into smaller fine-tuned models with rationales as auxiliary supervision; PAW shares the goal of replacing large-LM inference with small-model inference, but achieves it via per-task compile rather than per-task fine-tuning.

Neural programs.

Representing programs in neural networks is a long-standing direction ([48, 49]), with some work compiling formal code into network weights ([50, 51]). PAW differs in its training and use model: programs are not learned per task but produced on demand by a single compiler, then executed on a fixed interpreter and freely shared. A recent trend argues for replacing API LLMs with small, locally executed models ([3, 44, 45]); PAW is one realization. The crucial difference between PAW and "just use a small LLM" is that the small model's behaviour is configured per fuzzy function by a compiler rather than baked into a fine-tune. Practical on-device deployment of small models has been driven by post-training quantization (GPTQ ([52]), AWQ ([53]), QLoRA's quantization-aware finetuning ([38])) and lightweight inference runtimes (llama.cpp ([13]), in-browser via wllama ([54])); we use these directly. Most recent small-LM technical reports ([44, 45]) also adopt LoRA-flavoured PEFT for multimodal extensions and downstream adaptation, complementing the developer-API direction PAW pursues.

11. Conclusion

Section Summary: The authors introduce Program-as-Weights, a method that compiles flexible functions once into tiny neural networks run by a fixed small interpreter. Tests on their benchmark show this setup matches the accuracy of much larger models while using roughly 50 times less memory and running efficiently on a laptop. The same idea extends to image tasks by changing only the compiler, supporting a future where big models build tools and small ones run them locally.

We introduced Program-as-Weights, a programming paradigm in which a fuzzy function is compiled once into a small neural binary and executed locally on a fixed interpreter. On FuzzyBench, a 0.6B-parameter interpreter executing PAW programs matches Qwen3-32B prompting at $\sim$ 50 $\times$ less inference memory and runs at 30 tok/s on a MacBook M3 with quantized GGUF; we illustrate the paradigm through five case studies. The same abstraction extends to image-conditioned fuzzy tasks by swapping only the compiler for a vision-language model. We hope Program-as-Weights contributes to a future in which small LMs serve as the runtime ([3]), where large models compile and small models execute, and the role of foundation models shifts from per-input problem solver to per-function tool builder.

Acknowledgments

We thank Sasha Rush for his guidance and contributions to the earlier project that laid the foundation for this work. We also thank Saarang Agarwal, Austin Dong, Mohammad Jaffer Iqbal, Bihui Jin, Yinxi Li, Jiale Amber Wang, and the anonymous reviewers for their valuable comments and feedback.

This research was supported by a Starter Grant from the University of Waterloo and by the Natural Sciences and Engineering Research Council of Canada (NSERC) under grant numbers RGPIN-2024-04909 and RGPIN-2024-05178. Computational resources were provided by Compute Ontario (computeontario.ca) and the Digital Research Alliance of Canada (alliancecan.ca). We also thank OpenAI's Research Access Program for providing API credits. Wentao Zhang was supported in part by these sources and by the Dr. Derick Wood Graduate Scholarship, generously funded by Ms. Mary Chen.

Appendix

Section Summary: The appendix provides supporting details on tools and resources for the PAW system. It covers a hosted web interface that lets users compile natural-language fuzzy specifications into runnable programs, test them interactively, and download them for offline local use. It also describes the prompts used to build the FuzzyBench dataset, the different prompt styles for the compiler and interpreter, and supporting materials for experiments extending the system to image inputs.

A. Web Interface for PAW Compilation

We provide a hosted web interface that accepts a fuzzy specification, compiles it, lets the user test it interactively, and exports the compiled program as either a serialized weight file or a program identifier that can be loaded through the Python API. The three steps of the workflow are illustrated in Figure 5, Figure 6, and Figure 7. Compilation runs on a GPU-backed server so that users do not need to provision GPUs locally; once downloaded, the compiled program runs entirely offline on the local interpreter.

**Figure 5:** **Step 1: Compile a program from natural language.** The user specifies a fuzzy function in natural language. Image inputs are also supported.

**Figure 6:** **Step 2: Interactively test the compiled program.** Users can provide test inputs and inspect the corresponding outputs, enabling rapid validation and refinement before download.

**Figure 7:** **Step 3: Execute the program locally via Python.** Once compiled, the program can be loaded and invoked through a simple Python API; subsequent execution requires no internet access.

B. FuzzyBench Construction Prompts

Figure 8, Figure 9, and Figure 10 show the prompts used to generate the natural-language specifications. Half of the specifications are generated without exemplar examples (Figure 9) and half with examples (Figure 10); we found this mix to produce more diverse specifications than either style alone. Figure 11 and Figure 12 show the prompts used to generate input–output examples conditioned on a specification.

**Figure 8:** System prompt for generating specifications.

**Figure 9:** User prompt for generating specifications (no exemplar examples).

**Figure 10:** User prompt for generating specifications, with exemplar input/output pairs.

**Figure 11:** System prompt for generating input/output examples given a specification.

**Figure 12:** User prompt for generating input/output examples given a specification.

C. Compiler and Interpreter Prompts

We use two compiler prompt styles in this paper: minimal, which is a single [SPEC]…[END_SPEC] [PSEUDO_PROGRAM] wrapper, and examples, which produces task-description-plus-examples pseudo-programs. The examples style is used by the off-the-shelf compiler reference (Qwen3-4B-Instruct-2507) when generating reference rollouts at the start of training; the minimal style is what the trained PAW compiler uses at inference time. The interpreter uses a single minimal prompt that simply concatenates the pseudo-program with the input.

**Figure 13:** Compiler prompt, `minimal` style. Used by the trained PAW compiler at inference.

**Figure 14:** Compiler prompt, `examples` style. Used by the off-the-shelf reference compiler (Qwen3-4B-Instruct-2507) to generate the rollouts used during training.

**Figure 15:** Interpreter prompt, `minimal` style.

D. Image Processing

This appendix collects the materials supporting the multimodal generalization experiments in Table 3: the compiler and interpreter prompts used at compile and inference time (below), and a component-decomposition ablation of the prefix-tuning precursor on the same six image tasks (Appendix D.1). Recall that the image-task PAW pipeline replaces only the compiler base (Qwen3-4B-Instruct $\to$ Qwen3-VL-4B); the device-resident interpreter is the same Qwen3 0.6B used for text fuzzy functions, and image conditioning is fully encoded in the per-example PEFT module emitted by the VL compiler.

**Figure 16:** Compiler prompt for image-conditioned specifications.

**Figure 17:** Interpreter prompt for image-conditioned specifications.

D.1 Component decomposition of image-task PAW (prefix-tuning era)

Table 9 decomposes the prefix-tuning precursor PAW (Section 3.3) into its discrete and continuous components on the same six image tasks reported in Table 3. "Discrete pseudo only" uses the REINFORCE-trained compiler from the early prototype to emit only a pseudo-program, with no continuous PEFT injected; the small interpreter then runs on the pseudo-program alone. "Continuous KV-cache only" injects a per-example prefix-tuning KV cache from the compiler hidden states but feeds the interpreter the raw spec (no discrete pseudo-program). The full "PAW prefix-tuning" row is the same as in Table 3.

: Table 9: Image-task component decomposition (prefix-tuning precursor). EM/accuracy on the six image tasks. Adding the discrete pseudo-program helps on classification-style tasks (Circuit, Chemical, TextVQA) but hurts on long-form structured generation (Im2LaTeX, Im2SMILES), where "Continuous KV-cache only" is the strongest variant.

Variant	Circuit	Chemical	Music	Im2SMILES	Im2LaTeX	TextVQA
Discrete pseudo only (REINFORCE)	0.009	0.004	0.007	0.041	0.267	0.025
Continuous KV-cache only (no pseudo)	0.181	0.364	0.493	0.234	0.471	0.439
PAW prefix-tuning (both)	0.241	0.365	0.525	0.175	0.391	0.612

The cross-task pattern is consistent: when the output is a short phrase (Circuit/Chemical/Music understanding, TextVQA short-answer), the discrete pseudo-program is a strong inductive bias and adds 5–40 EM points on top of the continuous-only variant. When the output is a long structured sequence (Im2SMILES, Im2LaTeX), the discrete pseudo-program's input/output examples appear to crowd the small interpreter's context budget, and removing the pseudo-program returns 6–8 EM points. We read this as suggesting that future PEFT instantiations of PAW for image-to-markup-style tasks may want to either drop the pseudo-program at deployment time or re-design its content to be lighter (e.g., a paraphrase only, no examples).

E. Prefix-tuning Precursor Architecture

This appendix supplements Section 3.3 with a visual companion to the prefix-tuning precursor architecture, parallel to Figure 2 for the LoRA instantiation. The figure (originally the ICML-version overview of PAW) shows the high-level "compile / interpret" rhythm of the precursor: the compiler emits a compact KV prefix that constitutes the compiled program, and a frozen interpreter executes it locally as a callable function.

$**Figure 18:** **Prefix-tuning precursor architecture (Section 3.3).** *(a) Compile.* The user describes a fuzzy function (e.g., "extract the final answer"); the trained prefix compiler reads the description plus a handful of representative I/O examples and produces a per-example KV prefix — the "neural binary" that constitutes the compiled program. *(b) Interpret.* A small frozen interpreter loads the compiled KV prefix into its attention cache at every layer and processes user inputs locally as a callable function, in the manner of standard prefix-tuning ([4]). This is the prefix-tuning instantiation of the same compiler–interpreter abstraction depicted in Figure 2; only the mapping from compiler hidden states to per-example PEFT module differs (KV cache here, LoRA in Section 3.2). *Note*: our prefix-tuning precursor's exact training-time input format follows the same $[s \mid p_{\text{discrete}} \mid \texttt{EOS} \mid \tau_{1{:}T}]$ structure as the LoRA instantiation.$

F. FuzzyBench-10M Dataset Versions

FuzzyBench is built incrementally over 29 thematic versions, each adding 100K–500K examples covering a new family of fuzzy tasks. Table 10 summarizes the per-version size and the categories added at each stage. The full per-version category list (over 800 sub-categories) and the spec-generation commands used to create each batch are released alongside this paper.

\begin{tabular}{@rlr>{\raggedright\arraybackslash}p{0.77\linewidth}@}
\toprule
v & Size & Specs & Categories added (theme) \\
\midrule
1 & 2.50M & 81{,}920 & Core text processing (parsing, classification, NER, coref, sentiment, etc.; 277 base categories) \\
2 & 2.70M & +24{,}576 & New categories + freeform (data filtering, diff parsing, cron, resume parsing, \ldots) \\
3 & 3.00M & +32{,}768 & Fuzzy/soft matching (fuzzy search, approximate string match, phonetic, entity resolution) \\
4 & 3.25M & +32{,}768 & Format repair (JSON/XML/CSV/YAML/SQL repair, schema conformance, encoding repair) \\
5 & 3.50M & +32{,}768 & Natural-language commands (NL $\to$ shell/jq/awk/git/curl/find, command flag inference) \\
6 & 3.75M & +32{,}768 & Agentic/tool use (tool call generation, tool selection, schema generation) \\
7 & 4.00M & +32{,}768 & Custom classification/filtering (criteria-based, log filtering, salience, anomaly) \\
8 & 4.25M & +32{,}768 & File/code semantic classification (filepath classification, code purpose, commit message) \\
9 & 4.50M & +32{,}768 & DSL/code cleanup (NL-to-DSL, comment stripping, dialogue extraction, build error interp.) \\
10 & 4.75M & +32{,}768 & Log monitoring (importance detection, streaming anomaly, alert evaluation) \\
11 & 5.00M & +32{,}768 & NL constraint validation (constraint formalization, password policy, schema synthesis) \\
12 & 5.25M & +32{,}768 & Constraint satisfaction (boolean SAT, type checking, dependency order, API contracts) \\
13 & 5.50M & +32{,}768 & Operation safety/secrets (command risk classification, secret detection, redaction) \\
14 & 5.75M & +32{,}768 & Reversibility/output masking/token reduction (traceback distillation, output summarization) \\
15 & 6.00M & +32{,}768 & Auto-completion (context-aware, history-based, cwd-aware, domain-specific) \\
16 & 6.25M & +32{,}768 & Pseudo-program execution (execution trace, result prediction, NL-to-Python) \\
17 & 6.50M & +32{,}768 & Chemistry properties + domain commonsense (SMILES, reaction extraction, service-time est.) \\
18 & 7.00M & +65{,}536 & Domain-knowledge plugins + counterfactual reasoning (30 categories spanning STEM/health/social) \\
19 & 7.25M & +32{,}768 & Browser semantic matching + transcript cleanup + narrow translation \\
20 & 7.50M & +32{,}768 & Agent watchdog / wait interruption (process state classification, completion detection) \\
21 & 7.75M & +32{,}768 & AI text detection / authorship / style analysis \\
22 & 8.00M & +32{,}768 & Semantic search + explicit AI detection (relevance, query reformulation, snippet extraction) \\
23 & 8.25M & +32{,}768 & spaCy superset / custom NLP pipeline (custom NER, span labeling, dependency parsing) \\
24 & 8.50M & +32{,}768 & HTML understanding / browser intelligence (ad detection, boilerplate removal) \\
25 & 8.75M & +32{,}768 & Intent-based HTML + semantic equivalence / deduplication \\
26 & 9.00M & +32{,}768 & Agent tool store / streaming intelligence \\
27 & 9.25M & +32{,}768 & Intent-to-navigation / settings discovery \\
28 & 9.75M & +32{,}768 & Document-grounded QA / spec-based classification \\
29 & 10.0M & +32{,}768 & Smart search pipeline completion (keyword extraction, term weighting, reranking) \\
\bottomrule
\end{tabular}

G. Training Configuration

We use the following configuration for the Qwen3 0.6B and Qwen3.5 0.8B PAW runs (the GPT-2 124M run uses the same hyperparameters but with the GPT-2-specific target modules c_attn c_proj c_fc since GPT-2 fuses Q/K/V into a single projection):

Pseudo compiler $C_p$ (untrained). Qwen/Qwen3-4B-Instruct-2507, prompted with the examples template (Appendix C). Pseudo-programs for the entire 10M-example training set are pre-generated once with vLLM, indexed by spec, and stored in a JSONL file. During training, $p_{\text{discrete}}$ is read from disk for each example, never sampled from a live model.
LoRA compiler $C_L$ (trained). Qwen3-4B-Instruct-2507, fully unfrozen, learning rate 2 x 10^-5, bf16 parameters, gradient checkpointing on. Input format is the minimal spec wrapper followed by the pseudo from $C_p$ and a fixed sequence of $T = 64$ learned "prefix" tokens.
LoRA mapper. Kept in fp32 for numerical stability. Mean-pool aggregation, single residual MLP trunk, shared bases. Rank $r = 64$, $N = 64$ bases per module type, target modules q_proj k_proj v_proj o_proj gate_proj up_proj down_proj.
Interpreter. Frozen. Default Qwen/Qwen3-0.6B.
Training loop. 3 epochs over the 10M-example dataset; batch size 16, gradient accumulation 3 (effective batch 48); max $C_L$ sequence length 1280, max interpreter sequence length 1024. The loss is the negative mean-token log-likelihood of the target $y$ under the frozen interpreter Equation (4); no policy-gradient term, no group baseline.
Hardware. Single B300 (early stages) or 8 $\times$ H200 (later stages). The 0.6B Qwen3 run completed three epochs in $\sim$ 72 hours of training on 3 GPUs.

We use AdamW with the default PyTorch settings; no warmup; no LR schedule.

H. Additional Ablations

Table 11 reproduces the additional ablations summarized in Section 7 with their full numbers. Several rows are taken from earlier KV-prefix runs (where indicated) and serve to anchor the architectural transition described in Section 7.

\begin{tabular}{@lcc@}
\toprule
Ablation axis & Variant & EM (test\_clean) \\
\midrule
\multirow{3}{*}{Continuous component}
   {} & KV-prefix (epoch 2) & 0.5044 \\
   {} & LoRA $r{=}18$ (epoch 2) & 0.5652 \\
   {} & LoRA $r{=}64$, 7 modules (epoch 2) & \textbf{0.6572} \\
\midrule
\multirow{3}{*}{Compiler input for LoRA}
   {} & Spec only & 0.6411 \\
   {} & Pseudo only & 0.6165 \\
   {} & \textbf{Spec + pseudo (default)} & \textbf{0.6443} \\
\midrule
\multirow{2}{*}{Discrete-and-LoRA coupling}
   {} & Shared (one head) & 0.6350 \\
   {} & \textbf{Separate (default)} & \textbf{0.6443} \\
\midrule
\multirow{2}{*}{LoRA-mapper input norm}
   {} & With LayerNorm & 0.6377 \\
   {} & \textbf{Without (default)} & \textbf{0.6443} \\
\midrule
\multirow{2}{*}{Interpreter initialization}
   {} & Start from finetuned interpreter & 0.6038 \\
   {} & \textbf{Start from base (default)} & \textbf{0.6223} \\
\bottomrule
\end{tabular}

I. Compiler Scaling and Freezing (Inconclusive)

Table 12 reports test_clean exact match across compiler sizes (0.6B–32B), in both frozen and unfrozen variants, paired with a Qwen3.5 0.8B interpreter and otherwise identical training. We label this study inconclusive because the pattern is non-monotonic in ways we cannot yet attribute to a single cause: unfreezing the 4B compiler beats frozen 32B, and gpt-oss-20B as a frozen compiler underperforms a frozen Qwen3-4B-Instruct-2507. We have not run a controlled study at large data scales because each combination is expensive; we report the numbers descriptively rather than draw scaling claims.

: Table 12: Inconclusive compiler-scaling table. EM on test_clean (Qwen3.5 0.8B interpreter, 0.6M training examples, epoch 1). Reported as exploratory data.

Compiler	Frozen?	test_clean
Qwen3 4B	No	0.6455
Qwen3 4B	Yes	0.6128
Qwen3 4B + input norm	Yes	0.6228
Qwen3 14B	No	0.6257
Qwen3 32B	Yes	0.6174
gpt-oss-20B	Yes	0.5823
Qwen3.5 4B (hybrid)	Yes	0.5046

J. Per-Noise-Type Robustness

Table 13 reports the full noise-robustness numbers across light/medium/heavy intensity for all eight noise axes, for the Qwen3 0.6B interpreter at epoch 2.

\begin{tabular}{@lccc@}
\toprule
Noise axis & Light & Medium & Heavy \\
\midrule
Typos & 0.6709 & 0.6685 & 0.6621 \\
Grammar & 0.6687 & 0.6672 & 0.6731 \\
Ambiguity & 0.6731 & 0.6628 & 0.6511 \\
Formatting & 0.6702 & 0.6575 & 0.6526 \\
All noise (combined) & 0.6670 & 0.6650 & 0.6326 \\
\midrule
Terse (heavy only) & -- & -- & 0.6499 \\
Casual (heavy only) & -- & -- & 0.6675 \\
Paraphrase (heavy only) & -- & -- & 0.6614 \\
\bottomrule
\end{tabular}

K. Full Quantization Tables

Table 14 reports the full per-quant exact-match numbers for Qwen3 0.6B at the 4096-example scale, including the $\textsc{IQ4_XS}$ / $\textsc{IQ4_NL}$ I-quants. Table 15 and Table 16 report the corresponding figures for GPT-2 124M and Qwen3.5 0.8B at 36 examples (36-example numbers are not statistically meaningful below the 0.6B size; full at-scale validation is in progress for those interpreters).

\begin{tabular}{@lcccc@}
\toprule
Base format & bpw & Base size & Adapter size & EM (test\_clean) \\
\midrule
PyTorch bf16 & 16 & 1515 MB & -- & 0.6580 \\
fp16 & 16 & 1509 MB & 162 MB & 0.6594 \\
$Q8\_0$ & 8.5 & 805 MB & 162 MB & 0.6550 \\
$Q6\_K$ & 6.56 & 623 MB & 162 MB & 0.6558 \\
$Q5\_K\_M$ & 5.5 & 551 MB & 162 MB & 0.6499 \\
$Q4\_K\_M$ & 4.8 & 484 MB & 162 MB & 0.6460 \\
$\textsc{IQ4\_XS}$ & 4.25 & 430 MB & 162 MB & 0.6484 \\
$Q4\_K\_S$ & 4.5 & 449 MB & 162 MB & 0.6348 \\
$Q3\_K\_L$ & 3.9 & 416 MB & 162 MB & 0.6055 \\
$Q3\_K\_M$ & 3.5 & 395 MB & 162 MB & 0.5874 \\
\midrule
\multicolumn{5}{l}{$Q4\_0$ adapter (23 MB) instead of fp32 (162 MB):}\\
$Q6\_K$ + $Q4\_0$ & 6.56 & 623 MB & 23 MB & 0.6575 \\
$Q5\_K\_M$ + $Q4\_0$ & 5.5 & 551 MB & 23 MB & 0.6477 \\
$Q4\_K\_M$ + $Q4\_0$ & 4.8 & 484 MB & 23 MB & 0.6453 \\
$\textsc{IQ4\_XS}$ + $Q4\_0$ & 4.25 & 430 MB & 23 MB & 0.6462 \\
\bottomrule
\end{tabular}

\begin{tabular}{@lcccc@}
\toprule
Base format & bpw & Base size & tok/s & EM (36) \\
\midrule
fp16 & 16 & 252 MB & 100.7 & 21/36 \\
$Q8\_0$ & 8.5 & 137 MB & 115.7 & 23/36 \\
$Q6\_K$ & 6.5 & 107 MB & 111.0 & 22/36 \\
$Q5\_K\_M$ & 5.5 & 99 MB & 108.0 & 24/36 \\
$Q4\_K\_M$ & 4.8 & 91 MB & 110.5 & 24/36 \\
$\textsc{IQ4\_NL}$ & 4.5 & 85 MB & 107.1 & 24/36 \\
$\textsc{IQ4\_XS}$ & 4.25 & 82 MB & 115.4 & 24/36 \\
$Q3\_K\_L$ & 3.9 & 88 MB & 116.6 & 23/36 \\
$\textsc{IQ2\_M}$ & 2.7 & 63 MB & 192.5 & 16/36 \\
\bottomrule
\end{tabular}

: Table 16: Qwen3.5 0.8B (Mamba-attention hybrid) quantization sweep (36-example handcrafted set, Q4_0 16 MB LoRA adapter). Q4_K_S and below crash with llama_decode failed (code -3) due to the Mamba-hybrid architecture's incompatibility with aggressive quantization.

Base format	bpw	Base size	tok/s	EM (36)
fp16	16	1517 MB	6.1	30/36
Q8_0	8.5	774 MB	6.4	30/36
Q6_K	6.5	601 MB	6.7	30/36
Q5_K_M	5.5	551 MB	6.5	30/36
Q4_K_M	4.8	505 MB	6.5	31/36
Q4_K_S	4.5	505 MB	–	crash
Q3_K_L and below	–	–	–	crash

L. Qualitative Analysis

We hand-inspected the last 20 training rollouts from each of three interpreters (GPT-2 124M, Qwen3 0.6B, Qwen3.5 0.8B) to characterize where each succeeds and fails. The summary statistics are: GPT-2 12/20 perfect, 0.6B 8/20 perfect, 0.8B 13/20 perfect. The 0.8B's strengths are structured-output generation (JSON, CSV, BibTeX, DOT graphs), classification with multiple candidate labels, pattern matching and transformation, and logical reasoning with explicit cases (cycle detection, exclusivity violations, paraphrase detection). Its failure modes are precise numeric computation (off-by-small-amount unit conversions), character-level position tracking (span start/end indices off by a few), and creative reformulation (synonym replacement that changes meaning). The 0.6B has similar strengths but makes more span-offset errors and struggles with multi-step JSON construction. GPT-2 cannot do multi-step reasoning (sentiment timelines, rubric scoring) and cannot track precise positions, but is strong at pattern matching and classification when the answer space is small. We provide example transcripts for each model in the released supplementary material.

M. Full Case-Study Walkthroughs

Figure 19 sketches the multi-program library that the case studies below populate: each natural-language specification is compiled once into its own neural program, and the resulting programs are served by a single device-resident interpreter at run time. Below we provide longer walkthroughs of the five case studies summarized in Section 9, including the full specifications and the iteration histories.

M.1 Event-driven log monitoring (full walkthrough)

The final specification is:

Classify log lines. Return ONLY one word: ALERT or QUIET.

Input: [step 100] loss=0.05 lr=0.0001
Output: QUIET

Input: [Checkpoint] Saved model at step 1000
Output: ALERT

Input: Traceback (most recent call last):
Output: ALERT

Input: Training complete. Final loss: 0.11
Output: ALERT

The monitoring loop is a simple file-tailing wrapper that truncates each new chunk to fit the 2048-token context window, calls the PAW function, and surfaces ALERT chunks. A separate stall timer covers "no new output for $N$ minutes, " which the classifier cannot detect because it only sees what is written.

M.2 Intent-based site navigation (full walkthrough)

The page-classifier specification is:

Classify the user's intent. Return ONLY a single label.
playground = Create or compile something new
hub = Browse or search existing items
browser = Run something in the browser
docs = Read documentation
settings = Manage account or API keys
none = None of the above (likely a question)

Input: how do I get started
Output: docs

Input: browse community programs
Output: hub

Input: is it free?
Output: none

The full pipeline is five PAW functions in sequence: page classifier, question-type classifier, yes/no answerer, how/what answerer, and answer validator. Each program compiles in seconds and runs in milliseconds. The validator catches answers that are grammatically fine but do not address the question ("yes" to "what is the license?").

M.3 Semantic search reranking (full walkthrough)

The reranker template is:

You are a search matcher. Rate how well the candidate matches the query.
Match all constraints: constraint_types.
If the query excludes something, those candidates are not_relevant.

Query: "{query}"

Return ONLY one of: exact_match, highly_relevant, somewhat_relevant, not_relevant

The reranker is composed with a keyword search over a full-text index: keyword search returns the top 10–20 candidates, the PAW reranker scores each against the query into one of the four buckets (mapped to integer scores 3–0), and the candidates are returned in descending score order.

M.4 Tool calling pipeline (full walkthrough)

The pipeline uses 10 PAW functions: tc15-needs-tool, tc15-tool-router, tc15-impossible-check, tc15-second-tool, plus six parameter-extraction functions (tc15-extract-location, tc15-extract-ticker, tc15-extract-units, tc15-extract-search-query, tc15-extract-person, tc15-extract-translate). Date/time parsing is handled by a regex; OpenAI tool_calls JSON is built by deterministic Python. The proxy server handles multi-turn conversations by tracking which tools have already been called and threading data between steps. The single failed scenario (TC-13, an empty-results retry) was traced to overly aggressive loop-prevention logic in the proxy code, not to a PAW function; we report it as such in the main paper.

M.5 Word-guessing game (full walkthrough)

The English specification is:

You are playing a word-guessing game. The user describes a common English
word in their own words without saying the word itself. Your job is to
guess the word from the description.

Return ONLY the single word being described. Lowercase. No punctuation,
no explanation, no extra words.

Input: furry animal that meows and purrs
Output: cat

Input: yellow curved fruit, monkeys like it
Output: banana

Input: thing you use to unlock a door, metal, small, has teeth
Output: key

... (12 more)

The Chinese version is the same template translated to Mandarin with $\sim$ 20 hint $\to$ word examples. The hard part of the project, by far, was vetting a 361-word bank: a candidate-generation step that produced $\sim$ 4000 candidate words from gpt-5.4 across 40 themes; a simulated-playthrough step that prompted gpt-5.4-mini to play the role of a human describer and routed those descriptions through the deployed PAW program, keeping words solved within $\le 8$ rounds across $\ge 4$ of 5 random-seed trials; a commonness filter (Zipf $\ge 5.0$ on the wordfreq corpus); and a final manual pass.

N. Limitations

Coupled compiler–interpreter pairs.

A trained PAW system pairs one specific compiler with one specific interpreter family. Switching the interpreter (e.g., from Qwen3 0.6B to Qwen3.5 0.8B) requires retraining the compiler. This is a property shared with most parameter-generation methods ([21, 22]); PAW's main generalization axes are cross-task (one trained pair handles unboundedly many fuzzy specifications) and cross-modality (only the compiler is replaced, see Table 3).

Interpretability of the compiled program.

Once compiled, the only human-inspectable part of a PAW program is the discrete pseudo-program. The continuous PEFT component (LoRA or KV cache) is opaque. We see this as analogous to the inspectability gap between source code and compiled binaries; concrete tools for inspecting and debugging neural binaries are an open direction.

Single-step fuzzy functions.

All evaluations in this paper are single-step (one input, one output). Multi-step / long-horizon reasoning is not yet validated; in principle, PAW functions can be composed in user code (as in the case studies of Section 9), but learning a compiler that produces compositional programs is left for future work.

Synthetic training data.

FuzzyBench is generated by an LLM (gpt-5.2). The compiler we train is Qwen3-4B-Instruct-2507, a different model family, so the data is not aligned with the compiler's own biases; the test specifications are held out and verified by an independent strong model. Nonetheless, broader external validation is in progress; the five case studies in Section 9 are an initial step.

Task-dependent best PEFT.

We observe in Section 3.3, Table 3, and Appendix D.1 that the best PEFT instantiation is task-dependent: LoRA is strongest on text and on diagram-style image classification, while prefix-tuning (KV cache) is stronger on long-form structured image-to-markup generation. We do not yet have a principled rule for predicting which PEFT to deploy for a new task class without empirical comparison.

O. Broader Impacts

PAW shifts foundation-model use from per-input cloud invocation to per-function compilation followed by local execution. Positive impacts include reduced API dependency and cost (functions run on a $\sim$ 500 MB device-resident interpreter instead of round-tripping to a cloud LLM), reproducibility (a compiled program is a single versioned file), and offline availability (the in-browser path runs with no network). Negative impacts are constrained: the released interpreter is small ($0.6$ B parameters) and is fine-tuned per fuzzy function rather than for open-ended generation, so misuse risks comparable to those of general-purpose LLMs (disinformation generation, fraudulent text at scale) are limited; the training data is fully synthetic and contains no scraped or personal content. We see no direct path from this paradigm to a negative application that requires explicit mitigation, and we do not gate the released artifacts.

References

Section Summary: This references section compiles a numbered list of academic papers, technical reports, and conference proceedings primarily on artificial intelligence topics such as language models, vision-language systems, hypernetworks, and efficient model adaptation methods. The entries range from 2012 to 2025 and include works from sources like arXiv, ACL, NeurIPS, ICLR, and CVPR, each providing author names, titles, publication details, and links for further reading. They collectively support research on scaling AI capabilities through techniques like tuning, adapters, and synthetic data generation.

[1] Rubio Manzano, Clemente (2012). Design and implementation of a fuzzy logic programming language using weak unification. AI Commun.. 25(4). pp. 365–367.

[2] Kim et al. (2023). FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 14397–14413. doi:10.18653/v1/2023.emnlp-main.890. https://aclanthology.org/2023.emnlp-main.890/.

[3] Peter Belcak et al. (2025). Small Language Models are the Future of Agentic AI. https://arxiv.org/abs/2506.02153. arXiv:2506.02153.

[4] Li, Xiang Lisa and Liang, Percy (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 4582–4597. doi:10.18653/v1/2021.acl-long.353. https://aclanthology.org/2021.acl-long.353/.

[5] Huang et al. (2024). The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators. In Advances in Neural Information Processing Systems. pp. 62648–62672. doi:10.52202/079017-2003. https://proceedings.neurips.cc/paper_files/paper/2024/file/72802bef5cf1a3449e909b20c2ae18d5-Paper-Conference.pdf.

[6] Shuai Bai et al. (2025). Qwen3-VL Technical Report. https://arxiv.org/abs/2511.21631. arXiv:2511.21631.

[7] Yang et al. (2025). Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation. arXiv preprint arXiv:2502.14846.

[8] Deitke et al. (2024). Molmo and PixMo: Open weights and open data for state-of-the-art vision-language models. arXiv preprint arXiv:2409.17146.

[9] Yuntian Deng et al. (2017). Image-to-Markup Generation with Coarse-to-Fine Attention. In Proceedings of the 34th International Conference on Machine Learning. pp. 980–989. https://proceedings.mlr.press/v70/deng17a.html.

[10] Yuntian Deng et al. (2023). Markup-to-Image Diffusion Models with Scheduled Sampling. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=81VJDmOE2ol.

[11] Singh et al. (2019). Towards VQA Models That Can Read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8317–8326.

[12] AndesVL Team, OPPO AI Center (2025). AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Model. https://arxiv.org/abs/2510.11496. arXiv:2510.11496.

[13] Gerganov, Georgi and llama.cpp contributors (2023). llama.cpp: LLM inference in C/C++. https://github.com/ggerganov/llama.cpp.

[14] Ha et al. (2017). HyperNetworks. In The 5th International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1609.09106.

[15] Johannes von Oswald et al. (2020). Continual learning with hypernetworks. In International Conference on Learning Representations. https://arxiv.org/abs/1906.00695.

[16] Karimi Mahabadi et al. (2021). Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 565–576. doi:10.18653/v1/2021.acl-long.47. https://aclanthology.org/2021.acl-long.47/.

[17] Karimi Mahabadi et al. (2021). Compacter: Efficient Low-Rank Hypercomplex Adapter Layers. In Advances in Neural Information Processing Systems. https://arxiv.org/abs/2106.04647.

[18] Ye, Qinyuan and Ren, Xiang (2021). Learning to Generate Task-Specific Adapters from Task Description. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). pp. 646–653. doi:10.18653/v1/2021.acl-short.82. https://aclanthology.org/2021.acl-short.82/.

[19] Ivison et al. (2023). HINT: Hypernetwork Instruction Tuning for Efficient Zero- and Few-Shot Generalisation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 11272–11288. doi:10.18653/v1/2023.acl-long.631. https://aclanthology.org/2023.acl-long.631/.

[20] Phang et al. (2023). HyperTuning: Toward Adapting Large Language Models without Back-propagation. In Proceedings of the 40th International Conference on Machine Learning. pp. 27854–27875. https://proceedings.mlr.press/v202/phang23a.html.

[21] Rujikorn Charakorn et al. (2025). Text-to-LoRA: Instant Transformer Adaption. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=zWskCdu3QA.

[22] Tong Chen et al. (2025). Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=bc3sUsS6ck.

[23] Jiuding Sun et al. (2025). HyperSteer: Activation Steering at Scale with Hypernetworks. https://arxiv.org/abs/2506.03292. arXiv:2506.03292.

[24] Mu et al. (2023). Learning to Compress Prompts with Gist Tokens. In Advances in Neural Information Processing Systems. https://proceedings.neurips.cc/paper_files/paper/2023/hash/3d77c6dcc7f143aa2154e7f4d5e22d68-Abstract-Conference.html.

[25] Li et al. (2024). MEND: Meta Demonstration Distillation for Efficient and Effective In-Context Learning. In The Twelfth International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=2Y5kBPtU0o.

[26] Liu et al. (2026). SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass. https://arxiv.org/abs/2602.06358. arXiv:2602.06358.

[27] Trojan, Bartosz and Gębala, Filip (2026). HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning. https://arxiv.org/abs/2603.19278. arXiv:2603.19278.

[28] Charakorn et al. (2026). Doc-to-LoRA: Learning to Instantly Internalize Contexts. https://arxiv.org/abs/2602.15902. arXiv:2602.15902.

[29] Li et al. (2026). Latent Context Compilation: Distilling Long Context into Compact Portable Memory. https://arxiv.org/abs/2602.21221. arXiv:2602.21221.

[30] Huang et al. (2024). LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. In Proceedings of the First Conference on Language Modeling (COLM). https://arxiv.org/abs/2307.13269.

[31] Houlsby et al. (2019). Parameter-Efficient Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML). https://arxiv.org/abs/1902.00751.

[32] Pfeiffer et al. (2021). AdapterFusion: Non-Destructive Task Composition for Transfer Learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL). https://arxiv.org/abs/2005.00247.

[33] Lester et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). https://arxiv.org/abs/2104.08691.

[34] Liu et al. (2022). P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL). https://arxiv.org/abs/2110.07602.

[35] Edward J Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations. https://openreview.net/forum?id=nZeVKeeFYf9.

[36] Zhang et al. (2023). AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. In The Eleventh International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2303.10512.

[37] Liu et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. In Proceedings of the 41st International Conference on Machine Learning (ICML). https://arxiv.org/abs/2402.09353.

[38] Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Information Processing Systems. https://arxiv.org/abs/2305.14314.

[39] Liu et al. (2022). Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. In Advances in Neural Information Processing Systems. https://arxiv.org/abs/2205.05638.

[40] Wang et al. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). https://arxiv.org/abs/2212.10560.

[41] Honovich et al. (2023). Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). https://arxiv.org/abs/2212.09689.

[42] Gunasekar et al. (2023). Textbooks Are All You Need. https://arxiv.org/abs/2306.11644. arXiv:2306.11644.

[43] Xu et al. (2025). Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. In The Thirteenth International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2406.08464.

[44] Abdin et al. (2024). Phi-4 Technical Report. https://arxiv.org/abs/2412.08905. arXiv:2412.08905.

[45] Gemma Team (2025). Gemma 3 Technical Report. https://arxiv.org/abs/2503.19786. arXiv:2503.19786.

[46] Zhoujun Cheng et al. (2023). Binding Language Models in Symbolic Languages. In The Eleventh International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=lH1PV42cbF.

[47] Hsieh et al. (2023). Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. In Findings of the Association for Computational Linguistics: ACL 2023. https://arxiv.org/abs/2305.02301.

[48] Alex Graves et al. (2014). Neural Turing Machines. https://arxiv.org/abs/1410.5401. arXiv:1410.5401.

[49] Scott Reed and Nando de Freitas (2016). Neural Programmer-Interpreters. https://arxiv.org/abs/1511.06279. arXiv:1511.06279.

[50] Gail Weiss et al. (2021). Thinking Like Transformers. https://arxiv.org/abs/2106.06981. arXiv:2106.06981.

[51] Frédéric Gruau et al. (1995). A neural compiler. Theoretical Computer Science. 141(1). pp. 1-52. doi:https://doi.org/10.1016/0304-3975(94)00200-3. https://www.sciencedirect.com/science/article/pii/0304397594002003.

[52] Frantar et al. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. In The Eleventh International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2210.17323.

[53] Lin et al. (2024). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. In Proceedings of Machine Learning and Systems (MLSys). https://arxiv.org/abs/2306.00978.

[54] wllama contributors (2024). wllama: WebAssembly bindings for llama.cpp. https://github.com/ngxson/wllama.