One-sentence Summary

Google DeepMind introduces SkillSmith, an augmented LLM that treats model weights as a modality, synthesizing textual knowledge and prefix-tuning weights to directly generate instruction-steered parametric skill weights, significantly outperforming both text-only and weight-only baselines and unlocking performance gains unattainable by uni-modal adaptations.

Key Contributions

• SkillSmith treats model weights as a native modality, enabling an LLM to synthesize task-specific prefix weights from textual instructions and weight-space inputs.
• The augmented LLM, SkillSmith, acts as a hyper-network that processes prefix weights and textual metadata to directly generate new prefix weights, bridging textual reasoning and parametric adaptation.
• On Composite-SNI, SNI, and MMLU-ProX, SkillSmith outperforms text-only and weight-space-only baselines in zero-shot and data-sparse settings, and provides a stronger initialization for fine-tuning than standard weight merging or random initialization.

Introduction

Large language models are increasingly used as agents that must adapt from past experience to solve complex multi-step tasks. Currently, agent adaptation relies on two separate mechanisms: text-based reasoning (through self-reflection, memory, or prompt generation) and parametric skill libraries built via parameter-efficient fine-tuning. Prior work treats these as orthogonal pursuits, and weight-space merging methods use shallow arithmetic that does not capture semantic relationships between tasks. The authors propose SkillSmith, an augmented LLM that treats weight-space inputs as a native modality. By training on prefix-weights along with textual metadata and task descriptions, SkillSmith directly synthesizes task-specific parametric skills, enabling an agent's textual reasoning to steer compositional weight generation.

Dataset

The authors construct their training and evaluation data from three main sources: a synthetic composite dataset, a subset of Super-Natural Instructions (SNI), and a multilingual benchmark (MMLU-ProX). The data is used to train SkillSmith’s ability to compose capabilities from source tasks into new composite tasks.

Dataset composition and key details

• Composite-SNI (CSNI) – A synthetic dataset of ≈21k composite tasks. It is built by presenting pairs of SNI tasks to Gemini 2.5 Pro and prompting the model to generate a new task that naturally combines skills from both source tasks. The dataset is split into meta-train (≈17k tasks) and meta-eval sets. To dissect generalization, the meta-eval set is stratified into three subsets based on whether the constituent source tasks appear in the meta-train set: Both-Seen, One-Seen, and Neither-Seen.
• Super-Natural Instructions (SNI) – The original SNI benchmark contains 1,616 diverse NLP tasks across 76 categories. The authors select a subset of 875 tasks, each with 1,024 instances. These tasks serve as the source task library (the pool from which source KV caches are trained) and also as target tasks in the “wild” setting. When no ground-truth source mapping exists, a heuristic retrieval pipeline is used to assign each target task a pair of source tasks.
• MMLU-ProX – A multilingual benchmark spanning 14 subject categories and 29 languages from five geographic regions. The authors create a challenging evaluation split by holding out the three lowest-performing languages (according to Gemma 3 4B) and three randomly selected subject categories. The meta-training set thus consists of 26 languages and 11 categories, while the evaluation pool retains all 29 languages and 14 categories. For final evaluation, they sample six language-category pairs: wolof_math, wolof_health, zulu_physics, spanish_law, indonesian_law, and afrikaans_history.

How the data is used

• Training of source task modules: For each source task (e.g., from SNI), the authors train a prefix KV cache using the task’s (input, target) pairs. The input sequence length is randomly sampled from {32, 64, 128} to encourage diversity. The model is optimized over a grid of steps (200, 500, 1000) and learning rates (1e-2 to 1e-4), and the best checkpoint is selected based on validation NLL. Each source task module is accompanied by a “Source Text” – a task description together with 4–16 in-context demonstrations.
• Training SkillSmith: The meta-train split of Composite-SNI is used directly, with ground-truth source task pairs. For the SNI “wild” setting, target tasks are mapped to source task pairs via the retrieval pipeline (semantic retrieval with Gemini Embeddings followed by an LLM selection step using Gemini 2.5 Pro). For every target task, the authors generate a “Combination Text” by prompting Gemini 2.5 Pro to describe the semantic relationships between the source tasks and the target task.
• Evaluation: Composite-SNI’s meta-eval subsets measure how well SkillSmith generalizes to compositions with known, partially known, and unseen source tasks. MMLU-ProX is used solely for evaluation, testing cross-lingual and cross-domain generalization through the held-out language-category pairs.

Processing details

• Source tasks are processed into fixed-length prefix KV caches with randomly sampled lengths (32, 64, 128 tokens) to prevent overfitting.
• Source Text is constructed from a task description plus 4–16 few-shot demonstrations.
• For target tasks without ground-truth source mappings, a two-stage retrieval pipeline is used: first, Gemini Embeddings rank candidate source tasks; second, Gemini 2.5 Pro selects the most contextually relevant pair.
• Combination Text is obtained by prompting Gemini 2.5 Pro to describe the relationships between the selected source tasks and the target task.
• MMLU-ProX is split by both language and category using performance-based and random exclusion to create a challenging evaluation set.

Method

The authors cast the problem as bridging text and parameter modalities to synthesize a new PEFT module for a target task. Each previously encountered task $T_i$Ti​ is stored as a task bundle $b_i = (m_i, w_i)$bi​=(mi​,wi​), where $m_i$mi​ is a trained prefix‑tuning module (a learnable Key‑Value cache that modulates a frozen base model $M_\phi$Mϕ​) and $w_i$wi​ is textual metadata such as instructions, ICL demonstrations, or task reflections. This choice of prefix‑tuning is deliberate: text passages can be directly converted into KV‑caches by a forward pass through $M_\phi$Mϕ​, suggesting that relationships between the parametric and text modalities can be learned.

Given a new task $T_{new}$Tnew​, the goal is to produce $m_{new}$mnew​ by composing a small set of relevant source bundles $\mathcal{T}_{src}[T_{new}]$Tsrc​[Tnew​] and additional descriptive text. Unimodal baselines—either aggregating all text as in‑context examples or arithmetically merging existing weights—fail to fully exploit the complementary information in the two modalities. SkillSmith addresses this with an architecture that treats weight‑space inputs as a native modality.

As shown in the framework diagram, SkillSmith is an augmented pre‑trained language model (the coprocessor) that ingests interleaved sequences of text and adapted KV‑caches. The source bundles $\{b_i\}${bi​} are first projected into the latent space of the coprocessor via an input KV‑Adapter, implemented as an MLP. The projected caches are then serialized into a contiguous sequence using a small set of control tokens, as illustrated below.

The sequence begins with a Preamble Text that describes the composition objective, priming the model’s instruction‑following capabilities. Then, for each source bundle, the text metadata is prepended with a <src_start> token and immediately followed by its adapted parametric module $KV_i'$KVi′​, bounded by <kv_start> and <kv_end>. After all bundles, a Combination Text is appended; this can include descriptions of how the source tasks relate to the target, examples of the target task, or a plain description of the target. Finally, a <gen_start> token marks the beginning of cache synthesis: a fixed‑length sequence of placeholder latent tokens $(z_1, \dots, z_L)$(z1​,…,zL​) is placed, ending with <gen_end>.

The entire constructed sequence is passed through the coprocessor LLM. The KV‑caches corresponding to the placeholder tokens are isolated, stripped of positional information via inverse RoPE de‑rotation, and fed through an Output KV‑Adapter (another MLP) to produce the final synthesised weights $m_{new}$mnew​. This module can be directly plugged into the frozen base model $M_\phi$Mϕ​ to solve $T_{new}$Tnew​.

The authors train SkillSmith end‑to‑end using a meta‑learning objective. A meta‑training set $\mathcal{D}^{train}$Dtrain is bootstrapped from the pre‑constructed library $\mathcal{T}_{src}$Tsrc​. For each target task $T$T in $\mathcal{T}_{src}$Tsrc​, an entry contains a set of source bundles $\mathcal{T}_{src}[T]$Tsrc​[T], combination text $w$w, and example input‑output pairs. The generated module $m_T = \text{SkillSmith}_\theta(\{b_k\}, w)$mT​=SkillSmithθ​({bk​},w) is applied to the frozen base model, and the cross‑entropy loss (or a policy loss for reinforcement learning tasks) on the task data is backpropagated to update only the SkillSmith parameters $\theta$θ, keeping $M_\phi$Mϕ​ fixed:

During training, the output cache length is dynamically sampled from {16, 32, 64, 128} to improve regularization, while evaluation uses a fixed length of 32. The coprocessor LLM is initialised with Gemma 3 4B.

For real‑world deployment where ground‑truth source task mappings are unknown, a two‑stage heuristic retrieval pipeline is introduced. First, a semantic retriever based on Gemini Embeddings ranks all source tasks by relevance to $T_{new}$Tnew​ and groups them into candidate pairs. Second, an LLM selector (Gemini 2.5 Pro) chooses the single most contextually relevant pair, which becomes $\mathcal{T}_{src}[T_{new}]$Tsrc​[Tnew​]. This pipeline supplies the input bundles to SkillSmith without requiring manual annotation of task relationships.

Experiment

The evaluation spans three benchmarks—Composite-SNI, Super-Natural Instructions, and MMLU-ProX—to test SkillSmith’s ability to compose new tasks from text descriptions and learned prefix weights. Compared against weight-space merging, in-context learning, and direct prefix-tuning baselines, SkillSmith consistently outperforms, especially in zero-shot and data-limited fine-tuning settings where its cross-modal synthesis provides a strong initialization advantage. Ablations show that the model genuinely leverages both parametric and textual inputs, and its gains persist even when parent tasks are unseen, confirming that SkillSmith learns a generalizable composition rather than relying on extra textual context alone.

SkillSmith's performance improves as more input modalities are provided. Using only text metadata yields higher ELO ratings than using only K-V caches, and the combination of both modalities leads to the highest performance. Performance drops to its lowest when all inputs are removed, confirming the model's dependence on provided context. Text metadata alone proves more valuable than K-V caches alone, suggesting textual descriptions carry stronger task-relevant information. The best results emerge from jointly using text metadata and K-V caches, indicating that the model successfully integrates both modalities.

An ablation study on input modalities shows that SkillSmith's performance improves with more context, with text metadata alone being more valuable than K-V caches alone. The best results are achieved when both text metadata and K-V caches are combined, indicating successful integration of multimodal information. Removing all inputs leads to the lowest performance, confirming the model's reliance on provided context.