One-sentence Summary

The authors introduce PRefLexOR, a framework that integrates preference optimization with reinforcement learning for self-improving scientific reasoning through recursive step refinement, log-odds optimization between preferred and non-preferred responses, and dynamic knowledge graph contextualization, enabling even 3B-parameter models to solve open-domain biological materials science problems via multi-agent self-improvement.

Key Contributions

• PRefLexOR is a framework for self-improving scientific reasoning that combines preference optimization with reinforcement-learning concepts, recursive refinement of intermediate steps, and a dynamic knowledge graph that supplies retrieval-augmented context.

• The method optimizes the log odds between preferred and non-preferred responses using an in-situ dataset generation algorithm, and it uses rejection sampling, masked reasoning steps, and feedback-guided recursive optimization to focus training on discovery-oriented reasoning.

• Experiments show that 3B parameter models can self-teach deeper reasoning and solve open-domain problems, and the framework integrates into existing LLMs while improving reasoning in biological materials science through multi-agent self-improvement.

Introduction

The authors frame the research around an interdisciplinary connection between Hermann Hesse's Glass Bead Game and the hierarchical, dynamic nature of proteins. The context matters because both systems model reality as a network of interconnected elements, where simpler components combine into complex structures that can change and adapt. Prior discussions often treat the game as a philosophical or artistic construct and proteins as molecular biological objects, leaving limited opportunity to use their shared principles to explain complexity and adaptation. The main contribution is a conceptual bridge that leverages protein structure, conformational change, and molecular interaction as analogies for the game's evolving relationship between art, science, and reality.

Dataset

• Raw source corpus: The authors use 500 scientific papers from biological and bio-inspired materials as the underlying text corpus, converted into raw text for training data construction. The paper does not report a fixed number of generated examples or train/validation/test mixture ratios.

• On-the-fly dataset generation: Training data is generated during training rather than released as a static dataset. The process produces a question from a context, a correct answer, and a rejected or low-quality answer for comparison.

• Retrieval-augmented context enrichment: Raw text chunks are embedded with BAAI/bge-large-en-v1.5 and indexed using LlamaIndex. Generated questions are encoded as vectors, cosine similarity is used to retrieve relevant chunks, and the retrieved chunks are appended to the original context before answer generation.

• Generation model and variants: The dataset generation is primarily performed with mistralai/Mistral-Nemo-Instruct-2407, with some runs using meta-llama/Llama-3.1-8B-Instruct as an alternative.

• Example structure: Each generated example contains a question with an instruction to include a Thinking Section for Reasoning, a correct answer structured with that reasoning section, and a rejected answer. The rejected answer is designed to be incorrect and lack proper reasoning, or is generated from the current model state in certain training stages.

• Reflection processing: An optional reflection section can be added, where the model critiques its own reasoning and suggests improvements. This creates a structured introspective format for later reasoning data.

• Tokenized schema: Special tokens are added to the meta-llama/Llama-3.2-3B-Instruct tokenizer to organize generated responses into phases, including and for reasoning steps, and for reflection, and for the final answer, and optional and for intermediate notes.

• Dataset use in training: The authors use these structured examples to train the model to separate reasoning, reflection, and final answer generation. The paper states that no standalone dataset was generated or analyzed beyond this on-the-fly training construction.

Method

The PRefLexOR Framework: Architecture, Training Pipeline, and Recursive Inference

The authors present PRefLexOR, a multi-stage training and inference framework designed to endow a compact language model (in this case, meta-llama/Llama-3.2-3B-Instruct) with structured scientific reasoning capabilities, specifically tailored to the domain of biological and bio-inspired materials. The system is inspired by the adaptive and recursive optimization processes observed in biological systems, where iterative feedback loops progressively refine decision-making pathways. The framework integrates four core components: a customized tokenizer with special reasoning markers, an on-the-fly synthetic data generation pipeline leveraging retrieval-augmented generation (RAG), a two-phase preference-based training strategy, and a recursive multi-agent inference algorithm.

Special Tokens for Structured Reasoning

A foundational design decision is the extension of the base model's vocabulary with dedicated special tokens that explicitly demarcate reasoning phases within the model's output. The authors integrate the following token pairs into the tokenizer: <|thinking|>...<|/thinking|> for the reasoning phase, <|reflect|>...<|/reflect|> for the reflective evaluation phase, <|response|>...<|/response|> for the final answer boundary, and optionally <|scratchpad|>...<|/scratchpad|> for intermediate computations. These tokens serve as structural anchors that allow the training algorithm to selectively mask, include, or exclude portions of the generated sequence during loss computation, and enable the inference pipeline to parse and manipulate distinct reasoning components.

On-the-Fly Synthetic Data Generation

Rather than relying on static preference datasets, the authors construct an agentic data generation algorithm that produces question-answer pairs dynamically during training. The pipeline operates as follows. First, a pre-constructed index of text embeddings is built from the raw corpus using the BAAI/bge-large-en-v1.5 embedding model, where each text chunk $T_i$Ti​ is mapped to a dense vector via $\mathbf{v}_i = f_{\mathrm{embed}}(T_i)$vi​=fembed​(Ti​). At each training step, a random subset of text chunks (typically $n=3$n=3) is retrieved and concatenated to form a context. A domain-specific, expert-level question is then generated from this context by a teacher model. The context is further enriched through RAG: the query $Q$Q is encoded as $\mathbf{v}_q$vq​, and cosine similarity

is used to retrieve the most relevant additional chunks, which are appended to the original context. Structured information is then extracted across predefined categories (reasoning steps, relevant materials, design principles, hypotheses, and background), assembled into a "Thinking Section for Reasoning," and used to generate a correct answer. An incorrect (rejected) answer is also produced, either by prompting a model to deliberately omit logical reasoning or, during the second training phase, by sampling from the current trained model to keep the rejected response up to date and progressively reduce the margin between chosen and rejected samples.

Two-Phase Training Strategy

Phase 1: Structured Thought Integration Training (ORPO). In the first stage, the model learns to handle the new special tokens and produce well-structured reasoning outputs. The authors employ Odds Ratio Preference Optimization (ORPO), a reference-model-free method that combines supervised fine-tuning with a mild penalty on disfavored outputs. ORPO leverages the odds ratio to contrast favored and disfavored completions, teaching the model to recognize and generate the <|thinking|> and <|/thinking|> delimiters while aligning its internal reasoning representations with expert-level scientific responses. Low-Rank Adaptation (LoRA) with rank $r=64$r=64 and scaling factor $\alpha=64$α=64 is applied across the transformer's attention, feed-forward, embedding, and output projection layers, enabling efficient parameter updates.

Phase 2: Independent Reasoning Development (EXO). The second stage shifts focus from explicit reasoning structure to autonomous reasoning capability. Intermediate thinking tokens within the <|thinking|> sections are masked so the model must infer effective reasoning pathways solely from the final answer quality. The authors adopt Efficient Exact Optimization (EXO), a mode-seeking preference alignment method that minimizes reverse Kullback-Leibler divergence. For the binary case ($K=2$K=2), the EXO loss takes the form:

where $p_\theta$pθ​ and $p_r$pr​ represent empirical distributions from the learned policy and preference model over the ground-truth answer $y_w$yw​ and the rejected answer $y_l$yl​. Unlike DPO's forward-KL (mean-seeking) behavior, EXO's reverse-KL formulation concentrates probability mass on the most likely correct reasoning patterns, enabling the model to reconstruct hidden reasoning chains from the final answer signal. A new LoRA adapter ($r=64$r=64, $\alpha=64$α=64) is applied to attention and feed-forward layers (excluding embeddings and output head), with label smoothing ($5 \times 10^{-3}$5×10−3), scaling parameter $\beta=0.1$β=0.1, learning rate $5 \times 10^{-7}$5×10−7, and maximum gradient norm of 0.3. Training proceeds for one epoch per newly generated batch of 50 samples. Refer to the training curves below, which show the progressive increase in reward margins, steady convergence of the loss function, and rapid stabilization of accuracy near 1.0 within approximately 200 steps, confirming effective mode-seeking optimization.

Masking Strategies for Loss Computation

To handle reasoning tokens during preference optimization, the authors introduce two complementary mechanisms. In the token masking approach, a boolean mask $\mathbf{m}$m is applied such that inner thinking tokens receive $m_i = 0$mi​=0 while all other tokens (including the delimiters <|thinking|> and <|/thinking|>) receive $m_i = 1$mi​=1. The masked log-probabilities $\mathbf{p}_{\text{masked}} = \mathbf{p} \odot \mathbf{m}$pmasked​=p⊙m then feed into the DPO-style loss. A tunable parameter $\alpha \in [0,1]$α∈[0,1] controls the fraction of inner thinking tokens masked via $\lfloor \alpha \cdot n \rfloor$⌊α⋅n⌋ randomly selected tokens. In the dynamic final answer detection approach, the model identifies the position $\mathbf{t}_{\text{end}}$tend​ of the last <|/thinking|> token, and only the log-probabilities of tokens after this position contribute to the loss:

This provides flexibility for variable-length reasoning sequences while ensuring that preference comparisons focus exclusively on final output quality.

Inference-Time Recursive Reasoning with Reflection

At inference, the authors extend the basic thinking-only structure by incorporating a reflection phase triggered by <|reflect|>...<|/reflect|> tokens. As illustrated in the schematic below, the generation process begins with the <|thinking|> phase, where the model outlines reasoning steps, references relevant materials and concepts, and formulates a hypothesis. The <|reflect|> phase then revisits this reasoning, identifying improvements and corrections. The combination produces a refined final answer that benefits from both exploratory reasoning and critical self-evaluation.

To operationalize this structured output at test time, the authors design a Recursive Reasoning Algorithm implemented as a two-agent system. The Reasoning Model (the fine-tuned PRefLexOR model) generates initial responses containing both thinking and reflection components. The Critic Model (a general-purpose LLM) extracts the reflection, uses it to produce an improved thinking process, and feeds this back to the Reasoning Model for regeneration. This cycle repeats for $N$N iterations. The flowchart below depicts this iterative generation, extraction, critique, and regeneration loop, with an optional final integration step where the critic synthesizes all iterations into a comprehensive amalgamated response.

Quantitative evaluation of this recursive process, scored by gpt-4o across coherency, accuracy, depth of explanation, and clarity, demonstrates consistent improvement across iterations. The bar chart below shows that the initial response ($i=0$i=0) provides only a high-level summary, while the second iteration ($i=1$i=1) introduces specific mechanisms, and the final iteration ($i=2$i=2) achieves the highest scores in accuracy and depth, reflecting the algorithm's capacity for progressive refinement.

Multi-Agent Variant for Thinking-Only Models

The authors also demonstrate that recursive improvement does not strictly require the model to output reflection tokens. For models trained with thinking tokens only, a multi-agent strategy places the burden of critique entirely on the Critic agent. As shown in the alternative flowchart, the Critic first generates a reflection by analyzing the produced thinking process, then uses that reflection to improve the thinking, after which the Reasoning Model resamples a new response. This recursion proceeds identically to the two-phase agent loop, with the same integration and iteration-count logic, confirming that the recursive optimization paradigm is decoupled from the specific token structure of the reasoning model.

Experiment

Experiments evaluated a two-stage preference-optimized small language model, trained on synthetic biological-materials data with explicit thinking tokens, through qualitative inference tasks ranging from in-domain scientific questions to unseen interdisciplinary prompts, abstract summarization, and research proposal generation. The samples showed that the model could produce structured reasoning, connect abstract concepts to domain knowledge, and propose plausible research directions, validating that masked reasoning preference alignment transfers reasoning strategies beyond fact retrieval. Additional thinking and reflection iterations with critic-based improvement increased coherence, accuracy, depth, and clarity, while comparisons with non-fine-tuned baselines and a general-purpose model confirmed the advantage of domain-specific reasoning alignment.

The comparison shows that iterative refinement moves from a basic answer to a more comprehensive explanation of graceful biological failure. Later responses add detailed structural explanations, energy dissipation mechanisms, and concrete examples, with the final response including quantitative information and loading condition effects. The result indicates that reflection improves technical depth and coverage rather than just changing wording. Core concepts and hierarchy ideas are present from the first response. Thinking and reflection introduce detailed structural and energy dissipation explanations. The final response is most comprehensive, adding quantitative details and broader coverage.

The framework integrates concepts from the Glass Bead Game by using symbolic representation, interdisciplinary synthesis, reflective tokens, and multi-agent interaction to guide language model reasoning. It combines implicit model abstractions with explicit symbolic prompts and evolutionary memory updates, aiming for more interconnected, deliberate, and explainable responses. The described iteration process separates response generation from critique and improvement, allowing specialized reasoning and integration to refine outputs across cycles. Thinking tokens guide symbolic abstraction and help combine concepts across ideas. Multi-agent interaction brings diverse perspectives and collaborative refinement into the response process. Reflection tokens support deeper analysis of implications, principles, and successful patterns. Explicit prompts connect the model's inherent abstractions with controlled symbolic reasoning.

Special tokens are integrated into the updated Llama-3.2 tokenizer to structure model reasoning, reflection, and final answer generation. The token design separates thinking steps, reflective improvements, and response boundaries, while only the thinking and reflection token pairs are used in this work. The approach can also be extended to represent additional reasoning concepts through tokenizer-level control. Token pairs guide the model toward structured reasoning pathways rather than unconstrained generation. Reflection tokens mark a phase where the model evaluates and improves its initial reasoning. Response tokens demarcate the boundary between internal reasoning or reflection and the final answer. Only a subset of the introduced special tokens is used in this work, leaving the scheme extensible.

The structured categories convert free-form scientific answers into explicit components, such as reasoning steps, relevant concepts, design principles, material properties, and hypotheses. This organization helps a critic model inspect, compare, and refine successive reasoning iterations more effectively. It supports scientific reasoning by tying generated explanations to concrete context elements rather than unconstrained text. Reasoning steps isolate the logical chain behind a response. Material and design categories anchor answers to scientific context. Hypotheses provide a place for context-grounded proposed explanations.

The experiments evaluate an iterative reasoning framework for language models and validate that reflection-based refinement produces more comprehensive, technically grounded explanations rather than merely rephrased answers. The results show that special tokens and a multi-agent reflection process help separate thinking, critique, and final response generation, leading to more deliberate and interconnected reasoning. Structured categories for reasoning steps, design principles, material properties, and hypotheses further enable a critic model to inspect and improve outputs by anchoring them in scientific context.