Thinking-Models

Thinking vs Standard Decoding

1. Purpose

2. Definitions

• Standard decoding: step‑by‑step next‑token generation using methods like greedy, temperature sampling, or nucleus (top‑p) sampling. No explicit intermediate plan is maintained beyond what the model implicitly learns.
• Thinking: a mode or class of models that allocate additional internal steps for planning, scratch reasoning, self‑checks, and tool sequencing before emitting the final answer. The internal steps are typically not returned; only the end result is shown.

3. How they operate

Standard decoding

1. Read instructions and context within the window.
2. Generate the next token repeatedly until stop conditions are met.
3. Optional sampling controls (temperature, top‑p) shape style and determinism.

Thinking

1. Read instructions and context.
2. Allocate internal steps: draft a plan, decompose the task, perform self‑checks, and, if enabled, call tools in sequence.
3. Emit a final answer that reflects the internal reasoning, without exposing it verbatim.

4. When thinking helps

• Multi‑step reasoning and planning: tasks that require ordering steps (e.g., "first compute A, then transform B, finally compare C").
• Long‑range dependencies: questions that tie together distant parts of a document or multiple sources.
• Code and structured outputs: generation with self‑verification against a schema or tests; iterative correction before emitting results.
• Tool orchestration: selecting tools, deciding call order, and integrating tool outputs into a coherent response.

5. When standard decoding is sufficient

• Rewriting and summarization without strict multi‑hop logic.
• Classification and extraction into a fixed set of labels or a compact JSON.
• Short factual answers when grounding is straightforward and context is small.
• Format‑preserving transformations (e.g., redaction, normalization) with clear rules.

6. Trade‑offs

• Latency: thinking typically takes longer due to extra internal steps and possible tool calls.
• Determinism: more internal steps can introduce variance across runs; deterministic settings and fixed instructions mitigate this.
• Over‑reasoning risk: excessive internal steps can lead to unnecessary elaboration or deviation from the requested output shape.
• Traceability: internal chains are generally hidden; use external validation and logs to understand behavior.

7. Design patterns

• Plan‑then‑answer: instruct the model to plan internally and output only the final result in the specified format.
• Critique‑and‑revise: produce a candidate answer, apply a short checklist, then return a corrected version.
• Tool‑aware thinking: provide a catalog of tools with clear JSON I/O; allow limited internal planning to choose and order calls.

8. Minimal specification snippets

Goal: Extract fields into strict JSON. If a field is unknown, set it to null.
Output: ONLY JSON that matches the schema.
Schema: {...}

Goal: Solve the task using internal planning and self‑checks. Do not reveal intermediate steps.
Constraints: Obey the output schema; stop if required data is missing and set fields to null.
Output: ONLY JSON that matches the schema.

Attempt standard decoding first. If validation fails or multi‑step reasoning is detected, re‑attempt in thinking mode with a reasoning step limit of N and at most M tool calls.

9. Summary