> ## Documentation Index > Fetch the complete documentation index at: https://arkor-92aeef0e-eng-716.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # infer > Run inference against a checkpoint adapter from inside onCheckpoint. # `infer` `infer` is a function passed into [`onCheckpoint`](/sdk/callbacks#oncheckpoint-step-adapter-job-infer-artifacts-) on `CheckpointContext`. It runs an inference request bound to the just-saved checkpoint adapter and returns the raw `Response`. There is no top-level `infer` export; the SDK exposes it as a callback argument so that the call is automatically scoped to the right job + checkpoint step. ```ts theme={null} onCheckpoint: async ({ step, infer }) => { const res = await infer({ messages: [ { role: "user", content: "I can't log in." }, ], }); console.log(`step=${step} sample=`, await res.text()); } ``` ## Signature ```ts theme={null} type Infer = (args: InferArgs) => Promise; ``` ## Parameters ```ts theme={null} interface InferArgs { messages: ChatMessage[]; temperature?: number; topP?: number; maxTokens?: number; /** Default: true. Set false to get a single JSON body instead of SSE. */ stream?: boolean; /** OpenAI-compatible function-calling tool definitions. */ tools?: ToolDefinition[]; /** "auto" | "none" | "required" | { type: "function", function: { name } } */ toolChoice?: ToolChoice; /** OpenAI-compatible response_format (text / json_object / json_schema). */ responseFormat?: ResponseFormat; /** vLLM structured outputs (regex / choice / grammar) for cases response_format can't express. */ structuredOutputs?: StructuredOutputs; signal?: AbortSignal; } ``` | Field | Type | Notes | | ------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `messages` | `ChatMessage[]` | Chat history. Discriminated union over `system` / `user` / `assistant` (with optional `tool_calls`) / `tool` (with `tool_call_id`) — matches the OpenAI message shape so a tool-calling history can round-trip. | | `temperature` | `number?` | Sampling temperature. Backend default if omitted. | | `topP` | `number?` | Nucleus sampling. Backend default if omitted. | | `maxTokens` | `number?` | Maximum response tokens. Backend default if omitted. | | `stream` | `boolean?` | Default **true** (SSE). Set `false` for a single JSON body. | | `tools` | `ToolDefinition[]?` | Function declarations the model is allowed to call. When set without an explicit `toolChoice`, the OpenAI-compatible default `"auto"` applies; the underlying endpoint must be configured for auto-tool extraction or the request returns `400 tool_calling_not_configured`. | | `toolChoice` | `ToolChoice?` | `"auto"` / `"none"` / `"required"` / `{ type: "function", function: { name } }` — only `"auto"` (and the default when `tools` is present) needs the auto-extraction parser; the rest go through the guided-decoding path. | | `responseFormat` | `ResponseFormat?` | OpenAI's standard structured-output knob: `{ type: "text" }`, `{ type: "json_object" }`, or `{ type: "json_schema", json_schema: { name, schema, strict? } }`. Prefer this when expressible. | | `structuredOutputs` | `StructuredOutputs?` | vLLM extension for constraints `responseFormat` can't express. The whole field is optional, but **when provided, exactly one** of `json` / `regex` / `choice` / `grammar` / `json_object` / `structural_tag` must be set (vLLM's `__post_init__` rejects zero or more than one constraint, and the type encodes the same invariant via `ExactlyOne`). Field names are snake\_case (`json_object`, `structural_tag`, `disable_any_whitespace`, `whitespace_pattern`) to match vLLM's wire format. | | `signal` | `AbortSignal?` | Aborts the local fetch. Does not stop work on the backend; the model finishes generating but you stop reading. | ### Tool calling example ```ts theme={null} onCheckpoint: async ({ infer }) => { const res = await infer({ messages: [ { role: "user", content: "What's the weather in Tokyo?" }, ], tools: [ { type: "function", function: { name: "get_weather", parameters: { type: "object", properties: { city: { type: "string" } }, required: ["city"], }, }, }, ], toolChoice: "auto", stream: false, }); const data = (await res.json()) as { choices: Array<{ message: ChatMessage }> }; // data.choices[0].message may be { role: "assistant", tool_calls: [...] } }; ``` ### Structured-output example ```ts theme={null} const res = await infer({ messages: [{ role: "user", content: "Extract the user's email." }], responseFormat: { type: "json_schema", json_schema: { name: "user", schema: { type: "object", properties: { email: { type: "string", format: "email" } }, required: ["email"], }, strict: true, }, }, }); ``` ## Returns `infer` returns `Promise`: the raw Fetch `Response`. The SDK does not parse the body; you decide how to consume it: ```ts theme={null} // Streaming (default) const res = await infer({ messages }); for await (const chunk of res.body!) { // chunk: Uint8Array of one or more SSE frames } // Or read the whole stream at once const text = await res.text(); // Or, if you set stream: false, parse the JSON body const res = await infer({ messages, stream: false }); const data = await res.json(); ``` When `stream: true` (the default), the body is an SSE event stream in the same shape Studio's Playground consumes. The SDK does not currently expose a frame parser for this stream; if you need decoded text deltas, copy the small `extractInferenceDelta` helper from `packages/studio-app/src/lib/api.ts` or write a parser around `eventsource-parser`. ## Constraints * `infer` lives **only** on `CheckpointContext`. There is no equivalent for completed jobs from the SDK side; for that path use the cloud-api directly or trigger the run again. Studio's Playground is the UI-level route to chat with a completed adapter. * The call is scoped to `{ kind: "checkpoint", jobId, step }`. You cannot retarget it to a different checkpoint or a different model from inside `onCheckpoint`. * The function is not memoized: every call hits the backend. ## Use cases * **Sanity check during a run.** Compare a checkpoint at step 50 to one at step 100 against a fixed prompt. If the loss curve looks fine but outputs are degraded, you find out before the run finishes. * **Custom early-stopping.** Combine with a simple eval prompt: if outputs diverge, abort the run via `controller.abort()` (see [`abortSignal`](/sdk/trainer-control#abortsignal)) and call `trainer.cancel()` to stop the backend. See the [Early stopping recipe](/cookbook/early-stopping) for the full pattern. * **Live preview into your own UI.** Send the checkpoint output to Slack, an internal review queue, or your own app's preview channel. ## See also * [`onCheckpoint`](/sdk/callbacks#oncheckpoint-step-adapter-job-infer-artifacts-) for the callback that hands you `infer` * [Mid-run eval recipe](/cookbook/mid-run-eval) for an end-to-end pattern * [Early stopping recipe](/cookbook/early-stopping) for tying `infer` output to `abortSignal` + `cancel()` * [Trainer control](/sdk/trainer-control) for `abortSignal` and `cancel()`