Middleware

Overview

Middleware lets you hook into every model operation to run logic before and after each call. A middleware is an object with optional hooks that wrap operations like generate, stream, generateObject, streamObject, or embed. Each hook receives the call options, a reference to the model, and an execute function that continues the chain. Use middleware for:

Logging and telemetry
Input preprocessing and sanitization
Output validation and transformation
Custom error handling and retry logic
Caching
Access control

core-ai ships first-party OpenTelemetry and Langfuse middleware packages built on this same system. See Observability for details.

Applying middleware

Wrap a model with wrapChatModel to apply middleware. The returned model has the same ChatModel interface, so you can use it anywhere the original model was used.

import { wrapChatModel, generate } from '@core-ai/core-ai';
import type { ChatModelMiddleware } from '@core-ai/core-ai';

const logging: ChatModelMiddleware = {
    generate: async ({ execute, options, model }) => {
        console.log(`Calling ${model.provider}/${model.modelId}`);
        const result = await execute();
        console.log(`Finished with ${result.usage.outputTokens} output tokens`);
        return result;
    },
};

const wrappedModel = wrapChatModel({ model, middleware: logging });

const result = await generate({
    model: wrappedModel,
    messages: [{ role: 'user', content: 'Hello!' }],
});

You can pass a single middleware or an array:

const wrappedModel = wrapChatModel({
    model,
    middleware: [logging, validation, retry],
});

Writing custom middleware

A ChatModelMiddleware is an object with optional hooks. Each hook you define wraps the corresponding model operation. Hooks you omit pass through to the model unchanged.

type ChatModelMiddleware = {
    generate?: (args: {
        execute: (options?: GenerateOptions) => Promise<GenerateResult>;
        options: GenerateOptions;
        model: ChatModel;
    }) => Promise<GenerateResult>;
    stream?: (args: {
        execute: (options?: GenerateOptions) => Promise<ChatStream>;
        options: GenerateOptions;
        model: ChatModel;
    }) => Promise<ChatStream>;
    generateObject?: <TSchema extends z.ZodType>(args: {
        execute: (
            options?: GenerateObjectOptions<TSchema>
        ) => Promise<GenerateObjectResult<TSchema>>;
        options: GenerateObjectOptions<TSchema>;
        model: ChatModel;
    }) => Promise<GenerateObjectResult<TSchema>>;
    streamObject?: <TSchema extends z.ZodType>(args: {
        execute: (
            options?: StreamObjectOptions<TSchema>
        ) => Promise<ObjectStream<TSchema>>;
        options: StreamObjectOptions<TSchema>;
        model: ChatModel;
    }) => Promise<ObjectStream<TSchema>>;
};

Each hook receives three arguments:

execute — call this to continue the chain. You can call it with no arguments to pass the original options, or pass modified options to override them.
options — the options for the current call (messages, temperature, tools, etc.)
model — the underlying model, useful for reading provider and modelId

Example: input guardrail

This middleware checks user messages for blocked terms before calling the model:

import type { ChatModelMiddleware } from '@core-ai/core-ai';

const blockedTerms = ['password', 'secret-key'];

const inputGuardrail: ChatModelMiddleware = {
    generate: async ({ execute, options }) => {
        for (const message of options.messages) {
            if (
                message.role === 'user' &&
                typeof message.content === 'string' &&
                blockedTerms.some((term) => message.content.includes(term))
            ) {
                throw new Error('Message contains blocked content');
            }
        }

        return execute();
    },
};

Example: automatic retries

This middleware retries failed calls with exponential backoff:

import type { ChatModelMiddleware } from '@core-ai/core-ai';

const retry: ChatModelMiddleware = {
    generate: async ({ execute }) => {
        const maxAttempts = 3;

        for (let attempt = 1; attempt <= maxAttempts; attempt++) {
            try {
                return await execute();
            } catch (error) {
                if (attempt === maxAttempts) throw error;
                await new Promise((resolve) =>
                    setTimeout(resolve, 1000 * attempt)
                );
            }
        }

        throw new Error('Unreachable');
    },
};

Part metadata

Middleware can use part metadata to keep annotations next to the text they describe. This is useful for provenance, validation status, classification results, or text transformation spans. On outbound calls, read metadata from text-bearing messages:

import type { ChatModelMiddleware } from '@core-ai/core-ai';

const inspectPartMetadata: ChatModelMiddleware = {
    generate: async ({ execute, options }) => {
        for (const message of options.messages) {
            if (message.role === 'user' && Array.isArray(message.content)) {
                for (const part of message.content) {
                    if (part.type === 'text') {
                        console.log(part.metadata);
                    }
                }
            }
        }

        return execute();
    },
};

On inbound streaming calls, wrap the stream and attach metadata to text-end. createChatStream() copies that metadata to the matching AssistantTextPart in .result.

import { createChatStream } from '@core-ai/core-ai';
import type { ChatModelMiddleware, StreamEvent } from '@core-ai/core-ai';

const annotateText: ChatModelMiddleware = {
    stream: async ({ execute }) => {
        const chatStream = await execute();

        async function* annotate(): AsyncIterable<StreamEvent> {
            for await (const event of chatStream) {
                if (event.type === 'text-end') {
                    yield {
                        ...event,
                        metadata: { reviewed: true },
                    };
                    continue;
                }

                yield event;
            }
        }

        return createChatStream(annotate());
    },
};

GenerateOptions.metadata is call-level context. Part metadata is application-owned state that stays on individual text parts, reasoning parts, tool calls, and tool results. providerMetadata on reasoning parts is provider-owned data for provider round-trips.

Embedding and image middleware

EmbeddingModelMiddleware and ImageModelMiddleware follow the same pattern with their respective operations.

EmbeddingModelMiddleware

import { wrapEmbeddingModel } from '@core-ai/core-ai';
import type { EmbeddingModelMiddleware } from '@core-ai/core-ai';

const logging: EmbeddingModelMiddleware = {
    embed: async ({ execute, model }) => {
        console.log(`Embedding with ${model.modelId}`);
        return execute();
    },
};

const wrappedModel = wrapEmbeddingModel({
    model: embeddingModel,
    middleware: logging,
});

ImageModelMiddleware

import { wrapImageModel } from '@core-ai/core-ai';
import type { ImageModelMiddleware } from '@core-ai/core-ai';

const logging: ImageModelMiddleware = {
    generate: async ({ execute, model }) => {
        console.log(`Generating image with ${model.modelId}`);
        return execute();
    },
};

const wrappedModel = wrapImageModel({ model: imageModel, middleware: logging });

Composing middleware

When you pass an array of middleware, they execute in order from first to last. The first middleware in the array is the outermost layer — it runs first on the way in and last on the way out.

const wrappedModel = wrapChatModel({
    model,
    middleware: [logging, retry, validation],
});

// Execution order for generate():
// 1. logging (before) →
// 2.   retry (before) →
// 3.     validation (before) →
// 4.       model.generate()
// 5.     validation (after) ←
// 6.   retry (after) ←
// 7. logging (after) ←

Each middleware can:

Modify options before calling execute()
Inspect or transform the result after execute() resolves
Short-circuit the chain by returning without calling execute()
Catch and handle errors from execute()

First-party middleware packages

core-ai provides two observability middleware packages. These are built on the same middleware system documented above, and serve as good reference examples for writing your own middleware.

OpenTelemetry

Automatic tracing with OpenTelemetry spans for all model operations

Langfuse

Langfuse observability with generation tracking and usage reporting

Next steps

Set up tracing with Observability
Learn about Models and their interfaces
Configure generation options with Configuration

Get Started

Core Concepts

Guides

Observability

Examples

Overview

Applying middleware

Writing custom middleware

Example: input guardrail

Example: automatic retries

Part metadata

Embedding and image middleware

EmbeddingModelMiddleware

ImageModelMiddleware

Composing middleware

First-party middleware packages

OpenTelemetry

Langfuse

Next steps

​Overview

​Applying middleware

​Writing custom middleware

​Example: input guardrail

​Example: automatic retries

​Part metadata

​Embedding and image middleware

​EmbeddingModelMiddleware

​ImageModelMiddleware

​Composing middleware

​First-party middleware packages

OpenTelemetry

Langfuse

​Next steps

Overview

Applying middleware

Writing custom middleware

Example: input guardrail

Example: automatic retries

Part metadata

Embedding and image middleware

EmbeddingModelMiddleware

ImageModelMiddleware

Composing middleware

First-party middleware packages

Next steps