Deploy Runifold in a Rust web service

Put Runifold behind an application boundary

A web service should own provider credentials, user authentication, admission, Run construction, error mapping, and observability. The browser sends product inputs to your service; it does not call model providers with a long-lived key.

client → authentication → admission → RunContext → Agent → Provider
                                ↘ journal / metrics / traces

Create provider clients and runtimes once during startup. Create a fresh Agent or RunContext for request-specific policy.

Build a complete Axum service

cargo new runifold-api
cd runifold-api
cargo add runifold@0.9.0
cargo add runifold-providers@0.9.0 --features openai
cargo add axum@0.8
cargo add serde@1 --features derive
cargo add tokio@1 --features macros,rt-multi-thread,net

Replace src/main.rs:

use axum::{Json, Router, extract::State, http::StatusCode, routing::post};
use runifold::{ProviderModelExt, ProviderRuntime};
use runifold_providers::openai::OpenAiClient;
use serde::{Deserialize, Serialize};
 
#[derive(Clone)]
struct AppState { runtime: ProviderRuntime }
 
#[derive(Deserialize)]
struct PromptRequest { prompt: String }
 
#[derive(Serialize)]
struct PromptResponse { answer: String }
 
async fn prompt(
    State(state): State<AppState>,
    Json(request): Json<PromptRequest>,
) -> Result<Json<PromptResponse>, (StatusCode, String)> {
    let answer = state.runtime
        .agent("http-assistant")
        .system("Answer precisely and expose uncertainty.")
        .prompt_text(request.prompt)
        .await
        .map_err(|error| (
            StatusCode::BAD_GATEWAY,
            format!("model request failed: {error}"),
        ))?;
 
    Ok(Json(PromptResponse { answer }))
}
 
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let runtime = OpenAiClient::from_api_key(
        std::env::var("OPENAI_API_KEY")?
    )?.runtime("gpt-5")?;
 
    let app = Router::new()
        .route("/prompt", post(prompt))
        .with_state(AppState { runtime });
    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000").await?;
 
    axum::serve(listener, app).await?;
    Ok(())
}

Set OPENAI_API_KEY, run cargo run, then send a JSON request to POST /prompt. This complete service is compiled in documentation CI.

Add request policy before exposure

The minimal service proves integration, not production safety. Add:

• authenticated tenant and actor identity;
• input and body-size limits;
• per-tenant admission and concurrency limits;
• an explicit RunContext with token, turn, Tool, and wall-time budgets;
• capability grants derived from authorization, never request JSON;
• cancellation when the client disconnects;
• stable public error codes without leaking provider bodies.

Use RunContext, budgets, and capability security for the request envelope.

Choose streaming or background work

Use SSE or WebSocket when the user is waiting for incremental output. Preserve terminal success, refusal, usage, and failure semantics instead of flattening the stream to text chunks.

Use a durable Workflow when work must survive process loss, wait for a signal, or continue after the HTTP request ends. Return a task identifier and let a Worker own execution; do not keep an HTTP connection open for hours.

Pass the deployment checklist

Before routing production traffic, verify:

1. secrets enter only through the deployment environment;
2. health checks do not call a paid model;
3. graceful shutdown cancels or hands off active Runs;
4. retries happen only for errors marked safe;
5. logs correlate tenant, Run, provider, and model without sensitive content;
6. latency, error, refusal, budget, and saturation metrics have alerts;
7. one live smoke request validates the deployed network path.