Architecture

AI Runtime is one gateway, backed by a database and a cache, that sits in the path of your AI traffic. The same gateway runs in every deployment option; what changes is where it runs and who operates it.

Components

Component	Role
Gateway	One process that holds both planes: the data plane inspects traffic and applies policies, the control plane is the API the console uses to manage your organization. Each plane uses its own credentials. Stateless, replicated behind a load balancer
PostgreSQL	The source of truth: organizations, routes, policies, integrations with their encrypted provider secrets, consumers, MCP servers and tools, alerts, playbooks, the audit log and incidences
Redis	Shared runtime state: the gateway key map that authenticates the data plane, cached organization configuration, rate and token limits, the firewall blocklist and the response cache
Scheduler	Triggers the gateway's periodic work: grouped alerts, quiet hours, anomaly checks, and scheduled reports
ByteHide classification service	Classifies prompt injection and jailbreak attempts. ByteHide-hosted in every deployment option
ByteHide console and platform	The console your team uses, member sign-in, and delivery of notifications and SIEM events

Deployment Models

SaaS

ByteHide runs everything. Your applications call the shared gateway over the internet.

Dedicated

A single-tenant gateway in its own cloud account, operated by ByteHide and reachable only over private connectivity. See Dedicated & Self-Hosted.

Self-Hosted

Your team runs the gateway and its data stores in your own cloud account. See On-Premises Deployment.

In the self-hosted model the console runs in your team's browsers and calls your gateway's control API directly, so the control API only needs to be reachable from your operators' network.

Request Flow

Every protected request follows the same path, whether it arrives in gateway mode, from an MCP client or from an in-app SDK. In gateway mode it looks like this:

Not every request reaches the classification service: obvious attacks are blocked deterministically, trivially clean short text is skipped, only the new messages of each turn are classified, and verdicts are cached.

LLM Traffic in Gateway Mode

Your application points its SDK's base URL at a route. The gateway authenticates the caller, applies the route's policy to the input, forwards to the provider with the key stored on the integration, applies the policy to the output, and returns the provider's answer. Your application's own credentials are never forwarded. See Gateway Mode.

MCP Traffic

Agents call an MCP route with a consumer credential or a token from your identity provider. The gateway checks the caller's grants on the route, inspects the tool call, forwards it to the upstream MCP server with the upstream credentials you configured, and inspects the tool result before it goes back to the agent. See MCP Protection.

In-app Traffic

The App runtime SDK (formerly App runtime) intercepts your application's model calls in-process and asks the gateway for a verdict. The application keeps its own connection to the provider, so the gateway needs no route to model providers. See In-app Mode.

Control Plane and Data Plane

	Control plane	Data plane
Serves	The console and automation	Your applications, agents and SDKs
Endpoints	The control API	Route URLs, `/mcp/...`, verdict endpoints
Authenticated with	The member's ByteHide session	Gateway keys, consumer credentials, project tokens or your identity provider's tokens
Exposure	Private: your operators' network only	Wherever your callers are

Both run in the same gateway process. In a self-hosted or dedicated deployment, both run in your gateway: the console only manages them.

Where Policies Are Evaluated

Policies are evaluated by the gateway, in the request path, per route and detector by detector, with each detector's mode, sensitivity and scope (input, output or both). PII, secrets and every other detector run inside the gateway. Only prompt injection classification is delegated to ByteHide's classification service.

Inspection is always synchronous: there is no traffic mirroring or out-of-band mode.

Configuration changes made in the console reach every replica without a restart:

Change	Applied within
Organization configuration	Immediately
Firewall blocklist	10 seconds
Route settings	15 seconds
Alert definitions	30 seconds
Member permissions	60 seconds

Where Verdicts and Logs Are Stored

Data	Where
Incidences	PostgreSQL, one per inspected request: verdict, detections, route, policy, caller and latency. Prompt and response fragments are stored after PII redaction, and each route can skip storing them. Caller IP addresses are kept as their `/24`
Audit log	PostgreSQL, written in the same transaction as each configuration change, with a field-by-field diff. Secrets are never included. See Audit Log
Provider secrets	PostgreSQL, encrypted at rest, and never returned by the API
SIEM events	Forwarded by the ByteHide platform. They carry the verdict, detections, identifiers and request context, never the prompt or response text
Traces	Your OpenTelemetry collector, if you configure one

In a self-hosted deployment, PostgreSQL and Redis live in your cloud account, so your configuration and incidences stay there.

Failure Modes

If this is unavailable	The gateway
Classification service	Fails closed by default: answers `503` with `x-bytehide-fail-closed: true` instead of letting traffic through uninspected. Failing open is configurable, but then prompt injection goes uninspected during the outage
PostgreSQL	Stops serving: routes no longer resolve. In a self-hosted deployment, a replica does not start without it
Redis	Rejects requests by default. Configured to fail open, it keeps working on the last cached configuration
ByteHide platform	Keeps inspecting. Alerts are evaluated and recorded, but notifications and SIEM events are not delivered
IP reputation	Keeps inspecting. Incidences lose origin details
A model provider	Retries with exponential backoff, honoring `Retry-After`, and fails over to the next target when the route uses failover or load balancing. See Routes

Your clients should treat 503 as an outage and retry, not as a detection.