Consider one user with three devices: a recent phone that already has a system model ready, a tablet that is eligible but still downloading the assets, and an older device that will never run the model at all. Now consider one feature that behaves differently depending on the request: rewriting a sentence is harmless and local, but estimating a price needs current external data. Network, battery, language support, subscription status, and consent can all change between the moment the app launches and the moment the user taps the button.

So a single compile-time answer is rarely enough. The question worth designing around is narrower and more useful: for this operation, on this device, at this moment, which route can satisfy the product's promise without quietly changing its privacy, cost, or quality? This article is a framework for answering that, and for building the small amount of machinery that makes the answer safe.

Start with the operation, not the model

"AI assistant" is a label that hides decisions which should stay separate. Break a feature into the operations it actually performs, and the routing choices become obvious. Four common ones:

• completing a category from the user's recent history;
• rewriting one short sentence while keeping its language;
• grouping a batch of items into buckets the user already defined;
• interpreting an image and, when needed, consulting live sources.

The first may need no model at all — a rule can do it. The second is a good on-device candidate. The third fits either route depending on how much context it needs and how reliable the output must be. The fourth usually needs the cloud, but only because it requires live retrieval; plain image understanding is increasingly available on-device on newer hardware, so treat "needs vision" as device-dependent and "needs live data" as the durable reason to go remote.

Define the product-level contract once, and put the implementations behind it:

protocol AssistantService: Sendable {
    func rewrite(_ text: String) async throws -> String
    func classify(_ items: [ItemBrief]) async throws -> [Suggestion]
}

struct ItemBrief: Sendable {
    let id: UUID
    let title: String
    let currentGroup: String?
}

The input is a small, immutable, sendable snapshot — not an object owned by the view or the database. That one choice prevents a category of concurrency and lifecycle bugs, limits how much data can accidentally leak into a request, and lets both implementations be tested against the same fixtures. It also keeps routing out of the UI. A button should ask to rewrite text. It should not know how to build a vendor request or check whether the device's model is ready.

Decide with a matrix, and put hard constraints first

There are seven practical dimensions. Some are preferences. Some are disqualifiers, and those have to be checked first.

Capability. Can the route even do the job? A local text model cannot satisfy a need for live sources. A cloud text endpoint cannot interpret an image. Keep tool use, supported media, context size, structured-output support, and language coverage in an explicit capability record rather than scattering model-name checks through the code.

Availability. On-device availability is not a boolean. It has states: the OS is too old, the hardware is ineligible, the user has turned intelligence off, the assets are not ready yet, the runtime is briefly busy, or it is available. Cloud availability has its own states: offline, not authenticated, rate-limited, or disabled. Keep those distinctions long enough to choose the right fallback and show the right message.

Privacy and consent. Running locally can reduce how much data leaves the device, but it does not end your privacy work — inputs can still be logged, persisted, or backed up. A cloud route needs real data minimisation and, for free-form or sensitive content, a clear and revocable consent boundary. The rule that matters most: an automatic fallback must never turn "runs locally" into "uploads to a third party" without a policy the user actually accepted.

Economics. On-device inference has no metered network call, but it spends memory, battery, and engineering time. Cloud inference has a visible marginal cost and may need subscription gating. Route by the operation's cost class, not by whether the user is labelled premium — a free local path and a paid cloud fallback can sit behind the very same button.

Latency and lifecycle. Short local work can feel instant once a session is warm; the first call, before that, may not. Cloud calls add network variance but run on far more devices. Either way, the work must be cancellable when the view goes away, and a late result must never overwrite a newer one.

Quality and domain risk. An acceptable rewrite is a much lower bar than an acceptable extracted monetary amount. For higher-risk outputs, the route has to support enough validation, and often a human review step. "The cloud model is better" is not a contract; define what valid means for the domain.

Consistency and operations. A system or bundled model can shift with platform updates and device settings; a cloud model can change at the provider's edge. Cloud routes are easier to disable remotely. Local routes can keep working during a server incident — provided their entitlement and kill-switch checks are cached or fail open, rather than blocking on a server they can't reach. Neither route escapes drift, so both need tests and version-aware telemetry.

Evaluate the hard constraints before anything else. If the operation needs live retrieval, local is out. If there is no consent to upload, cloud is out. If the device model is not ready, local is out for an action that has to happen now. Only after the ineligible routes are removed do preferences like cost and latency decide between what's left.

Resolve a route, not a brand

A resolver should return more than the string "local" or "cloud". It should return a decision that carries its reason and the fallback it permits — and the fallback has to obey the same rules as a primary of the same kind:

enum Route {
    case deterministic
    case local
    case cloud(tier: Tier)
}

struct RouteDecision {
    let primary: Route
    let fallback: Route?
    let reason: Reason
    let uploadConsentRequired: Bool
}

func resolve(_ op: Operation, env: Environment) -> RouteDecision? {
    if op.canUseRules, env.rulesCanAnswer {
        return .init(primary: .deterministic, fallback: nil,
                     reason: .localRuleMatched, uploadConsentRequired: false)
    }

    if op.localCapabilities.isSubset(of: env.localCapabilities), env.localReady {
        // A cloud fallback is only offered when the cloud route is already
        // allowed, paid for, and consented to.
        let cloudFallbackOK = env.cloudAllowed && env.entitled && env.uploadConsent
        return .init(primary: .local,
                     fallback: cloudFallbackOK ? .cloud(tier: .small) : nil,
                     reason: .localCapable,
                     uploadConsentRequired: false)
    }

    guard env.cloudAllowed, env.entitled, env.uploadConsent,
          op.cloudCapabilities.isSubset(of: env.cloudCapabilities) else { return nil }
    return .init(primary: .cloud(tier: op.cloudTier), fallback: nil,
                 reason: .localUnavailable, uploadConsentRequired: true)
}

Two things about this. First, the local branch used to be where consent quietly leaked: it is tempting to attach a cloud fallback gated only on "is cloud allowed," but that lets a local failure upload data the user never agreed to send. Gate the fallback on the full contract, and the decision object stops lying — uploadConsentRequired is only ever false when no route in the decision can upload. Second, this resolver encodes a deliberate local-first preference among eligible routes; that is a simplification of the matrix, not the whole matrix.

In production, cloudAllowed should fold in a remote kill switch and backend admission, and entitled should not trust mutable client state alone. The decision's reason is also what feeds privacy-safe telemetry and support later.

Give the user an explicit setting, too: automatic, on-device only, or cloud allowed — stored per device, because hardware capability differs across the devices one person owns. "On-device only" should disable or hide unsupported actions with a specific reason. It must never silently fall back to the cloud. "Automatic" may fall back, but only inside the consent and subscription contract above.

Make local execution a real service

On-device code is too often a convenience call embedded in a view, and that throws away most of its value. It should implement the same protocol as the cloud path and normalise its failures into domain categories: assets unavailable, context too large, safety guardrail, decoding failure.

A long-lived session avoids paying the warm-up cost on every call. But be precise about how you protect a stateful session. Putting it "behind an actor" is not, by itself, enough: Swift actors are reentrant, so while an actor serialises synchronous access to its state, an actor-isolated async method gives up its executor at every await — which means two operations can each reach the session across a suspension point. Many stateful session APIs reject exactly that, throwing if you start a new response while one is in flight. So enforce one-at-a-time explicitly: check an in-flight flag, feed a single serial consumer, or reject and defer overlapping calls. Isolation plus an explicit gate, not isolation alone.

The rest follows from treating the session as a queue you own. Don't launch unbounded background work from the UI. Coalesce duplicate actions, cancel obsolete ones, and show progress for batches. When a response fails in a way that might have corrupted the session, discard the session so the next operation starts clean.

And local output is still untrusted. A model that wraps JSON in commentary still needs extraction and decoding. A rewrite should be trimmed, stripped of presentation quotes, rejected if empty, and treated as a no-op if it equals the original. A grouping result may reference only identifiers that were in the request.

One scoping note, because this framework invites it: the availability states above — especially "assets not ready" — describe the OS system model, whose download and readiness the platform manages for you. If you ship or download your own model, you own delivery: keep the model out of the base install, fetch it on opt-in to avoid bloating the first download and every update, and use your platform's current on-demand asset mechanism. Confirm the exact mechanism against current platform docs before you rely on it; that part moves.

Make cloud execution narrow

The cloud adapter should receive only what the operation needs. A classification request can send identifiers, short titles, and a few examples from existing groups — not full records, attachments, or an account history. A conversation can send a bounded recent window instead of growing forever.

Credentials belong behind an authenticated proxy, never in the app binary. The proxy should validate input type, length, count, and media size; verify app attestation where the platform supports it; enforce rate and entitlement policy; and translate provider errors into a stable application error vocabulary.

Retries should be selective. One authentication refresh after an unauthenticated response is reasonable. Retrying every failure is not. And because a timeout can arrive after inference was already accepted and billed, use an idempotency key when repeating a request could duplicate a cost.

Finally, localise the error at the edge. A backend resource_exhausted becomes "try again later." An offline state becomes a connectivity action. Malformed output becomes a safe no-change result or a review failure. Provider messages should not leak into the UI a customer reads.

Converge at the validation layer

Local and cloud implementations don't need identical prompts or APIs. They need equivalent domain behaviour. Once each route has produced decoded transport types, everything runs through one shared pipeline:

1. Validate identifiers against the set that was submitted.
2. Normalise enum values, whitespace, dates, and currency codes.
3. Reject impossible numbers and unsafe links.
4. Convert the result into a proposal the user can review.

(The extraction of a JSON region from prose, and the decode into transport types, are per-route pre-steps that feed this shared pipeline — do them before you converge, not after.)

This layer is what stops a route change from changing persistence semantics. If a cloud response names a group that doesn't exist, or a local response drops an item, both resolve to "keep unchanged" — never to data invented from a guess.

For batch operations, preserve input order and emit exactly one decision per item in a response you actually received; a missing or malformed entry becomes an explicit keep. Chunk-level failures are a different level: if one chunk fails as a transport, keep the successful chunks and leave the failed chunk's items pending for retry rather than marking them keep; if every chunk fails, report a service error. That distinction matters operationally — it avoids telling the user that everything was already correct during an outage.

Design the UX around uncertainty

The interface should expose the outcome, not infrastructure trivia. "Preparing on this device" is useful. A vendor-specific error string is not. And when routing is automatic, users deserve to know when their data will leave the device — especially if they chose the feature because it was described as local.

Route AI-generated changes through a review surface: a before/after diff, a selectable plan, an editable extraction. Save only what the user accepts. This keeps the user as the authority and, as a bonus, gives every route the same failure behaviour — the original stays intact no matter what.

Lifecycle handling is part of correctness, not polish. Store the in-flight task, cancel it when the sheet disappears, and cancel it before starting a replacement. Cooperative cancellation isn't enough on its own, so compare a generation token before applying any result:

generation += 1
let mine = generation
work?.cancel()
work = Task {
    do {
        let proposal = try await service.classify(snapshot)
        try Task.checkCancellation()
        guard mine == generation else { return }
        state = .review(validate(proposal, against: snapshot))
    } catch is CancellationError {
        // expected on replacement — do nothing
    } catch {
        guard mine == generation else { return }
        state = .failed(localize(error))
    }
}

This matters most when a user switches modes while a slower route is still running: only the result whose generation still matches is allowed to update the screen, and a failure is surfaced instead of vanishing into a spinner. (This assumes the state is touched from a single isolation context, e.g. the main actor.)

Observe decisions without observing users

Record the operation, the route, the decision reason, the availability state, a duration bucket, the validation outcome, and an error class. Do not record raw prompts, image contents, account text, or model output. An allow-list for analytics properties is safer than hoping everyone remembers what not to log, and free-text diagnostic fields should be redacted before upload.

Route telemetry answers concrete questions. How often is the local model unavailable? How often does automatic mode cross over to cloud? Does one language produce more validation failures? Do users abandon during model preparation? These are diagnostic questions, not performance claims — measure them in the deployed environment before you change policy.

Executable scenarios

Write these as tests and adapt them to your own domain.

Automatic mode respects local capability. Given a short rewrite, a ready on-device model, and no need for external data, when the resolver runs, then it picks the local service and never even evaluates cloud entitlement.

Local-only means no upload. Given the user chose on-device-only mode and the model assets are unavailable, when a classification is requested, then the app reports that local capability is unavailable and makes no cloud request.

Automatic fallback requires consent. Given local execution fails with a recoverable error and cloud is technically available, when upload consent has not been granted, then the app asks for consent or stops — it does not silently transmit the input.

Both routes enforce identifier membership. Given either implementation returns a suggestion referencing an identifier absent from the request, when the validator processes it, then that suggestion becomes "keep unchanged" and cannot reach persistence.

A late result cannot replace a newer one. Given request A is running and the user switches mode, starting request B, when B completes first and A completes later, then only B's matching generation may update the review state.

When this is not worth it

A runtime router is overkill when a feature has one safe execution environment, low device heterogeneity, low volume, and no meaningful difference in privacy, capability, or marginal cost. A simple provider-neutral protocol with a single implementation is enough. Don't add a local model just to claim "offline AI" if deterministic logic solves the task more reliably.

Hybrid execution earns its keep when device eligibility is genuinely mixed, cloud cost is material, upload consent matters, or different operations need different capabilities. At that point the router is not abstraction for its own sake. It is the place where the product makes its promises explicit — because where a request runs is a product decision, as much as which model runs it.

So teams reach for it early, and they are right to — right up until they wire an action to the stream. Here is the tension nobody puts in the demo: streaming and correctness pull in opposite directions. The user sees content as it arrives, but a validated, structured result cannot exist until the response is complete. If you render the stream as though it were final, you are showing the user — and sometimes acting on — data your app has not yet verified.

That is the lie in naive streaming: it presents partial, unvalidated output with the confidence of a finished answer. This piece is about keeping the speed without telling that lie. The trick is to separate what you show from what you trust, and to never let a half-formed response drive a consequence.

Two clocks in one response

A streamed response has two moments that teams love to collapse into one.

The first is first useful output — the instant something worth showing has arrived. This is what makes streaming feel fast, and it is fine to optimise for it. The second is completion and validation — the instant the whole response is present, parses, and conforms to your contract. This is the only moment the result becomes actionable.

Naive code treats the first moment as the second. Everything downstream — enabling a Save button, applying a change, committing structured data, firing a paid side effect — gets attached to the stream instead of to validation. Then the final tokens change the meaning, or the response fails to parse, or a safety filter truncates it, and the app has already acted on something that no longer exists.

Show the surface, gate the action

Split the response into what the user may see while it streams and what the app may do only after it validates.

Progressive display is fine for the human-readable surface. Render tokens as they arrive; let it feel live. But commit is gated. Any control that causes a consequence — apply, save, send, purchase, replace state — stays disabled until the complete response has parsed and validated against your contract.

That single rule kills a whole class of bugs. The user still perceives speed, because text is moving, while the app refuses to let anyone act on unverified data. When validation succeeds, you enable the action and, if you need to, reconcile the display: swap the optimistically streamed text for the validated version. Usually they match. When they do not, the validated version wins, and the user was never able to act on the difference.

Structure is what makes this non-negotiable

If the response is free-form prose that gets shown and forgotten, streaming is nearly risk-free. The danger scales with structure. A response that has to yield an object with an action, an amount and a list of targets cannot be trusted until the closing brace has arrived and the object has validated. Partial JSON is not "a little bit of an object." It is a syntactically incomplete string that will either fail to parse or — with a lenient parser — parse into something subtly wrong, which is worse.

So never act on a partial parse. If you want to give feedback during a structured stream, drive it from a separate, explicitly-partial text field, not from the machine object. Parse defensively, only on completion, and treat a parse or validation failure at the end as a full failure of the response — not a "mostly worked."

Cancellation is part of streaming, not an afterthought

A stream is a long-lived operation on a device that gets interrupted constantly. The user navigates away, backgrounds the app, edits the input, or starts a new request while the old stream is still flowing. Streaming without a currency rule just reintroduces the stale-response problem in a new place.

Bind each stream to the request lifecycle. Give every request an identifier. When a newer request starts, or the user cancels, mark the older stream stale. If a stale stream later completes, it must not be presented and must not commit — its tokens are discarded and the event is recorded as a stale-discard under the same request id. The interface only ever presents and commits the current request's completed, validated result.

Failing mid-stream without leaving a mess

Streams fail halfway. The connection drops, the provider errors after emitting some tokens, a safety filter cuts the response short. If you have been rendering optimistically, you now have partial content on screen and nothing valid behind it. Handle it explicitly:

• Keep partial display content clearly provisional until commit. On failure, discard it and show an honest failure or fallback — not a frozen half-answer.
• Commit nothing. No structured data, no side effect, because the response never validated.
• Make recovery a real state, not an exception swallowed into a spinner that never ends.

And guard side effects with idempotency. A retry after a mid-stream failure must not double-apply a change or double-charge a paid operation. The stream is a delivery mechanism; the effect has to be tied to a completed, validated, idempotent commit.

Model it as a state machine

A pile of booleans (isStreaming, isDone, hasError) drifts out of sync the moment two of them can be true at once. Model the response as a state machine instead, where every transition is explicit and observable:

The two states worth watching in production are discarded — are users routinely outrunning your latency? — and failed after validating — are completed streams failing the contract, which would mean streaming is masking a structure problem you would otherwise have caught.

Given, When, Then

• Given a streamed structured response still in progress, when the user views it, then a progressive preview renders but every consequential action stays disabled until the full response validates.
• Given a stream that fails contract validation at completion, when a preview was already shown, then the app discards the preview, commits nothing, and shows a clear failure or fallback.
• Given the user starts a new request or cancels mid-stream, when the older stream completes later, then its result is neither presented nor committed, and the discard is recorded under the same request id.
• Given partial structured output arriving token by token, when the app parses, then it tolerates incompleteness, acts on nothing until a complete valid parse, and never drives a side effect from a partial object.

When this is not worth it

If responses are short, streaming buys little perceived speed and the machinery is not worth it — wait for completion and render once. If the output is pure display text with no structure and no downstream action, optimistic streaming is essentially free, so skip the commit-gating. The full discipline pays off precisely when the response is structured, drives an action or a paid side effect, or lives in a UI where users routinely cancel and re-ask.

There, the goal is easy to state and easy to get wrong: let the interface feel as fast as the first token, while the application trusts nothing until the last one has arrived and validated.