The shared model runtime

The local-model runtime shared by every kumulativ desktop app — smartreader, smartwriter, smartdiary. One Go module, one package: engine.

The problem it solves

Each app used to carry its own copy of the same six files, its own models/ and engines/ directories, and its own llama-server sidecar. Installing a second app meant downloading the same weights a second time. Opening two apps meant two servers, each holding a model in graphics memory, with no idea of each other. And on macOS, where nothing ties a child to its parent, a crashed app could leave a server holding five gigabytes with no window left to close.

What it does instead

One store. Every app reads and writes <UserConfigDir>/kumulativ/:

kumulativ/
  models/      every GGUF any app has fetched — downloaded once, per machine
  engines/     the pinned llama.cpp build for this machine
  run/
    server.json   who is serving, on which port, and which app owns the process
    server.lock   held while deciding whether to start one
    presets.ini   per-model settings the server is started with

On first use an app moves anything found in its old private models/ and engines/ into the store — a rename, not a copy — so upgrading never re-downloads what is already on disk.

One server. The first app that needs a model starts a single llama-server in router mode (--models-dir), which serves every model in the store from one loopback port. A request names its model; the router loads it on first use, keeps at most one resident (--models-max 1), and puts it to sleep after a quiet minute (--sleep-idle-seconds), freeing its memory. All three apps run the same Qwen3.5 family, so a second app finds the weights already there; a model from another family would be swapped in on the next question, and nothing is held while nobody is asking — whichever app’s question it was.

Per-model settings (context window, reasoning off) live in presets.ini, written from the catalog in config.go when the server starts, so a model fetched by one app is served with the right settings whichever app started the server.

One owner. The app that started the server owns it: the server is that process’s child, enrolled in a Job Object on Windows so it cannot outlive it, and stopped by Runtime.Stop on a clean exit. Every other app finds it through run/server.json, checks that it is alive, and uses it as a client. When the owner exits the server goes with it; the next sibling that asks a question notices, takes the lock, and starts a fresh one — a few seconds of reload, paid once, and never two servers. On macOS and Linux, where a crash can leave a server running with no owner, the next app to start adopts it rather than starting another, and stops it at its own exit.

Using it from an app

import "github.com/stijlmassi/kumulativ-core/engine"

rt := engine.New(engine.Config{
    App:        "smartreader",
    Family:     engine.Qwen35,           // the family all three apps run
    LegacyDirs: []string{oldConfigDir},  // models/ and engines/ move to the store
})
rt.OnChange(func(up bool) { /* status chip */ })

// first run: both resumable, both report Progress
rt.EnsureEngine(ctx, onProgress)
rt.EnsureWeights(ctx, onProgress)

// before a question: server up, this app's model resident
rt.Ensure(ctx, onProgress)
text, err := rt.ChatStream(ctx, msgs, 0.3, func(tok string) { /* render */ })

// at exit: stops the server if this process owns it, else forgets it
rt.Stop()

Status() reports whether this app’s model is resident and on which backend (cuda, metal, cpu); a sleeping model reports as not running, because it holds no memory and the chip in the nav bar exists to say so.

Tests use engine.NewTestRuntime(cfg, fakeServerURL): a runtime pinned to a URL that never spawns, never polls and never stops anything.

Environment

Pinned build

LlamaRelease in binaries.go is b10146. The router flags this package relies on (--models-dir, --models-preset, --models-max, --sleep-idle-seconds) and the per-model preset keys (c, reasoning) were verified against that build; re-verify before bumping.