Darkwood / local inference infrastructure

NoLife Models

Exploring Local AI Runtime Infrastructure

@matyo91

Thesis

We are moving from “chatbot APIs” to AI runtime infrastructure.

Catalogs, local engines, evaluation harnesses, and exportable traces - the same primitives ops expects from any serious runtime.

01 | thesis @matyo91

Agenda

Five arcs - one pipeline.

Catalogs

models.dev · modalities · cost · context.

Runtimes

Ollama as HTTP surface · future adapters.

NoLife Models

Compare · benchmark · export.

Kronk

Embedded inference - not “better Ollama”.

Governance

Ports, traces, reproducibility.

Outlook

Orchestration & embedded edge.

02 | agenda @matyo91

Part 1 - Catalog plane

models.dev is a contract surface for the model economy.

Structured metadata: modalities, limits, pricing tiers, reasoning flags.

Provider
→
Model record
→
Tooling & SDKs
03 | models.dev @matyo91

Live - models.dev

models.dev

Public catalog in the browser. Some sites block iframes; if the frame is blank, open the URL in a new tab.

04 | models.dev · site @matyo91

Comparator plane

modles - local browser lab for rows from the catalog.

Filter, sort by $/M, context, capabilities.

05 | modles @matyo91

Live - modles

modles

Local comparator lab for catalog rows (this deck expects it on port 8001).

06 | modles · site @matyo91

Analogy - shipping vs inference

Docker ecosystem ↔ AI runtime ecosystem

Docker Hubmodels.dev
ImagesModel artifacts (GGUF, weights)
dockerdOllama / Kronk / vLLM…
KubernetesAgent & batch orchestration
ObservabilityTraces · exports · benchmarks
containerd / OCILocalModelRuntimeInterface
07 | analogy @matyo91

Part 2 - Process-local HTTP runtime

Ollama exposes models as local HTTP services.

Inventory

/api/tags - what’s pulled, which quant.

Inference

/api/generate - non-stream MVP in NoLife.

Metadata

Load time, eval counts, wall clock - rudimentary SRE signals.

08 | ollama @matyo91

Live - NoLife Models

Catalog explorer

Merged models.dev API + JSON overrides · live filters · modalities · $/M · context windows.

09 | demo · catalog @matyo91

Live - NoLife Models

Installed local models

Runtime discovery against your daemon - tags, quant, what’s actually callable.

10 | demo · ollama @matyo91

Part 3 - Reference app

NoLife Models - Symfony 8 + Symfony UX

DDD / hexagonal core: catalog ports, LocalModelRuntimeInterface, exporters, session-backed results.

UX Live

Catalog explorer (Live Component) - filters without round-trips.

Turbo

Fast morphing between compare / results.

11 | nolife stack @matyo91

Hexagonal slice

Application ↔ ports ↔ adapters

flowchart TB
  subgraph UI["UserInterface"]
    TW["Twig + Forms"]
    LC["LiveComponent"]
  end
  subgraph APP["Application"]
    RC["Compare handler"]
    RB["Benchmark handler"]
  end
  subgraph DOM["Domain"]
    LR(("LocalModelRuntime"))
    CR(("CatalogRepository"))
    EX(("Exporter"))
  end
  subgraph INF["Infrastructure"]
    OL["Ollama runtime"]
    MD["models.dev cache"]
    SE["Session + files"]
  end
  TW --> RC
  LC --> RC
  RC --> LR
  RC --> CR
  RB --> LR
  RB --> CR
  OL -.-> LR
  MD -.-> CR
  SE -.-> EX
12 | architecture @matyo91

Runtime port

LocalModelRuntimeInterface

interface LocalModelRuntimeInterface
{
    /** @return list<LocalModel> */
    public function listLocalModels(): array;

    public function generate(GeneratePromptCommand $command): ModelInferenceResult;
}
13 | port @matyo91

Live - NoLife Models

Compare two models

Same prompt · two tags · wall clock + token-ish metrics - engineering comparison, not vibes.

14 | demo · compare @matyo91

Evaluation plane

Latency is cheap to measure. Quality is expensive.

Metrics

Duration, load, eval counts - reproducible numbers.

Reasoning

Side-by-side text for human triage.

Next

Structured rubrics · model grading pipelines.

15 | compare concepts @matyo91

Live - NoLife Models

Benchmark suite

Micro-cases (EN/FR) · deterministic prompts · matrix results - orchestration over the same runtime port.

16 | demo · benchmark @matyo91

Orchestration

Benchmark flow

flowchart LR
  S[Suite definition] --> M[Model A / Model B]
  M --> P[Prompt cases]
  P --> R[Runtime generate]
  R --> T[Timing + text capture]
  T --> X[Matrix + exports]
17 | benchmark flow @matyo91

Live - NoLife Models

Exports

JSON · CSV · Markdown - stable schema/version fields for downstream tooling.

18 | demo · exports @matyo91

Governance

Inference without artifacts is faith, not engineering.

Audit trail

Exports + session IDs - what ran, when, with which tags.

Runtime governance

Policies on top of the same port: quotas, routing, kill switches.

19 | observability @matyo91

Part 4 - Embedded inference

Kronk is not “better Ollama”.

Ollama: models as local HTTP services. Kronk: push inference into the application process - GGUF via llama.cpp, streaming, OpenAI-compatible ergonomics where it fits.

20 | kronk @matyo91

Embedded stack

Primitives

GGUF

Single-file weights - ship like any binary asset.

llama.cpp

CPU/GPU inference core - predictable foot-print.

Streaming

Token deltas - UX + back-pressure aware servers.

OpenAI-shaped APIs

Interchange clients - ecosystem leverage without cloud lock-in.

21 | embedded primitives @matyo91

End-to-end

Catalog → runtime → evaluation → export

flowchart LR
  C[Catalog plane] --> R[Runtime plane]
  R --> E[Evaluation plane]
  E --> O[Observability plane]
22 | pipeline @matyo91

Part 5 - Outlook

Multi-runtime clusters · policy-aware routing · edge bundles with embedded weights.

The same governance patterns (ports, traces, exports) port from Symfony apps to agents to on-device runtimes.

23 | future @matyo91

Closing

We are moving from chatbot APIs to AI runtime infrastructure.

NoLife Models is a thin, opinionated slice through that stack - catalogs, local engines, benchmarks, exports - built as a real Symfony product surface.

Catalog
+
Runtime port
+
Evaluation
+
Artifacts
24 | outro @matyo91