Darkwood · PHP 8.5 · Flow · Symfony

Flow Tokens

Token optimization is
stream discipline

Symfony Console demo · flow-pipe

Noisy context

→

Pipeline

→

Budgeted signal

The problem

Agentic workflows consume too many tokens.

Engine logs

Verbose, repetitive, ANSI-colored.

Tool outputs

Stack traces, debug lines, boilerplate.

RAG dumps

Useful signal buried in noise.

Context floods the agent - before any reasoning starts.

Wrong instinct

The issue is not word shortening.

Abbreviate vocabulary

waterfall → wf
pipeline → pln
stream → strm

Destroys readability. Saves almost nothing.

Stream discipline

Strip noise
Chunk bounded segments
Compress duplicates
Apply a token budget

Root cause

Uncontrolled context.

flowchart TB
  LOG[Engine logs]
  TOOL[Tool outputs]
  TRACE[Repeated traces]
  CHUNK[Noisy chunks]
  CTX[Agent context window]
  LOG --> CTX
  TOOL --> CTX
  TRACE --> CTX
  CHUNK --> CTX

Data enters as one giant string - not as a disciplined stream.

Thesis

Token optimization is not word shortening.

Token optimization is stream discipline.

How data moves - not how words are spelled.

The Flow angle

Flow is not only about running tasks.

It controls how data moves:

as a pipe

as a stream

as chunks

as measurable context

as budgeted context

for humans · machines · agents

The demo

`flow-pipe`

A Symfony Console project that runs a token pipeline through Flow.

php bin/console app:flow-token-demo \
  --input=flow-engine-log --show-chunks

Deterministic. Local. No external API.

Architecture

Three layers. One pipeline.

flowchart TB
  CMD[FlowTokenDemoCommand\napp:flow-token-demo]
  SVC[TokenPipelineService]
  RUN[TokenPipelineFlowRunner]
  PAR[PipelineExpressionParser]
  REG[PipelineStepRegistry]
  CTX[PipelineContext]
  FIX[FlowFixtureProvider]
  FF[FlowFactory + FiberDriver]
  CMD --> SVC
  SVC --> PAR
  SVC --> RUN
  PAR --> REG
  RUN --> FF
  RUN --> CTX
  FIX --> CTX

Project structure

Application · Infrastructure · Domain

Command

FlowTokenDemoCommand

CLI I/O only

Application

TokenPipelineService

Parse + run facade

Infrastructure

TokenPipelineFlowRunner

Full pipeline via FlowFactory

TokenFlow domain

PipelineExpressionParser

PipelineStepRegistry

PipelineContext

FlowFixtureProvider

Step classes

Pipeline steps

Registered operations - not hardcoded branches.

sourceLoad fixture into stream

strip_ansiRemove escape sequences

remove_noiseDrop debug / trace lines

normalize_whitespaceCollapse blank runs

chunk:300Split into ~300-char segments

compressDeduplicate & collapse repeats

budget:1000Keep chunks until token limit

sinkFlush output & metrics

Pipeline expression

Declarative. Shareable. Readable.

source |> strip_ansi |> remove_noise |> normalize_whitespace
  |> chunk:300 |> compress |> budget:1000 |> sink

PipelineExpressionParser splits on |> and resolves names through PipelineStepRegistry.

Three |> semantics

Same symbol. Three layers.

Expression DSL

String pipeline language

"source |> sink"

PHP 8.5 |>

Step → ClosureJob mapping

$step |> (fn ($s) => ...)

Flow fn()

Runtime composition

$flow->fn($job)

Visual rhythm of the DSL - at Flow composition time.

Flow runtime

Full pipeline. One graph.

$toJob = fn ($s) => new ClosureJob(fn ($ctx) => $s->apply($ctx));

$flow = (new FlowFactory())->createFlow(
    $firstStep |> $toJob,
    ['driver' => $driver],
);

foreach ($remaining as $step) {
    $flow = $flow->fn($step |> $toJob);
}

$flow(new Ip($context));
$flow->await();

TokenPipelineFlowRunner - each step becomes a ClosureJob on PipelineContext.

The result

From flood to signal.

flowchart LR
  N[Noisy input\n17k chars]
  C[Cleaned stream\nstrip + normalize]
  K[Chunked data\n~300 char segments]
  Z[Compressed context\ndeduped lines]
  B[Budgeted output\n≤1000 tokens]
  N --> C --> K --> Z --> B

Fixture: flow-engine-log · heuristic ~1 token / 4 chars

Before / after

`flow-engine-log`

Before

4,488 est. tokens

17,398 characters

After

88 est. tokens

334 characters · 98% reduction

Domain vocabulary intact. Noise and repetition removed.

Signal vs noise

`flow-lexicon` resists compression.

Before

371 est. tokens

After

354 est. tokens · 4.6%

Semantic content carries signal - abbreviating words is not the strategy.

Tomorrow

From CLI demo to async streams.

Non-blocking streams

stream_select

Fibers

Process pipes

PTY / TTY modes

Event loop drivers

Async workers

Flow orchestration layer

Read process stdout as it arrives - not after wait().

Takeaway

Control the stream.
Not the spelling.

Expression

|>

Flow

|>

Budgeted context

Token optimization is stream discipline.

Resources

Go deeper.

darkwood/content/flow-pipe - Symfony demo repo
app:flow-token-demo - run the pipeline locally
darkwood/flow - orchestration package
PHP.Watch - PHP 8.5 pipe operator

Inspired by a LinkedIn post on agentic context - cited at the end of the article.

Noisy logs

→

Stream pipeline

→

Agent-ready context