OUTPAINT.SYS
A conditional diffusion model trained from scratch to extend images beyond their borders — one 128-pixel window at a time. It learned to hide the seam almost perfectly. It has not yet learned to remember what it was painting.
Growing an image sideways
Cropping an image is easy. Un-cropping it means inventing pixels that never existed — and the first pixel you invent sits right next to one that did.
The border is a cliff
Mirroring, clamping or blurring the edge outward produces wallpaper, not world. A real continuation has to extend structure — the branch keeps bending, the fur keeps flowing, the lighting stays consistent. That requires a generative model that actually understands what the visible pixels imply about the invisible ones.
Seven channels in, three out
The ADM-style U-Net sees the noisy canvas (3ch), the clean known pixels (3ch) and the mask (1ch) stacked together. It was trained on random crops of DIV2K + Flickr2K with a curriculum of masks — mostly one-sided bands, because outpainting is the job — predicting v with min-SNR-5 weighting on a cosine schedule.
Keep the known region noisy
The trick that makes seams disappear: during sampling, the known region is re-noised onto the forward trajectory at every DDIM step, so known and unknown pixels always share a noise level. Training reinforces it from the other side — loss is computed over the whole canvas with known pixels down-weighted to 0.1 instead of masked to zero, giving the boundary gradient signal from both directions.
Drift: the model forgets
Recursive expansion only ever sees the last window of committed pixels. Sixteen tiles in, the original subject has scrolled out of context and the panorama free-associates — a lion becomes weather. Halving the stride to keep 75% context doubles the compute and still drifts. This is the open problem.
One expansion, start to finish
The loop below runs once per committed tile — the interactive sim further down replays sixteen of them back to back.
WINDOW + MASK
Slide a 128px window over the canvas frontier. Pixels already committed are 'known' (mask=1); the strip to invent is 'unknown' (mask=0). The webapp picks whichever 4×4-tile window covers the target cell with the most accepted context.
RE-NOISE THE KNOWN REGION
At every sampling step the known pixels are pushed back onto the forward diffusion trajectory at the current noise level, so the network always sees a canvas that looks like its training data — no clean/noisy mismatch at the seam.
CONDITIONAL DENOISE
The U-Net reads 7 channels — noisy canvas, clean context, mask — and predicts v. 30 DDIM steps walk pure noise down to an image that agrees with the context on structure, color and grain.
CANDIDATE BATCH
Diffusion is stochastic: the same context with 4 different seeds gives 4 different futures. Candidates are sampled as one batch, so extra options are nearly free on the GPU.
SELECT + COMMIT
A human picks the best candidate (the webapp's accept/undo loop). Known pixels are composited back exactly — the model never gets to repaint what's already committed.
ADVANCE + REPEAT
Shift the window by the stride (64px default, 32px keeps more context) and go again. Each pass grows the canvas; nothing bounds how far it can extend — except drift.
What the run actually looked like
Full 250k-step training run on a single RTX 3090 — batch 20, BF16, 1.77 it/s, about 1.6 days of GPU time.
Real outputs, unretouched
Everything below came out of the trained checkpoint via sample.py on held-out DIV2K-valid images the model never saw in training.
Candidate batch. Leftmost: the context — a palm frond with the right half masked (shown black). Then four completions from four seeds over the identical context. Each invents a different but plausible continuation of the frond; this one-context-many-futures operation is the core primitive the interactive webapp is built on.
Harder context.Cluttered scene, purple glove. The seeds diverge much more here — some completions stay abstract, one invents new objects. When context under-determines the continuation, candidate variety goes up; that's exactly when having a human pick (or, someday, a scorer) matters most.
Progressive expansion, 8 tiles.Grown 128→640px from the left. The first strips genuinely continue the plant; by the right edge the foliage is pure invention but still on-theme. Note there is no visible seam at any of the eight commit boundaries — the re-noising trick doing its job.
Drift, exhibit A.Same recipe pushed to 16 tiles (128→1152px), seeded with a lion's face. Around tile 4 the lion has left the 64px context window and the panorama starts free-associating: fabric, weather, a red something. Locally every seam is clean; globally the plot is lost. This strip is the one you can replay in EXPANDER.SIM below.
Stride experiment.Same 16 tiles, but committing only 32 new pixels per pass (75% context kept, 2× the compute per distance). Both the lion and the plant hold coherence a little longer per pixel travelled — and both still end somewhere the seed image never implied. Smaller strides slow drift; they don't stop it.
EXPANDER.SIM
Watch the canvas grow tile by tile. Start from the 128px seed crop and press EXPAND to commit the next strip — exactly as the model generated it. Switch strides to compare how fast each run walks away from the lion.
replay mode: every strip you reveal is real model output — a stored 16-tile run of sample.py --mode expand on a DIV2K-valid crop (seed 128px, 30 DDIM steps per tile). The browser is only cropping the finished panorama; live in-browser sampling via ONNX export is a roadmap item. Watch the subject drift as the context window walks away from the original pixels.
Ledger
Honest accounting: what exists and runs, what is unresolved, what comes next.
- Full training run complete: 250k steps, 92M params, EMA checkpoints at 6 milestones.
- Masked DDIM sampler with forward-trajectory re-noising and exact known-pixel compositing.
- Candidate + recursive-expansion demos (sample.py), evaluated on held-out DIV2K-valid.
- FastAPI webapp: unbounded tile canvas, click-to-expand, 4-candidate picker, accept/undo — runs locally against the checkpoint.
- Multi-scale JPEG crop cache (8.9 GB) so Windows dataloader workers keep the 3090 fed.
- Automatic candidate scoring. Today a human picks; no metric yet distinguishes a good continuation from a plausible-looking wrong one.
- Drift. Content decoheres once the subject leaves the context window, at stride 64 and stride 32 alike. Long-range memory or global conditioning is unsolved here.
- Seam metrics: seams look invisible, but there's no quantitative boundary-artifact measure beyond eyeballing.
- ONNX export of the EMA U-Net so EXPANDER.SIM can sample live in the browser instead of replaying stored runs.
- Anchor experiments against drift: keep a downscaled global canvas as an extra conditioning channel.
- Scale past the 128px canvas — either a bigger window or coarse-to-fine expansion.