Skip to content

Commit 4f77cce

Browse files
authored
Refactor: Portable characters & one canvas for training (#57)
* feat: character scoring accuracy, encode progress, and an in-app Hugging Face token * feat: train and apply characters per model, in a resizable editor panel * moved models out of the app * feat: apply characters on Krea 2 via their trained adapter * merged training & studion tabs, new character training nodes, removed ui for character * fix: canvas edits, wiring, node sizing and the models panel, complete ui refactor * fix: exported graphs carry the models and settings needed to rebuild them * fix: report the character encoders, and stop serving a frozen registry cache * surface all models, export/import graph improvement, new connection for lora support * fixed trainer adapter, run state per node, download fix, strength param for adapter * model export fix, ti adapter * fix: training adapter for krea2
1 parent 0952262 commit 4f77cce

150 files changed

Lines changed: 12032 additions & 2987 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CLAUDE.md

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -27,10 +27,10 @@ plus closed models via fal.ai.
2727
>
2828
> **Hugging Face org: [`inlineresearch`](https://huggingface.co/inlineresearch)** - published models and datasets trained with the app.
2929
>
30-
> | Hugging Face repo | What it is |
31-
> | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
32-
> | [`inlineresearch/skin-lora-krea-2-raw`](https://huggingface.co/inlineresearch/skin-lora-krea-2-raw) | Skin-texture LoRA for Krea 2 RAW, trained in the Trainer tab |
33-
> | [`inlineresearch/krea2-skin-lora`](https://huggingface.co/datasets/inlineresearch/krea2-skin-lora) | The 26 image + caption pairs that LoRA trained on |
30+
> | Hugging Face repo | What it is |
31+
> | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
32+
> | [`inlineresearch/skin-lora-krea-2-raw`](https://huggingface.co/inlineresearch/skin-lora-krea-2-raw) | Skin-texture LoRA for Krea 2 RAW, trained on the canvas |
33+
> | [`inlineresearch/krea2-skin-lora`](https://huggingface.co/datasets/inlineresearch/krea2-skin-lora) | The 26 image + caption pairs that LoRA trained on |
3434
>
3535
> Website: [inlinestudio.art](https://inlinestudio.art).
3636
@@ -148,10 +148,11 @@ linking. Generation is Core nodes, installed-extension nodes, and fal nodes on t
148148
(`src/shared/nodes/`); their execution is Core's fal relay.
149149
- **Files & naming.** Components `PascalCase.tsx`, hooks `useX.ts`, one component per file,
150150
feature-foldered views. Keep files under ~300 lines without a good reason.
151-
- **Comments are short.** One or two lines, and only for the **why** a reader can't infer from the
152-
code - a non-obvious constraint, a rejected alternative, an ordering that matters. Never narrate
153-
what the code does or restate the line below. If the reasoning needs paragraphs, it belongs in a
154-
doc, not in the source.
151+
- **Comments are one line.** Not two, not a paragraph, and only for the **why** a reader can't infer
152+
from the code - a non-obvious constraint, a rejected alternative, an ordering that matters. This
153+
applies to file and component header comments too: one line, not a block. Never narrate what the
154+
code does or restate the line below. If the reasoning needs more, it belongs in a doc, not in the
155+
source.
155156
- **Icons, never emoji.** Never use emoji in the UI (no 🎬/🎵/✂/🔊 as glyphs). Use crisp,
156157
consistent line SVG icons (Lucide-style: `viewBox="0 0 24 24"`, `fill="none"`,
157158
`stroke="currentColor"`) that inherit color/size via `currentColor` + a size class. Follow the

README.md

Lines changed: 260 additions & 406 deletions
Large diffs are not rendered by default.

TRAINING.md

Lines changed: 28 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,14 @@ the canvas, see [LoRA training in the README](README.md#lora-training).
66

77
Inline Studio trains LoRAs for **Z-Image**, **Krea 2**, **FLUX.2**, **MiniMax H3** and **LTX-2.5**
88
on your own GPU, with no cloud step and nothing uploaded. Training is cheaper than generating: a
9-
16GB card trains all three image models at 512px, and a LoRA trained at 512 applies at any
10-
generation resolution. The two video models are the exception, and LTX-2.5 wants a 48GB card.
9+
16GB card trains all three image models at 512px, and MiniMax H3 as well if it has 64GB of system
10+
RAM behind it. A LoRA trained at 512 applies at any generation resolution. LTX-2.5 is the one
11+
exception and wants a 48GB card.
1112

1213
**Contents:** [The graph](#the-graph) · [Datasets and outputs](#datasets-and-outputs) ·
1314
[Stop and resume](#stop-and-resume) · [Trigger words](#trigger-words) ·
14-
[Architecture and base model modes](#architecture-and-base-model-modes) · [Install](#install) ·
15-
[Training on clips](#training-on-clips) · [Control LoRAs](#control-loras) ·
15+
[Architecture and base model modes](#architecture-and-base-model-modes) ·
16+
[Training on clips](#training-on-clips) · [Install](#install) · [Control LoRAs](#control-loras) ·
1617
[**Benchmark results**](#benchmark-results) ·
1718
[Dataset and adapter options](#dataset-and-adapter-options) · [Base precision](#base-precision)
1819

@@ -60,7 +61,7 @@ The Trainer's Adjust panel picks the **architecture** first (Z-Image, Krea 2, FL
6061
- **FL2VA** is the only base, and it is undistilled, so there is no adapter and nothing to drift. Put `minimax_h3_fl2va_bf16.safetensors` in `models/diffusion_models/`, train on stills, then wire the LoRA into any of the four H3 nodes. **It has to be the bf16 file.** The smaller `pruned` and `pruned_fp8_scaled` builds generate but cannot train: they ship no timestep path for the modulation basis to be derived from, and they would save nothing anyway, because the base trains at 4-bit whichever file it starts from. The trainer says so rather than failing part way in. It loads on the Reference to Video node too, which uses a different checkpoint file: the two partitions are the same architecture.
6162
- **Stills or short clips.** Drop images and it learns appearance: look, style, character, lighting. Drop video and it learns motion too. Sound is never learned either way, because the audio rows are empty. See [Training on clips](#training-on-clips).
6263
- **The base is 4-bit, always.** H3 is 40GB after the AdaLN factorisation and 11.7GB after quantisation, so full precision is refused rather than offered and then failing. There is no base-precision control for H3 for the same reason.
63-
- **A 24GB card is comfortable and a 16GB card works, slowly.** The run encodes latents and captions in two passes that never overlap, because H3's fp32 video VAE and its 32B conditioner cannot be resident together. On a card that holds the conditioner it peaks at 20.6GB; on one that does not, the conditioner runs on the CPU and the peak drops to 12.7GB while a step goes from 0.6s to 16s. Either way there is about seven minutes of startup, and 64GB of system RAM for the smaller card. See [Benchmark results](#benchmark-results) for the split. The download is about 124GB before any of that.
64+
- **A 24GB card is comfortable and a 16GB card works, slowly.** The run encodes latents and captions in two passes that never overlap, because H3's fp32 video VAE and its 32B conditioner cannot be resident together. On a card that holds the conditioner it peaks at 20.6GB; on one that does not, the conditioner runs on the CPU and the peak drops to 12.7GB while a step goes from 0.6s to 16s. Either way there is about seven minutes of startup, and 64GB of system RAM for the smaller card. See [Benchmark results](#benchmark-results) for the split. The download is about 139GB before any of that.
6465

6566
**Z-Image** is distilled either way:
6667

@@ -187,24 +188,11 @@ well as its own `0001.ref.mp4`, so a downloaded set can be trained on directly.
187188
> guessing: LTX needs Ampere or newer for the same reason generation does (see the runbook below),
188189
> and a 42GB bf16 base does not fit a 15GB card in any case.
189190
190-
### LTX-2.5 generation, measured on an L40S (44.4 GiB)
191-
192-
A 2 second clip at 960x576, distilled, same clip each time. The **cached** column is the one a
193-
sequence actually pays, for every shot after the first.
194-
195-
| | cold render | cached render | peak VRAM |
196-
| ------------------------ | ----------- | ------------- | --------- |
197-
| Streaming weights | 944.8s | 844.3s | 7.71 GiB |
198-
| Transformer resident | 538.7s | 534.2s | 21.90 GiB |
199-
| + shared weight registry | **465.4s** | **229.2s** | 32.19 GiB |
200-
201-
Three fixes got from the first row to the third, and each needed the one before it. The full
202-
write-up is in [docs/ltx-2-5.md](docs/ltx-2-5.md#performance-measured). The remaining cost is the
203-
prompt encoder, reloaded on every render.
191+
### Training VRAM by model
204192

205193
12 steps at rank 16, batch 1, gradient checkpointing on. The number is `torch.cuda.max_memory_allocated`, so leave headroom for the CUDA context and allocator slack.
206194

207-
| Model | Base mode | Res | Base precision | L40S (46GB) | T4 (15GB) |
195+
| Model | Base mode | Res | Base precision | L40S (48GB) | T4 (16GB) |
208196
| ---------- | --------------- | ---- | -------------- | ------------- | ------------- |
209197
| Z-Image | De-Turbo | 512 | bf16 | 13.1GB | 13.4GB |
210198
| Z-Image | De-Turbo | 1024 | bf16 | 14.9GB | out of memory |
@@ -242,7 +230,7 @@ On a card too small for the conditioner it never goes there at all, so the peak
242230

243231
**The bill arrives as time instead.** The conditioner runs on the CPU, and bitsandbytes only quantises on the move to CUDA, so it runs unquantised:
244232

245-
| | L40S (46GB) | L4 (24GB) | T4 (16GB, 64GB RAM) |
233+
| | L40S (48GB) | L4 (24GB) | T4 (16GB, 64GB RAM) |
246234
| ----------------------- | ----------- | --------- | ------------------- |
247235
| Peak VRAM, 512px | 20.6GB | 20.55GB | 12.7GB |
248236
| Seconds per step, 512px | 0.63 | 1.81 | 16.2 |
@@ -259,11 +247,11 @@ A training adapter is free: it is fused into the base before training starts, so
259247

260248
**LTX-2.5 is the opposite case: the base is nearly the whole bill, and it barely fits.** The 22B dev
261249
transformer lands at 38.0GB allocated before a step runs, and training peaks at 42.0GB allocated
262-
against 43.4GB reserved on a 46GB card. That leaves under 3GB of headroom, so a 48GB card is the
250+
against 43.4GB reserved on an L40S. That leaves under 3GB of headroom, so a 48GB card is the
263251
floor rather than a comfort. There is no 4-bit rung to fall back on: the loader refuses to quantise
264252
this architecture, and the trainer now says so instead of dropping the setting silently.
265253

266-
| | L40S (46GB) |
254+
| | L40S (48GB) |
267255
| -------------------------- | ----------- |
268256
| Peak VRAM, 512px | 42.0GB |
269257
| Seconds per step, 512px | 0.67 |
@@ -306,6 +294,22 @@ FLUX.2 is quicker than either. On an L40S, klein Base 4B trains at about 0.3s a
306294

307295
System RAM matters as well. Checkpoints are read tensor by tensor rather than mapped whole, so Krea 2 trains in about 3GB of host RAM. Without that, Linux refuses to map a file larger than physical RAM when there is no swap, and a 26GB checkpoint cannot be opened on a 16GB machine at all.
308296

297+
### LTX-2.5 generation, measured on an L40S
298+
299+
Generation rather than training, kept here because it is the same card and the same clip. A 2 second
300+
clip at 960x576, distilled. The **cached** column is the one a sequence actually pays, for every shot
301+
after the first.
302+
303+
| | cold render | cached render | peak VRAM |
304+
| ------------------------ | ----------- | ------------- | --------- |
305+
| Streaming weights | 944.8s | 844.3s | 7.71 GiB |
306+
| Transformer resident | 538.7s | 534.2s | 21.90 GiB |
307+
| + shared weight registry | **465.4s** | **229.2s** | 32.19 GiB |
308+
309+
Three fixes got from the first row to the third, and each needed the one before it. The full
310+
write-up is in [docs/ltx-2-5.md](docs/ltx-2-5.md#performance-measured). The remaining cost is the
311+
prompt encoder, reloaded on every render.
312+
309313
## Dataset and adapter options
310314

311315
Three settings shape what the adapter learns rather than what it costs:
@@ -322,7 +326,7 @@ Krea 2's base is 26GB at bf16, which is what makes it expensive to fine-tune. Th
322326
- **Full precision (bf16)** forces the unquantized base.
323327
- **4-bit (NF4)** forces the quantized base.
324328

325-
The setting appears for Krea 2 and FLUX.2, but it only pays off on Krea 2. Z-Image has no 4-bit path and does not need one: it trains in about 15 GB at 1024, so bf16 already fits the cards people have. FLUX.2 has the path and gains nothing from it, because klein 4B is smaller than its own text encoder and the peak sits in the caching pass either way, so Auto leaves it at bf16. See [Benchmark results](#benchmark-results).
329+
The setting appears for Krea 2 and FLUX.2, but it only pays off on Krea 2. Z-Image has no 4-bit path and does not need one: it trains in about 15GB at 1024, so bf16 already fits the cards people have. FLUX.2 has the path and gains nothing from it, because klein 4B is smaller than its own text encoder and the peak sits in the caching pass either way, so Auto leaves it at bf16. See [Benchmark results](#benchmark-results).
326330

327331
To keep the peak down, the VAE and text encoder are loaded first, used to cache latents and captions, then freed before the transformer loads, so the two never stack. Which half then owns the peak depends on the model: for Z-Image and Krea 2 it is the transformer, for FLUX.2 klein it is the caching pass. If you do hit an out-of-memory error, lower the training resolution before changing anything else.
328332

@@ -333,22 +337,3 @@ reference: [Krea 2](https://inlinestudio.art/lora-training/krea-2) ·
333337
[Z-Image](https://inlinestudio.art/lora-training/z-image) ·
334338
[FLUX.2](https://inlinestudio.art/lora-training/flux-2). Back to the
335339
[README](README.md), or the [full guide on the site](https://inlinestudio.art/lora-training).
336-
337-
### Runbook: measuring LTX-2.5
338-
339-
Two boxes, because they answer different questions: a **Tesla T4 (16GB, Turing)** and a **48GB
340-
A6000, L40S or A40**. Record for each run the resolved plan the engine logs, peak **allocated and
341-
reserved** VRAM (`nvidia-smi` shows only reserved, so a leaked reference and allocator cache look
342-
identical from outside), host RAM high-water, and wall clock per stage.
343-
344-
1. **Generation, fast mode.** 5 seconds at 1920x1088 and again at 960x544, distilled transformer.
345-
Confirm the MP4 has audio in it.
346-
2. **Generation, quality mode.** The dev transformer plus the distilled LoRA, same two shapes.
347-
3. **Training, Clip LoRA.** Rank 16, 512px, batch 1, gradient checkpointing on, the shortest
348-
grid-legal clip. Peak VRAM and seconds per step.
349-
350-
**On the T4, answer the numerics question before the speed one.** Turing has no bf16 acceleration
351-
and no fp8, and LTX is written for bf16 throughout. An fp16 cast of a bf16-trained 22B transformer
352-
is a numerics change rather than a placement one, and it fails by producing black or NaN frames
353-
rather than by raising. So render one short clip and look at it first. If it is broken, the honest
354-
result is that LTX-2.5 needs Ampere or newer, and that is what the table should say.

core/CLAUDE.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -212,12 +212,12 @@ real codec that moves tensors lives with the model runner.
212212

213213
- **Typed, strict.** `pyright` in strict mode (`[tool.pyright]`, `typeCheckingMode = "strict"`), all of
214214
`src` + `tests`. No silent `Any` leaks across component/graph boundaries.
215-
- **Comments are short.** One or two lines, and only for the **why** a reader can't infer from the
216-
code - a non-obvious constraint, a rejected alternative, an ordering that matters. Module
217-
docstrings: 1-3 sentences. Function docstrings: one line, or none when the signature says it.
215+
- **Comments are one line.** Not two, not a paragraph, and only for the **why** a reader can't infer
216+
from the code - a non-obvious constraint, a rejected alternative, an ordering that matters. Module
217+
docstrings: one sentence. Function docstrings: one line, or none when the signature says it.
218218
Never narrate what the code does, never write an essay in a docstring, never leave a comment that
219-
restates the line below it. If the reasoning genuinely needs paragraphs, it belongs in a doc, not
220-
in the source.
219+
restates the line below it. If the reasoning genuinely needs more, it belongs in a doc, not in
220+
the source.
221221
- **Lint.** `ruff` with `select = ["E", "F", "I", "UP", "B"]`, line length 100, target `py311`.
222222
- **Typed graph, validated before run.** Never execute an unvalidated graph. Edge type-checking
223223
(`graph/validate.py` + `port_satisfies`) rejects bad wiring at submit. New port kinds go in

0 commit comments

Comments
 (0)