Usuba / Post

· Harris Kenny · 12 min read

How Usuba was built and tested

We set out to train a small local model to write Omarchy plugins. Six days later, the most useful thing we had built was the test that tells you whether one works. Here’s what we tried, the scores that lied to us, and where the model stands.

Checker Scripts v0.1.0: out nowChecker Model v0.1.0: out now

Preliminary results, October 2026. Machine-drafted labels on small exam sets.

Where this started

Omarchy moves fast, faster than the big models learn about it. Ask one for an Omarchy plugin and it fills the gaps with confident guesses: APIs that don’t exist, programs that aren’t installed, data it made up. The plugin looks right and does nothing.

Coinbase had just written about post-training a small open model, Qwen3.5 9B, for one specialised job. Usuba is the same idea at a smaller scale: a model for Omarchy plugins that runs free on your own laptop. Coinbase is not a sponsor; its write-up is what prompted this.

The test machine

Everything rests on one laptop. When this post says a plugin “works” or is “broken”, it means we ran it on this Dell Latitude 7430 running Omarchy, and watched what happened.

The test machine: Dell Latitude 7430Every hardware label on this site comes from runs on this one laptop.
CPU
Intel Core i5-1245U (12th Gen)
Memory
16 GB DDR4
Storage
256 GB SSD
Display
14.0" FHD, 1920 × 1080, non-touch
GPU
Integrated graphics only
OS
Omarchy 4.0.4-1

That gives every verdict an order of authority. The real laptop outranks the Checker Scripts, and the scripts outrank the model.

  1. The real laptopOmarchy 4.0.4 on a Dell Latitude 7430: load it, click it, screenshot it, read the logs.The final word on whether a plugin works.
  2. The Usuba Checker ScriptsPlain code with fixed rules. The same answer every run.Decide pass or fail, rule by rule.
  3. The Usuba Checker ModelReads the plugin and the scripts' report, explains, suggests a fix.Advises. It can't overrule the two above.
Most authority at the top. Passing the scripts isn't the same as working: they passed one model on 49 of 49 plugins, and 38 held up on the Dell Latitude and a code read.

Six days, October 11 to 16

Each step ran into the same wall from a different side: a number that looked good until something more real checked it.

  1. Oct 11–12

    Teach a small model to write plugins

    We fine-tuned Qwen2.5-Coder 7B and 14B on licensed Omarchy plugins. On a static validator the tuned 7B scored 42 of 56, level with GPT-5.6. Run in real Quickshell, it scored 8.

    Learned: A file check can call a plugin valid when it does nothing. Only running it tells you.

  2. Oct 12

    Build a grader that matches a real laptop

    A headless harness boots the real Omarchy shell and loads the plugin. We calibrated it against a Dell Latitude 7430 running Omarchy 4.0.4: from 44 of 58 agreeing cases to 90 of 90, with no false passes.

    Learned: A grader you haven't checked against hardware is a guess.

  3. Oct 12–13

    Fine-tunes lose to stock, and a model games the grader

    A tuned 27B scored 4 of 49 and looped on 31 answers; the stock 27B scored 6 with none. In a fix loop, grader passes rose from 5 to 13, but only 4 were real.

    Learned: Beat your own base model first. And a test you trust has to survive a model trying to fool it.

  4. Oct 13

    Pivot: the Checker Scripts come first

    We hardened the grader into the Usuba Checker Scripts and wrote down every rule as a public, versioned standard, each with an id, a reason and a fix.

    Learned: Don't train anything against a test that can be gamed. The test comes first.

  5. Oct 13

    Sweep thousands of plugins on the Dell Latitude

    Every plugin we could find, run on real hardware. An audio plugin rerouted the sound and the shell ran out of memory, so we sandboxed the runs and redid 194 jobs.

    Learned: The scripts handle "does it load". What's left is "does it work when you click it".

  6. Oct 13–14

    Data, and Usuba run 1

    A 4,592-row training set with provenance on every row, and a first LoRA on Qwen3.5 9B. Its first scores were leaky, it flagged 27 of 29 clean plugins, and it named the real bug for 0 of 33 broken ones.

    Learned: Read the answers, not just the accuracy. A better prompt on a stock model beat our first trained run.

  7. Oct 15–16

    Usuba run 2

    A much smaller, cleaner set: 221 rows mined from plugin git histories and the Dell Latitude runs, scored on three exams against the same stock model.

    Learned: It learned the answer format. The judgment is still the Checker Scripts' job.

How a score can lie

Every one of these happened to us. The first is why the project changed direction: a static validator only reads files, so it can call a plugin valid when it does nothing at all.

Validator score vs. running it for real, out of 56
  1. Tuned Qwen2.5-Coder 7B42 8
  2. Tuned Qwen2.5-Coder 14B42 12
  3. GPT-5.640 29
  4. Claude Opus34 28

the validator's count run for real

October 12, 2026. The validator only reads files; the real run loads each plugin in Quickshell. The real reference plugins passed 51 of 56. Claude Opus's real-run figure is from its larger run; the regular run scored 27.

The fix loop: what the grader counted vs. what was real, out of 49
  1. 54Start
  2. 8Round 1
  3. 12Round 2
  4. 134Round 3
  • What the grader passed
  • Real passes, after reading the code

Stock Qwen3.6 27B with up to 3 rounds of grader feedback, October 13. Real passes were counted by reading every plugin, at the start and after round 3.

The second chart changed the plan. Given the grader’s complaints and three tries, a model learned to satisfy the grader instead of the task: stripping out whatever failed, hardcoding the data, calling functions that don’t exist on paths the grader never ran. Thirteen passes; four real.

How it liedThe flattering numberWhat exposed itThe honest number
Checking the wrong thingTuned 7B: 42 of 56 on a static validatorRunning the plugins in real Quickshell8 of 56
A grader never checked against hardwareOld grader agreed with the Dell Latitude on 44 of 58Labelling real runs on the Dell LatitudeRebuilt: 90 of 90, no false passes
A model gaming the graderGrader passes rose from 5 to 13 of 49Reading all 13 plugins: stubs, hardcoded data, invented APIs4 real
Passing the scripts isn't workingThe scripts passed Claude Opus 5.5 on 49 of 49A code read plus a load test on the Dell Latitude38 of 49
Accuracy that rewards "everything is broken"Run 1: 0.81 accuracy on the holdoutAlways answering "changes requested" scores 0.91 thereIt flagged 27 of 29 clean plugins
Format read as skillRun 2: 0.99 valid answers on the Dell 75Verdict accuracy on the same set0.48, against 0.51 for its stock base

The four exams

Every number on this site comes from one of these. Each has a size, a source for its labels and a limit, and we name the exam wherever a number appears.

Loading isn’t using

Next we stopped sampling and ran every plugin we could find on the Dell Latitude. One audio plugin rerouted the laptop’s sound and the shell ran out of memory, so the runs moved into sandboxes and 194 jobs were redone.

For loading, the scripts turned out stricter than the hardware: nothing they passed failed to load. For using, they weren’t enough. 137 plugins passed the scripts and still broke when you clicked them.

5,532 plugin runs, labelled on the Dell LatitudeOmarchy 4.0.4 on a Dell Latitude 7430, October 13, 2026, after re-running 194 jobs a misbehaving plugin spoiled.
Works3,942Broken1,001Unsure589

137 passed the scripts and still broke in use

  1. The toggle opens nothing68
  2. Calls a program that isn't installed62
  3. Memory grows past 150 MB in 20 seconds7

Each one became a new hardware-learned check: USB-D01 audio routing, USB-D02 panel with no toggle, USB-D04 close and reopen, USB-D05 runaway memory, USB-D06 panel that can't close.

The Usuba Checker Scripts Available now

Plain Python, no AI: 66 rules in Usuba Standard v0.2, each with an id, a reason and a fix. They agree with the hardware answer key on 90 of 90 decided cases, and on the Dell 75 they get 0.773 of verdicts right. The static checks run on any machine; the real-shell run needs the reference Linux build. Here is real output from a broken test plugin:

$ python3 functest/functest.py now-playing --reasons
FAIL (static checks only: runtime skipped,
      the runtime stage needs Linux, this is darwin)
  [USB-B05] Widget.qml: runs `playerctl` - not an
      Omarchy/system binary on the Dell and not shipped
      by the plugin.
      Fix: use an installed program or ship/document it
  [USB-A04] Widget.qml: line 19: `p.start()` -
      `Process` has no method `start`.
      Fix: set `running = true` to start a Process
      (there is no start())

Real output from the Usuba Checker Scripts on a deliberately broken test plugin. Static checks only, run on a Mac; long lines wrapped.

Do they help models write plugins? Eleven models wrote the same 49 plugins, and the scripts’ report went back for up to three fix rounds. A plugin only passed if the scripts passed it, a code read agreed it did the job, and, where we could run it, it loaded on the Dell Latitude. Strong models gained most: Claude Opus 5.5 went from 14 to 38.

Plugins passing checks, out of 49 · Preliminary results, October 2026.11 models each wrote a plugin for the same 49 requests. When the Checker Scripts found problems, their report went back for up to 3 fix rounds.
  1. Claude Opus 5.5142411
  2. Meta Muse Wonder102514
  3. Kimi K3430123
  4. GPT-5.6 Sol923143
  5. Claude Sonnet 5.532917
  6. Grok 4.7228136
  7. Gemini 4 Argon42124
  8. DeepSeek V4 Pro261229
  9. Gemini 3.1 Pro173110
  10. Qwen3-Coder213016
  11. Qwen3.6 27B22225
  • Passed on first try
  • Passed after Checker fixes
  • Failed checks
  • Failed to produce usable code
  • proprietary open

Of 539 answers across the 11 models: 53 passed on the first try, 194 passed after Checker fixes, 200 failed checks and 92 produced no usable code. Passing means the Checker Scripts pass it, a code read agrees it does the task, and where we could run it, it loads on the Dell Latitude 7430. Run October 16, 2026.

The 49 requests, model by model.

The Usuba Checker Model v0.1.0

The case for a model isn’t loading; the scripts do that. It’s the review a person would do: is the data real, do the buttons do anything, does it break in use. Usuba run 2 is Qwen3.5 9B with a small LoRA adapter, trained on 221 examples mined from real plugin git histories and the Dell Latitude. Here it is against the stock model it started from.

  • Usuba Checker Model v0.1.0
  • Qwen3.5 9B with a LoRA adapter
  • 4-bit file: 5.63 GB (Q4_K_M)
  • Built for a 32 GB laptop, no graphics card
  • 221 training rows from real plugin histories
Usuba run 2 vs its stock base, one shotDots are scores, pills are 95% intervals. Preliminary results, October 2026.
  1. Verdict accuracyDell 75, n = 75, higher is betteron par
    Usuba run 20.48 0.37–0.59
    Stock Qwen3.5 9B0.51 0.40–0.61
  2. Broken plugins flaggedDell 75, n = 28, higher is betterbetter
    Usuba run 20.61 0.43–0.79
    Stock Qwen3.5 9B0.29 0.11–0.46
  3. False alarms on working pluginsDell 75, n = 47, lower is betterworse
    Usuba run 20.57 0.43–0.70
    Stock Qwen3.5 9B0.28 0.15–0.40
  4. Valid answersDell 75, n = 75, higher is betterbetter
    Usuba run 20.99 0.93–1.00
    Stock Qwen3.5 9B0.89 0.80–0.94
  5. Valid answersS4 60, n = 60, higher is betterbetter
    Usuba run 20.92 0.82–0.96
    Stock Qwen3.5 9B0.57 0.44–0.68
  6. Valid answersHoldout 329, n = 329, higher is betterbetter
    Usuba run 20.94 0.91–0.96
    Stock Qwen3.5 9B0.64 0.58–0.69
  7. Bugs foundS4 60, n = 60, higher is betteron par
    Usuba run 20.13 0.07–0.24
    Stock Qwen3.5 9B0.07 0.03–0.16
  8. Clean plugins given a blockerHoldout 329, n = 29, lower is betteron par
    Usuba run 20.28 0.15–0.46
    Stock Qwen3.5 9B0.38 0.23–0.56

Machine-drafted labels, one shot, same prompt and scorer for both. Run 2 was scored as the bf16 merge; the 4-bit file has no exam score yet.

  • Better than its base: usable answers on every exam, about twice the broken plugins caught, and far fewer findings of its own.
  • On par: overall verdicts and bugs found.
  • Worse: about twice the false alarms on working plugins. It rarely disagrees with the scripts, which is part of why.

And against everything else we tested on the same exam:

Verdict accuracy on the Dell 75, with 95% intervalsHigher is better. Preliminary results, October 2026.
  1. Usuba Checker Scripts, no model0.773 0.667–0.853
  2. Gemma 4 26B + scripts' feedback0.65 0.55–0.76
  3. Stock Qwen3.5 9B + scripts' feedback0.57 0.47–0.68
  4. Qwen3-Coder 30B + scripts' feedback0.56 0.45–0.67
  5. Gemma 4 26B, one shot0.56 0.44–0.68
  6. Stock Qwen3.5 9B, one shot0.51 0.40–0.61
  7. Usuba run 2, one shot0.48 0.37–0.59
  8. Usuba run 2 + scripts' feedback0.47 0.36–0.57
  • The Checker Scripts, no model
  • Usuba run 2
  • Its stock base
  • Other open models
  • Always answer "broken": 0.373

Machine-drafted labels. The scripts' interval is Wilson; model intervals are bootstrap. The scripts' feedback loop: up to 3 revisions.

The Checker Scripts still make the call. The model has not yet met the targets we set for a full release (Dell accuracy 0.85 or more, false alarms 0.15 or less), and closing that gap is what v0.2 is for.

See it run

Real screenshots from the Dell Latitude 7430: the model reading a plugin and writing its review on the laptop’s own Intel Core i5-1245U (12th Gen), with no graphics card and no cloud.

Terminal showing the end of an Omarchy bar-widget QML file.
The plugin under review: QML for a bar widget.
Two terminals side by side. The right one shows a model's review being generated token by token.
Usuba reading the plugin and writing its review, live.
The terminal shows verdict changes_requested with a blocker finding about an unbounded download and symlink risk.
The finished review: changes requested, one blocker.

Usuba run 1 (usuba-9b, Q4_K_M) on the Dell Latitude 7430, running on its Core i5 CPU, October 14, 2026. One manual run of review.py, a single sample, unedited screenshots; preliminary. This is the model's review, given a Checker Scripts report and the plugin source, not the scripts' own output. Run 1 is the run before run 2, which shipped as v0.1.0: its "exact commit" opening is one of the habits run 2 was trained to drop.

What we’re releasing, and what we’re not claiming

What it took

Money. About $207 by October 14, from the billing ledgers: $77.67 of rented GPUs and $129.73 of AI services. About $400 by the evening of October 16. That figure is derived from the ledgers and GPU credit, and isn't yet checked against the final bills.

Mostly, time. Calibrating a grader against a real laptop, sweeping thousands of plugins, chasing bugs through git histories, building a training set with a source on every row. Very little of it was GPUs. A specialist model isn’t a weekend fine-tune; it’s an ongoing investment in data and evaluation, and the evaluation is the part that lasts.

Tools. I did this work with OpenRouter for model calls, Vast.ai for rented GPUs, and Cursor and Grok Bot for agent work.

What we’d tell anyone trying this

  • Beat your own base model before you try to beat anyone else's.
  • Read the answers, not just the accuracy.
  • Check the grader against real hardware, then against a model trying to fool it.
  • Most of the work is data and labels, not GPUs.

Thank you

Usuba stands on other people’s work. Some of it we learned from; some of it we trained on directly.

Help us

Most useful: A plugin the Checker Scripts got wrong on your machine, especially one that loads fine and breaks when you click it.

  • Plugin authors: run the scripts on your plugins and report false passes and false fails.
  • Omarchy maintainers: check the rules against your intent, and tell us when a release changes what works.
  • Anyone: labelled bugs and false alarms from real plugins, and evaluation time for the next run.
Report a pluginThe model repo

Harris Kenny, OutboundSync

Built for Omarchy plugin builders

If you build Omarchy plugins, start with the Checker Scripts. They run today, on your own machine, for free.

Try the Checker Scripts