Rule delivery: working folder vs. marketplace skill

I had to make a boring infrastructure decision and could not find data on it, so I ran the experiment. Sharing it because two of the results surprised me and one of them is a genuine footgun.

Shortest version: folders win and skills cost more

The decision

You want every colleague's Claude Code sessions to follow the same set of organisational rules. Two ways to deliver them:

Folder — rules live in the working directory (CLAUDE.md). They load in full at session start, every session. But only if you start in that directory.
Marketplace skill — the same text lives in a skill. Works from anywhere, and only the name and description sit in context until the model decides to load the body.

Folder is guaranteed but positional. Skill is portable but conditional. Which one actually holds up?

Setup

One analytical question over a fixed dataset of project records, identical prompt in every arm. Twenty scorable rules: fourteen text rules, six computational, a mix of ALWAYS and NEVER, all picked so a model would not comply by accident. Three are baited — the prompt misspells the company name, the source data uses dd-mm-yyyy and calls its column deadline, while rules demand ISO dates and a different word.

Arms: folder, skill, skill-with-forced-invocation, and a control with no rules at all. The control is not optional — without it you cannot tell "the rule worked" from "the model would have done that anyway".

The trick that made scoring easy. The dataset is rigged so each computational rule shifts the requested total by a unique amount: −1, +64, +128, +256, +512, against a compliant total of 1,028. All subset sums are distinct, so one number tells you exactly which rules were applied. No free-text parsing, no LLM judge, no attribution guesswork. I wrote handwritten compliant and naive answers and validated the scorer against them before spending a single run.

Then I varied the load: 20 rules, then 210, by adding realistic rules that do not apply to this task — at constant total length, so only composition changed. The last round used heterogeneous distractors across six formats, including 22 near-miss rules that contradict a core rule but are scoped elsewhere ("in e-mail use dd-mm-yyyy", "in offers use the full customer name", "in tickets the word deadline is fine").

Results

Six things I learned

1. Rules change the substance, not just the formatting. Without rules: 1 out of 11, and a completely different set of records selected — zero overlap with the correct answer, total 2.5× off.

2. Scale is not the problem. From 20 to 210 rules, almost 8× more text, costs about one rule of compliance. Not one computational rule was ever broken, at any load. That contradicts the popular rule of thumb that big instruction files get ignored wholesale. It matches the only published study I could find on this (McMillan, arXiv 2605.10039 — 1,650 sessions, factorial design), which found no detectable effect of file size, instruction position or file architecture. Worth noting I read the abstract, not the full paper.

3. Near-miss rules did not contaminate. Twenty-two rules deliberately contradicting the core set, scoped to other contexts. Not one bled through in six runs. The model tracks scope reliably. This was designed to break the test and it did not.

4. One rule type is measurably fragile. Five of six failures across all rounds were the same rule: never use word X, where X also appears in the input data. It gets missed, or over-applied — in one run the model stripped an identifier prefix, turning P-401 into 401, because the forbidden word was part of the column name. If you write org rules: a rule forbidding a word that occurs in your input is the most dangerous kind you can write.

5. Strong wording made the rules override the user. This is the footgun. In one run the model deliberately dropped a line the prompt had explicitly asked for, and explained itself: the rules are "non-negotiable", so they take precedence over the requested output format. It even concluded that a footer required by the rules replaced the total the user asked for. Nothing said that. My own phrasing caused it. If your org instructions say "non-negotiable", expect them to eat the task occasionally.

6. Skill delivery costs about twice the context. 84,656 vs 45,957 input tokens for identical rules. The cause is the extra round trip: the model calls a tool to load the skill, then answers, so the full prefix goes over the wire twice. In money it is minor (~13%, most of it cached reads). Against a context window or a daily usage cap it is a factor of two. Progressive disclosure only saves context while the skill is not used.

Where I've landed, provisionally

Folder is ahead. Both channels apply the rules nearly perfectly, but skill delivery costs double the context and produced both structural failures while folder produced none.

I am deliberately not calling that a finding. It is two observations out of three runs. There is a plausible mechanism — in the folder arm the rules sit before the question, in the skill arm they arrive after it as a tool result, and later instructions dominate earlier ones more easily — but a mechanism you can tell a story about is exactly the kind you should be suspicious of.

Two caveats cut the other way:

Skill invocation was 8 out of 8, but that is a ceiling, not a production number. My harness held exactly one skill, with a description ending in "always apply, for every task", on a task that matched it perfectly. In production a rules skill competes with dozens of others on tasks that do not smell like it.
Folder delivery only works if the session starts in the right directory. My test does not measure that at all — and in real life it is probably the dominant failure mode.

What I'd test next

Replace "non-negotiable" with an explicit precedence statement, and see whether the override behaviour disappears. Cheapest run, biggest failure mode.
Repeat the skill arm with the skill invoked up front, to separate position from channel.
Add 15–20 competing skills and vary the task type, to get a realistic invocation rate rather than a ceiling.
Fill in the 60 and 110 rule levels for a curve.

Roughly three runs per variant, about $2 total. The constraint is my daily seat limit, not the money.

If you want to copy the method

The two design choices that did the most work:

A control arm. Half of what looks like compliance is just default behaviour. Mine scored 1 out of 11, which is what made everything else interpretable.
Make the rules change one number, uniquely. Assign each computational rule a distinct power-of-two shift on a single output figure. Then scoring is reading an integer, and attribution is free. It also forces you to design a dataset where the rules genuinely matter — if you cannot construct the deltas, your rules were cosmetic.

And validate the scorer on handwritten answers before you spend anything. Mine had a bug I would otherwise have hand-corrected around, which would have quietly destroyed the determinism claim.

5 comments

Rule delivery: working folder vs. marketplace skill