They know and they do it. Ask them to help calibrate.
Method
Methodological note
For whoever runs a pilot or audits the instrument. How a pilot runs, which rules protect the data, and what it takes to go from an estimate to a placement.
Run a pilot · Evidence and rules · AI Act · Licences
The basic rule
The measurement never feeds an individual appraisal. If it did, people would answer what looks good, and the results would only show the pressure they are under.
Run a pilot
Six weeks, thirty to fifty volunteers.
The pilot first checks that the instrument measures what it should, before saying anything about people. Each step has a gate: you move to the next step only once its condition is met.
1
Frame
Week 1
Charter signed; two reviewers agree on 40 of 48 descriptors.
2
Dry run
Week 2
Eight people think aloud; no gesture where almost everyone does the same.
3
Collect
Weeks 3–4
60 % take part; under ten minutes.
4
Analyse
Week 5
Two reviewers read the same real work the same way: agreement of 0.6 or more.
5
Return, decide
Week 6
Participants first, then the decision to extend or stop.
Each step in detail, with its gate:
FrameWeek 1
- Charter signed by the sponsor, the data protection officer and IT: no link to appraisal, population results only, no group under eight, retention period.
- Staff representatives informed before any round.
- Reference frozen: version, situation pack, role profiles.
GateTwo reviewers, working separately, read at least 40 of the 48 descriptors the same way.
Dry runWeek 2
- Eight people answer in front of an observer and say how they understand each situation.
- Ambiguous situations are rewritten.
GateNo gesture where almost everyone does the same, since such a gesture tells no one apart.
CollectWeeks 3–4
- Open round, two reminders, participation tracked per population only.
- One real piece of work requested from a third of volunteers, reviewed independently by two reviewers.
GateParticipation of 60 % or more; median time under ten minutes.
AnalyseWeek 5
- Distributions, profile shape, would-versus-did gap.
- Agreement between the two reviewers on the reviewed work, measured by Cohen’s kappa: 0 is chance, 1 is perfect agreement.
GateKappa of 0.6 or more, the usual threshold. Below it, rewrite the cells that divide reviewers before going any wider.
Return and decideWeek 6
- Results to participants first, then to the sponsor.
- Two development priorities, chosen from the shape of the results rather than from requests.
GateOne page of results publishable with no group under eight. Decision: extend, or stop.
What the pilot checks on the test gestures
Each gesture has a reading key that turns what the person did into a level. The pilot checks three points.
- Levels build on each other. Whoever does the “higher” thing also does the “lower” ones: for example, whoever replaces an unsourced figure with a sourced one had also marked the figures to check. Checked on the pilot answers with a scale analysis, known as Guttman or Mokken, H ≥ 0.3 per gesture. A gesture that fails is rewritten before it is used for scoring.
- The four job versions are equivalent. At equal practice, the same gesture gives the same level in each version. Checked by comparing level distributions across job types; a version that drifts is rewritten.
- People recognise themselves. “Was that you?” answered yes by 80 % or more, and the “not my job” button used on fewer than one gesture in five. Both are counted in the pilot’s instance; the public site sends nothing. Below that, the job versions need rework.
Reviewed real work remains the only proof of the top level. On a sample, it is the yardstick for gesture levels.
Reading the results
Shares by level, and the gap between saying and doing
Each situation is asked twice: what would you do, then what did you do last time. Results are published as the share of people at each level, per population, with the gap between the two answers, and no average or ranking. Measure at baseline, at six months, then once a year.
How to read the gap. Cross the level of what people did with the distance to what they would do:
Narrow gap
Wide gap
High level
High · narrow · Fluent
High · wide · Environment
They know and do not do it: time, tools, permissions or pressure are in the way. Training will not change that.
Low level
Low · narrow · Learning
They do not know yet. This is the case where training helps.
Low · wide · Aspiration
They say more than they do, at a low level. The question was probably read as an exam: look at the situation before the person.
Four biases and how to limit them
- Social desirability. Ask about the last real case, and make every possible answer a defensible behaviour someone actually has.
- The least fluent overrate themselves most. Never ask people to place themselves on the scale; anchor on real episodes.
- Halo. A good impression on one competency colours the others. Publish the shape of the profile rather than an average.
- Self-selection. Report rates per population and describe the sample openly.
Risks, and what to do about them
| Risk | What to do |
|---|---|
| A manager asks for a team member’s result | The platform cannot display it, the charter forbids it, and the sponsor has said so in public. |
| “An estimate is not a measure” | State the limit yourself. The sample of reviewed work produces a published agreement rate. |
| People learn the right answers | Ask about the last real case, rotate situations at each version, follow the would-versus-did gap. |
| The vocabulary sounds like one vendor | Situations name no product, and descriptors apply to any tool. |
| Measured too often, it becomes a target | Baseline, six months, then yearly. |
Roles and data
Who holds what, and the data rules
Sponsor
Signs the charter, says once in public that results are never used in appraisals, and receives population results only.
Product owner
Owns the reference version, the situations, the role profiles and the calibration.
Reviewer panel
Two to six people. Two reviews of each sampled piece of work, arbitration of disputed cells, yearly review.
IT, data protection, staff reps
Hosting and identity; lawful basis and retention; information before any round.
- Voluntary. Nobody is enrolled by a manager. The participation rate is itself a trust indicator.
- Each person sees their own data. A person’s profile and history are visible to that person only.
- No group under eight. This applies to every cross-cut, sub-totals included. One more group is hidden when a small one could be worked out by subtraction.
- No conversation logs. The platform stores answers to situations and, on request, one piece of work. It never reads AI tool usage. The trace instrument is separate: it runs on the person’s own machine, at their request only.
- Retention. Individual answers are kept until the next measurement, then deleted; aggregates are kept.
- No raw export. Only aggregates leave the platform.
Evidence and rules
From an estimate to a placement
The test gives an estimate. A placement needs evidence from several families that agree, a rule for weighing them, and reviewers whose agreement is measured. Anthropic’s course on the framework names three ways to assess: what a person produces, how they work over time, and what they understand of their own practice. Each family below serves one of the three.
The weight says how much a family counts when evidence is combined: 1.0 for a case handled under observation, 0.1 for what a person says about themselves.
Tracesprocess
Your conversation history, read by the trace instrument. Sees: Description and Discernment, turn by turn. Blind to: All of Transparency, most of Diligence.
weight 0.6
Situationaloutcome + process
A standardised case handled with an assistant, observed and rated. Sees: All twelve, under constraint. Blind to: Real practice over time.
weight 1.0
Artifactsoutcome
Real deliverables and their control traces: who reviewed, what was disclosed, what was verified. Sees: Diligence and Delegation in real conditions. Blind to: The quality of the interaction itself.
weight 0.8
Reflectionreflection
An annotated log, a learning journal, a stated rationale for a delegation or a disclosure. Sees: The reasoning behind the behaviour. Blind to: Whether the stated practice is followed.
weight 0.4
Self-ratingreflection
An unstructured self-assessment or a knowledge quiz. Sees: Knowledge of the rules, and self-perception. Blind to: Actual application. It never places anyone.
weight 0.1
The five rules that decide a level, numbered as in the specification.
R4
Trace evidence alone caps every placement at level 2, whatever the conversations show.
R2
Delegation and Diligence cannot be placed above level 2 without situational, artifact or reflection evidence.
R5
Transparency is never placed from conversation traces. Disclosure happens outside the conversation.
R7
A profile counts as complete only when it draws on at least two of the three approaches.
R9
The trace instrument is self-service. An organisation may offer it, never require it. The result belongs to the person.
All ten rules and the placement method, in the specification
The trace instrument
Your conversations, read on your own machine
A small program to install, an MCP server, reads the export of your conversations on your own machine, whatever the assistant. It detects the eleven indicators of Anthropic’s AI Fluency Index and shows an excerpt for each detection, so you can contest it. Traces alone cap a placement at level 2: Delegation and Diligence happen outside the conversation.
Install, tools, and what leaves your machine
npm install && npm run build
node dist/stdio.js
describe_data_flows
analyze_export → sample_for_rating
record_evidence → build_profile → render_card
The server makes no network call. The only thing that can leave is what a tool returns to your assistant:
analyze_exportStatistics
Counts and rates only: behaviour rates, pattern mix, iteration and artifact effects. No conversation id, no title, no text.
Aggregates
FiltersBefore anything is read
Projects, dates and any term you name are excluded when the export is loaded. An excluded conversation is never read, sampled or counted.
Nothing
sample_for_ratingDefault
Ids, dates, pattern, marker hits, word counts. No title, no text.
Metadata
sample_for_ratingLocal sheet
Writes a rating sheet on your machine. You read it and record the evidence yourself; only the file path comes back.
Nothing
sample_for_ratinginclude_text, opt-in
Redacted turn text, into your assistant and so to its provider. Emails, phone numbers, URLs, IBANs, card-like numbers, handles and IP addresses are replaced. Names and content are not.
Redacted text
render_cardYour result
Written to a local file. It leaves your machine when you share it, and not before.
Nothing
Rule R9Self-service
An organisation may offer the trace instrument. It may not require your export, a sample or a trace-based placement. The result is yours.
Yours
AI literacy obligations
AI Act, Article 4
Article 4 of the EU AI Act asks providers and deployers of AI to ensure a sufficient level of AI literacy among their staff. This reference helps gather proportionate evidence, role by role. It does not declare anyone compliant.
Towards a common reference
Five steps, as the language framework did
A reference becomes common when a level given in Paris means the same thing in Milan or Seoul, whatever the tool. The Common European Framework for languages took about twenty years to get there, in this order.
| Step | What the language framework did | Where this reference stands |
|---|---|---|
| 1 · Graded descriptors | Can-do statements per level, about what people do rather than what they know. | Done: 48 descriptors, one per sub-competency and level. |
| 2 · A rule of evidence | Standard exams saying which tasks prove which level. | Done: five families, ten rules, a placement method. |
| 3 · Empirical validation | Independent reviewers placing the same productions until they agree. | Open: the first pilot must publish its agreement rate. |
| 4 · Local thresholds | Each institution sets the level it requires. | Designed: role profiles set by each organisation. |
| 5 · Neutral governance | A neutral body that versions the framework and has it recognised. | Open: an open specification, dated reviews, adoption by others. |
Licences
What is borrowed, what is open
The 4D framework is the work of Rick Dakan and Joseph Feller with Anthropic, under CC BY-NC-SA 4.0: internal adaptation with attribution is fine, commercial use needs the authors’ permission. The scale, the descriptors, the situations and the rules on this site are a separate layer, under CC BY 4.0. The code is under Apache 2.0.