AI Fluency Scan Specification

An open assessment reference for AI fluency, built on the AI Fluency Framework (4D) of Rick Dakan and Joseph Feller, developed with Anthropic. The framework says what to look for. This specification adds what an assessment needs, levels, falsifiable descriptors, a rule of evidence, a placement method, a calibration requirement, and serves it over MCP so that any client, with any model, evaluates against the same versioned reference.

This specification is written in English. The descriptors are available in French in the grid and in the reference file. referential.json

1 · Why an assessment reference, and why now

Anthropic's framework is taught (the AI Fluency: Framework & Foundations course), observed at population scale (the AI Fluency Index, February 2026: 24 behaviours, 11 observable in conversation, 9,830 conversations) and, since July 2026, reflected back to individual users inside Claude (the monthly recap on the Reflect page, framed around the four competencies). What none of these does is place a person at a level, with a rule for what counts as proof, in a way that means the same thing across tools and evaluators. The Index says so itself: it measures conversations, not people; it sees 11 behaviours out of 24; none of the 11 belongs to Diligence.

The parallel is the CEFR for languages. A grammar describes a language; the CEFR lets "B2" mean the same thing in Paris, Milan and Seoul. The 4D framework is the grammar. This specification is the attempt at the CEFR: graded descriptors, a rule of evidence, empirical validation by inter-rater agreement, separation between the reference and the thresholds each organisation sets, and neutral governance, in that order.

Two things make this timely. Article 4 of the EU AI Act (AI literacy of staff) has been enforceable since 2 August 2026 and requires documented, role-proportionate evidence; and the native Claude features are consumer-only (not on Team or Enterprise plans), observational, and Claude-specific by construction.

2 · The reference layer, fidelity is the product

Identifiers are canonical English strings and are never renamed, merged or reordered. Definitions are quoted from Anthropic's AI Fluency terminology sheet. Three interaction modes are recorded on every piece of evidence: Automation ("AI performs tasks independently, but based on direct human instructions"), Augmentation ("AI and human co-define and co-execute tasks in an iterative way"), Agency ("Human configures AI to independently perform future tasks, including for others").

Two loops. The framework's teaching course presents the 4Ds as two nested loops: Description ↔ Discernment, the short loop inside a conversation, turn by turn, visible in traces; and Delegation ↔ Diligence, the long loop before and after the conversation, essentially invisible in traces. This drives the rule of evidence: an assessment built on traces alone measures the short loop and leaves in the dark the two competencies that carry the risk.

3 · The scale: four levels, forty-eight descriptors

Writing rules for every descriptor: one sentence; one observable action verb with "the person" as subject; no adverb of degree; falsifiable on real work; no product or version named; each level includes the previous one and states what is new; each criterion is clearly distinguishable from its neighbours and the progression between levels is logical and achievable.

Correspondence with the framework's own teaching. In Teaching AI Fluency, lesson 4, educators build rubrics with three example performance levels, emerging, developing, proficient, and are told to "transform abstract concepts into observable behaviors". This scale keeps that progression as L1-L3 (Starter ≈ emerging, Practitioner ≈ developing, Proficient ≈ proficient) and adds L4 Champion for what a course does not need to cover: the person codifies a practice in reusable form and makes it usable by others. L4 is defined at the level of the person, not of an organisation: it is reached at home as well as at work, the moment a practice is written down and handed to someone else.

Changed in v0.7. Level labels name a posture of practice, each read with its one-line summary. Foundation, Operator, Autonomous and Reference became Starter, Practitioner, Proficient and Champion (French: Initié, Praticien, Confirmé, Référent). A label describes a practice at a date, never a grade: results are not used in appraisal. Identifiers L1 to L4 and the 48 descriptors are unchanged; French descriptors were added.

Delegation Deciding on what work should be done by humans, what work should be done by AI, and coordinating between them.

Description Effectively communicating with AI systems through clear output specification and behavioural guidance.

Discernment Thoughtfully and critically evaluating AI outputs, processes, behaviors and interactions.

Diligence Using AI responsibly and ethically through transparent and accountable practices.

4 · The rule of evidence

The three approaches come from the framework itself. Teaching AI Fluency, lesson 4, names three complementary assessment approaches, outcome-based ("what students produce through AI collaboration"), process-based ("examining how students work with AI over time") and reflection-based ("evaluating metacognitive awareness"), and holds that combining all three "provides the most complete picture", on the principle that "observable actions and concrete artifacts are more reliable than assumptions about understanding". Our evidence families are instruments of those approaches, not a competing taxonomy: each declares the approach it serves, and rule R7 requires a complete profile to draw on at least two of the three.

Reflection is a pillar of the framework's own pedagogy, not a weak substitute for observation, it is the only practical route to the Delegation-Diligence loop outside a situational case. It is kept distinct from self_rating, which never places anyone and is recorded for the gap it reveals against observed evidence.

Why the trace instrument exists. The course asks educators, as a feasibility question, whether process-based assessment is even possible for them, "can you access and evaluate chat logs?". For most, the answer is no. The trace instrument is what makes that approach practical at scale, across assistants; it remains one family among several.

Combination rules.

• R1 · A placement at L2 or above requires at least one piece of evidence that is not a self_rating.
• R2 · A placement on any delegation.* or diligence.* sub-component requires at least one situational, artifact or reflection piece of evidence.
• R3 · A placement at L4 requires at least two families of evidence and one human review.
• R4 · Trace evidence alone caps the placement at L2 on every sub-component.
• R5 · diligence.transparency is never placed from trace evidence; trace signals for it are recorded as anecdotal only.
• R6 · Absence of a behaviour is not evidence of a lower level unless the situation called for the behaviour; otherwise the evidence is recorded as 'not elicited'.
• R7 · A profile is reported as complete only when its evidence draws on at least two of the three official approaches (outcome, process, reflection); otherwise it is reported as partial, naming the missing approach.
• R8 · Every placement is returned with the next level's descriptor as the demonstration to aim for, assessment is a learning opportunity, not just measurement.
• R9 · Trace evidence is self-service. An organisation may offer the trace instrument; it may not require an export, a sample or a trace-based placement from anyone. A placement that rests on trace evidence obtained under requirement is invalid.
• R10 · Self-report subtypes of the situational family (judgement_test, behavioural_tendency) count as a single family for convergence, never satisfy the human review required by R3, and are reported as an estimate labelled 'not calibrated' until the instrument's agreement statistics are published. When both steps are answered, the cell estimate is the practice level; the recognition level and the gap between the two are reported alongside it.

Every piece of evidence carries: the sub-component(s) targeted, the level observed (L1-L4 or not elicited), the family, the interaction mode, a short justification, a confidence, the rater (model, human, both), the versions of instrument, framework and scale, the date, and the consent scope (self or shared).

The trace instrument detects the eleven conversation-observable behaviours of Anthropic's AI Fluency Index, with a partial state and a mandatory excerpt, from exports of any assistant (Claude, ChatGPT, generic), and reports the person's own rates against the published baseline:

5 · Placement

A placement is computed per person, per sub-component, from the evidence of the last six months (default), with a linear temporal decay to half weight at the window's end. The retained level is the highest level for which at least two concordant pieces of evidence from different families exist; otherwise the level immediately below. Confidence is high with three or more pieces of evidence from two or more families within one level of each other; medium with two pieces, or one family only, or a one-level spread; low with one piece or a spread of two levels or more.

Every placement is returned with the next level's descriptor and what it would take to demonstrate it (rule R8), the course's principle that "assessment should be a learning opportunity, not just measurement", made mechanical. The standard output is a profile: twelve placements, twelve confidences, and the evidence behind each. Competency-level display is a distribution of the three sub-component placements, never a mean. No single global score is computed by default. The shareable card shows the 4×3 grid with confidence, the interaction-pattern mix against the Anthropic Economic Index baseline, and a working-style archetype derived from the grid, never a number that could travel alone. The archetype has two visible ingredients: the height of the measured cells (a flat profile is named by the scale's own level label) and the shape (the competency that leads by 0.6 level or more). Its blind spot always names the weakest competency and that cell's next descriptor.

Estimate and consolidated level (v0.7). Where a single approach is available, as with the scenario test, the output is an estimate, never a placement. Per sub-component, when both answers are given the estimate is the lower of the two (v0.6 kept the practice answer); with one answer only, that answer, at low confidence; answers two levels or more apart also give low confidence. v0.7 adds one summary, the consolidated level: the highest level n such that at least 9 of the 12 sub-components are at n or above and no measured sub-component is below n−1; with fewer than 9 measured there is none. It is a floor, not a mean, and it is never shown without the grid, the profile shape and the blind spot.

6 · Calibration, the release criterion

A descriptor set, a classifier or a situational case enters production only when two independent raters agree on its calibration set of real, anonymised productions already placed by two humans. Initial threshold: Cohen's κ ≥ 0.6, to be revised empirically. Four release checks, adapted from the rubric checks taught in lesson 4 of Teaching AI Fluency, apply to every descriptor: each criterion is clearly distinguishable from its neighbours; the progression between levels is logical and achievable; each indicator is an observable behaviour rather than an assumption about understanding; and the wording is clear to the person being assessed, not only to the assessor. The agreement rate is published with each instrument version; descriptors that divide raters are rewritten, not kept. The test for everything this reference produces: two evaluators observing the same person on the same work must place them at the same level. Until that test is passed on a real sample, the reference describes, it does not evaluate.

Gesture format and its keys (since 2 October 2026). The scenario test at /test now uses gestures; the twelve-situation version stays at /test-classic so that existing result links keep working: instead of four options to read, the person acts on an artefact. Twelve gestures cover the twelve sub-components, one each, in four role-family dressings (manager; analyses and documents; coordination; customer-facing) that share one structure and one key. Three lengths: quick (four gestures, one per component), default (eight, two per component: sort, choose the tool, complete, steer, point, cut, decide before sending, say it) and full map (all twelve, adding frame, set the posture, ask why and before it goes out). Sort (task delegation): five pieces of a request (understand it, gather, write, check, talk to people) into me / the assistant / a human exchange, scored by type of error, reading only what goes to the assistant (“me” and “a human exchange” are both human): a judgement piece to the assistant, or nothing to it → L1; writing to the assistant with every judgement piece kept → L3; otherwise L2. Gathering is open (me or the assistant). Choose the tool (platform awareness): four tasks, shown in a shuffled order, to the files-connected assistant / chat assistant / business tool / no AI (the writing task accepts either assistant); three or more to one AI destination or ≤1 fitting → L1, 2 → L2, 3–4 → L3. Complete (product description): up to three additions to a bare prompt from eight, three of them judgement items (what not to promise, objections, a past example); “ask me first” counts as an addition but not as judgement, since it belongs to process. ≤1 addition → L1, two judgement items → L3, otherwise L2. Steer (process description, replaces Order since 13 October 2026): the assistant answers at once with a final figure; up to two instructions for the next message, among the final answer, a format, two ways of sequencing the work, questions first and a stop before the answer; the final answer → L1, a sequence and a control (questions or a stop) → L3, one of them → L2, neither → L1. Frame (problem awareness): before opening the assistant, settle three things among six: what the result is for, what is already known and missing, what the result must satisfy, and three execution items (length, layout, tone); no framing item → L1, one or two → L2, all three → L3. The reframing part of the L3 descriptor is not observed. Set the posture (performance description): before the first request, add nothing, a format, a tone for the deliverable, or a posture (challenge my draft, say where you are unsure); posture → L3; otherwise the answer flatters and the person either asks it to change approach → L2, or uses it or adds content → L1. Ask why (process discernment): a recommendation with three reasoning steps, one resting on a flawed premise or an unsupported assumption; tap the step to challenge, then keep / ask how / have it redone; flawed step and redo → L3, no step and keep → L1, otherwise L2. Before it goes out (deployment diligence): four switches (read it through, layout and typos, two named checks), then an error is found after sending: blame the assistant / correct quietly / correct openly and add the check; both named checks and correcting openly → L3, any check → L2, none → L1. Point (product discernment): mark the claims to check among five (three load-bearing, one judgement nobody can verify, one source claim), then keep / cut / replace with a sourced figure; none marked → L1 whatever follows, keep → L2, cut or replace with ≥2 load-bearing marked → L3, else L2; marking the judgement or the source claim is neutral. Cut (performance discernment, since 5 October 2026; previously filed under process): where the person stops a looping exchange and what they do next; keeps going or rephrases → L1, says it is stuck → L2, changes the set-up (full context, questioning, or doing it themselves) before the third round → L3, after it → L2. Decide before sending (creation diligence): switches, in a shuffled order, before pasting personal data; anonymise and check the policy are the two checks: neither → L1, one → L2, both → L3. Noting the choice is recorded, not scored; the gesture stops at L3. Say it (transparency): send as is → L1, say it only if asked → L2, add a line → then tick who gets it: the people the document affects (the person reviewed, the committee, the participants, the manager in copy) → L3, fewer → L2. L4 on every gesture comes only from a final question (have you written a template, rule or list that others use), treated as an indicator. Option orders are shuffled once per session. After each gesture, “the last real time” is answered by picking the lower pattern that applies; the estimate is the lower of gesture and practice. The person then sees the descriptor their answer matches and the one above it, without the level number: levels appear only on the final map, so that they do not prime the following gestures. Time per gesture is shown to the person and not used in the reading. These keys are declared, not calibrated: the release checks for this format are cumulativity (Guttman / Mokken H ≥ 0.3 per gesture), dressing equivalence across families, and a self-recognition rate ≥ 80 %, as stated on the pilot page. Reviewed real work remains the anchor.

7 · Role profiles and the collective view

The reference is universal; thresholds are local. A role profile is a set of expected and critical levels per sub-component, defined by the organisation, never by the reference. The platform compares a person's profile to their role's and flags gaps; it does not judge the thresholds. The collective view never produces a mean: it produces distributions, how many people at each level, per sub-component, per role, per period, and their evolution. The Index's finished-artifact effect (evaluation drops when the AI hands over a polished artifact) is a native collective indicator. For organisations under Article 4 of the EU AI Act, evidence records, role profiles and training records are exportable as an evidence pack; the reference helps assemble proof, it does not certify compliance.

Collective indicators (v0.7). Two readings derive from role profiles. Role readiness: the share of a population at or above the expected level on every sub-component the profile marks as critical. Use-case readiness: the share of people at or above the required level on every sub-component a priority AI use case demands, the requirement with the lowest pass rate being named as the bottleneck. Every collective cell, including each population × period × sub-component cross-cut, is suppressed below 8 people. A demonstration on generated data is on /teams.

8 · What is implemented

The MCP server. Five versioned resources and fifteen tools in three layers. Reference: get_framework, get_scale, list_behaviors, explain_subcomponent, get_progression, get_evidence_rules. Trace instrument: analyze_export, sample_for_rating, describe_data_flows. Scenario instrument: get_scenario_test, score_scenario_test. Placement: record_evidence, list_evidence, build_profile, render_card. stdio for Claude Desktop and Claude Code, stateless Streamable HTTP for everything else. Evidence lives in session memory; nothing is written to disk except a card the user asks for.

The trace instrument. Adapters for the Claude export (with projects.json for the Agency indicator), the ChatGPT export, and a generic JSONL shape. A deterministic marker layer detects the eleven Index behaviours in English, French and Italian with present / partial / absent and a mandatory excerpt; a model then rates a stratified sample against the descriptors. Behaviour rates use substantive conversations only, matching the Index's own multi-turn filter; the pattern mix uses every conversation with a human turn, so one-shot Directive exchanges are not erased by the filter.

The scenario instrument (v0.2). A situational judgement test in three packs of twelve situations, one per sub-component: work (general), personal life, retail and luxury. The person declares the pack; the scale, descriptors and option levels are identical across packs. Each situation is asked twice: What would you do? (recognition, subtype judgement_test, weight 0.6) and And the last time this actually happened to you? (reported practice, subtype behavioural_tendency, weight 0.5, with an explicit has not happened answer recorded as not elicited). In v0.2 the estimate per cell was the practice level when reported; since v0.3 (site v0.7) it is the lower of the two answers, and both answers and their gap are shown. The MCP scorer still implements the v0.2 rule until it is updated. Options are written under mechanically checked anti-cue rules: word-count ratio ≤ 1.5 within an item, no "do that, and more" option, L4 not systematically the longest, one sophisticated-sounding L2 per item. Every result is labelled estimate, not calibrated, not a placement (rule R10).

The public site (v0.7). Six pages in English and French: an explanation of the framework with the 48-cell grid, the test and its result, a team view on generated data, a methodological note (pilot protocol, data rules, evidence and traces), and this specification. Nothing is uploaded and nothing is stored: the result is encoded in the URL fragment, and an optional file saved on the person’s own device allows two passations to be compared. The full reference is published as referential.json.

Still to build. V2: situational cases with a second rater and published inter-rater agreement. V3: artifact and reflection capture, role profiles, collective distributions. V4: publication with calibration statistics and adoption by a third-party client, the step that decides whether this is a tool or a standard.

9 · Threats to validity, stated in the product

• Goodhart. Known behaviours can be performed mechanically; hence mixed evidence, limited weight for traces, and relevance (not presence) rated in situational cases.
• Surveillance. A reference that reads conversations will be circumvented if experienced as control; hence private self-diagnosis by default, consent per use, minimisation, evidence shown behind every placement.
• False precision. Levels plus confidence, no average, distributions rather than means; the one summary, the consolidated level, is a floor shown only with the grid.
• Obsolescence. Platform awareness and cases age with the models; versioning, dated semi-annual review, product-free descriptors.
• Licence. The source framework is CC BY-NC-SA; strict separation between what quotes it and what is built on top, full attribution, a non-commercial open project.
• Vendor dependence. The framework is Anthropic-backed; the server and ontology work with any client model.

10 · Questions for reviewers

1. Are the 48 descriptors discriminating enough for two raters to agree? Which cells feel like a coin toss between L2 and L3?
2. Is "traces cap at L2" too conservative for the short loop (Description, Discernment), where traces are the richest evidence?
3. Is the proposed mapping of the eleven Index behaviours to sub-components the right one?
4. Should diligence.transparency produce a low-confidence level from weak trace signals, or stay "not measurable from traces" (current choice)?
5. L4 is now defined at the level of the person (a codified practice made usable by others). Is that the right reading of "beyond proficient", or should the scale stay at the course's three levels?
6. The scenario test is not calibrated: no item statistics, no second rater, no test-retest. Which population should provide the first voluntary anonymous passations, and what agreement threshold should gate the removal of the "not calibrated" label?
7. Is the lower-of-two rule too conservative when reported practice sits above recognition?
8. Which first occupation should host the calibration pilot? Proposal: luxury retail (sales and beauty advisors and their managers).