Design the assessment
Start from a seeded template or write your own. Coding, document, research, and roleplay tasks all run through the same pipeline, so scores stay comparable across roles.
Debug a failing payment webhook
60 min
Vendor evaluation memo
90 min
Underperforming report 1:1
45 min
Invite the candidate
One tokenized link — no account to create, no password to reset. The clock starts when they open it, and the token only works once.
Invite link
/invite/8f2c…a91d
Sana Okafor — Senior Backend Engineer
Single-use token · no account, no password · timer starts on first click
They work the way they actually work
A real editor, a real terminal, and an AI assistant in the same window. Nothing is sandboxed away from them, because the point is to watch how they use it.
export async function handle(req) {
const sig = req.headers["x-sig"]
if (!verify(sig, req.rawBody))
throw new Error("bad signature")
return process(req.body)
}$ npm test
✗ 2 failing — raw body consumed
Every action is captured
Edits, commands, prompts, responses, accepted and rejected suggestions, pastes, tab switches, test runs. It is an append-only log, which is what makes the session replayable instead of merely summarized.
Two passes, one report
Deterministic metrics are counted from the log first, then a scoring model reads the transcript alongside them. Separating the two keeps the countable parts stable and the judgment parts accountable to quoted evidence.
Counted from the event log — prompt count, edit/AI ratio, time-to-first-test, rejected suggestions. No model involved, so it never drifts.
A scoring model reads the transcript and the metrics, then scores each dimension with quoted evidence from the session.
Read the evidence, not a vibe
One AI-Q score across five dimensions, each backed by moments you can jump to in the replay. A high score you disagree with is auditable — that is the whole idea.
Evidence: pushed back on the model's dedupe claim at 06:47 and verified against the failing test before applying the fix.
Compare and decide
Same task, same rubric, side by side. Prompt counts and dependency ratios sit next to the score so a fast finisher and a thoughtful one don't look identical.
| Candidate | AI-Q | Prompts | Own work |
|---|---|---|---|
| Sana OkaforStrong verify loop | 79 | 14 | 1.8× |
| Dev Raman | 71 | 31 | 0.4× |
| Mira Halvorsen | 64 | 6 | 3.1× |
Everyone leaves with something
A shareable credential for the candidate, and — in internal L&D mode — a learning path built from the same evidence that produced the score.
AI-Q Credential
79
Sana Okafor
Shareable, verifiable, and scoped to the assessment it came from
Learning path
- Verify before applying AI output
- Decompose before first prompt
- Reduce redundant re-prompting
Generated from the same evidence that produced the score
That's the path
The app is seeded with templates, an organization, and demo accounts, so you can run the same eight stages against real data.