Your Interview Scorecards Do Not Mean Anything Yet

A scorecard is a good idea implemented backwards. You ask every interviewer to rate the candidate 1 to 5 and give a yes or no, you average it, and you get a number that feels like a measurement. It is not one, unless everybody was measuring the same thing.

Consider a real Tuesday. The founder does a forty-minute conversation about the candidate's last startup. The tech lead spends an hour on system design. The delivery manager asks about notice period, commute and whether they are comfortable with client calls. All three submit a "3, maybe". Those three 3s share a scale and share nothing else. Averaging them produces a number with no referent.

This is not an argument against scorecards. ShortlistAI has shipped them for a while: after an interview, every interviewer -- including outside panelists who have no dashboard login -- gets a link, and submits a 1-5 rating, a recommendation of Strong yes / Yes / Maybe / No, and optional notes from their phone in under a minute. That part works, and the low friction is the reason feedback actually arrives.

The missing half was always the questions.

What an interview kit is

A kit is a named, reusable list of questions, each with optional guidance on what a good answer sounds like. That is the entire idea. It lives in /hire/kits as a shared library owned by your organisation, not by whoever created it -- a recruiter can edit a kit an admin wrote, because a question set nobody may fix is a question set that rots.

The guidance line is the part that does the actual work. A shared question list without it still produces private standards: two interviewers ask "tell me about a time you disagreed with your manager" and one is listening for conflict-resolution skill while the other is listening for deference. Writing down what a good answer sounds like is what makes two people score the same answer the same way.

Here is a real example from one of the kits the product ships with:

When did you last get something badly wrong at work? What happened next? A genuine mistake with a genuine cost, owned without deflecting, and a concrete change afterwards. "I work too hard" is a non-answer -- ask again.

That second paragraph costs thirty seconds to write and is the difference between a structured interview and a shared list of conversation starters.

Kits attach to a role and a round

A single role is not one interview. Hiring a backend engineer usually runs a phone screen, then a technical round, then a manager or values conversation -- three different conversations that need three different question sets. So a kit attaches to a role and a round, and you can attach several. The rounds available are Phone screen, Interview, Technical round, Values and ways of working, and Final round.

That also means a kit is a library asset rather than a property of one job. The same "Values and ways of working" kit gets asked for every role in the company, and each kit shows you which roles currently use it -- which is exactly the thing you want to see before editing a question set other people are relying on.

Attaching is how a kit reaches an interviewer. A kit sitting in the library that is not attached to anything is a document, not a process.

What ships out of the box

Opening the kits page for the first time seeds three starter kits rather than showing you an empty screen and a "Create kit" button:

1. Phone screen -- five questions. A twenty-minute first conversation that confirms the basics are real before anyone spends an hour on this person. Includes the one recruiters most often skip: checking the stated notice period against what was on the application form.
2. Role and experience deep dive -- six questions. Past the resume into what they personally owned, how they approach a hard problem, and what they want in two years that they are not doing now.
3. Values and ways of working -- five questions. Asked for every role, by someone the candidate has not met yet. How they work with people, not whether you would enjoy a drink with them.

Three, not thirty. They are meant to be edited. A wall of starter content reads as the product's opinion rather than a starting point.

If you delete all three, they stay deleted -- the seed runs once per organisation and is marked as done, so it will not quietly reappear on your next page load.

The boring guardrails

A kit holds at most 50 questions, because past that it is a script rather than an interview. A kit must have at least one question to be saved: an empty kit is the exact failure the feature exists to prevent, because an interviewer who opens one improvises while believing they are following a process.

Questions can be reordered by dragging or with arrow buttons. If your browser tab has been open since before a colleague added a question, a reorder is rejected with "the question list changed, reload and reorder again" rather than silently dropping the question your stale list did not know about.

Being honest about what this fixes

Structured interviewing being more predictive than unstructured conversation is long-standing, general industry knowledge in hiring research -- it is not something measured inside this product, and nobody should present it as such. What a kit does concretely is narrower and easier to verify yourself: it makes your interviewers' scores comparable, and it makes a disagreement between two interviewers a disagreement about an answer rather than about two different conversations.

Interview kits are part of Advanced Hiring, available on the Business plan and up, along with the rest of the hiring toolkit. The five-day trial includes them.