Reusable Technical Question Sets

The questions an experienced engineer would ask if they were in the room — encoded so a non-expert can ask them without you there. Your judgment, reused in twenty meetings you’ll never attend.

What it is

This is the question-sets pilot named in the intro: standard questions to bring into agency or vendor meetings — the ones an experienced engineer would ask if they were in the room.

The flagship example is the 25 Questions for Ground Truth — the ninety-minute discovery a deployed-team engineer runs walking into a new agency. A question set encodes the thing that’s hardest to transfer: the judgment about what to ask, in what order, and what a vague answer is really telling you.

These are artifacts, not a meeting. That’s what makes them pure leverage. A set you write once gets asked in every vendor meeting you’re not in — it’s the inheritance model from the intro in its cleanest form, paying down the next engineer’s cost without spending any of your time in the room.

How you run it

Curate and maintain a small library — small is the point. A focused set of question lists that get used beats a sprawling bank that nobody can navigate. Each set has a clear job: the discovery conversation, the vendor demo, the build-vs-buy decision, the security review.

Harvest new questions from the work. The best questions come out of office hours and architecture reviews — the question you found yourself asking a team is exactly the one to encode. When a review keeps surfacing the same gap, that’s a question the set is missing.

Keep them current and retire the stale ones. A question set is a living artifact with a review cadence, not a write-once document. Pruning matters as much as adding — a set carrying dead questions about a vendor nobody uses anymore teaches the reader to skim, and a skimmed checklist catches nothing.

How you show up

Your engineering judgment is the product. The value isn’t the list of words — it’s the thinking the list encodes: which question exposes a vendor hand-waving about scale, which answer means “we haven’t actually decided,” which follow-up an experienced engineer would reach for. Write down what you’d ask and why, so the reader asks it like they mean it.

Pressure-test the sets with real teams. A question that reads well to you and falls flat in an actual vendor meeting is a question that doesn’t work yet. Watch a non-expert use the set in a real room — the questions that earn a useful answer stay; the ones that get a blank look get rewritten or cut.

Templates

Authoring a set is mostly discipline: write the questions you’d actually ask, then for each one write down what a good answer sounds like next to what a dodge sounds like — that contrast is the judgment you’re transferring. These are the copy-paste starters for that. Swap anything in [brackets] for your specifics, and keep the solid-vs-evasive tells when you edit; they’re the part that does the work, not the questions alone.

Question-set skeleton

The standard shape for a new set. The “When to use it” line keeps the library navigable, and the per-question tells are non-negotiable — a question without them is the generic question this pilot warns against.

Question Set: [name — the conversation it's for]

When to use it: [the specific meeting — e.g. a vendor's first technical
demo, a build-vs-buy decision, a security review]. If you can't name the
room, the set is too broad.

—

Q1. [The question, as you'd actually say it out loud.]
    Why you're asking: [the one thing a bad answer here exposes.]
    Solid answer sounds like: [specifics — names a real mechanism, a
        number, a tradeoff they've actually weighed.]
    Evasive answer sounds like: [a brochure phrase, a deflection, "it just
        works," a number with no source.]

Q2. [...]
    Why you're asking: [...]
    Solid answer sounds like: [...]
    Evasive answer sounds like: [...]

(Three to seven questions. Stop when the set has teeth, not when it's long.)

Notice every question carries its own “why” and its solid-vs-evasive tells. That’s what keeps the set a thinking tool rather than a script to read aloud — strip the tells and you’ve built the checklist ritual this pilot warns against.

Worked example — a vendor’s deployment and release story

One filled-in set, so the shape is concrete. Hand a non-expert this and they can tell a real release process from a rehearsed answer.

Question Set: Evaluating a vendor's deployment & release story

When to use it: a vendor's technical demo, before you believe their
"enterprise-ready" slide.

—

Q1. Walk me through what happens between a developer merging code and that
    code being live for us. Who clicks what?
    Why you're asking: surfaces whether they actually have a pipeline or
        ship by hand on a Friday.
    Solid: a named CI step, automated checks, a deploy that one person can
        trigger and anyone can read the logs of.
    Evasive: "our team handles all of that for you" — the deflection that
        hides a manual, undocumented process.

Q2. Last time a deploy went wrong, what happened — and how fast were you
    back to a known-good state?
    Why you're asking: a real rollback story is the difference between
        resilience and hope.
    Solid: a specific incident, a rollback in minutes, a clear "known-good"
        they can return to.
    Evasive: "we haven't really had one" — either untrue or they're not
        looking.

Q3. How do we find out you've shipped a change that affects us?
    Why you're asking: tells you if you're a partner or a surprise
        recipient.
    Solid: a changelog, a notice window, a way to pin a version.
    Evasive: "you'll see it in the app" — you'll find out when it breaks.

Q4. What can't we do on your platform that a team like ours usually wants?
    Why you're asking: an honest limit is worth more than a feature list;
        it tells you they know their own edges.
    Solid: a real constraint, named without flinching.
    Evasive: "honestly, nothing" — nobody's product does everything, so
        this is a tell about candor, not capability.

Notice each question is the one an experienced engineer would actually reach for, and the evasive line names the exact phrase to listen for. The vendor can tell the difference between this and a brochure question — which is precisely the “the vendor can tell” outcome this pilot is built to produce.

Handoff note

The short message you send a team member right before they walk in, set attached. It does one job: reset the goal from reciting questions to weighing answers.

Subject: For your [vendor/agency] meeting [day] — question set attached

Hi [name],

Attached is the "[set name]" set for [day]. One thing before you go in: the
goal isn't to get through all the questions — it's to understand the
answers. Ask two or three, really listen, and follow the thread when
something sounds thin.

Each question has a "solid vs. evasive" note under it. That's the part to
read closely — it tells you what a real answer sounds like versus a
deflection, so you'll know in the room, not three months later.

If an answer feels off and you're not sure why, that feeling is the signal —
note it and we'll dig in after. You've got this.

[your name]

Notice it tells the reader to ask two or three questions and listen, not to run the list. A set read straight through is the checklist ritual; the value is in weighing the answer, which is the judgment this pilot exists to transfer.

What good looks like

A deployed-team member who isn’t a deep engineer walks into a vendor meeting and asks the right questions — and the vendor can tell.
Sets get reused across engagements instead of rebuilt each time.
A vague or evasive answer gets caught in the room, not three months later.
Questions trace back to real work — each one earned its place by catching something.
The library is small enough that people can find the right set fast.

Where it goes wrong

Too generic to be useful

The questions are so broad they fit any meeting and sharpen none. “Tell us about your architecture” gets a brochure answer and catches nothing. The tell: a non-expert using the set gets answers that sound fine and reveal nothing. Every question has to have a teeth — a specific thing a bad answer exposes. If you can’t say what a question catches, cut it.

Performative modernization

The library exists, looks rigorous, and nobody uses it. It’s a shelf of impressive-looking question banks that went stale the month after they were written. It looks like the work without being the work, and the existence of the doc reduces the pressure to make the questions actually good. The tell: the sets are present and current on paper, but no real meeting was run from one this quarter. A question set that isn’t getting used isn’t an artifact — it’s a prop. Either get it into rooms or retire it.

Checklist ritual

The set becomes a box-ticking exercise — every question asked, none of the answers actually weighed. It’s used, but as a script to read rather than a tool to think with. The tell: people read the set aloud and move on regardless of the answers. The fix is to encode the why and the follow-ups, so the set teaches judgment instead of replacing it. A question set is a thinking tool wearing a checklist’s clothes; the moment it’s only the clothes, it’s failed.

What it tells us

The gaps in the library are the signal. Where teams keep getting caught flat-footed in vendor or agency meetings is exactly where a question is missing — so every “I wish we’d asked that” after a bad deal is a question to add. The library, read as a whole, is a map of where the deployed teams’ judgment is thinnest.

That map feeds the four questions and the hiring signal. A category of question you keep needing and can’t fully encode — because it really does take an expert in the room — is a place a set can’t substitute for a hire, and naming that is as useful as filling the gap.