Who receives the errors?

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

One overall score hides several experiences

A fictional school equipment desk receives requests through two subgroups: online and in person. These groups describe collection channels, not protected characteristics or fixed kinds of people. A real audit would need relevant groups, intersections and input from affected users.

The lab uses 12 online requests and 8 desk requests. Six online and four desk requests truly need urgent review. Supplied scores rank requests; they are not learned probabilities. The true labels are treated as correct for the experiment. In practice, ask who supplied labels, how disagreements were resolved and whether the definition of urgency fits the intended use.

A threshold is only the first gate

The first gate admits requests whose score meets their channel’s threshold. The second gate fills limited review capacity. At the default threshold of 0.50 and capacity of eight, the highest eligible scores win, with dataset order breaking exact score ties.

The flow diagram separates all requests, eligible requests and actual reviews. Lowering a threshold may admit a request without providing a slot. Read the individual trace to distinguish “below threshold” from “capacity deferred.” Neither is an automatic permanent denial: the lab represents one review round, and later outcomes are outside its scope.

Equal counts are not equal rates

In the default ranking, six online requests and two desk requests receive review. Four of the six urgent online requests are caught; only one of the four urgent desk requests is caught. These are worked-example defaults; the workbench recomputes every count when you change controls.

Recall means the fraction of truly urgent requests that receive review. Review rate means reviewed requests divided by all requests in that channel. False-positive rate means ordinary requests receiving review divided by all ordinary requests. Precision means truly urgent requests divided by all reviewed requests. The counts are in requests; the rates are unitless fractions expressed as percentages.

Reserving four slots per channel gives equal counts, but review rates are 4 / 12 ≈ 33.3% online and 4 / 8 = 50% at the desk. Never compare rates without their denominators. If no examples of a group are present, performance is unknown. If there are no actual urgent cases, recall is undefined rather than measured zero.

Different criteria ask different questions

Equal review rates ask whether groups receive review at the same fraction. Equal recall asks whether urgent cases are caught at the same fraction. Equal false-positive rates ask whether ordinary cases receive unnecessary review at the same fraction. These criteria can favor different policies. Matching one does not establish that every relevant harm, benefit or access barrier has been addressed.

Try reserving two online and six desk slots while lowering the desk threshold to 0.30. Online recall falls to 2 / 6, while desk recall rises to 4 / 4. Explain whose urgent requests are delayed by that allocation. A decision needs a reason for the tradeoff, not just the word “fair.” The example is a way to inspect consequences, not a recommendation to use channel-specific thresholds in a real service.

Reservation has its own policy choice: this lab leaves unused reserved slots empty. Other policies could transfer them, rotate service over time or escalate exceptions. Those rules change outcomes and must be tested explicitly. The lab does not assume human review is perfect or that an urgent review guarantees equipment.

Examine the evidence behind the numbers

A group’s measured recall of 1 / 2 and another’s 10 / 20 both equal 50%, but the first is much more uncertain. Revisit sampling uncertainty before interpreting a gap or a match. The twenty synthetic examples here are development evidence, not representative samples that establish real service quality.

For a decision brief, record the population, available capacity, compared policy, per-group counts, chosen criterion, affected users and remaining uncertainty. Request evidence on missing groups, disputed labels, access barriers and appeals. Engineers can implement the two-stage allocator with tests for ties, empty eligibility and unfilled reservations. Independent assessment of an implementation or decision requires new evidence and review beyond this practice lab.