Same sample, many benches: how laboratories find out whether they can be believed

A laboratory can check its own work in a dozen ways, and every one of them shares a weakness: the laboratory is marking its own homework. Proficiency testing is the one routine check where somebody else holds the answer. This guide explains how it works, how to read the scores it produces, and what a good or bad round does and does not show.

The idea is simple enough to state in a sentence. An organiser prepares one uniform material, splits it into identical portions, and sends a portion to every participating laboratory. Each laboratory measures it as it would any routine sample and reports a number. The organiser compares all the numbers against a reference value and tells each laboratory, in confidence, how far off it was. Everything else in this guide is detail, and the detail is where the value lies.

Why a laboratory cannot fully check itself

Internal quality control is good at one job: showing that today's results look like yesterday's. A control material run with every batch, plotted on a chart, will reveal drift and sudden jumps. What it cannot reveal is an error that has always been there. If a reference solution was made up slightly wrong two years ago and every calibration since has been built on it, the control chart will be beautifully flat, and every result will be wrong by the same amount.

Analysts call this the difference between precision and trueness. Precision is agreement between repeated measurements. Trueness is agreement between the average of those measurements and the real value. A laboratory can establish its precision alone. It cannot establish its trueness without something from outside: a certified reference material, a second independent method, or a comparison with other laboratories. Proficiency testing is the comparison, organised and scored.

Repeating a measurement tells you how consistent you are. Only an outside value tells you whether you are right.

The international standard for testing and calibration laboratories, ISO/IEC 17025, reflects this. It requires a laboratory to monitor the validity of its results and, where schemes are available and appropriate, to do so by comparison with other laboratories. Accreditation bodies normally expect participation at a planned frequency across the laboratory's scope, and they read the scores during assessment.

Anatomy of a round

Schemes differ in material and frequency, but nearly all of them follow the same six steps. The organiser, called the provider, is itself expected to work to a standard: ISO/IEC 17043 sets out what a competent provider must do.

1. Prepare the material. A bulk quantity is made as uniform as possible and divided into portions, called test items.
2. Prove it is uniform and stable. A number of items are picked at random and measured in duplicate. If the differences between items are small compared with the spread the scheme will tolerate, the batch is accepted. A second check shows the material does not change during shipping and the reporting window.
3. Distribute blind. Participants receive an item with a code and instructions, and nothing about the expected value.
4. Measure as routine. The laboratory is meant to treat the item like any other sample: same analyst rota, same method, same number of repeats.
5. Report by a deadline. Results go to the provider, often with the method used and, in some schemes, the laboratory's own stated uncertainty.
6. Score and report. The provider sets the reference value, calculates a performance score for each participant and issues a report in which laboratories appear only as codes.

The fourth step is the one that decides whether the exercise means anything. A round measures the laboratory's ordinary performance only if the item gets ordinary treatment. Handing it to the most experienced analyst, running it five times and reporting the average, or checking the figure with a neighbouring laboratory before submitting all produce a pleasing score and no information. Standards treat collusion and the falsification of results as serious breaches, and providers design schemes to make it harder, for example by sending out items at two slightly different levels within one round.

Where the reference value comes from

The number every participant is compared against is called the assigned value. It is not always obtained the same way, and the route matters when judging how much weight a score can bear.

The last row deserves a second look, because it is the route most schemes use. A consensus value tells a laboratory whether it agrees with its peers. If most of the peers share the same method and that method reads low, the consensus reads low as well, and the one laboratory with the true answer is flagged. This is rare, and it is the reason a careful report always states how the assigned value was obtained and how uncertain it is.

The z-score, and what the lines mean

Most schemes report a z-score. It answers one question: how far was this result from the assigned value, counted in units of the spread the scheme considers acceptable?

Take the invented round in the first figure. The assigned value is 50.0 grams per kilogram and the scheme's standard deviation for proficiency assessment is 1.0. Laboratory C reported 50.9. Its z-score is the difference, 0.9, divided by 1.0, which is 0.9. Laboratory J reported 52.3 and scores 2.3. Laboratory M reported 46.6 and scores minus 3.4. The sign shows the direction and the size shows the distance.

Conventions as set out in ISO 13528, the statistical companion to the provider standard. A scheme's own protocol always takes precedence.

The limits at 2 and 3 come from the bell curve. If a laboratory has no bias and its scatter matches the scheme's standard deviation, about 95.4% of its results will fall within 2, and about 99.7% within 3. Two consequences follow, and both are routinely forgotten.

First, a perfectly competent laboratory will land between 2 and 3 roughly once in every twenty-three rounds through chance alone. A single warning signal is a reason to look, not a verdict. Second, a score beyond 3 happens by chance about once in 370 rounds. That is rare enough that an action signal almost always has a cause that can be found.

The denominator is a decision

The standard deviation used for scoring is chosen by the provider, and the choice shapes every score. It can be set from what the measurement is used for, so that a score of 2 corresponds to the largest error users of the result could tolerate. It can come from earlier rounds, from a general model of how spread grows as concentrations fall, or from the spread of the current round. The first option is the most meaningful, because the score then says something about fitness for purpose and not merely about rank within a group.

Why the middle is found with a median

When the assigned value comes from the participants, the provider has to find the centre and spread of a set of results that almost always contains a few stragglers. The ordinary mean and standard deviation are poor tools for this, because a single wild value drags both of them. Schemes use robust statistics instead, which are built to ignore the extremes.

The simplest robust pair is the median, the middle result when all are ranked, and the scaled median absolute deviation: the median distance of the results from the median, multiplied by 1.483 so that it matches a standard deviation for well-behaved data. The fourteen results from the invented round show the difference. In the second column, laboratory M has made a typing slip and reported 36.6 in place of 46.6.

Fourteen invented results in grams per kilogram. Only one value differs between the two columns.

One keystroke moves the mean by seven tenths of a unit and more than doubles the standard deviation. Had the scheme scored against those figures, every honest participant would have been shifted and the spread would have been so wide that almost nothing could fail. The median and its companion do not move at all, because the lowest result is still the lowest result however low it goes. More refined robust estimators exist and many schemes use them. They work on the same principle: extreme values are pulled in toward the pack before the average is taken.

Two samples say more than one

A single score cannot say what kind of error produced it. A neat way to find out is to send two similar items in the same round and plot each laboratory's two scores against each other. The picture is named after the statistician who proposed it, W. J. Youden.

A laboratory with a consistent bias reads both items high or both low, so it falls along the diagonal, far from the centre. Laboratory J is the example: 2.3 on one item and 2.0 on the other. Something in its method adds roughly the same amount every time, and the place to look is calibration, a reference solution, or a correction factor. A laboratory with a one-off blunder is right on one item and wrong on the other, so it falls away from the diagonal. Laboratory M scored minus 3.4 and 0.3. Its method is probably sound, and the place to look is the handling of that single item: a dilution, a transcription, a mislabelled tube.

When the whole cloud of points is stretched along the diagonal, as it is here, the message is about the field and not about any one participant: laboratories differ from each other more than they differ from themselves. That usually points to methods or calibrants that are not harmonised across the group.

When the uncertainty is on trial too

A z-score ignores what the laboratory says about its own uncertainty. Scores that include it ask a sharper question: is the claimed uncertainty honest? Laboratory C makes the point. Its result of 50.9 gives a comfortable z-score of 0.9. Suppose it reported that result with an expanded uncertainty of 0.8, and the assigned value carries an expanded uncertainty of 0.3. The E_n number is the difference, 0.9, divided by the two uncertainties combined as the root of the sum of their squares, which is 0.85. The answer is 1.05, just outside the limit of 1.

Nothing is wrong with the measurement by the scheme's standard. What fails is the claim. The laboratory said its result was good to within 0.8 and it was off by 0.9. Either the uncertainty budget is missing a contribution or there is a small bias nobody has accounted for. This is one of the most useful things a comparison can reveal, because an understated uncertainty travels with every result the laboratory issues.

After a bad round

A flagged score starts an investigation, and the order of that investigation matters. The cheapest explanations are checked first, because they are also the most common.

1. Reporting. Was the right number submitted, in the right units, on the right basis? Many action signals turn out to be transcription, unit or decimal errors.
2. The item. Was it stored and prepared as instructed, and within the stated time?
3. The calculation. Dilution factors, recovery corrections, and the spreadsheet cells nobody has opened for a year.
4. The run. Did the batch's own controls pass? Were calibration and blanks normal that day?
5. The method. Reference solutions, their age and preparation, instrument condition, and any recent change to the procedure.
6. The people. Training records and whether the procedure as written matches the procedure as performed.

A proper response ends with three things written down: the cause, the fix, and an assessment of whether routine results reported since the last good round could have been affected. The third is the uncomfortable one, and it is the reason the exercise exists. Where no cause can be found, the honest record says so and the next round is watched closely.

The trend matters more than the round

A laboratory that only asks "did we pass?" throws most of the information away. The figure above shows twelve rounds with no signal of any kind. It also shows nine scores in a row above zero, rising from 0.2 to 1.6. For a laboratory with no bias, nine consecutive scores on the same side would happen by chance less than once in two hundred sequences. Something is drifting, and at this slope the first warning signal is two or three rounds away. Plotting scores over time, and treating a run on one side as a finding, lets a laboratory fix a problem before any single round reports it.

What a good score does not prove

• It covers one material at one level. A satisfactory score near the middle of the working range says little about performance at its edges or in a more difficult material.
• It is a snapshot. One day, one analyst, one batch. Competence between rounds rests on internal controls.
• It may only show agreement. Against a consensus value, a good score means the laboratory reads like its peers.
• It assumes routine handling. A score earned with special care describes a laboratory that does not exist on ordinary days.
• It is not a ranking. A score of 0.2 is not meaningfully better than 0.7. Both are well inside ordinary scatter.

Questions worth asking of any scheme

• Is the provider accredited to ISO/IEC 17043, and for this scheme specifically?
• How is the assigned value obtained, and what is its uncertainty?
• How is the standard deviation for scoring set, and does it reflect what the results are used for?
• How were homogeneity and stability of the items demonstrated?
• Does the material resemble the laboratory's routine samples in type and level?
• How many participants take part, and how many use the same method?
• How often are rounds run, and does that match how quickly the method could drift?
• Does the laboratory keep a chart of its scores over time, and does anyone read it?

The short version

Proficiency testing puts a laboratory's routine result next to a value it did not choose. The score that comes back is a distance, counted in agreed units, with warning and action lines drawn where chance alone becomes an unlikely explanation. Read well, it separates bias from blunder, tests whether stated uncertainties are honest, and shows drift long before it becomes a failure. Read badly, as a pass mark to be collected, it shows nothing at all. The value is never in the score. It is in what the laboratory does with it.