In equipment tests where displayed values must be compared under the same conditions, as with breathalyzers, the fact that two devices gave different readings is not enough to conclude that the devices differ. Besides the device, readings can be affected by conditions that changed during the test, such as the time of testing, the person who prepared the sample, the room temperature, and the device's readiness. In particular, if all measurements on one device are done in the morning and all measurements on the other in the afternoon, the effects of the device and the time of day become mixed into a single lump.
Here, the items being compared are devices or sensor types such as A and B, and the run order means the position in which each measurement was placed. A block is a group of tests in which you try to keep conditions such as the operator, time, and environment as constant as possible during the comparison. The core of a comparison test is designing it so that the effect of switching devices does not get mixed with other changes, and recording both the planned order and the order actually carried out so the results can be interpreted later.
Do not make device differences and the passage of time the same variable
Consider, for instance, a case where A was measured continuously in the morning and B continuously in the afternoon. Even if the afternoon readings are higher or lower, the data alone can hardly separate whether that reflects a characteristic of B, a change in the sample over time, a change in room conditions, or a difference in how the test operator prepared things. That is because the placement of A and B overlaps completely with the time of day.
The measurement order is not just a matter of convenience in running the test; it is a condition for interpreting the results. If you keep only the average for each item compared, you cannot see which device was tested at which point, so you cannot trace whether values drifted over time. You therefore need a setup that stores the device name, block number, planned order, actual order, and whether a run was interrupted or repeated, all linked to the readings.
Put every item being compared into each block
A factor that can affect results but is not the subject of the current test is called a confounding factor. For example, the operator, test time, and room temperature are conditions that can affect readings more than the device comparison itself. If you define as a block a span in which such conditions are kept as similar as possible, and include both A and B in each block, you can look at differences between devices under the same set of conditions.
A block does not have to be a unit of time. It can be a group of measurements that one operator can carry out with the same preparation procedure in the same environment, or a group in which the same sample preparation conditions can be maintained. What matters is that no item being compared is missing from a block, and that conditions likely to differ between blocks are recorded separately.
| Item to check | Placement easily confused with device differences | Placement that makes comparison possible |
|---|---|---|
| Device placement | Only A in the first group of measurements, only B in the next | Include both A and B in each group of measurements |
| Operator | One operator runs only A, another runs only B | Where possible, run both devices in the same group |
| Environmental conditions | Morning and afternoon results split by device | Compare within groups with similar time and environment |
| Test changes | Order changed after a problem and the change left unrecorded | Record the original order, the reason for the change, and the actual order together |
For example, this is a hypothetical case. When one test operator needs to compare A and B in both the morning and the afternoon, they could run A then B in the morning and B then A in the afternoon. At the observation stage, record each reading along with the morning or afternoon group, the device order, and whether there was an interruption. At the interpretation stage, check whether the difference between A and B appears similarly in both groups, or whether there is a common shift between the device measured first and the device measured second.
A possible action is to put both devices into each group and decide the device order for each group in advance. The completion criterion is a state where A and B were actually measured in each group and the planned and actual order can be traced for every reading. This case alone does not establish a performance difference or statistical significance for a particular device, but it at least reduces the problem of the device and the order overlapping completely.
Decide the order before the test and distinguish random assignment
Randomization is a method of setting the run order of test units at random. Measuring A and B alternately once each does not automatically amount to randomization. For instance, if you always repeated A then B, the device and the measurement position stay linked, so a difference between earlier and later measurements can get mixed into the device difference.
It is better to set the order before the test, within the scope of the permitted procedure, and keep that plan on file. If there are more than two items being compared or there are repeated measurements, you can review the layout so that which device comes in which position in each block is not consistently skewed to one side. However, setting the order at random does not solve other problems such as too few repetitions, unsuitable samples, or environmental changes.
Record control of the preparation procedure together with any order changes
Mixing up the order must not change the approved test procedure, such as device preparation, handling of reference samples, and conditions between measurements. If there are constraints that require changing the order, it is important not to hide them and interpret the test as if it were a fully randomized test. Maintaining comparable conditions may come before arranging the order.
If the order changed during the test because of a device fault, a sample problem, or a schedule interruption, do not rewrite the original plan to erase it. Record which device you originally meant to measure, where things stopped, why you started again, and what the actual order turned out to be. If only values unfavorable to a particular device are excluded after the fact, the final comparison can be skewed even if the layout was fair beforehand.
Keep the run history ahead of the averages
The final results table should contain not only summary values per device but also information showing in which block and in which position each reading was taken. Only then can the analysis stage review changes over time, changes of operator, and the relationship between measurement order and values. If you pass on only one line of averages per device, such checks become practically impossible.
If the purpose of a comparison test is to see differences between devices, the starting point is to make sure that conditions other than the device do not attach to only one device. If you put the items being compared together in each block and record the pre-assigned order and the actual run order separately, you have a basis for reviewing whether an observed difference is due to the devices or to conditions that changed while the test was under way. These design records are not a document that guarantees one device is better; they are test data that let the results be analyzed without exaggeration.
