4.10. Are any other contradictions implied by the data?
There may be other instances of contradictory results that would not be detected using the other checks in this domain. Several examples have been identified, but appear to occur too infrequently to warrant routine checking. Should anomalies of this nature be observed, they may be recorded in response to this check. For example:
Overall cohort vs. subgroup descriptive statistics
Subgroup sample sizes, means, or Standard Deviations could conflict with results for the overall cohort.
Example
A manuscript reports a mean quality of life score for the intervention group overall, and also separately for participants aged 65 or over, or under 65. There are 41 participants in the intervention group overall, with 30 of these being aged less than 65 years and 11 being 65 years or older. The overall mean score is reported as 7.3. The mean score for the 30 participants younger than 65 is reported as 6.2, and the mean score for the 11 participants aged 65 years or older is 7.4. We may attempt to reproduce the overall mean score from the subgroup-specific means, by calculating (6.2×30 + 7.4×11) / 41 = 6.52, which is smaller than the reported value of 7.3. We must confirm that the discrepancy cannot be explained by rounding, by considering the values for the subgroup means that would make the recalculated value as large as possible (e.g. by setting the subgroup means to 6.2499 and 7.499 and recalculating the overall value). This returns a value of 6.57, which is clearly not compatible with the reported value of 7.3, even if the latter value has been rounded up. The reviewer records a response of “yes” for this check, and this response contributes to the domain-level judgement.
Tools for this check
- INSPECT-SR consistency checker for overall cohort vs. subgroups — Ian Hussey’s web app for checking multiple values, with guidance on interpretation and the ability to download a copy of the results. Built on Ian Hussey’s ‘recalc’ R package.
Bounds checks
For example, a mean value could fall outside of a reported range: the mean of a 1-7 self-report Likert scale must be between 1 and 7. Equally, the reported Standard Deviations could fall outside the mathematical bounds set by the scale’s reported range.
Consultation with a statistician may be useful to verify judgement.
Tools for this check
- INSPECT-SR granularity and bounds consistency checker for means and variances — Lukas Jung and Ian Hussey’s web app for checking multiple values with GRIM and GRIMMER, with guidance on interpretation and the ability to download a copy of the results. Built on Lukas Jung’s ‘scrutiny’ R package and Ian Hussey’s ‘strait’ R package.
Some combinations of outcome are not possible
For example, it is not possible to have more birth events (with one “birth event” defined as the birth of at least one child) than pregnancies.
The answer to this check should contribute to a domain-level judgement.