How should one define fairness of testing? There are countless ways to make tests unfair, but achieving fairness surely involves aligning what's being tested with the purpose of the education. And isn't the main purpose of education to give you skills for life?
Yet in the modern US-UK concept of fairness, questions with complete reproducibility of assessment trump questions that more accurately simulate real life but can't always get every marker awarding exactly the same marks.
For example, multiple choice tests can be marked with complete reproducibility, but when in real-life did you last pick from 4 or 5 answers one of which you knew "has to be" right? Rather, questions which need explanation and judgement calls can be much fairer tests of the student's ability at the real-life subject, even if they might garner some subjectivity of marking.
All this was brought up today when I looked at a book which helps testers set tests. Within the narrow confines of how US testing works, it was no doubt very helpful.
But thinking bigger picture, it was deeply frustrating--like castigating a question with "irrelevant information" ie. more than the minimum needed to calculate the answer, because it wasn't solely testing one core ability [doing a manual calculation]. Since when does real life only have exactly the amount of information you need--no more, no less? And isn't sifting information and using what's relevant a crucial, core ability---particularly since the internet?
Something that makes this all worse: today's tests have assumed an importance beyond their capability to judge. And that's had the unfortunate feedback of putting huge emphasis on reproducibility of marking...and therefore questions with definitively right or wrong answers.
Governments, others setting test guidelines, please remember: fairness ≠ reproducibility (and while you're about it, math ≠ calculating (!))
Or as a London cabby put it on not getting a tip "it may be correct but it ain't right".