> What, specifically, is an example of an exam not measuring proficiency
Not op, but a few examples:
1. Test structures that reward good time management. The paper SAT is a good example of this. The early items in a section are (were?) easy, the middle items were medium difficulty, and the toughest items were towards the end. A good test take would manage their time so that they spent the most amount of time on questions that could improve their scores — later questions for high achievers, but the middle questions for folks who were missing a lot of questions in that range.
2. Test structures that reward endurance. Again, SAT is a good example. How often have examinees sat down in a multi-hour high-stakes testing situation before taking the SAT. “Not often enough to feel comfortable” is usually the answer. I have taken some foreign language proficiency tests that were also multi-hour long endeavors. I practiced answer old exam questions in a testing environment, so the tests were easy for me. Some of my peers did not, and fatigue seemed to be a part of their challenge.
3. I took some “issue spotter” tests in college. I had never heard of or seen an issue spotter test before. I bombed my first one. My professor kindly walked me through things he knew I knew the answers to that would what given me credit. I aced all future issue-spotter exams. Side note: familiarity with test/question format seems to matter for better students, but it largely doesn’t for unmotivated students. Many studies on familiarity with test/question format show no correlation, but my personal experience (as a test giver) is that this is due to lumping the unmotivated and ambivalent examinees with the folks who notice, care, and take action.
There are many more answers to your question, but the above are few decent examples.