The EdTech Revolution Has Failed

What, specifically, is an example of an exam not measuring proficiency? If an exam is well designed, the student will need to figure out what is being asked and use their mastery to provide an answer.

A good example in the UK is teaching students the FOIL technique for algebraic expansion. Students typically can expand (ax+b)(cx+d) because they've learnt a recipe but cannot expand say (ax2+bx+c)(dx+e).

Many schools here focus on such tricks (nix the tricks was a great book focusing on such things) as schools here are judged on pass/fail rates.

In general, exams are an excellent way to assess students en masse at their ability to remember similar problems but not inherent problem solving techniques. The latter I've found is possible to teach 1to1 but far harder with a class of varying abilities.

> What, specifically, is an example of an exam not measuring proficiency

Not op, but a few examples:

1. Test structures that reward good time management. The paper SAT is a good example of this. The early items in a section are (were?) easy, the middle items were medium difficulty, and the toughest items were towards the end. A good test take would manage their time so that they spent the most amount of time on questions that could improve their scores — later questions for high achievers, but the middle questions for folks who were missing a lot of questions in that range.

2. Test structures that reward endurance. Again, SAT is a good example. How often have examinees sat down in a multi-hour high-stakes testing situation before taking the SAT. “Not often enough to feel comfortable” is usually the answer. I have taken some foreign language proficiency tests that were also multi-hour long endeavors. I practiced answer old exam questions in a testing environment, so the tests were easy for me. Some of my peers did not, and fatigue seemed to be a part of their challenge.

3. I took some “issue spotter” tests in college. I had never heard of or seen an issue spotter test before. I bombed my first one. My professor kindly walked me through things he knew I knew the answers to that would what given me credit. I aced all future issue-spotter exams. Side note: familiarity with test/question format seems to matter for better students, but it largely doesn’t for unmotivated students. Many studies on familiarity with test/question format show no correlation, but my personal experience (as a test giver) is that this is due to lumping the unmotivated and ambivalent examinees with the folks who notice, care, and take action.

There are many more answers to your question, but the above are few decent examples.

That, to me, is not a problem with the exam though. It's a problem of teaching to a special case and not the general case. If you want to find fault, it's in the incentive system. But I don't see how the exam itself is the problem.

“Well designed? is doing heavy lifting here. You can get very good at specific test formats in terms of time management, common tricks, etc.

Actual tests include things like: Multiple choice questions were providing answers aids answering the question. Short responses / fill in the blank generally mean people can just regurgitate answers they don’t understand. Essay responses sound great, except you can’t answer many questions and essays writing is a separate skill which heavily influences final scores.

More broadly tests are time limited so can’t test skills that take long periods of time to demonstrate a major issue for say programming.

I think test prep allowing people to increase SAT scores is actually a useful feature not a bug.

If you’re the kind of person who’s going to put extra effort to add a few points to your results, you are also the kind of person more likely to do well in collage classes.

Exams are a poor measure of proficiency. Proficiency is gained by doing and stretching a skill over time. You can measure that in small increments than a periodic exam. At the end of a period, a student would have a body of work to demonstrate proficiency rather than relying on a single day.

When I taught at university there was a disparity between exam grades and the physical body of work they had submitted over the years. You'd see the grade and be shocked the did so badly. The grade reflected proficiency in examinations, not in the subject.

Well I won't reiterate all of 'bad education' by Bryan Caplan but to my mind exams are imperfect because:

1. Schools are not equal. It's not fair to compare students when they usually have no choice over their teachers. 2. Exams cover an arbitrary syllabus controlled by undemocratic exam boards. 3. Topics are chosen by exam boards that can be examined not by importance. 4. Students who perform poorly under stress of exam conditions are punished for it. 5. Exams serve no real purpose. Children are not chickens being graded for sale. They're at best a weak signal of aptitude.

I would much prefer exams to serve as a prerequisite of sitting a future course rather than an assessment at the end. That way teachers can actually teach rather than continuously repeat the same content.

> More broadly tests are time limited so can’t test skills that take long periods of time to demonstrate a major issue for say programming.

Every programming class I've taken, in high school or college, was project based where the main source of grade were actual programs I wrote which actually did something.

The one exception perhaps being the AP test for AP CS.

I agree in general with what your idea, but this sort of assumes a few things that aren’t always true:

1. Examinee is aware of test prep and its potential benefits. Definitely not always true in low socioeconomic status (SES) families, especially ones with no family members who have been to college.

2. Examinee can afford said test prep courses and/or materials. Library and online are “free” options, but we are back to assuming that the examinee has easy access to a library and/or the internet and knows how to find said materials.

There are many, many students who would do well in college if they had some insight on how to do college. The US has a lot of wasted potential due to our public school focus on bringing up the low achievers to the exclusion of developing those with high achiever potential.

Some private schools and most public schools in “good” neighborhoods have programs and cultures that cater to those with high potential, but these schools only address a relatively small portion of the student population with high potential, imho.

I think the SAT is a ludicrous setup given that most people never see anything like it in their lives again but the practice of studying for passing an exam of some kind is an absolutely invaluable skill

It's not really being the "kind of person who’s going to put extra effort" and more the "kind of person who is privileged enough to have parents that care about their education, know that outside resources exist, and have the time and money to utilize those resources".

English exams for ESL students are a great example.

Getting good grades in those exams requires that you know the criteria for evaluating each part of the exam and how to tailor your answer to that criteria. For example, if the exam asks you to write a short movie review you are expected to follow the formula for reviews and show that you can use certain specific grammar constructs.

If you know English well but you don't practice the exam before you will get a mediocre grade, simply because you didn't follow the tacit guidelines that you are expected to know.

It biases results towards people that "studied how to do the test" rather than studying the material that is evaluated.

> I think the SAT is a ludicrous setup given that most people never see anything like it in their lives again

Yeah, I guess.

I think the issue is that the folks best suited to get the most out of college don’t really need to prep much to absolutely crush the SAT.

Most people are just woefully underprepared.

> but the practice of studying for passing an exam of some kind is an absolutely invaluable skill

Yep. Learning how to study, learn, and prep for a test are good skills to have.

That said, for folks who have done well in a semi-rigorous school environment and read the right things (high-brow periodicals), very little test prep is actually needed to get them close to their theoretical max score.

There are a few catches, though:

- Some people in the US have an incredibly irrational fear of math.

- The math curriculum is super-slow and limited for those who like math, often turning them away from it (or at least the high school math classes).

- Most folks have no idea what university-level literacy looks like. Doing things like reading “high-brow” periodicals gives high school students most of the vocab and text structures covered in the SAT.

the inability the generalize the foil procedure to an expression with more than 2 variables speaks more to the non mathematically oriented population just sucking at generalizing things. i have found this to be a very “you have it or you don’t” type of thing, not really something that can be taught

Then again, it may be because FOIL is stupid.

I've always had a difficult time wrapping my head around this acronym. What counts as "outer"? What counts as "inner"? And yes, when there are more than two items (not necessarily variables!) to be multiplied, you suddenly have to ignore this little trick, because now it's confusing to know what to do about the middle stuff -- and it doesn't take into account non-commutativity either.

And yes, some of the problem may be due to my (very recently diagnosed! at least, formally) autistic mind. But I cannot help but think that if someone with a PhD in math struggles with and largely ignores "FOIL", then the problem may be with the technique, and not with the people who don't understand it.

i never found foil stupid, but i also only have a bachelors in math. maybe this was because i had already been exposed to multiplying polynomial expressions, beyond just a 2 term * 2 term by the time i had learned it in school, but i never found it particularly complicated to grasp. foil was never taught to me as the only way to multiply polynomials, rather, an easy algorithm to apply in a certain case. the goal is for you to make the connection that oh, in a 3x2 case, u have to multiply each term in the 3 with each term in the 2, etc.

i think your problems with foil can be extended to the general way math is taught. at least for me, it was always full of tricks, little rules that can be broken sometimes, and i was constantly learning new things that made me realize my old teachers had taught us tricks to shortcut solutions.