Do AI detectors work? Students face false cheating accusations

The process of reading textual material, thinking about it, and then producing more textual material about the stuff you just read (and maybe connecting it to other stuff that you've read in the past) is a critical way of developing thinking skills and refining your ability to communicate to an audience.

The value of that shouldn't be overlooked just like the value of basic numeracy shouldn't be overlooked because we all carry calculators.

You're right that it would be better if post secondary institutions would test people's ability to think in more ways than just what they can regurgitate onto a piece of paper, if only because that can be easily cheated but that doesn't mean that there isn't personal benefit in the experience of writing an essay.

I may not be the best writer but I am a better writer because I wrote essays in university, and I may not be great at math but I can reason and estimate about a variety of things because I have taken many math courses. These things have ultimately made me a better thinker and I am grateful to have had that imparted to me.

So the obvious solution is to make students to talk with an AI, which would grade their performance. Or, maybe the grading itself could be done by a minimum wage assistant, while AI would lead the discussion with a student.

It seems to me that this is pretty much immune to plagiarism as well as being much better for the student.

For longer essays, you can just build in an oral examination component. This face time requirement is just not that hard to include given that even in lecture hall style settings you can rely on graduate student TAs who do not really cost anything. The thing is that the universities don't want to change how they run things. Adjuncts in most subjects don't cost anything and graduate students don't cost anything. They earn less than e.g. backroom stocking workers. This is also why they, by and large, all perform so poorly. 30 minutes of examiner time costs maybe $11 or less. Even for a lecture class with 130 students, that's under $1,500. Big woop.

There are some small changes to grading practices that would make life very hard for AI cheaters, such as even cite checking a portion of citations in an essay. The real problem is that US universities are Soviet-style institutions in which gargantuan amounts of cash are dumped upon them and they pretend to work for it while paying the actual instructors nothing.

In fact, the only reason I use the word 'importunate' to describe myself, is because that's what my undergrad advisor called me.

So I at least was able to get well over 30m with each professor to discuss whatever I wanted. But likely that's b/c there wasn't a lot of competition.

It is, because I'm becoming tired with the current AI hype. It lasts too long to be funny.

OTOH, professor talking with a student is a good way to assess the academic performance of the student, but there are some caveats beyond costs. For example, professor will struggle to be an objective judge. Moreover even if they succeed, they would face accusations of discrimination in any case.

AI could solve this problem, but I'm not sure if AIs will be up to a task of leading the discussion. Though maybe if you try to assess students on their ability to catch AI on a hallucinated bullshit...

Of course the current setup works badly for those who explain it much better by speaking.

Yeah, the General Problems exam was a nightmare, I think the professors competed each year to come up with the toughest questions. Getting 50% was an excellent score.

It did force you to learn all the material though, especially as at the end of 3-4 years you may have forgotten some of it, like Optics or whatever. It was pretty hardcore though, especially compared to my friends studying other subjects.

Why not have the testing done externally, by really neutral persons?

But AIs and especially LLMs are way too unreliable for the foreseeable future.

When you audition for a symphony you perform behind a curtain and are required to wear soft slippers (so they can't tell if you are a wearing high heals - female).

We can probably use voice changers so the examiner cannot tell who you are by your voice, but those tend to be fatiguing.

Those of you who severely lack any of the three will be penalized. Just like someone who can discuss a topic orally but could not write it up would be penalized.

    > Written assay evaluation is not and has never been an effective evaluation. 

I've kept a blog for almost 20 years now and one thing is for sure: well-structured writing is very different from an oral exam the writing allows for restructuring your thoughts and ideas as you go and allows for far more depth.

I don't think, for most folks, that they could have as much depth in an F2F as they could in their writing with the exception of true experts in their fields.

The written essay has a cohesiveness and a structure to it that provides a better framework for evaluation and conveyance of information.

I'm not so sure that writing takes more time than studying. For starters, you don't have to memorize anything, and you can limit yourself to the assigned topic.

Of course, it can be that students don't take studying for an oral exam seriously, and trust the teacher to only ask superficial questions.

Yes, there are people who have trouble with public speaking to a debilitating degree, but it would be excessively rare for someone to not at the very least in a one on one with their professor/teacher be able to be so badly affected as to not seem they've even heard of a topic or at least be able to prove they've worked on it to a certain degree.

I would be immediately skeptical of any student who claims they are completely unable to explain their knowledge unless they are allowed to work in complete isolation with nobody to monitor they aren't cheating in some way.

Lots of elite prep schools in the US use a similar system, for similar reasons.

However, +90% of students will not now do any of that work. I got out of teaching (coincidentally) before LLMs appeared, and even then +80% of students did not experience that benefit of the essay process even with a grade (and plagiarism consequences) to motivate them. Now that decent-ish prose is a few keystrokes or Siri-led "chats" away, that's what they're going to do. That's what they're going to do.

I know of - I think it's up to four, now - former colleagues taking early retirement, or changing careers, rather than continue teaching Humanities in a world of LLMs.

That strikes me as a workable bottom line.

I think the hard answer is that to some extent you just have to learn to. I mean, you could sit silently in supervisions if you really insisted, but to participate properly you just needed to build the confidence.

Is it fun? No, but it's a pretty accurate reflection of life after school: nobody in the real world gives you points for "couldn't say the right thing at the right time, but was thinking it"

However, you also mentioned a final project. You’d be shocked how much commissioning exists where people have their projects produced for them. I’m not talking an overly helpful study group, I mean straight up essay mills. Tools like ChatGPT make the bar for commissioning lower and cheaper. I don’t know how you can combat this and still have long-term projects like dissertations.

Average student debt after a 4 year degree is ~$35,000 after ~45 courses. Before even running the math it should be obvious the gigantic cost of higher ed over 4 years is entirely unrelated to what an instructor would be making for ~23 hours of work (barring a secret society of multi millionaires). I.e. the problem you're identifying is the vast majority of $ spent in higher ed is not going to time with your professors, not that doing so is itself expensive.

Hiring a n TA to delegate grading that’s hard to verify seems like will cost more than you think.

Could not disagree more. Researching, formulating arguments, can give a student a complete view of the subject that studying for tests misses. But, similarly to tests, it probably depends on the skill of the teacher in creating the right kind of written assignments.

Can you communicate it in real-time through writing? Maybe that's an accommodation that could be done?

I doubt that anyone could have produced a plausible report without actually doing the work. And to defend it one would have to understand the underlying physics and the work that was done. Plus I think my supervisor and the other two students who worked with me on the project would have remarked on my absence from the laboratory if I had simply bought the paper!

You can still have long term projects and dissertations so long as the degree is awarded for the defence of the dissertation rather than the dissertation itself; that is the student must demonstrate in a viva that they understand everything in the dissertation rather than merely regurgitate it.

Many years later I took a course in C# at a university in Norway and that was not merely open note but also open book (you could take the set book in). Again that gives the exam author the possibility to really discover who knows what.

I doubt that your rich middle-western student would have passed either of these

Similar re: exams, they were available but sticking rigidly to them didn’t help much.

Not saying this is a fatal flaw, but there is a bit of a tradeoff there.

There are certainly similar projects being completed by students every year, and doubtless those students are not cheaters, but for each dissertation like yours, there are probably 10 or more projects that are not collaborative and have no artefacts or supporting evidence other than a written report. Such projects are fairly easy to commission. For a reasonable price (potentially thousands of dollars) you can pay a poor research student in the same field as you to churn out a mid-tier dissertation. This can be detected with a viva, but the academics need to be very confident before accusing someone of cheating. More often than not, you can get away with it and just get a not great grade.

I think that in general the natural/formal sciences don't suffer nearly as much as social science and humanities do, simply because exams and labs tend to highlight irregularities, and cheaters are less likely to be drawn into "hard" fields. However, it still exists in every field.