Do AI detectors work? Students face false cheating accusations

AI is here to stay; new methods should be used to assess student performance.

I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with having 'calculators' in our pockets 24/7 now.

We need to accept that the world has changed; I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us.

This is overdue - we should be using interactive technology and not boring kids to death with a whiteboards.

Bureaucracy works to protect itself and protect ease of administration. Even organising hand on practical lessons is harder

I remember being told the same thing, but I happen to believe that it was a fantastic policy, with a lackluster explanation. The idea that you wouldn't have a calculator was obviously silly, even at the time, but underlying observation that relying on the calculator would rob you of the mental exercise the whole ordeal was supposed to be was accurate. The problem is that you can't explain to a room full of 12 year olds that math is actually beautiful and that the systems principles it imparts fundamentally shape how you view the world.

The same goes for essays. I hated writing essays, and I told myself all sort of weird copes about how I would never need to write an essay. The truth, that I've observed much later, is that structured thinking is exactly what the essay forced me to do. The essay was not a tool to asses my ability in a subject. It was a tool for me to learn. Writing the essay was part of the learning.

I think that's what a lot of this "kids don't need to calculate in their heads" misses. Being able to do the calculation was only ever part of the idea. Learning that you could learn how to do the calculation was at least as important.

If the level of effort made is high, but the outcome does not comply in some way, praise is due. If the outcome complies, but the level of effort is low, there is no reason for praise (what are you praising? mere compliance?) and you must have set a wrong bar.

Not doing this fosters people with mental issues such as rejection anxiety, perfectionism, narcissism, defeatism, etc. If you got good grades at school with little actual effort and the constant praise for that formed your identity, you may be in for a bad time in adulthood.

Teacher’s job is to determine the appropriate bar, estimate the level of effort, and to help shape the effort applied in a way that it improves the skill in question and the more general meta skill of learning.

The issue of judging by the outcome is prevalent in some (or all) school systems, so we can say LLMs are mostly orthogonal to that.

However, even if that issue was addressed, in a number of skills the mere availability of ML-based generative tools makes it impossible to estimate the level of actual effort and to set the appropriate bar, and I do not see how it can be worked around. It’s yet another negative consequence of making the sacred process of producing an amalgamation of other people’s work—something we all do all the time; passing it through the lens of our consciousness is perhaps one of the core activities that make us human—to become available as a service.

That basically never happens and the outcome is the result of some sort of struggle. Usually just a peaceful one in the courts and legislatures and markets, but a struggle nonetheless.

> new methods should be used to assess student performance.

Such as? We need an answer now because students are being assessed now.

Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all. Perhaps we're going to have to accept that and aggressively ration higher education by the limited amount of time available for human-to-human evaluations.

Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.

<< But that doesn't scale at all.

I realize that the level of effort for oral exam is greater for both parties involved. However, the fact it does not scale is largely irrelevant in my view. Either it evaluates something well or it does not.

And, since use of AI makes written exams almost impossible, this genuinely seems to be the only real test left.

My current best guess, is to hand the student stuff that was written by an LLM, and challenge them to find and correct its mistakes.

That's going to be what they do in their careers, unless the LLMs get so good they don't need to, in which case https://xkcd.com/810/ applies.

> Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.

Yup.

I hope the e/acc types are wrong, we're not ready.

It is new tech, but people do not treat it as such. They are not figuring it out. Its results are already being imposed. It is sheer luck that the individual in question choose to fight back. And even then it was only a partial victory:

"The grade was ultimately changed, but not before she received a strict warning: If her work was flagged again, the teacher would treat it the same way they would with plagiarism."

Isn't it easy to prevent students from using an AI if they are doing the exams in a big room? I mean when I was a student, most of my exams were written with just access to notes but no computers. Not that much resources needed to control that...

I am really quite confused about what you think the point of education is.

In general, the world (either the physical world or the employment world) does not care about effort, it cares about results. Someone laboriously filling their kettle with a teaspoon might be putting in a ton of effort, but I'd much rather someone else make the tea who can use a tap.

Why do we care about grades? Because universities and employers use them to quickly assess how useful someone is likely to be. Few people love biochemistry enough that they'd spend huge sums of money and time at university if it didn't help get them a job.

The problem is more with teachers lazily slapping an essay on a topic as a goto homework to eat even more of the already limited students' time with busywork.

That is not say there won't be cheaters ( they always are ), but that is what proctor is for. And no, I absolutely hated the online proctor version. I swore I will never touch that thing again. And this may be the answer, people need to exercise their free will a little more forcefully.

I think it's probably more concerning that spitting out the most generic mathematically formulaic bullshit on a subject is likely to get a decent mark. In that case what are we actually testing for?

Two decades ago, when I was in engineering school, grades were 90% based on in-person, proctored, handwritten exams. So assignments had enough weight to be worth completing, but little enough that if someone cheated, it didn't really matter as the exam was the deciding factor.

> Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all.

What? Sure it does. Every extra full-time student at Central Methodist University (from the article) means an extra $27,480 per year in tuition.

It's absolutely, entirely scalable to provide a student taking ten courses with a 15-minute conversation with a professor per class when that student is paying twenty-seven thousand dollars.

For a solution "now" to the cheating problem, regular exam conditions (on-site or remote proctoring) should still work more or less the same as they always have. I'd claim that the methods affected by LLMs are those that could already be circumvented by those with money or a smart relative to do the work for them.

Longer-term, I think higher-level courses/exams may benefit from focusing on what humans can do when permitted to use AI tools.

Finding errors in a text is a useful exercise, but clearly a huge step down in terms of cognitive challenge from producing a high quality text from scratch. This isn't so much an alternative as it is just giving up on giving students intellectually challenging work.

> That's going to be what they do in their careers

I think this objection is not relevant. Calculators made pen-and-paper arithmetic on large numbers obsolete, but it turns out that the skills you build as a child doing pen-and-paper arithmetic are useful once you move on to more complex mathematics (that is, you learn the skill of executing a procedure on abstract symbols). Pen-and-paper arithmetic may be obsolete as a tool, but learning it is still useful. It's not easy to identify which "useless" skills are still useful as to learn as cognitive training, but I feel pretty confident that writing is one of them.

Same with homework. If everyone has to solve the same 10 problems, divide and conquer saves everyone a lot of time.

Of course, you're only screwing yourself because you'll negatively impact your learning, but that's not something you can easily convince kids of.

In person oral exams (once you get over the fear factor) work best, with or without (proctored!) prep time.

Maybe it doesn't scale as well, but education is important enough not to always require maximal efficiency.

Can't perform any research, compare conflicting sources, or self-reflection.

When I was in university (Humanities degree), we had to do lots of mandatory essays throughout the year but they counted little towards your overall mark, maybe 10% iirc.

The majority of marks came from mid-year & end-of-year exams.

A simple change to negate AI is to not award any points for work outside exams — make it an optional chance to get feedback from lecturers. If students want to turn in work by AI, it's up to them

I completely agree, but then again it seems to me that society also functions according to many norms that were established due to historical context; and could / should be challenged and replaced.

Our education system was based on needs of the industrial revolution. Ditto, the structure of our working week.

My bet: We will see our working / waking lives shift before our eyes, in a manner that's comparable to watching an avalanche in the far distance. And (similarly to the avalanche metaphor) we'll likely have little ability to effect any change.

Fundamental questions like 'why do we work', 'what do we need' and 'what do we want' will be necessarily brought to the fore.

It doesn't scale if performed by a human. But what if... we employ AI to conduct the voice exams?

1. Whoever determined that he does not “deserve” this is wrong. There may be other constraints, but no one gets to frame it as “deserves” when a child wants to learn something.

2. If a teacher is unable to teach Johnny to write a for loop, despite Johnny’s genuine utmost motivation, I would question teacher’s competence or at least fit.

3. Like any mentor, a professor in higher ed may want to choose whom to teach so that own expertise and teaching ability is realized to the fullest. Earlier in life, elementary school teacher’s luxury to do so may be limited (which is why their job is so difficult and hopefully well-compensated), and one bailing on a kid due to lack of patience or teaching competence is detestable.

4. If Johnny continues to pursue this with genuine utmost motivation, he will most likely succeed despite any incompetent teachers. If he does not succeed and yet continues to pursue this to the detriment to his life, that is something a psychologist should help him with.

As for Timmy, if he learns to produce the expected result with least effort, for which he receives constant praise from the teacher, and keeps coasting this way, that does him a major disservice as far as mental mental and self-actualisation in life.

I still feel like AI detectors would work well if we have access to the exact model, output probabilities of tokens, We can just take a bit of given text, and calculate the cumulative probability that the AI would complete it exactly like that.

On the contrary; with AI it scales better than ever before.

We would need to increase the amount of teaching staff by well over 10x to do this. The costs would be astronomical.

It never was. It's just even more ineffective now that AI exists, than before.

The central example of this is college admissions statements. Some kids have the advantage both of parents who can afford to give them the experiences that look good on such an essay (educational trips to Africa, lessons in two musical instruments, one-on-one golf coaching, that kind of thing), and who can hire tutors to "support" them in writing the essay. AI just makes the tutor part accessible/affordable for a wider segment of the population.

It would be naive to assume that, pre-AI, there was not a "gray" essay-coaching market as well as the "dark" essay-writing as a service market. That market still works better than AI in many cases.

I believe it's still true today, but with future AI systems even highly abstract math is under the danger.

My sense is that if writing was entirely optional, it would be too easy for students to let it slide; having 10% count toward your grade is a good "nudge" to get honest students to actually do the work. I'd just give a little admonition at the beginning saying that I'm not going to bother checking if you use LLMs, but if you do you're an idiot, because you'll be completely unprepared for the hand-written essay you have to compose yourself in the exams.

All the low paid, physically laborious work is not affected by AI, so there will be plenty of work, especially with aging populations around the world.

The question is will it be worth doing (can the recipients of the work pay enough) without being able to provide the dream of being able to obtain a desk job for one’s self or their children.

I think that we'll see fundamental changes, but it will be based on cheaper consumer goods because all of the back end white collar labor that adds costs to them will be (for all intents and purposes) free.

But we will see the absolute destruction of the middle class. This will be the death blow. The work week will change, but only because even more people will work multiple part time jobs. We'll think about what we need, but only because we'll have cheap consumer goods, but no ability to prepare for the future.

I think it's bleak. Source: most of human history. We're not, as a species, naturally altruistic. We're competitive and selfish.

Countries have no problem spending astronomical amounts on old people. If the country wants productive young people, the country will find a way.

> The costs would be astronomical.

Those 200 students have paid the college $549,600 for your class.

The costs are already astronomical.

Is it so unreasonable for some of that money to be spent on providing education?

This assumes that homework helps kids learn, or that the knowledge required to succeed in school will help kids once they graduate.

The poem was assigned to us, but for some reason the subject matter really chimed with me personally. I thought about it a lot, and—as a result—ended up writing a great essay.

Because I did well, I was accused of cheating in front the class.

Teachers are definitely fallible.

(AI alone for assessment is even worse than AI proctoring!)

Let's not assume a lot right now. OpenAI and other companies are torching through cash like drunken socialist sailors. Will AI be here as a Big Data 2.0 B2B technology? Most likely, but a viable model where students and laypeople have access to it? To be seen.

We all mooched off of dumb VC money at one point or another. I acquired a few expensive watches at Fab dot com at 80% off when they were giving money away, eh.

Looking back on it, I think it could be weirdly precient.

Two classes of society; one living a life of leisure, the other fighting on the plains.

(.. maybe minus Connery in a mankini)

I especially agree that essay writing is hugely useful. I'd even go as far as saying, the ability to think clearly is fundamental to a happy life.

Low paid physically labourorious work has been increasingly automated since the wheel started helping potters.

The history of computers often ends up referencing Jacquard looms because it used punched cards.

LLMs have been demonstrated controlling various robots.

AI's coming for all jobs, it's just a question of which jobs cease first.

The process of reading textual material, thinking about it, and then producing more textual material about the stuff you just read (and maybe connecting it to other stuff that you've read in the past) is a critical way of developing thinking skills and refining your ability to communicate to an audience.

The value of that shouldn't be overlooked just like the value of basic numeracy shouldn't be overlooked because we all carry calculators.

You're right that it would be better if post secondary institutions would test people's ability to think in more ways than just what they can regurgitate onto a piece of paper, if only because that can be easily cheated but that doesn't mean that there isn't personal benefit in the experience of writing an essay.

I may not be the best writer but I am a better writer because I wrote essays in university, and I may not be great at math but I can reason and estimate about a variety of things because I have taken many math courses. These things have ultimately made me a better thinker and I am grateful to have had that imparted to me.

That threat may be enough to dissuade students from cheating with AI.

You can run GPT-4-equivalent models locally. Even if all software and hardware advancements immediately halt, models at the current level will remain available.

On a more serious note - US removed cursive from their curriculum almost two decades ago - something i cant wrap my head around as cursive is something the rest of the world(?) uses starting in middle school and onwards through the whole adult life.

Sorry, English is not my first langage what is this expression ? Why does the sailor as to be socialist ? Google didn't help me with this one.

So the obvious solution is to make students to talk with an AI, which would grade their performance. Or, maybe the grading itself could be done by a minimum wage assistant, while AI would lead the discussion with a student.

It seems to me that this is pretty much immune to plagiarism as well as being much better for the student.

If done right they would go from VC money with an expected exit to government money that overpays for incompetence because our only way out of deficit spending is through more debt and inflation.

I believe that to learn well, you need to be challenged, but not too much. Ability-based evaluation only does that for students whose abilities happen to line up with the expected standard. It is bad both for gifted students and for struggling students.

There is a sad, depressing world out there. One of my parents work at an elderly home, and the shit that happens there is just wild. Zero responsibility and accountability. Deliberate killing of people out of pure inconvenience, etc.

I am in favor of a "social support network".

[1] Requires money, e.g. pension, which is increasingly less, and they keep increasing the age.

For longer essays, you can just build in an oral examination component. This face time requirement is just not that hard to include given that even in lecture hall style settings you can rely on graduate student TAs who do not really cost anything. The thing is that the universities don't want to change how they run things. Adjuncts in most subjects don't cost anything and graduate students don't cost anything. They earn less than e.g. backroom stocking workers. This is also why they, by and large, all perform so poorly. 30 minutes of examiner time costs maybe $11 or less. Even for a lecture class with 130 students, that's under $1,500. Big woop.

There are some small changes to grading practices that would make life very hard for AI cheaters, such as even cite checking a portion of citations in an essay. The real problem is that US universities are Soviet-style institutions in which gargantuan amounts of cash are dumped upon them and they pretend to work for it while paying the actual instructors nothing.

Why bother training and educating the young people who are already here when you can just import them from poorer countries?

Here is the brand new method - asking verbal questions in person and evaluating answers. Also allow high tech aides in the form of chalk and blackboard

We all know they'll just exploit grad students rather than hire real teachers.

In fact, the only reason I use the word 'importunate' to describe myself, is because that's what my undergrad advisor called me.

So I at least was able to get well over 30m with each professor to discuss whatever I wanted. But likely that's b/c there wasn't a lot of competition.

Please expand?

It is, because I'm becoming tired with the current AI hype. It lasts too long to be funny.

OTOH, professor talking with a student is a good way to assess the academic performance of the student, but there are some caveats beyond costs. For example, professor will struggle to be an objective judge. Moreover even if they succeed, they would face accusations of discrimination in any case.

AI could solve this problem, but I'm not sure if AIs will be up to a task of leading the discussion. Though maybe if you try to assess students on their ability to catch AI on a hallucinated bullshit...

There are lots of things I spent a lot of time learning school that I rarely use, but see the value in having learnt. Cursive, beyond a very basic level, is not one of those things.

Though I’m no education expert, perhaps there is a subliminal value to spending all that time.

By your own logic, the student who fills the kettle with the spoon has produced the expected result. Fast enough with the spoon and sky’s the limit, right?

A good teacher, while praising the effort, would help them find out about the tap. Not praising the effort would give the opposite signal! You have worked hard, and through no fault of your own (no one has built-in knowledge about the tap) you were essentially told that was for nothing?!

And if you have learned the tap, do you want to be done with it? Or be pushed to keep applying the same effort as with the spoon, but directed more wisely knowing that there’s a tap? Imagine what heights would you reach then!

The worst teachers are in whose class 30% of the students are filling their kettle with spoons all their time, 30% simply dip them into the puddle and never get used to do the work, 30% give up because what is even the point of filling the kettle when their home has a hot water dispenser.

Love your analogy, by the way.

This is similar to how the AI companies mostly spending VC’s money buying these accelerators from nVidia.

For all their problems, 5000 word take home assignments in Microsoft Office have a lot in common with the activities of a junior management consultant, NGO writer, lawyer or business analyst. And same with for scientists but with Latex.

I’d rather hire a lawyer who could only do their job with AI than one who couldn’t use a computer to create documents or use digital tools to search case law.

Happens in tax filing too.

> Some kids are going to learn more quickly than others, and for them, no effort will be required.

If no effort is required, then the bar is wrong.

It's like looking at a hand carved match-stick judging the result as low effort, not knowing that they started with a seed.

Of course the current setup works badly for those who explain it much better by speaking.

Yeah, the General Problems exam was a nightmare, I think the professors competed each year to come up with the toughest questions. Getting 50% was an excellent score.

It did force you to learn all the material though, especially as at the end of 3-4 years you may have forgotten some of it, like Optics or whatever. It was pretty hardcore though, especially compared to my friends studying other subjects.

You are in a room with a sheet of paper and a pen. Go.

You’re acting as if 2010 was a hundred years ago.

Why not have the testing done externally, by really neutral persons?

But AIs and especially LLMs are way too unreliable for the foreseeable future.

But yes, I don’t know how clear must I be about it—this is learning (for very young humans still psychologically immature), that’s exactly why it has to be spelled out that evaluation must be on the effort, precisely because it is never on the effort in any other activity in adulthood.

When you audition for a symphony you perform behind a curtain and are required to wear soft slippers (so they can't tell if you are a wearing high heals - female).

We can probably use voice changers so the examiner cannot tell who you are by your voice, but those tend to be fatiguing.

But in principle this is not a problem for me, I already spend at least this much time grading papers, and an oral exam would be much more pleasant. The real problems will come up when (1) students are forced to schedule these 15-minute slots, and (2) they complain about the lack of time and non-objective grading rubric.

Those of you who severely lack any of the three will be penalized. Just like someone who can discuss a topic orally but could not write it up would be penalized.

    > Written assay evaluation is not and has never been an effective evaluation. 

I've kept a blog for almost 20 years now and one thing is for sure: well-structured writing is very different from an oral exam the writing allows for restructuring your thoughts and ideas as you go and allows for far more depth.

I don't think, for most folks, that they could have as much depth in an F2F as they could in their writing with the exception of true experts in their fields.

The written essay has a cohesiveness and a structure to it that provides a better framework for evaluation and conveyance of information.

I'm not so sure that writing takes more time than studying. For starters, you don't have to memorize anything, and you can limit yourself to the assigned topic.

Of course, it can be that students don't take studying for an oral exam seriously, and trust the teacher to only ask superficial questions.

Where do you get this from? The people I know with pure research positions get paid basically the same (after correcting for 'rank' and seniority) as those who split their time between research and teaching.

Yes, there are people who have trouble with public speaking to a debilitating degree, but it would be excessively rare for someone to not at the very least in a one on one with their professor/teacher be able to be so badly affected as to not seem they've even heard of a topic or at least be able to prove they've worked on it to a certain degree.

I would be immediately skeptical of any student who claims they are completely unable to explain their knowledge unless they are allowed to work in complete isolation with nobody to monitor they aren't cheating in some way.

Lots of elite prep schools in the US use a similar system, for similar reasons.

Extremely optimistic about the benefits of new tech, downplay all the risks, my experience of self-identifying e/acc people has generally been that they assume AI alignment will happen by default or be solved in the marketplace… and specifically where I hope they're wrong, is that many seem to think this is all imminent, as in 3-5 years.

If they're right about everything else then we're all going to have a great time regardless of when it comes, but I don't see human nature being compatible with even just an LLM that can do a genuinely novel PhD's worth of research rather than "merely" explain it or assist with it (impressive though even those much easier targets are).

I learned cursive, then reverted to print, but when I entered a phase of my life where I needed to write several pages a day I quickly went back to (a custom variant of) cursive because it was faster to write in a legible way than print.

When I rush print it quickly becomes illegible. When I rush my cursive it doesn't look quite as nice as it does when I'm writing steadily, but I can still read what I wrote ten years later.

From what I can tell it works because cursive letters are defined in a shape that lends itself to a quick moving pen. Once you learn that shape (both to write and read), you can quickly get words down on a page and then understand them later. If you just try to slur your print in an unprincipled way your letters distort in ways that make them harder to tell apart.

Now, I imagine someone could develop a slurred print that doesn't have connections between letters, but I'd probably call that a cursive anyway.

On setting up expectations for adulhood, I think this is exactly backwards:

> If you got good grades at school with little actual effort and the constant praise for that formed your identity, you may be in for a bad time in adulthood.

Praising a child for effort without results seems way more likely to set them up for a surprisingly bad time as an adult. My personal experience has been that the "good grades/rewards without effort" thing has continued and seems pretty likely to continue through adulthood as long as you go into some kind of engineering.

The top earning professors in the nation in mathematics are all very good research mathematicians

This will be it. [edit: for all education I mean, not just college] Computers are going to become a bigger part of education for the masses, for cost reasons, and elite education will continue to be performed pretty much entirely by humans.

We better hope computer learning systems get a lot better than they’ve been so far, because that’s the future for the masses in the expensive-labor developed world. Certainly in the US, anyway. Otherwise the gap in education quality between the haves and have nots is about to get even worse.

Public schools are already well on the way down that path, over the last few years, spurred by Covid and an increasingly-bad teacher shortage.

I did nearly all my exams on a computer.

At one point the best writing tool was the fountain pen. It was a great invention and it had an appropriate script: cursive, which was the natural thing to do given how the ink flowed.

However kids are messy and you really want them to use pencils because they don't have flowing ink. The reason for cursive in the first place was the flowing ink, so when we switched away from flowing ink, there was no reason to write in cursive.

Except of course to waste the only resource everybody agrees is okay to waste: kids time.

However, +90% of students will not now do any of that work. I got out of teaching (coincidentally) before LLMs appeared, and even then +80% of students did not experience that benefit of the essay process even with a grade (and plagiarism consequences) to motivate them. Now that decent-ish prose is a few keystrokes or Siri-led "chats" away, that's what they're going to do. That's what they're going to do.

I know of - I think it's up to four, now - former colleagues taking early retirement, or changing careers, rather than continue teaching Humanities in a world of LLMs.

That strikes me as a workable bottom line.

Calculators are just one analogy, there is no guarantee it will work out that way. It's just as likely that this over-technologization of the classroom will go the way of whole-language reading education.

This is an absolute disgrace. And then these are the people who lecture you on "inclusion".

While writing term papers is a skill that is only minimally useful in the real world (save for grant writers and post docs, pretty much), the patterns of thinking it encourages are valuable to everything that isn’t ditch digging.

Maybe we can outsource this part of our cognition to AI, but I’m skeptical of the wisdom of doing so. Are we all going to break to consult ChatGPT in strategy meetings?

That was just a simple quip to shut down student bellyaching. Even before we had pocket calculators, it was never a strong answer. It just had to hold over long enough so when you realized it was bad answer you weren't that teacher's problem anymore.

The actual answer was that they're complaining about a minor inconvenience designed for reinforcement, and if they really did need a calculator for the arithmetic on a test designed deliberately designed to be taken without a calculator, then they don't belong in that class.

I think the hard answer is that to some extent you just have to learn to. I mean, you could sit silently in supervisions if you really insisted, but to participate properly you just needed to build the confidence.

Is it fun? No, but it's a pretty accurate reflection of life after school: nobody in the real world gives you points for "couldn't say the right thing at the right time, but was thinking it"

It was not just "turn in the paper at the end" but turn in your topic with a paragraph describing it. Then make an outline, then bibliography of the sources we were using. During the process we had to use 3x5 index cards with various points, arguments, facts, and the specific pages in the books listed in our bibliography. We did this because this was later used to make footnotes in our paper.

By structuring the project this way and having each milestone count as 5-10% of the overall grade it made it much harder to cheat and also taught us how to organize a research paper.

I suppose you could ask ChatGPT to do the entire paper and then work backwards picking out facts and making the outline etc.

My office mate in engineering was trying to get funding for a start-up. I was trying to get a consulting business off the ground. Neither of us achieved those things, but whatever. He got a teaching gig at the community college, which is unionized and actually a pretty good situation. I found a regular day job through his network.

A friend of mine had an adjunct gig in the humanities, and used his off-time to learn how to code.

A lot of academic spouses get adjunct gigs, especially if they want to balance part time work with child care.

However, you also mentioned a final project. You’d be shocked how much commissioning exists where people have their projects produced for them. I’m not talking an overly helpful study group, I mean straight up essay mills. Tools like ChatGPT make the bar for commissioning lower and cheaper. I don’t know how you can combat this and still have long-term projects like dissertations.

There's a good question about the future and utility of long at-home research paper projects in school, but it's not a cornerstone of education.

In 9th grade I procrastinated the semestral paper so much that I bought an essay online that explored unexpected gay themes in Ray Bradbury's corpus of work. I was so lazy I didn't even read it first, only skimmed it, and then back to Runescape. So it's not like this is a new problem due to LLMs, and I think take-home semester projects are all quite bad for these reasons that predate LLMs.

(It turned out to be such a phenomenally audacious essay that my teacher started fascinated email correspondence with me about it and I was forced to not only study the essay but also read the quoted parts of his work. Ugh, backfire.)

The Time Machine is a very famous example.

Are they? Is there any evidence of correlation between these two groups of people?

But, for example, high school AP English exam is 3 45 minute essays (plus multiple choice). You have the read the passages, compare/contrast, etc.

Average student debt after a 4 year degree is ~$35,000 after ~45 courses. Before even running the math it should be obvious the gigantic cost of higher ed over 4 years is entirely unrelated to what an instructor would be making for ~23 hours of work (barring a secret society of multi millionaires). I.e. the problem you're identifying is the vast majority of $ spent in higher ed is not going to time with your professors, not that doing so is itself expensive.

Hiring a n TA to delegate grading that’s hard to verify seems like will cost more than you think.

2040? 2050?

Could not disagree more. Researching, formulating arguments, can give a student a complete view of the subject that studying for tests misses. But, similarly to tests, it probably depends on the skill of the teacher in creating the right kind of written assignments.

Where is the transition? At some point ability is more important than effort.

In isolation, I don't think a model necessarily becomes less useful over time. It'll still be as good at summarizing articles, translating text, correcting grammar, etc. for you as it is today.

If things do continue to advance and new models are released, which I think is likely, the old ones become less useful by comparison and in situations where there's competition. But then, through hardware/algorithmic improvements, better models also become feasible for universities/open-source groups/individuals - so you shouldn't be stuck with a 2024 era model.

Later I found out it was developed for use with a fountain pen, designed with the idea that a correctly-faced nib would make some strokes bolder and others very faint, and to keep the nib always moving in a kind of flow to avoid spots, and to make it natural to keep the nib faced the correct way(s), plus with even more attention to avoiding raising the pen than most cursives, for similar reasons of avoiding spotting. That made all the downsides make sense—it's far less ugly and easier to read when written with a fountain pen, and may well be faster than many other similarly-clean methods of writing with one.

Why the hell we were still learning that hand decades into the dominance of the ballpoint, remains a question.

At the end of the day cursive writing is a hobby, not a skill. We don't need it anymore. It wastes priceless learning time at a critical juncture in our intellectual development.

Can you communicate it in real-time through writing? Maybe that's an accommodation that could be done?

I'm currently a PhD candidate, and our program includes separate written and oral qualifying exams during the first year or two, and a public defense of the dissertation at the end. I thought some minor variation of this was nearly universal.

It's also my observation, by the way, that the public dissertation defense (and even the written dissertation itself) is less of a big deal than outsiders tend to think. What matters is doing the research that the advisor / committee wants, and working on some number of papers that get accepted into workshops / conferences / journals (depending on the field). Everything else seems to be kind of a check-the-box formality. By the time the committee agrees that someone has done enough to defend, it's pretty much a done deal.

My particular school and department is interesting because it doesn't have any hard requirement for publications, and it aims to have students finish a PhD in about three years of full-time work (assuming one enters the program with a relevant master's degree already in-hand). There has been some tension between the younger assistant professors (who are still fighting for tenure) and the older full professors (who got tenure in, say, the 1990s). In practice, the assistant professors expect to see their students publish (with the professors as co-authors, of course) and would strongly prefer to see a dissertation comprised of three papers stapled together, regardless of the what the school and department officially says. The full professors, on the other hand, seem to prefer something more like a monograph that is of "publishable" quality, maybe to be submitted somewhere after graduation. They argue that the assistant professors should be able to judge quality work for themselves instead of outsourcing it to anonymous reviewers. Clearly, there are different incentives at play.

One of the reasons I mentioned the viva was an example of how we can decouple production of some work from an assessment of quality and some reason to believe that some candidate is capable of the work without assistance.

It would be unreasonable to spend five or so years working under examination conditions. But that doesn’t mean we can’t subsequently examine a candidate to determine likely authorship amongst other things.

I doubt that anyone could have produced a plausible report without actually doing the work. And to defend it one would have to understand the underlying physics and the work that was done. Plus I think my supervisor and the other two students who worked with me on the project would have remarked on my absence from the laboratory if I had simply bought the paper!

You can still have long term projects and dissertations so long as the degree is awarded for the defence of the dissertation rather than the dissertation itself; that is the student must demonstrate in a viva that they understand everything in the dissertation rather than merely regurgitate it.

It didn't explain the goals well enough to parents, and many teachers didn't have the number sense themselves leading to many of the examples are passed around showing how the whole process is broken. There is also a question of if even works well, as it is somewhat akin to teaching someone the shortcut on how to do something before they have mastered the long way of doing it. Many experts in their fields have shortcuts, but they don't teach them directly to juniors in the field as there is value in learning how to do it the long hard way, as often times shortcuts are limited and only an understanding of the full process provides the knowledge of when best to apply different shortcuts.

Many years later I took a course in C# at a university in Norway and that was not merely open note but also open book (you could take the set book in). Again that gives the exam author the possibility to really discover who knows what.

I doubt that your rich middle-western student would have passed either of these

Similar re: exams, they were available but sticking rigidly to them didn’t help much.

The students paid me nothing. The university provides some TAs, that's it. But even if they gave me all of that money in cash to spend, this would be totally impossible.

I'm supposed to grade a student based on 1 conversation? Do you know how grading and teaching work? Can you imagine the complains that would come out of this process? How unfair it is to say that you have one 15 minute shot at a grade?

But fine, even if we say that I can grade someone based on 1 conversation. What am I supposed to ask during this 15 minute conversation? Because if I ask the every student the same thing, they'll just share the questions and we're back to being useless.

So now I need to prep unique questions for 200 people? Reading their background materials, projects, test results, and then thinking of questions? I need to do that and review it all before every session.

Even with a team of TAs this would be impossible.

But even if I do all of this. I spend hours per student to figure out what they did and know. I ask unique questions for 15 minutes so that we can talk without information leakage mattering. You know what the outcome will be? Everyone will complain that my questions to them were harder than those that I asked others. And we'll be in office hours with 200 people for weeks on end sorting this out and dealing with all the paperwork for the complaints.

This is just the beginning of the disaster that this idea would be.

It's easy to sit in the peanut gallery and say "Oh, wow, why didn't my arm surgery take 10 minutes, they just screwed two bones together right?" until you actually need to do the thing and you notice that it's far more complex than you thought.

A lot of adjuncts sit around in precarious financial situations, developing serious mental health issues, and drinking problems because the system taught them that this is a form of success.

Going to industry and making money? That's failure. That's an "alternate career". Not scraping by in a system that couldn't care less about you. That's success.

It's pretty vile. I've never had a student become an adjunct. It would be a personal failure that I haven't given them the tools to thrive.

I disagree.

I've been writing a novel now for… far too long, now. Trouble is, whenever I read it back, I don't like what I've done.

I could totally just ask an LLM to write one for me, but the hard part is figuring out what parts of those 109,000 words of mine sucked, much more so than writing them.

(I can also ask an LLM to copyedit for me, but that only goes so far before it gets confused and starts trying to tell me about something wildly different).

> It's not easy to identify which "useless" skills are still useful as to learn as cognitive training

Indeed. And you may also be correct that writing is one such skill even if only just to get the most out of an LLM.

What I'm describing here is very much a best guess from minimal evidence and the current situation; I would easily drop it for another idea if I saw even very minimal evidence for a better solution.

Not saying this is a fatal flaw, but there is a bit of a tradeoff there.

There are certainly similar projects being completed by students every year, and doubtless those students are not cheaters, but for each dissertation like yours, there are probably 10 or more projects that are not collaborative and have no artefacts or supporting evidence other than a written report. Such projects are fairly easy to commission. For a reasonable price (potentially thousands of dollars) you can pay a poor research student in the same field as you to churn out a mid-tier dissertation. This can be detected with a viva, but the academics need to be very confident before accusing someone of cheating. More often than not, you can get away with it and just get a not great grade.

I think that in general the natural/formal sciences don't suffer nearly as much as social science and humanities do, simply because exams and labs tend to highlight irregularities, and cheaters are less likely to be drawn into "hard" fields. However, it still exists in every field.

> The students paid me nothing.

Well gee, there I was thinking they were paying $27,480 per year for "tuition"

Later, people learned that and started to buy diploma: faster, cheaper, no risk of failing the final exam.

> My personal experience has been that the "good grades/rewards without effort" thing has continued and seems pretty likely to continue through adulthood as long as you go into some kind of engineering.

Based on my observation, people who are comfortable doing the work in engineering achieve completely different heights than people who got used to coasting. As applied ML spreads across industries, the difference between the competitiveness of those two categories will only become more pronounced. Furthermore, from my observation those who got used to coasting suffer from issues like perfectionism, narcissism, rejection aversion, and similar.

Sooner or later in adulthood not doing the work stops achieving results deserving praise.

> Praising a child for effort without results seems way more likely to set them up for a surprisingly bad time as an adult.

Not “without results”. Results are critical. However, if results do not comply with whatever requirements, that is not a factor in whether to reward, unless you reward compliance. Rewarding compliance has to happen to some degree, but should not be overdone unless your goal is to foster uncreative cogs.

There is no benefit in always being praised for your efforts if you cannot deliver the goods.