How University Students Use Claude

My wife is an accounting professor. For many years her battle was with students using Chegg and the like. They would submit roughly correct answers but because she would rotate the underlying numbers they would always be wrong in a provably cheating way. This made up 5-8% of her students.

Now she receives a parade of absolutely insane answers to questions from a much larger proportion of her students (she is working on some research around this but it's definitely more than 30%). When she asks students to recreate how they got to these pretty wild answers they never have any ability to articulate what happened. They are simply throwing her questions at LLMs and submitting the output. It's not great.

Target the cheaters with pop quizzes. Prof can randomly choose 3 questions from assignments. If students cant get enough marks on 2/3 of them they are dealt a huge penalty. Students that actually work through the problems will have no problems with scoring enough marks on 2/3 of the questions. Students that lean irresponsibly on LLMs will lose their marks.

Homework would still be assigned as a learning tool, but has no impact on your grade.

not those who did actually do the work, and learnt.

The change ought to be that students are allowed to be failed, and this should be a form of punishment for those who "cheat".

I'm more worried about those who will learn to solve the problems with the help of an LLM, but can't do anything without one. Those will go under the radar, unnoticed, and the problem is, how bad is it, actually? I would say that a lot, but then I realize I'm pretty useless driver without a GPS (once I get out of my hometown). That's the hard question, IMO.

...But then.

I don't see the former as that much of a problem.

We’re either handicapping our brightest, or boosting our dumbest. One part is concerning, the other encouraging .

If there is a difference, then fundamentally LLMs cannot solve problems for you. They can only apply transformations using already known operators. No different than a calculator, except with exponentially more built-in functions.

But I'm not sure that there is a difference. A problem is only a problem if you recognize it, and once you recognize a problem then anything else that is involved along the way towards finding a solution is merely helping you solve it. If a "problem" is solved for you, it was never a problem. So, for each statement to have any practical meaning, they must be interpreted with equivalency.

You'll acquire advanced knowledge and skills much, much faster (and sometimes only) if you have the base knowledge and skills readily available in your mind. If you're learning about linear algebra but you have to type in every simple multiplication of numbers into a calculator...

In some of the CS tests, coding by hand sucks a bit but to be honest, they're ok with pseudo code as long as you show you understand the concepts.

You as the employer are liable, a human has real reasoning abilities and real fears about messing up, the likely hood of them doing something absurd like telling a customer that a product is 70% off and them not losing their job is effectively nil. What are you going to do with the LLM, fire it?

Data scientist and people deeply familiar with LLMs to the point that they could fine tune a model to your use case cost significantly more than a low skilled employee and depending on liability just running the LLM may be cheaper.

As an accounting firm ( one example from above ) far as I know in most jurisdictions the accountant doing the work is personally liable, who would be liable in the case of the LLM?

There is absolutely a market for LLM augmented workforces, I don't see any viable future even with SOTA models right now for flat out replacing a workforce with them.

I have the base knowledge and skill readily available to perform basic arithmetic, but I still can't do it in my mind in any practical way because I, for lack of a better description, run out of memory.

I expect most everyone eventually "runs out of memory" if the values are sufficiently large, but I hit the wall when the values are exceptionally small. And not for lack of trying – the "you won't always have a calculator" message was heard.

It wasn't skill and knowledge that was the concern, though. It was very much about execution. We were tested on execution.

> If you're learning about linear algebra but you have to type in every simple multiplication of numbers into a calculator...

I can't imagine anyone is still using a four function calculator. Certainly not in an application like learning linear algebra. Modern calculators are decidedly designed for linear algebra. They need to be given the rise of things like machine learning that are heavily dependent on such.

Some people argue that it doesn’t matter if there is mistakes (it depends which actually) and with time it will cost nothing.

I argue that if we give up learning and let LLM do the assignments then what is the extent of my knowledge and value to be hired in the first place ?

We hired a developper and he did everything with chatGPT, all the code and documentation he wrote. First it was all bad because from the infinity of answers chatGPT is not pinpointing the best in every case. But does he have enough knowledge to understand what he did was bad ? And then we need people with experience that confronted themselves with hard problems and found their way out. How can we confront and critic an LLM answer otherwise ?

I feel student’s value is diluted to be at the mercy of companies providing the LLM and we might loose some critical knowledge / critical thinking in the process from the students.

As a comment upthread said, let them cheat on the take home as much as they want to, they're still going to fail the exam.

Spotify CEO is channeling The Two Bobs from Office Space: "What are you actually doing here?" Just in a nastier way, with a kind of prisoner's dilemma on top. If you can get by with an agent, fine, you won't bother him. If you can't, why can't you? Should we replace you with someone who can, or thinks they can?

Spotify CEO is not his employees' friend.

However I really don't need to implement some weird algorithms myself every time (ideally I am using a well tested Library) but the point is that you learn to be able to but also to be able to modify or compose the algorithm in ways the LLM couldn't easily do.

Depends on the country and educational system I suppose, but I do believe professors in many places get in trouble for failing too many students. It's right there in the phrasing.

If most students pass and some fail, that's fine. Revenue comes in, graduates are produced, the university is happy.

If most students fail, revenue goes down, less students might sign up, less graduate, the university is unhappy.

It's a tragedy of the commons situation, because some professors will be happy to pass the majority of students regardless of merit. Then the professors that don't become the problem, there's something wrong with them.

Likewise, if most universities are easy and some are really hard, they might not attract students. The US has this whole prestige thing going on, that I haven't seen all that much in other countries.

So if the students overall get dumber because they grow up over relying on tools, the correction mechanism is not that they have to work harder once the exam approaches. It's that the exam gets easier.

I'm running into similar issues trying to use LLMs for logic and reasoning.

They can do it (surprisingly well, once you disable the friendliness that prevents it), but you get a different random subset of correct answers every time.

I don't know if setting temperature to 0 would help. You'd get the same output every time, but it would be the same incomplete / wrong output.

Probably a better solution is a multi phase thing, where you generate a bunch of outputs and then collect and filter them.

Interesting! :D Do you mind sharing the prompt(s) that you use to do that?

Thanks!!

It's technically allowed on an individual basis, but the economics don't work for any institution to attempt to raise its bar.

If institutions X and Y grant credential Z, and X starts failing a third of its students, who would apply to go there?

An agent can't do it. It can help you like a calculator can help you, but it can't do it alone. So that means you've become the programmer. If you want to be the programmer, you always could have been. If that is what you want to be, why would you consider hiring anyone else to do it in the first place?

> Spotify CEO stated on X that before asking for more headcount they have to justify not being able to do the job with an agent.

It was Shopifiy, but that's just a roundabout way to say that there is a hiring freeze due to low sales (no doubt because of tariff nonsense seizing up the market). An agent, like a calculator, can only increase the productivity of a programmer. As always, you still need more programmers to perform more work than a single programmer can handle. So all they are saying is that "we can't afford to do more".

> The company will and is just using the agent as well …

In which case wouldn't they want to hire those who are experts in using agents? If they, like Shopify, have become too poor to hire people – well, you're screwed either way, aren't you? So that is moot.

Then they're on a video call and their vocabulary is wildly different, or they're very clearly a recent immigrant and struggle with basic sentence structure such that there is absolutely zero change their discussion forum persona is actually who they are.

This has happened at least once in every class, and invariably the best classes in terms of discussion and learning from other students are the ones where the people using AI to generate their answers are failed or drop the course.

In Italy university keeps getting easier because their funding is tied to not failing students.

I'm in a 100%-online grad school but they proctor major exams through local testing centers, and every class is at least 50% based on one or more major exams. It's a good way to let people use LLMs, because they're available, and trying to stop it is a fool's errand, while requiring people to understand the underlying concepts in order to pass.

As a society we need to be okay with failing people who deserve to fail and not drag people across the finish line at the expense of diluting the degrees of everyone else who actually put in effort.

  Where will all those new students find a job if :
  - they did not learn much because LLM did work for them 
  - there is no new jobs required because we are more productive ?

This assumes, of course, an institution is actively trying to raise the academic bar of its student population. Most schools are emphatically not trying to do this and are focused more on just increasing enrollment, getting more tax dollars, and hiring more administrators.

Never in the history of humans have we been content with stagnation. The people who used to do manual calculations soon joined the ranks of people using calculators and we lapped up everything they could create.

This time around is no exception. We still have an infinite number of goals we can envision a desire for. If you could afford an infinite number of people you would still hire them. But Shopify especially is not in the greatest place right now. They've just come off the COVID wind-down and now tariffs are beating down their market further. They have to be very careful with their resources for the time being.

> - they did not learn much because LLM did work for them

If companies are using LLMs as suggested earlier, they will find jobs operating LLMs. They're well poised for it, being the utmost experts in using them.

> - there is no new jobs required because we are more productive ?

More productivity means more jobs are required. But we are entering an age where productivity is bound to be on the decline. A recession was likely inevitable anyway and the political sphere is making it all but a certainty. That is going to make finding a job hard. But for what scant few jobs remain, won't they be using LLMs?

This model won't work for subjects that rely on students writing reports. But yes, universities frequently accept that failure rates for some courses will be high, especially for engineering and the sciences.

Keep your responses short and to the point. Use the Socratic method when appropriate.

When enumerating assumptions, put them in a numbered list. Make the list items very short: full sentences not needed there.

---

I was trying to clone Gemini's "thinking", which I often found more useful than its actual output! I failed, but the result is interesting, and somewhat useful.

GPT 4o came up with the prompt. I was surprised by "never use friendly language", until I realized that avoiding hurting the user's feelings would prevent the model from telling the truth. So it seems to be necessary...

It's quite unpleasant to interact with, though. Gemini solves this problem by doing the "thinking" in a hidden box, and then presenting it to the user in soft language.

What percentage of the dumbest will be boosted? What makes a person dumb? If they are productive and friendly, isn't that more important?

What percentage of the dumbest will fall farther or abandon heavy learning even earlier?

Were they wrong? People who rely too much on a calculator don't develop strong math muscles that can be used in more advanced math. Identifying patterns in numbers and seeing when certain tricks can be used to solve a problem (verses when they just make a problem worse) is a skill that ends up being beyond their ability to develop.

They really should modify it to take out that whole loop where it apologizes, claims to recognize its mistake, and then continues to make the mistake that it claimed to recognize.

Yes. People who rely too much on a calculator weren't going to be doing advanced math anyway.

My wife is surprisingly good at remembering routes, she'll use the GPS the first time, but generally remembers the route after that. She still isn't good at knowing which direction is east vs west or north/south, but neither am I.

Honestly it seems like we're doing both most of the time. It's hard to only optimize resources for boosting the dumbest without taking them away from the brightest.

Not really. While doing something to ensure that students are actually learning is important, plenty of the smartest people still don't always test well. End of semester exams also tend to not be the best way to tell if people are learning along the way and then fall off part way through for whatever reason.

It would be pretty neat if there was an LLM that guides you towards the right answer without giving it to you. Asking questions and possibly giving small hints along the way.

I think you can prompt them to do that, but that doesn't solve the issue of people not being willing to learn vs just jump to the answer, unless they made a school approved one that forced it to do that.

Same with education, for example you can financially force people to learn, say, computer science instead of liberal arts. Even when they don't like it. It's harder, less efficient, but possible.

I run it locally and read the raw thought process, find it very useful (can be ruthless at times) seeing this before it tags on the friendliness.

Then you can see it's planning process to tag on the warmth/friendliness "but the user seems proud of... so I need to acknowledge..."

I don't think Gemini's "thoughts" are the raw CoT process, they're summarized / cleaned up by a small model before returned to you (same as OpenAI models).

Then they suddenly become kinda stricter in high school, where your results decide if you can go to university and which.

But I've been to one of the top technical universities and compared to Italy it was very easy. It was obvious the goal was to try and have everyone pass. Still people managed to fail or drop out anyway, although not in the dramatic numbers I saw in Italy for math exams.

The students who want to learn, will learn. For the students who just want the paper so they can apply for jobs, we ought to give them their diploma on the first day of class, so they can stop wasting everybody's time.

It does seem similar in structure to Gemini 2.0's output format with the nested bullets though, so I have to assume they trained on synthetic examples.

There is a measurable decrease in critical thinking skills when people consistently offload the thinking about a problem to an LLM. This is where the primary difference is between solving problems with an LLM vs having it solved for you with an LLM. And, that is cause for concern.

Two studies on impact of LLMs and generative AI on critical thinking:

https://www.mdpi.com/2075-4698/15/1/6

https://slejournal.springeropen.com/articles/10.1186/s40561-...

Student being unable to unwilling to learn that knowledge or acquire those skills should mean they don't get that degree, they don't get those jobs, and they go work in fast food or a warehouse.

"Just give them the degree" is quite literally the worst possible solution to the problem.