Do AI detectors work? Students face false cheating accusations

The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. But any teacher doing this will end up with no students signing up for their class. The only solution I see is the Higher Learning Commission mandating this for all classes.

But even requiring in person proctored exams is not the full solution. Students are not used to doing the necessary work to learn. They are used to doing the necessary work to pass. And that work is increasingly cheating. It’s a clusterfuck. I have calculus students who don’t know how to work with fractions. If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated.

K-12 needs to be changed as well.

I was in high school when kids started getting cell phones with internet access and basically as soon as that happened it opened up rampant cheating even among the best of students. I can only imagine it being much worse nowadays than even 15 years ago when I was in high school.

Isn't it to either do that now, or to lose the signaling value of college degrees as indicating knowledge.

In Germany, all exams are like this. Homework assignments are either just a prerequisite for taking exam but the grade is solely from the exam, or you may get some small point bonus for assignments/projects.

> But any teacher doing this will end up with no students signing up for their class.

The main courses are mandatory in order to obtain the degree. You can't "not sign up" for linear algebra if it's in your curriculum. Fail 3 times and you're exmatriculated.

This is because universities are paid from tax money in Germany and most of Europe.

The US will continue down on the path you describe because it's in the interest of colleges to keep well-paying students around. It's a service. You buy a degree, you are a customer.

Same in the U.S. but you can sometimes find an online offering. If you don’t know what you are doing or don’t care then always take the online offering. Much easier to cheat.

My ex-girlfriend is German. She cheated on her exams to get her agricultural engineering degree at university. This was in the 80s.

I'm not sure knowledge is what a college degree signals to prospective employers. The alternative hypothesis, which AFAIK has a fair bit of support, is that it signals a willingness to do whatever it takes to fulfill a set of on paper requirements imposed by an institution, by hook or by crook.

...which is what we have today, where the most lucrative industries for people with good math skills are finance (= cheating dumb people out of their retirement), advertising (= cheating dumb people out of their consumer dollars), and data-driven propaganda (= cheating dumb people out of their votes).

/dystopia

The course might be mandatory but which professor you choose isn't. What if multiple professors teach it? Word gets around and everyone chooses the easy profs.

I like the phrasing you used.

Advertising absolutely works on you regardless of how smart or educated you are.

How it has to work to do that can change, but the idea that advertising only impacts dumb people is pernicious as shit.

There were exceptions to this rule (in both directions), of course.

I guess there is some sort of incentives that rewards institutions taking the easy way out.

There is one Linear Algebra course. You have to pass it to get your degree. Typically, it's taught by the same prof for many years, but it might also rotate between different chairs and profs (but only one in each semester and the "design" and requirements of the course stays largely the same).

You point to a true failure in incentives. And yet, the US has the highest density of renowned universities.

And at least when I was in college it was the same with respect to classes, you can't take the same class more than 3 times. Additionally if a course is required you either take it or make the case for an equivalent class.

You don't have the choice to not take the class, you just have choice with which professor you would like to take it with. And often you would have to get lucky anyway, since that session may be filled so you'd have to take it with the "harder" teacher anyway.

For example with the popularity of computer science and STEM in general, at my school there were often 2-3 teachers teaching linear algebra in any given semester. And same for popular classes like calculus or introductory physics. Students would often lookup online which teacher was considered easier, but they still had to take the class.

Long term I see education going this route, rather than preventing students from using AI tools, update course curriculum so that AI tools don't give such an advantage.

Almost every university in the US takes federal money and relies on federal loan guarantees to keep the high revenues pumping through. In exchange, the schools are subject to requirements by the government and they impose many. I think the bigger issue is the size and scope of higher ed here and if it's actually a good idea to to tell every school how to run their exams (and enforce it).

This is because 100-200 level math courses are not about teaching anything, but about filtering out students who can't do the work. Once you get past that level students have already formed bad habits and so still only do what it takes to pass. I don't know how to fix it, I don't know if it CAN be fixed.

The same for job interviews. I did a lot of technical interviews in the past as interviewer (hundreds) for Software Engineer positions (and still help companies to hire sometimes, as independent interviewer).

There is insane amount of cheating. I'd say at least 30% in normal companies are cheaters, and 50% and more in FAANG. I can prove it, in private groups, and forums people share tech assignments. And very large number of these people use some kind of assistance while interviewing.

It's interesting to see how sometimes questions that are intentionally sophisticated are getting solved in a few minutes the best way they can be solved. I see this over and over.

Especially something like Linear Algebra can easily have class sizes of 800+ people at big universities. Yes there is typically exactly one lecture hall for that and you have 30+ exercise groups. But still only one faculty

The real issue as I see it is that no one wants to face the reality that far too many incapable, incurious people are going to college. So I pretend to give real tests and pretend to give real grades and students feel good about themselves and my classes fill.

This is 100% incorrect.

In university I can recall a computer graphics course where literally everyone got 100+% on problem sets (there were bonus questions of course) and the median score on the midterm was below 50%. Leading up to the exam I remember the prof leading an exam prep session, opening the floor to questions, and getting a sincere request from one of the students to please go over the whole concept of "matrices" again.

This was a 400 level course, BTW. At one of the highest-rated universities in Canada. (I was taking it as an elective from a different program from the default, so I can't speak to the precise prerequisites to get there.)

This was over 20 years ago, BTW. I'm sure it's only gotten somehow even worse.

The renown has to do with a lot more than demonstrated ability of graduates.

Having the choice between different professors with supposedly different difficulties for what is supposed to be the same course seems absurd.

But I'll say where I went to school, and I hear its even worse now since enrollment in STEM is way up, there were often multiple thousands of students every quarter wanting to take just one class, so they split it up because we simply didn't have lecture halls with enough seats. There would often be 3-4 classes each of 500+ students all full, and still students struggling to get in due to the maximum amount per course. Usually there was around two teachers splitting the sessions, and they also have their other more advanced courses and/or research.

So its probably just practicality in terms of their time and resources. This wasn't an issue with more advanced courses where there was usually only one teacher per semester offering the class.

The midterm and final exam were in-person in bluebooks, and they were 60% of your grade. If you were just copying the problem sets, you would fail the exams and likely the class.

But I also mentioned that there are often thousands of students all trying to take one course. And the schools simply don't have the space to fit all of them in one session since I believe the rules are basically that it needs to be held in a lecture hall big enough to fit every enrolled student, and teachers don't have the time to teach 4 different sessions by themselves on top of their other duties. Maybe class sizes are just smaller elsewhere, but where I went to school it was not unheard of to have multiple thousands of students needing to take one class that was required for practically every STEM major in a given semester.

As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating.

As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted (the hard classes) and not cheating (as I did what I wanted) - I can say that there are consequences to this approach.

In my opinion, the solution is to reduce the reliance on assessments which are prone to cheating or which in the real world would be done by computer.

But that's just my personal experience. I don't know if it's different at other large universities.

If it's a job, the order will be school, school, major, everything else on the résumé, grades maybe.

If actually learning is valuable to them, independent of whether it will actually help them with prospective employers, then yes. But I don't think we can assume that all students value that.

That's not the cost of not getting an A, it's the cost of appearing to underperform compared to too many of your peers. Which is directly tied to how many of them cheat. If not enough cheaters got an A then the cost would no longer be tied to not getting an A, it would be tied to whatever metric they appeared to outperform you on.

Really focusing on stretching yourself necessarily means lower grades. Why is that penalized? TBH, in software engineering a lot of people with lower grades tutor the ones with 4.0 averages. The skillsets required to code and the skillsets required to get a good grade on a test are different.

There's a line of research that shows that high quality one-on-one instruction gets you up to 2stdev gains in learning performance.

If you can afford to increase the professor to student ratio and make them available for office hours, you probably do see increases in performance. Is it due to better motivation? Seeing an academic up close? Actually better explanations you get from an expert in the subject? Hard to say.

Very strongly depends on the school and major; there are both narrow-path degrees with lots of mandatory courses and wide-path degrees with very few specifically mandatory courses (instead having several n of m requirements) other than lower-division general education requirements.

Sure, but they don't set the rules; sure, they do much of the education, but much of the demand for them comess from bachelor’s-degree bound students, so the course selection is set by what bachelor’s degree granting institutions accept.

I know an equitable grading champion at an LAUSD school, I'll see if I can get material to share. EDIT: I just received [0][1][2][3].

[0] (5 page pdf) https://drive.google.com/file/d/1YO7SQEwisAbHHi6mfgj7XU9FcSB...

[1] (4m30s video) https://drive.google.com/file/d/10eWor4uhSxR8ZITA1w3kzqhTOX0...

[2] (audio interview) https://www.bamradionetwork.com/track/fair-grades-dropping-g...

[3] (article) https://ascd.org/el/articles/taking-the-stress-out-of-gradin...

I didn't realize that so many community colleges and state universities were basically online diploma mills.

K-12 specifically has it bad. Wake up 7am get to school for 8/9 fill your day with classes you don't have much interest in while also figuring out how to be a social human with other kids and all the stress that entails. Then we require them to go home and continue to do more schoolwork.

Of course they're gonna cheat. They're overworked and overstressed as it is.

Remember, in European countries students are admitted to study a specific subject at university, rather than being admitted to the university as a whole and expected to choose a major later on.

So there are multiple courses going on, with a lot of intersection between the topics covered. There's maths for computer scientists (heavy on the discrete maths), maths for engineers (heavy on the integrals and matrices), maths for social scientists (heavy on the statistics), and so on.

So both American and European universities split their year 1 maths courses so they can get a few thousand first-year undergraduates through the largest 300-500 seat lecture theatres. But in Europe it's a split by subject, rather than by choose-your-instructor.

Fundamentally, kids that are just trying to pass a class don't see the value in learning and it seems that the contributions towards the "pointless" school work are parts teacher attitudes, parts curriculum design, parts real-life applicability to the student's interests, parts framing.

We've been using tests and such for far too long as a proxy for competence, rather than developing the competencies in such a way that engages the kids.

I think we need to look at reframing fundamental parts of how education is structured. I don't think there needs to be drastic changes, just some small things that allow the education and curriculum to become more engaging.

They ended up changing the assignment to where you could just find an implementation of a FFT online and write about it or something.

That's not even getting into the students who copy-pasted Wikipedia straight into their papers in that same class.

The Higher Learning Commision is a farce. It’s purpose is for sinecures for its employees.

The nice thing about an all pass/fail system is you can formalize the 'new' way grades are actually done in which A means meets expectations and anything less means did not. Making pass mean A/B takes a lot stress off students and C/D is already failing for practical purposes as often you can't continue with less than a B.

-> GPA can be gamed, as laid out.

-> Take Home assessments can mostly be gamed, I want to assess how you think, now which tools you use.

-> Personality tests favor the outgoing/extroverts

-> On-location tests/leet code are a crapshoot.

What should be best practice here? Ideally something that controls for first-time interviewer jitters.

Some organizations still hire software engineers just based on resume and a nontechnical interview. This can easily be a disaster! You need to do a real assessment during the interview of how well software engineers can code.

The other thing you have to do is that you have to be willing to fire the people who are underperforming. It's just a natural consequence of the interview process being imperfect.

This system truly forced students to grind the hell out of science

I think the vast majority of people who say and think this haven’t taught in the classroom much.

That seems more of an indictment of your profession than anything to do with the students.

Admittedly, they are about teaching things. For example, teaching Laplace transforms to mechanical engineers. It certainly isn't true to say the "courses are not about teaching anything".

But if 20% of the class should decide to change majors to business? Well, there's been some filtering out of students too.

They aren't 'fair' in avoiding every false negative, but they at least tell me that the passing candidates know and can do something.

If I ask someone who claims to know Python or Java whether or not you can have a memory leak in them, and their answer is 'no' or 'Maybe, but I don't know how', I get a pretty good idea of whether or not they know anything about this topic.

If you can't do fizzbuzz, you probably aren't a good fit for a SWE position either, you should be aiming for something more director-level. Given how much people struggle with coding, I sometimes feel like I may as well ditch my regular question, and just ask them to write that.

When I was in college, this was every math class. You could cheat all you want on the 20% of your grade that came from homework, but the remaining 80% was from 3-4 in-class, proctored exams.

And every non-educational institution after that will see school, degree as a checkbox.

When I was at Google, most of these were filtered out before I got to see them, but for a while I was doing iOS development interviews and a lot of the candidates applying to Google clearly didn't know anything.

I'm not sure what your point is, but if you're trying to claim that GP is incorrect and companies don't ask for GPA, you are (unfortunately) wrong. There are plenty who do. It seems to be especially the bigger and/or more conservative companies so it's trending away, but it definitely happens.

So schools offer multiple sections of the same class to share the workload. E.g. in recent years Computer Science 101 - often the most popular class on campus.

Onsite interviews with a range of approaches seem to be the best I've found over the years. As much as it pains me, things like fizzbuzz are still useful, because people still lie about their ability to program in languages. If you claim to know C very well and can't knock that out in 5 minutes, and it takes you 45 minutes of prompting, well, you don't know C usefully.

I've seen good results with having a pre-done sort of template program that's missing functionality, and the person completes it out based on comments (for remote interviews), and you can generally tell by watching them type how familiar with the space they are. Again, perfection isn't the goal, but if someone claims to know C very well and is trying to make Javascript syntax work, well, they're full of crap about knowing C.

That said, probably the best approach I've seen for hiring junior dev sorts is a formal summer internship program - and some places have a pretty solid system for doing this, with 20-30 people coming in every summer for a few months. That's a far better way to get to know someone's actual technical skills. In the programs I interacted with, it's safe to assume that if you have 30 people, you'll have about 15 that are "Thank you for your time, good luck..." sorts, maybe 5 or 8 that are "Yeah, you'd probably be a good fit here, and can be trained up in what we need, you'd be welcome back next summer!" and if you're lucky, one or two "HIRE NOW!" sorts that leave the summer program with a job offer.

It's obviously a lot higher effort than interviewing, but the "Throw things at people for three months and see what they do, with a defined end of the program" process seems to be a really good filter for finding quality people.

Then I started interviewing, and... yeah. I get it now. It really is that simple, should take a competent coder a few minutes, and 80% of people interviewing will take 45 minutes to muddle their way through it.

I completely agree, but the entire higher ed system is moving to on-line instruction.

Basically, if the University of <xyz> follows your suggestion, all of the competing institutions will eat their lunch by offering on-line courses with the "convenience" of on-line assessments" and the University of <xyz> will lose enrollment.

:-(

If the school has a tuition, then there's even more of a conflict of interest. I've had parents/admins imply that we might be losing a student due to poor grades.

For me a proper exam is when you get a topic, spend 30 minutes in a classroom preparing, and then sit down with an examiner to tell him about this topic and answer all the follow-up questions.

We don't do multi-option tests at software interviews, and for a good reason. Why do them in a uni?

Students largely just want to pass. They mostly don't care that they don't know anything.

When I tell people that I never cheated, ever, in any class, through my entire degree, I get mostly surprise. You never? Not once?

But I paid for it, I think. Because it was not easy finding a first position out of school -- I certainly got filtered by GPA. It actually enrages me. What is the point of a degree? What exactly is the point of this thing, if most of the signal is false? Why did I work so hard?

Not even to mention -- many of my classmates (about 1 in 5, one in 6 or so?) were granted "accommodations" which granted them twice as much time to take their exams. There are online services: pay $80, get a letter certifying your ADHD, that you can give the school to get these accommodations. It's completely ridiculous.

I'd like to point out this has nothing to do with cheating. Cheating happens at all levels of academic performance.

I have not been in university for a while, but I do remember that it was rare that I did my best work for any individual class.

For me it was more of a "satisficing" challenge, and I had to make hard choices about which classes I would not get A's in.

I'm sure some professors might have interpreted my performance in their class as indicative of my overall abilities. I'm fine with that. I learned as much as I could, I maxed out my course load, and I don't regret it at all.

But if you went to a top university with brutal courses, and got a 2.7 GPA, then all I'm seeing is you're not elite material. The number otherwise does not help me one bit in evaluating you.

BTW, having spent a lot of time out of the US - it's still pretty laid back in the US. A person who is 2.7 GPA material in the US would simply not get admission in any decent university in some countries. And plenty of people in the US start all over at another institution and do well - something many countries don't allow (state funded, lack of resources, you have to move out of the way to let the younger batch in).[1]

[1] A good friend of mine totally flunked out of his university. He spent time off in the military. Then started all over at a new university. Got really high grades. Went to a top school for his PhD and is now a tenured faculty member.

I recently had an interview and a "skill test" in C. It was proctored by the interviewer in-person. I had so many questions about the questions. It was like, what is the output of "some code" and while obvious, there were some questions where specific CPU architecture mattered:

    #include <stdio.h>

    int main() {
        unsigned int x = 0x01020304;
        unsigned char *c = (unsigned char*)&x;

        printf("First byte of x: 0x%02x\n", *c);
        return 0;
    }

Sure, whatever. Probably dodged a bullet.

> I want to assess how you think, not which tools you use

suggests you have a more nuanced approach and aren’t just aiming for large numbers of drones.

What worked well for me (in a couple of smaller companies/teams) was:

- Talk to the candidates about their experiences in a project-oriented course where they had to work in a team. (Most CS programs have at least one of these. Get the name of that course ahead of time and just ask about it.) You want to find out if they can work in a team, divide up work and achieve interim goals, finish a project, deal with conflicts, handle setbacks and learn from mistakes, etc.

- Similarly, find out the names of some of the harder elective courses, and ask about their experiences in these. This gets at what they find interesting, how they think, and can help filter out GPA gamers.

- Talk to them about their experiences in whatever jobs, internships, volunteer work, or extracurricular activities they engaged in while at school. It doesn’t have to be directly related to your field—-you’re screening for work ethic and initiative.

Admittedly it’s been a while, but we used this approach for both on-campus recruiting and remote phone screens, and got pretty good at hitting these topics in a 15-20 minute conversation. We’d have one or two people screen maybe 30-50 candidates each recruiting season, identify 5-10 for on-site interviews with a larger team, and end up hiring about half of those.

This sort of bespoke screening does take some work on your part, and can be tough to scale. But we found it consistently identified solid candidates and led to outstanding hires.

Remember, outside CS no one else does whiteboard interviews or takehome tests. It's generally a few conversations and that's it. It's because experts been sniff out other experts in their domain fairly quickly. It's about *how* they think, not what they know.

I'll give you an example of something subtle but is a frequent annoyance for me and I'm sure many others. You're on a webpage that asks for your country. Easy, just put in a drop-down, right? But what's much much better it's to use the localization information of the browser to place a copy of that country at the top of the list (a copy, not move). Sure, it saves us just scrolling to the bottom, but my partner is Korean and she never knows if she's looking for K(orea), S(outh Korea), or R(epublic of Korea). This happens for a surprising number of countries. Each individual use might just save a second or two of time, but remember you also need to multiply that by the number of times people interact with that page, so it can be millions of seconds. It'll also just leave users far less frustrated.

I'm also very sympathetic to the jitters stuff, because I get it a lot and make dumb mistakes when in the spotlight lol. But you can often find these things in other work they've done if they include their GitHub. Even something small like a dotfiles repo. And if the interview is more about validation their experience, the attention to detail and deeper knowledge will still show up in discussions especially if you get them to talk about something they're passionate about.

I'd also say that GPA and school names are very noisy (incidentally that often means internships too, since these strongly correlate). I know plenty of people from top 3 schools who do not know very basic things but have done rounds at top companies and can do leet code. But they're like GPT or people who complain about math word problems, they won't generalize and recognize these things in the wild. Overfit and studies to the test (this is a subtle thing you can use while interviewing too)

But in the (hypothetical) limit where AI tools outperform all humans, what does this updated test look like? Are we even testing the humans at that point?

If, hypothetically, there's two candidates, one who is more knowledgeable but has no personal projects versus someone who has less knowledge but has worked on different side projects in various languages/domains, I'm always going to pick the latter candidate since they clearly have a passion, and that passion will drive them to pick up the knowledge more than someone who's just doing it for a paycheck and could care less about expanding their own knowledge.

To go one step forward, you can ask them to go into detail about their side project, interesting problems they faced, how they overcame them, etc. Even introverts who are generally worse at small talk are on a much more balanced playing field when talking about something they're passionate about.

Maybe worse was seeing the undergrads who passed on research opportunities out of fear it would distract them from keeping a high GPA.

I think you're right here but, to play devil's advocate... isn't there some survivorship bias going on here? I assume you've never tested the negative hypothesis and gone ahead and hired somebody who couldn't program fizzbuzz to validate your assumption.

I can, too. I wanted to learn, but I also wanted to achieve a high GPA. I had a privileged background, so I got to retake classes after earning Cs or Bs until I got an A, without cheating.

The consequences: My degree took a long time to get, cost more money than my peers in the same program, and I now have a deep-seated feeling of inadequacy.

2. screen for passion and general technical competency above all else. You're going to make arbitrary decisions & restrictions (ex: we're only hiring from these 3 schools) which is fine, then work within those constraints. Ask about favorite classes (and why), what they've done lately or are excited about, side projects, OS contributions, building/reading/playing. The best intern I've hired lately answered some high-level questions about performance by building a simple PoC to demo some of their ideas, with React - a technology they didn't know but that we use.

3. recognize some things on the hiring side that from the hunting side don't make sense or are really annoying: you're playing a numbers game, hiring is a funnel, it's better to miss a great hire than go with a poor candidate (i.e. very risk averse), most hiring companies are at the mercy of the market; they hire poorer candidates and pay more, then get very picky and pay less. In a tight market you can't do much internally to stand out, and when lots of people are looking you don't have to.

I agree with the first part, but I think the second follows from it.

Take a class like organic chemistry. When I was in school, the grade was based on 5 exams, each worth 20% of your grade. Worse still, anything less than an A was seen as a failure for most students dreaming of medical/vet school.

Of course you are going to have people that are going to cheat. You've made the stakes so high that the consequences of getting caught cheating are meaningless.

On top of that, once enough students are cheating, you need to cheat just to keep up.

  unsigned int x = 0x01020304;
  unsigned char *c = (unsigned char*)&x;

However, on a big endian system, that same value would be stored in memory as [01][02][03][04] - so, reading a byte at 0x100 would return 0x1.

Older ARM systems were big endian, and there are others that run that way, though it's rarer than it used to be. One of the perks of little endian is that if you want to read a smaller version of a value, you can read from the same address. To read that value as an 8, 16, or 32 bit value, I read at the same address. On a big endian system, I'd have to do more address math to do the same thing. It mostly doesn't matter, but it is nice to be able to have a "read of 8 bits at the address of the variable" do the sane thing and return the low order 8 bits, not the high order bits.

A former employer of mine inadvertently did! He wasn't asked to complete FizzBuzz, but I am confident he couldn't answer it as I worked on the same team as him. He was a very charismatic individual who always "needed help" from team mates on all tasks, no matter how small. He managed to collect a salary for 6 months. Some time after he was let go, the police called my employer enquiring after him, and we learned he was a conman with outstanding arrest warrants with no prior SWE experience at all. The name we all knew him by was just one of many aliases.

At least for my CS degree, this surprisingly wasn't the case. I remember our freshman class advisor gave a speech that said that grades don't really matter so long as if you pass, but we all laughed and dismissed him. I ended up getting a big tech internship with a ~2.8 GPA and an even better full time job with a ~3.2.

Obviously, your mileage may vary. I graduated in a hot tech market from a prestigious university with a reputation of being difficult. Even so, overall, almost all of my classmates were stressed over grades significantly more than they needed to be.

To do what you're asking, you'd do something like this:

  unsigned char c = (unsigned char)x;

I reiterate:

But even requiring in person proctored exams is not the full solution. Students are not used to doing the necessary work to learn. They are used to doing the necessary work to pass. And that work is increasingly cheating. It’s a clusterfuck. I have calculus students who don’t know how to work with fractions. If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated.

K-12 needs to be changed as well.

Going from high school to college grades are looked at a bit more, but that's because that, the essay, and the SAT are all they have.

I am troubled by this argument because it suggests people have no ethical core. If that is true then we are going to have problems with them regardless.

Unless, I suppose, sizeof was negative :).

1. Live coding, in Zoom or in person. Don't play gotcha on the language choice (unless there's a massive gulf in skill transference, like a webdev interviewing for an embedded C position). Pretend the 13 languages on the candidate's resume don't exist. Tell them it can be any of these x languages, which are every language you the interviewer feel comfortable to write leetcode in.

2. Write some easy problem in that language. I always go with some inefficient layout for the input data, then ask for something that's only one or two for loops away from being a stupid simple brute force solution. Good hygienic layout of the input data would have made this a single hashtable lookup.

3. Run the 45 minute interview with a lot of patience and positive feedback. One of the best hires in our department had first-time interview nerves and couldn't do anything for the first 10 minutes. I just complimented their thinking-out-loud, laughed at their jokes, and kept them from overthinking it.

4. 80% of interviewees will fail to write a meaningful loop. For the other 20%, spend the rest of the time talking about possible tradeoffs, anecdotes they share about similar design decisions, etc. The candidate will think you're writing in your laptop their scoring criteria, but you already passed them and generated a pop-sci personality test result for them of questionable accuracy. You're fishing for specific things to support your assessment, like they're good at both making and reviewing snap decisions and in doing so successfully saved a good portion of interview time, which contributed to their success. If it uses a weasel word, it's worth writing down.

5. Spend an hour (yes, longer than the interview) (and yes, block this time off in your calender) writing your interview assessment. Start with a 90s-television-tier assessment. For example, the candidate is nimble, constantly creating compelling technical alternatives, but is not focused on one, and they often communicate in jargon. DO NOT WRITE THIS DOWN. This is the lesson you want the geriatric senior management to take away from reading your assessment. Compose relatively long (I do 4 paragraphs minimum) prose that describes a slightly less stereotyped version of the above with plenty of examples, which you spent most of the interview time specifically fishing for. If the narrative is contradicted by the evidence, it's okay to re-write the narrative so the evidence fits.

6. When you're done, skim the job description you're hiring for. If there's a mismatch between that and the narrative you wrote, change your decision to no hire and explain why.

Doing this has gotten me eye rolls from coworkers but compliments at director+ level. I have had the CTO quote me once in a meeting. Putting that in my performance review packet made the whole thing worth it.

The problem is the lack of learning the material. You don't, IMO, directly care how they produced the answer, you care about it only as a proxy for them learning the material well enough to solve the problem.

And making people do them in person with no technology is unrealistic - not because it can't be done, but because at that point, it's not a reflection of how you'd use it outside of that classroom, and people are going to call it pointless, and IMO they'd be right. You would be correct that anyone who met that bar would have likely learned the material, but you'd also have excluded people who would have met the bar of "can use the material to the degree of familiarity needed going forward".

I think a reasonable compromise would be to let students collaborate on the exams in the classroom, without external access - while I realize some people learn better on their own in some subjects, as long as everyone contributes some portion of the work, and they go back and forth on agreeing what the right answers are, then you're going to make forward progress, even if that ruins the exam process as anything other than a class-wide metric. You could subdivide it, but then that gets riskier as there's a higher chance that the subset of people doesn't know enough to make progress. Maybe a hint system for groups, since the goal here is learning, not just grading their knowledge going in?

Not that there's not some need for metrics, but in terms of trying to check in on where students are every so often, I think you need to leverage how people often end up learning things "in the wild" - from a combination of wild searching and talking to other people, and then feedback on whether they decided you could build an airplane out of applesauce or something closer to accurate.

I've found that talking through projects is a weak indicator of competence. It's much easier to memorize talking points than to produce working code.

So it then becomes hard for me to make suggestions to juniors. It isn't difficult to sniff out those like you or me who are motivated by the rabbit holes themselves, nor difficult to tell those who are entirely driven by social pressures (money, prestige, family, etc), but what about those on the edge? I think it's the morally best option to encourage learning for learning but it's naive to also not recognize that their peers who will cheat will be rewarded for that effort. It's clear that we do not optimize for the right things and we've fallen victim to Goodhart's Law, but I just hope we can recognize it because those systems are self reinforcing and the longer we work in them the harder they are to escape. Especially because there are many bright students who's major flaw is simply a lack of opportunity. For me? I'm just happy if I can be left to do my research, read papers and books, and have sufficient resources -- which is much more modest than many of my peers (ML). But it'd be naive to not recognize the costs and I'm a big believer in recognizing incentive structures and systematic issues. Unfortunately these are hard to resolve because they're caused by small choices by all of us collectively, but fortunately that too means they can be resolved by small choices each of us make.

As an interviewer, I spot this and try to get them to ease up. I will talk about myself for a bit, about the work I do. I'm trying to get them to realize they are not in the spotlight, but whether we would be a good fit together; and thus both of us want them to work there.

BUT, my interviews tend to be about us solving a problem together, very rarely about actual code. For example, we might walk through how we would implement an email inbox system. We may discuss some of the finer details, if they come up, but generally, I'm interested in how they might design something they've basically used every day. How would we do search, what the database schema would look like, drafts, and so on.

I won't nudge them (to keep my biases in check), but I will help them down the path they choose, even if I don't like it. I'm not testing for the chosen path, but what "gotchas" they know and how they think though them. If you are a programmer, it shows. If you are an excellent programmer, it shows. If you are not a programmer, you won't make it 10 minutes.

If you don't want people to prioritize grades over everything else...

Some of their stuff works really well, and they have prof customers who love it. The CEO went on a tour to visit their biggest customers in person and several of them said they couldn't imagine going back.

Unfortunately as a whole the industry is not interested in it, aside from a few small niches and department heads who are both open minded and actually care about the integrity of the education. There have even been cases where profs want it and the dean or admin in charge of academic integrity vetoes its adoption. I've been privy to some calls I can only characterize as corrupt.

There is something deeply broken about higher Ed, the economics, the culture of the students, the culture of the faculty, the leadership... This isn't an AI problem it's a society problem.

When the students genuinely want to learn something and they are there for the knowledge, not the credit, cheating isn't a problem.

  > Really focusing on stretching yourself necessarily means lower grades.

It's overly dramatic, but I think gets the point across in an easy to understand way. It's exactly why you see the lower grade ones tutor the high grade ones (this even happened in my undergrad and I did physics[0]).

It's because learning happens when struggling. It happens at the edge. This is also a big reason some learn a lot faster than others or even why someone will say they don't understand but understand more than someone who says they do (and who believes it). Because expertise isn't about the high level general ideas, it's about all the little nitty gritty details, the subtle things that dramatically change things. But a big concern I have is that this is a skill to learn in of itself. I think it's not difficult to recognize when this skill is learned (at least if you have) but it's not something that'll be learned if we focus to much on scores. After all, they're just a proxy. Even the institutional prestige is a proxy (and I have an argument why it no longer matters though it did decades ago).

I do wonder if this is in part cause for the rise in enshitification. Similarly if this is why so many are bad at recognizing issues in LLMs and ML models. I'm sure it is but not sure how much this contributes or if it's purely a confounding variable.

[0] when I signed up to be a tutor at my university I got signed off my the toughest math professor. When I took the signature to the department the admin wasn't sure if I was trying to trick her because she immediately called the professor to confirm the signature. Then told me I could tutor whatever I wanted because I was one of two people he had ever signed off on. Admittedly, I'm sure a lot of that was because people were afraid of him (he wasn't mean, but he wouldn't let you be anything less than the best he thought you could be)

Can you blame them? If they do the necessary work to learn, but do poorly on an exam for some reason, will you still give them a passing grade?

This is true in the US as well. You can change your major, but you are admitted into a College in the University. Moving to another College is not guaranteed if you later change your mind.

If all my math professors had done this, I never would have earned my computer science degree or my minor in mathematics.

I have an immensely difficult time memorizing formulas and doing math by hand. I absolutely need to be able to prepare notes ahead of time, and reference them, to be able to complete a math test on paper. Even then, I'm a very slow in-person test-taker, and would often run out of time. I've honestly come around to the idea that maybe I have some sort of learning disability, but I never gave that idea much thought in college. So, I didn't qualify for extra time, or any other test-taking accommodations. I was just out-of-luck when time was up on a test.

The only reason I was able to earn my degree is because I was able to take almost all of my math classes online, and reference my notes during tests. (COVID was actually a huge help for this.)

And by "notes", I don't just mean formulas or solutions to example problems that I had recorded. I also mean any of the dozens of algorithms I programmed to help automate complex parts of larger problems.

The vast majority of the math classes I took, regardless of whether they were online or in-person, did not use multiple-choice answers, and we always had to show our work for credit. So I couldn't just "automate all the things!", or use AI. I did actually have to learn it and demonstrate how to solve the problems. My issue was that I struggled to learn the material the way the university demanded, or in their timeframe.

So as an otherwise successful student and capable programmer, who would have struggled immensely and been negatively affected mentally, professionally, and financially, had they been forced to work through math courses the way you prescribe, I'm asking you: please reconsider.

Please reconsider how important memorization should be to pass a math class, how strongly you might equate "memorized" to "learned", and what critical thinking and problem-solving could look like in a world where technology is encouraged as part of learning, not shunned.

Maybe here's a general question you can add (if you don't already use it) to bring out that thinking even if they're nervous. Since it's systems they are familiar with (my forum entry example is similar. I don't do front end), ask them what things they're frustrated with in tools they've used and how they could be fixed. It can help to ask if they've tried different solutions. With email that can be like if they just use Gmail via the Web, just use outlook or Apple Mail, or have tried things like Thunderbird, mux, or other aggregators. Why do they like the one they use? And if they've tried others I think that in itself is a signal that they will look for improvements on their own.

The things I think many interviews do poorly at is that they tend to look for knowledge. I get this, it's the easiest thing to measure because it's tangible. It's something you "have". While this matters, the job is often more dependent on intelligence and wisdom which are more about inference, attention, flexibility, and extrapolation. So I don't think it's so much about "gotchas" -- especially as many now just measure how "prepared" they are -- but, like you said, the way they think.

I'd much rather take someone with less knowledge (within reason) who is more intelligent, curious, and/or self driven by the work (not external things like money or prestige). Especially with juniors. A junior is an investment and thus more about their potential. As they say, you cannot teach someone who "already knows".

[EDIT]:

There's something else I should bring up about the "classic engineering" interview. Often they will discuss a problem they are actively working on. A reason for this is 1) it is fresh in their mind, 2) it gets at details, *but* 3) because it makes it easier for the interviewee to say "I don't know."

I think this is often an issue and sometimes why people will say weird erroneous things. They feel pressured to not admit they don't know and under those conditions, a guess is probably a better strategy. Since admitting lack of knowledge is an automatic "failure" while a guess has some chance, even if very small. At least some will admit to guessing before they do and you can also say its fine to guess and I see that often relax people and frequently results in them not guessing and instead reason through it (usually out loud).

(I'm an older grad student finishing up, so I frequently am dealing with undergrads where I'm teaching a class, holding office hours, or mentoring them in the lab. I've done interviews when I was a full time employee before grad school, and I notice there's a lot of similarities in these situations. That people are afraid to admit lack of knowledge when there is an "expert" in front of them. Even if they are explicitly there to get knowledge from said expert.)

So roughly 6x the amount of students.

Just one generation ago this was the norm. The only differences between how exams were given in my math classes were what size of note paper was allowed.

In general students hated the few classes that tried to use online platforms for grading, the sites sucked so much that students preferred pen and paper.

Also, it is a math class! The only thing that is needed is arguably a calculator, a pencil, and some paper. What the hell kind of technology are students using in class?

> The only solution I see is the Higher Learning Commission mandating this for all classes.

Colleges used to all have tech requirements, the big debate was to allow calculators with CAS or not.

> If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated.

What the heck are students doing in college then? I was paying good $$$ to go to college, I was there because I wanted to learn. Why the hell would I pay thousands of dollars to go to class and then not learn anything in the class, that would be a huge waste of my time!

Profs who were hard but fair never had a problem filling up their classrooms with students who self selected for wanting to learn.

The hard but not fair ones were just assholes IMHO.

The easy ones also had their classes filled up.

My community college had two history profs, one had all essay questions, one had multiple choice. The essay question prof was considered "hard", but so long as your essay justified your position and was well reasoned, you got full credit for the answer.

I hated the multiple choice prof. He gave the entire class his test bank every quarter and you just had to memory hundreds of questions and he'd pick 50 for the test. IMHO it took more time studying because I had to read the book and then memorize a bunch of pointless answers, vs reading the book and understanding what was going on, which I can typically do in the first pass.

I don't completely agree that multiple-choice questions can't test real knowledge. It is possible to write multiple-choice questions that require deep thinking and problem solving to select the correct answer (modulo a 25% chance of getting it right with a guess.)

It's true that MC questions can't evaluate the problem-solving process. You can't see how the student thought or worked through the problem unless you have them write things out. But again, that's a tradeoff with the time it takes to evaluate the students' responses.

I don’t care about the answer. I care about the thought process that went into finding the answer. The answer is irrelevant.

And making people do them in person with no technology is unrealistic - not because it can't be done, but because at that point, it's not a reflection of how you'd use it outside of that classroom, and people are going to call it pointless, and IMO they'd be right.

There’s body of knowledge a person trained in a given area ought to know without use of computers or notes. There are things a person who calls themself “an engineer” or “a physicist” ought to know off the top of their head. A person going into mechanical engineering ought to have some familiarity with how to integrate without using a computer. Such is my belief.

It is 100% societal. It is because society is focused on "get degree, get job, get money". It is because Western societies have gotten so damn competitive that if you don't succeed at any of the above, there is a non-trivial chance you won't even be able to afford a house to live in.

In America, I'll admit that No child left behind made it a lot worse, with tests left and right, which gives students the wrong impression of what learning is about.

Every class should be about critical thinking. Every single class. Multiple choice tests are a societal cancer and should be limited to a tiny fraction of tests given.

The point of school is to learn how to learn. That is it. What facts are taught are almost irrelevant. The point is to learn HOW to learn. Be that researching the history of fabric dyes in Ancient Egypt or making a scale drawing of one's house.

The "WHAT" IS NOT IMPORTANT.

The HOW is important.

How to write an essay, the topic doesn't matter.

How to learn about the culture of a country.

How to learn a new field of mathematics.

How to learn a new type of art.

How to give a presentation.

How to learn a hard science.

Yes the basics of physics and chemistry and such need to be taught. But the things that are learned should be inline with teaching the all important skill of how to learn.

Students want to be engaged in their coursework, but the universities aren't there to encourage or support it.

Then we need to teach them. You are doing the right thing for being a "hard" teacher, and it doesn't prevent you from also being known as a caring one.

From experience, acknowledging the students' difficulties with it and emphasising that it is because they were not taught how to think (as opposed to some innate inability to do maths) can go a long way.

Essentially when a scarcity increases, there will always be an imperfect heuristic of selection.

I guess this is more of a public policy area but it seems reasonable that anyone working full time should have access to economic security. Essentially cheating on university is the first symptom of lifetime of vastly unequal access to economic security.

Anytime students are allowed technology there is massive amounts of cheating. Knowing a certain body of knowledge off the top of your head is important in all areas of study.

I had a decent GPA and took reasonably hard classes. I had a required discrete math class that was awful. The professor would assign homework for the next chapter that we hadn't gone over yet and them grade it as if it were a test. WTF am I paying you to teach me if I have to learn it myself before you ever present it and test me on that? Assign reading beforehand - great. Assign upgraded, or completion-graded homework beforehand - great. Grad it like a test before teaching it - BS. I took it with another professor after dropping the first one and they had more normal practices and it went much better.

Can you blame students for optimizing for grades rather than "learning"? My first two years of undergrad, the smallest professor-led lecture course I took had at least 200 students (the largest was an econ 101 course that literally had 700 kids in it). We had smaller discussion sections as well, but those were led by TAs who were often only a couple years older than me. It was abundantly clear that my professors couldn't care less about me, let alone whether I "learned" anything from them. The classes were merely a box they were obligated to check. Is it so hard to understand why students would act accordingly?

wat

I interview a lot of people and I rarely see anything I'd describe as cheating. Maybe my company is not famous enough to be worth cheating at.

Since this compiles to FF 00 00 00 in big-endian and 00 00 00 FF in little-endian, it would work on both platforms.

If you’re reading a file in binary format from disk, though, you always have to know whether the byte you are reading is little-endian or big-endian on disk.

Perhaps another way to widen the scope of what is not cheatable (at the cost of more teacher work, ugh), is to require showing all work?

And I mean every draft, edit, etc.. All paper scratch-notes. Or on work on computer applications, a replayable video/screenshot series of all typing and edits, like a time-lapse of a construction site. Might even add opportunities to redirect work and thinking habits.

Of course, that too will eventually (probably way too soon) be AI-fakeable, so back to paper writing, typewriters, red pencils, and whiteout.

Just an idea; useful?

You need to use a bit mask in order to make this code endian-independent rather than a pointer alias. Like (uint8_t)(int & 0xFF), or something like that.

If someone writes "perfect" code from a take-home, you can ask them to explain what they did (and if they used GPT, explain how they checked it). Then ask them to extend or discuss what the issues are and how they'd fix it.

I think asking some probing questions about past projects is normally enough to discern bullshit. You do need to be good at interviewing though. If you really want an excellent candidate then there's the FANG approach of (perhaps unfairly) filtering people who don't perform well in timed interviews, provided your rubric is good and you have enough candidates to compare to. There is a trade off there.

Grad positions optimise for what you can test - people are unlikely to have lots of side projects or work experience so you end up seeing how well they learned Algorithms 101. For someone who's worked for 10 years asking about system design in the context of their work is more useful.

Note that PhD and academic positions very rarely ask for this sort of stuff. Even if you don't have publications. They might run through a sample problem or theory (if it's even relevant), but I've never had to code to get a postdoc.

Otherwise you put people on short probation periods and be prepared to let them go.

Realize there are practical limits to knowledge here. In the case of a new graduate, they are likely to have little or no job experience, so no one actually knows how they function in a workplace. Even if they were a personal family friend who you knew quite well, there would be considerable uncertainty.

I don't think GPA, take-home assignments plus an interview about them, personality tests, or on-location tests like leetcode or architecture interviews are measuring the same thing. Are you just looking for any means to winnow down the pool of applicants, or is there an underlying ability you're searching for?

But we are really great about rationalizing away ethical issues. I suspect is a good grade is worth more than a personal sense of ethics.

As much as a med school wants ethical students, they want students with 4.0s more.

There’s also the ‘having an opinion on things’ factor. Someone that thinks things should be done a certain way, and can motivate that, will always be higher on my ranking, regardless of what that opinion is.

An argument perhaps that there should not be hundreds in a room.

Rote memorization and examinations as being the basis of modern education is the problem here, and frankly I'm glad that many academics are struggling because it should show how terrible most educational programs truly are at actually teaching students and developing knowledge.

Sorry, I'm tired to hear about the crocodile tears from instructors who refuse to adapt how they teach to the needs of students and instead lashing out and taking the easy road out by blaming students for being lazy or cheaters or whatever.

When you can read about a classroom in the 1800s and in 2024 and you realize the model is exactly the same, then this should tell you that your entire model is broken. All of it. The rote lectures, the memorization, the prompting students to demonstrate knowledge through grading. All of it is useless and has been a cargo cult for a long time because (and this is especially bad in higher education) there's no interest or effort in changing the way business is done.

Yeah sorry, no sympathy from me here.