Do AI detectors work? Students face false cheating accusations

My personal take, we’ve made the cost of failure to high and cheating too easy.

As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating.

As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted (the hard classes) and not cheating (as I did what I wanted) - I can say that there are consequences to this approach.

In my opinion, the solution is to reduce the reliance on assessments which are prone to cheating or which in the real world would be done by computer.

I think grading is obsolete. Grade inflation increased a lot the past 30 years. Ironically, it has increased the least at the least prestigious colleges. Pass/fail is the way to go. Don’t know if this would mess up things like applying for graduate school or jobs but let’s end the farce that grading has become.

Disagree on the order, unless the next institution is also an educational one, which for undergraduates is mostly not the case.

If it's a job, the order will be school, school, major, everything else on the résumé, grades maybe.

> As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating.

That's not the cost of not getting an A, it's the cost of appearing to underperform compared to too many of your peers. Which is directly tied to how many of them cheat. If not enough cheaters got an A then the cost would no longer be tied to not getting an A, it would be tied to whatever metric they appeared to outperform you on.

This really can't be emphasized enough. Universities and the initial hiring process really optimize for a score and not for learning. Those could be, and sometimes are, correlated, but it isn't necessarily the case.

Really focusing on stretching yourself necessarily means lower grades. Why is that penalized? TBH, in software engineering a lot of people with lower grades tutor the ones with 4.0 averages. The skillsets required to code and the skillsets required to get a good grade on a test are different.

Agreed, I didn't know people put their GPA on resumes.

Getting rid of grading sounds crazy, but it's actually happening. Los Angeles Unified, the second largest school district in America, is moving to "equitable grading", which amounts (imo) to pass/fail with extra pageantry. Teachers are being retrained _right now_ to equitable grading.

I know an equitable grading champion at an LAUSD school, I'll see if I can get material to share. EDIT: I just received [0][1][2][3].

[0] (5 page pdf) https://drive.google.com/file/d/1YO7SQEwisAbHHi6mfgj7XU9FcSB...

[1] (4m30s video) https://drive.google.com/file/d/10eWor4uhSxR8ZITA1w3kzqhTOX0...

[2] (audio interview) https://www.bamradionetwork.com/track/fair-grades-dropping-g...

[3] (article) https://ascd.org/el/articles/taking-the-stress-out-of-gradin...

In a pass/fail system, what does a student need to do for a teacher to be willing to fail them? What is the minimum bar to pass?

except for a plethora of companies that require GPA disclosure on their submissions.

Sometimes it is even a form field on Linkedin job openings.

At most universities you can talk most classes pass/fail by choice which means A-D is pass and F is fail.

The nice thing about an all pass/fail system is you can formalize the 'new' way grades are actually done in which A means meets expectations and anything less means did not. Making pass mean A/B takes a lot stress off students and C/D is already failing for practical purposes as often you can't continue with less than a B.

Serious question from someone who is regularly tasked with hiring Juniors. What IS a good assessment for entry-level/right out of college positions?

-> GPA can be gamed, as laid out.

-> Take Home assessments can mostly be gamed, I want to assess how you think, now which tools you use.

-> Personality tests favor the outgoing/extroverts

-> On-location tests/leet code are a crapshoot.

What should be best practice here? Ideally something that controls for first-time interviewer jitters.

Employers need to wake up to this in hiring, too. You can get a 4.0 with a degree in computer science from a top school, and still not be able to program at all.

Some organizations still hire software engineers just based on resume and a nontechnical interview. This can easily be a disaster! You need to do a real assessment during the interview of how well software engineers can code.

You have to use on-location tests. Do your best to be fair and get a true evaluation of the candidate's skills. It's not perfect but the alternatives are worse.

The other thing you have to do is that you have to be willing to fire the people who are underperforming. It's just a natural consequence of the interview process being imperfect.

> On-location tests/leet code are a crapshoot

They aren't 'fair' in avoiding every false negative, but they at least tell me that the passing candidates know and can do something.

If I ask someone who claims to know Python or Java whether or not you can have a memory leak in them, and their answer is 'no' or 'Maybe, but I don't know how', I get a pretty good idea of whether or not they know anything about this topic.

If you can't do fizzbuzz, you probably aren't a good fit for a SWE position either, you should be aiming for something more director-level. Given how much people struggle with coding, I sometimes feel like I may as well ditch my regular question, and just ask them to write that.

> As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order.

And every non-educational institution after that will see school, degree as a checkbox.

Since this was at LA Unified, I suspect the bar for passing is extremely low. Not commenting on that district specifically, but not graduating from High School on time takes some doing. The system is very good at moving kids through, and it's why a high school diploma means so little.

I have hired for many positions over the years and never once asked for grades.

Back when I was at a small company doing a lot of new-grad interviews, it was really shocking how many people couldn't solve fizzbuzz or something equally trivial, like reversing an array in-place.

When I was at Google, most of these were filtered out before I got to see them, but for a while I was doing iOS development interviews and a lot of the candidates applying to Google clearly didn't know anything.

According to the equitable grading materials I just received (and posted above), that determination is... entirely up to the individual teacher's discretion? I might be misunderstanding.

> I have hired for many positions over the years and never once asked for grades.

I'm not sure what your point is, but if you're trying to claim that GP is incorrect and companies don't ask for GPA, you are (unfortunately) wrong. There are plenty who do. It seems to be especially the bigger and/or more conservative companies so it's trending away, but it definitely happens.

It's hard, and interviewing is better suited to answering "nope, not you!" questions than "yes, you'll be a good fit."

Onsite interviews with a range of approaches seem to be the best I've found over the years. As much as it pains me, things like fizzbuzz are still useful, because people still lie about their ability to program in languages. If you claim to know C very well and can't knock that out in 5 minutes, and it takes you 45 minutes of prompting, well, you don't know C usefully.

I've seen good results with having a pre-done sort of template program that's missing functionality, and the person completes it out based on comments (for remote interviews), and you can generally tell by watching them type how familiar with the space they are. Again, perfection isn't the goal, but if someone claims to know C very well and is trying to make Javascript syntax work, well, they're full of crap about knowing C.

That said, probably the best approach I've seen for hiring junior dev sorts is a formal summer internship program - and some places have a pretty solid system for doing this, with 20-30 people coming in every summer for a few months. That's a far better way to get to know someone's actual technical skills. In the programs I interacted with, it's safe to assume that if you have 30 people, you'll have about 15 that are "Thank you for your time, good luck..." sorts, maybe 5 or 8 that are "Yeah, you'd probably be a good fit here, and can be trained up in what we need, you'd be welcome back next summer!" and if you're lucky, one or two "HIRE NOW!" sorts that leave the summer program with a job offer.

It's obviously a lot higher effort than interviewing, but the "Throw things at people for three months and see what they do, with a defined end of the program" process seems to be a really good filter for finding quality people.

The first time I saw FizzBuzz, I immediately assumed it was some sort of "trap" or "trick" interview question - that there's some deviously subtle little thing in it that you'll miss at first or second glance, as a "gotcha." It literally never occurred to me that it was, in fact, a basic "Can you code your way out of a paper bag given a map?" sort of question to check for basic code competence in languages.

Then I started interviewing, and... yeah. I get it now. It really is that simple, should take a competent coder a few minutes, and 80% of people interviewing will take 45 minutes to muddle their way through it.

> My personal take, we’ve made the cost of failure to high and cheating too easy. This is so true. I was recently pondering about the impact of AI cheating in Africa and came up with the conclusion that it won't be as significant as in EU/US precisely because most evaluations in African countries are in person https://www.lycee.ai/blog/can-africa-leapfrog-its-way-to-ai-... Your take reminds me of Goodhart's law: "When a measure becomes a target, it ceases to be a good measure". Same is true with GPA and all. But I am pessimistic about seing that change in the medium to long term because it is so politically sensitive.

Passion. Juniors you want to hire will have a side project. That's all you need to see.

Most of the big professional sports already have this figured out. New college graduates have to compete for a spot at training camp. Hire them as temp contracts for two weeks to two months and let them play with the starting team.

Sounds roughly like an internship

If someone is willing to do their job well for a fair wage, why do you insist that they make their job their entire life outside work?

One classic approach is to over-hire and weed out. I find some form of this de-facto happens anyway, so managing more explicitly has some benefits.

And that is one of the best ways to hire new grads. Take the best of the crop of interns you've had.

The solution may also be not to make classes too hard. If, for example, your physics classes were of the same difficulty as the ones in my undergrad (easy to medium difficulty for the most part), then the 2.7 GPA is probably an accurate reflection of your abilities.

But if you went to a top university with brutal courses, and got a 2.7 GPA, then all I'm seeing is you're not elite material. The number otherwise does not help me one bit in evaluating you.

BTW, having spent a lot of time out of the US - it's still pretty laid back in the US. A person who is 2.7 GPA material in the US would simply not get admission in any decent university in some countries. And plenty of people in the US start all over at another institution and do well - something many countries don't allow (state funded, lack of resources, you have to move out of the way to let the younger batch in).[1]

[1] A good friend of mine totally flunked out of his university. He spent time off in the military. Then started all over at a new university. Got really high grades. Went to a top school for his PhD and is now a tenured faculty member.

Also you can hire people with 20+ years of experience that also can't code (where people claim to be a software engineer). FizzBuzz was a real filter for a while. It has amazed me how some people where able to slide by in larger organizations for years and then switch (internally or to another company) when the competency mattered. You can make a whole career it!

> If you claim to know C very well and can't knock that out in 5 minutes, and it takes you 45 minutes of prompting, well, you don't know C usefully.

I recently had an interview and a "skill test" in C. It was proctored by the interviewer in-person. I had so many questions about the questions. It was like, what is the output of "some code" and while obvious, there were some questions where specific CPU architecture mattered:

    #include <stdio.h>

    int main() {
        unsigned int x = 0x01020304;
        unsigned char *c = (unsigned char*)&x;

        printf("First byte of x: 0x%02x\n", *c);
        return 0;
    }

I was like, what architecture are we running on here? So, I answered that "it depends" and explained how it would depend on the architecture. They came back and said that I didn't know C.

Sure, whatever. Probably dodged a bullet.

It of course depends on what you’re hiring for, what qualities you value, and the scale you’re working at. But:

> I want to assess how you think, not which tools you use

suggests you have a more nuanced approach and aren’t just aiming for large numbers of drones.

What worked well for me (in a couple of smaller companies/teams) was:

- Talk to the candidates about their experiences in a project-oriented course where they had to work in a team. (Most CS programs have at least one of these. Get the name of that course ahead of time and just ask about it.) You want to find out if they can work in a team, divide up work and achieve interim goals, finish a project, deal with conflicts, handle setbacks and learn from mistakes, etc.

- Similarly, find out the names of some of the harder elective courses, and ask about their experiences in these. This gets at what they find interesting, how they think, and can help filter out GPA gamers.

- Talk to them about their experiences in whatever jobs, internships, volunteer work, or extracurricular activities they engaged in while at school. It doesn’t have to be directly related to your field—-you’re screening for work ethic and initiative.

Admittedly it’s been a while, but we used this approach for both on-campus recruiting and remote phone screens, and got pretty good at hitting these topics in a 15-20 minute conversation. We’d have one or two people screen maybe 30-50 candidates each recruiting season, identify 5-10 for on-site interviews with a larger team, and end up hiring about half of those.

This sort of bespoke screening does take some work on your part, and can be tough to scale. But we found it consistently identified solid candidates and led to outstanding hires.

It's subtle, but people who are self driven and learning for the sake of learning will talk differently. They tend to include more nuance and detail, addressing the subtle things. To be able to see those things requires internalization of what's learned, not just memorization. If you get good at it, you can do pretty well at recognizing these people even when they're in a different subject domain.

Remember, outside CS no one else does whiteboard interviews or takehome tests. It's generally a few conversations and that's it. It's because experts been sniff out other experts in their domain fairly quickly. It's about *how* they think, not what they know.

I'll give you an example of something subtle but is a frequent annoyance for me and I'm sure many others. You're on a webpage that asks for your country. Easy, just put in a drop-down, right? But what's much much better it's to use the localization information of the browser to place a copy of that country at the top of the list (a copy, not move). Sure, it saves us just scrolling to the bottom, but my partner is Korean and she never knows if she's looking for K(orea), S(outh Korea), or R(epublic of Korea). This happens for a surprising number of countries. Each individual use might just save a second or two of time, but remember you also need to multiply that by the number of times people interact with that page, so it can be millions of seconds. It'll also just leave users far less frustrated.

I'm also very sympathetic to the jitters stuff, because I get it a lot and make dumb mistakes when in the spotlight lol. But you can often find these things in other work they've done if they include their GitHub. Even something small like a dotfiles repo. And if the interview is more about validation their experience, the attention to detail and deeper knowledge will still show up in discussions especially if you get them to talk about something they're passionate about.

I'd also say that GPA and school names are very noisy (incidentally that often means internships too, since these strongly correlate). I know plenty of people from top 3 schools who do not know very basic things but have done rounds at top companies and can do leet code. But they're like GPT or people who complain about math word problems, they won't generalize and recognize these things in the wild. Overfit and studies to the test (this is a subtle thing you can use while interviewing too)

What architectures would it not be 04 on?

I think the best test for a Junior is to ask them to submit some of their OSS or personal fun projects they've worked on. From my perspective, especially with Juniors who aren't expected to be extremely knowledgeable, displaying a sense of curiosity and a willingness to learn is much more important.

If, hypothetically, there's two candidates, one who is more knowledgeable but has no personal projects versus someone who has less knowledge but has worked on different side projects in various languages/domains, I'm always going to pick the latter candidate since they clearly have a passion, and that passion will drive them to pick up the knowledge more than someone who's just doing it for a paycheck and could care less about expanding their own knowledge.

To go one step forward, you can ask them to go into detail about their side project, interesting problems they faced, how they overcame them, etc. Even introverts who are generally worse at small talk are on a much more balanced playing field when talking about something they're passionate about.

And it penalizes in many ways. Focusing too much on grades can be detrimental in graduate studies, despite graduate admissions focusing on GPA and test scores. I remember seeing 4.0 undergrads really struggle with research in grad school, sometimes to the point of dropping out. Certainly not always the case, but for the ones that did I think it speaks to your point about different skillsets.

Maybe worse was seeing the undergrads who passed on research opportunities out of fear it would distract them from keeping a high GPA.

> things like fizzbuzz are still useful

I think you're right here but, to play devil's advocate... isn't there some survivorship bias going on here? I assume you've never tested the negative hypothesis and gone ahead and hired somebody who couldn't program fizzbuzz to validate your assumption.

> As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted (the hard classes) and not cheating (as I did what I wanted) - I can say that there are consequences to this approach.

I can, too. I wanted to learn, but I also wanted to achieve a high GPA. I had a privileged background, so I got to retake classes after earning Cs or Bs until I got an A, without cheating.

The consequences: My degree took a long time to get, cost more money than my peers in the same program, and I now have a deep-seated feeling of inadequacy.

(older) ARM (aka, big-endian) it will be 01

You're right. When interviewing for a team that writes mostly in C and assembly (assembly for various different ISAs), we're not going to hire someone who claims to know C and fumbles through some basic problems and can't reason about hardware in the slightest.

IME: 1. build a co-op/intern program and hire out of that exclusively for junior. It's like an extended, two-way interview or try before you buy for both sides.

2. screen for passion and general technical competency above all else. You're going to make arbitrary decisions & restrictions (ex: we're only hiring from these 3 schools) which is fine, then work within those constraints. Ask about favorite classes (and why), what they've done lately or are excited about, side projects, OS contributions, building/reading/playing. The best intern I've hired lately answered some high-level questions about performance by building a simple PoC to demo some of their ideas, with React - a technology they didn't know but that we use.

3. recognize some things on the hiring side that from the hunting side don't make sense or are really annoying: you're playing a numbers game, hiring is a funnel, it's better to miss a great hire than go with a poor candidate (i.e. very risk averse), most hiring companies are at the mercy of the market; they hire poorer candidates and pay more, then get very picky and pay less. In a tight market you can't do much internally to stand out, and when lots of people are looking you don't have to.

> My personal take, we’ve made the cost of failure to high and cheating too easy.

I agree with the first part, but I think the second follows from it.

Take a class like organic chemistry. When I was in school, the grade was based on 5 exams, each worth 20% of your grade. Worse still, anything less than an A was seen as a failure for most students dreaming of medical/vet school.

Of course you are going to have people that are going to cheat. You've made the stakes so high that the consequences of getting caught cheating are meaningless.

On top of that, once enough students are cheating, you need to cheat just to keep up.

I just ran it on my M2 mac and got 04. Don't compilers typically take endianness into account for things like this anyway?

Most of this isn't even necessary; just look for passion and <anything> that gets them excited from a relevant technology area, then probe for legitimacy and learn about their interests. Being a jr. is all about the individual learning and skilling up, you really shouldn't be looking for existing expertise.

Anything big endian.

  unsigned int x = 0x01020304;
  unsigned char *c = (unsigned char*)&x;

Assume x is stored at 0x100. On a little endian architecture (x86, most modern ARM systems, etc), it will be stored in memory as [04][03][02][01], from bytes 0x100 to 0x103. If you assign char c to the address of x (0x100), it will read one byte, which is 0x4.

However, on a big endian system, that same value would be stored in memory as [01][02][03][04] - so, reading a byte at 0x100 would return 0x1.

Older ARM systems were big endian, and there are others that run that way, though it's rarer than it used to be. One of the perks of little endian is that if you want to read a smaller version of a value, you can read from the same address. To read that value as an 8, 16, or 32 bit value, I read at the same address. On a big endian system, I'd have to do more address math to do the same thing. It mostly doesn't matter, but it is nice to be able to have a "read of 8 bits at the address of the variable" do the sane thing and return the low order 8 bits, not the high order bits.

You'll have to be a lot more specific than "ARM" - Most newer ARM systems are little endian in practical operation, and ARM has been "flexible endian" (you can switch between big and little endian - SCTLR has the relevant bits to control the accesses on most recent ARM ISAs) for some long while now.

Do you know if compilers are smart enough to return 04 even on big-endian architectures nowadays? For some reason I'm under the impression that (at least clang and gcc) are able to change this from "first byte in x" to "least significant byte in x" but don't actually know why I think that. Maybe embedded compilers typically don't?

If I want to hire an artist, I'd like to see their portfolio. If they don't have commercial work they can show me, I'd like to see things they created on their own time.

> I assume you've never tested the negative hypothesis and gone ahead and hired somebody who couldn't program fizzbuzz to validate your assumption

A former employer of mine inadvertently did! He wasn't asked to complete FizzBuzz, but I am confident he couldn't answer it as I worked on the same team as him. He was a very charismatic individual who always "needed help" from team mates on all tasks, no matter how small. He managed to collect a salary for 6 months. Some time after he was let go, the police called my employer enquiring after him, and we learned he was a conman with outstanding arrest warrants with no prior SWE experience at all. The name we all knew him by was just one of many aliases.

The person above you teaches higher ed, and yet cannot articulate what you just did. Cheating isn't the problem, the system is.

> As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order.

At least for my CS degree, this surprisingly wasn't the case. I remember our freshman class advisor gave a speech that said that grades don't really matter so long as if you pass, but we all laughed and dismissed him. I ended up getting a big tech internship with a ~2.8 GPA and an even better full time job with a ~3.2.

Obviously, your mileage may vary. I graduated in a hot tech market from a prestigious university with a reputation of being difficult. Even so, overall, almost all of my classmates were stressed over grades significantly more than they needed to be.

No, and it would be wrong for it to do so, because you've given it a very explicit set of instructions about what to do: "Give me the value of the byte of memory at the start of x."

To do what you're asking, you'd do something like this:

  unsigned char c = (unsigned char)x;

That will give you the low order byte of x. But to do that, on a big endian system, when you've told it to get you the byte at the base address of x, is simply wrong behavior. At least in C. I can't speak to higher level languages since I don't work in them.

Can't or didn't? I had a different message to convey. You can't understand that. Or perhaps, you didn't understand that. Can't or didn't?

I reiterate:

But even requiring in person proctored exams is not the full solution. Students are not used to doing the necessary work to learn. They are used to doing the necessary work to pass. And that work is increasingly cheating. It’s a clusterfuck. I have calculus students who don’t know how to work with fractions. If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated.

K-12 needs to be changed as well.

When you graduate college all that people see is the degree; unless you go to graduate school and then they will look at grades but will notice many other things much more.

Going from high school to college grades are looked at a bit more, but that's because that, the essay, and the SAT are all they have.

The consequences of cheating could be made much more severe.

I am troubled by this argument because it suggests people have no ethical core. If that is true then we are going to have problems with them regardless.

To expand slightly on Syonyk said: the compiler cannot do it, because the object is stored between addresses c and c + sizeof(unsigned int). You can use this information to, for example, copy the object to another place with memcpy, and that of course wouldn't work if c wasn't pointing to the "leftmost" byte in the memory.

Unless, I suppose, sizeof was negative :).

My process is as follows:

1. Live coding, in Zoom or in person. Don't play gotcha on the language choice (unless there's a massive gulf in skill transference, like a webdev interviewing for an embedded C position). Pretend the 13 languages on the candidate's resume don't exist. Tell them it can be any of these x languages, which are every language you the interviewer feel comfortable to write leetcode in.

2. Write some easy problem in that language. I always go with some inefficient layout for the input data, then ask for something that's only one or two for loops away from being a stupid simple brute force solution. Good hygienic layout of the input data would have made this a single hashtable lookup.

3. Run the 45 minute interview with a lot of patience and positive feedback. One of the best hires in our department had first-time interview nerves and couldn't do anything for the first 10 minutes. I just complimented their thinking-out-loud, laughed at their jokes, and kept them from overthinking it.

4. 80% of interviewees will fail to write a meaningful loop. For the other 20%, spend the rest of the time talking about possible tradeoffs, anecdotes they share about similar design decisions, etc. The candidate will think you're writing in your laptop their scoring criteria, but you already passed them and generated a pop-sci personality test result for them of questionable accuracy. You're fishing for specific things to support your assessment, like they're good at both making and reviewing snap decisions and in doing so successfully saved a good portion of interview time, which contributed to their success. If it uses a weasel word, it's worth writing down.

5. Spend an hour (yes, longer than the interview) (and yes, block this time off in your calender) writing your interview assessment. Start with a 90s-television-tier assessment. For example, the candidate is nimble, constantly creating compelling technical alternatives, but is not focused on one, and they often communicate in jargon. DO NOT WRITE THIS DOWN. This is the lesson you want the geriatric senior management to take away from reading your assessment. Compose relatively long (I do 4 paragraphs minimum) prose that describes a slightly less stereotyped version of the above with plenty of examples, which you spent most of the interview time specifically fishing for. If the narrative is contradicted by the evidence, it's okay to re-write the narrative so the evidence fits.

6. When you're done, skim the job description you're hiring for. If there's a mismatch between that and the narrative you wrote, change your decision to no hire and explain why.

Doing this has gotten me eye rolls from coworkers but compliments at director+ level. I have had the CTO quote me once in a meeting. Putting that in my performance review packet made the whole thing worth it.

Most engineers, including good ones, that I've interviewed have no interesting GitHub contributions. GitHub is also game-able. Bootcamps, in particular, push their graduates to build an interesting GitHub portfolio.

I've found that talking through projects is a weak indicator of competence. It's much easier to memorize talking points than to produce working code.

I just want to second this (also did an undergrad in physics funny enough). I specifically sought out the harder professors in my undergrad and for the most part I'm happy I did it, but it's also a good thing that I'm not very motivated by money or prestige because I saw many of my colleagues who had gotten into better schools or jobs (even just the return calls on applications) who chose the easier routes or cheated. They are without a doubt wealthier. What mattered the most was the line items on their resumes and networking, but there is feedback in this so one begets the other. Fwiw, I had a 3.3.

So it then becomes hard for me to make suggestions to juniors. It isn't difficult to sniff out those like you or me who are motivated by the rabbit holes themselves, nor difficult to tell those who are entirely driven by social pressures (money, prestige, family, etc), but what about those on the edge? I think it's the morally best option to encourage learning for learning but it's naive to also not recognize that their peers who will cheat will be rewarded for that effort. It's clear that we do not optimize for the right things and we've fallen victim to Goodhart's Law, but I just hope we can recognize it because those systems are self reinforcing and the longer we work in them the harder they are to escape. Especially because there are many bright students who's major flaw is simply a lack of opportunity. For me? I'm just happy if I can be left to do my research, read papers and books, and have sufficient resources -- which is much more modest than many of my peers (ML). But it'd be naive to not recognize the costs and I'm a big believer in recognizing incentive structures and systematic issues. Unfortunately these are hard to resolve because they're caused by small choices by all of us collectively, but fortunately that too means they can be resolved by small choices each of us make.

> I'm also very sympathetic to the jitters stuff, because I get it a lot and make dumb mistakes when in the spotlight lol.

As an interviewer, I spot this and try to get them to ease up. I will talk about myself for a bit, about the work I do. I'm trying to get them to realize they are not in the spotlight, but whether we would be a good fit together; and thus both of us want them to work there.

BUT, my interviews tend to be about us solving a problem together, very rarely about actual code. For example, we might walk through how we would implement an email inbox system. We may discuss some of the finer details, if they come up, but generally, I'm interested in how they might design something they've basically used every day. How would we do search, what the database schema would look like, drafts, and so on.

I won't nudge them (to keep my biases in check), but I will help them down the path they choose, even if I don't like it. I'm not testing for the chosen path, but what "gotchas" they know and how they think though them. If you are a programmer, it shows. If you are an excellent programmer, it shows. If you are not a programmer, you won't make it 10 minutes.

One of the smartest people I know did 4 degrees in 4.5 years: undergrads in physics, chem, biochem, and math. He graduated with like a 3.2 gpa, low because he took 18-22 credits of hard classes every single semester, and couldn't get into med school. They made him take some stupid biochem masters, at which he excelled, particularly with a reduced course load. He then easily got admitted to med school.

If you don't want people to prioritize grades over everything else...

  > Really focusing on stretching yourself necessarily means lower grades.

I'm reminded of a saying/trope (whatever) I've seen in reference to surgeons and lawyers (I'm sure it's also been used in TV and movies). But the trope is that someone is looking for an expert and will be talking to a bunch of hotshots (let's say lawyers). They'll be bragging and then asked if they've ever lost a case, to which they proudly declare they have a spotless record. To which the person responds: then you've never taken a single risk.

It's overly dramatic, but I think gets the point across in an easy to understand way. It's exactly why you see the lower grade ones tutor the high grade ones (this even happened in my undergrad and I did physics[0]).

It's because learning happens when struggling. It happens at the edge. This is also a big reason some learn a lot faster than others or even why someone will say they don't understand but understand more than someone who says they do (and who believes it). Because expertise isn't about the high level general ideas, it's about all the little nitty gritty details, the subtle things that dramatically change things. But a big concern I have is that this is a skill to learn in of itself. I think it's not difficult to recognize when this skill is learned (at least if you have) but it's not something that'll be learned if we focus to much on scores. After all, they're just a proxy. Even the institutional prestige is a proxy (and I have an argument why it no longer matters though it did decades ago).

I do wonder if this is in part cause for the rise in enshitification. Similarly if this is why so many are bad at recognizing issues in LLMs and ML models. I'm sure it is but not sure how much this contributes or if it's purely a confounding variable.

[0] when I signed up to be a tutor at my university I got signed off my the toughest math professor. When I took the signature to the department the admin wasn't sure if I was trying to trick her because she immediately called the professor to confirm the signature. Then told me I could tutor whatever I wanted because I was one of two people he had ever signed off on. Admittedly, I'm sure a lot of that was because people were afraid of him (he wasn't mean, but he wouldn't let you be anything less than the best he thought you could be)

I think what I'm saying is more important to the type of interviews you do. And I think for the most part we agree (or I misunderstand?). Those interviews sound much closer to the classic engineering interview (as in not programming but like mechanical or civil engineering) or typical science interview. I think those are better interviews and more meaningful than live coding sessions or whiteboard problems.

Maybe here's a general question you can add (if you don't already use it) to bring out that thinking even if they're nervous. Since it's systems they are familiar with (my forum entry example is similar. I don't do front end), ask them what things they're frustrated with in tools they've used and how they could be fixed. It can help to ask if they've tried different solutions. With email that can be like if they just use Gmail via the Web, just use outlook or Apple Mail, or have tried things like Thunderbird, mux, or other aggregators. Why do they like the one they use? And if they've tried others I think that in itself is a signal that they will look for improvements on their own.

The things I think many interviews do poorly at is that they tend to look for knowledge. I get this, it's the easiest thing to measure because it's tangible. It's something you "have". While this matters, the job is often more dependent on intelligence and wisdom which are more about inference, attention, flexibility, and extrapolation. So I don't think it's so much about "gotchas" -- especially as many now just measure how "prepared" they are -- but, like you said, the way they think.

I'd much rather take someone with less knowledge (within reason) who is more intelligent, curious, and/or self driven by the work (not external things like money or prestige). Especially with juniors. A junior is an investment and thus more about their potential. As they say, you cannot teach someone who "already knows".

[EDIT]:

There's something else I should bring up about the "classic engineering" interview. Often they will discuss a problem they are actively working on. A reason for this is 1) it is fresh in their mind, 2) it gets at details, *but* 3) because it makes it easier for the interviewee to say "I don't know."

I think this is often an issue and sometimes why people will say weird erroneous things. They feel pressured to not admit they don't know and under those conditions, a guess is probably a better strategy. Since admitting lack of knowledge is an automatic "failure" while a guess has some chance, even if very small. At least some will admit to guessing before they do and you can also say its fine to guess and I see that often relax people and frequently results in them not guessing and instead reason through it (usually out loud).

(I'm an older grad student finishing up, so I frequently am dealing with undergrads where I'm teaching a class, holding office hours, or mentoring them in the lab. I've done interviews when I was a full time employee before grad school, and I notice there's a lot of similarities in these situations. That people are afraid to admit lack of knowledge when there is an "expert" in front of them. Even if they are explicitly there to get knowledge from said expert.)

It may be a result of personal preference, but I struggle to see how talking through challenges encountered with a personal project are a poor indicator of competence. If you ask some boilerplate list of questions, sure, but few if any candidates could memorize all of the random in-the-weeds architecture questions one could ask while talking through someone's project. For a junior specifically, even a non-answer to these questions provides valuable insight into their humility and self-awareness. I also think that it'd be pretty easy to visually weed out personal projects created for the sake of saying one has personal projects, like a bootcamp may push to create, versus an actual passion project, and even easier to weed out during any actual discussion. I suppose YMMV, but in my experience, the body language and flow of discussion are vastly different when someone is passionate about a subject versus not.

I wonder if we should take a look at how students, all paying tuition, have vastly unequal outcomes when it comes to job opportunities. Essentially there is a high scarcity of "good jobs" available to all but from the most selective universities.

Essentially when a scarcity increases, there will always be an imperfect heuristic of selection.

I guess this is more of a public policy area but it seems reasonable that anyone working full time should have access to economic security. Essentially cheating on university is the first symptom of lifetime of vastly unequal access to economic security.

In some cases, easier classes aren't a bad thing.

I had a decent GPA and took reasonably hard classes. I had a required discrete math class that was awful. The professor would assign homework for the next chapter that we hadn't gone over yet and them grade it as if it were a test. WTF am I paying you to teach me if I have to learn it myself before you ever present it and test me on that? Assign reading beforehand - great. Assign upgraded, or completion-graded homework beforehand - great. Grad it like a test before teaching it - BS. I took it with another professor after dropping the first one and they had more normal practices and it went much better.

You probably did dodge a bullet. The correct answer to every engineering question is "it depends". :)

Yeah, even just making people engage with source code from your system and answer questions about it or find bugs is better than asking them about their own portfolio.

If you wanted to return 04 on big-endian architectures, you can use a binary mask - (int &0xFF).

Since this compiles to FF 00 00 00 in big-endian and 00 00 00 FF in little-endian, it would work on both platforms.

If you’re reading a file in binary format from disk, though, you always have to know whether the byte you are reading is little-endian or big-endian on disk.

it doesn't to be on github or "interesting" though, if it's something that a person worked on in their free time - it's good enough to consider...

YUP

Perhaps another way to widen the scope of what is not cheatable (at the cost of more teacher work, ugh), is to require showing all work?

And I mean every draft, edit, etc.. All paper scratch-notes. Or on work on computer applications, a replayable video/screenshot series of all typing and edits, like a time-lapse of a construction site. Might even add opportunities to redirect work and thinking habits.

Of course, that too will eventually (probably way too soon) be AI-fakeable, so back to paper writing, typewriters, red pencils, and whiteout.

Just an idea; useful?

It also would be great if a person doing the interview could take the rest of the day off rather than jumping on a quick call with no time/interest to really try and understand the person on the other side of the desk. At the moment in most (all?) big companies an interview is something that noone wants to commit to and and when they have to - you understand the effort, dedication and focus that goes into it - that's right, none.

No, compilers don’t take endianness into account. (especially not C)

You need to use a bit mask in order to make this code endian-independent rather than a pointer alias. Like (uint8_t)(int & 0xFF), or something like that.

What counts as gaming? In my physics degree, for coding courses, we were allowed to use library algorithms directly provided we cited them. We were mostly tested on how (not) buggy and usable our program was. If you don't care what tools were used or how the solution came up, then that shouldn't be a problem.

If someone writes "perfect" code from a take-home, you can ask them to explain what they did (and if they used GPT, explain how they checked it). Then ask them to extend or discuss what the issues are and how they'd fix it.

I think asking some probing questions about past projects is normally enough to discern bullshit. You do need to be good at interviewing though. If you really want an excellent candidate then there's the FANG approach of (perhaps unfairly) filtering people who don't perform well in timed interviews, provided your rubric is good and you have enough candidates to compare to. There is a trade off there.

Grad positions optimise for what you can test - people are unlikely to have lots of side projects or work experience so you end up seeing how well they learned Algorithms 101. For someone who's worked for 10 years asking about system design in the context of their work is more useful.

Note that PhD and academic positions very rarely ask for this sort of stuff. Even if you don't have publications. They might run through a sample problem or theory (if it's even relevant), but I've never had to code to get a postdoc.

Otherwise you put people on short probation periods and be prepared to let them go.

When we talk about an ethical core, that sort of behavior exists between individuals. People in a family, or people who are friends, hopefully will and typically do adjust their behavior according to some sense of ethics. When we put people into a classroom, however, we’re implicitly putting them into competition with their peers for a limited set of opportunities that determine the extent to which their basic human needs, and those of their family, will be met in the future. Let me ask you, what is it about one’s ability to perform well in some arbitrary social role that makes them more entitled to their needs being met than another who lacks that particular ability? If you wanted to argue that a cheater is behaving unethically, you’d need to show that they do, in a moral and ethical sense, deserve less than their peers.

One of my teachers implemented a system like this. What they ended up doing was making it so that you had to score a (effectively) 9/10 on major assignments to pass the class (minor assignments were graded on completion), but had an infinite number of revisions with which to get this grade with feedback being provided each time you tried. Pretty much everyone passed, with more work required from some than from others. The only issue it ran into was with the final paper, where you (realistically) only had time to receive and make one to two revisions before the end of the semester and the deadline to submit grades.

You listed out what you think the options are. You have to pick one, so pick the least bad.

Realize there are practical limits to knowledge here. In the case of a new graduate, they are likely to have little or no job experience, so no one actually knows how they function in a workplace. Even if they were a personal family friend who you knew quite well, there would be considerable uncertainty.

What do you want out of your junior engineers? What is the actual skill, talent, or trait?

I don't think GPA, take-home assignments plus an interview about them, personality tests, or on-location tests like leetcode or architecture interviews are measuring the same thing. Are you just looking for any means to winnow down the pool of applicants, or is there an underlying ability you're searching for?

I think if you asked people who cheat if it was ethically wrong, they would admit it cheating is indeed unethical.

But we are really great about rationalizing away ethical issues. I suspect is a good grade is worth more than a personal sense of ethics.

As much as a med school wants ethical students, they want students with 4.0s more.

I think I judge these mostly by how much they know that falls outside the expected curriculum. It doesn’t even have to be related to the job, but the indication that they’ll learn without external motivation is a very large signal in their favor.

There’s also the ‘having an opinion on things’ factor. Someone that thinks things should be done a certain way, and can motivate that, will always be higher on my ranking, regardless of what that opinion is.

Why would it do that? You're asking for a raw memory address value.

It is not possible to differentiate someone who stretched themselves and got a lower grade from someone that got a lower grade for more mundane reasons.