Do AI detectors work? Students face false cheating accusations

Serious question from someone who is regularly tasked with hiring Juniors. What IS a good assessment for entry-level/right out of college positions?

-> GPA can be gamed, as laid out.

-> Take Home assessments can mostly be gamed, I want to assess how you think, now which tools you use.

-> Personality tests favor the outgoing/extroverts

-> On-location tests/leet code are a crapshoot.

What should be best practice here? Ideally something that controls for first-time interviewer jitters.

You have to use on-location tests. Do your best to be fair and get a true evaluation of the candidate's skills. It's not perfect but the alternatives are worse.

The other thing you have to do is that you have to be willing to fire the people who are underperforming. It's just a natural consequence of the interview process being imperfect.

> On-location tests/leet code are a crapshoot

They aren't 'fair' in avoiding every false negative, but they at least tell me that the passing candidates know and can do something.

If I ask someone who claims to know Python or Java whether or not you can have a memory leak in them, and their answer is 'no' or 'Maybe, but I don't know how', I get a pretty good idea of whether or not they know anything about this topic.

If you can't do fizzbuzz, you probably aren't a good fit for a SWE position either, you should be aiming for something more director-level. Given how much people struggle with coding, I sometimes feel like I may as well ditch my regular question, and just ask them to write that.

Back when I was at a small company doing a lot of new-grad interviews, it was really shocking how many people couldn't solve fizzbuzz or something equally trivial, like reversing an array in-place.

When I was at Google, most of these were filtered out before I got to see them, but for a while I was doing iOS development interviews and a lot of the candidates applying to Google clearly didn't know anything.

It's hard, and interviewing is better suited to answering "nope, not you!" questions than "yes, you'll be a good fit."

Onsite interviews with a range of approaches seem to be the best I've found over the years. As much as it pains me, things like fizzbuzz are still useful, because people still lie about their ability to program in languages. If you claim to know C very well and can't knock that out in 5 minutes, and it takes you 45 minutes of prompting, well, you don't know C usefully.

I've seen good results with having a pre-done sort of template program that's missing functionality, and the person completes it out based on comments (for remote interviews), and you can generally tell by watching them type how familiar with the space they are. Again, perfection isn't the goal, but if someone claims to know C very well and is trying to make Javascript syntax work, well, they're full of crap about knowing C.

That said, probably the best approach I've seen for hiring junior dev sorts is a formal summer internship program - and some places have a pretty solid system for doing this, with 20-30 people coming in every summer for a few months. That's a far better way to get to know someone's actual technical skills. In the programs I interacted with, it's safe to assume that if you have 30 people, you'll have about 15 that are "Thank you for your time, good luck..." sorts, maybe 5 or 8 that are "Yeah, you'd probably be a good fit here, and can be trained up in what we need, you'd be welcome back next summer!" and if you're lucky, one or two "HIRE NOW!" sorts that leave the summer program with a job offer.

It's obviously a lot higher effort than interviewing, but the "Throw things at people for three months and see what they do, with a defined end of the program" process seems to be a really good filter for finding quality people.

The first time I saw FizzBuzz, I immediately assumed it was some sort of "trap" or "trick" interview question - that there's some deviously subtle little thing in it that you'll miss at first or second glance, as a "gotcha." It literally never occurred to me that it was, in fact, a basic "Can you code your way out of a paper bag given a map?" sort of question to check for basic code competence in languages.

Then I started interviewing, and... yeah. I get it now. It really is that simple, should take a competent coder a few minutes, and 80% of people interviewing will take 45 minutes to muddle their way through it.

Passion. Juniors you want to hire will have a side project. That's all you need to see.

Most of the big professional sports already have this figured out. New college graduates have to compete for a spot at training camp. Hire them as temp contracts for two weeks to two months and let them play with the starting team.

Sounds roughly like an internship

If someone is willing to do their job well for a fair wage, why do you insist that they make their job their entire life outside work?

One classic approach is to over-hire and weed out. I find some form of this de-facto happens anyway, so managing more explicitly has some benefits.

And that is one of the best ways to hire new grads. Take the best of the crop of interns you've had.

> If you claim to know C very well and can't knock that out in 5 minutes, and it takes you 45 minutes of prompting, well, you don't know C usefully.

I recently had an interview and a "skill test" in C. It was proctored by the interviewer in-person. I had so many questions about the questions. It was like, what is the output of "some code" and while obvious, there were some questions where specific CPU architecture mattered:

    #include <stdio.h>

    int main() {
        unsigned int x = 0x01020304;
        unsigned char *c = (unsigned char*)&x;

        printf("First byte of x: 0x%02x\n", *c);
        return 0;
    }

I was like, what architecture are we running on here? So, I answered that "it depends" and explained how it would depend on the architecture. They came back and said that I didn't know C.

Sure, whatever. Probably dodged a bullet.

It of course depends on what you’re hiring for, what qualities you value, and the scale you’re working at. But:

> I want to assess how you think, not which tools you use

suggests you have a more nuanced approach and aren’t just aiming for large numbers of drones.

What worked well for me (in a couple of smaller companies/teams) was:

- Talk to the candidates about their experiences in a project-oriented course where they had to work in a team. (Most CS programs have at least one of these. Get the name of that course ahead of time and just ask about it.) You want to find out if they can work in a team, divide up work and achieve interim goals, finish a project, deal with conflicts, handle setbacks and learn from mistakes, etc.

- Similarly, find out the names of some of the harder elective courses, and ask about their experiences in these. This gets at what they find interesting, how they think, and can help filter out GPA gamers.

- Talk to them about their experiences in whatever jobs, internships, volunteer work, or extracurricular activities they engaged in while at school. It doesn’t have to be directly related to your field—-you’re screening for work ethic and initiative.

Admittedly it’s been a while, but we used this approach for both on-campus recruiting and remote phone screens, and got pretty good at hitting these topics in a 15-20 minute conversation. We’d have one or two people screen maybe 30-50 candidates each recruiting season, identify 5-10 for on-site interviews with a larger team, and end up hiring about half of those.

This sort of bespoke screening does take some work on your part, and can be tough to scale. But we found it consistently identified solid candidates and led to outstanding hires.

It's subtle, but people who are self driven and learning for the sake of learning will talk differently. They tend to include more nuance and detail, addressing the subtle things. To be able to see those things requires internalization of what's learned, not just memorization. If you get good at it, you can do pretty well at recognizing these people even when they're in a different subject domain.

Remember, outside CS no one else does whiteboard interviews or takehome tests. It's generally a few conversations and that's it. It's because experts been sniff out other experts in their domain fairly quickly. It's about *how* they think, not what they know.

I'll give you an example of something subtle but is a frequent annoyance for me and I'm sure many others. You're on a webpage that asks for your country. Easy, just put in a drop-down, right? But what's much much better it's to use the localization information of the browser to place a copy of that country at the top of the list (a copy, not move). Sure, it saves us just scrolling to the bottom, but my partner is Korean and she never knows if she's looking for K(orea), S(outh Korea), or R(epublic of Korea). This happens for a surprising number of countries. Each individual use might just save a second or two of time, but remember you also need to multiply that by the number of times people interact with that page, so it can be millions of seconds. It'll also just leave users far less frustrated.

I'm also very sympathetic to the jitters stuff, because I get it a lot and make dumb mistakes when in the spotlight lol. But you can often find these things in other work they've done if they include their GitHub. Even something small like a dotfiles repo. And if the interview is more about validation their experience, the attention to detail and deeper knowledge will still show up in discussions especially if you get them to talk about something they're passionate about.

I'd also say that GPA and school names are very noisy (incidentally that often means internships too, since these strongly correlate). I know plenty of people from top 3 schools who do not know very basic things but have done rounds at top companies and can do leet code. But they're like GPT or people who complain about math word problems, they won't generalize and recognize these things in the wild. Overfit and studies to the test (this is a subtle thing you can use while interviewing too)

What architectures would it not be 04 on?

I think the best test for a Junior is to ask them to submit some of their OSS or personal fun projects they've worked on. From my perspective, especially with Juniors who aren't expected to be extremely knowledgeable, displaying a sense of curiosity and a willingness to learn is much more important.

If, hypothetically, there's two candidates, one who is more knowledgeable but has no personal projects versus someone who has less knowledge but has worked on different side projects in various languages/domains, I'm always going to pick the latter candidate since they clearly have a passion, and that passion will drive them to pick up the knowledge more than someone who's just doing it for a paycheck and could care less about expanding their own knowledge.

To go one step forward, you can ask them to go into detail about their side project, interesting problems they faced, how they overcame them, etc. Even introverts who are generally worse at small talk are on a much more balanced playing field when talking about something they're passionate about.

> things like fizzbuzz are still useful

I think you're right here but, to play devil's advocate... isn't there some survivorship bias going on here? I assume you've never tested the negative hypothesis and gone ahead and hired somebody who couldn't program fizzbuzz to validate your assumption.

(older) ARM (aka, big-endian) it will be 01

You're right. When interviewing for a team that writes mostly in C and assembly (assembly for various different ISAs), we're not going to hire someone who claims to know C and fumbles through some basic problems and can't reason about hardware in the slightest.

IME: 1. build a co-op/intern program and hire out of that exclusively for junior. It's like an extended, two-way interview or try before you buy for both sides.

2. screen for passion and general technical competency above all else. You're going to make arbitrary decisions & restrictions (ex: we're only hiring from these 3 schools) which is fine, then work within those constraints. Ask about favorite classes (and why), what they've done lately or are excited about, side projects, OS contributions, building/reading/playing. The best intern I've hired lately answered some high-level questions about performance by building a simple PoC to demo some of their ideas, with React - a technology they didn't know but that we use.

3. recognize some things on the hiring side that from the hunting side don't make sense or are really annoying: you're playing a numbers game, hiring is a funnel, it's better to miss a great hire than go with a poor candidate (i.e. very risk averse), most hiring companies are at the mercy of the market; they hire poorer candidates and pay more, then get very picky and pay less. In a tight market you can't do much internally to stand out, and when lots of people are looking you don't have to.

I just ran it on my M2 mac and got 04. Don't compilers typically take endianness into account for things like this anyway?

Most of this isn't even necessary; just look for passion and <anything> that gets them excited from a relevant technology area, then probe for legitimacy and learn about their interests. Being a jr. is all about the individual learning and skilling up, you really shouldn't be looking for existing expertise.

Anything big endian.

  unsigned int x = 0x01020304;
  unsigned char *c = (unsigned char*)&x;

Assume x is stored at 0x100. On a little endian architecture (x86, most modern ARM systems, etc), it will be stored in memory as [04][03][02][01], from bytes 0x100 to 0x103. If you assign char c to the address of x (0x100), it will read one byte, which is 0x4.

However, on a big endian system, that same value would be stored in memory as [01][02][03][04] - so, reading a byte at 0x100 would return 0x1.

Older ARM systems were big endian, and there are others that run that way, though it's rarer than it used to be. One of the perks of little endian is that if you want to read a smaller version of a value, you can read from the same address. To read that value as an 8, 16, or 32 bit value, I read at the same address. On a big endian system, I'd have to do more address math to do the same thing. It mostly doesn't matter, but it is nice to be able to have a "read of 8 bits at the address of the variable" do the sane thing and return the low order 8 bits, not the high order bits.

You'll have to be a lot more specific than "ARM" - Most newer ARM systems are little endian in practical operation, and ARM has been "flexible endian" (you can switch between big and little endian - SCTLR has the relevant bits to control the accesses on most recent ARM ISAs) for some long while now.

Do you know if compilers are smart enough to return 04 even on big-endian architectures nowadays? For some reason I'm under the impression that (at least clang and gcc) are able to change this from "first byte in x" to "least significant byte in x" but don't actually know why I think that. Maybe embedded compilers typically don't?

If I want to hire an artist, I'd like to see their portfolio. If they don't have commercial work they can show me, I'd like to see things they created on their own time.

> I assume you've never tested the negative hypothesis and gone ahead and hired somebody who couldn't program fizzbuzz to validate your assumption

A former employer of mine inadvertently did! He wasn't asked to complete FizzBuzz, but I am confident he couldn't answer it as I worked on the same team as him. He was a very charismatic individual who always "needed help" from team mates on all tasks, no matter how small. He managed to collect a salary for 6 months. Some time after he was let go, the police called my employer enquiring after him, and we learned he was a conman with outstanding arrest warrants with no prior SWE experience at all. The name we all knew him by was just one of many aliases.

No, and it would be wrong for it to do so, because you've given it a very explicit set of instructions about what to do: "Give me the value of the byte of memory at the start of x."

To do what you're asking, you'd do something like this:

  unsigned char c = (unsigned char)x;

That will give you the low order byte of x. But to do that, on a big endian system, when you've told it to get you the byte at the base address of x, is simply wrong behavior. At least in C. I can't speak to higher level languages since I don't work in them.

To expand slightly on Syonyk said: the compiler cannot do it, because the object is stored between addresses c and c + sizeof(unsigned int). You can use this information to, for example, copy the object to another place with memcpy, and that of course wouldn't work if c wasn't pointing to the "leftmost" byte in the memory.

Unless, I suppose, sizeof was negative :).

My process is as follows:

1. Live coding, in Zoom or in person. Don't play gotcha on the language choice (unless there's a massive gulf in skill transference, like a webdev interviewing for an embedded C position). Pretend the 13 languages on the candidate's resume don't exist. Tell them it can be any of these x languages, which are every language you the interviewer feel comfortable to write leetcode in.

2. Write some easy problem in that language. I always go with some inefficient layout for the input data, then ask for something that's only one or two for loops away from being a stupid simple brute force solution. Good hygienic layout of the input data would have made this a single hashtable lookup.

3. Run the 45 minute interview with a lot of patience and positive feedback. One of the best hires in our department had first-time interview nerves and couldn't do anything for the first 10 minutes. I just complimented their thinking-out-loud, laughed at their jokes, and kept them from overthinking it.

4. 80% of interviewees will fail to write a meaningful loop. For the other 20%, spend the rest of the time talking about possible tradeoffs, anecdotes they share about similar design decisions, etc. The candidate will think you're writing in your laptop their scoring criteria, but you already passed them and generated a pop-sci personality test result for them of questionable accuracy. You're fishing for specific things to support your assessment, like they're good at both making and reviewing snap decisions and in doing so successfully saved a good portion of interview time, which contributed to their success. If it uses a weasel word, it's worth writing down.

5. Spend an hour (yes, longer than the interview) (and yes, block this time off in your calender) writing your interview assessment. Start with a 90s-television-tier assessment. For example, the candidate is nimble, constantly creating compelling technical alternatives, but is not focused on one, and they often communicate in jargon. DO NOT WRITE THIS DOWN. This is the lesson you want the geriatric senior management to take away from reading your assessment. Compose relatively long (I do 4 paragraphs minimum) prose that describes a slightly less stereotyped version of the above with plenty of examples, which you spent most of the interview time specifically fishing for. If the narrative is contradicted by the evidence, it's okay to re-write the narrative so the evidence fits.

6. When you're done, skim the job description you're hiring for. If there's a mismatch between that and the narrative you wrote, change your decision to no hire and explain why.

Doing this has gotten me eye rolls from coworkers but compliments at director+ level. I have had the CTO quote me once in a meeting. Putting that in my performance review packet made the whole thing worth it.

Most engineers, including good ones, that I've interviewed have no interesting GitHub contributions. GitHub is also game-able. Bootcamps, in particular, push their graduates to build an interesting GitHub portfolio.

I've found that talking through projects is a weak indicator of competence. It's much easier to memorize talking points than to produce working code.

> I'm also very sympathetic to the jitters stuff, because I get it a lot and make dumb mistakes when in the spotlight lol.

As an interviewer, I spot this and try to get them to ease up. I will talk about myself for a bit, about the work I do. I'm trying to get them to realize they are not in the spotlight, but whether we would be a good fit together; and thus both of us want them to work there.

BUT, my interviews tend to be about us solving a problem together, very rarely about actual code. For example, we might walk through how we would implement an email inbox system. We may discuss some of the finer details, if they come up, but generally, I'm interested in how they might design something they've basically used every day. How would we do search, what the database schema would look like, drafts, and so on.

I won't nudge them (to keep my biases in check), but I will help them down the path they choose, even if I don't like it. I'm not testing for the chosen path, but what "gotchas" they know and how they think though them. If you are a programmer, it shows. If you are an excellent programmer, it shows. If you are not a programmer, you won't make it 10 minutes.

I think what I'm saying is more important to the type of interviews you do. And I think for the most part we agree (or I misunderstand?). Those interviews sound much closer to the classic engineering interview (as in not programming but like mechanical or civil engineering) or typical science interview. I think those are better interviews and more meaningful than live coding sessions or whiteboard problems.

Maybe here's a general question you can add (if you don't already use it) to bring out that thinking even if they're nervous. Since it's systems they are familiar with (my forum entry example is similar. I don't do front end), ask them what things they're frustrated with in tools they've used and how they could be fixed. It can help to ask if they've tried different solutions. With email that can be like if they just use Gmail via the Web, just use outlook or Apple Mail, or have tried things like Thunderbird, mux, or other aggregators. Why do they like the one they use? And if they've tried others I think that in itself is a signal that they will look for improvements on their own.

The things I think many interviews do poorly at is that they tend to look for knowledge. I get this, it's the easiest thing to measure because it's tangible. It's something you "have". While this matters, the job is often more dependent on intelligence and wisdom which are more about inference, attention, flexibility, and extrapolation. So I don't think it's so much about "gotchas" -- especially as many now just measure how "prepared" they are -- but, like you said, the way they think.

I'd much rather take someone with less knowledge (within reason) who is more intelligent, curious, and/or self driven by the work (not external things like money or prestige). Especially with juniors. A junior is an investment and thus more about their potential. As they say, you cannot teach someone who "already knows".

[EDIT]:

There's something else I should bring up about the "classic engineering" interview. Often they will discuss a problem they are actively working on. A reason for this is 1) it is fresh in their mind, 2) it gets at details, *but* 3) because it makes it easier for the interviewee to say "I don't know."

I think this is often an issue and sometimes why people will say weird erroneous things. They feel pressured to not admit they don't know and under those conditions, a guess is probably a better strategy. Since admitting lack of knowledge is an automatic "failure" while a guess has some chance, even if very small. At least some will admit to guessing before they do and you can also say its fine to guess and I see that often relax people and frequently results in them not guessing and instead reason through it (usually out loud).

(I'm an older grad student finishing up, so I frequently am dealing with undergrads where I'm teaching a class, holding office hours, or mentoring them in the lab. I've done interviews when I was a full time employee before grad school, and I notice there's a lot of similarities in these situations. That people are afraid to admit lack of knowledge when there is an "expert" in front of them. Even if they are explicitly there to get knowledge from said expert.)

It may be a result of personal preference, but I struggle to see how talking through challenges encountered with a personal project are a poor indicator of competence. If you ask some boilerplate list of questions, sure, but few if any candidates could memorize all of the random in-the-weeds architecture questions one could ask while talking through someone's project. For a junior specifically, even a non-answer to these questions provides valuable insight into their humility and self-awareness. I also think that it'd be pretty easy to visually weed out personal projects created for the sake of saying one has personal projects, like a bootcamp may push to create, versus an actual passion project, and even easier to weed out during any actual discussion. I suppose YMMV, but in my experience, the body language and flow of discussion are vastly different when someone is passionate about a subject versus not.

You probably did dodge a bullet. The correct answer to every engineering question is "it depends". :)

Yeah, even just making people engage with source code from your system and answer questions about it or find bugs is better than asking them about their own portfolio.

If you wanted to return 04 on big-endian architectures, you can use a binary mask - (int &0xFF).

Since this compiles to FF 00 00 00 in big-endian and 00 00 00 FF in little-endian, it would work on both platforms.

If you’re reading a file in binary format from disk, though, you always have to know whether the byte you are reading is little-endian or big-endian on disk.

it doesn't to be on github or "interesting" though, if it's something that a person worked on in their free time - it's good enough to consider...

It also would be great if a person doing the interview could take the rest of the day off rather than jumping on a quick call with no time/interest to really try and understand the person on the other side of the desk. At the moment in most (all?) big companies an interview is something that noone wants to commit to and and when they have to - you understand the effort, dedication and focus that goes into it - that's right, none.

No, compilers don’t take endianness into account. (especially not C)

You need to use a bit mask in order to make this code endian-independent rather than a pointer alias. Like (uint8_t)(int & 0xFF), or something like that.

What counts as gaming? In my physics degree, for coding courses, we were allowed to use library algorithms directly provided we cited them. We were mostly tested on how (not) buggy and usable our program was. If you don't care what tools were used or how the solution came up, then that shouldn't be a problem.

If someone writes "perfect" code from a take-home, you can ask them to explain what they did (and if they used GPT, explain how they checked it). Then ask them to extend or discuss what the issues are and how they'd fix it.

I think asking some probing questions about past projects is normally enough to discern bullshit. You do need to be good at interviewing though. If you really want an excellent candidate then there's the FANG approach of (perhaps unfairly) filtering people who don't perform well in timed interviews, provided your rubric is good and you have enough candidates to compare to. There is a trade off there.

Grad positions optimise for what you can test - people are unlikely to have lots of side projects or work experience so you end up seeing how well they learned Algorithms 101. For someone who's worked for 10 years asking about system design in the context of their work is more useful.

Note that PhD and academic positions very rarely ask for this sort of stuff. Even if you don't have publications. They might run through a sample problem or theory (if it's even relevant), but I've never had to code to get a postdoc.

Otherwise you put people on short probation periods and be prepared to let them go.

You listed out what you think the options are. You have to pick one, so pick the least bad.

Realize there are practical limits to knowledge here. In the case of a new graduate, they are likely to have little or no job experience, so no one actually knows how they function in a workplace. Even if they were a personal family friend who you knew quite well, there would be considerable uncertainty.

What do you want out of your junior engineers? What is the actual skill, talent, or trait?

I don't think GPA, take-home assignments plus an interview about them, personality tests, or on-location tests like leetcode or architecture interviews are measuring the same thing. Are you just looking for any means to winnow down the pool of applicants, or is there an underlying ability you're searching for?

I think I judge these mostly by how much they know that falls outside the expected curriculum. It doesn’t even have to be related to the job, but the indication that they’ll learn without external motivation is a very large signal in their favor.

There’s also the ‘having an opinion on things’ factor. Someone that thinks things should be done a certain way, and can motivate that, will always be higher on my ranking, regardless of what that opinion is.

Why would it do that? You're asking for a raw memory address value.