Please stop the coding challenges

I recently ran an interview process for a relatively senior eng role at a tiny startup. Because I believe different interview methods work better for different people, I offered everyone a choice:

1. Do a takehome test, targeted to take about 4 hours but with no actual time limit. This was a non-algorithmic project that was just a stripped-down version of what I'd spent the last month on in actual work.

2. Do an onsite pairing exercise in 2 hours. This would be a version of #1, but more of "see how far we get in 2 hours."

3. Submit a code sample of pre-existing work.

Based on the ire I've seen takehome tests get, I figured we'd get a good spread between all three, but amazingly, ~90-95% of candidates chose the takehome test. That matches my preference as a candidate as well.

I don't know if this generalizes beyond this company/role, but it was an interesting datapoint - I was very surprised to find that most people preferred it!

Why would you even do any of that for a senior role? I wouldn't waste my time with it, and it shows you don't know how to interview/evaluate for a senior position.

I've come around to the idea that all people who will be expected to write code as part of their job need to be evaluated for their ability to write code as part of the interview process. I've seen too many people write convincing and impressive resumes without a single ounce of technical ability to their name.

Why wouldn’t you? How should one evaluate a senior position, in your opinion? Are you suggesting a senior dev shouldn’t need to demonstrate any coding skill? (If not, why not?) Or are you suggesting that there are other faster ways to show coding skills?

Interesting. I think I slightly prefer take homes, mostly because they're not as hectic. The problem ends up being, that I don't have any guarantee that the other side will spend any time on it. At least with an in person, I know that the company had to incur a significant cost so I know my time is less likely to be wasted. There's already a growing asymmetry with job searches, and this is one more trend that accelerates that. I still have a non technical friend who's annoyed with me for not completing a take home years ago, because, while the main ask was obviously trivial, there was enough slop in the directions that I could kill hours trying to decide if something was good enough to submit. On top of that it was expecting some fairly narrow set of techs, that, while not difficult to pick up, were not something I was interested in trying to learn enough about for a chance to maybe help some company that "couldn't find any technical talent".

What would you waste your time with?

I exclusively give pairing interviews, usually just 1 hr. The variance between good and bad candidates is amazing, and you wouldn't guess at all from resumes or casual conversations. Most candidates have given me very positive feedback. Why wouldn't you want to be interviewed like this?

Because for a senior role you need to hire a person with senior role skills, I suppose. Just N years of experience at X company doesn't provide that. I'd rather look at someone's github repo than talk to their reference tbh.

Interesting! I like the idea of choice, but as a hiring manager it makes my problem harder. How do you compare the results from different choices equitably? I find trying to compare candidates fairly to be quite difficult, even when they have the exact same interview.

Last time I did a coding interview for real, I had the choice of any programming language, and could choose between 3 different problems to solve. I liked that quite a bit, and was offered the job. Being able to choose Python, instead of, say, C++ in a time-bound interview almost feels like cheating.

> ~90-95% of candidates chose the takehome test

I was expecting "3. submit a code sample" to be the overwhelming winner here - did anybody choose that? Seems like a no-brainer, since it's already done...

I've never worked at a company with a public repo. I never understood this. I have a life outside of work

in one interview for a python job i was allowed to submit solutions to tests in pike which i had more experience with, but none of the devs at the company had ever seen. for python they had an automated testsuite but i remember my interviewer telling me that the devs were pouring over my test results to verify them manually while we continued other parts of the interview. i got the job.

The code I've written in personal projects have been pretty messy -- whereas code I've written professionally tend to be better (not perfect), but of course I can't share that code.

On the employer side, I would prefer the take home assignment over existing code, unless the existing code was super high quality or highly relevant to the company.

You're hiring for a senior role, not a junior or mid.

Your candidates aren't fresh out of school or just starting their careers. They already have plenty of work experience. They already know how to code and have been doing it for years. A conversation will tell you much more about their approaches to problems solving, team work, and mentoring than a coding challenge will. Looking at their GitHub, their work experience and references will tell you the rest.

It's like asking a doctor to go over basic human anatomy.

Neither did I. But I'd rather just "smuggle" out some past work and "rewrite it" to make it inconspicuous, if I didn't have my own personal work to show. The company I work for isn't well known so saying "I worked there in a senior role for 10 years" isn't going to get me the next job. If I didn't have anything to show for that I either made for fun or could display from a past job, I'd not apply for another until I had tbh.

People say "No I don't have any past hobby work, never contributed to OSS, and all my professional work is verboten to show" that's not unusual, but it's also not an encouraging sign to me when interviewing.

I responded to a different comment but it's also relevant here.

You're hiring for a senior role, not a junior or mid.

Your candidates aren't fresh out of school or just starting their careers. They already have plenty of work experience. They already know how to code and have been doing it for years. A conversation will tell you much more about their approaches to problems solving, team work, and mentoring than a coding challenge will. Looking at their GitHub, their work experience and references will tell you the rest.

It's like asking a doctor to go over basic human anatomy.

I'm a senior developer. Of course I know how to code. Do you know how many code challenges I've done over the course of my career? 99% of them where terrible at evaluating my ability to code. They are a waste of time at the senior level. You are much better off with a conversation, looking at past work and experience and references than you are doing code challenges.

Wouldn't that be a red flag if they don't have any public code at all as a senior developer? They aren't fresh out of school or just starting their careers. They should have something.

You’re underestimating the variance in senior devs, and you’re failing to look at this from the interviewer’s perspective. Sometimes doctors should be quizzed on anatomy, btw.

They need to be able to compare candidates, and senior devs will be expected to code. Their goal is to pick the best candidate. Asking them to demonstrate their coding skill isn’t just fair game, it’s what half the interviewees actually want. A lot of people prefer being evaluated on code than on soft skills (just read the threads here for tons of evidence.)

It’s useful to know whether a candidate is going to be on the principal engineer path or management path. It’s useful to know whether they are actually good at coding, and how good exactly. And it’s also useful to know if they are willing to do what needs to be done once hired. Someone interviewing for a senior coding position who refuses to code during the interview is about as big of a red flag as you can have, and I’ve been part of hiring teams that will politely excuse someone for that, and I agree with the reasoning.

This might not make sense until you’re on the hiring side of things, but having a resume does not entitle one to being hired for a higher title and more money.

> as a hiring manager it makes my problem harder. How do you compare the results from different choices equitably?

That makes sense, and it's the perspective that's being drilled into a lot of us. Implicit bias and all that. But in my experience, the comparison problem has never been that big of an issue in practice. I guess it depends on the hiring climate, but I'm much more familiar with spending a lot of time going through mediocre candidates and looking for someone who's "good enough". And this is when the recruiter is doing their job, and I understand why each candidate made it to the interview stage. Sure, sometimes it's a hot position (or rather, hot group to work for) and we get multiple good candidates, but then the decisionmaking process is more about "do we want A, who has deep and solid experience in this exact area; or B, who blew us away with their breadth of knowledge, flexibility, and creativity?" than something like "do we want A who did great on the take-home test but was unimpressive in person, or B whose solution to the take-home test was so-so but was clearly very knowledgeable and experienced in person?" The latter is in a hypothetical case where everyone did the same stuff, just to make it easier to compare, but even in that setup that's an uncommon and uninteresting choice. You're comparing test results and trying to infer Truth from it. Test results don't give a lot of signal or predictive power in the first place, so if you're trying to be objective by relying only on normalized scores, then you're not working with much of value.

Take home tests or whiteboard tests or whatever are ok to use as a filter, but people aren't points along a one-dimensional "quality" line. Use whatever you have to in order to find people that have a decent probability of being good enough, then stop thinking about how they might fail and start thinking about what it would look like if they succeed. They'll have different strengths and advantages. Standardizing your tests isn't going to help you explore those.

>They already know how to code and have been doing it for years.

I'm telling you from my own experience that this is observably not true. Twice now I've been burned by internal hires who objectively had years of experience (easy to verify when they have been working within my division) and yet couldn't code their way out of a paper bag.

And tangentially, with some of the health care experiences I've had I very well might try to administer a basic anatomy quiz to potential doctors if I thought I could get away with it.

After interviewing ~50 people using this technique - I'm sorry but there's no "of course". I cannot tell by looking at your past work experience or talking to you whether you know how to code.

I know this, because I used to look at resumes and have long conversations before I ran my pairing interview. And I found the results almost uncorrelated. Now I save the fun conversations for people who make it through the screening. But by then I already know if I want to hire them.

My pairing interview (it's a pairing session, not a code challenge) covers aspects of engineering that I specifically care about. It's not terribly hard. But at the end I'll have a pretty good idea if I want to turn this person loose in my codebase.

I'm a senior developer and my github is half guitar tabs. I'm not interested in peacocking. Maybe it's because Hackaday refused to put my name on the article with my senior design project years ago and I just don't want to play the game.

Highly agree with all of that, perhaps save the conclusion. I still try to standardize the interview process, but we have enough different kinds of interview phases to capture different strengths and weaknesses of candidates. I still want the interview to be fair even when people respond very differently. You’re right that it doesn’t often come anywhere close to a tie. But sometimes candidates aren’t vocal and don’t vouch for themselves strongly but they are great coders, and sometimes people are talkative and sell themselves very well but when it comes down to technical ability they aren’t amazing. The choice of coding interview could obscure some of that, so mainly I include other parts to the interview, though I’m still interested in how to compare someone who does take home coding to someone who does the live pair programming, for example. I kinda want to see how candidates handle both. ;)

Yes exactly. The code IS the cv of a developer.

The elitism is rampant. As an embedded sector guy everyone thinks every other RTOS isn't a real RTOS and the knowledge and wisdom don't translate.

Every time I find some open source I would contribute to, what I wanted to implement is already there or is already in the works. I'm not going to change shit just to change it. I'll buy them a coffee tip and leave.

And that might be hurting you in the long run. ¯ \ _ ( ツ ) _ / ¯

All of the meaningful code I write is private for the company I work for. This is how most code is stored.

Yeah, I saw that coming a mile away. Don't you remember uni? Students always breathed a sigh of relief when the final exam was a take home project.

Now add this to the fact that a vast majority of your applicants are going to be feeding your assignment directly into an assistive LLM.

As an interviewer, you really can't win.

- If you have a take-home test, people will either complain that it's too involved, and time consuming.

- If you do whiteboarding, people will claim that it's not representative of the actual job.

- If you do paired programming, people will claim it's unfair to those who don't test well under pressure.

Did you ask why they chose the take home?

I'd favor 1 and 2 over 3, because I feel that you understanding the problem I'm solving gives a better context to appreciate my approach.

If I take it home and hand it later, you can compare my solution to yours (or others) and better grasp my coding style, even if it's different to yours, since we worked on essentially the same task.

If we're working as a pair, you can observe my thought process. I can better express my reasoning for picking certain idioms.

With past code, there's almost no context. It's just code that was written for some other project. In my opinion, this can be very useful in preliminary steps, when I'm sending out my resume and you need to validate that I can indeed write code. In later stages, I think it can be impressive if I have a working app and can walk you down the code for a particular feature. But who has working software laying around? We all have code that works with a few hacks, sorta...

I'd favor 1 over 2, because it reduces the probability to blunder. Between spending 2 hours on-site with a higher risk to mess up (due to pressure, Murphy's law, etc) and take the assignment home to work on it for an extra 2 hours, with ample time to freely research whatever I don't understand. It's a no-brainer for me.

99.9% of the code I’ve written is proprietary.

Oops, sorry, I think I mangled my conclusion a bit. I'm not against standardizing tests. If it doesn't get in the way of other attributes, standardization is very valuable. It's just that those results aren't enough by themselves.

> I still try to standardize the interview process, but we have enough different kinds of interview phases to capture different strengths and weaknesses of candidates.

100% agree with this approach.

> Why would you even do any of that for a senior role?

I used to think like that ... then we hired a batch of 3 "senior" devs who could not do simple, everyday coding tasks, even in their ostensible languages of preference. All had come with exceptional resumés and personal recommendations.

So now, no one at any level one gets hired without demonstrating that they can at least write a fizzbuzz and commit it to a repository.

Now, I doubt that senior engineers in other fields have to do this sort of thing. But, most other engineering fields have licensing/accreditation with accompanying post-graduate academic curricula. We don't (but probably should).

Mine too. Or 99.99% of it at least. But if I were to apply for a job, I'd focus more on having at least a little repo of something to show, than on polishing my CV or doing leetcode excercises. That 0.01% code I wrote which is my github is my CV. That 99.99% I wrote for my last employer won't be seen by my next employer.

You'd think so, but during the interviews I did, there are plenty of candidates to senior positions who just cannot code.

They talk the great talk about problem solving and team work and mentoring, but once it's time walk the walk, they just can't seem to write any code - and we are not talking advanced algorithms, we are talking Fizz-Buzz level.

Maybe it's different for doctors, but as long as such people exist and apply to jobs, we need to ask programming questions.

This also means that if _you_ are applying to a job in a larger team for a senior position and they did not ask you for any coding questions, they either:

- Really good for firing people fast for low performance

- Have people at "senior" position which just give advice and nasty code reviews, but don't actually write any code.

Either way, it's a red flag for the company.

If I'm interviewing for people to work in some specific tech stack or business, it's just as good to see completely different things they can show. Could be a php site for their book club or whatever. But the key is to have some code to discuss that is their code. Because I want to see them describe and reason about code they know, and it works 10x better if it's code they know, rather than some code I present and say "here, let's reason about this code".

Based on my interviewing, there's a ~25% likelihood that you're flat out lying, and a ~75% likelihood that you're overstating your skills.

The number of people with impressive looking resumes, and good command of the jargon, but who still can't successfully code hello world is shocking.

I don't believe you (general you, not just the parent) can even code until you demonstrate that to me, or until the industry comes up with a reputable way to tell me that you can code. Until then I'll be making sure you can at least write code that compiles and finishes a brain-dead simple task.

Work samples are a trap. Real problems are always way more hairy than contrived ones, which is not appreciated by people who were just introduced to the problem 30s ago. Since you wrote it while trying to actually accomplish something rather than polish up a perfect little nugget to impress someone, you probably spent way less time per line.

That's because these three methodologies test different things. Take-home is the best representative, but it's heavily skewed by the prize. Whiteboarding only test for algorithms knowledge and a bit of problem-solving. Paired programming relies more on communication. And the last two add time pressure that rarely exists in real world scenario.

The fact is, it all depends on the person you need and the team they will be slotted in. But it seems that the ones who know the criteria is rarely involved in these matters.

That's a great signal (that you have a hobby playing guitar). If the other half is also interesting, it sounds like a great portfolio.

Why not discuss a problem you have? You're obviously hiring for someone to solve your problem. Looking to check if they can code (in a specific language) seems like the XY problem.

I'm your average skilled introverted engineer. But after more than a dozen years of experience and problem solving, I'd go with #2. I feel i'd be able to explain myself much more easily, have to do much less work, and probably have much more ways to impress the interviewers with face to face. I have also been on the receiving side of take-home tests and I know how hard it is to impress someone with those.

> The problem ends up being, that I don't have any guarantee that the other side will spend any time on it.

This is a key thing for me. I consider it a moral obligation to give material feedback to anyone who does a takehome test for me - to invest at least some serious time evaluating it. Likewise, I would never give a takehome test until the candidate has at least had a phone screen with someone at the company and there's some level of investment on both sides.

On the other hand, I know a few junior devs right now who are submitting resumes and getting sent takehome tests off-the-bat. And, of course, after spending hours on those challenges, they get only form-letter rejections. I understand why companies do that - there's a glut of junior devs right now and any early-career role gets flooded with resumes that you need to somehow pare down - but I still consider it unconscionable.

I understand why you might not want to risk dealing with that.

I second all of that heartily. Of course the senior eng with 10 years hands-on experience at a household name tech company knows how to code... except that they don't. At all, apparently. That seems so ludicrously unlikely unless you've actually interviewed a lot of people, then it's just the sad fact.

To others reading along, I'm not talking about someone not being able to write a multitasking OS on their own. I mean, I'm not sure how these people possibly graduated college not knowing how to write a trivial program in the language of their choosing. Turns out you can get surprisingly far into a software engineering degree without every writing code. I wouldn't have believed it. Evidence proves it though.

Have you ever tried to hire developers? I have, many times. It's shocking how incapable most people with a history of working as senior developers for 10+ years are at basic programming questions. I'm talking about stuff at the level of Fizz Buzz, or barely more difficult, being solved in a language of their choice with ample time.

Through all of the technical interviews I've run on behalf of multiple companies, I've become convinced that our industry is full of imposters. A terrifying number of people who are employed as developers cannot code, beyond making tiny edits to existing code, or slinging StackOverflow answers and AI generated content around. They cannot think for themselves. They rely heavily on their coworkers to make any substantial progress. If anything, modern AIs have made these people more difficult to spot. I've found these imposters within the companies I've worked for too, and if it's not possible to move them to a non-technical role where they can be productive, then it's best to fire them. If that isn't possible, then keep them out of the way of any critical work, but know that they'll continue to be a drag on their team, and will decrease the moral of those on their team who have to keep covering for them.

Once I even worked on a team (as a junior) where only 3 members out of ~16 even knew the language that was exclusively used for the job. The remaining team members accomplished nothing every day. Those stand-up meetings were painful. Drawn out to 45+ minutes of standing in a circle while the unproductive team members found new ways to explain why they haven't made progress. Then I'd see them return to their desks, and not even open their IDEs for weeks at a time. They seemed to keep their jobs because their boss didn't want to fire them, and it cost him nothing personally to keep them around, offering them a sort of corporate welfare. Perhaps having such a large team even helped him politically, but it's unclear if he cared about that since he also spent almost all of his time slacking off, playing games in the break room, while the 3 productive team members carried the rest of the team.

Because they aren't familiar with or enthusiastic about the code I give them, they could be about code they have written. It's the classic "describe one time you did something poorly/well"-question, but with code. I want to see them critique some code, or explain what's so great, or why it ended up the way it is.

Reasoning about a problem i have (even showing some code) is also a good part of an interview. But it's my side of the field. I just found it's much better to move the interview to the interviewee's side of the turf, because they are more comfortable there.

> A conversation will tell you much more about their approaches to problems solving, team work, and mentoring than a coding challenge will.

Having hired many developers over multiple decades, let me assure you that you should definitely make sure they can actually code. Do definitely have those conversations, too, but trust me when I say that a technical conversation, a degree and several listed years of experience as a software developer on a resume are not enough to guarantee that they can actually code.

I only had one code sample.

As others have pointed out, there are a lot of problems:

- Many engineers don't write code outside of work and so don't have much code they can show off without breaching their employer's trust

- Those that do often don't have recent code.

- Those that have recent, personal code to show off have often written it to accomplish some goal and the code isn't necessarily a great example they feel like showing off

Really, the only people I've seen have good code samples are those who do extensive open-source work. And for those people, I'm probably already aware of that work when I checked their resume + opened their github.

Because all evidence[0] I've found has shown that work-sample tests and repeatable, structured indicators are the best indicator of job performance. I understand that a lot of devs think they can get a feel for a candidate by just having a good ol' conversation with them, but the body of study on the subject says that that's just wrong. And so I have people do a task that's as close to the actual job as humanly possible and evaluate them on that.

https://www.researchgate.net/publication/283803351_The_Valid...

Yeah, that was the explicit tradeoff: I'm losing the ability to compare apples-to-apples. I decided it was worth the risks associated with that - that I'd wind up with two candidates who picked different options and I couldn't decide between them because I got different signal between them. As it turned out, it didn't really come up - nearly everyone chose the takehome. On a larger scale, though, you'd definitely have to grapple with that.

That's why I offered the choice: different people do well under different circumstances. I figured that it'd allow me to hire a great dev who hates timed challenges or a great dev that hates take-home busywork. In practice, though, everyone like the take-homes, so ¯\_(ツ)_/¯

As for feeding it into an LLM, I go back and forth on that. With their current capabilities, the project is slightly too complex for that to work. You could get a lot of help from an AI, but you still have to put the pieces together yourself. And I'm fine with that! If AI tooling makes you a faster/better dev on a test, I'd expect it to do the same in real work.

Longer-term, though, I worry that LLMs still won't be able to "write the whole thing" for real work, but will be able to "write the whole thing" for takehome tests. And as such, I'll have to figure out how to handle that.

I would like to just be asked. I can't post my code on github but I can talk about my projects at length.

I appreciate it. After thinking about this I think I just need to get over my hangups about duplicating things already done or things that aren't really important.

I just take tabs and rearrange them for personal use. Learn from my mistake and just post whatever to your github, if you fret over its usefulness or purpose you might just never grow your portfolio at all. This was a mistake. You don't have to have some gnu front-page exploratory project.

If you want to shred guitar look up Troy Grady to grok efficient mechanics that won't break your wrist (we type alot for work as well)

> Take home tests or whiteboard tests or whatever are ok to use as a filter, but people aren't points along a one-dimensional "quality" line.

They are unless you're going to hire everyone who ever applies. At some point, you're choosing between two candidates, and the only way you can make that choice is by projecting them onto a one-dimensional line. The approach you describe:

> stop thinking about how they might fail and start thinking about what it would look like if they succeed. They'll have different strengths and advantages.

is one method of doing that, but not an especially effective one.

For me problem word I see here is „impress”.

When I hire I don’t need someone to impress me, he has to present himself as able to do the job.

I think a lot of devs think they have to hire someone that impresses them and we end up with insane tests and adversarial interviews.

Yeah of course there will be people who will blow through the task much faster than others with better quality than average and they will impress me and they will get the offer in first place - but often times they already get multiple offers and after negotiation they will pick some higher offer and we have to get someone we can afford to do the job.

For all this thread solution is simple. Small take home assignment where you write the code to discuss during the interview.

I know people are vocal about not wanting to do take homes. But if take home is reasonable and used as a talking piece it checks all the boxes for good tool.

I think in reality I had single person that outright refused and of course bunch of people who didn’t bother to deliver - great candidates delivered it the same day, busy great candidates delivered it over the weekend.

That sounds like a really challenging situation, and I can see how it would be disheartening to work in an environment like that. It makes me wonder if part of the problem lies in how companies approach technical growth and team dynamics. For example, instead of just identifying unproductive team members, do you think those people could have benefitted from structured mentorship or training programs? Or maybe clearer performance metrics that highlight where they’re struggling?

It’s also interesting to think about how managers could be supported better. If your old boss had been more engaged, perhaps they could’ve restructured the team or helped those who were falling behind find roles that suited their strengths better. Have you ever seen a company handle this type of situation well, or is this kind of dysfunction just too common?

I have seen team members fall below expectations, or burnout, or otherwise become unproductive, but then recover. In every case, their manager was a key part of the solution (and sometimes a key part of the original problem). If a person is really stuck on their current work, it can be demoralizing to have it taken away and assigned to others, but it can also be a relief. One thing that I've seen help is to let devs be more self-directed. Try to nudge them towards things that are important for the business, but focus on doing that by really convincing them of the value of that work. If they don't want to do that work, try to understand why, and consider changing plans based off of that.