←back to thread

The FAA’s Hiring Scandal

(www.tracingwoodgrains.com)
739 points firebaze | 2 comments | | HN request time: 0.485s | source
Show context
apical_dendrite ◴[] No.42950494[source]
I'm fairly certain this was an example of overfitting and Freedman's Paradox, not deliberate cheating.

Let's say you have a completely random data set. You generate a bunch of random variables x1 through xn and a random dependent variable y. Then you poke around and see whether any of the x variables look like they might predict y, so you pick those variables and try to build a model on them. What you end up with is a model where, according to the standard tests of statistical significance, some of the xs predict the y, even though all the data is completely random.

This is a much more likely explanation for why the answer weights on the biographical assessment were so weird than some conspiracy between the contractors who developed the test, the FAA staff, and the black employee organization.

They had a dataset that was very skewed because historically there have been very few black controllers, and so was very prone to overfitting. The FAA asked the contractor to use that dataset to build a test that would serve as a rough filter, screen in good candidates, and not show a disparate impact. The contractor delivered a test that fulfilled those criteria (at least in the technical sense that it passed statistical validation). Whether or not the test actually made any sense was not their department.

replies(4): >>42950888 #>>42950950 #>>42953610 #>>42954852 #
tim333 ◴[] No.42954852[source]
Have you looked at the info on the test here https://kaisoapbox.com/projects/faa_biographical_assessment/? (copied from another post)

To pass the test you have to click A on all 62 questions apart from question 16 where you have to click D to say your lowest grade in school was in history. The thing's a complete travesty.

replies(2): >>42956772 #>>42959086 #
1. tstrimple ◴[] No.42956772[source]
Did you get a different question/answer than I did? For me it showed science as the only correct answer awarding 15 points.
replies(1): >>42957764 #
2. flak48 ◴[] No.42957764[source]
it's science for school and history for college