←back to thread

437 points Vinnl | 3 comments | | HN request time: 0.608s | source
Show context
explodes ◴[] No.43984193[source]
Wouldn't it be nice if policy changes were accompanied by an A/B testing plan to evaluate their impact? I have always thought so. I have also seen a major pitfall of A/B testing that real humans can hand-pick and slice data to make it sound as positive or negative as wanted. Nonetheless, the more data the better.
replies(9): >>43984911 #>>43984957 #>>43985195 #>>43985261 #>>43985635 #>>43986010 #>>43986433 #>>43990930 #>>43996358 #
aclatuts ◴[] No.43986433[source]
The real world isn't A/B tests. No government is going to spend millions on equipment and infrastructure on a congestion zone because some engineers are like "Let's just test this out. I have done zero research on what could possibly happen, but it would be fun to see what the results are."
replies(1): >>43989611 #
1. listenallyall ◴[] No.43989611[source]
When you write it out like that, it seems to make total sense! But then you read grant proposals that get funded - in things like the social sciences and humanities, and even conventional science and health - millions of dollars essentially just throwing darts to see what sticks.
replies(1): >>43989863 #
2. czzr ◴[] No.43989863[source]
Surely you see the difference between working in a development environment and working in production?
replies(1): >>43992193 #
3. listenallyall ◴[] No.43992193[source]
The comment to which I replied was referring to the cost, not to the implementation