←back to thread

437 points Vinnl | 1 comments | | HN request time: 0.237s | source
Show context
explodes ◴[] No.43984193[source]
Wouldn't it be nice if policy changes were accompanied by an A/B testing plan to evaluate their impact? I have always thought so. I have also seen a major pitfall of A/B testing that real humans can hand-pick and slice data to make it sound as positive or negative as wanted. Nonetheless, the more data the better.
replies(9): >>43984911 #>>43984957 #>>43985195 #>>43985261 #>>43985635 #>>43986010 #>>43986433 #>>43990930 #>>43996358 #
sc68cal ◴[] No.43985261[source]
We already had A/B testing of congestion pricing. The A test was without congestion pricing in NYC, and has been tested for decades.
replies(3): >>43985550 #>>43985682 #>>43990138 #
bunderbunder ◴[] No.43985550[source]
That's not an A/B test because it has no way of controlling for broader economic trends over time. How do you figure out if what you're seeing is because of that one thing that changed, or the enormous list of other things that also changed around the same time?

A more valid design would be randomly assigning some cities to institute congestion pricing, and other cities to not have it. Obviously not feasible in practice, but that's at least the kind of thing to strive toward when designing these kinds of studies.

replies(3): >>43985801 #>>43991066 #>>43994453 #
1. sorcerer-mar ◴[] No.43994453[source]
Everyone knows how you can conduct good experiments in a land of frictionless spherical cows.