←back to thread

My 16-month theanine self-experiment

(dynomight.net)
817 points dynm | 2 comments | 09 Mar 25 03:08 UTC | HN request time: 0.546s | source
Show context
mg ◴[09 Mar 25 08:26 UTC] No.43307263[source]▶
This is great. The author defines their own metrics, is doing their own A/B tests and publishes their interpretation plus the raw data. Imagine a world where all health blogging was like that.

Personally, I have not published any results yet, but I have been doing this type of experiments for 4 years now. And collected 48874 data points so far. I built a simple system to do it in Vim:

https://www.gibney.org/a_syntax_for_self-tracking

I also built a bunch of tooling to analyze the data.

I think that mankind could greatly benefit from more people doing randomized studies on their own. Especially if we find a way to collectively interpret the data.

So I really applaud the author for conducting this and especially for providing the raw data.

Reading through the article and the comments here on HN, I wish there was more focus on the interpretation of the experiment. Pretty much all comments here seem to be anecdotal.

Let's look at the author's interpretation. Personally, I find that part a bit short.

They calculated 4 p-values and write:

    Technically, I did find two significant results.
I wonder what "Technically" means here. Are there "significant results" that are "better" than just "technically significant results"?

Then they continue:

    Of course, I don’t think this
    means I’ve proven theanine is harmful.
So what does it mean? What was the goal of collecting the data? What would the interpretation have been if the data would show a significant positive effect of Theanine?

It's great that they offer the raw data. I look forward to taking a look at it later today.

replies(14): >>43307304 #>>43307775 #>>43307806 #>>43307937 #>>43308201 #>>43308318 #>>43308320 #>>43308521 #>>43308854 #>>43309271 #>>43310099 #>>43320433 #>>43333903 #>>43380374 #
1. kqr ◴[09 Mar 25 10:56 UTC] No.43307937[source]▶
> So what does it mean? What was the goal of collecting the data? What would the interpretation have been if the data would show a significant positive effect of Theanine?

I think what the author is saying is that for them to bother with theanine on a permanent basis, it would have to have shown an effect large enough to be apparent just from plotting.

In other words, they mean technical significance as opposed to clinical significance. A small effect can be statistically verifiable without being meaningful in practice.

replies(1): >>43308016 #
2. mg ◴[09 Mar 25 11:11 UTC] No.43308016[source]▶

    an effect large enough to be
    apparent just from plotting
And how large is that? Without putting a number on it, how do we come to the conclusion that the effect is not large enough? That it didn't show in their sample of data points could have been just random chance.

But before we take the measured effects at face value, I think it's important think about them more. They report significant p values of their success in predicting if the capsule holds Theanine and also for the effect of the capsule when it holds Theanine. Both negative correlations. My first thought reading this is that the Placebo tasted more like Theanine and thinking they took Theanine had a positive effect on the outcome.