My 16-month theanine self-experiment

This is great. The author defines their own metrics, is doing their own A/B tests and publishes their interpretation plus the raw data. Imagine a world where all health blogging was like that.

Personally, I have not published any results yet, but I have been doing this type of experiments for 4 years now. And collected 48874 data points so far. I built a simple system to do it in Vim:

https://www.gibney.org/a_syntax_for_self-tracking

I also built a bunch of tooling to analyze the data.

I think that mankind could greatly benefit from more people doing randomized studies on their own. Especially if we find a way to collectively interpret the data.

So I really applaud the author for conducting this and especially for providing the raw data.

Reading through the article and the comments here on HN, I wish there was more focus on the interpretation of the experiment. Pretty much all comments here seem to be anecdotal.

Let's look at the author's interpretation. Personally, I find that part a bit short.

They calculated 4 p-values and write:

    Technically, I did find two significant results.

I wonder what "Technically" means here. Are there "significant results" that are "better" than just "technically significant results"?

Then they continue:

    Of course, I don’t think this
    means I’ve proven theanine is harmful.

So what does it mean? What was the goal of collecting the data? What would the interpretation have been if the data would show a significant positive effect of Theanine?

It's great that they offer the raw data. I look forward to taking a look at it later today.

This is an N=1 trial. Dressing your N=1 trial up with lots of pseudo controls and pseudo blinding and data collection does not make it better. In fact: putting this much effort into any medication trial makes it much more likely that you’re going to be incentivized to find effects that don’t exist. I think it’s nice that the author admits that they found nothing, but statistically, worthless drugs show effects in much better-designed trials than this one: it’s basically a coin toss.

Complete injustice to this lovely study. Why do you say unblinded? Why do you insult a time series study as “dressing up with lots of data”? Would you rather see less data? Or are you volunteering to be test subject #2? Show us how to do it right Dr. M.!

In my opinion this is an exemplary N=1 study that is well designed and thoughtfully executed. Deserve accolades, not derision. And the author even recognizes possible improvements.

Unlike most large high N clinical trials this is a high resolution longitudinal trial, and it is perfectly “controlled” for genetic difference (none), well controlled for environment, and there is only one evaluator.

Compare this to the messy and mostly useless massive studies of human depression reviewed by Jonathan Flint.

https://pubmed.ncbi.nlm.nih.gov/36702864/

Because....

>While I was blinded during each trial, I saw the theanine/D result when I wrote it down. Over time I couldn’t help but notice that my stress dropped even when I took vitamin D, and that I was terrible at predicting what I’d taken

That is not blinding

I would have taken a well-calibrated photo of the cup each time without looking, maybe with a color card in the bottom, and only entered results at the end of the trial.

Given that there is no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location (quiet island or next to a construction site), etc into account, the data seems useless.

As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment?

Perhaps the correct course of action would be to ask for feedback in the design phase of an N=1 trial, especially a longer one, to avoid some basic mistakes.

If your sample is blinded yet you consistently guess correctly, then presumably either 1. you failed at blinding or 2. there is a strong and discernible effect, regardless of what your other metrics might say (your other metrics could always be flawed after all).

> no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location

Assuming the blinded samples are uniformly randomly distributed, and assuming the study goes on long enough, then you'd expect that stuff to average out.

But I agree, it should be recorded nonetheless. That way you can verify at the end that it did, in fact, average out as you expected. If it didn't then your data is invalid.

He has 94 data points. Not nearly enough to average out so many potentially confounding variables, and there is no way to know they would. That will be the case in almost all N=1 experiments. Perhaps he takes the pill at the onset of stress, and stress almost always tends to build afterwards. This would be a probable case for many people trying to use theanine in this way. I could never accept that we should presume it would average out, and thus I consider logging potentially confounding variables essential to a valid experiment of this type of uncontrolled experiment. Your hypothesis would be much more relevant in a larger experiment of course.