My 16-month theanine self-experiment

Personally, I have not published any results yet, but I have been doing this type of experiments for 4 years now. And collected 48874 data points so far. I built a simple system to do it in Vim:

https://www.gibney.org/a_syntax_for_self-tracking

I also built a bunch of tooling to analyze the data.

I think that mankind could greatly benefit from more people doing randomized studies on their own. Especially if we find a way to collectively interpret the data.

So I really applaud the author for conducting this and especially for providing the raw data.

Reading through the article and the comments here on HN, I wish there was more focus on the interpretation of the experiment. Pretty much all comments here seem to be anecdotal.

Let's look at the author's interpretation. Personally, I find that part a bit short.

They calculated 4 p-values and write:

    Technically, I did find two significant results.

Then they continue:

    Of course, I don’t think this
    means I’ve proven theanine is harmful.

It's great that they offer the raw data. I look forward to taking a look at it later today.

Too much other stuff is changing in a single persons life that could account for all observed side effects.

You also have latent side effect issues. A person could smoke for 10 years, not smoke for another 10, and then conclude that smoking doesn’t cause cancer. Then they get lung cancer 20 years later.

Excellent data and statistics is not sufficient for a good experiment

It's a bit cumbersome, but not too much. But yeah, I look forward to when I finally get around to writing an app for it.

It depends on the setup of the experiment.

Imagine an experiment where a person's thumb gets randomly hit with either a hammer or a feather once per day. And they then subjectively rate the experience. After 1000 days of collecting data, I doubt that we would wrongly come to the conclusion that the hammer treatment leads to the nicer outcome.

The setup of the Theanine experiment which is the basis of this thread looks good on first sight. I have the feeling that the interpretation could use more thought though.

I think what the author is saying is that for them to bother with theanine on a permanent basis, it would have to have shown an effect large enough to be apparent just from plotting.

In other words, they mean technical significance as opposed to clinical significance. A small effect can be statistically verifiable without being meaningful in practice.

    an effect large enough to be
    apparent just from plotting

But before we take the measured effects at face value, I think it's important think about them more. They report significant p values of their success in predicting if the capsule holds Theanine and also for the effect of the capsule when it holds Theanine. Both negative correlations. My first thought reading this is that the Placebo tasted more like Theanine and thinking they took Theanine had a positive effect on the outcome.

That is the point I am making: Experiments of a single person can be useful.

The critics of single person experiments usually come up with examples vastly different than the Theanine experiment described here. With long term experiments which are only conducted once. But the Theanine experiment was looking for a short term effect and can be conducted many times. The hammer experiment I made up would be an extreme example of this type of experiment which leads itself well to be conducted by a single person.

What I am trying to point out is that if you are a skeptic, it would be better to try and find weaknesses in the experiment at hand. Not making up completely different experiments.

Or “not using a one-way t test”.

The most appropriate null hypothesis in this lovely study is “does theanine REDUCE anxiety”, not “does theanine change anxiety either up or down”.

What impressed me most is the suggestion for an improved experimental design to remove his temporal drift by using 100 pre-loaded envelopes and only decoding the results at the end.

I disagree with this. You have a prior belief that theanine might reduce anxiety; if you wanted to you could codify that subjective belief and perform some variety of Bayesian hypothesis test [1] and compute a Bayes factor. The main reason that one-sided tests are advocated for is power; that is often the same as having a prior belief in disguise. Why not quantify it?

However, scientifically, if the data conclusively show that "theanine increases anxiety" that is a meaningful, non-artefactual result: it is hugely important to be sensitive to the answer 'you are wrong' and may well ironically spur development in a direction to help understand what is going on. I personally think that one sided tests are best avoided except in the case where it is physically impossible to have an effect in the other direction. Examples of this are rare, but they do occasionally exist.

[1] https://mspeekenbrink.github.io/sdam-book/ch-Bayes-factors.h...

In my opinion this is an exemplary N=1 study that is well designed and thoughtfully executed. Deserve accolades, not derision. And the author even recognizes possible improvements.

Unlike most large high N clinical trials this is a high resolution longitudinal trial, and it is perfectly “controlled” for genetic difference (none), well controlled for environment, and there is only one evaluator.

Compare this to the messy and mostly useless massive studies of human depression reviewed by Jonathan Flint.

https://pubmed.ncbi.nlm.nih.gov/36702864/

It is great if you can solve things with diet, mindfulness, etc. But sometimes you need medical intervention and yes sometimes that means you need to take medication in the form of pills. There are millions of people who need that regardless of how they change their lifestyle or regulate their emotions/mental health without them.

Basically I don’t like the idea that you are implying medication is a bandaid and not ever the actual solution. If I misreading your comment my apologies

TFA describes a protocol for doing just that. The author randomly selected between the treatment dose and placebo. They didn't reveal the choice until after the effect should be complete so they could record the results.

Anecdotal data, n=1 trials of varying quality, correlations studies, double blind studies (with small and large cohorts), studies without attempted replication and studies with heavy replication - they are all provide evidence of varying quality and can inform the holistic scientific picture. They can all serve a purpose such as inspiring further research, providing fodder for meta analyses, etc. It simply isn’t true that gathered evidence ought to be casually discarded if it doesn’t attain the highest levels of the hierarchy of evidence. Neither is it true that some small study showing (or not showing) some supposed effect should drastically change all our lifestyle habits. The truth lies somewhere in the middle. The concept of a hierarchy of evidence can help us navigate these apparently mixed signals so prevalent in popular science discussions.

In Vim, I use multiple shortcuts that I made for this use case.

I've been using theanine for a long time, but never for any of these purported benefits. And the benefits I do use it for would be near impossible to measure. I just use it to make a over-caffinated monkey brain state tend toward a "lock-in" mental state. That's super hard to measure, and just as likely the theanine is a trigger for a mental deep dive that could just as well be sugar. But the ritual works, and that's what's important to me. It just took the intial "It helps mellow out caffeine for deep focus" idea to establish the ritual.

Science? No. Effective? Yeah, I think so.

Now you can argue that there isn’t enough time between samples, or he needs more subjects but he was blind to whether he was taking it that day or not.

We should avoid extreme polarization of our judgments in general. The study deserves some amount of praise for things it did somewhat well (like the method of blinding which is clever, but not applicable to everyone), and criticism for things it did not do well, such as designing your own study methodology for your own mood. That alone will affect the results. Simply RUNNING an experiment can affect your mood because it's interesting (or even maybe frustrating). The subject probably felt pride and satisfaction whenever they used their pill selection technique, which could improve mood on its own. Neither accolades nor complete derision are appropriate, although trying to claim too strong a result from this study is kinda deserving of derision if you claim to be science-minded.

The study was well-meaning and displayed cleverness.

Effect size is the key element of this question. If the substance was alprazolam (Xanax) instead of theanine, we would almost certainly see a strong effect here. The same would be true for heroin, ethanol, or cocaine.

It's not trying to test whether there might be a strong effect for most people or whether there are any side effects. Other experiments have been done with theanine seeking to answer those questions; the answers appear to be no and no.

It is clearly a step forward from what you can watch about theanine on YouTube or TikTok. I consider this a work of citizen science. While it should not be taken for more than it is, it’s a great example of how someone can experiment without a high burden.

Alternatively, may I suggest the DailyLog app in F-Droid?

The nuanced truth is that our medical industry is flawed; since most illnesses are defined by their set of apparent symptoms (with most root causes not fully understood) the standard approach is to treat "symptoms" with medicines with sides effects rivaling the original ailment!

What if the lack of proper diet ('proper' varies wildly with populations), drugs/alcohol, sleep, stress, and exercise were the originating cause? Rushing to the medication treatment without fixing those vitals first eliminates the opportunity.

Our bodies have much more adaptive self-healing resilience properties innate to our species development than our own species hubris seems to acknowledge. And most people severely underestimate the importance big four.

The entire blog post was anecdotal.

Also, you asked why he said "unblinded", and I think you now have the answer to that.

Comments like yours expose a particularly distasteful amount of hubris.

Neither of those is a hypothesis, which require a prediction not just a question.

The null hypothesis for this experiment would be "Theanine has no effect on stress".

How else are you going to find out whether a particular diet or medication works for you specifically? It's ALWAYS N=1.

>While I was blinded during each trial, I saw the theanine/D result when I wrote it down. Over time I couldn’t help but notice that my stress dropped even when I took vitamin D, and that I was terrible at predicting what I’d taken

That is not blinding

Given that there is no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location (quiet island or next to a construction site), etc into account, the data seems useless.

As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment?

Perhaps the correct course of action would be to ask for feedback in the design phase of an N=1 trial, especially a longer one, to avoid some basic mistakes.

It’s unblinded because the subject is preparing the concoction under study. There is no way they can create a blind experiment if they’re the ones preparing the control. The placebo effect is nothing if not pernicious and cunning, able to exploit even the most subtle psychological signal - like minuscule differences in the amount of powder in a capsule.

Blinded studies have independent doctors prepare and dispense the candidate drug so they know whether its the real thing or a placebo, but their patients dont. In double blinded studies, neither the doctor nor the patient have any idea about what they’re getting because a third party prepares the drugs.

It gives a hint of how well the blinding worked.

Sure, OP addressed that and said it would be especially useful more people did it and "if we find a way to collectively interpret the data".

Citizens scientists rarely post “facts” but rather interesting avenues for research or further investigation. Part of being an educated citizen scientist is to understand the limitations of your knowledge, data, and methods.

Quacks and cranks, on the other hand, are always making grand new “discoveries” lol.

Anyone who has a decent education can make observations apply the scientific method. I say this coming from a family of actual scientists from molecular biology to particle physics, who will tell you the same, and also give credit to the multitude of citizen scientists who have done just that.

As for myself, I’ll stick to engineering.

Re-reading such a negative critique of a solid home-brew experiment is unwarranted. There are several word here worth red flags.

>This is an N=1 trial. Dressing your N=1 trial up with lots of pseudo controls and pseudo blinding and data collection does not make it better. In fact: putting this much effort into any medication trial makes it much more likely that you’re going to be incentivized to find effects that don’t exist. I think it’s nice that the author admits that they found nothing, but statistically, worthless drugs show effects in much better-designed trials than this one: it’s basically a coin toss.

> no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location

Assuming the blinded samples are uniformly randomly distributed, and assuming the study goes on long enough, then you'd expect that stuff to average out.

But I agree, it should be recorded nonetheless. That way you can verify at the end that it did, in fact, average out as you expected. If it didn't then your data is invalid.

but this isn't the case! i linked in an answer to my post what science knows when measuring stress... it's far away from a self-reported subjective question

i think the problem is whatever you post on the internet is politic. it may have a huge reach, and so far so good people trying far from lethal doses of theanine. now, what if an influencer wants to make their stuff look intellectual and go buy pure caffeine? how about the people coming after wards?

one thing is a 3D-print project at your garage on keyboards ergonomics, home automation, another is rock-climbing gear and another is substances with reachable lethal doses where the masses can buy (not the case here but again, i cited caffeine but there's much more). the author has a stellar presentation but they seriously researched what science does when measuring stress? they didn't even considered The Perceived Stress Scale (PSS), which is used on professional research. i stand against public posts with badly methods (or the complete lack of research) suppressed by fancy graphics on substance use/abuse. i also bet isn't that hard to buy empty pills with opaque color

> You also have latent side effect issues. A person could smoke for 10 years, not smoke for another 10, and then conclude that smoking doesn’t cause cancer. Then they get lung cancer 20 years later.

But that's not the actual experiment here. Your criticism is invalid. The author randomised every day which supplement to take.

That's pretty close to a blocked design.

I think most people could criticize the carbon out of a corpse if they themselves weren't being criticized into one.

If we devolved from apes, maybe apes devolved from piranhas.

The New Yorker's paywall has successfully obscured the origin of the joke, so that's on them, as far as I'm concerned.

Well, instead of an actual web link, you could just mention that it's from the New Yorker.

The problem is that he is comparing two very different things, the level when he took the pill and the level one hour after that. So it's not surprising that they are different. Let's imagine a very very very stupid experiment, where the problem is more obvious.

Does Coca Cola or Pepsi improve luck? N=1000000, double blind randomized controlled trial.

1) Each subject flips a coin. tail=0, head=1.

2) They drink a glas of soda, 50% Coke or 50% Pepsi, that is served in a hidden place and nor the subject or the experimenter know which one.

3) They roll a dice (an usual one, D6)

Results:

* Average before Coke = 0.5002

* Average after Coke = 3.5005

* Average before Pepsi = 0.5004

* Average after Pepsi = 3.5003

So the conclusion is that Coke improves the average (p<1E-a-lot) and Pepsi improves the average (p<1E-a-lot). Both are "technically" statistically significant (but it's caused by a horrible experiment design).

Unsurprisingly, the difference in the average after drinking Coke or Pepsi is not statistically significant (p<.something).

(I'm too lazy to run a simulation now, but it's not difficult to get realistic averages and p values.)

In conclusion, the useful result is the comparison of the anxiety after taking both drugs, not the difference of before and after taking them.

As the article says:

>> So I propose a new rule: Blind trial or GTFO.

(From a vaguely-remembered 1990s-era ad campaign that I thought was excruciatingly self-indulgent at the time, but which evidently worked.)

Some version of that still exists among people with a naturalist bent recording observations of reptiles, birds, plants, etc. But yeah, we don't really have backyard chemists analyzing things anymore.

This is only an appropriate criticism in so far as you want to make conclusions about theanine as an intervention in the broader population.

It is however, perhaps much BETTER than large N trials if the author wishes to draw conclusions about how theanine affects THEM.

Another interesting aspect of citizen science is replication of existing scientific research, Often with experimental modifications that make the experiment much more approachable for amateurs. Sometimes this even leads to process improvements that facilitate industrial application.

That would not be a problem regarding the averaging I referred to, although it could well pose a problem for measurement depending on how it interacted with the selected metrics.

Note that the averaging I refer to is not regarding all possible values of some metric, but rather any discrepancy in the distribution of metrics which we expected to follow the same distribution between the sample and the control.

I think maybe there's a misunderstanding? It seems that we both agree that a variety of additional variable should be logged. I was not suggesting to omit them, but rather to use discrepancies in them to detect fundamental issues with the data or study design. I would also expect larger studies to do the same where possible.

At 94 data points it is entirely possible that there would be outliers that would have averaged out for a larger N but did not. In such a scenario the presence of such outliers should then be taken to indicate a problem with the data (ie the more discrepancies you observe, the less you should trust the data).

I of course agree that logging them is basic scientific methodology - in order to detect issues with the experiment, and even hopefully to see the signal through the noise.

Loved reading this as I have been on a similar self tracking journey of my own.

I am curious about your thoughts on mobile support? Specifically, have you implemented anything to make adding/editing events while away from your computer easier? Or perhaps you feel it is not important for your use case?

Personally, I've found that being able to log an event in an instant, or just on a whim, is invaluable for capturing the sort of data I care about tracking the most (like what I just ate or my mood), and I find too much friction causes many events simply go unlogged as they almost always feel too unimportant or mundane in retrospect to keep a mental note of and track later.

I started out with a text file system similar to yours but I've since begrudgingly resorted to google forms for most of my tracking needs. It's nice that I can arbitrarily add and remove fields while still keeping everything relatively structured and parseable, and it does have (albeit inferior) revision history. But I dislike not having ownership and control over where my data is stored and absolutely *dread* waiting for the form to load every time I want to log an event, especially on mobile.