When you don't have any adrenal stimulants in your diet at all, even a small amount is noticable.
Caffeine also has a metabolic half life of roughly 5 hours in the body, if I remember correctly. A few berries might not do much, but surely a handful will be enough.
Here is an article that further describes how we believe caffeine synthesis evolved in multiple land plant lineages.
https://www.pnas.org/content/113/38/10613
Caffeine synthesis involves several enzymes, but the enzyme family (called the SABATH family) involved in the final stages of its synthesis can trace its origins back to the first land plants. These first enzymes are thought to have been very promiscuous (capable of having activity with several molecules), partially contributing to how caffeine synthesis managed to evolve independently multiple times throughout the evolutionary history of land plants.