Claude 3.7 Sonnet and Claude Code

I like to make up my own tests, that way you know it is actually thinking.

Tests that require thinking about the physical world are the most revealing.

My new favourite is:

You have 2 minutes to cool down a cup of coffee to the lowest temp you can.

You have two options: 1. Add cold milk immediately, then let it sit for 2 mins.

2. Let it sit for 2 mins, then add cold milk.

Which one cools the coffee to the lowest temperature and why?

Phrased this way without any help, all but the thinking models get it wrong

The fact that the answer is interesting makes me suspect that it's not a good test for thinking. I remember reading the explanation for the answer somewhere on the internet years ago, and it's stayed with me ever since. It's interesting enough that it's probably been written about multiple times in multiple places. So I think it would probably stay with a transformer trained on large volumes of data from the internet too.

I think a better test of thinking is to provide detail about something so mundane and esoteric that no one would have ever thought to communicate it to other people for entertainment, and then ask it a question about that pile of boring details.

Out of curiosity, what is the answer? From your comment, it seems like the more obvious choice is the incorrect one.

EDIT: By the more obvious one, I mean letting it cool and then adding milk. As the temperature difference between the coffee and the surrounding air is higher, the coffee cools down faster. Is this wrong?

No need for thinking, that question can be found discussed and explained many times online and has almost certainly been part of the training data.

Simple Bench goes in this direction: https://simple-bench.com/

That is the correct answer. Also there is a lot of potential nuance, like evaporation or when you take the milk out of the fridge or the specific temperatures of everything, but under realistic settings adding the milk late will get you the colder coffee.

> Phrased this way without any help, all but the thinking models get it wrong

I C&P'd it into Claude 3.7 with thinking, and it gave the correct answer (which I'm pretty sure is #2). Including the CoT, where it actually does math (which I haven't checked), and final response.

# THINKING

Let's analyze the two options.

Option 1: Add cold milk immediately, then let it sit for 2 mins.

Option 2: Let it sit for 2 mins, then add cold milk.

To determine which option leads to the lowest final temperature, I need to consider the physics of cooling.

First, let's understand how cooling works in these scenarios:

1. When a hot liquid sits, it cools according to Newton's Law of Cooling, which states that the rate of heat loss is proportional to the temperature difference between the object and its surroundings.

- The rate of cooling is faster when the temperature difference is greater

- As the liquid cools, the rate of cooling decreases

2. When we add cold milk to hot coffee, we get an immediate temperature drop due to mixing liquids of different temperatures. The final temperature after mixing can be calculated using the principle of thermal equilibrium.

Now, let's analyze each option:

Option 1: Add cold milk immediately, then let it sit for 2 mins

- The coffee temperature drops immediately when milk is added

- Then this coffee+milk mixture cools for 2 minutes

- The cooling rate will be based on the temperature difference between the coffee+milk mixture and room temperature

Option 2: Let it sit for 2 mins, then add cold milk

- The black coffee cools for 2 minutes

- Then milk is added, causing an immediate drop in temperature

- The cooling rate during the 2 minutes will be based on the temperature difference between the black coffee and room temperature

The key insight is that hot coffee cools faster than warm coffee because the rate of cooling is proportional to the temperature difference. Since black coffee starts at a higher temperature than coffee with milk, it will lose more heat energy during the 2-minute cooling period.

Let's make this more concrete with some theoretical numbers:

- Let's say black coffee starts at 90°C

- Room temperature is 20°C

- Cold milk is at 5°C and makes up 10% of the final volume

Option 1:

- Adding milk immediately might bring the temperature to about 81.5°C (90 × 0.9 + 5 × 0.1)

- This mixture then cools for 2 minutes from 81.5°C to some lower temperature, let's call it T1

Option 2:

- The black coffee cools for 2 minutes from 90°C to some temperature, let's call it T2

- Then we add milk, which further reduces the temperature to 0.9 × T2 + 0.1 × 5

The question is: which is lower, T1 or 0.9 × T2 + 0.1 × 5?

According to Newton's Law of Cooling, the black coffee in Option 2 will lose more heat during the 2 minutes because it starts at a higher temperature and thus has a greater temperature difference with the surroundings. This means the drop from 90°C to T2 will be greater than the drop from 81.5°C to T1.

Let's think about this mathematically:

According to Newton's Law of Cooling: T(t) = Ts + (T0 - Ts) × e^(-kt) Where:

- T(t) is the temperature at time t

- Ts is the surrounding temperature

- T0 is the initial temperature

- k is the cooling constant

For Option 1:

T1 = 20 + (81.5 - 20) × e^(-k × 2) = 20 + 61.5 × e^(-2k)

For Option 2, before adding milk:

T2 = 20 + (90 - 20) × e^(-k × 2) = 20 + 70 × e^(-2k)

After adding milk in Option 2, the temperature would be:

0.9 × T2 + 0.1 × 5 = 0.9 × (20 + 70 × e^(-2k)) + 0.5

= 18 + 63 × e^(-2k) + 0.5

= 18.5 + 63 × e^(-2k)

Now we compare:

T1 = 20 + 61.5 × e^(-2k)

Final temp in Option 2 = 18.5 + 63 × e^(-2k)

For any positive value of k (which it must be for cooling), e^(-2k) is between 0 and 1.

If e^(-2k) = 1 (no cooling at all):

T1 = 20 + 61.5 = 81.5

Final temp in Option 2 = 18.5 + 63 = 81.5 (They would be the same)

If e^(-2k) = 0 (instant cooling to room temperature): T1 = 20 Final temp in Option 2 = 18.5 (Option 2 would be better)

For values in between, we need to compare 20 + 61.5 × e^(-2k) vs 18.5 + 63 × e^(-2k)

Subtracting: (T1 - Final temp in Option 2) = 1.5 - 1.5 × e^(-2k)

This will be positive (meaning T1 is higher) when e^(-2k) < 1, which is true for all positive k.

So, mathematically, Option 2 (letting it sit, then adding milk) results in a lower final temperature.

This makes physical sense because:

1. The hotter black coffee loses more heat during the 2 minutes than the cooler coffee+milk mixture would

2. The cold milk provides the same cooling effect whether added at the beginning or end

Therefore, Option 2 (letting the coffee sit for 2 minutes, then adding cold milk) will result in the lowest final temperature.

# ANSWER

# Coffee Cooling Analysis

## Option 2 wins: Let it sit for 2 mins, then add cold milk

This will give you the lowest final temperature, and here's why:

### The physics behind it:

1. *Newton's Law of Cooling*: The rate at which something cools is proportional to the temperature difference between the object and its surroundings.

2. *Key insight*: Hotter objects cool faster than cooler ones because the temperature gap with the environment is larger.

### Comparing the options:

*Option 1 (Add milk first, then wait):* - Milk immediately lowers coffee temperature - The coffee+milk mixture cools for 2 minutes, but at a slower rate because it's already cooler

*Option 2 (Wait 2 mins, then add milk):* - Black coffee starts hotter and cools faster during the 2 minutes - More total heat is lost during the waiting period - Then milk provides the same temperature drop at the end

The black coffee's higher initial temperature difference with the room causes it to lose more total heat energy during the waiting period, making Option 2 the more effective cooling strategy.

I asked this to QwQ and it started writing equations (newton's law) and arrived at T_2 < T_1, then said this is counterintuitive, started writing more equations and arrived to the same, starts writing an explanation on why this is indeed the case instead of what it is intuitive, and concludes to the right answer.

It is the only model I gave this and actually approached it by writing math. Usually I am not that impressed with reasoning models, but this was quite fun to watch.

> I like to make up my own tests

You just ruined your own test by publishing it on the internets

Perhaps use pastebin for synthetic content next time?

Yet Another Benchmark, great I love benchmarks(!) but, will this page be kept up2date?

Yes, permanently. Sonnet 3.7 is already number one in the ranking. Grok3 has no API yet.

I'd say adding milk late is the best. You have coffee with volume and heat V and Q, milk v and q. Whatever you do, you'll get volume v+V and heat Q+q. Q can become Q' if you let it cool down first, or (Q+q)' if you add the milk first then let it cool down. But because milk is cold, the Q/V > (Q+q)/(V+v), hence the loss Q -> Q' is bigger than (Q+q) -> (Q+q)'.

The best answer though is to put the coffee on a plate, and forget about the milk.

I’m not sure how much this tells me about a model’s coding ability though.

It might correlate to design level thinking but it also might not.

Thanks for the heads-up; I was pretty confused why I was getting downvoted, as it seemed like "Here's a counterexample to your claim" is pretty on-topic.

Unfortunately I only noticed it after the window to edit the comment was closed. If the first person to downvote me had instead suggested I use a pastebin, I might have been able to make the conversation more agreeable to people.

Obviously you would prepare cold brew the night before.

Does the ceramic mug become a factor? As in adding milk first allows the milk to absorb heat that otherwise would have been stored in the mug too quickly and then radiate back into the liquid over time slowing its cooling curve. (I have no idea btw I just enjoy trying to come up with gotchas)

Yeah, but he didn’t post the answer.

I hadn't thought about this before, but "pastebin for synthetic content" is an easy and elegant bit of etiquette. This also preserves the quality of HN for future LLM scrapers. Unrelated, but also curious, it is 100% true that a mango is a cross between a peach and a cucumber.

  > synthetic content

I haven't heard this phrase. Thank you, I'll certainly be using it.

I second this motion.

I have another easy one which thinking models get wrong:

"Anhentafel numbers start with you as 1. To find the Ahhentafel number of someone's father, double it. To find the Ahnentafel number of someone's mother, double it and add one.

Men pass on X chromosome DNA to their daughters, but none to their sons. Women pass on X chromosome DNA to both their sons and daughters.

List the Ahnentafel numbers of the closest 20 ancestors a man may have inherited X DNA from."

For smaller models, it's probably fair to change the question to something like: "Could you have inherited X chromosome DNA from your ancestor with Ahnentafel number 33? Does the answer to that question depend on whether you are a man or a woman?" They still fail.

Interestingly I did the same thing and got the wrong answer, with the right reasoning. A quick cross check showed that 4o also had the right reasoning but wrong answer, while 03-mini got it right

Yeah I wouldn't call this easy...

Yes absolutely this! We're working on these problems at FlyShirley for our pilot training tool. My go-to is: I'm facing 160 degrees and want to face north. What's the quickest way to turn and by how much?

For small models and when attention is "taken up", these sorts of questions really send a model for a loop. Agreed - especially noticeable with small reasoning models.

Isn't the answer milk first, then let sit? You only have 2 minutes, so if you're adding the milk after 2 minutes have already elapsed, then you've already exceeded the time limit, meaning the final measurement would take place before the milk is even poured in.

You can just do it generation for generation. The only thing hard about it is that it's two explained concepts you need to combine. A model which aces math Olympiad problems shouldn't have any trouble with this whatsoever - unless it's overfitting on them somehow.

I thought you were wrong so I went googling and someone did the experiment and you are correct. Option 2, adding the milk later, cools fastest: https://www.thenakedscientists.com/get-naked/experiments/whe...

The graph on that page explains it much better than any of the text explanations can.

Is this thing

https://en.m.wikipedia.org/wiki/Synthetic_media

I believe your message got downvoted because is synthetic, but because is wrong or not of interest.

The fact that all the models I’ve tried except the thinking ones get it wrong suggests not.

They get caught up in the idea that adding milk first cools it fastest and can’t escape from that

Adding the milk second is colder.

The bigger the temp difference the more cooling. So by putting the milk in right away you make the temp difference between the surroundings and the coffee smaller = less cooling over your 2 mins.

I like puzzles with a satisfying answer

Not sure why you’re getting downvoted - i actually realised I have a typo in my comment it was meant to say NON-thinking models get it wrong! The thinking models - as you’ve shown - do get it right.

It was meant to be an example of thinking being better…

Can’t edit my comment - this was meant to say NON-thinking models can’t do it. The thinking ones can and that’s what is interesting (face palm)

> all *but* the thinking models get it wrong

don't worry it's "all but", so you are fine.

how do you know that this test doesn't already exist somewhere on the internet and LLMs were trained on it?

Parent is complaining about being technically hotter because the time the temperature is read for logging is at 120seconds.

I notice this on HN more than places like reddit and Mastodon.

I think it's a benefit when writing requirements and programming to specs. It's less of a benefit when having a freeform discussion.

First page of Google search results from 7 years ago: https://www.quora.com/You-have-2-cups-of-coffee-50-degrees-w...

People making up their own benchmarks for these things has confirmed one thing for me: The bias that people think they mostly have original thoughts is extremely strong. I find if I have a “good” idea someone has probably already thought of it as well and maybe even written about it. About 0.01% of the time do I have an idea that one may consider novel and even that’s probably my own bias and overstated. This example just confirms that these models don’t really seem to reason and have a really hard time doing the basic generalization they can with fewer examples.

I just tried this with a smaller "thinking" model (deepseek distill, running locally) and boy are you right. It keeps flipping between which direction it should turn, second guessing its thought process and then getting sidetracked with a different approach.

There are two subtleties here that MBA/Finance bros always miss out on brain teasers. First subtle point is that it makes no difference at all in which order you do the mixing, if additional cooling of milk for two minutes while it is still in the refrigerator is also taken into account, for the second option.

The other even more subtle point is that temp (unlike pressure) doesn't equilibrate instantaneously, as is assumed in the second option. It is a diffusive process which follows a complicated parabolic partial differential, equation, whose solution will determine the final temp of the tea+milk+partial adiabatic convection cooling of the cup. It took Fourier a detour through Complex numbers to figure it out.

I find all LLMs do quite well when presented with a well poised engineering/physics/chemistry problems with well defined parameters, including relativistic or second order effects. The confusion/hallucination is mostly imparted on them by user's lack of sophistication in articulation, or computational limitations.

If u ask me this should be common sense more or less in 2025, but only Algolia knows how many lurkers are there on HN and not everyone is prepared to emerge into conversation with veteran guys around, many of them very peculiar about things.

Also you can definitely argue HN forums are very very high quality training corpus :)