GPT-4 and professional benchmarks: the wrong answer to the wrong question

In an example [1] referenced in the article, ChatGPT (maybe v3.5) did a poor job of answering the following question.

    Boll is a new company founded by three recent MBAs. The company plans to buy used automobiles at auctions and then sell them at four times the purchase price direct to consumers (i.e., a car that is purchased for $1,000 would be sold at $4000).  Based on an analysis of similar companies, the owners of the company expect to turn their inventory 1.5 times per year. To achieve an annual revenue of $3,000,000, how much inventory, expressed in dollars, should the company expect its operation to require?

How does GPT4 fare?

------

Let's break down the problem step by step.

The company wants to achieve an annual revenue of $3,000,000. They plan to sell cars at 4 times the purchase price. They expect to turn their inventory 1.5 times per year. First, we need to find out how much revenue they will generate per inventory turnover. To do this, we'll divide the annual revenue by the expected number of inventory turnovers per year:

Annual revenue / Inventory turnover rate = Revenue per inventory turnover $3,000,000 / 1.5 = $2,000,000

Now, we know that they plan to sell the cars at 4 times the purchase price. So, we need to find out the purchase price of the cars that will generate $2,000,000 in revenue per inventory turnover:

Revenue per inventory turnover / Markup factor = Purchase price of cars $2,000,000 / 4 = $500,000

Therefore, Boll should expect its operation to require $500,000 worth of inventory to achieve an annual revenue of $3,000,000.

------

That V4 is able to do this is in itself just as important as everything said in the article. We live in interesting times.

[1]: https://aiguide.substack.com/p/did-chatgpt-really-pass-gradu...

What's even more interesting, I gave GPT-4 a follow-up instruction:

> Please provide a single mathematical equation that could be used to solve this problem.

And it gave me the following answer:

> Sure, let's represent the required inventory as I, the desired annual revenue as R, the markup factor as M, and the inventory turnover as T. We can create a single equation to solve this problem:

> I = (R / M) / T

> In this problem, R = $3,000,000, M = 4, and T = 1.5. Plugging these values into the equation will give you the required inventory (I).

To me that a language model can do this is simply mind-blowing.

How do we know Mitchell's counterexample isn't in GPT-4's training set?

To truly test GPT-4 you must be sure to give it a problem which is not in its training set and which differently worded enough from anything it can recall. A variation that a human would understand but GPT-4 wouldn't.

I bet this can still be done, it's just that this particular example is now tainted.

FWIW Bard's reply:

To calculate the required inventory, we first need to calculate the annual sales. The company expects to turn its inventory 1.5 times per year, and they want to achieve an annual revenue of $3,000,000. This means that they need to sell $3,000,000 / 1.5 = $2,000,000 worth of inventory per year.

Now that we know the annual sales, we can calculate the required inventory. The company plans to buy used automobiles at auctions and then sell them at four times the purchase price. This means that the purchase price of each car is $1,000 and the sale price is $4,000. To sell $2,000,000 worth of inventory, they need to buy $2,000,000 / $4,000 = 500 cars.

The required inventory is therefore $500 * $1,000 = $500,000.

In other words, the company needs to have $500,000 worth of inventory on hand at all times in order to achieve its annual revenue goal.