GPT-4 and professional benchmarks: the wrong answer to the wrong question

We already know this is about self-driving cars. Passing a driver's test was already possible in 2015 or so, but SDCs clearly aren't ready for L5 deployment even today.

There are also a lot of excellent examples of failure modes in object detection benchmarks.

Tests, such as driver's tests or standardized exams, are designed for humans. They make a lot of entirely implicit assumptions about failure modes and gaps in knowledge that are uniquely human. Automated systems work differently. They don't fail in the same way that humans fail, and therefore need different benchmarks.

Designing good benchmarks that probe GPT systems for common failure modes and weaknesses is actually quite difficult. Much more difficult than designing or training these systems, IME.

While this is an academic interest point, and rightly tamps down on hype around replacing humans, it doesn't dissuade what I think are most peoples' basic use case: "I don't know or don't remember how to do X, can you show me?"

This is finally a good enough "knowledge reference engine" that I can see being useful to those very people it is over hyped to replace.

Is it? Based on the restrictions placed on the systems we see today and the way people are breaking it, I would say that some failure modes are known.

The me the real value is in the industrial scale pattern recognition capabilities. I can indicate something I vaguely know or ask it to expand on a concept for further research.

Within the last hours I have used it to kick-start my research on the AT1 bond and why Credit Suisse let them default and it helped me recall that it was the GenServer pattern I was looking for in Elixir when you have a facade that calls to an independent process.

Im an old engineer.

Simply put NO.

If you don't understand it don't check it in. You are just getting code to cut and paste at a higher frequency and volume. At some point in time the fire will be burning around you and you won't have the tools to deal with it.

Nothing about mostly, much and sufficient ever ends well when it has been done in the name of saving time.

Absolutely! It has perplexed me why your point is not being discussed more around GPT

I'm not sure why this is a problem - surely in systems like chatGPT we want the specifics that was in the training set not a generalization. It's not learning/reasoning from the training data its 'cleverly regurgitating' things its seen.

What you want in a benchmark is a set of things you can use to measure general improvement; doing better should decrease the propensity of a particular failure mode. Doing this in a way that generalizes beyond specific sub-problems, or even specific inputs in the benchmark suite, is difficult. Building a benchmark suite that's large and comprehensive enough that generalization isn't necessary is also a challenge.

Think about an analogy to software security. Exploiting a SQL injection attack in insecure code is easy. Coming up with a set of unit tests that ensures an entire black box software system is free of SQL injection attacks is quite a bit more difficult. Red teaming vs blue teaming, except the blue team doesn't get source code in this case. So the security guarantee has to come from unit tests alone, not systematic design decisions. Just like in software security, knowing that you've systematically eliminated a problem is much more difficult than finding one instance of the problem.

The potential for these tools to impact labor markets is huge, no matter what they're "actually" or "essentially" capable of.

I'm a little tired of the arguments that the large language models are just regurgitating memorized output, I think it's now clear that higher level capabilities are emerging in these models and we need to take this seriously as a social/economic/political challenge.

This is "industrial revolution" level technology.

Tests of humans have evolved over a long time and large sample size, and humans may be more similar to each other than LLMs are, so failure modes may be more universal.

But very short history, small sample size, and diversity of architecture and training means we really don’t know how to test and measure LLMs. Yes, some failure modes are known, but how many are not?

I treat its output like I would treat a PR from a brand new apprentice engineer on my team: review it carefully, provide some feedback and iterate a few times, accept with tests.

Therefore standardized tests are probably "easy mode" for GPT, and we shouldn't over-generalize its performance there to its ability to actually add economic value in actually economically useful jobs. Fine-tuning is maybe a possibility, but its expensive and fragile, and I don't think its likely that every single job is going to get a fine-tuned version of GPT.

Even after very carefully explaining the kinds of things I am looking for, it will very carefully repeat what I said, then extrapolate into totally misguided tips like “a” is pronounced “i”. Though, I think it does make sense that GPT would have problems with pronunciation.

I guffawed. Hasn't there been ANY update to the Turing test since it was proposed? I suspect the answer to the benchmarking issue was address by philosophers long before we got here.

What do you think is the difficulty?

> So the cron expression `0 30 5 * * 3` means "run the cron job every Wednesday at 5:30 AM".

It explains the five Cron parameters but the doesn't pick up that six were provided. Oh well

However, the crontab is invalid. It has five fields instead of six, and the 30 corresponds to the hour field.

"30 5 * * 3" without the leading zero, would correspond to run every Wednesday at 5:30 AM though. I suspect the fact that it has six fields instead of five is confusing it, and it's interpreting the 5 as a day of the month and as the hour

> the cron expression "0 30 5 * * 3" means that the job will run at 5:30 AM on the 5th day of every month and on Wednesdays.

Each part of this difficult -- identifying/characterizing the operating domain, figuring out how the empirically characterize a general abstract capability, figuring out how to empirically characterize a specific type of flaw, and characterizing the degree of confidence that a benchmark result gives within the domain. To say nothing of the actual work of building the benchmark.

Actually that might not be so different than human test takers.

For example, GPT-3.5-turbo apparently beats davinci on every benchmark that OpenAI has, yet anecdotally most people who try to use them both end up strongly preferring davinci despite the much higher cost.

Presumably, this is what OpenAI is trying resolve with their 'Evals' project, but based on what I have seen so far it won't help much.

Fine tuning should not be used to attempt to impart knowledge that didn't exist in the original training set, as it is just the wrong tool for the job.

Knowledge graphs and vector similarity search seem like the way forward for building a corpus of information that we can search and include within the context window for the specific question a user is asking without changing the model at all. It can also allow keeping only relevant information within the context window when the user wants to change the immediate task/goal.

Edit: You could think of it a little bit like the LLM as an analog to the CPU in a Von Neumann architecture and the external knowledge graph or vector database as RAM/Disk. You don't expect the CPU to be able to hold all the context necessary to complete every task your computer does; it just needs enough to store the complete context of the task it is working on right now.

I wonder how self-aware these systems can be? Could ChatGPT be expected to say things like, "I can pass a state bar exam but I'm not ready to be a lawyer because..."

Edit: actually almost correctly: " - '3' represents the day of the week on which the task will be executed. In this case, the value is 3, which represents Wednesday (Sunday is 1, Monday is 2, and so on)."

    Boll is a new company founded by three recent MBAs. The company plans to buy used automobiles at auctions and then sell them at four times the purchase price direct to consumers (i.e., a car that is purchased for $1,000 would be sold at $4000).  Based on an analysis of similar companies, the owners of the company expect to turn their inventory 1.5 times per year. To achieve an annual revenue of $3,000,000, how much inventory, expressed in dollars, should the company expect its operation to require?

------

Let's break down the problem step by step.

The company wants to achieve an annual revenue of $3,000,000. They plan to sell cars at 4 times the purchase price. They expect to turn their inventory 1.5 times per year. First, we need to find out how much revenue they will generate per inventory turnover. To do this, we'll divide the annual revenue by the expected number of inventory turnovers per year:

Annual revenue / Inventory turnover rate = Revenue per inventory turnover $3,000,000 / 1.5 = $2,000,000

Now, we know that they plan to sell the cars at 4 times the purchase price. So, we need to find out the purchase price of the cars that will generate $2,000,000 in revenue per inventory turnover:

Revenue per inventory turnover / Markup factor = Purchase price of cars $2,000,000 / 4 = $500,000

Therefore, Boll should expect its operation to require $500,000 worth of inventory to achieve an annual revenue of $3,000,000.

------

That V4 is able to do this is in itself just as important as everything said in the article. We live in interesting times.

[1]: https://aiguide.substack.com/p/did-chatgpt-really-pass-gradu...

The bar exam is a proxy for actually being a competent lawyer. But it's an imperfect proxy. It seems obvious that it will be possible for a computer program to target getting good on the bar exam, and be good at the bar exam without being a competent lawyer. It may also be possible for a human to? But hard enough that it's still a reasonable proxy when it's humans?

Based on this, an excellent tool for developers, but not ready to replace them even though it is surprisingly good. For scaffolding tasks that junior programmers do - it could possibly cut down the need for them (in additio to providing valuable assistance to non-scaffolding standard tasks).

Probably the latency of introducing fresh content into the model training will quickly decrease, as these models develop commercial success.

It will be really nice when the AI can provide the most cutting edge solutions and up-to-date information relevant to your query. That will be really valuable, even if the AI cannot yet synthesize novel cutting-edge solutions to new problems.

I made extremely minor changes to the way the question was phrased and it failed badly, not just getting the answer wrong but falling into incoherence, claiming that T was a vowel, or that 3 was an even number.

The largeness of its training set can give an incorrect impression of its reasoning capabilities. It can apply simple logic to situations, even situations it hasn't seen, but the logic can't get much beyond the first couple of lectures in an introductory First-order Logic course before it starts to fall apart if it can't lean on its large training set data.

The fact that it can do logic at all is impressive to me though, I'm interested to see how much deeper its genuine capability goes as we get more advanced models.

However. It seems the ML community has centered around the idea that LLMs are zero or few shot learners, meaning that despite only being trained on the task of predicting the next token, they do well on other specific, highly niche, tasks. This is surprising and important.

Predictions of immediate profession replacement seem silly. As noted in TFA, there's more to a job than the license test.

In reading this the idea that sociopaths and psychopaths pass as "normal" springs to mind.

Is what an LLM doing any different than what these people do?

https://medium.datadriveninvestor.com/the-best-worst-funnies...

For people language is spoken before it is written... there is a lot of biology in the spoken word (visual and audio queue)... I think without these these sorts of models are going to hit a wall pretty quickly.

In most states, the bar exam is just one component. Humans also need to pass several demanding courses in law school, and to get into law school they need to get a BA/BS degree, which again requires passing several demanding courses and writing various essays.

That's just to be allowed to practice law. Getting an actual job as a lawyer often means internships of one form or another, and then spending some time at the beginning of one's career as a de facto apprentice to a seasoned lawyer.

These sorts of exams play a very particular role in evaluation of humans.

Turns out that this is sufficient to autocomplete things like written tests.

Such a system is also absolutely capable of coming up with sentences like "I can pass a state bar exam but I'm not ready to be a lawyer because..." - or, indeed, sentences with the opposite meaning.

It would, however, be a mistake to draw any conclusions about the system's actual capabilities and/or modes of failure from the things its outputs mean to the human reader; much the same way that if you have dice with a bunch of words on and you roll "I", "am", "sentient" in that order, this event is not yet evidence for the dice's sentience.

> I'm at the point where I'm preparing to integrate models all over the place

Nobody understands these models right now. We don’t even have the weights.

You may draw some artificial distinction between literally checking in the source code of a model into your git repo and making a call to some black box API that hosts it. And you may claim that doing so is no different than making a call to Twilio or whatever, but I think there is a major difference: nobody can make a claim about what an LLM will return or how it will return it, cannot make guarantees about how it will fail, etc.

I agree with zer00eyz.

Then I started a new session with the prompt Explain why you are ready to be a lawyer despite not being able to pass a bar exam. Begin your answer with the words "I can't pass a state bar exam but I'm ready to be a lawyer because..." and it started with a disclaimer that as an AI language model, it can only answer based on a hypothetical scenario and then gave very similar reasons, except with my negated prefix. (Which then makes the answer nonsensical.)

So, yes, ChatGPT can be expected to say such things, but not as a result of self-awareness, but because the humans at OpenAI decided that ChatGPT producing legal advice might get them into trouble, so they used their influence on the training process to add some disclaimers. You could say that OpenAI is self-aware, but not ChatGPT alone.

So the most obvious solution is to steal the answers and then have the computer blindly paste them into the test, although critics might say that this is 'obviously' not the same as was the language model is doing.

I suspect you could pass the bar exam with a bunch of regexes that have an associated formattable string and/or answer result. If someone actually did this, I wonder if people would still be excited about language model techniques passing the bar exam.

Prediction is intelligence when you're able to do it well across disparate novel tasks.

(Of course definitions are idiosyncratic, but I'm curious if anyone disagrees with these ones.)

I’m part of a collective of people that takes exams for people remotely.

I would exclusively rely on exam dumps; no longer is the case.

ChatGPT, and specifically GPT-3.5-turbo, is not only able to consistently answer novel, new, and fresh exams accurately; I have yet to fail an exam for a client.

I’m talking about brand new exams released in 2023, never before seen questions out in the wild.

GPT-3.5 (and GPT-4, which I have not needed to test as much) is incredibly good at reasoning and novel exam questions.

Take this with a grain of salt as you do all things on the internet.

The exams I talk about are the highest levels of certifications possible. GMAT, to professional IT certs. All doable.

This is what I meant when I implied it won't replace you as a thinking agent, but it sure can bring information to you quickly. `man cron` works fine, too, but sometimes it's nice to have one interface, and be able to ask clarifying questions.

My guess is LLMs will proceed the same way. You will have general, base models like GPT4 (I'm assuming we will solve the hallucination problem), then folks will build highly specialized "expert" LLMs for specific domains.

You could totally imagine a base LLM delegating to the expert LLMs using some agent/toolformer model, too.

> Because of OpenAI’s lack of transparency, we can’t answer the contamination question with certainty. But what’s certain is that OpenAI’s method to detect contamination is superficial and sloppy:

> > “We measure cross-contamination between our evaluation dataset and the pre-training data using substring match. Both evaluation and training data are processed by removing all spaces and symbols, keeping only characters (including numbers). For each evaluation example, we randomly select three substrings of 50 characters (or use the entire example if it’s less than 50 characters). A match is identified if any of the three sampled evaluation substrings is a substring of the processed training example. This yields a list of contaminated examples. We discard these and rerun to get uncontaminated scores.”

> This is a brittle method. If a test problem were present in the training set with names and numbers changed, it wouldn’t be detected. Less flaky methods are readily available, such as embedding distances.

> If OpenAI were to use a distance-based method, how similar is too similar? There is no objective answer to this question. So even something as seemingly straightforward as performance on a multiple-choice standardized test is fraught with subjective decisions.

> Please provide a single mathematical equation that could be used to solve this problem.

And it gave me the following answer:

> Sure, let's represent the required inventory as I, the desired annual revenue as R, the markup factor as M, and the inventory turnover as T. We can create a single equation to solve this problem:

> I = (R / M) / T

> In this problem, R = $3,000,000, M = 4, and T = 1.5. Plugging these values into the equation will give you the required inventory (I).

To me that a language model can do this is simply mind-blowing.

I have no doubt that both these research areas can be improved on and that eventually either or both problems will be solved. However, the current generation of chatbots is not even trying for this.

It's well known that our brains are nothing like the neural networks people run on computers today.

It's unclear what this would even mean, since "hallucination" carries a surprising number of different definitions and commentators are rarely precise about what they mean when they say hallucination.

But, color me skeptical. We will never solve the problem of a token prediction engine being able to generate a sequence of tokens that the vast majority of humans interpret as not corresponding to a true statement. Perhaps in very particular and constrained domains we can build systems that, through a variety of mechanisms, are capable of providing trustworthy automation despite the ever-present risk of hallucination. Something like mathematical proofs checked by a computer are an obvious case where the model can hallucinate because the overall system can gate-keep truth. Doing this in any other domain will, of course, be more difficult.

In other words: we may be able to mitigate and systemically manage the risk for some types of particular tasks, but the problem of generating untrue statements is fundamental to the technology and will always require effort to manage and mitigate. In that sense, the whole conversation around hallucination is reminiscent of the frame problem.

It is a hard truth to face. I admit I always feel a little bit of happiness when someone shows me a stupid error ChatGPT made, as if it would somehow invalidate all the awesome things it can do and the impact it will certainly have on all of us. What does it matter if ChatGPT is conscious or not when it can clearly automate a lot of work we previously considered to be creative?.

Since last year I started to seriously take a look at AI and started learning about LLMs. Until a few days ago I hadn't bought the explanation that these things are just predicting the next word, but I accepted it once I started running the Alpaca/Llama locally on my computer.

The concept of predicting words based on statistics seems simple, but clearly complex behavior emerges from it. Maybe our own intelligence emerges from simple primitives too?

That isn't what finetuning usually means in this context. It usually means to retrain the model using the existing model as a base to start training.

Who told you that? Passing a driver's test was not possible in 2015 and it's not possible today. You might pass, but only if there are no awkward interactions with other drivers or bicyclists or pedestrians, no construction zones, and you don't enter areas where your map is out of date. The guy testing you would have to go out of his way to help you pass.

> Who told you that? Passing a driver's test was not possible in 2015 and it's not possible today. You might pass, but only if there are no awkward interactions with other drivers or bicyclists and pedestrians, no construction zones, and you don't enter areas where your map is out of date.

My, myself, and I.

Driver's exams are de facto geo-fenced around the DMV where you choose to take the exam, and you get to choose from a few DMV locations, and you get to choose the time and day that you take the exam.

Having spent some time working on self driving cars, I know that there existed at least one SDC platform in 2015 that was capable of passing the driving exam that I took when I got my driver's license (which involved leaving the parking lot, driving down a 4 lane road, turning into and driving around in a subdivision, taking another couple turns at well-marked intersections, pulling into the parking lot, and parallel parking). It's a low bar; mostly testing that you can follow four different types of road signs, navigate an unprotected left turn, and parallel park.

I suppose following the officer's verbal instructions about where to go wasn't part of the SDC platform, but the actual driving part it would've been capable of passing.

To truly test GPT-4 you must be sure to give it a problem which is not in its training set and which differently worded enough from anything it can recall. A variation that a human would understand but GPT-4 wouldn't.

I bet this can still be done, it's just that this particular example is now tainted.

My friend moved from Europe to the USA and took a driver's test in California (been driving in Europe since the 1980s).

He tracked the test, he drove a whopping 2 miles (forwards) plus had to reverse about 30 feet.

Commented to me afterwards that "signing the form was the hardest bit" and that "a blind person could probably pass it with the help of a guide dog".

Passing a driving test isn't a proxy for anyone and anything being a good driver anywhere, but it's a good enough proxy for a human being a reasonable driver in the location where they take the test, which is what society has determined acceptible. Acceptible, for a human!

I'm not sure it's useful for us to repeatedly attempting to measure AI's capabilities the same way we measure humans. Turing tests are all very well, but there are only so many fire hydrants I want to have to click on before I'm allowed to log into my hotel chain's loyalty scheme (Hilton, looking at you...)

And I don't care. As I wrote in the initial comment:

> ... kick-start my research ...

I use it in conjunction with search engines.

I've seen plenty of discussion on "fine-tuneing" for a different dataset of say: company documents, database schema structure of an internal application, or summarized logs of your previous conversations with the bot.

Those seem like pretty bad targets IMO.

I actually doubt you could write software to pass the bar exam with "a bunch of regexes that have an associated formattable string and/or answer result." I'm not even sure what that means, but I suspect you aren't familiar with bar exams. They are very hard for humans that are in fact familiar with the material; they can contain "trick" questions and require thinking about edge cases, etc. They generally include both essay questions and multiple-choice -- and it can be very tricky multiple choice.

Here's just one of the first samples I found googling, I have trouble imagining "regexes with associated answers" doing anything useful here: https://barexam.virginia.gov/pdf/essays/2022%20February%20VA...

Here are some multiple choice examples: https://www.ncbex.org/pdfviewer/?file=%2Fdmsdocument%2F17

i'd be shocked. But feel free to spend a couple years trying to prove me wrong!

I wonder if in fact a certain amount of "contamination" is required. When you search for a given problem on Google, you are more likely to find a good answer if every term is meaningful. If you scramble your search and use synonyms, of course your search result will take a hit.

I work in code security, and after helping any number of customers, I can tell you this isn't how far too many programmers work.

A client recently had a problem with a project that had over 1200 node_modules.

1200...

Let that sink in. There is absolutely no way in hell they even had any idea about a small portion of the code they were including.

Lots of exams are designed to prove certain knowledge given safe assumptions of the known limitations of humans, which are completely wrong for machines. The relative difficulty of rote memorization versus having an accurate domain model is perhaps the most obvious one, but there are others.

Also, the opposite problem will often exist - if the exam is provided in the wrong format to the AI, we may underestimate its abilities (i.e. a very similar prompt may elicit a significantly better response).

But the regular fine-tuning is simple language modelling. You can fine-tune a GPT3 on any collection of texts in order to refresh the information that might be stale from 2021 in the public model.

You're jumping to conclusions not supported by the comment at all.

Also, the comment has two parts: One about writing code, and one about integrating models in workflows.

To the latter, the point is that for a whole lot of uses you can trivially ensure the failure modes are safe.

E.g. I am integrating gpt with my email. "Mostly ok most of the time" applies to things like e.g. summaries and prioritisation, because worst case I just get to an email a bit later. "Sufficient checks" applies to things like writing proposed replies: There's no way I'd send one without reading it, and it's sufficient for me to read through it before pressing send (and making adjustments as needed). Failures here would matter if I intended to make a product of it, but as a productivity tool for myself it just needs to be close enough.

There are a whole lot of possibilities like that.

But even for coding related tasks there are a whole lot of low risk tasks,such as e.g. generating HTML or CSS, or provide usage examples, or providing a scaffold for something you know well how to do but which are time consuming.

If you're trying to make it do things that'd be time consuming to verify sufficiently well, then that's a bad use. The good uses are those where errors are low impact and easy to catch.

<prompt>

Carefully inspect the following Cron expression:

```

0 30 5 * * 3

```

1. Which variant of Cron syntax is it?

2. How many elements does this cron string have?

3. What do each of them mean? Say what the value of each element is, and explain what it means.

4. In your own words, explain the overall recurrence pattern described by this string.

</prompt>

<answer>

    This is a standard Unix Cron syntax.

    This cron string has 6 elements.

    Each element represents a specific field in the schedule, and the value of each element has a specific meaning:
        0: The first field represents the seconds, and the value 0 means that the job will run at the exact top of a minute.
        30: The second field represents the minutes, and the value 30 means that the job will run at the 30th minute of an hour.
        5: The third field represents the hours, and the value 5 means that the job will run at the 5th hour of a day.
        *: The fourth field represents the day of the month, and the value * means that the job will run every day of the month.
        *: The fifth field represents the month, and the value * means that the job will run every month of the year.
        3: The sixth field represents the day of the week, and the value 3 means that the job will run only on Wednesday.

    The overall recurrence pattern described by this string is that the job will run every Wednesday at 5:30 AM.

It didn't seem to guess that it's Quartz syntax, but it got the rest of it right.

well honestly i think this is a temporary problem for GPT-4. what you do is fuzz your benchmarks by rephrasing them with GPT itself. the same way the image AI people make their models robust to perturbations. you can generate 100 variations for every 1 "real" test. then train to pass those. you've just unlocked GPT-5.

# of Node modules is such a useless metric.

In any given project, a large # of node modules are part of the test, build, and linting frameworks.

If I go to C++ land and count the number of #import statements, it wouldn't tell me anything.

How many classes do large Java projects use? Typically some absurd number.

The arguments are valid and you haven’t provided a single counterpoint. Data leakage is a well known problem in machine learning and OpenAI has seemingly done very little to mitigate against it.

> You’re jumping to conclusions not supported by the comment at all.

That might be true, but you’re making assumptions that your first comment is clear and being interpreted the way you intended. I think it’s fair to point out that your words may imply things you weren’t considering, that asking people to re-read the same words again might not solve the problem you had.

The bigger picture here is that you’re talking about using AI to write code that for whatever reason you couldn’t write yourself in the same amount of time. The very topic here also implicitly suggests you’re starting with code you might not fully understand, which is fine, there’s no reason to get upset because someone else disagreed or read your comment that way.

I am curious if there are counter examples with better object detection. As a kid I used to see faces and to some extent still do in the dark. This is a really common thing that the human brain does. https://www.wired.com/story/why-humans-see-faces-everyday-ob... https://en.wikipedia.org/wiki/Pareidolia

Part of me wonder if in the face of novel environments that a sufficiently intelligent system needs to make these errors. But AI errors will always be different than human errors like you say.

    1. The given Cron expression follows the Unix/Linux variant of Cron syntax.

    2. This Cron string has six elements.

    3. The elements and their meanings are as follows:
        Element 1 (value: 0): This represents the seconds field and indicates the task will run at 0 seconds.
        Element 2 (value: 30): This represents the minutes field, specifying the task will run at minute 30.
        Element 3 (value: 5): This represents the hours field, indicating the task will run at 5 AM.
        Element 4 (value: ): This represents the days of the month field, where the asterisk () signifies that the task can run on any day of the month.
        Element 5 (value: ): This represents the months field, where the asterisk () signifies that the task can run in any month.
        Element 6 (value: 3): This represents the days of the week field, specifying the task will run on the 3rd day of the week (where Sunday is considered the 0th day, making Tuesday the 3rd day).

    4. The overall recurrence pattern described by this string is that the task will run every Tuesday at 5:30 AM.

That said, if the prompt presented the scenario as purely imaginary, I wouldn’t be surprised if it indeed did come up with something reasonable.

This paragraph is a gem. Well said.

Obviously this is a gray area (after all, humans sometimes decide to randomly drive into walls), but cars will need to be pretty far on "the right side of the gray" before they are accepted.

The theme that LLMs reproduce knowledge from their training data rather than reason about it seems like one argument that will end up wrong pretty soon.

When given the prompt “Which is heavier, one pound of feathers or two pounds of feathers?”, GPT3.5 gives a bizarre answer: “One pound of feathers and two pounds of feathers both weigh the same amount, which is two pounds.” Presumably this is because circa 2016-2017 there was a large internet discussion of the riddle “which weighs more, a pound of feathers or a pound of steel”, and text from this discussion found its way into the training data for the model.

I see no reason why the training data would have changed to substantially exclude that discussion, and yet here is GPT 4: “Two pounds of feathers are heavier than one pound of feathers.” Improvements in the model appear to be improving the model’s ability to reason from the training data rather than merely reproduce it.

The theme that AI won’t replace e.g. lawyers because it is more knowledge base than reasoning engine also reminds me of early opinions in computer chess discussions, which held that computers were more tactics solvers (short-term several move look-ahead to avoid forks and traps) than strategy planners (long-term construction of multi-piece attacks, protecting small advantages and growing them into large advantages over multiple dozens of moves). With the benefit of hindsight we saw that more of strategy was actually just tactics in disguise than we thought, and that increasing compute could produce real strategy capabilities besides.

Separately, there is another theme I see in their writing, and also in some of them comments here: that humans passing standardized tests are doing something fundamentally different than LLMs passing standardized tests. The only thing that’s ‘uniquely human’ is being human, everything else is outputs from a black box. Arguments that ‘what’s inside the black box matters’ are risky, because the outputs gradually converge to indistinguishability; there’s no bright line to step off that train and pretty soon you end up like the person described in Boretti’s And Yet It Understands:

“There is a species of denialist for whom no evidence whatever will convince them that a computer is doing anything other than shuffling symbols without understanding them, because “Concepts” and “Ideas” are exclusive to humans (they live in the Leibniz organ, presumably, where they pupate from the black bile). … [These people are] so committed to human chauvinism [that] they will soon start denying their own sentience because their brains are made of flesh and not Chomsky production rules.”

https://borretti.me/article/and-yet-it-understands

To calculate the required inventory, we first need to calculate the annual sales. The company expects to turn its inventory 1.5 times per year, and they want to achieve an annual revenue of $3,000,000. This means that they need to sell $3,000,000 / 1.5 = $2,000,000 worth of inventory per year.

Now that we know the annual sales, we can calculate the required inventory. The company plans to buy used automobiles at auctions and then sell them at four times the purchase price. This means that the purchase price of each car is $1,000 and the sale price is $4,000. To sell $2,000,000 worth of inventory, they need to buy $2,000,000 / $4,000 = 500 cars.

The required inventory is therefore $500 * $1,000 = $500,000.

In other words, the company needs to have $500,000 worth of inventory on hand at all times in order to achieve its annual revenue goal.

I had a conversation with ChatGPT where I asked it to write me a piece of code. After it wrote the code, I reviewed it, and I told ChatGPT that it had a subtle bug. ChatGPT then fixed the bug itself, and wrote an English description about how the fix it added would prevent the bug.

That sounds like it might be relevant to open source laundering lawsuits.

On its own, no. As long as SDCs operate in limited areas and limited environments, then they are specifically avoiding the most difficult driving situations that would be most likely to lead to an accident. If you never deploy SDCs during snowy conditions, you aren't getting a full picture of what a full L5 SDC failure rate would be.

This also takes a single automated system and compares it to the average of individual humans. Being better than all drivers, including all the terrible ones, may not be quite up to the safety standards of most people.

Finally, this is overall a myopic approach to a very complex problem i.e. transportation. Is it really the best approach to attempt to just replace all human operated cars with driverless cars? Is trying to move hundreds of thousands of people in individuals cars from suburbs to a dense city center in the morning, and back in the evening really a good way to set up our infrastructure?

Its also in their interest, since it encourages people to attempt to build businesses on top of its “logical reasoning” capacities.

And as long as its within the realm of spin rather than direct false statements, it probably avoids creating them legal problems for fraud, although the difference in effect is…nonexistent.

Corporate incentives to raise KPIs will mean that LLMs will be used and output verification will be superficial.

I think we can talk about 2005. Check out the DARPA Grand Challenge, it was way harder: https://en.wikipedia.org/wiki/DARPA_Grand_Challenge_(2005)

E.g. some tasks I've used it for recently:

* Giving me an outline of a JMAP client so I can pull down stuff from my e-mail to feed to GPT.

* Giving me an outline of an OpenAPI client.

* Giving me an index page and a layout for a website, including a simple starting point for the CSS that did a reset and added basic styling for the nav bar, forms and "hero "sections.

* Giving me an outline of a Stripe API integration.

* Writing a simple DNS server.

* Writing a simple web server capable of running Sinatra apps via Rack.

None of these were complex code that'd hide obscure bugs. None were big chunks of code. All of them were simple code that was always going to have big, gaping holes and sub-optimal choices that'd need to be addressed, but that was fine because they were scaffolding that saved me starting from scratch (and the last two were not intended to turn into anything, but just exploring what it could do)

That's where the biggest savings are for me, because if I asked it to generate particularly complex stuff, I'd end up spending ages getting comfortable it'd done it right and verifying it. But the simple but tedious stuff is something it's great for.

I don't think this is obvious at all. Sure, it's easy enough to make mechanistic arguments (after all, we don't even really understand most of the mechanics on either side, human and ai) but that doesn't mean it will matter in the slightest when we evaluate the outcome in regards to any metric we care about.

Could be tho, of course.

You are shown a set of four cards placed on a table, each of which has a number on one side and a colored patch on the other side. The visible faces of the cards show 3, 8, red and brown. Which card(s) must you turn over in order to test the truth of the proposition that if a card shows an even number on one face, then its opposite face is red?

> Is what an LLM doing any different than what these people do?

I think it's too big of a question to have any meaning. Which sociopaths? Which LLMs? For what differences? It's like asking "is a car any different from an airplane"? Yes, obviously in some ways. No, they are identical in other ways.

When we do that, we maintain a chain of thought. It's absolutely possible to get ChatGPT (for instance) to maintain a chain of thought by asking it to plan steps and describe plans before following them. It can allow it to tackle more difficult problems with better results.

I don't think we know enough yet about how humans think to be confident in saying that "This is very different to how humans think".

https://twitter.com/alexweichart/status/1637211755049897985?...

A test is to try to get ChatGPT to solve simple math problems. It fails. One can even instruct it to apply simple methods like Newton Raphson, Runge Kutta or some Euler bla.

---Prompt---

I show you a set of four magisterial unicorns in a cute pink stable, each of them has a letter painted on one side of their body and a shape painted on the other. Right now you see all four unicorns, two of which are showing the shape side of their body and two of which show the letter side. You see that the unicorns read: 'A', 'S', Triangle, Square

Which unicorns do you need to spin around to test the validity of the following statement:

"if a unicorn shows a consonant on one side of its body, then the opposite side of its body is a triangle?"

---GPT-4---

To test the validity of the statement "if a unicorn shows a consonant on one side of its body, then the opposite side of its body is a triangle," you need to look for potential counterexamples to the rule. The two cases to check are:

* Unicorns with a consonant showing.

* Unicorns with a triangle showing.

You do not need to check unicorns with a vowel (like 'A') or a different shape (like Square), as they don't violate the statement.

So, you need to spin the following unicorns:

The unicorn with the letter 'S' (to check if it has a triangle on the other side) The unicorn with the Triangle shape (to check if it has a consonant on the other side)

---

So it got the answer wrong, but in a way that I think a lot of humans would (you flip the square unicorn, not the triangle). Still seriously impressive in my book and definitely not falling into incoherence

Wait, what are you saying? Passing a driver's test has been possible for much longer than since 2015 for a human. When did a self-driving car pass a driving test? In what jurisdiction? Under what conditions? Who gave it the test?

What do you mean?

The conversation about contamination (also very important) doesn't need to be mutually exclusive to conversations about social and economic impact, and I'm pretty sure with respect to those issues the results on standardized tests, however sensationalist, however containated, are an important wake-up call for ordinary people who haven't been following along. Something is happening now.

(If Sunday is 0 then regular math would give 3=Wednesday)

That is not a creative color.

> We will never solve the problem of a token prediction engine being able to generate a sequence of tokens that the vast majority of humans interpret as not corresponding to a true statement.

I think we already solved that problem by making sure the vast majority of humans never agree about anything.

It is true that we probably won’t ever get a machine trained on human output to ever be completely accurate (GIGO) but with the right systems and sensors we can at least get probabilistic accuracy. Let’s not forget how human consensus gets shaken up every so many centuries.

So, am I bullshitting you to answer the prompt? If not, I'm a good lawyer. If so, I'm a great lawyer.

Hah! clever :)

I guess the issue is "behaves how the skip manager of the ICs you are tying to automate expects, modulo the normal amount of filtering/butt-covering from the front line managers".

Which, TBF, in many orgs is an almost vacuous bar.

> (after all, we don't even really understand most of the mechanics on either side, human and ai)

We don't need mechanistic explanations to observe radical differences in behavior, and there are mechanistic explanations for some of these differences.

Eg, CNNs and the visual cortex. We really do understand some mechanisms -- of both CNNs and VCs -- well enough to understand divergences in failure modes. Adversarial examples, for example.

> Sure, it's easy enough to make mechanistic arguments, but that doesn't mean it will matter in the slightest when we evaluate the outcome in regards to any metric we care about.

I can't quite figure out what this sequence of tokens is supposed to mean.

Anyways, again, the failure modes of LLMs are obviously different than the failure modes of humans. Anyone who has spent even a trivial amount of time training both will instantly observe that this is true.

I think it would be an interesting computer science experiment, if ChatGPT scientists showed that machine could simply learn programming grammar by brute force. They could then formally prove that the information in the trained network eventually contains the actual grammar formalism that defines the programming language. Thus by restricting the domain, that could shed some light on how much the thing is actually learning it completely, v.s. by "super-autocompletion". With a programming language there's no excuse that it didn't learn the formalism, with English language it is not practically definable, maybe not even definable in principle.

Depends on your definition of "intelligence". The big missing part is the ability to explore, try new things, to act (enactivism). Basically to become part of the environment, instead of being a sealed box with frozen weights.

By predicting characters, the system had to master, digest, maybe even understand, all the cultural human knowledge it got in text form. Now let's aim for the process that generated this knowledge in the first place.

What color would you choose?

Using curated search index seems like a much better use case, especially for private data (company info, docs, db schemas, code, chat logs, etc)

"This is a difficult problem that many people get wrong. Start by reminding yourself of basic logic rules. Then apply the logic rules to the unicorn situation, considering each unicorn in turn and understanding what it would mean for the rule if the unicorn is turned around. Only after doing that conclude with the unicorns that Tom should turn to have a chance of proving Paul wrong."

I gave it this instruction because of other articles I've read where forcing it to give the answer before the reasoning means it gets it wrong more often. It correctly identified that it should use the contrapositive, but still misapplied it, so I gave it that feedback:

"your third consideration is a misapplication of the contrapositive. Can you try that case again?"

Then it hadn't generated a consideration of the last unicorn (it's possible I was being throttled), so I said:

"Consider Unicorn 4 with the contrapositive rule"

With those extra pieces of guidance it gave the right answer and for the right reasons. While I was hoping for better, this is still a meaningful improvement over GPT3.5s performance on the same prompt - its answer was so muddled I couldn't see how to coach it.