GPT-4 and professional benchmarks: the wrong answer to the wrong question

Good luck getting ChatGPT to explain a cron expression like "0 30 5 * * 3". I mean, it will explain, but mixing up everything. How many other mistakes it might make?

[...]

> So the cron expression `0 30 5 * * 3` means "run the cron job every Wednesday at 5:30 AM".

It explains the five Cron parameters but the doesn't pick up that six were provided. Oh well

Isn't that an invalid cron expression? It has six fields instead of five, and says to run at the 30th hour of the day.

I got a similar but incorrect result from ChatGPT: "So, the expression "0 30 5 * * 3" means that the command should be executed at 5:30 AM on the 5th day of every month, but only if that day is a Wednesday"

However, the crontab is invalid. It has five fields instead of six, and the 30 corresponds to the hour field.

"30 5 * * 3" without the leading zero, would correspond to run every Wednesday at 5:30 AM though. I suspect the fact that it has six fields instead of five is confusing it, and it's interpreting the 5 as a day of the month and as the hour

I don’t know what that means. This is what GPT says. Is it correct?

> the cron expression "0 30 5 * * 3" means that the job will run at 5:30 AM on the 5th day of every month and on Wednesdays.

It's a valid Quartz scheduler task where it starts with the seconds. But right, I shouldn't have asked ChatGPT about cron when it's actually quartz - and indeed with the changed prompt it will describe it correctly.

Edit: actually almost correctly: " - '3' represents the day of the week on which the task will be executed. In this case, the value is 3, which represents Wednesday (Sunday is 1, Monday is 2, and so on)."

Probably the most fundamental problem of ChatGPT is that it refuses (or is even unable) to admit, "I don't know".

I imagine kind of defect can be fixed with more fine tuning / RHLF

It truly has achieved human-level intelligence!

I agree, there's rough edges everywhere. But you can rephrase the question as "how does a cron expression work", and do the interpretation yourself. It returns perfectly sensible results that would enable a perfectly sensible person to quickly write and interpret a cron expression.

This is what I meant when I implied it won't replace you as a thinking agent, but it sure can bring information to you quickly. `man cron` works fine, too, but sometimes it's nice to have one interface, and be able to ask clarifying questions.

Almost certainly true but they'd have to use a new cost function. It's not just about collecting examples where the model should say "I don't know".

FWIW GPT-4 gets it completely correct.

From ChatGPT-3.5:

<prompt>

Carefully inspect the following Cron expression:

```

0 30 5 * * 3

```

1. Which variant of Cron syntax is it?

2. How many elements does this cron string have?

3. What do each of them mean? Say what the value of each element is, and explain what it means.

4. In your own words, explain the overall recurrence pattern described by this string.

</prompt>

<answer>

    This is a standard Unix Cron syntax.

    This cron string has 6 elements.

    Each element represents a specific field in the schedule, and the value of each element has a specific meaning:
        0: The first field represents the seconds, and the value 0 means that the job will run at the exact top of a minute.
        30: The second field represents the minutes, and the value 30 means that the job will run at the 30th minute of an hour.
        5: The third field represents the hours, and the value 5 means that the job will run at the 5th hour of a day.
        *: The fourth field represents the day of the month, and the value * means that the job will run every day of the month.
        *: The fifth field represents the month, and the value * means that the job will run every month of the year.
        3: The sixth field represents the day of the week, and the value 3 means that the job will run only on Wednesday.

    The overall recurrence pattern described by this string is that the job will run every Wednesday at 5:30 AM.

</answer>

It didn't seem to guess that it's Quartz syntax, but it got the rest of it right.

Lol, GPT exposing bugs in the wetware

For reference, here's GPT-4's response:

    1. The given Cron expression follows the Unix/Linux variant of Cron syntax.

    2. This Cron string has six elements.

    3. The elements and their meanings are as follows:
        Element 1 (value: 0): This represents the seconds field and indicates the task will run at 0 seconds.
        Element 2 (value: 30): This represents the minutes field, specifying the task will run at minute 30.
        Element 3 (value: 5): This represents the hours field, indicating the task will run at 5 AM.
        Element 4 (value: ): This represents the days of the month field, where the asterisk () signifies that the task can run on any day of the month.
        Element 5 (value: ): This represents the months field, where the asterisk () signifies that the task can run in any month.
        Element 6 (value: 3): This represents the days of the week field, specifying the task will run on the 3rd day of the week (where Sunday is considered the 0th day, making Tuesday the 3rd day).

    4. The overall recurrence pattern described by this string is that the task will run every Tuesday at 5:30 AM.

A beautiful illustration, thank you.

(If Sunday is 0 then regular math would give 3=Wednesday)

Actually, no. It gave a wrong answer in full confidence then HN analyzed it to expose the bug(s).