GPT-4 and professional benchmarks: the wrong answer to the wrong question

Memorization is absolutely the most valuable part of GPT, for me. I can get natural language responses to documentation, basic scripting / sysadmin, and API questions much more easily than searching other ways.

While this is an academic interest point, and rightly tamps down on hype around replacing humans, it doesn't dissuade what I think are most peoples' basic use case: "I don't know or don't remember how to do X, can you show me?"

This is finally a good enough "knowledge reference engine" that I can see being useful to those very people it is over hyped to replace.

The main problem with using GPT-3 (and maybe 4 I dunno) in that way is it will happily bullshit you to the point of making up fake references. For example it quoted me "Section 6.2" of the Go Programming Language Specification to support its answer when I asked it how a particular conversion is specified.