Although (attributed) GenAI content is permitted on this site, many community members are hostile towards GenAI, and few are motivated to analyse such content. Please see physics.meta.stackexchange.com/q/14801/123208

LLMs like ChatGPT operate on language structure. As I said 2 years ago on SO, "ChatGPT generates plausible text, consistent with its training data and input, but it doesn't know what it's talking about, and it has no way of representing or evaluating the truth of its utterances. Yes, it can say true things, but it can also say complete nonsense, and it can't tell the difference". Sure, there have been some improvements since then, but the core GPT mechanism is still the same.

Fair point, @J.Delaney However, it's possible to educate people by interacting with them. In contrast, current LLM chat programs don't learn anything from their interactions with users. Yes, their behaviour can be adjusted by RLHF, but that doesn't affect the core GPT process. You need to retrain the transformer to do that, and that's an expensive time-consuming process.

@J.Delaney My core point is that while ChatGPT et al can sound like they know what they're talking about, that's (mostly) illusory. Unlike an AI Expert System (eg Wolfram | Alpha) which is designed to try to represent knowledge in specific fields, an LLM just represents language structures. It can't even evaluate the logical consistency of its utterances, just their statistical plausibility relative to the training data. So you need more than an LLM to build an AI that can be trusted on science topics. Please see the Wolfram links at the end of my linked MSO answer for further details.

@PM2Ring There are approaches to continual training of LLM's, so I don't think that's a major difficultly. As for logical reasoning, if the LLM sees in its training data for example patterns like "A is true, A->B, therefore B is true" repeated many times, then logical consistency can emerge from statistical plausibility (if A is true and A->B, then probably B is true). The layered structure of those transformers seems to give them a very good abstraction capability, that allows them to detect such patterns.

@PM2Ring By the way, speaking of who/what generates complete nonsene ...

@J.Delaney True, but that's kind of a side-effect of learning the language patterns, and it's not robust. As Wolfram points out, even relatively simple tasks like balancing parentheses get difficult for GPT-3 beyond 7 or so levels of nesting.

@J.Delaney Wolfram is a smart guy, but he's also a bit of an arrogant a$$hole, with a reputation for appropriating the work of others, especially his employees. And as for using cellular automata to model physics, IMHO, that's doomed.

I love playing with cellular automata, especially Conway's Game of Life. See conwaylife.com/forums/viewtopic.php?f=2&t=81#p281

However, even in the very simple world of Life, it's easy to construct patterns whose evolution is non-trivial. If you want to know what the pattern does, you have to evolve it, although there may be short-cuts for evolving simple periodic components of the pattern (which algorithms like hashlife can exploit).

So I don't see any computational advantage in trying to model real-world physics as a cellular automata.

FWIW, the conwaylife people have been exploring the patterns from a random "soup" on a 16×16 grid. They found some interesting & surprising new patterns with infinite growth. But of course they cannot exhaustively explore every pattern on a 16×16 grid.

So thinking that CA techniques would somehow help to compute practical, non-trivial real-world physics problems is a joke, IMHO.

@J.Delaney I'm not claiming that LLMs are useless. Even in their current form, they can perform useful tasks, if you're aware of their limitations. And they could certainly be a vital component of more capable AI systems.

However, people (like the OP) are using current LLMs like some kind of all-knowing but error-prone oracle. I guess that can be an interesting way to get some insights into a topic. But it needs expert help to sort the truth from the BS. And few experts are willing to do that kind of work on a voluntary basis...

Balancing parentheses with 7 nesting levels is hard even for humans! that's why we have sophisticated IDE's that find those errors for us. I agree that integrating LLM's with external tools is probably the way to go, but that's what everyone already seem to be doing now anyway with all the talk about agents. (if you ask chatGPT for example to multiply two numbers it will already now use an external calculator)

Conway's Game of life is cool, I particularly admire the guy who made this, I just don't see any meaningful contribution that Wolfram made to this (or any other) field

@PM2Ring I don't think that it is inconceivable that sometime in the not so far future, an LLM trained for example on the entire physics literature, would be able to answer physics questions at an expert level

@J.Delaney Perhaps. But I maintain that it would be more effective with an AI system that somehow tries to represent physics knowledge explicitly, not just as a side-effect. But sure, LLM can be a component of such a system.

Even now, LLMs can do very well on exams. But when they get stuff wrong, they do so in really stupid ways that demonstrate that they don't actually understand what they're saying, and their only "true" knowledge is linguistic.

Scott Aaronson has some info about that on his blog, eg scottaaronson.blog/?p=7209

Scott is an expert in quantum computing. But he recently spent a year working for OpenAI, on projects related to AI safety.

From Scott's blog: My AI Safety Lecture for UT Effective Altruism