The article's perspective reminds me of math class when I was a kid. The teacher didn't want the right answer alone, she wanted to see how I got there. In an absolute sense, the right answer is the right answer with or without my understanding, but that's not how the class was graded. We're approaching a point where AI is to science as my old calculator is to elementary school math -- it's capable of leapfrogging to answers faster and more accurately than we can.
Thing is, though, verification is knowledge itself. And if we can get to a point where the important questions have answers, we can surely have the verifiers explain those to us in ways we can understand them.
This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.
I think this is an artefact of how we train them, and they could be trained to be better at this if we focused on it, but at the same time, it shows its possible for these things to be able to do things they cant explain to us (granted, they can't explain them well - you can bash your way to understanding with them anyway but they aren't good at it)
> This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.
It's amazing to me that language models are still such poor writers and communicators. The mathematics community wrestled with this recently, when many solutions dropped into their laps but were not explained very well...
Thing is, (1) generic LLMs will improve, and (2) their lack of ability in this area has a lot to do with the rhetorical and structural habits they pick up during RLHF/instruction-tuning, which trained them for general-purpose comms. Zero-training base models can presumably be trained specifically to communicate scientific concepts. Interesting times.
(Edited to add: With respect to (1), compare Opus 5 to 5.5. 5 was a terrible communicator, even of software/coding concepts, whereas 5.5 is excellent and leaves very little to be desired.)
> 5.5 is excellent and leaves very little to be desired.
Hm, that may have to do with (faking) confidence. I've often enough with 5.5 still run into cases where it confidently explained something, at the surface it kinda made sense, but the more I dug and verified, the more it became apparent that it was entirely wrong with the most crucial core concepts. It has gotten to a point where you really need a domain expert though to realize this, which makes it tough for using it in areas where you are not an expert and don't have easy access to one either.
I don't think this is the issue. Most people are mediocre mathematicians, too, yet the companies have selected well enough for math literacy in training data (presumably). I think its a matter of training - they are often post trained by other agents (which read everything), human reviewers not paid enough to care (easily wowed by 10 paragraph length essay with correct punctuation), and deterministic scripts (checking for presence of a specific point, so agents are incentivised to write as much as possible to have more chances of hitting the unknown thing the script is looking for)
As far as I know labs try to train the other way, else we'd have monstrously long and useless outputs, but its hard to overcome these natural biases for length
I don't think that's a suitable mental model of how LLMs work. Their output is not just the statistical average of all the training data. If it was then the output would be quite useless.
And in many cases it is not really the new solution that is important. It is the new techniques, abstractions, or other sense-making or investigative tools that made the solution possible that matter.
And the abstractions do still matter, since in many cases it is an issue of tractability. Perhaps an LLM can force its way to a solution through 3 million mind numbingly boring and convoluted steps, but if the number of steps increases exponentially without the right abstraction that means it can only handle a slightly larger problem than a human with pen and paper.
Meaning requires integration. But I assume AI can make meaning of it, now or in the future. I assume the same with people. However I don’t see the point of knowledge without meaning.
I would prefer relation or context instead of integration. Meaning is contextual. Understanding is the process of attaing the meaning or a meaning. Knowledge in this perspective is the final outcome that can be verified in some way or you assume so.
Relation can be a link. I don’t think that’s sufficient for meaning. It needs to have an effect — and arguably a positive one — on the broader coherence.
I don't think we're there yet for science. Perhaps for math proofs, but that's a very different domain ultimately since we know the rules of the game in a way we don't with science.
First math and CS, where formal logic dominate, and AI can check/verify its own answers very quickly. Then physics, downstream of math. Then materials science, which in some cases is practically a branch of physics. Then chemistry more generally. Then biology. Then various social sciences.
A big difference is that in mathematics, you can do all your experimentation purely in theory and at arbitrary speed, while for example chemistry needs a lab and plenty of time.
Math is really super suitable for quick iterations by a machine.
Fields medalist and as Tao listed as one of the most important mathematicians of our age.
Scholz is known to find
„precisely the correct and cleanest way to incorporate all the previously done work and find an elegant formulation for that — and then, because he found really the correct framework, go way beyond the known results“
In a way that man is the complement of an AI, he generates understanding rather than just proofs.
Maybe he is the right template for the mathematicians of the future.
Even Peter Scholz is not always sure of correctness of extremely complicated mathematical statements and welcomes computer assistance in verification of his results.
"
I spent much of 2019 obsessed with the proof of this theorem, almost getting crazy over it. In the end, we were able to get an argument pinned down on paper, but I think nobody else has dared to look at the details of this, and so I still have some small lingering doubts.
"
In a sense, mathematics is not very different from software engineering.
One bit stands out to me: to better understand, Scholz will work the problem backwards. Sometimes that strategy is just so spot on.
--
To my eternal shame, philosophy (like this OC) makes my head hurt. Pretzels my brain, ouroboros like (aka "head up my own butt").
Caveats covered, here's my cave man cold take on two aspects of this topic.
#1 As we're all trying to paraphrase here: answers are great; understanding is even better. I like the word "legibility" (h/t James Scott). The current models are not very legible.
Which directly leads to:
#2 Understanding enables predictions.
If LLMs are stochastic parrots, then pi.dev (GPT + reasoning + ralph loop) is a drunken sailors walk (thru a solution space). A kind of messy heuristic for optimization. With some kind of gate for declaring "done!" Pretty useful.
However, I don't expect Pi.dev to make predictions. That'd require something else. Maybe that something is being worked on; am noob, so I wouldn't know. (I'm still unclear about whatever Jeff Hawkins is doing, in practical terms.)
--
Wrapping up: Automated prediction would be cool. That's very different from finding solutions to problems. And the current cohort of AI can't even explain why, much less make predictions.
Assuming predictions requires understanding. Maybe there are other strategies to generate predictions. Like a meta drunken sailor's walk, with drunken sailors all the way down.
(Someone please correct me as needed. TIA.)
[Aside: That article has an ad for their podcast They Joy of ₍₎. Perfect. I have no idea if it lives up to its promise.]
Surely the way forwards is to take the results obtained by brute force with AI, for example that certain conjectures are true or false, then to use that knowledge to further human understanding.
Knowing that something is true or false will at least stop humans wasting time trying to prove or disprove it, and allow them to find a more elegant proof, or investigate the implications of the true/false result.
Maybe but it might also be useless. Knowing a very large number is the product of two very large primes doesn’t at all help the reader to understand factorization, for example.
"Is science about humans achieving insight, or the archiving of verifiable solutions?"
I wonder if this dichotomy fully captures the value of scientific discoveries. One could argue that their practical impact can be independent of whether humans fully understand how they were reached.
Practical impact has always been linked but ultimately independent of knowledge. You can have very sophisticated technology based on zero actual knowledge or understanding, like roman concrete or borosillicate glass. And you can have a very deep understanding without practical impact, like lasers for decades or galactic algorithms for multiplication.
Indeed, if a machine had told Heisenberg in 1943 (or whatever?) that "the critical mass for an atomic bomb is much less than you think", we could very well have had a different world today, regardless of Heisenberg understanding it or not.
The problem is trading off short term and long term progress. A solution alone can be useful in the short term, but if humans don't also build up understanding it will probably prevent long term progress from happening. It's much more difficult to build on top of something you don't understand.
I would rephrase it as: "Verification is not human knowledge", but that doesn't really matter. We are going to be several steps behind AI as it reaches a level of intelligence far beyond human capabilities, and we are not going to be able to catch up with it. It's like trying to beat AI at chess: we are bound to be left behind as AI continues to expand its knowledge in mathematics, physics, computer science, and other fields. Eventually, no human being will be able to reach that level of knowledge within their lifetime.
No need to start worrying about AI reaching a level of intelligence far beyond human capabilities until it actually reaches the level of humans and assuming that ever happens it could be a very very long way off. All of the intelligence "AI" has right now comes from the human data that it was trained on. It's been given enough power to crunch numbers better than we do, as any computer can, and more data to play with than any one human can retain, but none of that equals intelligence. Speculating on a sci-fi future however, if AI is ever able to create new and more advanced forms of AI, which can also create new and and more advanced forms of AI we'll be outpaced quickly.
As long as humans are in charge it's vital for us to understand what is going on - be it in science or coding, once you give up all knowledge (yes knowledge = power) not just work to AI, you can be manipulated/controlled/fooled with ease.
I have doubts about that. You don't need AI (understood as the current LLMs) to make a powerful chess engine, or computer vision or anything that reacts faster and better than a human. We've had those forever.
A good word predictor happens to be good with closed systems with very defined rules for which a huge training corpus exists and we are suddendly freaking out about how it's gonna become so much better than us at everything that we are basically obsolete.
When in doubt about where AI stands, ask it to create a poem, a joke or a riddle. And see how little actual intelligence it has.
How do we know if/when that happens if it’s beyond us to comprehend. An open ended and broad claim like this can always become true one day, similar to flying cars.
Ive never heard such drivel in my life. Are you a shill? The current "AI" is not intelligent, it can beat humans in certain areas but weve always had that with things like Deep Blue. I dont believe llms will reach human intelligence and certainly not super intelligence.
Thing is, though, verification is knowledge itself. And if we can get to a point where the important questions have answers, we can surely have the verifiers explain those to us in ways we can understand them.
This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.
I think this is an artefact of how we train them, and they could be trained to be better at this if we focused on it, but at the same time, it shows its possible for these things to be able to do things they cant explain to us (granted, they can't explain them well - you can bash your way to understanding with them anyway but they aren't good at it)
It's amazing to me that language models are still such poor writers and communicators. The mathematics community wrestled with this recently, when many solutions dropped into their laps but were not explained very well...
Thing is, (1) generic LLMs will improve, and (2) their lack of ability in this area has a lot to do with the rhetorical and structural habits they pick up during RLHF/instruction-tuning, which trained them for general-purpose comms. Zero-training base models can presumably be trained specifically to communicate scientific concepts. Interesting times.
(Edited to add: With respect to (1), compare Opus 5 to 5.5. 5 was a terrible communicator, even of software/coding concepts, whereas 5.5 is excellent and leaves very little to be desired.)
Hm, that may have to do with (faking) confidence. I've often enough with 5.5 still run into cases where it confidently explained something, at the surface it kinda made sense, but the more I dug and verified, the more it became apparent that it was entirely wrong with the most crucial core concepts. It has gotten to a point where you really need a domain expert though to realize this, which makes it tough for using it in areas where you are not an expert and don't have easy access to one either.
As far as I know labs try to train the other way, else we'd have monstrously long and useless outputs, but its hard to overcome these natural biases for length
And the abstractions do still matter, since in many cases it is an issue of tractability. Perhaps an LLM can force its way to a solution through 3 million mind numbingly boring and convoluted steps, but if the number of steps increases exponentially without the right abstraction that means it can only handle a slightly larger problem than a human with pen and paper.
> https://en.wikipedia.org/wiki/Hierarchy_of_the_sciences
First math and CS, where formal logic dominate, and AI can check/verify its own answers very quickly. Then physics, downstream of math. Then materials science, which in some cases is practically a branch of physics. Then chemistry more generally. Then biology. Then various social sciences.
Math is really super suitable for quick iterations by a machine.
Fields medalist and as Tao listed as one of the most important mathematicians of our age.
Scholz is known to find „precisely the correct and cleanest way to incorporate all the previously done work and find an elegant formulation for that — and then, because he found really the correct framework, go way beyond the known results“
In a way that man is the complement of an AI, he generates understanding rather than just proofs.
Maybe he is the right template for the mathematicians of the future.
https://www.quantamagazine.org/peter-scholze-and-the-future-...
See the Liquid tensor experiment:
https://xenaproject.wordpress.com/2020/12/05/liquid-tensor-e...
https://www.ma.imperial.ac.uk/~buzzard/xena/pdfs/liquid_tens...
" I spent much of 2019 obsessed with the proof of this theorem, almost getting crazy over it. In the end, we were able to get an argument pinned down on paper, but I think nobody else has dared to look at the details of this, and so I still have some small lingering doubts. "
In a sense, mathematics is not very different from software engineering.
More precisely: Curry–Howard correspondence
https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...
One bit stands out to me: to better understand, Scholz will work the problem backwards. Sometimes that strategy is just so spot on.
--
To my eternal shame, philosophy (like this OC) makes my head hurt. Pretzels my brain, ouroboros like (aka "head up my own butt").
Caveats covered, here's my cave man cold take on two aspects of this topic.
#1 As we're all trying to paraphrase here: answers are great; understanding is even better. I like the word "legibility" (h/t James Scott). The current models are not very legible.
Which directly leads to:
#2 Understanding enables predictions.
If LLMs are stochastic parrots, then pi.dev (GPT + reasoning + ralph loop) is a drunken sailors walk (thru a solution space). A kind of messy heuristic for optimization. With some kind of gate for declaring "done!" Pretty useful.
However, I don't expect Pi.dev to make predictions. That'd require something else. Maybe that something is being worked on; am noob, so I wouldn't know. (I'm still unclear about whatever Jeff Hawkins is doing, in practical terms.)
--
Wrapping up: Automated prediction would be cool. That's very different from finding solutions to problems. And the current cohort of AI can't even explain why, much less make predictions.
Assuming predictions requires understanding. Maybe there are other strategies to generate predictions. Like a meta drunken sailor's walk, with drunken sailors all the way down.
(Someone please correct me as needed. TIA.)
[Aside: That article has an ad for their podcast They Joy of ₍₎. Perfect. I have no idea if it lives up to its promise.]
Knowing that something is true or false will at least stop humans wasting time trying to prove or disprove it, and allow them to find a more elegant proof, or investigate the implications of the true/false result.
I wonder if this dichotomy fully captures the value of scientific discoveries. One could argue that their practical impact can be independent of whether humans fully understand how they were reached.
See: most of pharma
A good word predictor happens to be good with closed systems with very defined rules for which a huge training corpus exists and we are suddendly freaking out about how it's gonna become so much better than us at everything that we are basically obsolete.
When in doubt about where AI stands, ask it to create a poem, a joke or a riddle. And see how little actual intelligence it has.