This is deeply disturbing on many levels, which I will not dive into here, but perhaps in another post / topic, because I think there's an interesting conversation to be had here around AI proofs...
This is deeply disturbing on many levels, which I will not dive into here, but perhaps in another post / topic, because I think there's an interesting conversation to be had here around AI proofs and the future of mathematics in general.
Nonetheless, OpenAI disgusts me.
While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.
Yeah, right. As of the initial publishing of the paper, it only had 16 citations, which is laughable for a proof of this size, not to mention it didn't credit the team that has been working on this problem for years and were close to solving it, Diego Córdoba and Luis Martínez-Zoroa.
OpenAI now amended the paper to include additional references, but at best, this is blatant plagiarism. Why do we allow this? Why are they not held accountable? Why can years or decades of hard work be stolen so easily? Breaks my heart.
Here is a link to the originally published paper with its 16 references. 16 references for 165 pages.
There’s a lot of backlash against OpenAI on this and, to be clear, I really dislike OpenAI as a company. But knowing how ai training works, the way I read that sentence from their article is a...
There’s a lot of backlash against OpenAI on this and, to be clear, I really dislike OpenAI as a company.
But knowing how ai training works, the way I read that sentence from their article is a CYA. Probably maybe there can have been some formulation in the final result that was influenced by this training because it was part of the chain and they used 10k agents. But, it’s not at all realistic to think that this was actually meaningfully influenced by this usage, and it’s so ridiculously far from being plagiarised that it doesn’t make sense to write what you wrote.
Like, people don’t understand how little influence individual datapoints have over ai training. It’s microscopic. The whole point is there’s billions of these datapoints to learn from… usually what has the most influence gets weighted out a ton.
I recall once in grad school, a prospective professor was showing their work (it was even fluid related too). Part of it involved fitting data to a line. They used a polynomial with like 8 terms,...
I recall once in grad school, a prospective professor was showing their work (it was even fluid related too). Part of it involved fitting data to a line. They used a polynomial with like 8 terms, none of the coefficients had any connection to real world properties, not derived from first principles at all. But it drew the line well, and they were proud of it.
One professor there in the same subfield had derived an equation from first principles that fairly accurately represented those graph types based on actual physical properties reduced to only one fitting parameter. It wouldn't draw the line quite as well though.
Nonetheless, the field seemed to favor these ugly polynomials when it came to what was used in papers. Computer simulations with more silicon, more digits, not looking at the real world.
So what I've been keen on finding out is, are these AI solutions more like the prospective prof's, where they just threw more numbers on the board until everything fit expectations regardless of the underlying physics, or our prof, where they actually model the behavior based on principles? Is there a novel model, or are they over fitting the data to get a solution? The "smooth" aspect seems analogous at the least.
(Fun aside: my profs also complained of how scientists didn't get how to mathematically model vortexes accurately, so of course they show up in the solution.)
By design, they are the first type. The Universal approximation theorem, which is the foundational theory in ML/LLMs, is all about universal approximation without care for subject matter knowledge...
are these AI solutions more like the prospective prof's, where they just threw more numbers on the board until everything fit expectations regardless of the underlying physics, or our prof, where they actually model the behavior based on principles?
By design, they are the first type. The Universal approximation theorem, which is the foundational theory in ML/LLMs, is all about universal approximation without care for subject matter knowledge or understanding. Quoting wikipedia:
the universal approximation theorems (UATs) state that neural networks with a certain structure can, in principle, approximate any continuous function to any desired degree of accuracy.
Universality is achieved by increasing the number of neurons in the hidden layer, making the network "wider." Other versions of the theorem show that universality can also be achieved by keeping the network's width fixed but increasing its number of layers, making it "deeper."
That's kind of conflating multiple different concepts. For one, the foundational theory in ML is not the universal approximation theorem. Neural networks are just one area; most of ML is about...
That's kind of conflating multiple different concepts.
For one, the foundational theory in ML is not the universal approximation theorem. Neural networks are just one area; most of ML is about convex optimization, for one. A gradient forest or linear regression has no relationship to a neural network, and certainly does not have the same number of parameters. There's the whole field of generative models (which, unfortunately, now has a namespace collision with generative AI), which are models where you explicitly define the prior probabilities rather than just trying to recreate the posterior probability.
As an aside, I would also say that I wouldn't even consider that the "foundational theory" in neural networks. It's a somewhat trivial proposition. What is more relevant is empirical observation that local minima are unlikely to occur in high-dimensional spaces, since the obvious theoretical issue with high dimensional models is that the optimization space is absolutely NOT concave, and therefore optimizing it is a NP-complete problem. What makes them work is that you can use a greedy approach and mostly get away with it in high dimensions.
Secondly, although that has relevance to the generated text, the way that an LLM would go about solving something like Navier-Stokes has nothing to do with it. As you can see from their whitepaper, they were iterating on closed-form solutions and made heavy use of Lean to ensure it was formally correct.
The Navier-Stokes equations account for the fact that fluids can have viscosity, or friction. (Fluids with more viscosity, like honey, flow slowly, while those with less viscosity, like water, flow more quickly.) A simpler, related set of equations called the Euler equations describe fluids with zero viscosity, which flow without friction. [...] in 2013, when Thomas Hou of the California Institute of Technology and Guo Luo, now at the Hang Seng University of Hong Kong, derived a groundbreaking result showing that the Euler equations can “blow up,” as mathematicians like to say, in a cylinder if the top and bottom halves are set spinning in opposite directions.
[...]
There are a few intellectual steps to get from there to the present day. The first is the question of a boundary. The Millennium Prize version of the problem asks what happens in three-dimensional space that extends indefinitely in all directions. [...] The next major step involves modeling the forces that cause fluids to move. [...] [Mathematicians have] sometimes tried to introduce awkward, ungainly forcing functions to get fluids to behave in odd ways. But the Millennium Prize version of the problem asks what can happen when the forcing function is mathematically well-behaved, or “smooth.”
[...]
By 2023, the pair [Martínez-Zoroa and Córdoba] had proved that a version of the Euler equations with a messy forcing function displayed singularities. [...] In broad outline, the pair’s technique relies on creating an infinite sequence of “layers,” each of which is a non-singular solution to the equation they are studying. [...] They then combine those solutions in what Martínez-Zoroa calls an “infinite cascade” to produce a new solution.
That new solution, they showed, contains the desired singularity. However, even though each individual layer relies on a smooth forcing function, combining them together can cause the forcing function to have undesirable mathematical properties. The remaining hurdle was to figure out how to create a similar infinite cascade that resulted not only in a singularity, but also in a smooth forcing function.
That’s the step that both competing AI groups appear to have had success with.
This is deeply disturbing on many levels, which I will not dive into here, but perhaps in another post / topic, because I think there's an interesting conversation to be had here around AI proofs and the future of mathematics in general.
Nonetheless, OpenAI disgusts me.
Yeah, right. As of the initial publishing of the paper, it only had 16 citations, which is laughable for a proof of this size, not to mention it didn't credit the team that has been working on this problem for years and were close to solving it, Diego Córdoba and Luis Martínez-Zoroa.
OpenAI now amended the paper to include additional references, but at best, this is blatant plagiarism. Why do we allow this? Why are they not held accountable? Why can years or decades of hard work be stolen so easily? Breaks my heart.
Here is a link to the originally published paper with its 16 references. 16 references for 165 pages.
There’s a lot of backlash against OpenAI on this and, to be clear, I really dislike OpenAI as a company.
But knowing how ai training works, the way I read that sentence from their article is a CYA. Probably maybe there can have been some formulation in the final result that was influenced by this training because it was part of the chain and they used 10k agents. But, it’s not at all realistic to think that this was actually meaningfully influenced by this usage, and it’s so ridiculously far from being plagiarised that it doesn’t make sense to write what you wrote.
Like, people don’t understand how little influence individual datapoints have over ai training. It’s microscopic. The whole point is there’s billions of these datapoints to learn from… usually what has the most influence gets weighted out a ton.
I recall once in grad school, a prospective professor was showing their work (it was even fluid related too). Part of it involved fitting data to a line. They used a polynomial with like 8 terms, none of the coefficients had any connection to real world properties, not derived from first principles at all. But it drew the line well, and they were proud of it.
One professor there in the same subfield had derived an equation from first principles that fairly accurately represented those graph types based on actual physical properties reduced to only one fitting parameter. It wouldn't draw the line quite as well though.
Nonetheless, the field seemed to favor these ugly polynomials when it came to what was used in papers. Computer simulations with more silicon, more digits, not looking at the real world.
So what I've been keen on finding out is, are these AI solutions more like the prospective prof's, where they just threw more numbers on the board until everything fit expectations regardless of the underlying physics, or our prof, where they actually model the behavior based on principles? Is there a novel model, or are they over fitting the data to get a solution? The "smooth" aspect seems analogous at the least.
(Fun aside: my profs also complained of how scientists didn't get how to mathematically model vortexes accurately, so of course they show up in the solution.)
By design, they are the first type. The Universal approximation theorem, which is the foundational theory in ML/LLMs, is all about universal approximation without care for subject matter knowledge or understanding. Quoting wikipedia:
a.k.a., just chuck more variables at it.
That's kind of conflating multiple different concepts.
For one, the foundational theory in ML is not the universal approximation theorem. Neural networks are just one area; most of ML is about convex optimization, for one. A gradient forest or linear regression has no relationship to a neural network, and certainly does not have the same number of parameters. There's the whole field of generative models (which, unfortunately, now has a namespace collision with generative AI), which are models where you explicitly define the prior probabilities rather than just trying to recreate the posterior probability.
As an aside, I would also say that I wouldn't even consider that the "foundational theory" in neural networks. It's a somewhat trivial proposition. What is more relevant is empirical observation that local minima are unlikely to occur in high-dimensional spaces, since the obvious theoretical issue with high dimensional models is that the optimization space is absolutely NOT concave, and therefore optimizing it is a NP-complete problem. What makes them work is that you can use a greedy approach and mostly get away with it in high dimensions.
Secondly, although that has relevance to the generated text, the way that an LLM would go about solving something like Navier-Stokes has nothing to do with it. As you can see from their whitepaper, they were iterating on closed-form solutions and made heavy use of Lean to ensure it was formally correct.
[...]
[...]