Man the models can’t store verbatim its training data, the amount of data is turned into a model that is hundreds or thousands of times smaller than the original source data. If it was capable of simply recovering everything that it was trained on this would be some magical compression algorithm and that by itself would be extremely impressive.
Oh ok, you want to claim this is compressing the entirety of the internet in a model that isn’t even 1 terabyte of data and be unimpressed that is something.
But it isn’t compression. It is a mathematical fact that neural networks are universal function approximators, this is undisputed, and analytic functions are continuous so to be an analytical function approximator it must be able to fill in the gaps between discrete data points by itself, which necessarily means spiting out data outside of the input distribution, data it has not seen.
It is a partial analogy, it takes into consideration the outputs which are related to some specific training data and disconsiders the outputs which cannot be directly related to any specific training data.
For example, make up a new meme template and a new joke on the spot, it couldn’t have seen it before if you make sure your joke and template are new. If the AI can explain it then compression is a horrendous analogy.
Lossy compression explains outputs being similar but not identical when trying to recover the original data, it doesn’t explain brand new content that makes sense standalone. Imagine a lossy audio compression resulting in a brand new song midway through playback, or a lossy image compression resulting in a brand new coherent image being overlayed onto some pixels of the original image. That is not what happens, lossy audio compression results in noise, lossy image compression results in noise, not in coherent unheard songs and unseen images.
Not sure why you feel the need to put words in my mouth. It wasn’t trained on “the entirety of the Internet,” but rather less than a terabyte of it. So yeah, that would probably take up less than a terabyte.
Arguing over this is just dumb, you can yourself take any picture you want at this very moment or come up with a brand new meme template on the spot and upload it to ChatGPT to see you are wrong, it is free btw.
They do not store anything verbatim; They instead store the directions in which various words and related concepts relate to one another in some gigantic multidimensional space.
I highly suggest you go learn what they actually do before you continue talking out of your ass about them
You said it matches text to its training data, which it does not do.
Your single-phrase statement only works for very short, non-repetitive phrases. As soon as your phrase repeats a token more than a few times, the statistics for the tokens change and could result in nonsensical output that repeats through subsections of the training data.
And even then for that single non-repetitive phrases, the reason you would get that single phrase back is not because it would be “matching on” the phrase. It is because the token weights would effectively encode that the statistical likelihood of the “next token” in the generated output is 100% for a given token when the evaluated token precedes it in the training phrase. Or in other words: Your training data being a single phrase maniplates the statistics so that the most likely output is that single phrase.
However, that is a far cry from simple “matching” against the training data. Which is what you said it does.
Analysis. It uses it, but not by “matching it”. The training data is not included in the final model. No GPT can access its training data at runtime.
Training analyzes the contents of the training data and creates a statistical model representing the likelihoods of various tokens based on a complex series of mathematical transformations that encode various attributes of the tokens making up the training data.
3Blue1Brown has a great series on the actual math behind it, I would highly recommend educating yourself on what GPTs actually do. It’s way more interesting than simple matching.
That’s not how GPTs work
That’s literally how they work
Man the models can’t store verbatim its training data, the amount of data is turned into a model that is hundreds or thousands of times smaller than the original source data. If it was capable of simply recovering everything that it was trained on this would be some magical compression algorithm and that by itself would be extremely impressive.
Congratulations on discovering compression
Oh ok, you want to claim this is compressing the entirety of the internet in a model that isn’t even 1 terabyte of data and be unimpressed that is something.
But it isn’t compression. It is a mathematical fact that neural networks are universal function approximators, this is undisputed, and analytic functions are continuous so to be an analytical function approximator it must be able to fill in the gaps between discrete data points by itself, which necessarily means spiting out data outside of the input distribution, data it has not seen.
TBF, compression is related to ML. Hence, the Hutter Prize. Thinking of LLMs as lossy compression algorithms is a decent analogy.
It is a partial analogy, it takes into consideration the outputs which are related to some specific training data and disconsiders the outputs which cannot be directly related to any specific training data.
For example, make up a new meme template and a new joke on the spot, it couldn’t have seen it before if you make sure your joke and template are new. If the AI can explain it then compression is a horrendous analogy.
Lossy compression explains outputs being similar but not identical when trying to recover the original data, it doesn’t explain brand new content that makes sense standalone. Imagine a lossy audio compression resulting in a brand new song midway through playback, or a lossy image compression resulting in a brand new coherent image being overlayed onto some pixels of the original image. That is not what happens, lossy audio compression results in noise, lossy image compression results in noise, not in coherent unheard songs and unseen images.
Not sure why you feel the need to put words in my mouth. It wasn’t trained on “the entirety of the Internet,” but rather less than a terabyte of it. So yeah, that would probably take up less than a terabyte.
Then why did I just make this meme up right now and chat gpt can explain it?
https://i.ibb.co/NYHRnTY/Screenshot-20240531-072008-Chat-GPT.jpg
Arguing over this is just dumb, you can yourself take any picture you want at this very moment or come up with a brand new meme template on the spot and upload it to ChatGPT to see you are wrong, it is free btw.
They do not store anything verbatim; They instead store the directions in which various words and related concepts relate to one another in some gigantic multidimensional space.
I highly suggest you go learn what they actually do before you continue talking out of your ass about them
If you trained a GPT on a single phrase, all you’d get out of it would be the single phrase.
The mechanism of storage doesn’t need to be just the verbatim source material, which is not even close to what I said.
You said it matches text to its training data, which it does not do.
Your single-phrase statement only works for very short, non-repetitive phrases. As soon as your phrase repeats a token more than a few times, the statistics for the tokens change and could result in nonsensical output that repeats through subsections of the training data.
And even then for that single non-repetitive phrases, the reason you would get that single phrase back is not because it would be “matching on” the phrase. It is because the token weights would effectively encode that the statistical likelihood of the “next token” in the generated output is 100% for a given token when the evaluated token precedes it in the training phrase. Or in other words: Your training data being a single phrase maniplates the statistics so that the most likely output is that single phrase.
However, that is a far cry from simple “matching” against the training data. Which is what you said it does.
If it doesn’t use its training data, what’s the training data for?
Analysis. It uses it, but not by “matching it”. The training data is not included in the final model. No GPT can access its training data at runtime.
Training analyzes the contents of the training data and creates a statistical model representing the likelihoods of various tokens based on a complex series of mathematical transformations that encode various attributes of the tokens making up the training data.
3Blue1Brown has a great series on the actual math behind it, I would highly recommend educating yourself on what GPTs actually do. It’s way more interesting than simple matching.
God forbid I use simpler language to describe what it does.
It’s pattern matching with extra steps.
Simpler language is fine when it’s accurate.
Your simplification is inaccurate and could mislead people into thinking GPTs are just advanced regex matching engines.
They are not. They are closer to autocorrect on steroids.