Want to wade into the sandy surf of the abyss? Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid: Welcome to the Stubsack, your first port of call for learning fresh Awful youāll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cutānāpaste it into its own post ā thereās no quota for posting and the bar really isnāt that high.
The post Xitter web has spawned soo many āesotericā right wing freaks, but thereās no appropriate sneer-space for them. Iām talking redscare-ish, reality challenged āculture criticsā who write about everything but understand nothing. Iām talking about reply-guys who make the same 6 tweets about the same 3 subjects. Theyāre inescapable at this point, yet I donāt see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldnāt be surgeons because they didnāt believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I canāt escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard for starting this.)
One thing Iāve heard repeated about OpenAI is that āthe engineers donāt even know how it works!ā and Iām wondering what the rebuttal to that point is.
While it is possible to write near-incomprehensible code and make an extremely complex environment, there is no reason to think there is absolutely no way to derive a theory of operation especially since any part of the whole runs on deterministic machines. And yet Iāve heard this repeated at least twice (one was on the Panic World pod, the other QAA).
I would believe that itās possible to build a system so complex and with so little documentation that on its surface is incomprehensible but the context in which the claim is made is not that of technical incompetence, rather the claim is often hung as bait to draw one towards thinking that maybe we could bootstrap consciousness.
It seems like magical thinking to me, and a way of saying one or both of āwe didnāt write shit down and therefore have no idea how the functionality worksā and āwe do not practically have a way to determine how a specific output was arrived at from any given prompt.ā The first might be in part or on a whole unlikely as the system would need to be comprehensible enough so that new features could get added and thus engineers would have to grok things enough to do that. The second is a side effect of not being able to observe all actual input at the time a prompt was made (eg training data, user context, system context could all be viewed as implicit inputs to a function whose output is, say, 2 seconds of Coke Ad slop).
Anybody else have thoughts on countering the magic āthe engineers donāt know how it works!ā?
well, I canāt counter it because I donāt think they do know how it works. the theory is shallow yet the outputs of, say, an LLM are of remarkably high quality in an area (language) that is impossibly baroque. the lack of theory and fundamental understanding presents a huge problem for them because it means āimprovementsā can only come about by throwing money and conventional engineering at their systems. this is what Iāve heard from people in the field for at least ten years.
to me that also means it isnāt something that needs to be countered. itās something the context of which needs to be explained. itās bad for the ai industry that they donāt know what theyāre doing
EDIT: also, when i say the outputs are of high quality, what i mean is that they produce coherent and correct prose. im not suggesting anything about the utility of the outputs
I think I heard a good analogy for this in Well Thereās Your Problem #164.
One topic of the episode was how people didnāt really understand how boilers worked, from a thermal mechanics point if view. Still steam power was widely used (e.g. on river boats), but much of the engineering was guesswork or based on patently false assumptions with sometimes disastrous effects.
another analogy might be an ancient builder who gets really good at building pyramids, and by pouring enormous amounts of money and resources into a project manages to build a stunningly large pyramid. āim now going to build something as tall as what will be called the empire state building,ā he says.
problem: he has no idea how to do this. clearly some new building concepts are needed. but maybe he can figure those out. in the meantime heās going to continue with this pyramid design but make them even bigger and bigger, even as the amount of stone required and the cost scales quadratically, and just say heās working up to the reallyyyyy big buildingā¦
I mean if you ever toyed around with neural networks or similar ML models you know itās basically impossible to divine what the hell is going on inside by just looking at the weights, even if you try to plot them or visualise in other ways.
Thereās a whole branch of ML about explainable or white-box models because it turns out you need to put extra care and design the system around being explainable in the first place to be able to reason about its internals. Thereās no evidence OpenAI put any effort towards this, instead focusing on cool-looking outputs they can shove into a presser.
In other words, āengineers donāt know how it worksā can have two meanings - that theyāre hitting computers with wrenches hoping for the best with no rhyme or reason; or that they donāt have a good model of what makes the chatbot produce certain outputs, i.e. just by looking at the output itās not really possible to figure out what specific training data it comes from or how to stop it from producing that output on a fundamental level. The former is demonstrably false and almost a strawman, I donāt know who believes that, a lot of people that work on OpenAI are misguided but otherwise incredibly clever programmers and ML researchers, the sheer fact that this thing hasnāt collapsed under its own weight is a great engineering feat even if externalities it produces are horrifying. The latter is, as far as Iām aware, largely true, or at least I havenāt seen any hints that would falsify that. If OpenAI satisfyingly solved the explainability problem itād be a major achievement everyone would be talking about.
Another ironic pointā¦ Lesswrongerās actually do care about ML interpretability (to the extent they care about real ML at all; and as a solution to making their God AI serve their whims not for anything practical). A lack of interpretability is a major problem (like irl problem, not just scifi skynet problem) in ML, you can models with racism or other bias buried in them and not be able to tell except by manually experimenting with your model with data from outside the training set. But Sam Altman has turned it from a problem into a humble brag intended to imply their LLM is so powerful and mysterious and bordering on AGI.
Not gonna lie, I didnāt entirely get it either until someone pointed me at a relevant xkcd that I had missed.
Also I was somewhat disappointed in the QAA teamās credulity towards the AI hype, but their latest episode was an interview with the writer of that āAGI as conspiracy theoryā piece from last(?) week and seemed much more grounded.
the mention in QAA came during that episode and I think there it was more illustrative about how a person can progress to conspiratorial thinking about AI. The mention in Panic World was from an interview with Ed Zitronās biggest fan, Casey Newton if I recall correctly.