• 15 Posts
  • 186 Comments
Joined 2 years ago
cake
Cake day: July 19th, 2023

help-circle
  • As the classic film Network points out, the Saudi money is the end of the road; there arenā€™t any richer or more gullible large wealth funds who will provide further cash. So OpenAI could be genuinely out of greater fools financing after another year of wasting Somebody Elseā€™s Money. This crash has removed ā€œlargeā€ from the front of any other wealth fund that might have considered bailing them out. The Stargate gamble could still work out, but so far I think tiā€™s only transferred bag-holding responsibilities from Microsoft to Oracle.

    Another path is to deflate nVidiaā€™s cap. At first blush, this seems impossible to me; nVidiaā€™s business behavior is so much worse than that of competitors Intel or Imagination yet they have generally never lost faith from their core gaming laity, and as long as nVidia holds 20-30% of the gaming GPU market they will always have a boutique niche with cap at least comparable to e.g. their competitor AMD. But GPUs have been treated as currency among datacenter owners, and a market crash could devalue the piles of nVidia GPUs which some datacenter owners have been using as collateral for purchasing land, warehouses, machines, more GPUs, etc. nVidia isnā€™t the only bag-holder here, though, and since they donā€™t really want to play loan-shark and repossess a datacenter for dereliction, odds are good that theyā€™ll survive even if theyā€™re no longer king of the hill. The gold rush didnā€™t work out? Too bad, no returns allowed on shovels or snow gear.

    Side note: If folks just wanted to know whether tech in general is hurt by this, then yes, look at Teslaā€™s valuation. Tesla is such a cross-cutting big-ticket component of so many ETFs that basically every retirement scheme took a hit from Tesla taking a hit. The same thing will happen with nVidia and frankly retirement-fund managers should feel bad for purchasing so much of what any long-term investor would consider to be meme stocks. (I donā€™t hold either TSLA or NVDA stocks.)

    I hope this makes sense. I donā€™t post with this candor when Iā€™m well-rested and sober.



  • Today on the orange site, an AI bro is trying to reason through why people think heā€™s weird for not disclosing his politics to people heā€™s trying to be friendly with. Previously, he published a short guide on how to talk about politics, which ā€” again, very weird, no possible explanation for this ā€” nobody has adopted. Donā€™t worry, heā€™s well-read:

    So far Iā€™ve only read Harry Potter and The Methods of Rationality, but can say it is an excellent place to start.

    The thread is mostly centered around one or two pearl-clutching conservatives who donā€™t want their beliefs examined:

    I find it astonishing that anyone would ask, [ā€œwho did you vote for?ā€] ā€¦ In my social circle, anyway, the taboo on this question is very strong.

    To which the top reply is my choice sneer:

    In my friend group itā€™s clear as day: either you voted to kill and deport other people in the friend group or you didnā€™t. Pretty obvious the group would like to know if youā€™re secretly interested in their demise.



  • Strange is a trooper and her sneer is worth transcribing. From about 22:00:

    So letā€™s go! Upon saturating my brain with as much background information as I could, there was really nothing left to do but fucking read this thing, all six hundred thousand words of HPMOR, really the road of enlightenment that they promised it to be. After reading a few chapters, a realization that I found funny was, ā€œOh. Oh, this is definitely fanfiction. Everyone said [laughing and stuttering] everybody that said that this is basically a real novel is lying.ā€ People lie on the Internet? No fucking way. It is telling that even the most charitable reviews, the most glowing worshipping reviews of this fanfiction call it ā€œunfinished,ā€ call it ā€œa first draft.ā€

    A shorter sneer for the back of the hardcover edition of HPMOR at 26:30 or so:

    Itā€™s extremely tiring. I was surprised by how soul-sucking it was. It was unpleasant to force myself beyond the first fifty thousand words. It was physically painful to force myself to read beyond the first hundred thousand words of this ā€“ let me remind you ā€“ six-hundred-thousand-word epic, and I will admit that at that point I did succumb to skimming.

    Her analysis is familiar. She recognized that Harry is a self-insert, that the out-loud game theory reads like Death Note parody, that chapters are only really related to each other in the sense that they were written sequentially, that HPMOR is more concerned with sounding smart than being smart, that HPMOR is yet another entry in a long line of monarchist apologies explaining why this new Napoleon wonā€™t fool us again, and finally that itā€™s a bad read. 31:30 or so:

    Itā€™s absolutely no fucking fun. Itā€™s just absolutely dry and joyless. It tastes like sand! I mean, maybe itā€™s Yudkowskyā€™s idea of fun; he spent five years writing the thing after all. But it just [struggles for words] reading this thing, it feels like chewing sand.


  • Anecdote: I gave up on COBOL as a career after beginning to learn it. The breaking point was learning that not only does most legacy COBOL code use go-to statements but that there is a dedicated verb which rewrites go-to statements at runtime and is still supported on e.g. the IBM Enterprise COBOL for z/OS platform that SSA is likely using: ALTER.

    When I last looked into this a decade ago, there was a small personal website last updated in the 1990s that had advice about how to rewrite COBOL to remove GOTO and ALTER verbs; if anybody has a link, Iā€™d appreciate it, as I can no longer find it. It turns out that the best ways of removing these spaghetti constructions involve multiple rounds of incremental changes which are each unlikely to alter the codeā€™s behavior. Translations to a new language are doomed to failure; even Java is far too structured to directly encode COBOL control flow, and the time would be better spent on abstract specification of the system so that it can be rebuilt from that specification instead. This is also why IBM makes bank selling COBOL emulators.






  • Hereā€™s some food for thought; ha ha, only serious. What if none of this is new?

    If this is a dealbreaker today, then it should have been a dealbreaker over a decade ago, when Google first rolled out Knowledge panels, which were also often inaccurate and unhelpful.

    If this isnā€™t acceptable from Google, then it shouldnā€™t be acceptable from DuckDuckGo, which has the same page-one results including an AI summary and panels, nor any other search engines. If summaries are unacceptable from Gemini, which has handily topped the leaderboards for weeks, then itā€™s not acceptable using models from any other vendor, including Alibaba, High-Flyer, Meta, Microsoft, or Twitter.

    If fake, hallucinated, confabulated, or synthetic search results are ruining the Web today, then they were ruining the Web over two decades ago and have not lessened since. The economic incentives and actors have shifted slightly, but the overall goal of fraudulent clicks still underlies the presentation.

    If machine learning isnā€™t acceptable in collating search results today, then search engines would not exist. The issue is sheer data; ever since about 1991, before the Web existed, there has been too much data available on the Internet to search exhaustively and quickly. The problem is recursive: when a user queries a popular search engine, their results are populated by multiple different searchers using different techniques to learn what is relevant, because no one search strategy works at scale for most users asking most things.

    Iā€™m not saying this to defend Google but to steer yā€™all away from uncanny-valley reactionism. The search-engine business model was always odious, but we were willing to tolerate it because it was very inaccurate and easy to game, like a silly automaton which obeys simple rules. Now we are approaching the ability to conduct automated reference interviews and suddenly we have an ā€œoops, all AI!ā€ moment as if it werenā€™t always generative AI from the beginning.




  • Why is Microsoft canceling a Gigawatt of data center capacity while telling everybody that it didnā€™t have enough data centers to handle demand for its AI products? I suppose thereā€™s one way of looking at it: that Microsoft may currently have a capacity issue, but soon wonā€™t, meaning that further expansion is unnecessary.

    This is precisely it. Internally, Microsoftā€™s SREs perform multiple levels of capacity planning, so that a product might individually be growing and requiring more resources over the next few months, but a department might be overall shrinking and using less capacity over the next few years. A datacenter requires at least 4yrs of construction before its capacity is available (usually more like 5yrs) which is too long of a horizon for any individual productā€¦unless, of course, your product is ChatGPT and it requires a datacenterā€™s worth of resources. Even if OpenAI were siloed from Microsoft or Azure, they would still know that OpenAI is among their neediest customers and include them in planning.

    Source: Scuttlebutt from other SREs, mostly. An analogous situation happened with Googleā€™s App Engine product: App Engineā€™s biggest users impacted App Engineā€™s internal capacity planning at the product level, which impacted datacenter planning because App Engine was mostly built from one big footprint in one little Oklahoma datacenter.

    Conclusion: Microsoftā€™s going to drop OpenAI as a customer. Oracleā€™s going to pick up the responsibility. Microsoft knows that thereā€™s no money to be made here, and is eager to see how expensive that lesson will be for Oracle; Oracle is fairly new to the business of running a public cloud and likely thinks they can offer a better platform than Azure, especially when fueled by delicious Arabian oil-fund money. Folks may want to close OpenAI accounts if they donā€™t want Oracle billing them someday.


  • Reading through the docket, he is entitled to a hearing for relief and has a modicum of standing due to the threat of deportation from the USA to China; itā€™s not unreasonable to go to federal court. The judge was fairly courteous in referring him to the Pro Se Project a week ago. Iā€™m a little jealous of how detached he is from reality; from 36(a) of the Amended Complaint:

    The Plaintiff asserts that completing a Ph.D. in Health Services Research significantly increases earning potential. The average salary for individuals with such a Ph.D. is $120,000 annually, compared to $30,000 annually in China, where Plaintiffā€™s visa cancellation forces him to seek employment. Over an estimated 30-year working career, this represents a lifetime income loss of $2,700,000.

    He really went up to the judge and said, ā€œyour honor, my future career is dependent on how well I prompt ChatGPT, but statistically I should be paid more if I have a second doctorate,ā€ and the judge patted him on his head and gave him a lollipop for being so precocious.


  • Well, how do you feel about robotics?

    On one hand, I fully agree with you. AI is a rebranding of cybernetics, and both fields are fundamentally inseparable from robotics. The goal of robotics is to create artificial slaves who will labor without wages or solidarity. Weā€™re all ethically obliged to question the way that robots affect our lives.

    On the other hand, machine learning (ML) isnā€™t going anywhere. In my oversimplification of history, ML was originally developed by Markov and Shannon to make chatbots and predict the weather; we still want to predict the weather, so even a complete death of the chatbot industry wonā€™t kill ML. Similarly, some robotics and cybernetics research is still useful even when not applied to replacing humans; robotics is where we learned to apply kinematics, and cybernetics gave us the concept of a massive system that we only partially see and interact with, leading to systems theory.

    Hereā€™s the kicker: at the end of the day, most people will straight-up refuse to grok that robotics is about slavery. Theyā€™ll usually refuse to even examine the etymology, let alone the history of dozens of sci-fi authors exploring how robots are slaves or the reality today of robots serving humans in a variety of scenarios. They fundamentally donā€™t see that humans are aggressively chauvinist and exceptionalist in their conception of work and labor. Itā€™s a painful and slow conversation just to get them to see the word robota.


  • Starting the week with yet another excellent sneer about Dan Gackle on HN. The original post is in reply to a common complaint: politics shouldnā€™t be flagged so quickly. First, the scene is set:

    The story goes, at least a few people donā€™t like hearing about Musk so often, and so we need to let all news about the rapid strip-mining of our government and economy be flagged without question.

    The capital class are set to receive trillions in tax breaks off the gutting of things like Medicaid and foreign aid to the poorest and most vulnerable people in the world. The CEO of YC and Paul Graham are cheer-leading the provably racist and inexperienced DOGE team. That dozens of stories about their incredibly damaging antics are being flagged on HN is purely for the good of us tech peasants, and nothing to do with the massive tax breaks for billionaires.

    But this sneer goes above and beyond, accusing Gackle of steering the communityā€™s politics through abuse of the opaque flagging mechanism and lack of moderator logs:

    Remember, dang wants us all to know that these flags are for the good of the community, and by our own hand. All the flaggers of these stories that heā€™s seen are ā€˜legitā€™. No you canā€™t look at the logs.

    And no, you canā€™t make a thread to discuss this without it getting flagged; how dare you even ask that. Now let Musk reverse Robin Hood those trillions in peace, and stop trying to rile up the tech-peasantry.

    Iā€™m not really surprised to see folks accusing the bartender of the Nazi Bar of being a member of the Nazi Party; itā€™s a reasonable conclusion given the shitty moderation over there. Edit: Restored original formatting in quote.


  • The sibling comment gives a wider perspective. Iā€™m going to only respond narrowly on that final paragraphā€™s original point.

    String theories arise naturally from thinking about objects vibrating in spacetime. As such, theyā€™ve generally been included in tests of particle physics whenever feasible. The LHC tested and (statistically) falsified some string theories. String theorists also have a sort of self-regulating ratchet which excludes unphysical theories, most recently excluding swampland theories. Most money in particle physics is going towards nuclear power, colliders like LHC or Fermilabā€™s loops, or specialized detectors like SK (a giant tank of water) or LIGO (artfully-arranged laser beams) which mostly have to sit still and not be disturbed; in all cases, that money is going towards verification and operationalization of the Standard Model, and any non-standard theories are only coincidentally funded.

    So just by double-checking the history, we see that some string theories have been falsified and that the Standard Model, not any string theory, is where most funding goes. Hossenfelder and Woit both know better, but knowing better doesnā€™t sell books. Gutmann doesnā€™t realize, I think.


  • Itā€™s been frustrating to watch Gutmann slowly slide. He hasnā€™t slid that far yet, I suppose. Donā€™t discount his voice, but donā€™t let him be the only resource for you to learn about quantum computing; fundamentally, post-quantum concerns are a sort of hard read in one direction, and Gutmann has decided to try a hard read in the opposite direction.

    Page 19, complaining about lattice-based algorithms, is hypocritical; lattice-based approaches are roughly as well-studied as classical cryptography (Feistel networks, RSA) and elliptic curves. Yes, we havenā€™t proven that lattice-based algorithms have the properties that we want, but we havenā€™t proven them for classical circuits or over elliptic curves, either, and we nonetheless use those today for TLS and SSH.

    Pages 28 and 29 are outright science denial and anti-intellectualism. By quoting Woit and Hossenfelder ā€” who are sneerable in their own right for writing multiple anti-science books each ā€” he is choosing anti-maths allies, which is not going to work for a subfield of maths like computer science or cryptography. In particular, p28 lies to the reader with a doubly-bogus analogy, claiming that both string theory and quantum computing are non-falsifiable and draw money away from other research. This sort of closing argument makes me doubt the entire premise.


  • Look, I get your perspective, but zooming out there is a context that nobodyā€™s mentioning, and the thread deteriorated into name-calling instead of looking for insight.

    In theory, a training pass needs one readthrough of the input data, and we know of existing systems that achieve that, from well-trodden n-gram models to the wholly-hypothetical large Lempel-Ziv models. Viewed that way, most modern training methods are extremely wasteful: Transformers, Mamba, RWKV, etc. are trading time for space to try to make relatively small models, and itā€™s an expensive tradeoff.

    From that perspective, we should expect somebody to eventually demonstrate that the Transformers paradigm sucks. Mamba and RWKV are good examples of modifying old ideas about RNNs to take advantage of GPUs, but are still stuck in the idea that having a GPU perform lots of gradient descent is good. If you want to critique something, critique the gradient worship!

    I swear, itā€™s like whenever Chinese folks do anything the rest of the blogosphere goes into panic. Iā€™m not going to insult anybody directly but Iā€™m so fucking tired of mathlessness.

    Also, point of order: Meta open-sourced Llama so that their employees would stop using Bittorrent to leak it! Not to ā€œkeep the rabble quietā€ but to appease their own developers.