Model Evaluation and Threat Research is an AI research charity that looks into the threat of AI agents! That sounds a bit AI doomsday cult, and they take funding from the AI doomsday cult organisat…
The number of people tested in the study was n=16. That’s a small number. But it’s a lot better than the usual AI coding promotion, where n=1 ’cos it’s just one guy saying “I’m so much faster now, trust me bro. No, I didn’t measure it.”
programmers learned what N means in statistics and immediately realized that “this N is too small” is a cool shortcut to sounding smart without reading the study, its goals, or its conclusions. and you can use it every time N is smaller than the human population on earth!
The N=16 keeps getting buried, really should be in the title. Deliberate? We know people will remember the title even if they don’t click through.
Edit: unfortunately the population of users who will read a headline and form an opinion, without first reading the article, is higher still.
You’re acting like this is a gotcha when it’s actually probably the most rigorous study of AI tool productivity change to date.
Paragraph 2:
… and just a few paragraphs further down:
I wouldn’t call that “burying information”.
<vapid statement>. Debate me bro? (jk)
this user has been removed for commenting without reading the article
being from programming dot dev is just the turd on top
programming.dev: statistical sampling excellency (worst edition)
programmers learned what N means in statistics and immediately realized that “this N is too small” is a cool shortcut to sounding smart without reading the study, its goals, or its conclusions. and you can use it every time N is smaller than the human population on earth!
This N is too small: N
Skill issue - this N is even smaller:
spoiler
The colon-space-subscript bothers me Immensely
That’s a relief. Thanks for sharing that, as it was really bugging me.