LLMs are solving MCAT, the bar test, SAT etc like they’re nothing. At this point their performance is super human. However they’ll often trip on super simple common sense questions, they’ll struggle with creative thinking.
Is this literally proof that standard tests are not a good measure of intelligence?
Long before. Even in 1930 the eugenics-motivated creator Carl Brigham recanted his original conclusions only years ago that had led to the development of the SAT, but by then the colleges had totally invested in a quick and easy way to score students, even if it was inaccurate. Change is hard, but I think the bigger influence here was money since it hadn’t been around that long at that point.