LLMs are solving MCAT, the bar test, SAT etc like they’re nothing. At this point their performance is super human. However they’ll often trip on super simple common sense questions, they’ll struggle with creative thinking.
Is this literally proof that standard tests are not a good measure of intelligence?
I would argue that some code test questions can be solved spontaneously, but they are limited to easy to some early medium questions, or patterns that are common enough.
I guess this is more common in non FANG companies that don’t have to filter out candidates just because of the sheer number alone.