Running AI models without matrix math means far less power consumption—and fewer GPUs?

Running AI models without matrix math means far less power consumption—and fewer GPUs?

  • blindsight@beehaw.orgEnglish
    3·
    5 months ago

    This could be huge, but we’ll need to wait and see. The economic and ecological footprint of LLMs is problematic.

    That said, will this actually help, or will they just use 3T parameter models to outcompete competitors 1T parameter models using GPUs? Really, this is more about small-scale models competing with midsize models. Like, this could bring a model as big as GPT 3.5 down to be something you could run on affordable hardware, right?

    That would be really compelling for my sector (education) where there’s a lot of concern about student data privacy. I could definitely pitch building a local $5K-cost LLM server that could handle a dozen or so simultaneous users. That would be enough for a small school district.