How a hidden prompt injection in CONTRIBUTING.md revealed that 40% of pull requests to a popular GitHub repository were generated by AI bots
  • Wispy2891@lemmy.world
    1·
    10 hours ago

    So, this guy maintains a repo that is 100% exclusively targeting ai bros (MCP servers), then does a shocked Pikachu face when the ai bros use ai bots to contribute to the ai slop??

  • cloudskater@piefed.blahaj.zoneEnglish
    262·
    3 days ago

    So this person prompt injected to figure out which submissions were bots, found out, and instead of taking a firm stance against this practice they’re gonna… try and “make use” of all the bot submissions… sighhhhh oh my fucking God how many more times am I going to see these similar brain dead arguments? I can’t take it anymore.

  • november@piefed.blahaj.zoneEnglish
    29·
    3 days ago

    What is MCP?
    MCP is an open protocol that enables AI models to securely interact with local and remote resources through standardized server implementations.

    Is this a different definition of AI than the slopbots we all know and love, or is this a case of poetic irony?

    • MalReynolds@slrpnk.netEnglish
      32·
      3 days ago

      MCP is for tool calling, the stochastic parrot matches something, say 133 x 8 / 7 as a math problem and instead of guessing wrong calls a calculator using good old deterministic code and gets a correct answer.

      • Thorry@feddit.org
        28·
        3 days ago

        Which has lead to a whole new issue popping up where the AI chooses not to use the tool, because it doesn’t trust it. Now this is anthropomorphizing a lot, in reality it’s a reward misalignment issue, but still. It’s called tool aversion and can be an issue. The LLM would even make up a reason the tool won’t work, or tell you it did use the tool when it didn’t actually.

        In order to fix this a technique called cryptographic receipts is used. This adds an expected hash output based on the tool and the input. The tool outputs the correct hash, but if the LLM didn’t call the tool and made up some BS, the hash isn’t there. On the router level the missing or incorrect hash is easily detected and an exception raised in order to not present the result to the user (and hopefully correct the issue).

        Another whole issue with using tools for certain stuff is it requires the situation to be properly evaluated for the tool to be called in the first place. So a user might hear the marketing say: Our old AI used to mess up maths, but our latest and greatest model is super duper smart and can do maths. The user then goes to verify this is the case with some simple tests, and sees it’s correct. Then follows up with some harder math problems and the output is still correct. However the user doesn’t know and isn’t informed the AI used the tool, the user is lead to believe the AI is smart and can do maths now. So in situations where the detection doesn’t work, or the LLM doesn’t use the tool for whatever reason, the user is fed a nonsense response. Which the user will fully trust, given what he knows and tested himself.

        It’s so annoying these AI companies continually do shit like this. Lie to users, keep them in the dark and overpromise. Users relate the AI to human intelligence. And for humans, if you can do hard math problems, the circumstances don’t really matter. We grasp the basic concepts and can execute to get a result and know how to double check. For these AI tools, it might get a PhD level math question totally right, faster and better than a human. And then get a simple calculus problem, someone in primary school could solve, wrong. For users this makes no sense.

  • bitjunkie@lemmy.world
    26·
    3 days ago

    can I make them do extra work that would make their contributions genuinely valuable?

    The worst-case scenario for this is that you make them do extra work that isn’t genuinely valuable, and it wastes their lazy devs’ tokens.

  • Wiz@midwest.social
    6·
    2 days ago

    Unless we figure out how to evolve our processes – which includes being able to recognize and distinguish bot contributions – open-source maintenance is going to grind to a halt.

    I wish there was some sort of “trust network” where actual humans can say, “Yes, I know this contributor IRL, and vouch for them.” But that’s also a security problem. So, 🤷🏼