ChatGPT can offer coding solutions, but its tendency for hallucination presents attackers with an opportunity. Here's what we learned.

From https://twitter.com/llm_sec/status/1667573374426701824

  1. People ask LLMs to write code
  2. LLMs recommend imports that don’t actually exist
  3. Attackers work out what these imports’ names are, and create & upload them with malicious payloads
  4. People using LLM-written code then auto-add malware themselves
  • pwshguy (mdowst)English
    5•1 year ago

    I’ve used ChatGPT and Copilot to help with PowerShell in the past. For the most part I’ve found it, okay. But I can definitely see how that could happen. I’ve had a few instances where is tried giving me cmdlets that don’t exist. This means it is just taking pieces of code from someone else’s project and not understanding how it all fits together. So, if I search that cmdlet there is no telling the number of results I could get from good, bad, or irrelevant sources. It would be better if it told you where that code came from. Then I could look at the original source and see if they created custom functions that these AI are considering to be built-in cmdlets.