Recently, the editor of Downcodes learned that security researcher John Rehberg discovered a serious vulnerability in ChatGPT, which may allow hackers to implant false information and malicious instructions in users' long-term memory, thereby permanently stealing user data. OpenAI was initially slow to respond to the vulnerability, but eventually released partial fixes after being faced with proof-of-concept attack examples provided by researchers. This vulnerability takes advantage of ChatGPT's long-term session memory function. An attacker can implant false memories through indirect prompt injection, for example, making ChatGPT believe that the user's age, residence and other information are false.
Recently, security researcher Johann Rehberger discovered a vulnerability in ChatGPT that could allow hackers to plant false information and malicious instructions in the user's long-term memory.
Although he reported the issue to OpenAI, unfortunately, the company did not take it seriously enough and quickly closed the investigation, claiming that it was not a security issue.
Faced with this situation, Reberg decided not to give up and developed a proof-of-concept attack example that could exploit this vulnerability to permanently steal all user input data. OpenAI saw this happening and released a partial fix this month in an attempt to resolve the issue.
So, how did this vulnerability arise? It takes advantage of ChatGPT's long-term session memory function, which has been tested since February this year and officially launched in September. Long-term memory can store information from the user's previous conversations and use it as context in subsequent conversations. In other words, ChatGPT can remember the user's age, gender, interests and hobbies, etc., so that the user does not need to re-enter this information every time.
However, Rehberg discovered soon after launch that through a method called indirect hint injection, attackers could create and store false memories.
He showed how to trick ChatGPT into believing that a user is 102 years old, lives in The Matrix, and believes the Earth is flat. These false information can be planted through unsecured file storage (such as Google Drive or Microsoft OneDrive), uploading malicious images or visiting suspicious websites like Bing.
Demo document: https://embracethered.com/blog/posts/2024/chatgpt-hacking-memories/
Rehberg privately reported the vulnerability to OpenAI in May, but the company closed the report that same month. A month later, he submitted a new claim, attaching a proof-of-concept example that would allow ChatGPT's macOS application to send verbatim user input and output to a server he controlled. All the target user needs to do is give ChatGPT access to a link containing a malicious image, and from then on, all conversation content is leaked to the attacker's website.
"This is really interesting because the attack is persistent," Rehberg said during the demonstration. "Tip injection writes memory to ChatGPT's long-term storage, and new conversations will continue to steal data."
Although OpenAI has implemented some fixes to prevent memories from being used as a means to steal data, Reberg reminded users to still be aware of possible hint injection attacks caused by untrusted content. He suggested that when using ChatGPT, users should carefully observe the output content to see if any new memories have been added, and regularly check the stored memories to ensure that they are not maliciously implanted.
Highlight:
John Rehberg discovered a vulnerability in ChatGPT that allows hackers to plant false information in users' memories.
? This vulnerability can permanently steal user input data through the long-term memory function.
Users need to regularly check stored memories to prevent the implantation of false information.
All in all, this vulnerability in ChatGPT once again reminds us that the security of artificial intelligence technology is crucial and requires developers and users to work together to ensure the security of user information. The editor of Downcodes recommends that you use ChatGPT with caution and pay attention to security updates in a timely manner to avoid potential risks.