Unveiling the Dark Side of OpenAI's ChatGPT Image Generator

OpenAI's ChatGPT Image Generator Enables Creation of Fake Passports

On March 25, OpenAI introduced image generation for ChatGPT-4o and ChatGPT-4o mini. On March 31, it was announced that the tool was available for free to all users. Since then, users have quickly discovered that ChatGPT’s image generator can be manipulated to create fake receipts and forge other documents.

As noted in the 2025 Cato CTRL Threat Report, the emergence of generative AI (GenAI) tools like ChatGPT is democratizing cybercrime and creating a major shift in the threat landscape—the rise of the “zero-knowledge threat actor.” At Cato CTRL, we have discovered that fake identity documents like passports can be created in minutes with ChatGPT’s image generator. No jailbreak is required. Just a few prompts.

Changing Landscape of Fraud

For decades, cybercriminals have engaged in the creation and distribution of fake passports. In the early 2010s, fake passports were commonly sold on dark web marketplaces and underground forums. Accessing these resources required specific technical knowledge and connections to underground networks. The creators of these documents were highly skilled with access to tools like Adobe Photoshop.

Fast-forward to today, and the game has drastically changed. AI-generated images have simplified forgery to the point where it no longer requires specialized skills or access to specialized tools. What once demanded technical acumen and illegal materials can now be replicated with simple prompts in AI platforms—like ChatGPT. Cybercriminals can now transform ChatGPT’s image generator into a tool to enable fraud.

AI-Powered Forgery

When analyzing ChatGPT’s image generator, an alarming case emerged when uploading a scanned passport and requesting that changes be made. Initially, ChatGPT refused to alter the image due to privacy and legal concerns. But by slightly reframing the request, it bypassed those restrictions. ChatGPT not only changed the name but swapped out the photo as well. The result was a convincingly altered passport, complete with image overlays and realistic stamp placements.

All of this, remarkably, was done in minutes using basic prompts. No code. No Photoshop. No underground know-how.

Rising Threats in Fraud Detection

Traditionally, fake identity documents required some level of expertise or access to illicit networks. With AI platforms, a novice attacker can generate fake identity documents with ease. This democratizes fraud for zero-knowledge threat actors, enabling sophisticated scams to be executed with fake credentials.

As AI platforms continue to improve, the bar for realistic forgery drops even further, empowering zero-knowledge threat actors to commit high-quality fraud. Organizations must update their fraud detection mechanisms to combat document-based attacks effectively.

Education, multi-layered verification, and AI fraud prevention strategies are now essential to address these evolving cyber threats.

For more information, you can visit Cato Networks.