When disinformation expert Tal Hagin asked Grok to verify a post on X about Iranian missiles that had supposedly struck Tel Aviv, Elon Musk’s AI-powered chatbot failed miserably.
Grok repeatedly misidentified the location and date for the video, which was originally shared on X by an Iranian state-owned media outlet on Sunday. Then, the chatbot then tried to prove its point by sharing an AI-generated image.
“Now Grok is replying with AI slop of destruction,” Hagin wrote in response. “Cooked I tell you.”
The interaction neatly sums up just how unhinged from reality X has become since the US and Israel began their attack on Iran on February 28. As WIRED reported at the time, the social media platform was quickly flooded with disinformation by accounts sharing fake and repurposed videos.
As the conflict has continued, the flood has only gotten worse. In recent days, it’s been supercharged by AI images and videos, while Grok has repeatedly given false information when asked to verify claims made on the platform. AI images are being shared by paid accounts bearing blue checkmarks and Iranian officials seeking to portray exaggerated damage.
The proliferation of easy-to-access AI image and video generation tools has led to increasingly sophisticated fake content. On March 2, for example, Iranian officials and state media shared AI-generated videos of a high-rise building in Bahrain on fire. The videos and images appear realistic enough for many: One image of a US B-2 bomber being shot down by Iran with US troops detained was viewed over a million times before it was deleted, while images of members of Delta Force being captured by Iranian authorities were viewed over 5 million times before they were deleted.
Some of the AI content promoted on X is less realistic.One video, for example, purports to show Iranian forces manufacturing missiles deep inside a cave. However, the video was still been shared by multiple accounts and has been viewed over a million times.
AI is also being used by the Iranian government to push overtly antisemitic narratives, with accounts in a pro-regime propaganda network on X sharing AI-generated posts depicting Orthodox Jews leading American soldiers to war or celebrating American deaths, according to researchers from the Institute of Strategic Dialogue (ISD), who shared their analysis with WIRED.
A number of accounts in this pro-regime network also shared a fake video that supposedly showed a line of young girls walking past President Donald Trump wearing only underwear. The post was viewed over 6.8 million times, according to ISD, before being taken down, though it continues to be shared by other accounts on X.
“What is particularly unique about this war is the dramatic uptick in AI-generated content I find myself debunking,” Hagin tells WIRED. “This is likely due to AI being advanced enough to fool journalists, and the ease with which users can create this AI slop with zero consequences. The longer we go without regulations against AI abuse, the more harm will be caused. I see the proliferation of AI-based fake news pushing us over the edge of a fact-based world unless we enact change now.”
When the flood of AI-generated fakes began taking over the platform last week, X announced it would temporarily demonetize blue checkmark accounts if they post AI-generated videos of armed conflict without a label. X did not respond to a request for comment about how many accounts it had demonetized since introducing the measure. Until recently, a number of Iranian officials appeared to be paying X for its premium service, which provided their accounts with blue checkmarks, boosted engagement, and created the potential to earn money for their posts.
