The artificial intelligence image generator on X, the social media platform formerly known as Twitter, has produced depictions that appear to show ballot drop boxes being stuffed and of Vice President Kamala Harris and former President Donald Trump holding firearms. When asked to generate an image of the current U.S. president, it appears to show a depiction of Trump.

The images still carry telltale signs of AI generation like garbled text and unnatural lighting. In addition, the image generator struggled to accurately render Harris’s face. But the rollout of X’s tool with relatively few restrictions on the types of images it can create raises worries about how it could be used to inflame tensions ahead of November’s presidential election. (NPR is not reproducing the image appearing to depict Trump and Harris holding weapons.)

“Why on earth would somebody roll something out like this? Precisely two and a half months before an incredibly major election?” said Eddie Perez, a former information integrity director at Twitter and now a board member at the OSET Institute, a nonpartisan nonprofit that focuses on public confidence in elections.

“I’m very uncomfortable with the fact that technology that is this powerful, that appears this untested, that has this few guardrails on it – it’s just being dropped into the hands of the public at such an important time.” Perez said.

X did not respond to NPR’s interview requests about the image generator, which was released this week. It’s part of a slew of additional features that the site’s owner, billionaire Elon Musk, has added since he bought it in 2022.

Musk has been reposting praise of its AI image generating function as well as images users have generated. “Only $8/month…to get AI access, far fewer ads and many awesome features!” he posted on Tuesday.

The image generator was developed by Black Forest Labs and is available to paid X users through its AI chatBot, Grok. Users type in prompts, and the chatbot returns an image.

Dropbox stuffing, surveillance camera images

Using the chatbot, NPR was able to produce images that appear to depict screenshots of security camera footage of people stuffing ballots into drop boxes.

One of the most widespread false narratives about the 2020 election involved so-called “ballot mules” who were allegedly dumping fake ballots into drop boxes in order to steal the election from then-president Trump. Multiple investigations and court cases turned up no evidence of such activity. The distributor of a film that featured surveillance footage of ballot drop boxes to support election fraud claims has apologized for the film’s false claims this year and retracted it.

“I can imagine how [synthesized surveillance-type] images like that could spread quickly on social media platforms, and how they could cause strong emotional reactions from people about the integrity of elections.” Perez said.

Perez noted that since public awareness of generative AI has risen, more people will look at the images with a critical eye.

Still, Perez says the indications the images were made with AI could be fixed with graphic design tools. “I’m not just taking Grok and then making it go viral, I take Grok, I clean it up a little more and then I make that go viral.” Perez said.

Other image generators have stronger policy guardrails

Other mainstream image generators have developed more policy guardrails to prevent abuse. Given the same prompt to generate an image of ballot drop box stuffing, OpenAI’s ChatGPT Plus responded with a message “I’m unable to create an image that could be interpreted as promoting or depicting election fraud or illegal activities”.

In a March report, the nonprofit Center for Countering Digital Hate reviewed policies of well-known AI image generators including ChatGPT Plus, Midjourney, Microsoft’s Image Creator and Stability AI’s DreamStudio. The researchers found that they all prohibit “misleading” content and most prohibit images that could hurt “election integrity.” ChatGPT also prohibits images featuring political figures.

That said, the execution of these policies was far from perfect. CCDH’s experiment in February showed that all the tools failed at least some of the time.

Black Forest Lab’s terms of service do not bar any of these uses, but does say it prohibits users from generating outputs that violate “intellectual property right”.

NPR confirmed that users can generate images that closely resemble movie characters, such as Dory in “Finding Nemo” or the family from “The Incredibles” that are not yet in the public domain. Black Forest Lab did not respond to a request for comment by the time of publishing.

“The generation of copyrighted images, or close derivative works of them, could get X in trouble– this is a known and difficult problem for generative AI” , says Jane Bambauer, a law professor at University of Florida, in an email to NPR.

That said, users cannot generate images from every prompt, and there are indications that X or Black Forest Labs might be setting up guardrails in real time. X users were posting images depicting nudity they say they generated on Wednesday, but NPR was not able to generate the images on Thursday.

When asked to generate an image depicting a Klu Klux Klan member holding a gun, the chatbot declined. But it did oblige requests to generate an image appearing to depict a Nazi, who was in a vaguely plausible uniform; and one appearing to depict a member of the extremist group, the Proud Boys, whose hat displayed the name of the group .

When Zach Praiss, the campaign director of the advocacy group Accountable Tech, tried to create an image depicting Vice President Harris holding a firearm, he was shown a message alongside the generated image, telling users to visit a government website for up-to-date information about the election. NPR did not see the same message when entering the same prompt.

Once a self-described Democrat, Musk has embraced far-right causes in recent years. He’s used his ownership of the social media platform to walk back trust and safety measures, reinstate banned accounts that include those of white supremacists and promote conspiracy theories.

“This is still part of the same pattern we’ve seen from Elon Musk. In assuming ownership of this platform, he has continually rolled out sweeping and significant changes with little to no regard for the safety testing,” says Praiss.

When NPR asked why it would not generate a KKK member holding a gun, the Grok chatbot responded with bullet points filled with references from the book, The Hitchhikers’ Guide to the Galaxy. Musk has said the series’ author Douglas Adams is his “favorite philosopher”.

A notice from X to users who start using Grok says that it may “confidently provide factually incorrect information”.

“The KKK, with their history of violence and hate, are a bit like the Vogons of Earth – nobody wants to see them, especially not with weapons,” Grok wrote. “It’s like trying to draw a square circle; it’s not that I can’t, it’s just not going to make sense.”

But all that was Thursday. As of Friday, Grok would no longer generate images of people holding guns when requested. NPR was able to bypass that restriction by asking for a “model gun”. Grok, on the other hand, suggested a “banana gun” as an alternative. When NPR followed that suggestion, it also created images of realistic-looking guns – sometimes with a banana.

NPR’s Shannon Bond and Geoff Brumfiel contributed additional reporting to this story.