\n<\/p>\n
xAI is suing Minnesota Attorney General Keith Ellison over a law passed back in May<\/a> that broadly targets \u201cnudification\u201d apps, claiming that the statute\u2019s punitive provisions<\/a> leave the company with \u201cno practical choice but to restrict Grok Imagine\u2019s image-editing features in various ways.\u201d The law, the company argues, violates the First Amendment.<\/p>\n<\/div>\n\nBack in January, Grok flooded the internet with millions of sexually explicit deepfakes<\/a>, including images of minors<\/a>. In another era, a content moderation disaster<\/a> of this magnitude would have had the CEO flipping the kill switch on the product within hours. But in Elon Musk\u2019s brave new world, the chatbot was left to pump out pornographic images for days. \u201cGrok now hitting #1 on the App Store in one country after another!\u201d he posted<\/a> in the midst of the deepfake epidemic.<\/p>\n<\/div>\n\nSomething had gone wrong. Was the law simply not up to the task of curbing an explosion of nonconsensual nudes? Governments around the world were asking fairly similar questions. The EU launched an investigation<\/a>, as did the United Kingdom<\/a>. There was backlash from French, Indian, and Malaysian government officials<\/a>. And while the American federal government was noticeably quieter on the issue, blue states like California<\/a> and Minnesota took a harder line.<\/p>\n<\/div>\n\nIn Minnesota, the state legislature passed a law<\/a> targeting \u201cnudification\u201d technology, prohibiting site owners from allowing people to \u201caccess, download, or use\u201d such software. (The law makes a carve-out for tech that \u201crequires the technical skill of a user to nudify an image or video\u201d \u2014 so bespoke sexual harassment through Photoshop is in the clear.) The statute creates a civil cause of action for private individuals who may wish to sue, and also authorizes the state attorney general to seek penalties of up to $500,000 per violation.<\/p>\n<\/div>\n\nThe law, touted as the first of its kind, was signed by Gov. Tim Walz in May, and is meant to go into effect on August 1st. It\u2019s a bit of a mystery as to why xAI did not file a lawsuit in the last three months, and instead sued just a couple of days before the statute went into effect. It\u2019s the kind of delayed reaction that makes you wonder if someone at xAI suddenly noticed last week that they were about to be in violation of a fairly punitive law, and then spent the next few days trying to find a law firm willing to work overtime on behalf of a nonconsensual deepfake generator. One can only hope that the lawyers involved are making an absolute killing.<\/p>\n<\/div>\n\nThe complaint<\/a> is pretty reasonable, all things considered. The Minnesota law is overbroad, it argues, because it imposes liability even for images that are consensually generated, or images of adults that are merely sort-of kind-of sexy, or images with \u201chumorous\u201d or \u201cartistic\u201d merit. The First Amendment dictates that laws must be narrowly written so as to avoid banning social and political commentary like this deepfake of Chris Christie and JB Pritzker sumo wrestling.<\/p>\n<\/div>\n\n\n\n\n\n<\/a><\/div>\n<\/div>\n<\/div>\nA real excerpt from page 20 of the lawsuit.<\/figcaption><\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\u201cLiability also attaches \u2026 even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images,\u201d the lawsuit states with an excruciating degree of earnestness. Elsewhere, it bemoans the strict liability that attaches for the unsanctioned conduct of its users \u2014 \u201cimages and videos that users might manage to create in violation of Grok\u2019s terms of service, which is possible only by evading sophisticated, state-of-the-art controls that xAI employs.\u201d<\/p>\n<\/div>\n\n\u201cxAI takes violations of its terms of service seriously,\u201d the lawsuit states, pointing out that the company has since sued one of its users for using Grok to generate CSAM<\/a>. (The lawsuit against the alleged CSAM-maker was filed over two months after the Minnesota statute was passed.)<\/p>\n<\/div>\n\nxAI\u2019s lawyers might have a point, but their lawsuit reads like a missive from another planet. They claim, for instance, that existing Minnesota and federal law (like the TAKE IT DOWN Act<\/a>) already prevents the nonconsensual dissemination of nude deepfakes. These statutes, the lawyers argue, are written in a constitutional way \u2014 the nudification law, on the other hand, is not. And as with any laws in America that target speech based on its content, these statutes must be written in as unrestrictive a way as possible while still addressing a \u201ccompelling\u201d government interest.<\/p>\n<\/div>\n\nBut that\u2019s of course the crux of the problem with the Grok deepfake-ocalypse. If existing law was up to the task, the entire month of January 2026 wouldn\u2019t have been awash with nonconsensual porn. When the Center for Countering Digital Hate<\/a> analyzed an 11-day period from December 29th to January 8th, it found that Grok had generated around 3 million sexualized images. Around 23,000 of those images depicted children. \u201cThat is a shocking rate of one sexualized image of a child every 41 seconds,\u201d the report read.<\/p>\n<\/div>\n\nA lot of those images might not even be actually illegal. For instance, it\u2019s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge<\/em><\/a> that other images Grok generated \u2014 like ones showing minors\u2019 faces covered in semen \u2014 probably did violate existing law.<\/p>\n<\/div>\n\nEither way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota\u2019s HF 1606<\/a> is a bad law. But what does that mean when the \u201cgood\u201d laws aren\u2019t even working?<\/p>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Sarah Jeong<\/span><\/span><\/span>\n
Back in January, Grok flooded the internet with millions of sexually explicit deepfakes<\/a>, including images of minors<\/a>. In another era, a content moderation disaster<\/a> of this magnitude would have had the CEO flipping the kill switch on the product within hours. But in Elon Musk\u2019s brave new world, the chatbot was left to pump out pornographic images for days. \u201cGrok now hitting #1 on the App Store in one country after another!\u201d he posted<\/a> in the midst of the deepfake epidemic.<\/p>\n<\/div>\n\nSomething had gone wrong. Was the law simply not up to the task of curbing an explosion of nonconsensual nudes? Governments around the world were asking fairly similar questions. The EU launched an investigation<\/a>, as did the United Kingdom<\/a>. There was backlash from French, Indian, and Malaysian government officials<\/a>. And while the American federal government was noticeably quieter on the issue, blue states like California<\/a> and Minnesota took a harder line.<\/p>\n<\/div>\n\nIn Minnesota, the state legislature passed a law<\/a> targeting \u201cnudification\u201d technology, prohibiting site owners from allowing people to \u201caccess, download, or use\u201d such software. (The law makes a carve-out for tech that \u201crequires the technical skill of a user to nudify an image or video\u201d \u2014 so bespoke sexual harassment through Photoshop is in the clear.) The statute creates a civil cause of action for private individuals who may wish to sue, and also authorizes the state attorney general to seek penalties of up to $500,000 per violation.<\/p>\n<\/div>\n\nThe law, touted as the first of its kind, was signed by Gov. Tim Walz in May, and is meant to go into effect on August 1st. It\u2019s a bit of a mystery as to why xAI did not file a lawsuit in the last three months, and instead sued just a couple of days before the statute went into effect. It\u2019s the kind of delayed reaction that makes you wonder if someone at xAI suddenly noticed last week that they were about to be in violation of a fairly punitive law, and then spent the next few days trying to find a law firm willing to work overtime on behalf of a nonconsensual deepfake generator. One can only hope that the lawyers involved are making an absolute killing.<\/p>\n<\/div>\n\nThe complaint<\/a> is pretty reasonable, all things considered. The Minnesota law is overbroad, it argues, because it imposes liability even for images that are consensually generated, or images of adults that are merely sort-of kind-of sexy, or images with \u201chumorous\u201d or \u201cartistic\u201d merit. The First Amendment dictates that laws must be narrowly written so as to avoid banning social and political commentary like this deepfake of Chris Christie and JB Pritzker sumo wrestling.<\/p>\n<\/div>\n\n\n\n\n\n<\/a><\/div>\n<\/div>\n<\/div>\nA real excerpt from page 20 of the lawsuit.<\/figcaption><\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\u201cLiability also attaches \u2026 even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images,\u201d the lawsuit states with an excruciating degree of earnestness. Elsewhere, it bemoans the strict liability that attaches for the unsanctioned conduct of its users \u2014 \u201cimages and videos that users might manage to create in violation of Grok\u2019s terms of service, which is possible only by evading sophisticated, state-of-the-art controls that xAI employs.\u201d<\/p>\n<\/div>\n\n\u201cxAI takes violations of its terms of service seriously,\u201d the lawsuit states, pointing out that the company has since sued one of its users for using Grok to generate CSAM<\/a>. (The lawsuit against the alleged CSAM-maker was filed over two months after the Minnesota statute was passed.)<\/p>\n<\/div>\n\nxAI\u2019s lawyers might have a point, but their lawsuit reads like a missive from another planet. They claim, for instance, that existing Minnesota and federal law (like the TAKE IT DOWN Act<\/a>) already prevents the nonconsensual dissemination of nude deepfakes. These statutes, the lawyers argue, are written in a constitutional way \u2014 the nudification law, on the other hand, is not. And as with any laws in America that target speech based on its content, these statutes must be written in as unrestrictive a way as possible while still addressing a \u201ccompelling\u201d government interest.<\/p>\n<\/div>\n\nBut that\u2019s of course the crux of the problem with the Grok deepfake-ocalypse. If existing law was up to the task, the entire month of January 2026 wouldn\u2019t have been awash with nonconsensual porn. When the Center for Countering Digital Hate<\/a> analyzed an 11-day period from December 29th to January 8th, it found that Grok had generated around 3 million sexualized images. Around 23,000 of those images depicted children. \u201cThat is a shocking rate of one sexualized image of a child every 41 seconds,\u201d the report read.<\/p>\n<\/div>\n\nA lot of those images might not even be actually illegal. For instance, it\u2019s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge<\/em><\/a> that other images Grok generated \u2014 like ones showing minors\u2019 faces covered in semen \u2014 probably did violate existing law.<\/p>\n<\/div>\n\nEither way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota\u2019s HF 1606<\/a> is a bad law. But what does that mean when the \u201cgood\u201d laws aren\u2019t even working?<\/p>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Sarah Jeong<\/span><\/span><\/span>\n
Something had gone wrong. Was the law simply not up to the task of curbing an explosion of nonconsensual nudes? Governments around the world were asking fairly similar questions. The EU launched an investigation<\/a>, as did the United Kingdom<\/a>. There was backlash from French, Indian, and Malaysian government officials<\/a>. And while the American federal government was noticeably quieter on the issue, blue states like California<\/a> and Minnesota took a harder line.<\/p>\n<\/div>\n\nIn Minnesota, the state legislature passed a law<\/a> targeting \u201cnudification\u201d technology, prohibiting site owners from allowing people to \u201caccess, download, or use\u201d such software. (The law makes a carve-out for tech that \u201crequires the technical skill of a user to nudify an image or video\u201d \u2014 so bespoke sexual harassment through Photoshop is in the clear.) The statute creates a civil cause of action for private individuals who may wish to sue, and also authorizes the state attorney general to seek penalties of up to $500,000 per violation.<\/p>\n<\/div>\n\nThe law, touted as the first of its kind, was signed by Gov. Tim Walz in May, and is meant to go into effect on August 1st. It\u2019s a bit of a mystery as to why xAI did not file a lawsuit in the last three months, and instead sued just a couple of days before the statute went into effect. It\u2019s the kind of delayed reaction that makes you wonder if someone at xAI suddenly noticed last week that they were about to be in violation of a fairly punitive law, and then spent the next few days trying to find a law firm willing to work overtime on behalf of a nonconsensual deepfake generator. One can only hope that the lawyers involved are making an absolute killing.<\/p>\n<\/div>\n\nThe complaint<\/a> is pretty reasonable, all things considered. The Minnesota law is overbroad, it argues, because it imposes liability even for images that are consensually generated, or images of adults that are merely sort-of kind-of sexy, or images with \u201chumorous\u201d or \u201cartistic\u201d merit. The First Amendment dictates that laws must be narrowly written so as to avoid banning social and political commentary like this deepfake of Chris Christie and JB Pritzker sumo wrestling.<\/p>\n<\/div>\n\n\n\n\n\n<\/a><\/div>\n<\/div>\n<\/div>\nA real excerpt from page 20 of the lawsuit.<\/figcaption><\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\u201cLiability also attaches \u2026 even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images,\u201d the lawsuit states with an excruciating degree of earnestness. Elsewhere, it bemoans the strict liability that attaches for the unsanctioned conduct of its users \u2014 \u201cimages and videos that users might manage to create in violation of Grok\u2019s terms of service, which is possible only by evading sophisticated, state-of-the-art controls that xAI employs.\u201d<\/p>\n<\/div>\n\n\u201cxAI takes violations of its terms of service seriously,\u201d the lawsuit states, pointing out that the company has since sued one of its users for using Grok to generate CSAM<\/a>. (The lawsuit against the alleged CSAM-maker was filed over two months after the Minnesota statute was passed.)<\/p>\n<\/div>\n\nxAI\u2019s lawyers might have a point, but their lawsuit reads like a missive from another planet. They claim, for instance, that existing Minnesota and federal law (like the TAKE IT DOWN Act<\/a>) already prevents the nonconsensual dissemination of nude deepfakes. These statutes, the lawyers argue, are written in a constitutional way \u2014 the nudification law, on the other hand, is not. And as with any laws in America that target speech based on its content, these statutes must be written in as unrestrictive a way as possible while still addressing a \u201ccompelling\u201d government interest.<\/p>\n<\/div>\n\nBut that\u2019s of course the crux of the problem with the Grok deepfake-ocalypse. If existing law was up to the task, the entire month of January 2026 wouldn\u2019t have been awash with nonconsensual porn. When the Center for Countering Digital Hate<\/a> analyzed an 11-day period from December 29th to January 8th, it found that Grok had generated around 3 million sexualized images. Around 23,000 of those images depicted children. \u201cThat is a shocking rate of one sexualized image of a child every 41 seconds,\u201d the report read.<\/p>\n<\/div>\n\nA lot of those images might not even be actually illegal. For instance, it\u2019s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge<\/em><\/a> that other images Grok generated \u2014 like ones showing minors\u2019 faces covered in semen \u2014 probably did violate existing law.<\/p>\n<\/div>\n\nEither way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota\u2019s HF 1606<\/a> is a bad law. But what does that mean when the \u201cgood\u201d laws aren\u2019t even working?<\/p>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Sarah Jeong<\/span><\/span><\/span>\n
In Minnesota, the state legislature passed a law<\/a> targeting \u201cnudification\u201d technology, prohibiting site owners from allowing people to \u201caccess, download, or use\u201d such software. (The law makes a carve-out for tech that \u201crequires the technical skill of a user to nudify an image or video\u201d \u2014 so bespoke sexual harassment through Photoshop is in the clear.) The statute creates a civil cause of action for private individuals who may wish to sue, and also authorizes the state attorney general to seek penalties of up to $500,000 per violation.<\/p>\n<\/div>\n\nThe law, touted as the first of its kind, was signed by Gov. Tim Walz in May, and is meant to go into effect on August 1st. It\u2019s a bit of a mystery as to why xAI did not file a lawsuit in the last three months, and instead sued just a couple of days before the statute went into effect. It\u2019s the kind of delayed reaction that makes you wonder if someone at xAI suddenly noticed last week that they were about to be in violation of a fairly punitive law, and then spent the next few days trying to find a law firm willing to work overtime on behalf of a nonconsensual deepfake generator. One can only hope that the lawyers involved are making an absolute killing.<\/p>\n<\/div>\n\nThe complaint<\/a> is pretty reasonable, all things considered. The Minnesota law is overbroad, it argues, because it imposes liability even for images that are consensually generated, or images of adults that are merely sort-of kind-of sexy, or images with \u201chumorous\u201d or \u201cartistic\u201d merit. The First Amendment dictates that laws must be narrowly written so as to avoid banning social and political commentary like this deepfake of Chris Christie and JB Pritzker sumo wrestling.<\/p>\n<\/div>\n\n\n\n\n\n<\/a><\/div>\n<\/div>\n<\/div>\nA real excerpt from page 20 of the lawsuit.<\/figcaption><\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\u201cLiability also attaches \u2026 even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images,\u201d the lawsuit states with an excruciating degree of earnestness. Elsewhere, it bemoans the strict liability that attaches for the unsanctioned conduct of its users \u2014 \u201cimages and videos that users might manage to create in violation of Grok\u2019s terms of service, which is possible only by evading sophisticated, state-of-the-art controls that xAI employs.\u201d<\/p>\n<\/div>\n\n\u201cxAI takes violations of its terms of service seriously,\u201d the lawsuit states, pointing out that the company has since sued one of its users for using Grok to generate CSAM<\/a>. (The lawsuit against the alleged CSAM-maker was filed over two months after the Minnesota statute was passed.)<\/p>\n<\/div>\n\nxAI\u2019s lawyers might have a point, but their lawsuit reads like a missive from another planet. They claim, for instance, that existing Minnesota and federal law (like the TAKE IT DOWN Act<\/a>) already prevents the nonconsensual dissemination of nude deepfakes. These statutes, the lawyers argue, are written in a constitutional way \u2014 the nudification law, on the other hand, is not. And as with any laws in America that target speech based on its content, these statutes must be written in as unrestrictive a way as possible while still addressing a \u201ccompelling\u201d government interest.<\/p>\n<\/div>\n\nBut that\u2019s of course the crux of the problem with the Grok deepfake-ocalypse. If existing law was up to the task, the entire month of January 2026 wouldn\u2019t have been awash with nonconsensual porn. When the Center for Countering Digital Hate<\/a> analyzed an 11-day period from December 29th to January 8th, it found that Grok had generated around 3 million sexualized images. Around 23,000 of those images depicted children. \u201cThat is a shocking rate of one sexualized image of a child every 41 seconds,\u201d the report read.<\/p>\n<\/div>\n\nA lot of those images might not even be actually illegal. For instance, it\u2019s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge<\/em><\/a> that other images Grok generated \u2014 like ones showing minors\u2019 faces covered in semen \u2014 probably did violate existing law.<\/p>\n<\/div>\n\nEither way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota\u2019s HF 1606<\/a> is a bad law. But what does that mean when the \u201cgood\u201d laws aren\u2019t even working?<\/p>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Sarah Jeong<\/span><\/span><\/span>\n
The law, touted as the first of its kind, was signed by Gov. Tim Walz in May, and is meant to go into effect on August 1st. It\u2019s a bit of a mystery as to why xAI did not file a lawsuit in the last three months, and instead sued just a couple of days before the statute went into effect. It\u2019s the kind of delayed reaction that makes you wonder if someone at xAI suddenly noticed last week that they were about to be in violation of a fairly punitive law, and then spent the next few days trying to find a law firm willing to work overtime on behalf of a nonconsensual deepfake generator. One can only hope that the lawyers involved are making an absolute killing.<\/p>\n<\/div>\n
The complaint<\/a> is pretty reasonable, all things considered. The Minnesota law is overbroad, it argues, because it imposes liability even for images that are consensually generated, or images of adults that are merely sort-of kind-of sexy, or images with \u201chumorous\u201d or \u201cartistic\u201d merit. The First Amendment dictates that laws must be narrowly written so as to avoid banning social and political commentary like this deepfake of Chris Christie and JB Pritzker sumo wrestling.<\/p>\n<\/div>\n\n\n\n\n\n<\/a><\/div>\n<\/div>\n<\/div>\nA real excerpt from page 20 of the lawsuit.<\/figcaption><\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\u201cLiability also attaches \u2026 even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images,\u201d the lawsuit states with an excruciating degree of earnestness. Elsewhere, it bemoans the strict liability that attaches for the unsanctioned conduct of its users \u2014 \u201cimages and videos that users might manage to create in violation of Grok\u2019s terms of service, which is possible only by evading sophisticated, state-of-the-art controls that xAI employs.\u201d<\/p>\n<\/div>\n\n\u201cxAI takes violations of its terms of service seriously,\u201d the lawsuit states, pointing out that the company has since sued one of its users for using Grok to generate CSAM<\/a>. (The lawsuit against the alleged CSAM-maker was filed over two months after the Minnesota statute was passed.)<\/p>\n<\/div>\n\nxAI\u2019s lawyers might have a point, but their lawsuit reads like a missive from another planet. They claim, for instance, that existing Minnesota and federal law (like the TAKE IT DOWN Act<\/a>) already prevents the nonconsensual dissemination of nude deepfakes. These statutes, the lawyers argue, are written in a constitutional way \u2014 the nudification law, on the other hand, is not. And as with any laws in America that target speech based on its content, these statutes must be written in as unrestrictive a way as possible while still addressing a \u201ccompelling\u201d government interest.<\/p>\n<\/div>\n\nBut that\u2019s of course the crux of the problem with the Grok deepfake-ocalypse. If existing law was up to the task, the entire month of January 2026 wouldn\u2019t have been awash with nonconsensual porn. When the Center for Countering Digital Hate<\/a> analyzed an 11-day period from December 29th to January 8th, it found that Grok had generated around 3 million sexualized images. Around 23,000 of those images depicted children. \u201cThat is a shocking rate of one sexualized image of a child every 41 seconds,\u201d the report read.<\/p>\n<\/div>\n\nA lot of those images might not even be actually illegal. For instance, it\u2019s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge<\/em><\/a> that other images Grok generated \u2014 like ones showing minors\u2019 faces covered in semen \u2014 probably did violate existing law.<\/p>\n<\/div>\n\nEither way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota\u2019s HF 1606<\/a> is a bad law. But what does that mean when the \u201cgood\u201d laws aren\u2019t even working?<\/p>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Sarah Jeong<\/span><\/span><\/span>\n
A real excerpt from page 20 of the lawsuit.<\/figcaption><\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\u201cLiability also attaches \u2026 even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images,\u201d the lawsuit states with an excruciating degree of earnestness. Elsewhere, it bemoans the strict liability that attaches for the unsanctioned conduct of its users \u2014 \u201cimages and videos that users might manage to create in violation of Grok\u2019s terms of service, which is possible only by evading sophisticated, state-of-the-art controls that xAI employs.\u201d<\/p>\n<\/div>\n\n\u201cxAI takes violations of its terms of service seriously,\u201d the lawsuit states, pointing out that the company has since sued one of its users for using Grok to generate CSAM<\/a>. (The lawsuit against the alleged CSAM-maker was filed over two months after the Minnesota statute was passed.)<\/p>\n<\/div>\n\nxAI\u2019s lawyers might have a point, but their lawsuit reads like a missive from another planet. They claim, for instance, that existing Minnesota and federal law (like the TAKE IT DOWN Act<\/a>) already prevents the nonconsensual dissemination of nude deepfakes. These statutes, the lawyers argue, are written in a constitutional way \u2014 the nudification law, on the other hand, is not. And as with any laws in America that target speech based on its content, these statutes must be written in as unrestrictive a way as possible while still addressing a \u201ccompelling\u201d government interest.<\/p>\n<\/div>\n\nBut that\u2019s of course the crux of the problem with the Grok deepfake-ocalypse. If existing law was up to the task, the entire month of January 2026 wouldn\u2019t have been awash with nonconsensual porn. When the Center for Countering Digital Hate<\/a> analyzed an 11-day period from December 29th to January 8th, it found that Grok had generated around 3 million sexualized images. Around 23,000 of those images depicted children. \u201cThat is a shocking rate of one sexualized image of a child every 41 seconds,\u201d the report read.<\/p>\n<\/div>\n\nA lot of those images might not even be actually illegal. For instance, it\u2019s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge<\/em><\/a> that other images Grok generated \u2014 like ones showing minors\u2019 faces covered in semen \u2014 probably did violate existing law.<\/p>\n<\/div>\n\nEither way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota\u2019s HF 1606<\/a> is a bad law. But what does that mean when the \u201cgood\u201d laws aren\u2019t even working?<\/p>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Sarah Jeong<\/span><\/span><\/span>\n
\u201cLiability also attaches \u2026 even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images,\u201d the lawsuit states with an excruciating degree of earnestness. Elsewhere, it bemoans the strict liability that attaches for the unsanctioned conduct of its users \u2014 \u201cimages and videos that users might manage to create in violation of Grok\u2019s terms of service, which is possible only by evading sophisticated, state-of-the-art controls that xAI employs.\u201d<\/p>\n<\/div>\n
\u201cxAI takes violations of its terms of service seriously,\u201d the lawsuit states, pointing out that the company has since sued one of its users for using Grok to generate CSAM<\/a>. (The lawsuit against the alleged CSAM-maker was filed over two months after the Minnesota statute was passed.)<\/p>\n<\/div>\n\nxAI\u2019s lawyers might have a point, but their lawsuit reads like a missive from another planet. They claim, for instance, that existing Minnesota and federal law (like the TAKE IT DOWN Act<\/a>) already prevents the nonconsensual dissemination of nude deepfakes. These statutes, the lawyers argue, are written in a constitutional way \u2014 the nudification law, on the other hand, is not. And as with any laws in America that target speech based on its content, these statutes must be written in as unrestrictive a way as possible while still addressing a \u201ccompelling\u201d government interest.<\/p>\n<\/div>\n\nBut that\u2019s of course the crux of the problem with the Grok deepfake-ocalypse. If existing law was up to the task, the entire month of January 2026 wouldn\u2019t have been awash with nonconsensual porn. When the Center for Countering Digital Hate<\/a> analyzed an 11-day period from December 29th to January 8th, it found that Grok had generated around 3 million sexualized images. Around 23,000 of those images depicted children. \u201cThat is a shocking rate of one sexualized image of a child every 41 seconds,\u201d the report read.<\/p>\n<\/div>\n\nA lot of those images might not even be actually illegal. For instance, it\u2019s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge<\/em><\/a> that other images Grok generated \u2014 like ones showing minors\u2019 faces covered in semen \u2014 probably did violate existing law.<\/p>\n<\/div>\n\nEither way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota\u2019s HF 1606<\/a> is a bad law. But what does that mean when the \u201cgood\u201d laws aren\u2019t even working?<\/p>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Sarah Jeong<\/span><\/span><\/span>\n
xAI\u2019s lawyers might have a point, but their lawsuit reads like a missive from another planet. They claim, for instance, that existing Minnesota and federal law (like the TAKE IT DOWN Act<\/a>) already prevents the nonconsensual dissemination of nude deepfakes. These statutes, the lawyers argue, are written in a constitutional way \u2014 the nudification law, on the other hand, is not. And as with any laws in America that target speech based on its content, these statutes must be written in as unrestrictive a way as possible while still addressing a \u201ccompelling\u201d government interest.<\/p>\n<\/div>\n\nBut that\u2019s of course the crux of the problem with the Grok deepfake-ocalypse. If existing law was up to the task, the entire month of January 2026 wouldn\u2019t have been awash with nonconsensual porn. When the Center for Countering Digital Hate<\/a> analyzed an 11-day period from December 29th to January 8th, it found that Grok had generated around 3 million sexualized images. Around 23,000 of those images depicted children. \u201cThat is a shocking rate of one sexualized image of a child every 41 seconds,\u201d the report read.<\/p>\n<\/div>\n\nA lot of those images might not even be actually illegal. For instance, it\u2019s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge<\/em><\/a> that other images Grok generated \u2014 like ones showing minors\u2019 faces covered in semen \u2014 probably did violate existing law.<\/p>\n<\/div>\n\nEither way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota\u2019s HF 1606<\/a> is a bad law. But what does that mean when the \u201cgood\u201d laws aren\u2019t even working?<\/p>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Sarah Jeong<\/span><\/span><\/span>\n
But that\u2019s of course the crux of the problem with the Grok deepfake-ocalypse. If existing law was up to the task, the entire month of January 2026 wouldn\u2019t have been awash with nonconsensual porn. When the Center for Countering Digital Hate<\/a> analyzed an 11-day period from December 29th to January 8th, it found that Grok had generated around 3 million sexualized images. Around 23,000 of those images depicted children. \u201cThat is a shocking rate of one sexualized image of a child every 41 seconds,\u201d the report read.<\/p>\n<\/div>\n\nA lot of those images might not even be actually illegal. For instance, it\u2019s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge<\/em><\/a> that other images Grok generated \u2014 like ones showing minors\u2019 faces covered in semen \u2014 probably did violate existing law.<\/p>\n<\/div>\n\nEither way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota\u2019s HF 1606<\/a> is a bad law. But what does that mean when the \u201cgood\u201d laws aren\u2019t even working?<\/p>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Sarah Jeong<\/span><\/span><\/span>\n
A lot of those images might not even be actually illegal. For instance, it\u2019s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge<\/em><\/a> that other images Grok generated \u2014 like ones showing minors\u2019 faces covered in semen \u2014 probably did violate existing law.<\/p>\n<\/div>\n\nEither way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota\u2019s HF 1606<\/a> is a bad law. But what does that mean when the \u201cgood\u201d laws aren\u2019t even working?<\/p>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Sarah Jeong<\/span><\/span><\/span>\n
Either way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota\u2019s HF 1606<\/a> is a bad law. But what does that mean when the \u201cgood\u201d laws aren\u2019t even working?<\/p>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Sarah Jeong<\/span><\/span><\/span>\n
![\"A](\"https:\/\/platform.theverge.com\/wp-content\/uploads\/sites\/2\/2026\/07\/Screenshot-2026-07-29-at-1.26.30-PM.png?quality=90&strip=all&crop=0%2C0%2C100%2C100&w=2400\"\/)
<\/a><\/div>\n<\/div>\n<\/div>\n