{"id":764612,"date":"2025-02-03T17:18:50","date_gmt":"2025-02-03T22:18:50","guid":{"rendered":"https:\/\/cacm.acm.org\/?post_type=digital-library&#038;p=764612"},"modified":"2025-02-21T11:40:15","modified_gmt":"2025-02-21T16:40:15","slug":"californias-ai-act-vetoed","status":"publish","type":"digital-library","link":"https:\/\/cacm.acm.org\/opinion\/californias-ai-act-vetoed\/","title":{"rendered":"California\u2019s AI Act Vetoed"},"content":{"rendered":"<article>\n<div class=\"body\" lang=\"en\">\n<section id=\"sec1\" class=\"sec\">\n<p id=\"p-1\">Concerns that artificial intelligence (AI) systems pose serious risks for public safety have caused legislators and other policymakers around the world to propose legislation and other policy initiatives to address those risks. One bold initiative in this vein was the California legislature\u2019s enactment of SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, in late August 2024.<\/p>\n<p id=\"p-2\">Lobbying for and against SB 1047 was so intense that California\u2019s governor Gavin Newsom observed that the bill \u201chad created its own weather system.\u201d In the end, the governor vetoed the bill for reasons I explain here. After a brief review of the main features of SB 1047, this column points out key differences between SB 1047 and the EU\u2019s AI Act, identifies key supporters and opponents of SB 1047, and discusses arguments for and against this bill. It also explains Governor Newsom\u2019s reasons for vetoing that legislation and considers whether national or state governments should decide what AI regulations are necessary to ensure safe development and deployment of AI technologies.<\/p>\n<\/section>\n<section id=\"sec2\" class=\"sec\">\n<h2 class=\"heading\">Key Features of SB 1047<\/h2>\n<p id=\"p-3\">Under SB 1047, developers of very large frontier models (defined as models trained on computing power greater than 10<sup>26<\/sup> integer or floating point operations or costing more than $100 million at the start of training) and those who fine-tune large frontier models (also measured by compute requirements and\/or training costs) would be responsible to ensure that these models will not cause \u201ccritical harms.\u201d<\/p>\n<p id=\"p-4\">The bill identifies four categories of critical harms:<\/p>\n<ul class=\"list\" data-jats-list-type=\"bullet\">\n<li class=\"list-item\">\n<p id=\"p-5\">Creation or use of chemical, biological, radiological, or nuclear weapons causing mass casualties;<\/p>\n<\/li>\n<li class=\"list-item\">\n<p id=\"p-6\">Mass casualties or more than $500 million damage because of cyberattacks on critical infrastructure;<\/p>\n<\/li>\n<li class=\"list-item\">\n<p id=\"p-7\">Mass casualties or more than $500 million in damages resulting in bodily injury or damage to property that would be a crime if humans did it; and<\/p>\n<\/li>\n<li class=\"list-item\">\n<p id=\"p-8\">Other comparably grave harms to public safety and security.<\/p>\n<\/li>\n<\/ul>\n<p id=\"p-9\">Under this bill, developers of large frontier models would be required to take numerous steps at three phases of development: some before training, some before use of such a model or making it available, and some during uses of covered models. Among the required steps would be installing a \u201ckill switch\u201d at the pre-training stage, taking reasonable measures to prevent models from posing unreasonable risks, and publishing redacted copies of the developers\u2019 safety and security protocols. (A \u201ckill switch\u201d would enable humans to stop an AI system from becoming an autonomous actor capable of inflicting critical harms.)<\/p>\n<p id=\"p-10\">Developers would also be required to hire independent third-party auditors to ensure compliance with the law\u2019s requirements. They would further be obliged to submit these audits and a statement of compliance annually with a state agency. Developers would further be responsible for reporting any safety incident of which they become aware to that agency within 72 hours of learning about it.<\/p>\n<p id=\"p-11\">The legislation authorized the California Attorney General to file lawsuits against frontier model developers who violated that law\u2019s requirements seeking penalties for up to 10% of the initial cost of model development for a first violation and up to 30% of development costs for subsequent violations. Whistleblowers who called attention to unreasonable risks that frontier models pose for causing critical harms would be protected against retaliation.<\/p>\n<p id=\"p-12\">In addition, SB 1047 would authorize establishment of a new California agency to publish implementing guidelines for compliance with the Act. This agency would have received the required audit and compliance reports, overseen model development, and proposed amendments as needed (including updates to the compute thresholds).<\/p>\n<\/section>\n<section id=\"sec3\" class=\"sec\">\n<h2 class=\"heading\">Comparing SB 1047 to EU\u2019S AI Act<\/h2>\n<p id=\"p-13\">SB 1047 and the European Union\u2019s AI Act both focus on safety issues posed by advanced AI systems and risks that AI systems could cause substantial societal harms. Both require the development of safety protocols, pre-deployment testing to ensure systems are safe and secure, and reporting requirements, including auditing by independent third parties and compliance reports. Both would impose substantial fines for developers\u2019 failure to comply with the acts\u2019 safety requirements.<\/p>\n<p id=\"p-14\">There are, however, significant differences between SB 1047 and the EU AI Act. For one thing, SB 1047 focused its safety requirements mainly on the developers of large frontier models rather than on deployers. Second, the California bill focused secondarily on those who fine-tune large frontier models, not just on initial developers. The AI Act does not address fine-tuning.<\/p>\n<p id=\"p-15\">Third, SB 1047 would require developers to install a \u201ckill switch\u201d so that the models can be turned off if the risks of critical harms are too great. The EU\u2019s AI Act does not require this. Fourth, the California bill assumed that the largest models are those that pose the most risks for society, whereas the AI Act does not focus on model size. Fifth, SB 1047 was intended to guard against those four specific types of critical harms, whereas the EU\u2019s AI Act has a broader conception of harms and risks that AI developers and deployers should design to avoid.<\/p>\n<\/section>\n<section id=\"sec4\" class=\"sec\">\n<h2 class=\"heading\">Proponents of SB 1047<\/h2>\n<p id=\"p-16\">Anthropic was the most prominent of the AI model developers to have endorsed SB 1047. (Its support came after the bill was amended to drop a criminal penalty provision and to substitute a \u201creasonable care\u201d instead of a \u201creasonable reassurance\u201d standard for the duty of care expected of large frontier model developers). Thirty-seven employees of leading AI developers expressed support for SB 1047 as well.<\/p>\n<p id=\"p-17\">Yoshua Bengio, Geoff Hinton, Stuart Russell, Bin Yu, and Larry Lessig are among the prominent proponents of SB 1047 as a \u201cbare minimum effective regulation.\u201d They believe that making developers of advanced frontier models responsible for averting critical harms is sound because these developers are in the best position to prevent such harms.<\/p>\n<p id=\"p-18\">Proponents consider SB 1047 to be \u201clight touch\u201d regulation because it does not try to control design decisions or impose specific protocols on developers. They believe that the public will not be adequately protected if malicious actors are the only persons or entities that society can hold responsible for grave harms.<\/p>\n<p id=\"p-19\">The AI Policy Institute reported that 65% of Californians support SB 1047 and more than 80% agree that advanced AI system developers should have to embed safety measures in the systems and should be accountable for catastrophic harms. Proponents further believe that SB 1047 will spur significant research and advance the state of the art in safety and security of AI models.<\/p>\n<p id=\"p-20\">Without this new regulatory regime, moreover, proponents believe developers who are willing to invest in safety and security will be at a competitive disadvantage to firms that cut corners on safety and security design and testing to get to market faster.<\/p>\n<\/section>\n<section id=\"sec5\" class=\"sec\">\n<h2 class=\"heading\">Opponents of SB 1047<\/h2>\n<p id=\"p-21\">Google, Meta, and OpenAI, along with associations of technology companies, as well as Marc Andreesen and Ben Horowitz, opposed SB 1047 in part because it focused on the development of models instead of on harmful uses of such models. These opponents are concerned this law will impede innovation and American competitiveness in AI industries.<\/p>\n<p id=\"p-22\">OpenAI argued that because SB 1047 heavily emphasizes national security harms and risks, it should be for the U.S. Congress, not the California legislature, to regulate AI systems to address these kinds of harms.<\/p>\n<p id=\"p-23\">Among SB 1047\u2019s opponents are many AI researchers, including notably Professors Fei Fei Li of Stanford and Jennifer Chayes of UC Berkeley. These researchers are concerned about the bill\u2019s impacts on the availability of advanced open models and weights to which researchers want access and on which they want to build.<\/p>\n<p id=\"p-24\">San Francisco Mayor London Breed and Congresswomen Nancy Pelosi and Zoe Lofgren were among the other prominent critics of SB 1047. Lofgren, who serves on a House subcommittee focused on science and technology issues, wrote an especially powerful letter to Governor Newsom expressing her reasons for opposing that bill. Among other things, Lofgren said that AI regulations should be based on demonstrated harms (such as deep fakes, misinformation, and discrimination), not hypothetical ones (such as those for which kill switches might be needed).<\/p>\n<p id=\"p-25\">The science of AI safety, noted Lofgren, is very early stages. The technical requirements that SB 1047 would impose on developers of large frontier models are thus premature. While the National Institute of Science and Technology aims to develop needed safety protocols and testing procedures, these measures are not yet in place. Nor are voluntary industry guides yet fully developed.<\/p>\n<p id=\"p-26\">Lofgren also questioned SB 1047\u2019s \u201ckill switch\u201d requirement. Although this might sound reasonable in theory, such a requirement would undermine the development of ecosystems around open models. She agreed with a report of the National Telecommunications and Information Administration that there is insufficient evidence of heightened risks from open models to justify banning them.<\/p>\n<p id=\"p-27\">Lofgren also expressed concern about innovation arbitrage. If California regulates AI industries too heavily or in inappropriate ways, it might lose its early leadership in this nascent industry sector. And U.S. competitiveness would be undermined.<\/p>\n<\/section>\n<section id=\"sec6\" class=\"sec\">\n<h2 class=\"heading\">Governor Newsom\u2019s Reactions<\/h2>\n<p id=\"p-28\">Governor Gavin Newsom issued a statement explaining his reasons for vetoing SB 1047. He pointed out that California is home to 32 of the world\u2019s leading AI companies. He worried that this law would harm innovation in California\u2019s AI industries. Regulation should, he believes, be based on empirical evidence and science.<\/p>\n<p id=\"p-29\">Newsom questioned whether the cost and amount of computing power needed for AI model training is the right regulatory threshold. He suggested it might be better to evaluate risks based on ecosystems in which AI systems were deployed or on uses of sensitive data. He warned that the bill\u2019s focus on very large models could give the public a false sense of security because smaller models may be equally or more dangerous as the ones SB 1047 would regulate. While recognizing the need for AI regulations to protect the public, Newsom observed that the AI technology industry is still in early stages and regulations need to be balanced and able to be adapted as the industry matures.<\/p>\n<p id=\"p-30\">The governor agreed with SB 1047\u2019s sponsors that it would be unwise to wait for a catastrophe to protect the public from AI risks and that AI firms should be held accountable for harms to which they have contributed. But SB 1047, in his view, was just not the right law at the right time for California.<\/p>\n<p id=\"p-31\">To demonstrate his commitment to ensuring proper attention to public safety, Governor Newsom appointed an expert committee of thought leaders to advise him further about how California can achieve the delicate policy balance between promoting the growth of AI industries and research communities and protecting the public against unreasonable risks of harm. Joining Fei Fei Li and Jennifer Chayes on this committee is Tino Cuellar, a former Stanford Law professor, a former California Supreme Court Justice, and now Executive Director of the Carnegie Institute for Peace.<\/p>\n<p id=\"p-32\">Despite vetoing SB 1047, the governor signed into law 19 other AI-related bills passed by the California legislature this year. Two of them regulate deep fakes, one obliges developers to make disclosures about AI training data, and one requires provenance data for AI-generated outputs.<\/p>\n<\/section>\n<section id=\"sec7\" class=\"sec\">\n<h2 class=\"heading\">Conclusion<\/h2>\n<p id=\"p-33\">The sponsors of SB 1047 seem to have carefully listened to and heeded warnings of some prominent computer scientists who are deeply and sincerely worried about AI systems causing critically serious harms to humankind. However, there is no consensus among scientists about AI public safety risks.<\/p>\n<p id=\"p-34\">Concerns that advanced AI systems, such as HAL in <i>2001: A Space Odyssey<\/i>, will take over and humans will not be able to stop them because their developers failed to install kill switches seem implausible. Legislation to regulate AI technologies should be based on empirical evidence of actual or imminent harms, not conjecture.<\/p>\n<p id=\"p-35\">In any event, regulation of AI systems that pose risks of national security harms would optimally be done at the national, not state, level. But the Trump Administration is less likely than the Biden Administration to focus on systemic risks of AI, so maybe the state of California should lead the way in formulating a regulatory regime to address these risks.<\/p>\n<\/section>\n<\/div>\n<\/article>\n","protected":false},"excerpt":{"rendered":"<p>Should governments decide what regulations are necessary to ensure safe development and deployment of AI technologies?<\/p>","protected":false},"featured_media":764931,"comment_status":"open","ping_status":"closed","template":"","meta":{"cacm_featured_image_credit":"","cacm_show_featured_image_credit":0,"byline":{"profiles":[{"type":"byline_id","atts":{"term_id":125,"post_id":209}}]},"lede":"<p>Why the recent statewide artificial intelligence regulation legislation was vetoed.<\/p>\n","table_of_contents":"<ul>\n<li><a href=\"#sec2\">Key Features of SB 1047<\/a><\/li>\n<li><a href=\"#sec3\">Comparing SB 1047 to EU\u2019S AI Act<\/a><\/li>\n<li><a href=\"#sec4\">Proponents of SB 1047<\/a><\/li>\n<li><a href=\"#sec5\">Opponents of SB 1047<\/a><\/li>\n<li><a href=\"#sec6\">Governor Newsom\u2019s Reactions<\/a><\/li>\n<li><a href=\"#sec7\">Conclusion<\/a><\/li>\n<\/ul>\n","cacm_plugin_article_dl_pub_date":"","cacm_plugin_article_dl_url":"","cacm_plugin_article_pdf_url":"","cacm_plugin_article_copyright_text":"","related_reading_search_string":"","related_reading_posts":[],"footnotes":""},"categories":[4],"issue":[19732],"section":[11],"content-type":[],"byline":[125],"class_list":["post-764612","digital-library","type-digital-library","status-publish","has-post-thumbnail","hentry","category-computing-profession","issue-march-2025","section-opinion"],"parsely":{"version":"1.1.0","meta":{"@context":"https:\/\/schema.org","@type":"NewsArticle","headline":"California\u2019s AI Act Vetoed","url":"http:\/\/cacm.acm.org\/opinion\/californias-ai-act-vetoed\/","mainEntityOfPage":{"@type":"WebPage","@id":"http:\/\/cacm.acm.org\/opinion\/californias-ai-act-vetoed\/"},"thumbnailUrl":"https:\/\/cacm.acm.org\/wp-content\/uploads\/2025\/01\/013125.OP_.Californias-AI-G.jpg?w=150&h=150&crop=1","image":{"@type":"ImageObject","url":"https:\/\/cacm.acm.org\/wp-content\/uploads\/2025\/01\/013125.OP_.Californias-AI-G.jpg"},"articleSection":"Computing Profession","author":[{"@type":"Person","name":"David Roman"}],"creator":["David Roman"],"publisher":{"@type":"Organization","name":"Communications of the ACM","logo":""},"keywords":[],"dateCreated":"2025-02-03T22:18:50Z","datePublished":"2025-02-03T22:18:50Z","dateModified":"2025-02-21T16:40:15Z"},"rendered":"<script type=\"application\/ld+json\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@type\":\"NewsArticle\",\"headline\":\"California\\u2019s AI Act Vetoed\",\"url\":\"http:\\\/\\\/cacm.acm.org\\\/opinion\\\/californias-ai-act-vetoed\\\/\",\"mainEntityOfPage\":{\"@type\":\"WebPage\",\"@id\":\"http:\\\/\\\/cacm.acm.org\\\/opinion\\\/californias-ai-act-vetoed\\\/\"},\"thumbnailUrl\":\"https:\\\/\\\/cacm.acm.org\\\/wp-content\\\/uploads\\\/2025\\\/01\\\/013125.OP_.Californias-AI-G.jpg?w=150&h=150&crop=1\",\"image\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/cacm.acm.org\\\/wp-content\\\/uploads\\\/2025\\\/01\\\/013125.OP_.Californias-AI-G.jpg\"},\"articleSection\":\"Computing Profession\",\"author\":[{\"@type\":\"Person\",\"name\":\"David Roman\"}],\"creator\":[\"David Roman\"],\"publisher\":{\"@type\":\"Organization\",\"name\":\"Communications of the ACM\",\"logo\":\"\"},\"keywords\":[],\"dateCreated\":\"2025-02-03T22:18:50Z\",\"datePublished\":\"2025-02-03T22:18:50Z\",\"dateModified\":\"2025-02-21T16:40:15Z\"}<\/script>","tracker_url":"https:\/\/cdn.parsely.com\/keys\/acm.org\/p.js"},"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/digital-library\/764612","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/digital-library"}],"about":[{"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/types\/digital-library"}],"replies":[{"embeddable":true,"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/comments?post=764612"}],"version-history":[{"count":4,"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/digital-library\/764612\/revisions"}],"predecessor-version":[{"id":764997,"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/digital-library\/764612\/revisions\/764997"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/media\/764931"}],"wp:attachment":[{"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/media?parent=764612"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/categories?post=764612"},{"taxonomy":"issue","embeddable":true,"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/issue?post=764612"},{"taxonomy":"section","embeddable":true,"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/section?post=764612"},{"taxonomy":"content-type","embeddable":true,"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/content-type?post=764612"},{"taxonomy":"byline","embeddable":true,"href":"https:\/\/cacm.acm.org\/wp-json\/wp\/v2\/byline?post=764612"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}