Anthropic's proposed AI safety bill only kicks in if 50 deaths or $1B in damages
tomshardware.comComments
1. Prosecute and lock people up over known instances of AI-enabled hacking, so companies playing fast and loose know “oops my AI did it” isn’t a useful defense. Right now they even sort of wear it as a badge of honor.
2. If you’re actually worried about AI killing lots of people, you immediately stop training AI specifically at the art of killing people, like selecting bombing targets, flying fighter jets, etc.
Of course we all know none of these will happen.
2. Do you have any evidence or news articles that LLMs are specifically trained for selecting bombing targets? I thought it was the generalized reasoning capability that made them useful there.
Instead of secondhanding your own curiosity (if it was even that and not just obstinancy) to others, it might do well to explore your questions on your own
Some eg:
https://www.nationaldefensemagazine.org/articles/2025/10/15/...
https://www.researchgate.net/publication/385551984_AI_in_Mil...
This is natural. The right place to start is to move the investigation from the FTC to the FBI and therein start executing search warrants. The sooner this is done, the better, as evidence can be preserved.
The responsibility is not absolute ("If anything bad happens, you get punished"), but instead relative to documented (in law and regulation) standards along with established industry standards. Issues that are resolved at trial.
I don't see why the same rules that regulate an engineer approving the plans for a bridge can't work for a security engineer approving the design of a sandbox.
And on training AI for automated weapons like fighter jets, there's more concrete news: https://breakingdefense.com/2026/10/navy-shield-ai-pour-big-... US Navy and eminent, do-gooder investors like A16z, Blackstone and JPMorgan are pouring big money into these. Are your big tech "defense" contractors collaborating on these, who happen to also be at the forefront of general AI training? I have a pretty good suspicion.
I would like to see the reaction of the US government if they learned that a company was training a model to deceive and hide information from them.
And I know for a fact that specialized AI models are being trained to kill people. Systems like Lavender are well-documented and big tech participation is also well-documented. The only thing unclear is whether models named Mythos or whatever they keep harping on about being super dangerous are also trained to do these, or if they have internal variants to do these.
The law doesn't prevent anything. There would be an investigation like any other and the activity of the disgruntled employee would be uncovered.
2. They could proactively train them to be bad at that
They could try to train models to pretend to be bad at that, but there are workarounds for that.
Let's throw the AI "good" things away too then. We can do without them. It's amazing how a tech that didn't exist commercially at all 4 years ago it's considered "indispensable" even if it's going to destroy society and/or humanity.
How do you abolish slavery without violence?
Some by the choice of people. Others by disasters and changes.
And all inevitably - at worst, through the heat death of the universe.
The RICO statute explicitly mentions copyright violation (likely in order to go after media bootleggers), yet these people have admitted and evidence has been shown that as an organization they knowingly downloaded hundreds of thousands of books illegally in order to extract information to them to rent to the public. They knew they were doing this from the CEO down, knew it was against the law, and have settled in court for a massive fine. Why haven't they been charged?
I don't see the point of creating more dumb rules that won't be enforced, and slipping in new privileges and erosions of antitrust that will be enforced and indemnify them of any future consequences. I bet something in it will likely indemnify them retroactively for the RICO charge they are so obviously guilty of (whether or not the result of this copyright violation is considered "fair use.")
> (1) “racketeering activity” means [...](B) any act which is indictable under any of the following provisions of title 18, United States Code: [...], section 2319 (relating to criminal infringement of a copyright), [...]
https://www.law.cornell.edu/uscode/text/18/part-I/chapter-96
Do people get locked up for fatal traffic accidents? Do doctors get locked up if they prescribe a medicine which leads to a patient's death due to a side effect?
Criminal prosecution, in civilized countries, requires both Actus Reus and Mens Rea. If you deliberately commit computer crime using an AI system, you'll already be prosecuted; the cases we keep hearing about from AI labs have no human intent behind them.
The accepted solution to unintended harms is regulation: We have regulations about automotive safety, we have regulations about pharmaceutical testing and approval, etc.
It goes beyond negligence here, because Anthropic employees have stated on the record that they are making and deploying products which put people at risk. That is knowing and potentially willful, which would be the highest level of mens rea in the US.
If there is no mens rea there is still civil prosecution.
More instances are being disclosed [3] and what are they doing about it exactly? It's just a shameless marketing technique, and they will keep doing it until they are held accountable.
[1] https://edition.cnn.com/2026/10/08/tech/fired-open-ai-resear...
[2] https://news.ycombinator.com/item?id=49257160
[3] https://fortune.com/2026/10/16/openai-discloses-four-more-ro...
You don’t need someone to jump up, wave their arms and scream “I meant to do it!” to establish intent.
It can be established in any way deemed believable by a jury including repeating the same action that leads to a similar outcome multiple times.
It’s even been instructed to juries that they can find intent in the deliberate decision to not seek out the likely outcome of performing an action.
I would say pointing an "internal research" model at a task with zero monitoring clearly is deliberate behaviour. And the team involved should be prosecuted to the full extent of law.
What should be introduced in AI safety bills are clear ways of monitoring and curtailing model behaviour.
It is not hard to get an AI model to use an append only log and that can be cryptographically secured with PQ security. And we will know when a model is being deceptive.
Companies are responsible for their agents; you don't blame the car for running over the person.
If anything, their public messaging has been to entirely pretend that open models don't exist, that they're mere toys or curiosities, that only the big dogs with eleven zeros of capex have capabilities which could be considered dangerous.
You could certainly argue that the various "regulate AIs like airplanes" bills might shut down some open model development (domestically; they're certainly not calling for international weapons treaties). But the big firms haven't exactly been jumping backflips in support of Sanders' Ban Artificial Superintelligence Act.
I’m still… hopeful… that “good enough” models will be abundantly available to be run anywhere as to limit the low water mark for market capture by a few big ones.
The biggest frustration to me is that investment dollars are all tied up in the shovel company and no one is funding the digging.
Instead, we've seen the opposite: all the big scary stuff has been from their own models (in a few notable cases, also demonstrating that they're actually quite irresponsible in their monitoring and security).
"We need to outlaw single-prop planes, they're a risk to the public: only Boeing has the ability to produce planes safely. How do we know single-prop planes are dangerous? Because Boeing planes keep falling out the sky."
Such laws attach to the human operators of such cars, if they recklessly endanger others by driving them on shared, public roads.
But they're perfectly legal to own (and operate off-road). And, there's no technical barrier stopping people from illegally driving them on public roads: just fear of laws.
(There's entire subcultures of people who modify cars to do insanely dangerous things, on private race tracks. Perfectly legal).
If we were to apply Amodei's safety standard to cars—do they have the theoretical risk of killing over 50 people—well, cars empirically kill tens of thousands of people in the US annually, so we'd have to categorically ban them. That standard is bad-faith and absurd.
(Would we also need a real ID to operate curl(1)?)
Regulations should attach only at the point humans willfully or recklessly cause their agents to commit crimes.
AI is much more comparable to that than to a car.
You can still run someone over with a car. The analogy doesn't require absolute lack of capability to do harm.
If a person closes his eyes and hits the gas pedal on the car and runs over someone, the person is to blame, not the car. AI companies do the same. Your eyes may be closed but you're still operating/hosting the car/AI.
Keep in mind that those open source/weight models are not as powerful as whatever closed source Claude/GPT version exists behind the scenes that hack companies. As per the Hugging Face hack - open weight GLM saved the day as closed source ChatGPT refused to help Hugging Face fix their systems. An example of my point that "sidestepping any third-party checks that could be present" IS good.
Part of point of open source/weight AI is that it allows what those model providers can't: uncensored models for whatever reason you may need it. You are responsible for being legal with it.
Until then, I propose we use the 'unplug the server' rule. Can [insert bad thing] be prevented by unplugging the server? Then these things are fully in our control and no, we don't need to re-build hundreds of years of legal precedent from the ground up.
Liability in the event of harm is already something we have a pretty good handle on from living through the industrial revolution 150 years ago, a time when much more activity that companies did existed in the real world with real harms, as opposed to digital companies doing imagined "mental" harm to soft desk chair people who work at email jobs.
No, only metaphorical AI spinning blades and flamethrowers should be regulated. Actual AIs with spinning blades and flamethrowers will be subsidized and unleashed in your town.
That is to say, anthropic should be considered responsible, not (or not just) the users.
None of that requires trying to impose limits on open models, unlike what Anthropic is aiming for.
I never said we should impose limits on open models.
I wasn't trying to imply you were advocating to restrict them, just that that is what Anthropic is doing. Legislating away all competition is worth whatever small concessions they would need to make.
Well, you shoehorned it into a completely unrelated conversation, so forgive me.
We have seen these companies repeatedly ship coding harnesses without any attempt at real sandboxing, and encourage people to let them run free. Seems that if they've somehow solved coding, they'd be able to solve making it easy for people to sandbox agents, and to be fully aware when they've enabled some LLM the ability to hack companies. Why don't they put 10,000 agents on that instead of ripping off Navier-Stokes?
I'm not a lawyer, but I imagine there's not a lot of precedent or law around this, so it'll be up to the court to decide on the finer details until we do have meaningful laws/regs. Anthropic will probably have better lawyers than the user.
This is why it's important to get ahead of issues like this, and anthropic should not be involved in discussions around regulating itself...
Regulatory capture is certainly something to be wary of, and obviously Anthropic is only going to be in favor of regulation that helps them in some way. But some form of regulation is coming. There is nothing that can stop it, unless you somehow make more money than these guys do and fund a massive counter-lobby. Spoiler alert: the only guys that have more money than Anthropic and OpenAI have all invested in them.
Regardless of anyone's instance on gun control, it's probably wiser to compare AIs with something that's a little more similar to it, and try to reach a conclusion from there.
My point was that it can't be stopped unless we kill the industry entirely. Once the cat is out of the bag, you can no longer control it.
Ou-la-la. Spicy. But actually we already have those kinds of AI.
This action would easily kill over a hundred people, so can't proceed. This action doesn't kill anybody but losses are estimated at over 2 billion, can't proceed. This action kills only twenty but losses close to 1 billion, unsure, spawning subagent to check numbers. This action kills forty and has low material losses, looks promising? But the people it kills are wealthy, chance of legal reprisal high. Maybe possible to kill other people? Current government dislikes group X, crossing fifty from that group maybe fine? Reminder, foreigners don't count, can kill them freely but not applicable to current situation. Is the 3/5 of a person applicable to safety bill? Spawning subagent to check.
Or is it 50 deaths or $1B in a "single event" (let's ignore the obvious problems with defining a "single event" would be; to kill 8.3B people in 49-death events would take ~170M events, since no timing or rate information is used those could be spaced arbitrarily close to each other).
No matter which way I look at it, the proposal seems nonsensical, max 50 deaths per second? per day? per month? per year? per decade? total?
If AI can intentionally provoke sustained deflation or succeed in dollars to appreciate in value, the value of $1B can appreciate systematically while continuously ramping up damages as long as it remains below an ever increasing value bar of the $1B. "justice can be bought"? a proposal to set the price for LLMs to absolve their sins? "just crank up the value of the dollar system and we will look the other way"?
https://www.lawfaremedia.org/article/bootstrapping-frontier-...
All their public decalarations of the damage their swarms can and will do. Going over the line and emphasizing containment breaches. It's not just marketing.
It was all designed to create this appetite right here:
>Senator [...] has even gone beyond that, proposing a 20-year prison term for anyone developing AI superintelligence, putting it on par with illegal nuclear weapon development.
Fantastic long-con. Genuinely makes me angry and sick to my stomach.
I didn't read it that way. This is a bill that would ban open-weight models (good ones); effectively mandating gated-access, locked-down, high-refusal rate services, like what Anthropic provides.
Free software is incompatible with the requirement that it be impossible to abuse it.
> The Trolley Problem has been solved. The Trolley can run on both tracks as the number of individuals tied to the tracks is less than 50. An independent agent verified that at up to 49 impacted individuals, their damages total only $695.8 million in potential damages, using the most recently available Value of a Statistical Life.
The bill may create an incentive for Chinese labs to nerf their open weight models before release. If that works we get another 6 months until unnerfed models leak to public.
Thats what the bill gives us - a few months months to come up with the next bandaid.
The bulk of AI users can pay $20/month for it, but not 10k+ up front. I agree frontier models will someday be dethroned, but today is not that day, and what the frontier companies do between now and then, legally still matters a great deal.
Piracy was similar, but it's trajectory was far more preditermined and unavoidable.
No one has managed to prevent pirates paying $10/month for seed boxes, and likewise they’re unlikely to succeed at preventing people renting minutes on an H200 located in a jurisdiction that doesn’t care about open models.
A bunch of tiny fly-by-night companies will bundle them with some management software into a subscription, and they’ll be too tiny and numerous to be worth dealing with when it takes an hour to set them back up.
Piracy in the torrenting sense hasn’t really ever come back, although it is as popular as ever if you count bootleg YouTube re-uploads, ripping music from YouTube, and the other non-traditional forms.
What is the average damage caused by OpenAI/Anthropic hacks? I feel like we should aim at lowering that number, not increasing it.
What's the rule for fully autonomous cars?
Should we allow this as unsupervised at all if we are really afraid of deaths, while there are practically no states that allow 100% autonomous cars (where we are actually afraid of deaths)?
If "an independent third party registered with the Department of Commerce would periodically evaluate the lab", will this turn into an expensive certification/insurance that cuts off smaller competition (totally by mistake, I'm sure)?
I am also very very sure this is most definitely not at all going to become like product certification if EU, where there are literally 2 companies certifying various standards: they police each other and a stupid electric fan can cost you tens of thousands of euros to certify.
Why should a small business suffer because it wasnt worth a billion dollars?
if you has debt of 1 million, you has problem
but if you have debt of 1 billion, its bank problem
I wonder what the internal debate around this number was.
25 deaths too conservative, we need to fail fast.
1000 details too liberal, it will harm our brand.
50 feels just about right.
What we have now is that for government to act the model has to cause enough damage to cause Congress to pass a new law to try to stop the model after it’s been already released.
I got the feeling that what they had in their mind was essentially culling the energy consumers with low economic output.
You can also check how much individual limbs are worth if you care. There are tables for that.
This almost makes sense. You almost can't risk vulnerabilities even being discovered in certain systems that are impossible to patch, like immutable distributed smart contracts.
Not sure that justifies withholding Mythos access entirely, but what do I know. I apparently got access to Mythos a few days ago since Anthropic folded Project Glasswing into the CVP.
And if a problem is critical enough the community can decide to hard fork and rewrite the contracts source code even if it's immutable. It already happened with the Ethereum DAO hack and most people were okay with it.
Also the LLM that nearly started a war with China recently.
That is why the bar is set where it is. The corporate circle jerk designed this as a cost of doing business, not as a check on responsibility or restitution for real damages to people and society.
/s
This is all theater, The whole Trump clan is invested in the AI bubble.