As some of you may know I frequently read Sci-Fi author Charles Stross's blog. This is his latest entry:
http://www.antipope.org/charlie/blog-static/2023/11/dont-create-the-torment-nexus.html

One of the comments said this, which I found pretty illuminating and yeah well... :

"Another issue is the nature of AI. I’d posited an old school super brain n a box, whose tragedy was that they’d know how to fix climate change, but they wouldn’t have the resources to even save themselves. Instead we’ve basically got horribly complicated maximum likelihood calculators that literally can’t learn from each other so far.

So you give a modern AI the problem of, say, dealing with billions of people on the move due to climate change while protecting national sovereignty and property rights, AND the answer must also protect the company’s finances and the manager’s job status.

Am I the only one who thinks that the maximum likelihood model might well choose triggering a nuclear war as the best way to meet all those goals? Its not necessarily smart in ways that allow it to avoid batshit options like trying to zero out functions and end simulation runs to meet give problem parameters."