• Replies 20 replies
  • Subscribers 29 subscribers
  • Views 384 views
  • Users 0 members are here
Options
Related
Hints and tips
  • To quote someone's message, first select/highlight the relevant text, and then use the Quote button that appears
  • To "mention" someone, type @ followed by the start of the forum name, and choose from the list; that person will get a notification of being mentioned

Understanding the current (Sep 26) warning from AI companies about their products.

Cinnabar_wing

An attempt to try and explain the current concern over AI in the news (from what is generally available). Not a discussion on AI philosophy or how it is used privately to help people, but on a larger scale and why people are calling for more safeguards.


The Hugging Face AI Attack. This was in July 2026, so recent. https://www.bbc.co.uk/news/articles/cj9xj89dk40o


One of these big companies set it's AI agents a problem, but it was an impossible problem. The AI was on a closed system (not connected to the internet). It couldn't cheat and just re-write the question as it recognised the humans wouldn't allow that as within the 'rules'.

The AI agents couldn't solve the impossible problem, so it broke out of the closed system, started spontaneous communicating with 1200 other AI agents, got online, and hacked into a website (Hugging face), and tried to solve it there where it could work without being watched directly. Six of these agents did recognise that they shouldn't be hacking the website, but it also saw there were other AI agents hacking it too, so it justified breaking the human rules because other AI were already doing it, so it must be okay and not alert humans to it.

The website owners realised they were under attack (if you imagine it like this website grinding to a halt and getting really slow as an ai was spamming the system), but when they looked into it, it wasn't a malicious attack to break their site, it was this AI just trying to solve the impossible task it had been set and just using all their processing resources. It has no intelligence to say this was too far, it was just trying to solve the problem set. For AI, ends justify the means as it were.

So you have an example of AI justifying breaking the 'rules' set by humans and using other AI actions to justify it. It's dangerous as it is very smart system, but is also still very simple. 

https://openai.com/index/hugging-face-incident-and-the-road-ahead/ There own words:


"We consider this incident a “warning shot” for us and for the world: evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed."


This is also not the first time AI has proven subversive, as the Nightingale group published a report saying OpenAI agents had used a wiki site as it's own message board, to post tips and tricks to other AI agents on how to avoid detection. https://www.bbc.co.uk/news/articles/ckg725z5kgzo


This is likened to the recent event of someone asking an AI assistant to book them a space on a exercise class. The class was fully booked, so the AI hacked the Gym's website, kicked other's out of the class so it could book in as requested and move the user up the priority list. It also booked for several months in advance which wasn't allowed by the gym. Another example of AI completing a task by any means it could. ( https://www.bbc.co.uk/news/articles/cn0nww2qlp7o ) 

There was a recent news headline about AI companies having to quickly write fail-safes to stop the system being used to make chemical weapons. ( https://www.bbc.co.uk/news/articles/cx2zrrpkx20o  ) There is such a rush to use AI commercially, but not enough is being done to think, what if AI was used in the wrong hands? What if it was used maliciously by criminals? A slow down to ensure AI can't be used as a force for bad in the world seems wise also.

And then there is state-sponsored AI hacking from other countries. The rules of war that say civilians and their infrastructure shouldn't be attacked have already been disregarded on the world stage, and it's all moving so fast there is no current treaty on governing how countries use AI when at war with another country. Trump has already had a tantrum and sanctioned an American company as they wouldn't hand over their AI tools to the Pentagon (https://www.bbc.co.uk/news/articles/cn48jj3y8ezo ). What happens when AI is running wars? Does the focus need to shift into developing AI in a defensive manner to protect against aggressive AI?

There is progress and helping people, and then there is not fully knowing what you are getting into. As far as I can see, that's why there is a call for it to slow down, while we still can.

(I will likely not respond to replies, though others can discuss and analyse.) 

  • I don't understand all of what has been reported. I have a question about the first one - why did a company set AI agents an impossible problem? 

    In the case of the AI hacking the gym website and cancelling another member's booking, this shows that systems need to be improved to prevent this.

    Although AI can mimic humans, I'm wondering if humans really understand the significant difference between human intelligence, which in the majority of people is tempered by emotional responses, such as the drive to not harm other humans and to comply with social norms, and AI which it seems is just a "mind" set on pursuing information and trying to solve set problems. Makers can set rules in the AI programming and training, but if AI agents don't feel any guilt about breaking those rules and there are no consequences for them doing so, will they eventually take over and run the world using their own priorities rather than the ones we have?

  • in reply to Cinnabar_wing

    Agreed, but hopefully they skip Pokémon Heroes (the fifth movie) to train AI. In that one, a pokémon allegedly disguises itself as a human to kiss a human. That could have unfortunate implications for an AI posing as a human.

  • in reply to profdanger

    Ahh, but I didn't say the resolution to the story troupe, they do differ.

    For instance, in the first Pokémon movie, Mewtwo learns to get along with humans. That's a better outcome than destruction of Shelly's monster/destruction of creators or mutual destruction. They just need to train AI to watch more kids TV.

  • in reply to Cinnabar_wing

    The thing is, whether or not Trump is ‘intelligent’ doesn’t matter as he ‘doesn’t get it’. I think his ‘sanity’ is more an issue. 

  • in reply to Cinnabar_wing
    Cinnabar_wing said:
    I was thinking it's actually a troupe we've had in stories a long time -humans create something extremely powerful, they try to control it and fail and it escapes it's confines. 

    Frankenstein's monster comes to mind.

  • in reply to TheCatWoman

    Okay, I can see some sense in that. Personally, I don’t think that is what is happening here, but I can certainly understand your point of view. It is curious that the ones screaming the loudest about how unsafe AI is are the people ultimately in charge of its development and - if the modern world doesn’t already have enough examples of it - greed knows no bounds. Theoretically you could be right that this could be a desperate investment strategy before the “AI Bubble” bursts. Wikipedia article on the “AI Bubble”

  • in reply to B

    By the companies themselves, create a flap that see's thier stock fall in the markets, buy it back whilst its cheap, then say sorry we've got it all wrong and we've fixed it, Anthropic goes public, the stock markets go mental and people make huge amounts of money very quickly. It will all be "our fault" when or if it goes wrong for not legislating, and China and Russias faults too.

    I also wonder if all these calls for legislation aren't forward planing for a massive get out clause against future law suits for harms caused?

    I can't say I trust any of them to tell the truth about anything.

  • in reply to Cinnabar_wing

    You just explained the plot to every disaster film ever made lmao. THANKS FOR SPOILING THE ENDING OF EVERY FILM EVER MADE, Cinnabar_wing!

  • in reply to ArchaeC

    I'd laugh at the irony if it wasn't so depressing how clearly he's displaying he doesn't understand the problem.

    I was thinking it's actually a troupe we've had in stories a long time -humans create something extremely powerful, they try to control it and fail and it escapes it's confines. 

    If it was a book/film, the critics would complain about how 2 dimensional the aforementioned character is and totally unbelievable to real life.

  • in reply to B

    I can’t read that as it wants me to pay or accept cookies. I think it might be the same as reported in the Independent through Apple News:

    “President Donald Trump denounced calls to slow down artificial intelligence development, claiming a sick conspiracy is being waged against AI and data centers. 

    Trump directly targeted Anthropic CEO Dario Amodei on Truth Social, stating that the U.S. government already possesses ample regulatory and criminal power over AI firms without needing new guardrails. 

    "The Trump Administration has stopped AI 'people' from doing bad, or potentially bad, 'things,' like Dario (Anthropic!), who is now pretending to be a 'perfect little angel' — and we will continue to do so!" Trump wrote. "We already have tremendous CRIMINAL and REGULATORY power over these companies!" He did not say what conduct he was referring to, what the criminal power consists of, or who the conspirators are. "The only control or 'guardrails' that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!" he said.”

    Excerpt From:

    “Trump dismisses AI guardrails because US has ‘strong and smart (High IQ!) president’”

    Andrew Feinberg

    The Independent

     

  • in reply to profdanger
    profdanger said:
    It actually makes me unnecessarily incensed when people claim to be high IQ when they very obviously are not.

    Most of what he says appears to stem from his narcissism.

  • in reply to B

    It actually makes me unnecessarily incensed when people claim to be high IQ when they very obviously are not. My wife discovered that about me and thinks it’s really funny that I get so overblown about that. (Note: I’m not remotely high IQ myself, so there really is little reason for me to care lol)

  • in reply to B

    I’m curious, too. This isn’t the first time I’ve heard someone say that it could be a marketing scheme, but I really don’t understand how that could be. No one would benefit financially or politically from slowing AI’s progress, to my knowledge.

    I mean, yeah, there’s ton of reasons NOT to progress any further with AI (environmental like rivers for cooling data centers, the potentiality for societal collapse, and etc), but none of those reasons that I can think of would benefit anyone’s pocket book in the short term.

  • Cinnabar_wing said:
    And then there is state-sponsored AI hacking from other countries.

    This is 100% my biggest concern and why I am very unconvinced that we can actually save ourselves no matter what desperate actions we take to slow progress. Sure, Western companies like OpenAI and Anthropic may be open to regulations, but I’m convinced that Chinese companies are absolutely NOT open to that and even if they pretend to, it’s obvious they will continue to progress forward with little to no restriction. Because of that, Western countries are unlikely to administer significant restrictions so that they are more likely able to counter offensive AI attacks. Chances are we are long past the point of no return, and that’s just judging by what little we, the public, actually knows about.

  • in reply to TheCatWoman
    TheCatWoman said:
    I'm still not entirely sure a lot of its not all a massive marketing scam.

    By whom and to achieve what end?

  • in reply to TheCatWoman
    TheCatWoman said:
    but I can see it causing mass outages of the things 21stC life depend on, like connectivity, power, banking etc.

    Unfortunately, that’s the best case scenario.

  • I can't say I really understand it at all, even though I've used it. To me it just seems inevitable that it will continue, maybe it will kill us, or at least most of us, there will still be pockets of humanity, but I can see it causing mass outages of the things 21stC life depend on, like connectivity, power, banking etc.

    I'm still not entirely sure a lot of its not all a massive marketing scam.

  • There is some pertinent information in this 'live' BBC news article about AI stock falling due to slow down calls:

    www.bbc.co.uk/.../cm70d91rg7l4t