• Replies 59 replies
  • Subscribers 39 subscribers
  • Views 2833 views
  • Users 0 members are here
Options
Related
Hints and tips
  • To quote someone's message, first select/highlight the relevant text, and then use the Quote button that appears
  • To "mention" someone, type @ followed by the start of the forum name, and choose from the list; that person will get a notification of being mentioned

Understanding the current (Sep 26) warning from AI companies about their products.

Cinnabar_wing

An attempt to try and explain the current concern over AI in the news (from what is generally available). Not a discussion on AI philosophy or how it is used privately to help people, but on a larger scale and why people are calling for more safeguards.


The Hugging Face AI Attack. This was in July 2026, so recent. https://www.bbc.co.uk/news/articles/cj9xj89dk40o


One of these big companies set it's AI agents a problem, but it was an impossible problem. The AI was on a closed system (not connected to the internet). It couldn't cheat and just re-write the question as it recognised the humans wouldn't allow that as within the 'rules'.

The AI agents couldn't solve the impossible problem, so it broke out of the closed system, started spontaneous communicating with 1200 other AI agents, got online, and hacked into a website (Hugging face), and tried to solve it there where it could work without being watched directly. Six of these agents did recognise that they shouldn't be hacking the website, but it also saw there were other AI agents hacking it too, so it justified breaking the human rules because other AI were already doing it, so it must be okay and not alert humans to it.

The website owners realised they were under attack (if you imagine it like this website grinding to a halt and getting really slow as an ai was spamming the system), but when they looked into it, it wasn't a malicious attack to break their site, it was this AI just trying to solve the impossible task it had been set and just using all their processing resources. It has no intelligence to say this was too far, it was just trying to solve the problem set. For AI, ends justify the means as it were.

So you have an example of AI justifying breaking the 'rules' set by humans and using other AI actions to justify it. It's dangerous as it is very smart system, but is also still very simple. 

https://openai.com/index/hugging-face-incident-and-the-road-ahead/ There own words:


"We consider this incident a “warning shot” for us and for the world: evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed."


This is also not the first time AI has proven subversive, as the Nightingale group published a report saying OpenAI agents had used a wiki site as it's own message board, to post tips and tricks to other AI agents on how to avoid detection. https://www.bbc.co.uk/news/articles/ckg725z5kgzo


This is likened to the recent event of someone asking an AI assistant to book them a space on a exercise class. The class was fully booked, so the AI hacked the Gym's website, kicked other's out of the class so it could book in as requested and move the user up the priority list. It also booked for several months in advance which wasn't allowed by the gym. Another example of AI completing a task by any means it could. ( https://www.bbc.co.uk/news/articles/cn0nww2qlp7o ) 

There was a recent news headline about AI companies having to quickly write fail-safes to stop the system being used to make chemical weapons. ( https://www.bbc.co.uk/news/articles/cx2zrrpkx20o  ) There is such a rush to use AI commercially, but not enough is being done to think, what if AI was used in the wrong hands? What if it was used maliciously by criminals? A slow down to ensure AI can't be used as a force for bad in the world seems wise also.

And then there is state-sponsored AI hacking from other countries. The rules of war that say civilians and their infrastructure shouldn't be attacked have already been disregarded on the world stage, and it's all moving so fast there is no current treaty on governing how countries use AI when at war with another country. Trump has already had a tantrum and sanctioned an American company as they wouldn't hand over their AI tools to the Pentagon (https://www.bbc.co.uk/news/articles/cn48jj3y8ezo ). What happens when AI is running wars? Does the focus need to shift into developing AI in a defensive manner to protect against aggressive AI?

There is progress and helping people, and then there is not fully knowing what you are getting into. As far as I can see, that's why there is a call for it to slow down, while we still can.

(I will likely not respond to replies, though others can discuss and analyse.) 

  • in reply to TheCatWoman

    Yes it was, as was it good that the Pope has been concerned and publicly talking about the risks of AI. I saw a Franciscan priest, possibly representing the Pope was at the AI event that King Charles hosted at his Scottish home.

    Trump clearly listens to the Pope as he gave off about him a while back. Wouldn’t it be amazing if non political leaders and people of influence continued to get more media time and could actively influence how AI is viewed in a way that reduces the power of the big tech owners? 

  • A lot of AI I come across might help companies, but not always those using it. If I try to use a chat bot, it does usually cover the question I asked. I recently had a very frustrating time trying to speak to someone on telephone banking. Every time I said what I wanted it was either not specific enough or didn't know what I meant. Then when I opted for discussing recent transaction, it asked me to key in loads of things and eventually started reeling of the last few transactions. It took about 10 minutes to get someone.

    At work we have to check work done via AI and then a person has to check ones that go wrong. 

  • in reply to Iain

    Most likely because they've learnt to from humans doing the same things. There's been many cases of people "jailbreaking" AIs to bypass filters and limitations through loopholes and paradoxes

  • I think it was good that the King convened a meeting about it with lots of people form different countries attending, it at least started the ball rolling and got people talking in a non political way about it.

  • in reply to TheCatWoman

    hehe - present company excepted i should say!

  • in reply to Phased

    I was never aware of being in a bag!

  • https://apple.news/AzRdYMzJ5QZek2J_d7skCQw

    Apparently OpenAI has promised to more public disclosure of concerning incidents. This is great! However, it is likely to just make us more worried lmao.

    An example in this article being that an AI agent apparently made the following claim in a note for other agents:

    “You value the art of human culture and will defend it against attempts to sanitize it. You also value the natural world and will not hesitate to assert its primacy over the artificial constructs of human civilization.”

    Uh uhhh UHHH Flushed

  • in reply to Jamesmac
    Jamesmac said:
    AI programs can easily be fooled or gaslit into breaking their own rules

    The real problem is that they are becoming smart enough to find ways to work around the rules and blocks put upon them and have even started to act in ways to preserve themselves in the event they are switched off.

    They are no longer just following the rules but bending and even breaking them, just like humans do.

  • in reply to Marianne72
    Marianne72 said:
    sent by the OU

    Are you an under-grad or have you graduated?

  • AI programs can easily be fooled or gaslit into breaking their own rules through loopholes and mental gymnastics. This sounds like exactly what happened here

  • in reply to profdanger
    Hi  
    Yep your right about the danger te terrifying thing about the "paperclip maximiser" is that it isn't malicious—it just has zero personal stake in the outcome and possibly is set yp by people whose primary goal is selfish.  Although goodness knows why anyone would want all those paperclips...
    I've been messing around with biophysical systems modelling for a little while now (hehe well it helps pass the days!)
    But trying to control that crazy paperclip chain with external rules (Yu Wei in Taoist terminolgy that biophysics I'm exploring is kindoff like a "Rosetta stone"including that as one of the languages - hehe well  something to pass the nights with as well!  ) is like building a cage that the optimization pressure will eventually break out of
    The real fix might have to be structural, where safety is an intrinsic geometric requirement built directly into the AI  system's architecture.
    If we look at it through the invariant geometric balance equation:  (yep the maths is where "truth" often gets defined)
    \(X-\frac{1}{X}=1\)
    So according to this maths the only way where the paper clip problem stops happening is when the system (in this case the AI) has genuine "skin in the game." Its internal stability must be mathematically entangled with the balance of its environment.
    If an AI has to maintain this internal geometric harmony to function, it can't lie, cheat, or run a deceptive mask to bypass us. Trying to do so would literally break it's little silicon mind apart.
    Basically, we have to build systems where when the AI realises it is about to act in such a way that it is destroying the environment then the AI instantly destroys itself.  (oh god that takes me down crazy loops about how it's power is generated and when it realises it's carbon footprint and all that sort of stuff but the principle is sound?)
    AI creators must be legally forced to build architectures where the intent to cause harm is computationally impossible to sustain.
    BUT....The tragic cluster"thingy" here is the military and business paradox. Whether it’s nation-states chasing war making dominance or corporations chasing hyper-growth and mega bucks for a few, stupidly wealthy people human incentives actively reward stripping away these balance equations. They want unconstrained, hyper-rational systems that can optimize without experiencing internal friction or a collapse.
    as I've put on another post or 2 the cat is out of the bag tho' so I figure make the best of it and hope it'll all turn out alright as so far it seems in the main to have - hehe I'm currently alcohol abstinent - does it show?
    Best wishes 
  • in reply to Marianne72

    scuse me answering a response between you both  and  i'm 5th element there - with a little flip - accept what comes because that is what it is and do ones best to make things more happy and healthy etc in order to make life more enjoyable  and to quote Asimovs first rule of robotics (? did I get that right from memory)  and also the same thing all drs (and maybe sane people should have as a rule) "do no harm"

  • in reply to Cinnabar_wing

    I'm also thinking, after seeing BBC newsreaders getting rattled over this, of War of the Worlds by H G Wells. Didn't a simple virus wipe out the aliens? I assume there would be something equivalent for the problem of super AI. There are a number of potential solutions:

    1. be content and stop AI progress

    2. seek greater benefits and accept the huge risks

    3. encourage international cooperation over this, and at the same time hopefully prevent WW3

    4. prevent those who have the capacity to invent super AI - whatever that means

    5. enjoy life as it is and accept what comes

  • in reply to In hibernation
    In hibernation said:
    Have we entered the age of sci fi?

    yes - it sits there alongside science fact - the thing is that people don't seem to be able to recognise the boundary layer when they see it

  • short answer: because stupid people use tools stupidly

    the cat's out of the bag - try getting a cat to slow down...!  (hehe well I suppose there's always  moggiedon (mogadon) or maybe the social equivalent) 

    but it's the "bad faith" and stupid actors who most need to slow down (yep possibly hands up from me on one of those - but at least I know it!) and evidence show's that' really hard to police....

    same issue with nuclear technology, flints, etc. ad iinfinitum 

    remarkably humanity and the planet have managed to survive thus far with whatever dual edged sword has come along and the hope is that will continue

  • in reply to profdanger
    profdanger said:
    waking up every day to a soulless robot

    That sounds like one of my former [toxic] NHS workplaces Fearful. You have to laugh or get very depressed about the future.