Last week was amazon, now its microsoft

D said:

Do you know how long it would take to replace a cable?

Service degradation lasted months there so I guess this would be the average repair time.

The cables are not typically modular so the whole thing would have be be laid again from scratch which takes weeks just to feed it from the back of a ship, never mind the route planning around obstacles like underwater volcanoes.

The cost of the cable itself it likely to be in the millions of pounds so an expensive proposition in an area like the Pacific Ring of Fire.

That's quite interesting about the undersea cables and volcanic activity. Do you know how long it would take to replace a cable?

Iain said:

I guess the vendors are choosing convenience over inclusion since it makes financial sense for them.

I was talking to an owner of a cashless bar once and they noted how it made a lot of things much easier. Not having to check for forged notes, no need to make sure they had the correct change, no taking the money to the bank or having it collected, no cash left on the premises, etc. I guess the savings/simplicity of that offset the transaction fees. 

The part of it I don't like is tips. When you're ordering and paying for things on an app or website, they will often ask if you at the point of ordering if you want to leave a tip, but I don't want to leave a tip and then find the service is bad. If the service is good though, I'm then left without a way to leave a tip. They should let you scan the QR codes again or something. 

I do somewhat miss terrestrial tv and having to get a tv guide to know what was going to come on the screen. You’d know a certain film was coming on in the week and made a note to make sure you switched on for it. There’s too much choice nowadays and we’ve been spoilt so much by technology to the point that we rely on it for everything. I’d miss the internet because I’m a habitual user of it but wouldn’t say I am addicted, I am not a user of any social media other than YouTube but that’s more for entertainment purposes. 

less related, the submarine cable map is something i like to look at sometimes.

this is more technical, but may be interesting for some members (DDoS, CloudFlare).

TheCatWoman said:

This is the sort of thing we all get told so often and yet when one big server goes down it takes lots of things with it, what happens if you get a chain reaction of overloaded servers?

This wasn't one big server that went down. Having "one big server" is extremely rare now. This was probably human error, and probably was a chain reaction as you mention. One incorrect configuration which replicated to many places. That's probably why it took so long for them to correct it.

Microsoft and Amazon will have mechanisms in place so that these types of configuration are limited to one region though, and they have a lot of regions. Also the first thing that usually happens with a large outage is you stop making changes as that might make things worse, and then you have to untangle everything. The chances of an entire cloud like Microsoft or Amazon going down are extremely slim. The recent outages are about as large as they should.

The only thing I can think of that might be bigger would be very intense solar activity, but at that point we've probably got bigger problems that the internet going down. We'd probably have to get the power grid up first. 

TheCatWoman said:

what if it's a hostile actor who cuts undersea cables? 

As with the servers, there is redundancy built in. They don't just run one cable they run many. They also run them along different paths, and connect them to different places on the shore. Companies like Google even run their own dedicated cables. I believe there are also satellite links, but those are slower. The undersea cables have been broken before by ship anchors.

https://youtu.be/pJU-KYMREbQ

As in my other comment, all of this doesn't mean you shouldn't be resilient. The whole internet going down might be unlikely, and it might be unlikely for individual servers to cause issues, but these large outages are probably inevitable, and we should be prepared for them. Have debit/credit card from multiple banks, ideally across mastercard and visa, have cash on hand, etc.

It's also worth considering that a lot of infrastructure is reliant on the internet as well. For example, I'm looking at an EV charger at the moment and most of them require an internet connection for some functions. I asked a company what happens if I turn the charger off via the app, and then their services go down. How do I turn the charger back on? Unsurprisingly they didn't have a good answer.

You're asking great questions that I wish more people would ask. I do like technology and think it can help with accessibility and quality of life, but far too much is not only dependent on it but dependent on it being connected to the web. Not everything needs a technical solution. Sometimes the simple things like writing something on pen and paper are far better. 

I think a closely related problem to what you ask, is what happens if power plants and energy import lines are damaged; in that case, you will struggle as if the internet were shut down. Because most (users) depend on electricity being available.

Also, in many cases you don't need complete shutdown to cause a disaster. It may be enough to make the available resources less than the demand. In a DDoS attack, this can happen as well. You get zone-blackouts, and also seemingly the internet goes down, just because the remaining servers can not respond to the demand / requests.

I don't think the resilience the electric grid and power generation mechanisms is so good. It is one of the reasons why, had I enough money, I would get a solar panel, and maybe a fuel-based electricity generator.

Just remember what happened in Spain a few months ago. If I remember correctly, their grid was not very resilient, and for a zone of the country there was little or no electricity. 

My point is a bit vague so in a sentence is: if a service does not go down completely, it does not mean it is accessible or useful. It can still be as if it were down completely, because of lack of response, or lack to respond to the demand.

There are a few ways people work around that issue, but causes a lot of disruption. They may get a generator or solar panels, and in some areas the may still get some access to internet, maybe satellite internet if antennas are damaged, but what about those that don't?

TheCatWoman said:

what if it's a hostile actor who cuts undersea cables? 

There is a whole network of redundant connectivity that should kick in and still let the traffic flow around the world, just more slowly.

When I used to work in Bali (Indonesia) there would often be undersea volvanic / tectonic activity that would break the undersea cables so they would switch the the satellite connectivity which, while slower, was still able to get emails flowing and web pages loading.

With the likes of Starlink and other companies there is a lot more bandwidth that does not rely on the subsea cables any more and most big data transfter companies (Amazon, Apple, Netflix etc) have their data in different geographic locations so are not impacted by this sort of thing.

TheCatWoman said:

places that don't take cash annoy me, especially those who expect everyone to have a smart phone, know how to use it and have all the different payment apps and internet banking etc.

From the vendors perspective, the only significant portion of the UK who may not have a smartphone or type of electronic payment (debit card etc) is the 65+ group who are unlikely to be big consumers for them. I suspect the loss of potential earnings is small enough to justify the convenience of not needing to handle the notes/coins, maintain change levels and have the hassle of cashing up and going to the bank.

Even amongst the 65+ group I believe almost 80% have smart phones and probably a lot more have cards to pay with, so the numbers are statistically insignificant for the vendors.

In theory you only need the digital wallet that is installed on all smartphones to pay if you don't have a card and no banking app is needed. I think all bank current accounts offer at least a debit card which can be used instead of a smart phone too.

I guess the vendors are choosing convenience over inclusion since it makes financial sense for them.

This is the sort of thing we all get told so often and yet when one big server goes down it takes lots of things with it, what happens if you get a chain reaction of overloaded servers?

Like Robert, I too believe we should be more resilient, mistakes are bad enough, but what if it's a hostile actor who cuts undersea cables? 

We have some cash only shop here, theres a cash only chippy down the road, luckily it's opposite a cash machine, but places that don't take cash annoy me, especially those who expect everyone to have a smart phone, know how to use it and have all the different payment apps and internet banking etc.

Absoltely! My comment was more pointing out that these large scale outages are typically rare, and the chances of the entrie internet going down are very slim. However, these outages are also probably inevitable. It's the nature of them that when things do go wrong to the level that it causes problems, it causes lots of problems.

It makes absolute sense to keep some cash for these situations, and even split your money across banks.

I understand what you're saying but as individuals we also need to be prepared to be more resilient.  Web based systems go down all the time,  fortunately it's only for a few hours or days at most.   So when I go out I carry debit cards, credit cards and £100 in cash and small change.

In my travels I've come across many cash only businesses and contactless only businesses. 

For example,  in Haworth,  home of the Bronte sisters, I got caught short and the public toilets in the park were 30p contactless only payment, no option for cash.  My credit card was invaluable. 

In Llandudno the public toilets were cash only, 50p, but the doors accepted £1 coins and no cash change given, I was lucky to have small change.

I am a software developer. The chances of the entire internet crashing are extremely slim. There is a lot of redundancy built in.

Any company with common sense doesn't run a website or service on one machine (server). If one server goes down, the others pick up more of the load until another one starts up. Think of it like going to a fast food restaurant where 3 people are serving at the countrr with a little time to spare between customers. If one goes on break, the other two just serve more customers until they get back.

Companies will usually use at least 3 servers but often more, and they will often be spread so that a single failure will not take down the service. They might be placed on different power supplies so that a power cut doesn't take them all out, putting them in different cities, or I think the most extreme one I've heard is making sure they are on different tectonic plates for protection against earthquakes. Think of this like having two restaurants in case one of them is closed; or in the worst case, one burns down. The food can still be available via the other restaurnat. 

So with all those protections why did so much start going wrong? Human error most likely. News sites have reported DNS was the culprit in both cases, which is very suseptible to this kind of issue. DNS is like an address book or telephone directory. If you want to contact someone and know their name (or web address), you can look up the details of how to contact them in the address book. Someone has to keep the address book updated though. Most likely someone updated some records incorrectly, which meant the servers didn't know how to find each other.

Thankfully these kinds of issues tend to be rare, and no doubt Microsoft and Amazon will be looking into what went wrong, how to reduce the risk of it in future, and how to recover more quickly if it does happen. Aside from anything else, there will be significant financial losses to motivate them as a result. They're public companies so their stock will have tanked, companies pay for server time so they won't be getting paid for that, but it can also be written into some contracts they they have to pay their customers if they have significant outages.

The other point is probably that the internet is an extremely complicated system with many moving parts, with old things being removed and new things added every moment. It is probably inevitable that we will have these issues every so often.

When it comes to stocking up, I've realised that I've gone too far.

I have ten months stock of toilet paper, six months worth of toothpaste and toothbrushes, 16 bars of soap, 4 bottles of liquid soup, plenty of tinned and dried food,  several months worth of tea bags and coffee. All sorts of cleaning fluids.

That is sensible.

I always keep a bit of a stock of food and other supplies at home to allow for a few weeks worth of unavailability - sort of a reaction to the coronavirus lockdown and realisation that it is way too easy to end up with the supermarkets out of goods.

The food is rotated and gets used well within its use-by dates so I'm not really wasting anything.

It also helps if I fall ill and can't get out for a while - I have everything I need to last me for the duration of most things.

I also worry about the Internet going down,  will debit and credit cards be accepted.   So I always keep at least £100 in cash at home for emergencies. 

Iain said:

What happened to Microsoft?

I take it back - the BBC posted this a few minutes ago:

https://www.bbc.com/news/articles/c3rj45n4x5eo

Heathrow, NatWest and Minecraft are among some of the sites and services experiencing problems amid a global Microsoft outage.

As expected it is the NDS at fault:

It said this was due to "DNS issues" - the same root cause of the huge Amazon Web Services (AWS) outage last week.

I have engaged the services of DNS protection companies in the past who protect against DDOS (distributed denial of service) attacks but these tend to only work for individual sites, not the DNS services themselves.

Maybe the companies will evolve techniques to protect agains this but then the attackers will evolve new ways of attacking them. This has always been the way it works on the web.

I don't see there being any way to prevent it from happening from time to time so we may just have to live with the expectation that there will be outages.

What happened to Microsoft? I didn't see any headlines of major service outages for them or notice anything stopped working. 

TheCatWoman said:

shouldn't we have some sort of national back up plan.

The nature of the technology does not lend itself to backups in the conventional sense. The centralised services like DNS which failed for Amazon are a single point of failure for most systems and by their nature don't work well with distributed backups to come online if one fails. 

They already have this built in but if that ability to locate the backup fails from the same root cause then it is pointless.

TheCatWoman said:

Will dinosaurs like me who remember how to do things with pen and paper suddenly be in demand?

Alas no because nobody will be able to get a hold of you - their phones won't be working to search for you.

Besides, they will want stuff that is online and you can't provide anyway.

Should the internet go down in a long term way then society will take a massive hit. Logistics for getting food etc sent to shops will fail, payment mechanisms will fail, factories probably won't be able to operate and their supply chains will fail too.

Healthcare can work to a degree but so much is on computers or needs the NHS systems online to run the scanners etc that only basic services could run.

You can see why the internet is a target for any military action these days.

It only really effects companies who have outsources parts of the IT Infrastructure, so most of the internet is OK.

Some companies are considering bringing such IT infrastructure back in-house, but it would be very expensive for a large company.

The actual internet is a distributed system so if part of it fails, the rest of it should just carry on regardless.