Anyone stuck on the ground?

Posts
1,586
Likes
3,153
Obviously, there's a bunch of tongue and cheek here. I think some of the actual criticism is a bit overstated. The FAA certainly has aging infrastructure and legitimate modernization issues, but older technology doesn't automatically mean unreliable technology. Copper-based communications, paper flight strips, and even older computer systems can remain perfectly capable of performing their intended functions. The bigger concern is often long-term maintainability, particularly when replacement components and technical support become increasingly difficult to obtain.

And it's not as though the FAA hasn't been modernizing. ERAM, ADS-B, STARS, Data Comm, and electronic flight strips are examples of improvements already implemented, although not necessarily at every facility. You can certainly question the pace and management of modernization, but replacing interconnected, safety-critical systems across an operation the size of the U.S. ATC network requires staged implementation and extensive validation.

Ironically, the preliminary explanation for this particular outage involves telecommunications maintenance during a copper-to-fiber transition, with abnormal router traffic disrupting the network. Newer technology isn't inherently immune to failures either.

I'd also consider the FAA's actual operational record. We're talking about a system handling roughly 45,000 flights daily across an enormous and complex airspace. How frequently do these infrastructure failures occur, what consequences have they produced, and what have investigations established as their causes?

That doesn't excuse aging equipment or delays in modernization. But there is a considerable difference between a system having maintenance and sustainability problems and a system being demonstrably unsafe or incapable of performing its job. Its performance over time deserves consideration alongside the age of its technology.

And for what it's worth, I work for the government, although not the FAA. I've operated equipment and been involved in specifying, maintaining, troubleshooting, and developing repair instructions for it. Now, working on the engineering and design side, I've seen both sides of the problem.

I know firsthand how frustratingly slow the government can be when it comes to repairing, replacing, or modernizing equipment. Sometimes you question why an outdated system is still being used, and frankly, sometimes there isn't a particularly good answer.

But I've also learned that replacing equipment, especially in a complex, interconnected system, isn't as simple as finding something newer that looks like it'll do the job and plugging it in. There's a tremendous amount of engineering, integration, testing, and validation involved, often while the existing system has to remain operational. That doesn't excuse unnecessary delays, but it does put some of them into perspective.
 
Posts
5,427
Likes
55,739
Obviously, there's a bunch of tongue and cheek here. I think some of the actual criticism is a bit overstated. The FAA certainly has aging infrastructure and legitimate modernization issues, but older technology doesn't automatically mean unreliable technology. Copper-based communications, paper flight strips, and even older computer systems can remain perfectly capable of performing their intended functions. The bigger concern is often long-term maintainability, particularly when replacement components and technical support become increasingly difficult to obtain.

And it's not as though the FAA hasn't been modernizing. ERAM, ADS-B, STARS, Data Comm, and electronic flight strips are examples of improvements already implemented, although not necessarily at every facility. You can certainly question the pace and management of modernization, but replacing interconnected, safety-critical systems across an operation the size of the U.S. ATC network requires staged implementation and extensive validation.

Ironically, the preliminary explanation for this particular outage involves telecommunications maintenance during a copper-to-fiber transition, with abnormal router traffic disrupting the network. Newer technology isn't inherently immune to failures either.

I'd also consider the FAA's actual operational record. We're talking about a system handling roughly 45,000 flights daily across an enormous and complex airspace. How frequently do these infrastructure failures occur, what consequences have they produced, and what have investigations established as their causes?

That doesn't excuse aging equipment or delays in modernization. But there is a considerable difference between a system having maintenance and sustainability problems and a system being demonstrably unsafe or incapable of performing its job. Its performance over time deserves consideration alongside the age of its technology.

And for what it's worth, I work for the government, although not the FAA. I've operated equipment and been involved in specifying, maintaining, troubleshooting, and developing repair instructions for it. Now, working on the engineering and design side, I've seen both sides of the problem.

I know firsthand how frustratingly slow the government can be when it comes to repairing, replacing, or modernizing equipment. Sometimes you question why an outdated system is still being used, and frankly, sometimes there isn't a particularly good answer.

But I've also learned that replacing equipment, especially in a complex, interconnected system, isn't as simple as finding something newer that looks like it'll do the job and plugging it in. There's a tremendous amount of engineering, integration, testing, and validation involved, often while the existing system has to remain operational. That doesn't excuse unnecessary delays, but it does put some of them into perspective.
Absolutely agree with you, to run such a vast interconnected network with these sort of daily traffic volumes and to do it seemingly almost flawlessly for such a length of time with it's aged equipment is bordering on a miracle.
 
Posts
180
Likes
159
As a fibre optic engineer of some years i can tell you that this must have been one incredible balls up. Whilst we never get involved in massive infrastructure changes like this project , we do a lot of work for Banks, Government and Financial institutions in the UK.

A Chopover like this would never be allowed without full resilience testing, It would mean a fully running system mirroring the original system and load tested for months with many hundreds of hours of IT labour time making 100% sure its working exactly correctly and as anticipated.

We undertook a large NHS hospital IT infrastructure change and the commissioning and resilience checking was ongoing for 18 Months after it had all be installed

Only then after after all this due diligence would the new system take over and the old system now mirroring the new system, so that in the event like happened it could quickly and seamlessly be handed over in full and then and only then the old system decommissioned.

The contractor who was in charge of the project will face some severe financial consequences i guess
 
Posts
180
Likes
159
Sec Transportation blames the grounding on a contractor in a small, remote location for cutting a copper wire. (True.)

The conversation:

"Hey, George, what wire did you cut?"

"The yellow one, why?"

"Just got a text, says all planes stopped flying."

"So you think I did it?!! NO WAY!"

"Try splicing them back together."

"Okay...Done. Anything?"

"Planes are flying again."

"I don't believe it. I'm disconnecting the wire..."

"Planes are grounded."

"Holy crap! Here, I put the wire together."

"Planes are flying."

"Anyone know we're here? Let's get out of here."
In the UK we say a "Big Boy broke it and ran away." . I work in the telecommunications industry and i would be amazed even on a old system if there was not fall over systems that jumped in when it all went south.
 
Posts
6,562
Likes
22,415
In the UK we say a "Big Boy broke it and ran away." . I work in the telecommunications industry and i would be amazed even on a old system if there was not fall over systems that jumped in when it all went south.
Exactly. The former sec of transportation made a similar statement, that there were triple redundancies. When one failed, another took over.

This is incompetence cloaked in spinelessness. They know it's a fragile and ancient system, and have said so themselves, so they should have anticipated the impact. If they didn't take precautions, they either didn't care or didn't understand, both of which are bad. And to try to put the blame on a contractor when it's their responsibilty to manage the system, that's saying we, the public, are too stupid to understand.

The system is a disgrace. The current administration inherited a problem that should have been addressed by Congress raising money decades ago. No healthy society would be satisfied with a transportation system like ours, or health care sysrem like ours. And now we have impetuous children who are enamoured with new AI toys in charge. I'm not hopeful that when the adults take over that they will do any better. Maybe they think we'll all have flying Teslas. What a nightmare.
 
Posts
180
Likes
159
Exactly. The former sec of transportation made a similar statement, that there were triple redundancies. When one failed, another took over.

This is incompetence cloaked in spinelessness. They know it's a fragile and ancient system, and have said so themselves, so they should have anticipated the impact. If they didn't take precautions, they either didn't care or didn't understand, both of which are bad. And to try to put the blame on a contractor when it's their responsibilty to manage the system, that's saying we, the public, are too stupid to understand.

The system is a disgrace. The current administration inherited a problem that should have been addressed by Congress raising money decades ago. No healthy society would be satisfied with a transportation system like ours, or health care sysrem like ours. And now we have impetuous children who are enamoured with new AI toys in charge. I'm not hopeful that when the adults take over that they will do any better. Maybe they think we'll all have flying Teslas. What a nightmare.
This it what is says on the BBC site in Uk.

"
The transport secretary explained that a router that was being worked on had started flooding other air traffic control systems and created communication issues with planes, prompting the shutdown on safety grounds.

The government was now trying to understand whether the root cause was the wire replacement or a fault with the router."


Either they have a really bad IT team who cannot resolve a loopback on a network without actually having to physically find it. or The poor contractors are being used as a scapegoat. No one piece of equipment on a network should be so important that the network fails when it fails without double or even triple redundancy.

If it really was something as simple as a router caused such a massive network issue that is very worrying
 
Posts
180
Likes
159
This is a very good explanation of what coukd possibly have happened. This i have copied and pasted from a trade related forum im a member of.

Exactly. The former sec of transportation made a similar statement, that there were triple redundancies. When one failed, another took over.

This is incompetence cloaked in spinelessness. They know it's a fragile and ancient system, and have said so themselves, so they should have anticipated the impact. If they didn't take precautions, they either didn't care or didn't understand, both of which are bad. And to try to put the blame on a contractor when it's their responsibilty to manage the system, that's saying we, the public, are too stupid to understand.

The system is a disgrace. The current administration inherited a problem that should have been addressed by Congress raising money decades ago. No healthy society would be satisfied with a transportation system like ours, or health care sysrem like ours. And now we have impetuous children who are enamoured with new AI toys in charge. I'm not hopeful that when the adults take over that they will do any better. Maybe they think we'll all have flying Teslas. What a nightmare.

Major airports aren't running on raw copper: Every major airport and regional Air Route Traffic Control Center (ARTCC) has had heavy, redundant dark and lit fiber rings fed into their demarcation points for decades. What the FAA actually has left on copper (about 30% of their circuits) are legacy TDM (Time-Division Multiplexing) T1/DS1 and DS3 circuits feeding remote radar towers, outer markers, and VHF ground-to-air radio sites out in the sticks.
"Analog" is the wrong word: Politicians keep saying "analog to digital," when the contractor (L3Harris) is actually migrating TDM to IP (circuit-switched to packet-switched) under the FAA's Enterprise Network Services (FENS) contract. T1s over copper are still digital signals—just ancient, fixed-bandwidth ones.
Splicing fiber doesn't take down a national grid: According to the FAA's own preliminary data, L3Harris was cutting over a remote site from copper to fiber on Friday afternoon. During that cutover, a router started blasting over 1,000 connection attempts every 4 to 5 seconds across the network. That isn't a "bad copper wire" problem—that's a broadcast storm, a BGP/routing table loop, or a misconfigured interface with zero rate-limiting or storm control.
The real culprit is 20-year-old core hardware and zero isolation: Sources inside the investigation noted the cascade hit legacy Cisco 600/2600-series era routers (manufactured between 2001 and 2006) with tiny memory buffers. When the remote cutover went sideways, those legacy routers ran out of memory and started spitting out corrupted/unsynchronized radar and voice packets. When one regional hub choked, automated failover pushed the flood right onto the neighboring regional hubs, knocking them over like dominoes.
Anyone who has built physical plant or managed a NOC knows that cutting over a single remote spur—especially during peak Friday afternoon traffic instead of a 2:00 AM maintenance window—should never be able to flood the core backbone and ground 3,500+ flights coast-to-coast.
Blaming "old 1980s copper" is a great pitch when asking Congress for billions more in infrastructure funding, because "our contractor botched a live cutover with no sandbox isolation and triggered a nationwide packet storm on 22-year-old routers" doesn't sound nearly as sympathetic to the public.
 
Posts
11,110
Likes
55,783
I think your "neglected" is a bit of an understatement.

"It was reported last year that several FAA sites were still running Windows 95. The FAA also uses decades-old Compaq computers that are so out of date that replacement parts have to be bought from eBay. Another legacy technology still in use is paper flight strips, the pieces of cardstock on which key details of a single flight - such as call sign, aircraft type, and altitude - are printed or handwritten.


"We're moving away from outdated, manual processes and replacing technology that once relied on floppy disks with a modern system that allows controllers to instantly share critical flight alerts, restricted airspace schedules and pilot reports," FAA Administrator Bryan Bedford said in a statement."

Obviously, there's a bunch of tongue and cheek here. I think some of the actual criticism is a bit overstated. The FAA certainly has aging infrastructure and legitimate modernization issues, but older technology doesn't automatically mean unreliable technology. Copper-based communications, paper flight strips, and even older computer systems can remain perfectly capable of performing their intended functions. The bigger concern is often long-term maintainability, particularly when replacement components and technical support become increasingly difficult to obtain.

And it's not as though the FAA hasn't been modernizing. ERAM, ADS-B, STARS, Data Comm, and electronic flight strips are examples of improvements already implemented, although not necessarily at every facility. You can certainly question the pace and management of modernization, but replacing interconnected, safety-critical systems across an operation the size of the U.S. ATC network requires staged implementation and extensive validation.

Ironically, the preliminary explanation for this particular outage involves telecommunications maintenance during a copper-to-fiber transition, with abnormal router traffic disrupting the network. Newer technology isn't inherently immune to failures either.

I'd also consider the FAA's actual operational record. We're talking about a system handling roughly 45,000 flights daily across an enormous and complex airspace. How frequently do these infrastructure failures occur, what consequences have they produced, and what have investigations established as their causes?

That doesn't excuse aging equipment or delays in modernization. But there is a considerable difference between a system having maintenance and sustainability problems and a system being demonstrably unsafe or incapable of performing its job. Its performance over time deserves consideration alongside the age of its technology.

And for what it's worth, I work for the government, although not the FAA. I've operated equipment and been involved in specifying, maintaining, troubleshooting, and developing repair instructions for it. Now, working on the engineering and design side, I've seen both sides of the problem.

I know firsthand how frustratingly slow the government can be when it comes to repairing, replacing, or modernizing equipment. Sometimes you question why an outdated system is still being used, and frankly, sometimes there isn't a particularly good answer.

But I've also learned that replacing equipment, especially in a complex, interconnected system, isn't as simple as finding something newer that looks like it'll do the job and plugging it in. There's a tremendous amount of engineering, integration, testing, and validation involved, often while the existing system has to remain operational. That doesn't excuse unnecessary delays, but it does put some of them into perspective.
Ya, it helps to look at the whole picture. I tend only to see the negative in our flight systems, highway infrastructure and electrical grid.
 
Posts
24,844
Likes
55,271
Blaming "old 1980s copper" is a great pitch when asking Congress for billions more in infrastructure funding, because "our contractor botched a live cutover with no sandbox isolation and triggered a nationwide packet storm on 22-year-old routers" doesn't sound nearly as sympathetic to the public.
From the very first press conference after the ground stop, it was clear to me that the Kung Fu Plumber and Sean Dummy were clearly trying to absolve themselves of blame and to spin it as a way to put pressure on congress to appropriate funding. The logic doesn't hold up if you think too much about it.

Some cynics may suggest that triggering the shutdown deliberately might have the same effect, along with providing a useful distraction from the election results.
 
Posts
11,110
Likes
55,783
Oh boy are conspiracy theories starting with this event to? First I heard of it.
 
Posts
5,540
Likes
9,479
Oh boy are conspiracy theories starting with this event to? First I heard of it.
Yes. Supposedly the 2025 plan (Heritage Foundation?) includes privatization of the flight control systems. I admittedly have not looked it up tp confirm it is true and not just a baseless internet claim, although it does fit in well with the general gameplan of the current regime and their billionaire backers.
 
Posts
6,562
Likes
22,415
To believe in this conspiracy, a person would need to believe they are competent enough to pull it off.
 
Posts
171
Likes
449
To believe in this conspiracy, a person would need to believe they are competent enough to pull it off.
Well, they are good at the first required part--breaking stuff.

The idea is to screw up stuff, convince people government is incompetent and can do nothing and needs to privatize everything, and then sell it off to their pals.

Anyway, Walrus.... that's the stated intention and probably the root of the conspiracy theories.