The most feared thing about the transporting of a carrier was not malfunctions, but the discovery of a failure that left the preset in its form was simply useless. Early in the year, our WiFi certification system, which had been approved by one of our clients, failed to provide extensive authentication, affecting thousands of users. When it was done, it was discovered that several key steps in the pre-project were not met and that everyone on the site was busy working on them.
The de-classification program for certification service machines cannot be"Toggler"
The first of many plans says:"Switch automatically to backup node when authentication server malfunctions". The word is fine on paper, but there are a few pits in practice.
The first pit is the main preset switch not being fully automated. Some WiFi certification systems call it"High Available"It's actually cold.——The service is not running, although the gear is on. When you find the main node, you start with the preset, start the service, check configuration consistency, confirm data synchronization, and it takes five to ten minutes for skilled people to know about the process. In this time, all users have failed their authentication requests.
The second pit is a data sync delay. If the main setup is made with an astrologic copy, the primary node twitching moment, the last few seconds and even dozens of seconds of authentication logs and billing data may not be synchronized to the backup. If you cut the gear up at this point, you lose it. Worse still, the main node automatically returns after it slows down, and then two nodes do not match the data."Brain crack."Status, where the amount of work to repair data later is much greater than the conversion itself.
So we wrote a much more detailed piece in the preset: master-to-change is not just cut nodes, but also synchronized data consistency. If the primary node is due to the database being hung up rather than the process being hung up, then the node is cut up and it is broken off to prevent it from restoring brain fractures. These operational details are not clearly written in the preset, and nobody would have thought about them at the time of the failure.
Single point distress check in authentication links
WiFi certification system is a chain: terminal → AP → AC → Radius → Database/Cache → Certification portal → SMS Channel. If there's a single malfunction on this chain, the entire authentication is broken.
We found a very hidden single point when we checked the malfunction: although Radius service was prepped, two Radius servers were connected to the same database. The database was over-timed because of the high IO on disk and two Radius were dead at the same time. So what's called high utility is half redundant.——The application layer is redundant and the data layer remains single point.
Later, at the structural review stage of WiFi certification systems, we drew a complete authentication chain chart showing each node on what it relies on for downstream resources and examined each case: is there any redundancy in this downstream resource? Is the switch between redundancies automatic or manual? Will data be lost during the conversion process? After drawing this map, many of the hidden single points are exposed.
Downgrading strategy during failure
If the first part of the plan is..."How?"The second part of the story should be"What if you can't recover?"This logic is not something many people would like to think about, because it means recognizing that the system may not recover in certain circumstances. But the truth is, some of these malfunctions are not fixed in minutes.——Like text-message access is completely disabled, like an export fiber-optic cut by a construction project.——At that time, you need an interim downgrading programme to minimize the operational impact.
We have several downgrading strategies in the WiFi authentication system. When SMS channels are not available, they switch to"Unaccredited mode"——But strictly control access to this mode for a duration of time (e.g. one hour per time) and speed-limit bandwidth (e.g. two trillions), while recording the MAC addresses and IPs of all users accessing the model, recouping authentication information afterwards. When Radius servers are completely unavailable, AC side automatically switches to local MAC certification as a back-up.——Although security has been reduced, it has at least ensured basic connectivity.
The conditions for the execution of these downgrading strategies, trigger logic, and restore processes are written in the presets, and they are rehearsed at least once every quarter. Instead of saying and writing, it is necessary to re-examine whether the presets apply after each system upgrade and structural adjustment."Who's making the decision?"and"Who can ping?"The most important thing about malfunctioning is not technical uncertainty, but the fact that all are waiting for a technical officer to be ready and the technical manager cannot contact him. We made clear in our plan that there was a chain of authority: if the first level chief fails to respond within five minutes, delegation of authority automatically to the second level; no secondary level is connected directly by the duty engineer, who reports back after reporting. This authorization mechanism is more important than any technology programme because it ensures that someone can do it at a critical time, rather than everyone is no longer talking to anyone."Error Notification Template"The communication costs are often higher than the cost of technical repair, and an improper external notification may cause a greater crisis of trust than the failure itself.