In October of a year, the information section of a university suddenly received a large number of complaints from students: the dormitory WiFi was connected, but the browser jumped to any authentication page and then entered the username password."Authentication successful"The information section then performed a complete check and retrace of the malfunction, some of which are worth recording.
A precise description of the malfunction is important.
The first lesson to be learned from this is:"WiFi is not working."When complaints are made, there is no rush to reopen the equipment and to clarify the situation."WiFi can't get on."and"WiFi's connected, but not online."Two completely different things, and the latter can be broken down into two."DNS Parsing Failed"、"No response to HTTP request"、"Redirected to authentication page"Three completely different situations, and the corresponding ranking direction is quite different.
The malfunction was accurately described as normal for wireless connections (the cell phone shows WiFi is connected, IP addresses are correct), authentication pages open (indicates DNS resolution and basic HTTP is available), authentication operations responsive (indicate the authentication server online), but after authentication, it remains reoriented (states that the session status is not correctly written)."Network Layer"It's just a straight shrink."Session management for authentication systems"。
Logical progression of the screening process
Based on the description of the above, the Information Section engineers began to prioritize each possible cause.
First check the status of the authentication server. Log on NATSHEL_AUTH_GATEWAY management page, see that the authentication service is running, RADIFUS port is listening, CPU and memory usage is normal."The certification service is dead."Possibility.
Then check the logs of the authentication server. In the certification log, a large number of requests for accreditation can be found and the result fields are shown."Access-Accept"——It means the RADIUS server is telling Natshell that"This user-certification has passed.". The system is not implemented after the students have lost their password on Portal page"Release the user's traffic"This move.
This phenomenon suggests that RADIUS certification itself was successful, but after it was successful."Let's do it."The certification success and flow release are two separate steps, the former being handled by RADIUS and the latter by NATSHEL_AUTH_GATEWAY (usually sent to AC or firewalls under a strategy) in the framework of the WiFi network billing system on campus.
Check further the NETSHELL_AUTH_GATEWAY session sheets. The number of records in the session sheets has increased considerably in a short time, approaching the system’s maximum limit. When the session forms are full, new requests for certification success cannot be recorded in the new session records, leading to authentication passing but user traffic is still intercepted.
Root: The table ceiling is set too low
The ultimate root cause of the situation is that NATSHEL_AUTH_GATEWAY has been set a ceiling of 20,000 for its session, while students in various dormitory buildings were online on the same day because of an event (school-based regeneratives) and the total number of sessions exceeded this limit.
The figure of 20000 is the default value for equipment out of the plant, and no one specifically checked the size of this school when it was on the line. There are 23,000 students in this school, and if the number of live-in areas reaches 80% or more, the number of live-room meetings will be close to 20,000, which, together with online users from the teaching area and other regions, would be too late.
The more dangerous is that the upper limit of this session sheet is shown in the management interface, but no warning threshold is configured——The information section does not know the risk when the number of conversations is nearing the upper limit.
Repairs and improvements
The temporary repairs on the same day were to raise the maximum number of conversations to 50,000 and manually clean up a long, inactive body session.
The after-action improvements are divided into three directions. First, the parameter review: a systematic analysis of all key capacity parameters in the WiFi network billing system on campus site was performed, including meeting table ceilings, RADIUS number of simultaneous connections, billing database connection pool size, log writing queue length etc., checking whether the set values match the actual size of the school. Several parameters were found to be default values and not consistent with the actual school situation.
Second is monitoring blindness: several key indicators have been added to the system, including alarm rules for meeting forms with over 70% use, RADIUS requests per second that exceed the threshold and certification failure rate that exceeds 5%."None of these are in the default monitoring template."- I'm gonna make it up.
Third is the pressure remix: a simulation of the start-up season was performed during summer break, pushing the number of QEs to 3,000/minutes, observing where the system's bottlenecks were. This test also revealed another risk.——The high-high chain waiting problem of the RADIUS database is subsequently resolved by optimizing indexing and introducing query caches.
A few lessons to remember.
Several lessons were repeated in the process of this malfunctioning project.
The default parameter for the device or system is a manufacturer's."General scene"It's not for your specific environment. Before you go online, you have to perform a dedicated capacity-planning check that will allow you to cross all parameters associated with the simultaneous distribution and storage caps and adjust them against actual size and peak expectations of the school.
Second, the alarm blind area is a hotbed of failure. No warning system. If there's a problem, you can only find it by using user complaints."Students are harassed to reflect.". The warning thresholds for key indicators are well-suited, a service infrastructure and not an option.
Three is deep on the fail-record. The immediate cause of this failure is that the session was full, but even deeper because the project was not systematically checked for capacity parameters when it was online."The larger parameters are solved."Similar problems will be repeated in other parameters.