The first two articles, on routine peacekeeping failures and technical docking. Many schools are built in a very robust way, and no one is running them until they have problems finding the delivery system empty.“It's done on the line.”On the contrary, it's the start of a conveyor.
Which indicators do day-to-day surveillance depend on?
The surveillance of the billing system can't be just a look.“Whether the service is running”. The service does not function properly. I suggest monitoring the following core indicators:
Certification success rate: normally more than 99%. If you suddenly fall to less than 95%, it means there is a problem. Do not look at averages alone, depending on real-time values. Some problems are intermittent and the average does not see them.
Accuracy of the billing: this is the hardest to monitor. My approach is to set one.“Experience Account”, which automatically logs in regularly, uses fixed traffic, exits, and then checks if the billing records are consistent with expectations. If not, the calculation logic is problematic.
Database performance: The system’s database is often read and written, which can easily become a bottleneck.
Radius response time: The billing system interacts with the authentication device, and the response time is monitored. Normally at millisecond level, and if more than one second of it, user experience will decline significantly.
I've put a surveillance on a school.“Service run”and“CPU usage”, which results in a slow-to-research stack of databases and user certifications that are slow but without alarm. Then the problem is immediately detected with Radius response time and database slow-to-seek monitoring.
Common fault check thinking
The campus network billing system is mostly centralized in a few areas.
Fault one: Large area authentication fails. Checks the Radius connection, then checks the database state, and then check if the billing policy is configured wrongly. The order of checking is important, moving up from the most basic level of network, without suspecting software bugs at once.
Fault two: Authentication but not bad billing. Checks the Radius billing attribute configuration, then checks the billing strategy, and then checks database records.
The problem is usually at the user management level, not at the system level.
Fault four: The system slows down. First check the database performance, then check whether log files are full of disks and then check for abnormal flow shocks. The system slows 90% by database or disk.
I've summarized a line of check principles: configuration, then performance, and finally code. Configuration issues are 70%, performance issues 25%, real codes only 5%.
Log analysis is a key capability.
The logs are very large, but many people don't analyse them. Logs are not used for this purpose.“Archive”It's for use.“Checking questions”Yeah.
Certification log: records each authentication success/failure, time, account number, IP, cause of failure. Analyses the certification log to detect authentication patterns, equipment malfunction trends, and unusual attack attempts.
Logs for billing: log every time the bill starts/stops, traffic uses, and costing. Analyses the bill log to detect charges abnormalities, traffic anomalies, and policy errors.
Operating log: all records administrator operations. This is both a security audit requirement and a failure regression key. I suggest that the log be detailed.“Who, when, who, what object, what changes?”。
Log analysis cannot be manually retouched. Use tools: Linux below grep/awk/sort/uniq, Windows under PowerShell string. More advanced, import logs into Elasticsearch and use Kibana for visualization analysis.
I did a log analysis for a school, and found that every morning at 2-4 a.m. there were a lot of failed certifications, all from the same IP section. It was discovered that a student wrote an automatic script to crack the password. Then the account locking strategy was added and the problem disappeared.
Account management pits, avoid ahead.
The campus network has a large number of users and is subject to frequent changes, and account management is the main focus of the operation.
New students are enrolled in schools: a batch of entries is to be recorded before the start of school.
Graduates leave school: account numbers are cancelled and resources cleaned in a timely manner.
This work is done by establishing a linkage mechanism with the personnel services, which cannot wait for the information services to find out.
Password management: The initial password is not too simple to force the user to log in for the first time. It also requires a password complexity requirement and a periodic modification strategy. These appear minor, but they are directly related to network security.
Performance optimized as the number of users grows
Not the system.“I've been on the line.”This is sustained and optimized as the number of users increases and business complexity increases.
Database optimization: regular cleansing of historical data (which can be archived in cold storage), establishment of appropriate indexes, optimal slow searching and, where necessary, splitting of databases (e.g. separate authentication and billing libraries).
Aggression of the layer: resizes the thread pool, connects the pool, and cache policies. These parameters are adjusted to the dynamics of actual operations and cannot be configured for several years.
Network layer optimization: ensure that the network path between the billing system and authentication equipment is low-delayed and high-reliability.
I saw a school that had its users growing from 3,000 to 8,000, and the system did not optimize performance, which resulted in the system Caden during the final exam, where students were unable to check their grades online. The database was split and the application layer optimized, with three times the performance.
Experience with manufacturers
The system is problematic and often requires technical support from the manufacturer. How to deal with the manufacturer efficiently is part of the transportation capability.
First, the question is to be precise. Don't just say it.“The system is slow.”Yes.“The certification response time increased from 200ms to 2000ms, at 10-11 p.m. daily, affecting about 2,000 users”. The more specific the information, the easier it is for the manufacturer to locate the problem.
Second, the logs are available quickly. The processor is checking the logs. You need to be able to quickly retrieve the authentication logs, billing logs, system logs for a given time period. If it takes half an hour to find the logs, the problem is very inefficient.
Third, the version should be upgraded with caution. Do not raise as soon as you have a new version, see if the change logs solve your concerns and whether new questions are introduced. I suggest that the upgrade should be validated in the test environment before production is confirmed.
Summary
The campus network billing system is a process that requires a combination of technology, processes, and communication.
The best mode of operation is that problems are detected before they occur, and the failure recovery time is at the minute level, with users almost unaware of the existence of system maintenance.
The next article will address the differences in application of the campus network billing system across different settings, including higher education institutions, vocational colleges, and the needs and priorities of primary and secondary schools.