Seven years since I've been running campus network billing systems, I have witnessed a variety of failures and valuable experience. I want to share some impressive examples today, hopefully helping us to keep you from going the wrong way.
The first case is about database performance. In 2018, we were responsible for operating a billing system at a 10,000-person college. One day, the school phone called and said that the certification system was completely disabled and that all students could not access the Internet. We had an emergency remote login server to check out and found that the CPU usage of the database had been 100% and there was a huge backlog of queries.
The problem is that the database indexing was not rational. The original system was designed without consideration of the performance problems associated with the increase in the number of users and no indexes were created in key fields, resulting in extremely inefficient queries. After we found the problem, the database was optimized overnight, necessary indices added and historical data cleaned up. The system was restored to normal after optimization, with authentication time reduced from 15 seconds to less than 3 seconds.
This thing is a wake-up call.
Since then, we have established a system for maintaining databases on a regular basis, which examines database performance every month, cleans up invalid data and updates statistical information. At the same time, we have developed a database monitoring tool to monitor database operations in real time and automatically alert if anomalies are detected.
The second case concerns security. In 2021, we discovered that the system of billing in a school was often attacked, and the attackers tried to break the student account codes and then embezzled the network resources. We thought it was a weak password problem, requiring students to set complex codes, but with little effect.
We introduced a multifactor certification mechanism.
Students need to enter a mobile phone identification code in addition to entering a password when certifying. At the same time, we have deployed an intrusion detection system that automatically identifies unusual login behaviour, such as simultaneous login at different locations with the same account number and multiple login failures within a short period of time.
The third case concerns backups. In the summer of 2019, a school’s data centre was overheated by air conditioning failures, server auto-disable protection, and billing systems were shut down for a whole day. School teaching and examinations were affected, and the principal asked about it himself.
We helped the school to build a backup system for disaster relief.
Core equipment is both hot and co-ordinated to offsite backup centres in real time. At the same time, we have developed detailed contingency plans, including a failover process, data recovery process, notification process, etc.
I have learned from the years of operation that campus network billing is not just a software system, but also a service that requires continuous commitment and careful maintenance. Only by building up a well-developed transportation system with advanced technological tools can it be ensured that the system functions reliably and efficiently, providing quality online services to teachers and students.
In terms of surveillance and early warning
We have a complete system of surveillance, including systematic monitoring, network traffic control, security incident surveillance, operational functionality monitoring. All surveillance data are displayed on the screen in real time and automatically alerts the carrier if an anomaly occurs, so that it can be processed in a timely manner.
In terms of automated transport, we have developed a large number of automated scripts and tools to automate routine transportation. For example, system configuration backups, log file clean-up, performance data collection, security patch installation, etc. can be done through automated tools. Not only does it increase efficiency but also reduces the occurrence of human error.
Knowledge management
We have a well-developed knowledge base system that includes all of the dimensions of experience, failures, solutions, etc. into the knowledge base to facilitate access and learning. We also organize technology sharing sessions on a regular basis to allow operators to share experiences and jointly improve their skills. Through knowledge management, we avoid duplication of effort and improve the speed and quality of problem resolution.
In terms of teamwork, we have adopted the DevOps concept to break down barriers between development and transport. Developers are not only responsible for code preparation but also for system deployment and transportation; they are not only responsible for system maintenance, but also for demand analysis and architecture design.
Continuous improvement
We have established the PDCA (Plan-Implementation-Examination-Action) revolving mechanism, which continuously captures lessons learned and optimizes the process. Every quarter we review the operation, identify problems and shortcomings, develop improvements and follow up on them. Through continuous improvement, our levels of support are constantly increasing, and systems availability is rising year by year.
In the area of user services, we insist on a user-centred approach to improving the quality of services. We have established a user service line that answers user queries at any time; we have an online service platform which facilitates user submission of failure declarations and requests for advice; and we have organized regular user satisfaction surveys to receive user comments and suggestions and continue to improve our work.
In general, the campus network billing system is a complex and important task that requires solid technical expertise, experience and high level of responsibility on the part of operators.