You are here: Home > Notices and News > Archive > Internet Performance Problems

Internet Performance Problems

Update: 19 September 2000

Router Misconfiguration. Over the weekend, a misconfiguration of an upstream router in Net.Work.Virginia caused severe packet loss between Virginia Tech and other institutions connected to Net.Work.Virginia. The underlying problem was finally diagnosed and corrected around 10:00am, Monday, 18 September 2000. Additional refinements to the routing of outbound traffic from Virginia Tech were completed by CNS network engineers around 2:00pm, Tuesday, 19 September 2000. These changes should further reduce latency and packet loss between Virginia Tech and other institutions connected to Net.Work.Virginia.

Sprint Backbone Bandwidth Updgrade. The first of three expected backbone bandwidth upgrades to the Internet service (Net.Work.Virginia) used by Virginia Tech and many other universities and colleges in the Commonwealth has been completed. This upgrade brings relief to the congestion related problems experienced over the last several weeks. Technical and administrative problems delayed this first upgrade, which was finally brought online on Friday, 15 September 2000. Additional upgrades are expected to come online around 1 December 2000.


Event Timeline

14 September 2000: Access Server Bug. CNS is aware of a system bug that allows a memory leak on the access servers resulting in occasional interruptions of modem pool sessions. Some sessions may have been affected over the last week or so. The access servers are being monitored for memory utilization and are reloaded when necessary to work around this problem. Normally, problem access servers are identified before they become unusable, are configured to not accept further calls, and reloaded after all existing sessions have expired. In this way, high service levels can be maintained without interrupting calls. We will continue to do all we can to minimize these events, and we hope to have the system bug resolved as quickly as possible.

14 September 2000: Net.Work.Virginia SprintLink Access. The backbone upgrade to higher speed access for Net.Work.Virginia is in progress and expected to be completed by early next week (18-22 September 2000). Please refer to the 8 September 2000 update. Until this is completed, there will continue to be severe congestion between Net.Work.Virginia and SprintLink affecting Internet performance. The circuit to Abilene for Internet2 sites is working satisfactorily.

14 September 2000: Static IP Host Naming. Previously, it was announced that all static asynchronous hosts would be re-named to PID.async.vt.edu early in August 2000 when the IP addresses for these hosts changed. This announcement was distributed via email to vt_modempool_announce@listserv.vt.edu subscribers on 7 July 2000, Subject: Modem Pool Static Address Renumbering. This change took effect Thursday, 14 September 2000. Users with static asynchronous hosts may want to note this change, as any static hostname that is not PID.async.vt.edu will be converted to the new naming convention. CNS estimates that approximately 60 static asynchronous hosts (out of over 9000 modem pool subscribers) will be affected.

8 September 2000. Sprint provided a progress report on their pending backbone improvements. Much progress has been made in bringing the additional bandwidth online. Some work remains to be done in order to move production Internet traffic to the new higher bandwidth links. Sprint expects to complete that work and be ready to put the circuits into production by close of business Monday (10 September 2000). Internet traffic to/from Virginia Tech will be among the first to be routed over the new circuits as soon as they are ready

5 September 2000. CNS has been working with Virginia Tech's major Internet service provider (ISP) to resolve the campus network problems. The ISP failed to meet scheduled mid-August delivery dates for critical upgrades to the service provided to Virginia Tech and many other educational institutions in Virginia. As a result, the facilities used to provide Internet service to many Virginia schools cannot handle current traffic loads, which are especially intense with the fall semester underway at the Commonwealth's university and college campuses.

7:00pm, Monday 4 September 2000. The Sprint-managed router that provides Internet access to Virginia Tech (and virtually every other educational institution in Virginia) had finally been stabilized. Technicians from Sprint as well as network engineers from Virginia Tech worked throughout the holiday weekend diagnosing and correcting problems to bring the router back under control.

Bandwidth is still a problem, but the router is now stable. Bandwidth upgrades are still on target for 8 September 2000.

3:00am, 1 September 2000. Sprint and Cisco technicians install scheduled hardware upgrades which are intended to improve the performance and stability of the router after nearly a two weeks of instability. Not all of the desired upgrades can be completed because the required hardware did not ship in time for the maintenance window. Nonetheless, the upgrades that were installed made a small improvement in the stability of the router.

5:00am, 3 September 2000. Sprint and Cisco technicians install the remaining upgrades using hardware that arrived late Friday afternoon. Initially, stability is substantially better.

10:00am, 3 September 2000. Virginia Tech engineers identify a software bug which is causing 50% packet loss on all flows switched through the router. A software feature is disabled, which avoids the packet loss bug at the expensive of lower packet switching performance and greater likelihood of instability. The bug is a known issue, which is corrected in a later version of the software.

11:00am, 3 September 2000. One of the new hardware modules begins to fail, causing the router's bus to be repeatedly reset. Software is suspected. A later software revision is loaded, but fails to improve the situation. Sprint technicians open a case with Cisco to get replacement hardware.

4:00pm, 3 September, 2000. Cisco determines that a replacement module is not available anywhere in the region. The replacement must be shipped from San Jose via an overnight carrier, with an expected arrival of Noon, 4 September 2000.

3:00am, 4 September 2000. Sprint technicians complete the task of migrating all customer feeds from the faulty module to another module in the same router. This is a labor intensive process, which began around 6:00pm on 3 September 2000. This configuration re-establishes connectivity to all customers, but doubles the traffic load of an already-congested interface module.

3:00am, 4 September 2000. Inexplicably, the remaining interface module (now carrying all customer feeds) stops forwarding traffic. No progress is made in diagnosing the problem.

6:00am, 4 September 2000. Engineers from Virginia Tech investigate the new problem and discover a workaround (without finding the problem source). The workaround is only partially effective, and the interface periodically stops forwarding traffic.

11:00am, 4 September 2000. Engineers from Virginia Tech discover the source of the new problem. Appropriate corrective measures are applied. A software bug report is pending with Cisco for the new problem. The interface module is stabilized, but is seriously congested due to the fact that it is carrying the traffic load for the faulty module in addition to its usual load.

5.00pm, 4 September, 2000. Sprint and Cisco technicians replace the faulty module. The delivery of the replacement parts was delayed due to some confusion on the part of the courier. Virginia Tech's Internet service is immediately moved to the new interface module. The new module is stable and appears to be fault-free.

7.00pm, 4 September, 2000. Sprint technicians complete the task of moving customer feeds back to the appropriate interface module. The router is operating at nominal performance levels. Close monitoring of all aspects of the router's operation continue.

8 September 2000. The ISP is scheduled to complete a facilities upgrade that will make substantially more bandwidth available to Virginia Tech's Internet service. This upgrade should provide adequate Internet service for the university's needs throughout the remainder of this academic year.

Please bear with us as we work with the university's ISP to resolve these problems. Your patience is greatly appreciated!


We welcome your comments, questions and suggestions.

Please send feedback and questions to: CNSWebMaster@vt.edu
Please send network related questions and concerns to: ASK-CNS@listserv.vt.edu
Please send questions about student rates and policies to: stunet@vt.edu