Production Exchange potential service impact Dec 2-6 - courtesy reminder
During the week of December 2-6, 2013, a Microsoft Premier Support Engineer will be on-site assisting the university's E-Communications Services and Windows Administration Services /ECS-WAS and Microsoft: Secure Infrastructure Services/M:SIS teams with extending the on-premise Exchange environment to support a hybrid cloud infrastructure. This is Phase One of the VT Office 365 project which includes extending our Exchange environment to support the ability to live migrate user mailboxes into the Microsoft Office 365 Exchange 2013 environment.
During this timeframe, the production on-campus Exchange service may encounter brief mail delivery delays, active sync and calendar slowness and the potential for auto discovery changes to occur. We appreciate your flexibility during this deployment and we will work to minimize any production impacts as we improve this service. For critical service issues related to Exchange, we will utilize the computing.vt.edu "Current Computer Status" banner and cross-post to the technical support listserv (techsupport).
VT AD child domain compromise and active directory exposure
For more detailed information, including best practices and technical analysis, VT affiliates may refer to: https://secure.hosting.vt.edu/www.support.vt.edu/ADcompromise/
On Friday morning, October 4th, 2013, staff from the central IT Systems Support department discovered unauthorized services running on one of the domain controllers for Central Services (cntrlsrvs). Investigation of other systems searching for the evolving fingerprint of the attack began immediately.
Over the course of the next several hours we determined that the following showed evidence of the attack:
- one or more domain controllers in CNTRLSRVS, AIS, and UC,
- VT WSUS servers,
- the licensing server for the ESRI ARC GIS software
- one of the IAS servers associated with VT-Wireless, and
- some test systems.
A domain admin account was used on the domain controllers.
The exact vector is unknown, and awaits forensic analysis of the system snapshots by the IT Security Office, although a brute force attack on the admin account is suspected. There was a period where Group Policy Object (GPO) was partially unconfigured. We do see traces on some machines back to at least Aug 14, 2013.
Please note that no evidence is present that the password hashes were exported, or that the domain account was used to do anything other than install a proxy service on those machines. In fact, currently no machines without that service user have been involved.
With the concurrence of IT management and the Computer Incident Response Team (CIRT), our mitigation plan did not implement the usual recommendation of "Burn it down and start over." This would have crippled storage.vt.edu, as well as 1000+ systems belonging to various departments, notably including Athletics, the VT Police Dept., and the Office of Emergency Management at 5:15pm on the Friday of homecoming weekend.
At this point all admin credentials, policies, group memberships, etc., have been reviewed, accounts have been temporarily locked where possible, and passwords have been changed on the domains in question. No local user accounts were locked. Several temporary router blocks at the border have been implemented to assist with containment and detection.
All the Domain Controllers that possess the compromised service account (whether known to have been compromised or not) have been replaced with fresh systems with Active Directory data. Clean replacement systems are being built for the non-critical machines and Systems Support will work with the appropriate application administrators to bring services back into production.
WHAT DID *NOT* HAPPEN:
Allow us to stress some key things that are *NOT* part of this incident.
This incident is:
- NOT an exposure of HOKIES accounts
- NOT an exposure of the general domain forest
- NOT, to the best of our knowledge, an exposure of any personal, or PII, or PCI information.
More technical details on the attack fingerprint will be forthcoming separately.
In the interim, we strongly urge administrators who have systems that are joined to the affected domains:
to please perform due diligence and check your systems for unusual activity. We also recommend that you change the password of any local users you created in your OUs.
Again, we have no evidence to cause us to believe that the problems extend beyond the systems that were already discovered, but good security practices dictate that we assume otherwise.
We will distribute more information via the usual channels as we have it.
Send questions or information to ASK-CNS@listserv.vt.edu.
Urgent maintenance of load balancing equipment Saturday morning, October 5, 2013
To address network problems encountered on Thursday, September 26, 2013, with the campus Server/Applications Load Balancing equipment (in the ISB datacenter), maintenance is scheduled for tomorrow, Saturday, October 5th, between 5 a.m. and 7 a.m. The backup load balancing equipment will be serviced first, then brought online. Once that step successfully completes, the primary load balancing equipment will be serviced, then returned to operation.
Various campus services may be impacted--we anticipate briefly--including www.vt.edu, ldap, auth, CAS, my.vt.edu, hosting.vt.edu, banner, email, filebox, etc. Expected downtime will be minimal, though the full maintenance window is reserved in the event of unforeseen problems.
Following the maintenance, network problems should be reported to the Virginia Tech Operations Center by calling 1-6780.
Late summer telecommunications orders
As you undoubtedly are at this time of the year, Communication Network Services (CNS) is busy completing work related to summer renovation projects or faculty/staff moves before the start of the Fall semester. This year the volume of telecommunications summer work is unusually heavy and CNS is especially busy with the ongoing implementation of Unified Communications services on campus.
So please understand if you have not already submitted an Interdepartmental Communications Request (ICR) for telecommunications service, CNS cannot guarantee that new requests for service can be completed prior to the start of the Fall semester. We will do everything possible to accommodate newly submitted orders, but even expedited ICRs may be challenging to fulfill due to the current volume of orders. Please forward questions to CNS Ordering and Provisioning via Ask-CNS@vt.edu or by calling 540-231-6460.
Campus Internet access to be interrupted
UPDATE:The campus core and border maintenance has been postponed until August 2.
Campus access to the Internet will be interrupted on Friday, August 2, 2013, to upgrade equipment at the core of the university's data network, along with the switches that connect the campus to the Internet. The upgrade will begin at 11:00 p.m. and is scheduled to conclude by 7:00 a.m. on Saturday, August 3, 2013. This interruption will impact both incoming and outgoing Internet traffic, but will have only a minor effect (two to five minutes of disruption) on network traffic that stays local to the campus. The university homepage (www.vt.edu) will be available from our emergency recovery site, but most other services which require connectivity beyond the university campus will be unavailable.
The campus data network is the foundation for the university's Internet communications. These “core and border” network components require an upgrade in order to support the expanding and increasingly sophisticated requirements of Virginia Tech's academic, research, administrative, and public safety functions.
Following the upgrade, the university will be better-positioned to conduct network maintenance, with higher reliability, shorter resolution times and without service impacts, all of which will maximize flexibility for responding to current and future university communication needs.
We will continue to keep the university community informed and provide reminders as the date draws closer. Questions may be referred to NI&S via Ask-CNS@vt.edu or by calling 540-231-6460.
University Wireless Network Changes
Beginning July 1, 2013, Virginia Tech will transition to a new service to simplify the initial connection of a device to the university's wireless network. When a student, faculty or staff member connects a Wi-Fi enabled device to the campus network for the first time, they should select the CONNECTtoVT-Wireless (replaces VT_WLAN) network and use Cloudpath's XpressConnect service. The XpressConnect wizard will provide simple on-screen steps to authenticate the user, configure the device and quickly connect to the secure VT-Wireless network.
After the device's initial configuration and connection, each subsequent connection to the university's VT-Wireless network will be automatically authenticated and connected. VT-Wireless uses the IEEE 802.1x authentication standard, which provides increased security by encrypting network traffic between the user's computer and the wireless access point. The XpressConnect wizard will minimize configuration errors and ensure reliable connections.
Concurrent with this transition, NI&S will standardize the combination of a Network ID (same as your PID) and network password as the single set of authorization credentials to simplify network access. These credentials will serve all your network access needs, both wireless and VPN. All users who have been connecting to VT_WLAN are required to set up a network password on my.vt.edu; information is linked on the XpressConnect webpage for setting up one's network password.
NOTE: If you are currently using your Hokies credentials to access VT-Wireless, they will continue to work after the change. You will still use your PID and PID password to access services that require PID and PID password credentials.
Please forward questions or comments to 4Help online or by calling 540-231-4357.
Possible brief interruptions of Outlook Webmail, Active-sync
A Microsoft Engineer will be on site this Wednesday and Thursday (6/19 and 6/20) to work with the university mail team on configuring Exchange to work with Office 365. Though not planned, there may be a need over the two days to briefly interrupt service to Outlook Web Access (weboutlook.vt.edu) and active-sync serving mobile devices. Interruptions should last only 10-15 minutes. Questions may be referred to 4Help online or via phone: 540-231-4357.
University E911 System Upgrade Scheduled
Virginia Tech's E911 telephone system will be updated by Network Infrastructure and Services (NI&S) on Saturday, June 8, 2013, between 10 a.m. and 6 p.m. 911 services will be continuously available during the maintenance effort. Questions about the upgrade may be referred to NI&S' 4Help Center at 540-231-4357.
Telecommunications Billing Scam
Network Infrastructure and Services (NI&S) has learned of a telecommunications billing scam that some university departments have inquired about. A PDF copy of a bill from the alleged company, “UST,” for a “Telecom Maintenance Agreement” may be viewed here. Should you have questions about your university telecommunications charges, feel free to send inquiries to email@example.com or, call 540-231-6460 and ask for a NI&S Accounts Receivable staff member.
Place spring/summer telecommunications orders early
Summer is an especially busy time for Communications Network Services. College, department, and office staff planning summer renovations or moves, which will involve new or changed telecommunications services, may find the following helpful:
- If you have telecommunications work you wish to be billed during the fiscal year ending June 30th, 2013, your Interdepartmental Communications Requests (ICRs) should be received by CNS no later than April 20th, to ensure sufficient time to complete the work by May 25th.
- Should you require installation of equipment or services, or assistance with a move to be completed before August 31st, which does not need to be billed during the fiscal year ending June 30th, 2013, CNS will be better able to fulfill your requirement if we **receive your ICRs before the end of June**. ICRs received by CNS later than this may not be completed until after the fall semester begins.
The Interdepartmental Communications Request form may be accessed via COLA
Larger departmental moves and most renovation projects should be specified via CNS' “Request For Estimate" form
Departments, understandably, may elect to delay ordering telecommunications services until the spring semester concludes, increasing the volume of summer orders. Additionally, this summer's telecommunications work will be impacted by the university's ongoing implementation of Unified Communications (UC) services.
The Unified Communications transition schedule is regularly updated and may be viewed at
By sharing this information with staff who order your department's telecommunications services, you will significantly improve CNS' ability to serve you in a timely manner.
Visit the CNS website or contact a CNS Ordering & Provisioning team member at 1-6460. We look forward to providing for your telecommunications needs!
Enterprise Storage System Maintenance, March 9, 2013
Virginia Tech's enterprise EMC VNX5700 storage system is scheduled for maintenance, March 9, 2013.
Scholar.vt.edu WILL be available during the maintenance.
Beginning at 12:01 a.m. through noon Saturday, March 9th, all virtual machines on the storage system will be UNAVAILABLE, interrupting the following services:
- Email clients in general, and especially those used off campus (which access auth.smtp.vt.edu), will not be able to send email during the maintenance. Incoming email will still be received and processed.
- Data Warehouse
- Banner production database
- Banner applications:
- Internet Native Banner
- Self-Service Banner (aka Hokiespa)
- Banner Workflow
- Travel & Expense
- Effort Reporting
- Learning Technologies Applications
- FDI Registration
- SPOT Survey
- Timeclock Plus
- VT Office Communications Server
- VT Windows Software Update Service/VTWSUS
- Network Backup Service including TSM and NetWorker
- answers.vt.edu and computing.vt.edu
Please contact 4Help at http://4help.vt.edu/ or call 540-231-4357, with questions.
CNS Phishing Scam Reminder
Virginia Tech e-mail system users are often harassed by online phishing messages. Phishing is the act of sending an e-mail falsely claiming to be an established legitimate enterprise in an attempt to scam the user into surrendering private information that will be used for identity theft ("phishing" www.webopedia.com). Private information requested in phishing e-mail messages often includes user names, account passwords, credit card, social security, or bank account numbers.
Do Not Respond!
Although Virginia Tech continually warns e-mail system users about phishing scams, some users are still responding to phishing messages. The University does not monitor e-mail content and cannot determine whether e-mail system users are providing their credentials, but responding to these scams in any way can have a negative effect and is likely to increase the amount of spam received.
Never Share Your PID & Password
Virginia Tech will never request your PID and password. No matter how eloquent, realistic, or grammatically correct a note appears to be, if it requests your password or other personal information, it is fake. Do not respond to any e-mail messages requesting your Virginia Tech PID and password. Anyone who requests this information intends to use it in a malicious manner.
More E-Mail Security Tips
- Never send any passwords of any type via e-mail - don't be part of the problem!
- Change your PID password occasionally
- Don't use your VT PID password as a password for other acccounts, especially if you use your PID as your user/account name.
- Consider using an e-mail alias instead of your PID as your e-mail address. For more information, see Using E-mail Aliases With Your VT Mail Account.
To keep up-to-date with current phishing schemes (in email and social network sites like Facebook, Twitter, LinkedIn and others) take a look at http://www.millersmiles.co.uk/.