Monthly Maintenance Plan

Information Technology has consolidated HSC monthly maintenance for networking equipment and all enterprise servers to occur on the third Saturday of each month, at 7:00 AM. This is a change from the previous monthly maintenance that covered only half of HSC’s enterprise servers, did not include Unix servers or Networking equipment, and occurred the second Sunday of each month at 11:00 PM.

Background

Periodic maintenance on IT systems is mandatory. Here are some of the reasons why:

Security patches
Hardware upgrades
Software patches and upgrades
Software and component installations
Re-configurations
Server reboots
Availability and fail-over testing

These are preventive measures that are essential to provide stable and secure systems for the campus. Maintenance usually lasts anywhere from four hours to over seven or eight hours when problems are identified. Many times, if a server is going to fail, it will happen during the reboots that are required during maintenance.

Monthly Maintenance Plan

The plan now defines a single monthly time for maintenance across Information Technology that will be scheduled from 7:00 AM until 2:00 PM on the third Saturday of each month. Even though the maintenance period calls for seven hours, interruptions to customers are typically brief. Customers are notified via email in advance of system down time during the week prior to the maintenance procedure. A System Alert is placed on the home page of the HSC main campus Web site on the day prior to Monthly Maintenance, and a message is recorded on the System Alert Hotline (405-271-4747 or 800-992-5411).

Justification
This proposal seeks to satisfy the following criteria:

Suitable vendor support. Experience has taught us that overall quality of support during overnight hours is unacceptable, and that availability of experienced engineers is significantly better during the day. Since risks and the resulting need for support is highest during maintenance procedures (e.g. system reboots, patch installation and software upgrades), maintenance should be scheduled during daytime hours to minimize downtime.
Minimize service interruption to customers. Although the maintenance is scheduled for a seven hour time period, interruption to customers is brief for several reasons.
- · Maintenance on each individual server may last from five to 20 minutes on average. The plan schedules seven hours due to the total number of servers being maintained.
- · The level of redundancy inherent in the current infrastructure allows support personnel to perform maintenance on one or more servers while other servers continue to provide services to customers. Examples of this include email and authentication servers.
Identify an overall low network utilization period for maintenance. Evenings tend to show higher utilization than Saturday and Sunday mornings due to evening remote user access.
Sufficient time to complete maintenance and resolve problems when they occur. The previous maintenance schedule pushed into Monday morning’s start of business when no problems were encountered. When problems did occur, interruption to business customers was practically assured. A maintenance schedule beginning earlier in the weekend reduces or avoids interruptions during the business week altogether.
Avoid schedule conflicts with database and system backups. All system and database backups run during the evening and early morning hours. This includes email and Peoplesoft systems. Previously, backups for Windows servers were delayed until after monthly maintenance. This caused the backups to run into Monday morning’s business day – a heavy utilization period resulting in the potential for degraded performance. PeopleSoft servers must be backed up at night under the existing methodology since PeopleSoft databases must be down during backups. The logical time to avoid schedule conflicts with backups is in the morning after backups have completed.
Avoid academic and administrative calendar conflicts. Finals complete prior to the third Saturday of both Fall and Spring semesters. A scheduled maintenance for the third Saturday of the month avoids conflicts for students preparing for exams. (e.g. Finals for Fall ’02 occur Monday-Friday, December 16-20, and finals for Spring ’03 occur Monday-Friday, May 5-9)
Alert system administrators. Maintenance procedures are a full day’s work. Starting these at 11:00 PM, when staff was fatigued, placed our servers at risk. Further, the new plan promotes the availability of a full and alert staff on Monday mornings.
A single consolidated maintenance period for all IT HSC enterprise systems and networking. A single monthly maintenance period will eliminate confusion for customers, minimize the number of service disruptions, and provide coordination for the HSC IT infrastructure groups.