On February 22 if you have AT&T mobile service, you may have been one of millions that faced an outage that spanned started around 3:30 a.m. EST and resolved almost 12 hours later at 3 p.m.
While the FCC has opened a formal investigation into what happened, and affected customers receive a $5 refund for the inconvenience, early speculation is the outage was created by a software upgrade gone awry. Today, we wanted to take a minute not to point fingers, but rather feed into the speculation and share our own best practices for software upgrades.
At infinIT, we’re diligent with our processes because we’re committed to ensure business outages don’t occur for our customers. We aren’t saying that accidents don’t happen, but rather we advise the following to be put in place on your own or with a trusted advisor like infinIT.
infinIT’s ‘must have’s’ in your IT change management policy.
- Maintenance Window. Establish a window, preferably after-hours and even overnight if possible. Make sure the window is communicated with a variety of communication mediums, multiple times, prior to an upgrade or reboot.
- Standby. Even if it is after hours and/or an inconvenient time, make sure that your internal, external (or both) support staff is on standby to quickly address interruptions and or outages.
- Test. Don’t rollout any changes without trying the upgrade/changes in a test environment. This was configuration changes or updates can be verified and implemented without impacting production.
- Ask for Help. Don’t make changes in a vacuum. It is okay to have peer review.
- Be Ready to Backtrack. Make sure you have backed up your current (working) configurations before applying any updates and be ready to roll back if an outage or interruption should occur.
- Track IT. Update the change management policy to document the changes.
- Say Something. Make sure you keep everyone up-to-date following your successful change/upgrade so the appropriate teams know what to expect.
Do you have a change management policy or are you, like most organizations ‘winging it?’
Our team can assist your organization with a straightforward approach to strengthening your business continuity.
