When it comes to upgrading the world's largest payment processing system, failure is not an option, said Scott Thompson, executive vice president of Visa International's technology practice, Inovant LLC. Thompson recently shared some of his best practices with eWEEK Labs.

Teamwork. Two years ago, Visa determined that completed projects would be more efficient and more flexible if quality assurance and engineering staff work closely together on the development of code.

Visa made a strategic decision to make engineering divisions responsible for quality assurance. This way, everyone involved with the project is equally vested in its outcome, Thompson said. This is especially important because the amount of time it takes to write an October release has jumped from 40,000 worker hours three years ago to more than 150,000 worker hours this year with no increase in personnel.

Responsibility. One of the main reasons Visa has been able to pull off its network upgrades is that everyone involved on the projects knows exactly how their decisions will affect the business. During the planning process, business managers and engineers discuss the types of changes they'd like to see and then determine what needs to be changed within the code to make those changes happen.

"The fact that everyone knows exactly how these changes are going to impact Visa as a business is paramount to our success," said Robin Owens, division head for Visa's Member Information Clearing and Settlement.

Work closely with partners. The company can't rely on its 21,000 merchant and bank partners to make changes as quickly as it does. As a result, Visa often throttles back its project to ensure that all its partners are on the same page and can make the upgrade jump together.

Visa also works closely with partners to help them prepare for the network updates.

Stick to schedule. At any point during the implementation, Visa engineers know exactly where they are in the upgrade cycle. Before the upgrade begins, a script is distributed to employees throughout the organization so that everyone knows at exactly what time a particular data center will be upgraded and exactly how long that upgrade will take.

"We don't do something and hope it's going to take 2 hours; we know it's going to take 2 hours," Thompson said. "We know exactly what's going to happen."