VoIP Industry Newsletter: Understanding the Network Operations Center (NOC)

Dec 2010

Every telecom operator – large or small – operates a Network Operations Center (NOC). The purpose of the NOC is to monitor and control a carrier’s network assets and interconnections, and to minimize service interruptions and quality-of-service affecting events. The use of VoIP technology in telecommunications networks introduces new variables. Namely, you are dealing with a less predictable network, and it can be challenging to collect specific monitoring information. It can also be more difficult to replicate events for diagnosis. Essentially, though, NOCs rely on the same fundamentals.

In our role as a Managed Service provider to VoIP carriers and service providers, VoIP Logic is deeply engaged in daily operational interaction with our customers, primarily through their NOC. This experience gives us exposure to many engineers, numerous systems and, most importantly, multiple different ways of tackling the same issue— network monitoring and management. I’d like to discuss some of the attributes we have found most compelling as we deal with over 150 carriers in 28 countries. Beyond the differences in language, work culture, team experience and company/NOC scale, there remain several common lessons for what succeeds in network management and what does not.

Network operations personnel are tasked with maintaining performance and equilibrium on the network. Invariably, there are problems that arise—from basic human errors to catastrophic power, bandwidth and equipment failures. A successful NOC is one that resolves these inevitable problems quickly and succinctly. There are three components to meeting these objectives: capable personnel, robust monitoring systems and clear process.

A strong team of NOC engineers is not hired; it is architected. Of course, anyone that has interacted with engineers or worked in a NOC can see a how disparate skill-sets are more or less successful in pursuit of smooth network management. Let me clarify. You can definitely find and hire talented individuals. However, if the goal is efficiency, every team member must have the opportunity and training to perform, a clear understanding of how to resolve each network event that occurs, and a holistic understanding of the goals of the NOC, rather than single systems expertise. Specifically, cross training on multiple systems generally leads to faster issue diagnosis. Clear procedures—including a knowledge base of historic and pervasive events, commonly accessible escalation information, frequent internal training and a cross-functional process that integrates the commercial team that relies on your NOC—all contribute to managing personnel to a successful outcome. Finally, if the goal is efficiency, it is much more cost effective to architect your team, rather than pay top price for a team of all-stars.

While a strong team of NOC engineers is crucial to success, the fallibility of human hands is often a limiting factor. Machines that assist in network monitoring, reporting, alarming and even self-healing are the factor that can take a NOC from good to great. An issue we frequently see centers around diagnosis. Better systems provide more information to NOC personnel, and they allow for more time spent treating causes rather than symptoms, not unlike the field of medicine. There is undoubtedly a capital expense to bringing your software and hardware automation to a state-of-the-art standard, though capital outlay should result in operational expense savings immediately if invested smartly. The types of systems we see most frequently on VoIP networks for this purpose are involved in routing optimization—routing around network or traffic quality issues, middleware that allows single portal presentation of data, multilayer network surveillance and notification systems, and integration with Customer Relationship Management (CRM) systems to automate case management. Experience helps guide your choice of components and tools needed for your NOC. Each system or software you purchase and operate for your NOC is a capital and most likely on-going operating expense. In your financial modeling, you should be able to map out the operational efficiencies (and savings) gained from the capex. Ask yourself, Have we increased our ability to now operate the network easily using these systems in a streamlined, scalable and repeatable NOC process? Of course, while it is less expensive to defer or avoid spending your hard-earned profits on NOC automation, there is a level of automation appropriate for any budget, and you should always have a wish list for further investment as you start to see more value in reinvesting in your network capabilities.

Most NOCs start small and grow, like the networks and the companies for whom they work. Hiring personnel and installing automation are discrete steps to improving the capacity and information sharing. The last item I want to discuss, clear process and procedure, is less tangible and often minimized. This is a mistake. Clear process, especially if you have cross-trained personnel and automation systems allowing for faster diagnosis, is the hallmark of a professional organization. From a first report or alarm on a network effecting event all the way through de-briefing customers and vendors after normal network operations have resumed, there are steps and decisions at each juncture. These steps—triage, isolation, escalation, notification, invasive intervention, emergency response & management and possibly disaster recovery—must be crisply defined and constantly refined to reduce overall Mean Time To Restore (MTTR) of a production network. From the smallest of companies (a one person NOC) to the largest NOCs in the world, quality is determined by the team’s ability to move in synchronization. This feat is accomplished by adjusting process as warranted and keeping all stakeholders aware and abreast of these changes through documentation.

In 2011, VoIP Logic will begin offering NOC services to carriers and service providers using VoIP technology While we hope to offer our NOC as a service, we are more interested in raising the standards of network operations in general by shining a light on common weaknesses. I hope this article inspires you to think about how you can improve or build your own NOC.

Micah Singer, CEO
  Next>>