Inside RDK

DevOps: A Deeper Dive

(Excerpted from the Spring/Summer 2016 RDK Report exclusively for

The RDK community is assuredly not alone in its curiosity and involvement in “DevOps” – the blend of product development and day-to-day operations. While it used to be that DevOps was a buzz phrase that defined the leaders in web-based services — Amazon, Etsy, Facebook and Netflix are all heavy partakers — it’s now in play in the automotive, insurance, and health care industries, noted participants at the 2016 RDK Users Conference.

“People think DevOps is something you can buy. It is not,” said Vikul Gupta, Director, Quality Engineering & Assurance for Cognizant. “DevOps is a transformation journey.”

As the RDK developed over the past four years, and as adoption ramped up, a number of ancillary areas emerged as ways to further enhance its applicability. One of the hottest is DevOps, for its contributions to the RDK by way of service agility, analytics, troubleshooting, and customer care.

It hasn’t been a journey without bumps, especially for people for whom quality assurance/QA is job #1. “One of the things that people don’t initially believe about DevOps is that it improves quality and stability,” noted Anthony Troy, VP/Business Operations for Comcast, who added that a big fear when rolling out the DevOps program at Comcast was that quality assurance /QA teams wouldn’t be able to “check every box” before launching, “and then anarchy was going to break out.”

Not so, he said. “It’s important for the teams to not only be accountable, but responsible. People tend to think it’s cool to go to DevOps — ‘developers are in charge now!’ — but they have to be responsible for the design, development, deployment, support and metrics,” Troy added.

That’s why it’s important to view DevOps as a way to analyze the complete value chain, from the time requirements are defined, until the time the product is delivered and supported, said Anant Krishnamachari, Head of Quality, Broadcast Business Unit, Tata Elxsi. “Many organizations today face chronic product issues” — like an “unknown reboot” problem on RDK boxes within one provider’s footprint. By analyzing logs and correlating them with the glitch, Tata’s technologists were able to reproduce the error, which delivered the path to fix it. “Triaging is an area where analytics will play a key role — it’s emerging as a category of ‘intelligent DevOps,’” he said.

The fusion of DevOps and analytics can also inspire creative action in marketing and customer care, said Sef Tuma, Managing Director and Chief Strategy Officer for Accenture Digital Video. “A bad customer experience can become a marketing opportunity with analytics — but in a DevOps environment, with analytics, you can see that immediately, and offer instant remediation — ‘we’re sorry, please accept this free thing, as a token of our appreciation.’”

The Power of Traffic Shaping

Comcast takes an iterative approach to the service velocity that comes with DevOps by “traffic shaping” its software releases — meaning small rollouts to 1% of its RDK footprint, then 5%, then 50%, all the way up to 100%. “That way, you can see how it goes, and pull it back if problems arise,” said Comcast’s Troy. “Once you roll it out to that first 1%, which we randomly select, you can correlate it with your call-in and truck roll data. If it starts to teeter, we pull it back. If not, it keeps on going, to 5%, 50%, 100%.”

Cognizant’s DevOps program takes a similar approach. “In the spirit of continuous integration, you build in the rollbacks,” said Cognizant’s Gupta. “You have to be prepared to fall back, if there’s an issue.”

It follows that testing, and especially automated testing, is a big part of a solid DevOps environment. “Testing is really about confidence,” Comcast’s Troy explained. “It’s about how to not break things with your code.”