Ask the Experts: Making Sense of the DevOps / ITSM Intersection (Part 2)


Welcome back for part 2 of the series with our resident DevOps expert, Rob Hayes.

Rob has spent the last 10 years focusing on delivering successful ServiceNow implementations and projects. He has experience in most areas of the platform including system integrations, developing custom scoped applications, and is a ServiceNow Certified Application Developer and Implementation Specialist in ITSM and Service Mapping. Prior to specializing in ServiceNow, Rob has experience in various roles within IT including Application Development, Project Management, Team and Operational Leadership.

Rob recently participated in an itSMF panel discussion and shared his unique perspective on what it means to embrace DevOps from an ITSM viewpoint. In Part 1, Rob covered the definition of DevOps and Agile from the position of an ITSM practitioner and talked culture change.

In Part 2 (below), Rob discusses ITSM revitalization through DevOps and the associated risks and opportunities.

How is ITSM positioned with DevOps?

You still have the ITSM foundation—the same tenets are there. The concepts and the incidents don’t go away. The problems don’t go away, the changes don’t go away. The thing to understand is how you are going to streamline, how you are going to get closer to the code, closer to the developers, closer to the community that is using your process. As you do, continue to ask who you are serving and what the business value is of what that process is providing.

The most important practice to reiterate is that we’re always trying to minimize our risk. We’re always trying to understand how we can improve processes and not serve the process for the sake of the process.

In the past, that’s how we got buy-in from the DevOps teams to actually use the process. If we could build the process touch points instream with their current CI/CD process, it didn’t feel so heavy. One brilliant idea was to build in plugins that were added in to the deployment process to automatically trigger changes. Because we were doing small changes, small deployments, we could now do those twenty times a day.

Find your touch points and ask if they are essential. If they are, then ask if they can be automated so they can become consumable. Another DevOps meetup participant spoke to the challenge facing their incident management team. They were struggling with a lack of visibility into the “mircoservices effect” of their DevOps applications, small, self-contained things with their own specific dashboards and telemetrics.

This organization decided to build their own service to allow registration of the microservices to occur. Now they register each service when they do a release and they know where the dashboard is because everything is pulled together in one place. Brilliant.

That’s an example of taking the same concepts you already have and meeting the developers where they live, meeting your requirements from an audit standpoint. There is an opportunity to rethink the processes and revitalize them, streamline them so they are truly meaningful for your organization.

How does this impact testing?

Test-driven development is something that developers are starting to embrace. It’s part Agile and a little bit of DevOps in that you write your tests at the same time as when you’re writing your functional code. Then after you’re finished with your unit tests, you write tests to do your integration tests. If you’re coding everything, a developer should get to the point where it’s shocking if anybody finds an error.

One of the things we did as part of a former development team was to make sure we had 90%+ code coverage on all of our tests. So if something failed, we had a test for it already in our suite. So in this model segmented roles for testing are really going away. And that requires a different level of cultural commitment to quality embedded in the dev culture.

You need to develop a culture where developers are accountable for qa and basically feel a pride of ownership to the degree where they are saying “I put that out there. If you get a QA defect, I would be shocked.”

The idea behind DevOps is that you’re deploying things into your production environment and taking advantage of things like A/B releases, blue-green releases and canary releases so you can put something out there and only send a fraction of the traffic to it or not any traffic at all. Then it’s a configuration change to actually release it and make it available.

One of the famous stories about the DevOps movement is Facebook Messenger. Messenger was in production for a full year having shadow traffic go through it before it was ever released to the public, so when Facebook released it, they were confident it was going to work. That release was really just a matter of saying ok, we’re good, turn it on.

And that’s one of the core concepts of DevOps; trying to say let’s reduce our risk by putting stuff out there and allowing ourselves to have these deployments where we only are doing a fractionable piece and when it’s good, we’re ready to turn it on.

What are the risks?

I think the biggest risk when you start in DevOps is if teams go in different directions and the potential for lost visibility if they choose to not utilize your ITSM tool.

When you start out, typically you don’t change everything right away. Instead, you start out with small, coordinated groups and if that’s not well managed, you’re going to start to see duplicate logging solutions, duplicate monitoring solutions, multiple other solutions happening. You’re going to find yourself as an ITSM person asking, “Didn’t we do this before? Didn’t we go through a consolidation effort to do these things?”

There’s a tendency to say this is new, so let’s have everything be new, and we want to push accountability down to the teams so they can decide certain things. But there are certain items that ITSM has that we want them to plug into, and if they go in different directions, visibility goes away, and visibility is one of the tenets of DevOps that you most want to have. You want to be more public with your stuff, so your entire organization can see what’s going on.  And then determine if you can share it external to your organization, and do so.

The solution is building out a service management layer that can be plugged into and shared across the organization for these DevOps teams. Once you start to “get” DevOps, everybody sees the value, and you start to have things like DevOps sharing. But you have to have the shared ITSM infrastructure so that you have visibility.

At some level you have to have the perspective of “I’ll let that team do their thing. But we’re not negotiating this piece.” You have to still feed into the ITSM process. We’ll negotiate how. We’ll negotiate how do we make that better, faster, cheaper, easier. That to me is what needs to happen because I’ve seen it not happen and you have a mess on your hands. And now you’re trying to figure out how to pull it all together.

Are there lessons from history that ITSM professionals can use to create success when adopting DevOps?

None of it is new. Can anybody remember when you didn’t have good processes in the organization, and there was chaos? People said there’s got to be a better way, and we found a better way. This is the same thing, just the next evolution. To anybody looking to embrace DevOps and understand the controls that need to be in place for a regulated environment, you’ve been there and you’ve done that. It’s just this is the next iteration of that.

Ultimately, I think you’ll still use your tools, you’ll still use the process tools and the ability to have metrics in place. Here’s the cool thing: Let’s have the machines generate the metrics. Then all I have to do is see that oh, that thing ran and here’s what it took. Automation is going to help with that. I think all of the tools that you have right now are still valued and valuable in the future. Let’s allow the robots to do some of the work that we used to do manually.

Rob Hayes

Solution Architect, AERITAE