The Key to Large-Scale Mainframe DevOps Transformation: Building the Right Dedicated Team

Transformations don’t just happen.

Especially big transformations of decades-old processes.

Table of contents

They can’t be pushed through by ad hoc efforts from internal resources with many other responsibilities on their plate. These types of transformation require the dedicated time, effort, and attention of a team designed to bring them to life.

Unfortunately, one of our clients learned this lesson the hard way.

In this article, we will explore the initial challenges our client faced attempting to produce a major transformation without first creating a team to drive it.

We will describe their initial efforts, explain the changes that we made, and provide a clear picture of how to build a DevOps team to drive a successful transformation effort like theirs.

Quick Recap: Global Giant Attempts Mainframe DevOps Transformation

The whole series is:
1. Bringing DevOps to Mainframe Application Development: A Real-World Story
2. The Key to Large-Scale Mainframe DevOps Transformation: Building the Right Dedicated Team
3. Choosing the Right First Steps in DevOps Mainframe Transformation
4. Improving the Initial Mainframe Automation and Creating a “Two Button Push” Process
5. How to Select and Implement the Right Tools for Mainframe DevOps Transformation
6. Picking the Right Tools: A Deep Dive Into the Most Popular Options for Mainframe DevOps
7. Completing the DevOps Pipeline for Mainframe: Designing and Implementing Test Automation
8. Making Change Stick: How to Get Mainframe Teams Onboard with New Strategies, Processes, and Tools

Our client was transforming the way they developed, tested, and deployed applications for their mainframe.

It is a huge global company with a strong technology focus. It generates tens of billions of dollars in revenue every year, and employs hundreds of thousands of people. Their decades-old mainframe sits at the heart of everything they do. And for the last 20+ years they have followed the same manual, legacy approach to build all of the applications for their mainframe.

The result: our client took far too long to develop, test, and deploy new applications for their mainframe. And when they did launch a new application, it was often buggy with serious quality issues, and had to be rolled back almost immediately.

Eventually, this client had enough. They knew they needed to fix this problem at the root. To do so, they decided to adopt a modern, enterprise-wide DevOps process to develop, test, and deploy applications for their mainframe.

To date, we have worked on this project for over 18 months. Together, we have made significant progress bringing modern DevOps practices to their mainframe environment.

In our first article, we provided an overview of the project, and one of the most significant challenges we faced throughout this project, and how we overcame it.

In this second article, we will explore one of the biggest changes we made at the start of this project. This change impacted every other step of this client’s transformation journey, and set their transformation up for success.

The change: We established a dedicated DevOps team within the client’s organization.

Setting the Stage: Lots of Mainframe Teams, Little Agreement

To be fair, this client had attempted to create their own DevOps team prior to bringing us on board. They had simply followed an approach that did not result in a unified, effective group of DevOps champions that were capable of driving their organization-wide transformation.

Here’s what went wrong.

The client entered their transformation effort with 20+ global teams who were responsible for the development, testing, and deployment of all mainframe applications throughout their organization.

Each of these teams had their own DevOps engineer who specialized in mainframe infrastructure. These engineers were selected naturally, and not very intentionally. Every group had to deploy code, and many of them attempted to automate as much of their work as possible. Over time, one member of each group came to specialize in this work, and became their de facto DevOps lead.

However, these DevOps leads did not communicate well with each other. They often built temporary solutions that only applied to their applications, and that could not be used by anyone else.

They also lacked code review and input from their colleagues. Each of them had to both develop and test their own pipeline, and they were unable to maintain quality levels for their work.

Overall, each group’s DevOps lead followed their own siloed, ad hoc approach. They did not maintain a consistent level of process automation, and the work they produced could not be applied to solve more than one problem for more than one project.

By following this scattered, individualistic approach to DevOps adoption, this client was unable to:

• Design, implement, and maintain a holistic architecture for their mainframe DevOps transformation.
• Drive the dramatic cultural change needed to make their transformation a success.
• Achieve continuous engagement and buy-in from development groups, operations groups, and business stakeholders.
• Develop and maintain sufficient patience, endurance, and motivation to make it through their transformation.

Clearly, this approach could not drive a consistent organization-wide transformation in how mainframe applications were developed, tested, and processed.

Course Correction: The Initial (Failed) Attempt to Unify the Mainframe DevOps Teams

This client saw the flaws in their approach. They knew that their current situation would not result in a unified, organization-wide DevOps transformation. So they attempted to correct course by unifying their existing DevOps leads.

To do so, they began to host DevOps specific calls and meetings. They invited all of the individual DevOps leads to discuss how the organization as a whole could move to a unified culture.

Together, they selected the tools that they would all agree to use, and they attempted to consolidate all of their existing automation scripts under one roof.

But that’s about as far as they got.

They soon saw that just selecting a tool for everyone to use would not be enough.

They quickly realized that they would need to write a lot more automation scripts.

And they came to understand that, above all, their existing DevOps leads just could not allocate enough of their time and attention to ensure the constant progress of, and ongoing governance over, their new processes. No one involved in this first attempt could drop everything else on their plate to write these scripts, let alone to manage the entire transformation effort.

That’s when they realized they would not be able to untangle this knot on their own.

And that’s when they asked for our help, and we worked with them to establish their dedicated DevOps team.

One Quick Note: Do You Need a Dedicated Team?

Before we proceed further, we wish to make one point clear. In this article we are providing a set of recommendations that are primarily focused on helping out large, mainframe-driven enterprises that will be building out DevOps practices around a massive volume of legacy code.

The broader distributed world of developers may not benefit as much from our discussion here. These developers are part of a much bigger and more open community. They have a lot of easy-to-integrate tools, plugins, and enterprise-grade solutions at their disposal. In their distributed world, one DevOps engineer can build their own DevOps pipelines, and solve all of their CI/CD problems through a quick Google search, and the deployment of some community-provided answers.

Unfortunately, developers and operations team members in the mainframe DevOps world are not so lucky. They are a much smaller community with much more specific problems. They will have a hard time finding working solutions to their problems through Google searches. They have to do a lot more of the work and problem-solving themselves, and thus require a team-based approach that others in the distributed DevOps world may not need to follow.

With that being said, let’s return to our story, and define what an optimal mainframe DevOps team might need to look like.

Building a Mainframe DevOps Team: The First Step to Transformation

To begin, we worked with this client’s existing DevOps leads to better define their needs.

We scoped out their transformation. We defined the gaps between their current capabilities and their desired end state. We mapped a practical set of requirements.

And we defined what their new Mainframe DevOps team would need to look like, and what responsibilities they would need to fulfill.

This team would need to:

• Establish and maintain the new DevOps culture.
• Design, implement, and govern the complete transformation.
• Unite the 20+ distributed teams through a set of shared processes to develop, test, and deploy mainframe applications.
• Produce a flexible set of solutions that could proliferate and be re-used across groups and implemented for many future projects.
• Become leaders who would be the first ones to adopt and model the changes everyone else would need to make.
• Review each other’s work, improve each other’s processes, and ensure applications were delivered faster and at a higher quality standard.
• Offer a single source of truth, and provide consistent answers to anyone who had any question about this transformation.

Ultimately, we created a team with set roles and responsibilities, who we developed specifically to spread the mainframe DevOps transformation throughout the organization.

But it wasn’t easy.

After all, the team we created had to provide coverage for a rare set of technical skills, and an even rarer set of soft skills.

Building the Dedicated Mainframe DevOps Team: Technical Requirements

DevOps is a complex subject area that is challenging to train.

DevOps for mainframe is even more complex and difficult to train.

It requires a wide range of technical skills that are near-impossible to find within any one person. DevOps team members who work in mainframe need to cover capabilities that include:

• A strong understanding of the DevOps methodology, in general.
• The ability to integrate modern, distributed tools that were not designed for mainframe into a mainframe environment.
• A deep understanding of the technical details of every currently-running mainframe project.
• Significant knowledge about the z/OS mainframe infrastructure, as well as mainframe languages (such as COBOL, PL/1, REXX, and JCL).
• The ability to write, test, and deploy new code from scratch, and to also be able to improve old code that was written in a legacy language.
• Critical analysis of problems in current development lifecycles, as well as the ability to both extract requirements for them and to translate those requirements into relevant, reusable solutions.

Even if you have access to comprehensive training in all of the subject area’s required skills, it is unrealistic to think that you could create a DevOps team for mainframe applications where everyone can do everything.

With that in mind, we worked with to create a DevOps team that as a whole provided coverage for all of the technical skills they required.

Together we assembled and trained a group of mainframe engineers with different specialization, who together met all the technical requirements of the project.

Building the Dedicated Mainframe DevOps Team: Soft Skill Requirements

For this project, technical skills were not enough.

Remember, this client’s DevOps team had to do more than just develop, test, and deploy high-quality applications for their mainframe.

They also had to take the lead and drive this sensitive, end-to-end transformation.

They needed to manage giant change initiative. They needed to constantly communicate with many different individuals, groups, and corners of the organization. They had to get everyone on board and keep them aligned with the new way of doing things.

Technical skills alone wouldn’t cut it. The DevOps team also needed to leverage considerable soft skills to construct and manage all of the right relationships.

To be clear, some of this did not need to be trained. We did our best to select engineers for the team who already had very good pre-existing relationships with multiple other mainframe applications teams. And we also did our best to select engineers who were already good at getting others to accept the changes being asked of them.

This helped. But it was not enough.

We also needed to train every member of the new DevOps team around the two sets of soft skills that are most critical to leading a DevOps transformation like this.

1/  The Ability to Network

DevOps methodology involves finding a way to get multiple different groups to work well together, including Management, Developers, Testers, and Business Analysts.

Designing and implementing DevOps automation also demands the collection and analysis of a lot of information from a lot of people.

In short: DevOps engineers need to be able to collaborate productively at many different stages of both the organization’s cultural transformation, and during the development of each new automation.

We placed special emphasis on training the DevOps team with the ability to make and maintain positive, productive connections throughout the entire organization.

2/  The Ability to Work Under Pressure

DevOps engineers also face a significant amount of pressure at all times.

They must develop, test, and deploy business critical applications that often have a high budget and that are typically delivered just prior to their deadline.

If anything goes wrong or if there is nervousness in the lead-up to the delivery, it is the DevOps engineer who receives the pressure (even though they are limited by the other elements of the pipeline).

What’s more, these DevOps engineers had to transform the entire development lifecycle, making them the “bad guy” among the other groups involved.

Given this, we also placed special emphasis on training the DevOps team to work well under pressure, and to not break when things went wrong and they were forced to take the blame.

Together, we curated and trained a team that could not only deliver on the technical requirements, but who could also ensure a smooth transition from the old ways of doing things to the new reality.

Building the Dedicated Mainframe DevOps Team: What a Team Looks Like

So far, we have provided an overview of why it’s needed to have a dedicated team to drive mainframe DevOps transformation, and what skills that team required to fulfill their challenging role.

Before we wrap up this piece, we’d like to provide an overview of what the composition of such a team might look like.

While we can’t share the precise makeup of the DevOps team we developed, we can provide a rough picture of what such a team might look like. We hope it is sufficient to help you begin to brainstorm how you might build out your own such team.

Overall, the DevOps team should follow a T-shaped team structure, where different individuals perform different tasks, such as one team member writing the processes, another writing the documentation, and another writing the integration scripts.

This type of team might look like:

Again, do not take this as a perfect representation of what your team might need to look like. Everyone’s context and requirements are different. But just consider this a starting point as you consider how a dedicated DevOps team like the one discussed in this article might be structured.

Bringing DevOps to Mainframe: Next Steps

The dedicated team we built with this client became the cornerstone of their DevOps transformation. Establishing this team was challenging, but it made every other step of this client’s DevOps transformation possible. We will outline each of these steps in detail over the remaining articles in this series.

In the next article, we will outline the first tangible change we made in how this client developed, tested, and deployed applications for their mainframe — the automation of REXX scripts.

If you would like to learn more about this project without waiting, then contact us today and we will talk you through the elements of driving a DevOps transformation for mainframe that are most relevant to your unique context.

How we can help you