The Easy Way to Bring DevOps to Mainframe Software Development

Table of contents

You are trying to modernize your mainframe software development group. You know that doing so will increase your speed and efficiency. But you don’t know where to start.

So, you search the web for some way to bring DevOps practices to your mainframe group. You find many different methods that sound good on paper but fail in practice.

Frustrated, you try to create your own tools, integrations and processes by scratch. But the whole project takes more time and effort than you think it’s worth, and you ultimately give up on the whole idea of bringing DevOps to mainframe.

It’s a painful cycle, and we wrote this article to help you break it.

In this article, we will give you a good, functional solution to the problem of how to bring DevOps to mainframe software development. We will do so in three steps.

• First, we will discuss the overall topic of how to bring DevOps practices to Mainframe software development, and tell you  the best way to do so.
• Second, we will provide a concrete example of our approach, and walk you through how to connect a mainframe with Jenkins pipeline using USS.
• Finally, we will provide some additional tools you can use to bring more DevOps practices to more aspects of mainframe software development.

Let’s get started.

Why You Must Bring DevOps to Mainframe Software Development

At a fundamental level, this is just something that you have to do. Every corner of mainframe has begun to adopt DevOps practices, and sooner or later you and your software development group will face some pressure to modernize as well.

The reason for this is simple — DevOps brings a lot of benefits to all things mainframe, especially mainframe software development. In particular DevOps can dramatically accelerate many of your core processes. For example, traditionally it has been normal to write several lines of code for a monolithic structure and then wait hours for it to compile. With DevOps practices you can complete this work in seconds.

• Automated builds
• Automated (and consistent) deployment
• Automated testing
• Automated code quality verification
• Automated monitoring

All of this adds up to one big benefit — with DevOps you can deliver higher quality software, more consistently, and much faster to your customers. And even if this doesn’t matter to you, it certainly matters to your senior stakeholders in the business, and they will pressure you to modernize your software development practice.

In sum: Even the most “old school” mainframe software engineer will eventually have to find some way to bring DevOps practices to their work. But unfortunately, many mainframe people are following a suboptimal method of doing so.

The Wrong and Right Way to Unite DevOps and Mainframe

As we outlined in a previous article, DevOps and mainframe are not always a perfect fit. What’s more, DevOps has come to mainframe later than it transformed other corners of IT. This means that “DevOps for mainframe” is still a few years behind the broader trends and most of the tools developed for this market are less than perfect.

Because of this, many mainframe software development teams are following the “wrong” way to bring DevOps practices to their work. They are looking at the market. They are seeing that “DevOps for mainframe” tools will not work for them. And they are choosing to create their own homegrown tools from scratch.

However, we have come to see that this approach is often fundamentally flawed for one big reason — most teams do not create reusable internal DevOps solutions that they can apply again and again to solve a wide range of problems. Instead, most teams build a single-use tool that can only solve the exact, immediate, here-and-now problem that they are struggling with. As a result, most teams burn through a lot of time, labor, and budget developing products that only work under specific conditions, and which must be thrown away as soon as those conditions change.

Now, like we said, this is an understandable approach and it’s not anyone’s fault.

But thankfully, there is a simple solution to this problem and a better approach.

While there are not a lot of great “DevOps for Mainframe” tools on the market, there are tons of great general-purpose DevOps tools on the market. And even though these great general-purpose DevOps tools were not designed specifically for mainframe, most of them can still be adapted for use within mainframe environments.

Ultimately, we have found that this is the best approach to bring DevOps practices to mainframe. Overall it is much faster, simpler, and more effective to just adapt existing general-purpose DevOps tools to work within a mainframe environment than it is to try and reinvent these tools and create new approaches from scratch.

In the next section of this article, we will demonstrate how to adapt general-purpose DevOps tools to mainframe environments. To do so, we will walk you through the concrete, real-world example of how to connect a mainframe with Jenkins pipelines using USS.

While everyone has their own unique mainframe with its own unique challenges to overcome, we will show you that you don’t need a unique solution to do so. We will show you how a general-purpose tool like Jenkins can solve most of your problems and automate many of your manual mainframe processes — fast, simple and easy.

Meet Jenkins: A General-Purpose DevOps Tool that Works Great with Mainframe

Jenkins brings a lot of benefits to mainframe software development, especially compared to creating a homegrown solution. Jenkins:

• Has already solved many automation problems and can automate many tasks for almost any solution, regardless of the scope of the solution’s infrastructure.
• Was developed to coordinate software building, testing, and deployment processes across multiple processes across multiple platforms.
• Has a huge number of plugins and integrations, extending its reach.
• Uses groovy scripting-based pipelines.
• Uses REST API to allow information feedback and pipeline triggering.
• Offers most of these capabilities out-of-the-box.

Most important, Jenkins is an active open-source project with a large and engaged community. It already has a lot of documentation, examples, and support, and the tool is well-maintained and constantly expanding to tackle new automation challenges.

In sum: Jenkins makes it fast, simple, and easy to bring a lot of DevOps practices to mainframe software development. It’s a great example of how you can save yourself a lot of time, effort and budget by using a general-purpose DevOps tool instead of a homegrown solution, and it’s worth finding a way to bring it to mainframe environments.

Here’s how to do just that.

How to Bring Jenkins to Mainframe: Imperfect Options

1. Running Jenkins directly on mainframe
2. Using the IBM z/OS Plugin for Jenkins
3. Building custom solutions

Let’s look at each with a little more depth and break down why they are all imperfect.

Option One: Running Jenkins directly on mainframe

• It requires a lot of very expensive CPU time.
• Other external services need to be available at the same time as Jenkins and you must find somewhere to keep them.
• You need to open additional access for external services, which is costly and unreasonably difficult, and creates additional security challenges.

Due to these challenges, among others, teams shouldn’t run Jenkins on their mainframe itself. Instead, they should keep Jenkins an external service that they run on a separate server.

Of course, running Jenkins as an external service creates a new challenge — how to make contact between Jenkins and the mainframe. Options two and three attempt to provide a solution to this challenge.

Option Two: Using the IBM z/OS Plugin for Jenkins

However, this plugin has some limitations. It only allows you to use Jenkins to interact with the mainframe SCLM. While this makes it a good solution if you are still storing SCLM on the mainframe, most modern teams have abandoned SCLM and now work through GitHub, BitBucket, or other similar services — which makes this plugin obsolete for many teams.

Option Three: Building Custom Solutions

Overall, while both of these methods have been useful, they are just too complicated, too time-intensive, and too high-effort to be used at scale. Mainframe software development teams need a modern and proven solution to connect Jenkins with mainframe — and we have developed that better approach.

Our Solution: A Simpler Way to Connect Jenkins to Mainframe

One day, we read carefully through Jenkins’ documentation. We learned that Jenkins already had some useful tools in its existing implementation methods. We realized we could use those tools to connect Jenkins to the mainframe in just a couple of steps and without developing any new plugins, simply by using the existing Jenkins agent.

The Jenkins agent is just a Java program that runs under USS. The Jenkins controller is the original node in the installation, and the controller administers the Jenkins agent and orchestrates its working, including scheduling jobs on the agent and monitoring them.

This agent can be connected to the Jenkins controller using either local or cloud computers. All it needs is a Java installation and a network connection to the Jenkins controller, and it provides a simple way to connect Jenkins and mainframe.

• Self-sufficient. You don’t need to install any additional tools on the mainframe, except the Java program to make everything work.
• Lightweight and simple. This solution will not take up too much disk space, and it will not overload the processor with a lot of work.
• Not loaded with business logic. It is  just a simple automation tool, which allows you to implement it and abandon it very quickly and easily.
• Secure. With this agent you are implementing automation into the mainframe with minimal external dependencies and a small impact on security.

Of course, the Jenkins agent is not perfect, and it does have some limitations. You won’t be able to perform certain tasks on the mainframe other than basic access to it, and it requires Java on USS. But it is no problem to put Java on USS and the problem of implementing more serious tasks has long been solved by other external components. Ultimately, the limitations of this approach are much, much smaller than the benefits you will gain by using agents to connect Jenkins to your mainframe.

Now, let’s dig into the details of how this approach works, and how you can connect the Jenkins agent to your mainframe as quickly and easily as possible.

How to Connect the Jenkins Agent to Mainframe: Step-by-Step

• Have Java under USS
• Open access so the agent and Java can interact
• Configure your initial settings in Jenkins
• And manage the agent through Jenkins

Most of these challenges — and most other challenges that you might encounter — already have a simple solution developed. You can find these solutions and more by reading through Jenkins’ existing documentation, which you can find here:

• https://www.jenkins.io/doc/book/using/using-agents/
• https://github.com/jenkinsci/remoting/blob/master/docs/inbound-agent.md

To get you started even faster and easier, here’s the exact step-by-step approach that our team uses. Our team has followed this approach to connect Jenkins to the mainframe on an IBM CustomPac project for the last two years. These steps give you a simple and time-effective way to set up and interact with components.

First, here are the steps to create the agent and insert it in USS

1. Open Jenkins, go to the Nodes tab, where you will find all of your mainframe nodes, where Jenkins is running in the mainframe’s USS.
2. To run Jenkins on the mainframe, click to create a new node, make it a permanent agent, and fill out the name and description nodes.
3. Assuming Java is installed in the mainframe, create an SSH connection, place a remote root directory, the directory of the user Jenkins in mainframe where the agent performs its work. This is the place Jenkins jobs, files, and stores on the mainframe side. Our directory on the USS side on the mainframe.
4. Choose “Launch agent by connecting it to the master.”
5. Save the node and Jenkins will automatically prepare a command line to run on the mainframe side.
6. In the mainframe, change the current user to Jenkins. Find the Jenkins workspace. You will find all Jenkins directories related to sources and scripts and so on.
7. Draw on the command, and Jenkins will automatically connect from the mainframe and the agent will connect to the Jenkins node.

While this method works, it will sometimes trigger some problems in Jenkins and force Jenkins to restart. Here’s a better way to connect the Jenkins agent to Jenkins by an SSH connection.

1. Create another node with its launch method set as “Launch agent via SSH”, host address as the address of your mainframe, credentials to connect to the mainframe and authorize on it, and place the SSH port and Java path for the USS services on the mainframe.
2. If all of your SSH keys are valid you can click the launch button in Jenkins, and Jenkins on itself will initiate the connection with the agent on the mainframe.

Second, here are the steps to use your agent to connect Jenkins and mainframe.

1. Go to the Jenkins web panel and set up your configuration to establish connections to the outside world.
2. Start your agent — either through the Jenkins web interface to launch it through USS, or to manually start the agent (if you don’t want Jenkins to make certain actions on your users’ mainframe).
3. If you configured your agent correctly, then the agent will throw the message to Jenkins that it’s ready to work and to acquire commands, and Jenkins will send commands back to the agent.

Third, here are the steps in a common use case — automating build, test, deploy

1. Make changes to your code in your IDE.
2. Make some commits to GitHub, or your other version control system.
3. Use web hooks to send everything required to Jenkins.
4. Start the pipeline that includes the Jenkins agent. This will grab everything from the GitHub commit, performs the automation, and sends everything to the end users.

Going Beyond Jenkins: Additional General-Purpose DevOps Tools for Mainframe

At the same time, you will learn — firsthand — that the easiest and most reliable way to introduce DevOps practices to your mainframe is to simply identify an existing and proven general-purpose DevOps tool, and to find a way to use it in your environment.

With that in mind, we recommend you take some time to get familiar with a wide range of open-source, general-purpose DevOps tools that have already been used extensively in the real world and proven their ability to solve automation problems.

Here are a few more of our favorite general-purpose DevOps approaches that you can adapt for mainframe:

• Azure DevOps for Mainframe: We have used this technology in our DevOps as a Service for Mainframe project. You can learn more about it by watching this example video.
• Shell Scripting: This method retains a lot of manual settings and is good if you only want to automate a few steps of your DevOps process. You can learn more about it here.
• UCD: This is a CI/CD tool from IBM that is primarily used to create pipelines for code deployment. You can learn a lot more about it by reading our detailed series of DevOps articles.

Or, if you would like some hands-on help using Jenkins in your mainframe software development environment, or some personalized advice on which open-source, general-purpose DevOps tools you should bring to your mainframe group.