Greetings from kaizenOps.io; my name is Nate and I’ve been in the application performance and availability space for 15+ years. Though looking back, one could say I’ve been working in this space as far back as my first bona fide corporate job as a college intern for Travelers Property Casualty in their IT management division. It was a fascinating introduction to the world of application delivery in what we now call the old-school data center age. I worked with a legion of folks that put together and ran the data centers and desktops that enabled Travelers’ insurance agents to electronically sell and process insurance policies.
Distributed computing was the only type of computing I had ever been exposed to when I arrived, but it turned out one gentleman I worked for was also the manager for mainframe administrators. One day the boss asked him a question about the status of a mainframe application, and that’s how I first saw a Mainframe green screen.
I’d heard stories of green screens, and quips like “Mainframes are the only thing keeping large Enterprises running”, but I had no real understanding of what that meant. Lucky for me, this manager took the time to explain the weekly paper mail evaluation application. It was a simple process that only did one thing – it went through the entire customer base and checked whether this was the week an individual customer was supposed to get their paper bill sent to them. If it was, the application would look up the customer’s address and send the name and address to another application.
Initially, I had a hard time visualizing this as an application – I mean, it doesn’t do anything, right?!?! But after a few discussion points, I finally started to ‘get it’. While simple, this application operated at massive scale; every single customer was processed. That meant checking hundreds of thousands (millions?) of customers, and the decision on whether to send an envelope had to be done both correctly and in a timely manner.
If the decision was wrong then either the customer did not get a bill or it would send the customers duplicate mail. Duplicate mail is a real problem as it would both annoy and confuse customers, and waste real money generating physical mail. Exponentially worse than duplication, however, was not finishing in a timely manner. If this application-that-did-nothing was too slow or broken, business would literally stop as no one would get a bill!
I finally connected the dots for what “Mainframes run Enterprises” meant and a few of the implications. Here was some highly tuned compiled mainframe code written in the 70’s processing a simple decision at massive scale, and you better hope to god that it doesn’t break and you never have to touch it.
After the application whiteboard session, he logged into his dumb terminal ‘green screen’ to have the mainframe tell him its status and performance. From what I recall, it looked a little like this:
He picked out a few CPU and memory metrics and checked on how long the process was running, and that was it.
It wasn’t until many years later that I realized the significance of what I saw and learned that day. A simple terminal that monitored a few infrastructure metrics was the genesis of the first 30-40 years of Enterprise Operations.
In this scenario, putting an application into production was basically starting a wind-up top above a resource pool of CPU and memory and letting go. The hope was the top would spin through the resource pile as fast as possible to help it finish its task. When the success of the business is relying on speedy execution, this screams at you to simply keep tabs on a few data points and get out of the way. If the boss asks about the status of the application, you check if is running, for how long, and a few CPU and memory numbers.
Handling the scenario this way makes a lot of sense, but applications have evolved in many ways where simple monitoring like this falls very short. I’ll talk about how the applications have changed and I’ll touch on different monitoring evolutions in future blogs.
How about you? What was your first monitoring experience? Was it on a green screen? Please let me know if the comments.