Cloud Watching

I’m a big fan of clouds. There, I said it. No, I am not talking about those cottony white puffs of condensed water vapor dotting an otherwise dull blue sky, beautiful as they can be. I’m talking about digital clouds, also known as Cloud Computing or On-Demand Computing.

Cloud Computing is one of the few game changers that is actually worth the name “game changer”. In essence, it’s all about maximizing the effectiveness of shared resources and applying economics of scale.

Working as an IT Manager in the energy sector, that concept certainly rings a bell. During the Industrial Revolution every factory had its own steam generator to supply power to its machines. With the advent of technological innovation, i.e. the ability to transport electricity, it didn’t take too long before people figured out centralized power generation might make more sense, and the first power plants soon came into existence.

Pearl St. Power PlantThe Pearl Street power plant commenced operation on Sept. 4, 1882

The Information Age followed much the same path. Initial efforts focused solely on individual campus or enterprise data processing facilities. And again it was advancement of technology, primarily the global commercial rise of the Internet, that ignited a shift towards centralization, leading to the centralized compute capacity we today call “the Cloud”.

Now, although most people will perceive these changes as change for the better, we can’t ignore that every coin has two sides. Change almost always brings about fear, uncertainty and doubt. In case of the availability of electricity, if you’re in any way familiar with the “War of the Currents“, you’ll appreciate what I’m talking about.

Killed by ACThe deadly accident of lineman John Feeks on Oct. 11, 1889 fueled concern about the dangers of power transmission

And where our forefathers feared the clouds would fall on their heads, digital Clouds carry their own share of fear, uncertainty and doubt. Major technology vendors started evangelizing: “If you are a systems administrator, start looking for another job. Soon all your systems will be run by a single guy running thousands upon thousands of servers at the same time”. Or playing the blame game: “All you competitors are moving to the Cloud already, you are wasting your company’s resources if you stick to your own local infrastructure”. And then hardware manufacturers cheer in: “You surely don’t want all your data to be out there in the open, do you?”, and immediately offering the helpful “we know your boss is nagging you about this Cloud thing, so buy this box here from us, and tell him you invested in a Private Cloud. All the benefits of the public Cloud, without any of the risks!” Sounds familiar?

And of course, there is not just one Cloud. You have Amazon’s AWS, Microsoft’s Azure, Google’s Cloud Computing, and hundreds of smaller clouds most of us never heard about and probably never will. Oh, and compatibility is for cheesy matchmaking services, not for serious technology vendors, so you’ll get to bet on your favorite flavor and stick with it. It’s the War of the Currents all over.

At this point you might be inquiring “Wait, didn’t you start by saying you loved the Cloud?”. Yes, I did, and I meant it too. But I’ll let you in on a secret: it sure took me some time to get over my own fears, uncertainties and doubts.

My tipping point came when I was working on some graphical design project. I needed to render a couple of poster-size images I created in Blender/Cycles. You see, working as an IT Manager, I don’t get much of a chance to use any hands-on IT skills anymore. Using PowerPoint and Excel are the two notable exceptions. So I kinda like to make up my own little pet projects at home. I had three decommissioned servers gathering dust for more than a year, but I remembered they had lots of CPU power and memory. So I plugged them into a power socket, attached a screen and keyboard, and switched them on.

Now, two things happened. First, they started making enough noise to scare away entire bird populations in a 5 mile radius. Second, the attached screen remained as black as Batman’s darkest suite on a dim moonless night.

Always up for a challenge, I opened up the servers, and started removing all their innards. I soon had stacks of CPU’s, DIMM’s, fans, power supplies and hard disks all over the floor. And blood all over my hands and arms. I’m pretty sure that whoever designs computer cases must have a profound hate for humankind. Or own large amounts of stock in adhesive bandage manufacturing companies.

Anyway, I then started the tiresome process of swapping individual components in and out, rebooting each server after each swap. After several hours, and one server, two CPU’s and several DIMM’s retired to the scrap yard, I finally ended up with two working, albeit still extremely noisy servers. Well, “working” is actually somewhat too optimistic. So far I only managed to get an “Operating System not found” message on the screen.

Installing an O.S. isn’t too much of a hassle, and at least doesn’t come with any massive blood loss hazards. Copy your favorite Linux distro to an USB stick, configure RAID setup, partition your hard drives, select which services you want to install, and voila, within half an hour or so, job done.

Lastly, download Blender, copy all your source models, textures and scripts to your servers, start your render job, and watch the magic happening before your eyes. Or not. At least not within the first couple of hours. Rendering, and certainly Physically Based Rendering as I was using here, requires massive amounts of compute power. Or massive amounts of patience, if you lack the budget for massive amounts of compute power. But after a couple of hours, still not much had happened. The only visible result so far was that exotic animals had started nesting in my office, loving the tropical temperatures reached by running eight multi-core CPU’s non-stop at max power in a confined space. With the server fans now producing more noise than a wall of Marshall amps at an AC/DC concert, I opened all windows in the room, and retreated to the safety of the living room.

Not one easily distracted from the chosen path, I left the servers crunching numbers for several days, leaving the neighbors bewildered as to why, as they must have assumed, I was vacuuming the house 24/7 for days on end. But all good things must come to an end. Whilst my ego and I might today still be waiting for those rendered images to appear, months after I started the render job, after about ten days my wife urged me to pull the plug on my little experiment. It’s clear my wife has always been the more sensible one among the two of us.

Still fixated on finishing the job, I started looking for other ways to get my images rendered. I needed massive amounts of compute power, but only for a short period of time. Now, doesn’t that sound like a usage pattern right up the Cloud’s alley? I already had an AWS account, but never actually used it before, so this would be a perfect learning opportunity!

Looking at the AWS EC2 price list, I started comparing “Compute Optimized” instances when I suddenly noticed the availability of “GPU” instances. If you don’t know your CPU’s from your GPU’s, just think of a GPU as a CPU tuned for one specific purpose: massive parallel processing of computer graphics. Now, I could have configured an EC2 GPU instance from scratch, but sometimes it pays to be lazy: just visit the AWS Marketplace, and search for a readily available instance matching your requirements. I only needed to install Blender myself, and about half an hour later, my first image was happily rendering away on four high-performance NVIDIA GRID GPUs in the cloud. I created three extra instances, so all four images could be rendered simultaneously, each one on its own EC2 GPU instance. About three hours later, all four images were fully rendered. That’s right: three hours.

Now all of sudden this “Cloud” thing started to make sense to me. We could try to do the actual math. But my initial attempt at building my own render farm out of spare parts didn’t cost me anything as far as hardware goes. I did invest a lot of my free time though. Quality time with one’s family is priceless, as I’ve been told, and calculating with “priceless” as the hourly rate makes for overly complex math. So let’s skip the math entirely, and just look at some interesting “lessons learned”.

  • My initial DIY attempt was based on me being a cheapskate. I recycled existing infrastructure and skills, trying to build a “poor man’s” render farm. And although, in the end, the hardware was working, and my skills sufficiently adequate for the job at hand, the final result was abysmal. Would this not have been a pet project done in my free time, but a professional decision at work instead, such reliance on existing infrastructure and skills, would have made for “yet another project over time and over budget”.
  • I could have bought fit-for-the-job hardware. But these NVIDIA GRID GPU’s don’t come cheap. It would take a large bite out of the family budget. And I only need such rendering capacity once every blue moon, so most of the time those GPU’s would just sit there gathering dust. Again, this translates very well to a typical enterprise IT challenge. Before the advent of the Cloud, a company’s IT infrastructure was most likely scaled for the most intensive job it has to handle. Typically a monthly big-ass bill run or super-complex salary calculation. The other 99% of the time you probably only require between 20% to 50% of your peak IT capacity. That’s a lot of inefficiency. In Cloud terms, this is called “elasticity”. I just call it “money in the pocket”.
  • The hands-on IT skills I built up over the past decades didn’t suddenly become obsolete with the arrival of the Cloud. I still needed my Linux admin skills to get the EC2 instance to actually do anything. I still needed my analytical troubleshooting skills to find out why initially the thing insisted on ignoring the available GPU’s, instead using the CPU’s capacity to do the rendering computations. I still needed to have a profound understanding of security best practices to prevent others from abusing my AWS account. The only skill that became obsolete was my ability to unscrew computer hardware. But given the amount of bandages required, it’s safe to say that it never was a prime skill of mine anyway. And besides, should the urge arise, there’s plenty of other appliances in the house to savagely attack with a screwdriver. So, give your company’s System Administrators some peace of mind, they are as valuable as ever. Unless using a screwdriver is their prime skill.
  • When I first looked at the Cloud as an alternative to my DIY render farm, my initial thinking was to fire up some EC2 Linux instances and customize those for the task at hand. Whilst this approach makes sense, it would not have been the most optimal approach. If your company decides to start making use of the Cloud, make sure you don’t fall into that trap either. Don’t just migrate existing physical or virtual servers to the Cloud as-is. I predict you will actually end up spending more money than you do today, without much of a benefit. Do not think of the Cloud as “externalized virtualization infrastructure”. Instead, start splitting your IT landscape into small building blocks or services, and see which ones are readily available as part of your Cloud providers offering. For example, don’t set up a database server, use existing services like AWS RDS or Azure SQL Database instead. It is not unwise to seek expert guidance when navigating unfamiliar terrain.

Whilst actually playing around with the Cloud, I’ve found most of my fears, uncertainties and doubts to be unwarranted. Sure, the Cloud isn’t a “one-size-fits-all” solution, there’s plenty of scenarios where other strategies will make more sense. But I would certainly recommend you investigate for yourself what promises this new land might hold for you and your company. Get to learn what services are out there, what procurement strategies fit your requirements and taste, and do some Proof-of-Concepts to learn where the Cloud’s true value is. Happy exploring!

P.S. The careful reader might have spotted I wrote “most of my fears, uncertainties and doubts”. Indeed, that implies I certainly have some fears left. But that will be the subject of another post.


me About the author:

Hi, my name is Koen. I have been passionate about IT since I was a little kid writing silly programs on a Tandy TRS-80. In my professional career, so far I’ve worked as a Field Support Engineer, System Administrator, Webmaster, Developer, Solutions Architect, Project Manager, and currently I’m the IT Manager at Besides IT, I like creativity and innovation. Feel free to visit if you want to know more!