My father is a structural engineer, my brother installs and configures industrial fire alarms, and I studied computer engineering in college. It suffices to say, I have had some exposure to engineering in various forms. Today however, I walk the razor’s edge between operations and development. Something folks in big iron houses might call systems engineering. But you web 2.0 cloud kids would call it DevOps, because let’s be honest… you’ve never heard of ITIL. And that’s okay as per my previous post.
But I think what’s interesting from the operations aspect of IT, is that we deal with real engineers more than most of the other regions. We get involved in datacenter planning. We know how the local 3 goes about doing its job, and we know when they will be very angry with us for going about doing our job. We also know that when you walk into a real datacenter ( not yuri’s hosting ran out of a loft in williamsburg ) what you see is order among complexity. And the reason you see that, is because of the magic word… engineering.
Let’s for the moment consider a house being built. Many houses are built off fundamentally similar design concepts. When an architecture firm builds a new home they usually build based off of known generic designs have been built before. Then they customize with designs that have been built before. And all the while they adhere to building planning codes, electrical codes, civil engineering requirements, and a world of other bureaucratic mess that in many cases exists solely to rob you of your cash and allow some skell to not have to get a real job. But, what those codes and standards also do is they fit into a framework that’s been established by engineers, or general contractors of varying skill levels, as well as unions and their disciplined approach to their craft. When a civil engineer goes to be certified as a licensed engineer they have to learn a hell of a lot more than the formulas for beam tension and computing viscosity of a fluid. They need to be well versed in those codified rules that everyone in that industry adheres to when constructing buildings that people will enter and trust their lives to not falling down on top of them or giving way beneath them.
Now, let’s pretend that there is a virtual city. Filled with these buildings. Some of them are residential cookie cutter designs. Some of them are waste treatment, or power plants, and while they are composed of standard technology sets they see a lot of customization for efficiency and meeting the local environmental / city ordinance needs. And then there’s other stuff like bridges, and stadiums that require a little bit more of the science in engineering to meet the requirements than most structures. All the while, attention has to be given to traffic flow, fire prevention, and natural hazards. Now when building out this virtual city it’s alright to accidentally collapse a building here and there on your virtual people, because heck they are virtual. But the goal is to not do that as much as possible. It’s not quite the same level of “risk” that you see in actual civil engineering at all. So the game changes fundamentally. And that’s a hell of a lot like what we do in software engineering. Sometimes you are designing an efficient loading dock for your trucks to channel datasets across your motorways to awaiting web shops, or maybe you are filtering packets for nasty stuff at a packet flow treatment plant, or maybe you are just building a park for virtual kittens and their fans. But a lot of the overall approach to the engineering in software is not all that dissimilar from what we see in other areas of engineering. And that’s not surprising. The way in which we as a species abstract complex sets to divide them into more manageable pieces is pretty much always going to be somewhat the same since it’s focused around what we can handle, and not what we’re trying to handle. We know where limitations are on our end of the equation so we begin laying out our problem with that side of the equation set up the way we want and then we focus on making the other side equate to what we need.
Now, Joel on software as well as Terse systems have some great blog posts about the job market for software engineers, and the engineers themselves.
- Explainging the tech job market – Terse systems
- Everyong thinks they are hiring the top 1% – Joel on software
The simple fact is when you look at programmers across the entire IT sector, you’ll see a few things. One, there are a lot of programmers. I mean a hell of a lot. Millions in fact. When you consider the digital divide that currently exists in the world, and how low the penetration has been of automation in so many industries the sheer volume of programmers is somewhat staggering. Whole industries are unable to benefit from the hardware that exists today simply because the world cannot field enough programmers to even begin to penetrate those markets effectively. And this is why any competent programmer is never without a job for long, though often they may despise the work.
In operations and in information security you get to see entirely too much of the really bad mistakes that programmers make. It’s usually very unfair the sort of opinions that security and operations teams have of developers. When you spend every day cleaning up after some problem child code project that’s long since been orphaned and fallen into your operations team lap, or spend day after day explaining usually to the same programmers over and over that they need to use SSL on their authentication connections you can begin to develop some seriously fringe rants about the industry at large. I’ve certainly participated in a few of those. Hell ask me my opinion of Java and you’ll discover my bigotry towards the language and its proponents occasionally make even me blush. This is more because of bad experiences than anything else. And that’s unfair to the technology and the people who support it.
Now you might be wondering why I am saying this. The terse systems link for instance discussed how there are in their opinion developers who actually accomplish the amazing task of doing negative work. And while I’ve certainly met a few who would certainly meet and exceed that qualification, the vast majority of developers aren’t incompetent. They are actually usually very competent. But, we’re still seeing a hell of a lot of waste in the software engineering industry. If you’ve ever worked in a large enterprise you’ve probably seen some of the worst of it. I’ve seen the same software written, rewritten, auditted, scrapped, rewritten, modified, unmodified, rewritten, finally put into staging, rewritten, pushed to production and maintained for 3 arduous years until it’s EOLed for some other project following the exact same line of development and doing the exact same thing just as bad. The problem in this isn’t the developers in my opinion. It’s that there is no actual software engineering being done at all. What you have is programmers running around like chickens with their heads cut off doing whatever the hell damned fool thing that makes sense to them at the time.
Now there’s probably a couple of programmers who already clicked this tab shut, but there’s probably a couple still reading along counting prepositional phrases and mounting a serious array of ammunition to use in your invective commentary. Ask yourself what did you think of the code you wrote three years ago when you viewed it recently. Odds are no matter how talented you are, you were horrified. In fact, odds are the more talented you are the more horrified you were. Now, the reason I see for this sort of thing happening is fairly simple. Developers aren’t engineering their software.
When you look at a building being built, the vast majority of the people involved in the project aren’t calculating stress points, and figuring out where to place central support columns. They are digging through their bolt collection to find that standard bolt for attaching a 2 x 4 to a 1 x 2. They don’t think about all the math that went into deciding why that 2 x 4 and that 1 x 2 need to be joined the way the building code requires. They just follow the damned rules. And they get trained to do that. From the day they start the job to the day they teach it to someone else all they are focused on is following the standards to the best of their ability. And because of that they can spot faults in design, and send a notification up the chain to the architects and on site engineers who can make a design change to work around the issue. And those guys doing that, while competent as hell don’t need to be Donald Knuth to do their job. And as a result everyone involved in what is a massively complex task can focus on a simplified set of things that they can easily handle. And at the end what’s built is modular, stable, and fairly easy to audit for problems.
Now go look at a major software project. You have a team of developers, some of them are older wiser and more capable. Some of them entry level and down right unsafe. And you’ve got Yuri in the corner who works part time at your company because his hosting company really hasn’t taken off yet, and as long as you keep him away from the vodka he’s got stashed in the locked routing cabinet his code is downright beautiful. Much like contractors developers are a colorful group of people who take issue with anyone that suggests they aren’t damned good at what they do. And much like contractors most of them aren’t doing something that hasn’t been done before. Sure some of it is unique. But laying out a form in a gui or a website is always calling the same methods, and handling that data requires the same scrubbing. Running an authentication query against an auth source will always use the same calls and require the same SSL usage. So on and so forth. And in a really good software engineering shop, developers will be reusing modular code. Sort of ad hoc developing their own standards that will become an intellectual asset of the company, and one more hook into their poor customers.
Imagine now if you will that your developers know what they have to do when they come to building a form, because they were taught it when they apprenticed under an experienced developer, or learned it when they went for their software engineering degree / license at school. Imagine if they know what the security reqs are off the top of their head when they come to authenticating off an ldap or AD source. Imagine if they know when setting up an RPC server that data should be in JSON. Imagine if every developer was being taught the same approaches from job site to job site. Imagine a world in which all software followed some set of codeified standards.
Go back to your virtual city. And rebuild it now. See how much easier it is.
Of course you are saying, well okay that’s both horrible and great. You eliminate all the innovation from the field and do so by relying on something that is about as likely to happen as unicorns hunting down lions in the Serengeti. Before I make my proposal for how, I suggest that innovation never dies. There will always be kitten parks for people to innovate in, labs, libraries, and contests for people to show off their awesome design approaches and get better at making them. What I am suggesting is that we don’t use Citibanks trading systems as Yuri’s personal algorithm proving grounds. In fact, today most flight control systems are written in ADA for the express purpose of ensuring that programmers stick to doing things in a very strict fashion. Where there is risk, there is incentive to do things in a highly sane fashion.
Now, how do I bring about this magic world of sane standards. The answer is the same as it is in Civil Engineering. It starts with universities. Colleges and universities need to stop pushing programmers into ridiculous Computer Science majors. Software engineering is NOT computer science. Hell computer science isn’t even a real term. It doesn’t mean anything. Since most comp sci is taught as a math dept course it should be called digital algorithms and be the end of it. And I don’t think that should go away. I just feel it would be better as a math dept graduate course. Software engineers should be taught at the engineering school by engineers. And the professors and phd students should be focused on defining the standards that are ultimately codified on their scholarly merit by state and federal regulation. And that regulation should apply those codes and standards only where it is necessary. Now obviously that won’t be entirely perfect. But I think the net result would be massively positive for everyone involved.
And I say all of this as a guy who never even finished his bachelors degree, and to be honest hates the entire structure of academia. The reason is, because this really is the right way to be doing this. I also say this as a damned near libertarian in most things. To quote the lawyer I work with “This is Matt Joyce, I am pretty sure he’s an anarchist.” Again, I do this for a living, and this is the RIGHT way to do this. This is how it is done in every other engineering industry. And this is how it should be done in software as well. And if you are looking at the track record of regulation in the software industry and are horrified by what MIGHT happen? Consider this, establishing standards and regulations of this nature will pretty much force reform in software patents.
I’d love to hear from others in the industry what they think about this. So please send comments or post follow on blog posts on your own blog or just send a note saying “great idea, will never happen but it would be nice.” I just want to know I’m not overlooking some obvious horror here.