“There are no mistakes, just happy accidents” – Bob Ross
I don’t know why I am writing this. Honestly, I am ignoring a chunk of code to write this right now, that I really shouldn’t. But, I guess reading some tweets by old NASA co-workers has me reminiscing about past adventures. I guess I can’t really not take a moment to look back at OpenStack like everyone else. Hopefully, however, this blog post won’t end up on wired and it will be something you happen onto by accident and enjoy. A “happy accident” if you will.
Regarding the Culture of OpenStack
“Something isn’t real, until it’s been around for five years” – Bre Pettis
I don’t know if Bre remembers saying this. But around one of the first NYC Resistor meetings, Bre said NYC Resistor wasn’t real until it was five years old. The reality is with small groups, five years will gut your community. People will have children, move on to bigger and better things, or simply disappear. Resistor is 5 now. I’ve moved on to California, along with about six or seven other former resistor members. Several more have moved elsewhere. Many simply drifted away though still in NYC. But, Resistor is still strong. There has been churn in membership, and today it is as lively and successful as ever.
OpenStack is now three years old. It, unlike Resistor, has exploded as a community. It has also undergone radical changes across the board. But, it’s culture has been an evolution that’s been guided by some very good values.
“The culture of OpenStack was defined by drinking and unit test coverage” – Josh McKenty
A quote from Josh McKenty concerning the early days of openstack, really does drive to the heart of why OpenStack is successful. Josh said recently in a Register article “The culture of OpenStack was defined by drinking and unit test coverage“. Funny, I know, and not entirely without a grain of truth. But it’s important to understand that from day one, testing became a necessary fact of life. So much changed so quickly from day to day, that comprehensive testing was literally the only way to keep the development environment usable by developers. That and making the environment completely ephemeral. The focus on testing bled into OpenStack, when the foundation formed. Contributions by folks like Monty Taylor, and other members of the Infrastructure team, have put OpenStack in the shadow of it’s supporting Continuous Integration infrastructure at times. The CI team for OpenStack has built an undisputed master example of CI working amazingly well. Our CI is the envy of every development shop on earth, and if your shop doesn’t envy it, you should. That’s indicative of a culture that’s focused on culture of development first and foremost. A great deal of OpenStack’s success follows from that.
“It’s a means of managing the fire hose that is OpenStack development” – Monty Taylor
In my own experience at Cloudscaling, there was never a focus on developing the Culture of the development team. It was a free for all in which developers strove for their own greatest level of success. There was guidance in the product development, but there wasn’t a focus on building a process that promoted a functional collaboration between developers. Ultimately, this lead to balkanization of the development team into sub groups. This did not reflect the code architecure, and is more indicative of a pervading problem in Technology companies that also affected Cloudscaling.
The reality is, you don’t need the best and brightest developers to build the worlds best code. You need the best community. Looking back to NYC Resistor and comparing it to it’s west coast sister space Noisebridge, you can see two hackerspaces with radically different design patterns. Where Resistor focused on building a community of hackers, and made the space an after thought, Noisebridge focused on building an infrastructure that would be free for all to make use of. In this focus on infrastructure, rather than community, there was a belief that meritocracy would ultimately lead the space to success on it’s very own. This has not entirely been the case. I consider Resistor to be an unmitigated success, where noisebridge is an unmitigated disaster. And some will disagree, to them I say only this… Resistor has never had a problem with mole men. Monty Taylor alluded to the need for structure and rules in an article he did for Wired. He’s right, but it’s not the rules, so much as the structure that is created organically by the OpenStack community and the values that pervade it. The execution works, but that’s because the culuture promotes successful execution of ideas. Lots of unit tests, lots of quantified reasoning, lots of drinking =P.
“We want to give developers as much freedom as we can, but if you give them too much freedom, it turns into anarchy. You have to have a certain amount of structure and rules.” – Monty Taylor
Recently, there’s been some discussion in the news regarding the bellicose nature of the Linux Kernel community. As we get compared to Linux regularly these days, it’s kind of interesting to compare the two communities. Both are successful, Linux more so than OpenStack by a large margin. Clocking in at over 20 years old, Linux is one of the all time great open source projects, if not the acme of Open Source success. However, OpenStack is not executing their development processes in a way that was intended to follow on the heels of their success.
OpenStack is entirely open in it’s development. If you’ve got a signed contributor license agreement, and an ssh key, you can register to submit code and within hours your accounts for gerrit will be enabled. From there it’s just a matter of getting a couple of core reviewers to look at your patch and approve it for merge. Compared, to say, Debian’s apprenticeship based community, this is remarkably more democratic. Linux is similar, however their existing methodology is dated by comparison, and highly custom having been built years prior to community standardized methodology for unit and functional testing. They did not have the benefit of jenkins ( hudson ) when they first began building their community. Ultimately both are very open in terms of allowing anyone to submit code. And both require existing and recognized experts to approve merger of code.
“There aren’t enough swear-words in the English language, so now I’ll
have to call you perkeleen vittupää just to express my disgust and
frustration with this crap.” – Linus Torvalds
Where the biggest difference exists, to my mind, is in the leadership. Linus is the defacto dictator for life of the project. The Che of Linux if you will. Even the project is named after him, like some form of digital Stalingrad. Unlike Stalin, Linus is actually a pretty nice guy when he’s not calling you an idiot. Mikko Hyponen explains this as being a very finnish, and crudely named, cultural dogma. Ot’s called Management by Perkele, and you don’t have to be Finnish to have experienced it. Just ask any Marine. It IS a management style. And it can be effective in specific scenarios.
OpenStack on the other hand promotes an open and accepting culture. We’re conscious of our failures, and we move to rectify them. And while we do occasionally have our diplomatic break downs, honestly developers occasionally lose it. But, we do try to keep people civil. Unlike say, herr dictator for life and middle finger to nvidia lord of all code he surveys. We also elect our leaders. The TC which is responsible for engineering related efforts of the foundation is elected by active contributors. Active contributors being defined as folks who have committed and merged code to OpenStack official projects sometime in the past two releases. That means, our leadership is ephemeral, and scales horizontally, just like OpenStack.
Consider the drinking part of OpenStack. It’s not about the booze. It’s about the community being able to have a friendly conversation even after hours of back and forth over competing blueprints. Even after vicious diatribes are invected upon competing ideas. Ideas that have code invested in them, sometimes business models. Ideas that can make or break careers. The ability for our community to stomach each other extends from a basic respect that pervades the culture of OpenStack. And it’s one of those qualities I hope to see even stronger at year five.
Cloud and the Food Services Industry
This section is part of the reason I am wasting time writing this. I was looking at my own hosting bills the other day. The money I pay to keep a node up in a datacenter for my personal use. And truthfully, I drop about 400 a year on that. Figure I drop another 30 a month on a data plan for my phone. And I drop another 100 or so a month on a home connection. My data related service costs are almost 2 grand a year. That’s a ton. So, as far as my wallet is concerned, data services are very good at separating me from my money.
But, more importantly, they share some structural similarities. Stop laughing, I am serious. You go into any restaurant or pub, and you have a service area. The stuff you can see. The aestehtic, the interfaces, the user experience. Then you have the product deliverables, in this case food. And then you have the add ons, beverages and deserts. But, this stuff is generally assembled in a kitchen. And, much like a cloud, you generally don’t have any idea what’s going on in that kitchen. If the food is bad, you don’t come back. If you get sick, you don’t come back. But, that doesn’t stop a provider from risking the safety of their customers, or providing a terrible experience more often than not.
“And that’s the deciding factor. We can’t win against obsession. They care, we don’t. They win.” – Ford Prefect
It kind of makes you think, was Opscode thinking of this exact analogy when they named Chef and Knife? More importantly, when Slartibartfast explained bistronomics, whas he explaining the Cloud economy? Probably best discussed over the latest batch of homebrew.
In the US, we have now, this grading system for restaurants. They post in their window a letter grade that’s been earned during their health inspections. Many restaurants are quite proud of their A ratings. In the the land of cloud, we have a couple of competing rating systems. Stuff like NIST, and FEDRamp spring to mind. And I am sure there’s a great more to follow. We even have some general government oversight via FISMA, HIPPA, and Sarbanes Oxley. We’ve got our own set of financial auditors who are the poorly paid beaurocratic health inspectors of well.. fiduciary oversight.
I used to live in Brazil. There, they have churrascarias. We’re starting to get them in the US now, so maybe you’ve been to one. It’s basically a Brazillian steakhouse, and it’s an experience. But, Brazil more than even the US, is a very corrupt country. Health Inspectors weren’t really doing much to curb bad practices in kitchens at these establishments, and when you deal in a lot of meat, you can easily begin dealing in death. So, the government needed as way to ensure these places were doing their due diligence in the kitchen. Being Brazil, they took the seemingly insane and radical approach of passing a law saying that any patron can inspect their kitchen at any time. Sounds nuts right? Could you imagine tom startup demanding that Amazon let him inspect their backend infrastructure? That’d totally go over well. But, for the churrascarias, some of them at least, it became a point of pride. Their kitchens were no longer closed off behind doors, they were on display. They were over engineered, spotless, and they were as much as part of the user experience as they were a means of ensuring the integrity of the food handling.
OpenStack is that open kitchen. And deployers, will eventually come to see the benefit of making sure that everyone sees their OpenStack core, and all it’s supporting code, is on display. A mark of pride in their engineering, and a promise to you the customer that their wares are delivered as safely as possible with the best ingredients possible.
As I said in a prior piece, Open Source isn’t Free Software. What it is, is showing pride in your work. It’s a promise to your users, that you’ve built a quality product.
More importantly, and probably coming up in a blog post for secstack, this sort of open display case model for back end cloud design can help promote a cloud with verifiable integrity. And that’s worth it’s weight in gold. Especially in a market where your biggest competitors are terrified of you seeing their kitchens.
Now a moment of personal repose. The past three years of my life have been a roller coaster. Time at Nebula was tumultuous. It was at times incredible, and exciting. And other times devestatingly depressing, and throughly humbling. In general Open Source is like this, but maybe a little less so. At least that’s how it was for me. When I came out to work on Nebula, I didn’t expect that I’d ever be in California working for anyone but NASA. But, budgets were cut, and truthfully, I wanted to follow OpenStack out into the world. I’ve been in California three years, and I am pretty much sick of it. I have nothing against California. It’s a wonderful state, with some great cities. My favorites are San Diego, and San Francisco. But, I miss my family. I miss my hackerspace. I miss having the drive to be creative. And I think, in part the reason for that drive falling away, as of late, is because I have become a bit depressed about where I am in life. And it’s at times like that, that you look around and try to see what else you have in your life outside work, and the reality is I don’t have much in California. I got to work for NASA, and be part of an incredibly successful Open Source project. One that is compared today to Linux no less. And I was there on the ground floor, in the lab when it was being first developed. I still want to take OpenStack out into the world and show it breaching barriers and changing the way people think about how we use processorts all over again. But, I want to do that while pursuing a life that I want for myself outside of that. I want to be happy again when I do turn my head from the screen. I want to leave California behind.
I don’t know if I will return to NYC, or if I will pursue another adventure that’s on the list of things to do before I die. But, OpenStack has taught me that sitting around waiting for things to get better isn’t a good enough answer. You need to take your destiny into your own hands, and make the changes you want to see made. To see the better world you believe in, you have to stand up an take the risks.
No matter what, it’s been a great adventure. And I’m better for having been on that adventure.