<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" dir="auto" class=""><div class="">The HTML version is here:</div><div class=""><a href="https://s.cassiba.com/2017/02/14/making-the-kitchen-great-again-a-retrospective-on-openstack-chef" class="">https://s.cassiba.com/2017/02/14/making-the-kitchen-great-again-a-retrospective-on-openstack-chef</a></div><div class=""><br class=""></div><div class="">This was influenced by Graham Hayes' State of the Project for Designate:</div><div class=""><a href="http://graham.hayes.ie/posts/openstack-designate-where-we-are/" class="">http://graham.hayes.ie/posts/openstack-designate-where-we-are/</a></div><div class=""><br class=""></div><div class="">I have been asked recently "what is going on with the OpenStack-Chef project?",</div><div class="">"how is the state of the cookbooks?", and "hey sc, how are those integration</div><div class="">tests coming?". Having been the PTL for the Newton and Ocata cycles, yet</div><div class="">having not shipped a release, is the unthinkable, and deserves at least a</div><div class="">sentence or two.</div><div class=""><br class=""></div><div class="">It goes without saying, this is disheartening and depressing to me and</div><div class="">everybody that has devoted their time to making the cookbooks a solid</div><div class="">and viable method for deploying OpenStack. OpenStack-Chef is among the</div><div class="">oldest[1] and most mature solutions for deploying OpenStack, though it is</div><div class="">not the most feature-rich.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">*TL;DR* if you don't want to keep going -</div><div class="">OpenStack-Chef is not in a good place and is not sustainable.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">OpenStack-Chef has always been a small project with a big responsibility.</div><div class="">The Chef approach to OpenStack historically has required a level of</div><div class="">investment within the Chef ecosystem, which is a hard enough sell when you</div><div class="">started out with Puppet or Ansible. Despite the unicorns and rainbows of</div><div class="">being Chef cookbooks, OpenStack-Chef always asserted itself as an OpenStack</div><div class="">project first, up to and including joining the Big Tent, whatever it takes.</div><div class="">To beat that drum, we are OpenStack.</div><div class=""><br class=""></div><div class="">There is no *cool* factor from deploying and managing OpenStack using Chef,</div><div class="">unless you've been running Chef, because insert Xzibit meme here and jokes</div><div class="">about turtles. Unless you break something with automation, then it's</div><div class="">applause or facepalm. Usually both. At the same time.</div><div class=""><br class=""></div><div class="">As with any kitchen, it must be stocked and well maintained, and</div><div class="">OpenStack-Chef is no exception. Starting out, there was a vibrant community</div><div class="">producing organic, free-range code. Automation is invisible, assumed to be</div><div class="">there in the background. Once it's in place, it isn't touched again unless</div><div class="">it breaks. Upgrades in complex deployments can be fraught with error, even</div><div class="">in an automated fashion.</div><div class=""><br class=""></div><div class="">As has been seen in previous surveys[2], once an OpenStack release has chosen</div><div class="">by an operator, some tend to not upgrade for the next cycle or three, to get</div><div class="">the immediate bugs worked out. Though there are now multinode and upgrade</div><div class="">scenarios supported with the Puppet OpenStack and TripleO projects, they do</div><div class="">not use Chef, so Chef deployers do not directly benefit from any of this</div><div class="">testing.</div><div class=""><br class=""></div><div class="">Being a deployment project, we are responsible for not one aspect of</div><div class="">the OpenStack project but as many as can be reasonably supported.</div><div class=""><br class=""></div><div class="">We were very fortunate in the beginning, having support from public cloud</div><div class="">providers, as well as large private cloud providers. Stackalytics shows a</div><div class="">vibrant history, a veritable who's-who of OpenStack contributors, too many to</div><div class="">name. They've all moved on, working on other things.</div><div class=""><br class=""></div><div class="">As a previous PTL for the project once joked, the Chef approach to OpenStack</div><div class="">was the "other deployment tool that nobody uses". As time has gone by, that has</div><div class="">become more of a true statement.</div><div class=""><br class=""></div><div class="">There are a few of us still cooking away, creating new recipes and cookbooks. The</div><div class="">pilot lights are still lit and there's usually something simmering away on the</div><div class="">back burner, but there is no shouting of orders, and not every dish gets tasted.</div><div class="">We think there might be rats, too, but we’re too shorthanded to maintain the traps.</div><div class=""><br class=""></div><div class="">We have yet to see many (meaningful) contributions from the community, however.</div><div class="">We have some amazing deployers that file bugs, and if they can, push up a patch.</div><div class="">It delights me when someone other than a core weighs in on a review. They are</div><div class="">highly appreciated and incredibly valuable, but they are very tactical</div><div class="">contributions. A project cannot live on such contributions.</div><div class=""><br class=""></div><div class="">October 2015</div><div class=""><br class=""></div><div class=""> <a href="https://s.cassiba.com/images/oct-2015-deployment-decisions.png" class="">https://s.cassiba.com/images/oct-2015-deployment-decisions.png</a></div><div class=""><br class=""></div><div class="">Where does that leave OpenStack-Chef? Let's take a look at the numbers:</div><div class=""><br class=""></div><div class=""><div class=""> +------------+------------+</div><div class=""> | Cycle | Commits |</div><div class=""> +------------+------------+</div><div class=""> | Havana | 557 |</div><div class=""> +------------+------------+</div><div class=""> | Icehouse | 692 |</div><div class=""> +------------+------------+</div><div class=""> | Juno | 424 |</div><div class=""> +------------+------------+</div><div class=""> | Kilo | 474 |</div><div class=""> +------------+------------+</div><div class=""> | Liberty | 259 |</div><div class=""> +------------+------------+</div><div class=""> | Mitaka | 85 |</div><div class=""> +------------+------------+</div><div class=""> | Newton | 112 |</div><div class=""> +------------+------------+</div><div class=""> | Ocata | 78 |</div><div class=""> +------------+------------+</div></div><div class=""><br class=""></div><div class="">As of the time of this writing, Newton has not yet branched. Yes, you read</div><div class="">correctly. This means the Ocata cycle has gone to ensuring that Newton *just</div><div class="">functions*. In a virtual quasi-vacuum, without input from larger scale</div><div class="">deployments, who are running releases older than Newton, reporting bugs we've</div><div class="">fixed in master. Supporting Newton required implementing support for Ubuntu</div><div class="">16.04, as well as client and underlying cookbook changes, due to deprecations</div><div class="">that started prior to Newton. Here is the output from *berks viz* for a top-down</div><div class="">view into the complexity on just the Chef side.</div><div class=""><br class=""></div><div class=""> <a href="https://s.cassiba.com/images/openstack-chef-dependency-graph.png" class="">https://s.cassiba.com/images/openstack-chef-dependency-graph.png</a></div><div class=""><br class=""></div><div class="">For the Pike cycle, Jan Klare will be reprising the role of PTL. I do not</div><div class="">intend to speak for him, but there are few paths forward in the Big Tent:</div><div class=""><br class=""></div><div class="">* Branching stable/newton and stable/ocata with the quickness.</div><div class="">* Improve OpenStack CI to the point of being able to trust it again for</div><div class=""> testing patches, as well as extend testing scenarios (including multinode).</div><div class=""><br class=""></div><div class="">For branching stable/newton, the external CI has been proving useful in overall</div><div class="">confidence in cutting a release. We're way behind schedule, but nearly there. I</div><div class="">have begun working on implementing some basic multinode gates, as our allinone</div><div class="">no longer fits within the confines of the 8GB instances. But, it’s Chef, so</div><div class="">triangle wheels, yo. Some of the cross-project efforts translate to Chef, but</div><div class="">not all. With square spinners.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">So... how did this happen?</div><div class="">-----------------------------------</div><div class=""><br class=""></div><div class="">As was in the case of Designate, as is in the case of OpenStack-Chef. There is</div><div class="">no one single reason or cause that arrived us at this point.</div><div class=""><br class=""></div><div class="">The main catalyst was internal support shifting, which impacted the sponsored</div><div class="">developers and contributors. OpenStack-Chef became less and less a priority,</div><div class="">and one by one they shifted to other focuses. At the Austin 2016 Summit, we said</div><div class="">farewell to all but the PTL and one core. This put OpenStack-Chef in a bad place</div><div class="">given its mission and scope, but onward we go.</div><div class=""><br class=""></div><div class="">Due to the volume of work done by this small group and the lack of feedback</div><div class="">during development, it became more and more difficult to tell when a release</div><div class="">could be considered "done". We could no longer trust our CI framework, as the</div><div class="">developers with intrinsic knowledge had been refocused, with little more than</div><div class="">commit history to go on.</div><div class=""><br class=""></div><div class="">Users were okay with leaving us work, which we added to the heap. This, with</div><div class="">the departure in contributors, resulted in the majority of the development</div><div class="">being funded by just two companies, which left the project at risk to changes</div><div class="">in direction by those companies. Without regular feedback or guidance beyond</div><div class="">release notes and the occasional chat in another project's channel, the focus</div><div class="">shifted away from features to just ship it, as long as it passes allinone and/or</div><div class="">multinode locally, if there’s time. Does it pass lint/unit/style? Fuck it. Ship</div><div class="">it, deal with the fallout. Yeah. This is bad on *so many* levels.</div><div class=""><br class=""></div><div class="">The Big Tent really did not do as much as advertised for OpenStack-Chef, as harsh</div><div class="">of an opinion as that sounds. Larger, more well-funded projects have since created</div><div class="">processes, frameworks and test suites that were developed for their own use cases,</div><div class="">not necessarily taking into account Chef's own blend of automation. That left</div><div class="">us having to go and discover how to make fire on our own to make the cookbooks</div><div class="">work on each supported platform and release of OpenStack. In the Big Tent, we</div><div class="">were effectively left to our own devices. Just another OpenStack project. We</div><div class="">numbered nine cores when we moved from StackForge to the Big Tent. Developer</div><div class="">peak, though we did not know it yet.</div><div class=""><br class=""></div><div class="">Initially, the cookbooks had a very heavy dependency: Chef Server. If not Chef</div><div class="">Server, Chef Solo, which still had its own quirks, and nobody liked Chef Solo</div><div class="">anyway. Not even Chef Solo liked itself. During the Juno cycle, we switched to</div><div class="">the Chef Development Kit, which gave us chef-provisioning. This decreased</div><div class="">turnaround time for testing patches being submitted, and boosted confidence all</div><div class="">around. Until Juno, it was difficult to run functional tests against the</div><div class="">cookbooks. That's when we discovered how to create fire. We could run</div><div class="">OpenStack! In virtual machines! On our laptops! OMyG you guys! Suddenly,</div><div class="">OpenStack on the laptop became easy, push button, single command, automated. We</div><div class="">could test a patch without a long spin-up. With that, came integration gates,</div><div class="">and periodic jobs. From days to minutes. We were cooking with gas! But... let’s</div><div class="">not make those integration gates voting... *yet*.</div><div class=""><br class=""></div><div class="">Mitaka brought a significant overhaul and simplification, with the introduction</div><div class="">of a multinode chef-provisioning recipe and more modular cookbooks. The pieces</div><div class="">finally existed, but the damage had been done, and unfortunately, this momentum</div><div class="">did not last. Internal priorities changed within companies sponsoring</div><div class="">developers, many of which could not be fully committed in the first place, and</div><div class="">we started shedding contributors, which happens. At this point is usually where</div><div class="">*someone* comes in to either play Grim Reaper or lifesaver. By Austin, our</div><div class="">numbers waned until just two cores remained, Jan and myself. We could not be in</div><div class="">the worst of locations to communicate, he in Germany and I in California.</div><div class=""><br class=""></div><div class="">In the Newton and Ocata cycles, development progressed in a lurching capacity</div><div class="">without a team. Due to the overhaul in Mitaka, patches slowed in frequency from</div><div class="">the outside community, many of who continued deploying and running on older,</div><div class="">EOL branches, or got frustrated enough to switch to other automation flavors.</div><div class="">The remaining team had little overlapping time to communicate, being on</div><div class="">different continents in conflicting time zones. What was difficult to do with a</div><div class="">larger team spread across three continents and five time zones became</div><div class="">impossible with just two. Day jobs increasingly took priority over</div><div class="">OpenStack-Chef care and feeding, and some cookbooks started to go rancid</div><div class="">(sorry, Ironic, Sahara, Swift and Trove. nobody was able to support a</div><div class="">deployment with you). Interaction within the development team was limited to an</div><div class="">hour or two a day, eventually down to once or twice a week if we had time. Day</div><div class="">jobs proceeded to consume the development team, with sporadic development as</div><div class="">the months ticked on.</div><div class=""><br class=""></div><div class="">Over the Newton cycle, one cookbook was offered, EC2, with inadequate coverage</div><div class="">for our support matrix. The most desired integration API. In the end, it was</div><div class="">not integrated due to time and commitment to support such a feature, having</div><div class="">inadequate resources at our disposal. During Ocata, the project had one</div><div class="">cookbook contributed from the community, Murano, that could be integrated, and</div><div class="">grew an appendage in the form of the client cookbook. It is the closest anyone</div><div class="">has gotten to new features since the Mitaka cycle. We added one core reviewer</div><div class="">during this time.</div><div class=""><br class=""></div><div class="">Communication is a big part of any project, particularly a geographically</div><div class="">diverse effort like OpenStack. Prior to the Big Tent, we held weekly meetings</div><div class="">using Hangouts, which were open and publicized for the mailing list subscribers</div><div class="">and channel denizens. Upon joining the Big Tent, we gave up the regular</div><div class="">face-time in favor of text-based IRC meetings, per governance. Without the high</div><div class="">bandwidth requirement of a video call once a week, one by one, cores had day</div><div class="">job meetings do what they do, and take priority. In the Newton cycle, we</div><div class="">relinquished our weekly time slot after it became apparent that neither of us</div><div class="">could make the meetings. We have not held a scheduled meeting since then, as it</div><div class="">is next to impossible to carve out adequate overlapping time.</div><div class=""><br class=""></div><div class="">We still have many of these problems to this day. Documentation is a mess or</div><div class="">nonexistent. Despite the flexibility of the tooling, users have but two</div><div class="">representative deployment examples: allinone or a rudimentary multinode.</div><div class="">OpenStack-Chef gained modularity at the expense of features, and there was an</div><div class="">overwhelming non-reaction to the deprecation of those features.</div><div class=""><br class=""></div><div class="">All of this results in a project that is not very friendly to new users, and</div><div class="">Chef does not look as attractive as a deployment option as other, more</div><div class="">feature-rich flavors. This has real business decisions behind it. One only need</div><div class="">look at the steady usage decline in the surveys to see how negatively things</div><div class="">appear to existing and new users. This, for a project that has roots in the</div><div class="">very cubicles OpenStack was born.[1]</div><div class=""><br class=""></div><div class="">But it's not all bad</div><div class="">--------------------</div><div class=""><br class=""></div><div class="">For all the negativity, this story has upsides. I call to the people who</div><div class="">actually use this software in their deployments, in whatever shape it's in, to</div><div class="">not abandon OpenStack-Chef, or retire it to bitrot. We need help, not funeral</div><div class="">arrangements. Share your pain, so that we may find a way forward together, not</div><div class="">alone.</div><div class=""><br class=""></div><div class="">October 2016</div><div class=""><br class=""></div><div class=""> <a href="https://s.cassiba.com/images/oct-2016-deployment-decisions.png" class="">https://s.cassiba.com/images/oct-2016-deployment-decisions.png</a></div><div class=""><br class=""></div><div class="">In my time with the project, I’ve gotten to know that there are some pretty big</div><div class="">names that leverage Chef in their deployments, and some of them even use it for</div><div class="">OpenStack. Some cookbook forks also exist, all serving to solve the same</div><div class="">problems that face OpenStack-Chef. Without feedback from real-world</div><div class="">deployments, OpenStack-Chef will continue to wither on the vine. This</div><div class="">fragmentation harms more than it solves.</div><div class=""><br class=""></div><div class="">What do we need? It’s easier to list what we don’t need. Developers, tooling,</div><div class="">documentation, testing, any and all are welcome and greatly appreciated. We</div><div class="">don’t need much, but what we are able to do is limited by our size and time</div><div class="">available.</div><div class=""><br class=""></div><div class="">We need developers with time and funding budgeted for contributing within the</div><div class="">OpenStack ecosystem. We need better representation at events such as the PTG.</div><div class="">For the first PTG, no OpenStack-Chef members will be attending, though we</div><div class="">intend to meet up in Boston, maybe. Given our physical locations in proximity</div><div class="">to Atlanta, it made sense to stay home and communicate over IRC/code review. It</div><div class="">doesn’t mean our development has ceased, we’re just too few and far away for it</div><div class="">to make sense.</div><div class=""><br class=""></div><div class="">We also need help from cross-project teams to understand and assimilate the</div><div class="">work done to solve the problems that we are working to solve. We are all</div><div class="">working toward the same goal, though with a different dialect and coat of arms.</div><div class=""><br class=""></div><div class="">OpenStack needs choice in *how* to OpenStack. Limiting to a few dialects of</div><div class="">automation makes things look less like an ecosystem and more like a distro,</div><div class="">which is fine, if that’s how people want to go about it. We can talk about that,</div><div class="">too.</div><div class=""><br class=""></div><div class="">I am happy to talk to anyone about how they can help. OpenStack-Chef has roots</div><div class="">in the pioneers of OpenStack, and, in my (not so humble) opinion, is way too</div><div class="">nifty to just let fall to the wayside.</div><div class=""><br class=""></div><div class="">I do not have team visuals to represent how the team has grown and shrank over</div><div class="">the years, so let me leave you with some mental visuals. At the end of Mitaka,</div><div class="">we numbered nine. At the start of Ocata, we numbered just two. In Boston, we</div><div class="">anticipate there will be, hopefully, all three of us, representing the sixteen</div><div class="">active subprojects that consist OpenStack-Chef.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">[1]: [openstack-compute::default commit history](<a href="https://github.com/openstack/cookbook-openstack-compute/commits/eol-grizzly/recipes/default.rb" class="">https://github.com/openstack/cookbook-openstack-compute/commits/eol-grizzly/recipes/default.rb</a>)</div><div class="">[2]: [April 2016 OpenStack User Survey: greater than 50% of production deployments were still on releases circa Kilo or older](<a href="https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf" class="">https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf</a>)</div><div class="">[3]: [OpenStack Survey Report](<a href="https://www.openstack.org/analytics" class="">https://www.openstack.org/analytics</a>)</div></div></body></html>