[openstack-dev] [tc][all][designate] The way forward for Designate

Fei Long Wang feilong at catalyst.net.nz
Tue Feb 21 21:56:17 UTC 2017

It would be nice to record this discussion/summary on an etherpad or
somewhere, because it's not just the problem for Designate, IMHO, many
other big tent projects are facing the same issue. Thanks.

On 22/02/17 09:25, Hayes, Graham wrote:
> For those that are interested, we will have an hour discussion about
> how designate can move forward tomorrow (wed 22nd) at the PTG.
> We will be in Savannah 3 @ 11:00 for an hour.
> Thanks,
> Graham
> On 09/02/2017 14:22, Hayes, Graham wrote:
>> The HTML version of this is here:
>> http://graham.hayes.ie/posts/openstack-designate-where-we-are/
>> I have been asked a few times recently "What is the state of the Designate
>> project?", "How is Designate getting on?", and by people who know what is
>> happening "What are you going to do about Designate?".
>> Needless to say, all of this is depressing to me, and the people that I have
>> worked with for the last number of years to make Designate a truly useful,
>> feature rich project.
>> *TL;DR;* for this - Designate is not in a sustainable place.
>> To start out - Designate has always been a small project. DNS does not have
>> massive *cool* appeal - its not shiny, pretty, or something you see on the
>> front page of HackerNews (unless it breaks - then oh boy do people
>> become DNS
>> experts).
>> A line a previous PTL for the project used to use, and I have happily
>> robbed is
>> "DNS is like plumbing, no one cares about it until it breaks, and then
>> you are
>> standing knee deep in $expletive". (As an aside, that was the reason we
>> chose
>> the crocodile as our mascot - its basically a dinosaur, old as dirt, and
>> when
>> it bites it causes some serious complications).
>> Unfortunately that comes over into the development of DNS products
>> sometimes.
>> DNSaaS is a check box on a tender response, an assumption.
>> We were lucky in the beginning - we had 2 large(ish) public clouds that
>> needed
>> DNS services, and nothing currently existed in the eco-system, so we got
>> funding for a team from a few sources.
>> We got a ton done in that period - we moved from a v1 API which was
>> synchronous to a new v2 async API, we massively increased the amount of DNS
>> servers we supported, and added new features.
>> Unfortunately, this didn't last. Internal priorities within companies
>> sponsoring the development changed, and we started to shed contributors,
>> which
>> happens, however disappointing. Usually when this happens if a project is
>> important enough the community will pick up where the previous group
>> left off.
>> We have yet to see many (meaningful) commits from the community though. We
>> have some great deployers who will file bugs, and if they can put up patch
>> sets - but they are (incredibly valuable and appreciated) tactical
>> contributions. A project cannot survive on them, and we are no exception.
>> So where does that leave us? Let have a look at how many actual commits we
>> have had:
>> Commits per cycle
>>      +--------------+---------+
>>      | Havana       | 172     |
>>      +--------------+---------+
>>      | Icehouse     | 165     |
>>      +--------------+---------+
>>      | Juno         | 254     |
>>      +--------------+---------+
>>      | Kilo         | 340     |
>>      +--------------+---------+
>>      | Liberty      | 327     |
>>      +--------------+---------+
>>      | Mitaka       | 246     |
>>      +--------------+---------+
>>      | Newton       | 299     |
>>      +--------------+---------+
>>      | Ocata        | 98      |
>>      +--------------+---------+
>> Next cycle, we are going to have 2 community goals:
>> * Control Plane API endpoints deployment via WSGI
>> * Python 3.5 functional testing
>> We would have been actually OK for the tempest one - we were one of the
>> first
>> external repo based plug-ins with `designate-tempest-plugin`_
>> For WSGI based APIs, this will be a chunk of work - due to our internal code
>> structure splitting out the API is going to be ... an issue. (and I think it
>> will be harder than most people expect - anyone using olso.service has
>> eventlet imported - I am not sure how that affects running in a WSGI server)
>> Python 3.5 - I have no idea. We can't even run all our unit tests on python
>> 3.5, so I suspect getting functional testing may be an issue. And,
>> convincing
>> management that re-factoring parts of the code base due to "community goals"
>> or a future potential pay-off can be more difficult than it should.
>>   http://graham.hayes.ie/images/oct-2016-projects-prod.jpg
>> We now have a situation where the largest "non-core" project [1]_ in the
>> tent
>> has a tiny number of developers working on it. 42% of deployers are
>> evaluating
>> Designate, so we should see this start to increase.
>> How did this happen?
>> ====================
>> Like most situations, there is no single cause.
>> Certainly there may have been fault on the side of the Designate leadership.
>> We had started out as a small team, and had built a huge amount of trust and
>> respect based on in person interactions over a few years, which meant that
>> there was a fair bit of "tribal knowledge" in the heads of a few people, and
>> that new people had a hard time becoming part of the group.
>> Also, due to volume of work done by this small group, a lot of users /
>> distros
>> were OK leaving us work - some of us were also running a production
>> designate
>> service during this time, so we knew what we needed to develop, and we had
>> pretty quick feedback when we made a mistake, or caused a bug. All of this
>> resulted in the major development cost being funded by two companies, which
>> left us vulnerable to changes in direction from those companies. Then that
>> shoe dropped. We are now one corporate change of direction from having no
>> cores on the project being paid to work on the project. [2]_
>> Preceding this, the governance of OpenStack changed to the `Big Tent`_
>> While this change was a good thing for the OpenStack project as a
>> whole it had quite a bad impact on us.
>> Pre Big Tent, you got integrated. This was at least a cycle, where you moved
>> docs to docs.openstack.org, integrated with QA testing tooling, got packaged
>> by Linux distros, and build cross project features.
>> When this was a selective thing, there was teams available to help with
>> that,
>> docs teams would help with content (and tooling - docs was a mass of XML
>> back
>> then), QA would help with tempest and devstack, horizon would help with
>> panels.
>> In Big Tent, there just wasn't resources to do this - the scope of the
>> project
>> expansion was huge. However the big tent happened (in my opinion - I have
>> written about this before) before the horizontal / cross project teams were
>> ready. They stuck to covering the "integrated" projects, which was all
>> they could do at the time.
>> This left us in a position of having to reimplement tooling, figure out
>> what tooling we did have access to, and migrate everything we had on our
>> own. And, as a project that (at our peak level of contribution) only
>> ever had
>> 5% of the number of contributors compared to a project like nova,  this put
>> quite a load on our developers. Things like grenade, tempest and horizon
>> plug-ins, took weeks to figure out all of which took time from other vital
>> things like docs, functional tests and getting designate into other tools.
>> One of the companies who invested in designate had a QE engineer that
>> used to
>> contribute, and I can honestly say that the quality of our testing
>> improved 10
>> fold during the time he worked with us. Not just from in repo tests, but
>> from
>> standing up full deployment stacks, and trying to break them - we
>> learned a lot
>> about how we could improve things from his expertise.
>> Which is kind of the point I think. Nobody is amazing at everything. You
>> need
>> people with domain knowledge to work on these areas. If you asked me to do a
>> multi-node grenade job, I would either start drinking, throw my laptop
>> at you
>> or do both.
>> We still have some of these problems to this day - most of our docs are
>> in a messy pile in https://docs.openstack.org/developer/designate
>> while we still have a small amount of old functional tests that are not
>> ported
>> from our old non plug-in style.
>> All of this adds up to make projects like Designate much less attractive
>> to users - we just need to look at the `project navigator`_ to see what
>> a bad
>> image potential users get of us. [3]_ This is for a project that was ran
>> as a
>> full (non beta) service in a public cloud. [4]_
>> Where too now then?
>> ===================
>> Well, this is where I call out to people who actually use the project -
>> don't
>> jump ship and use something else because of the picture I have painted.
>> We are
>> a dedicated team, how cares about the project. We just need some help.
>> I know there are large telcos who use Designate. I am sure there is tooling,
>> or docs build up in these companies that could be very useful to the
>> project.
>> Nearly every commercial OpenStack distro has Designate. Some have had it
>> since
>> the beginning. Again, developers, docs, tooling, testers, anything and
>> everything is welcome. We don't need a massive amount of resources - we
>> are a
>> small ish, stable, project.
>> We need developers with upstream time allocated, and the budget to go to
>> events
>> like the PTG - for cross project work, and internal designate road map,
>> these
>> events form the core of how we work.
>> We also need help from cross project teams - the work done by them is
>> brilliant
>> but it can be hard for smaller projects to consume. We have had a lot of
>> progress since the `Leveller Playing Field`_ debate, but a lot of work is
>> still optimised for the larger teams who get direct support, or well
>> resourced
>> teams who can dedicate people to the implementation of plugins / code.
>> As someone I was talking to recently said - AWS is not winning public cloud
>> because of commodity compute (that does help - a lot), but because of the
>> added services that make using the cloud, well, cloud like. OpenStack
>> needs to
>> decide that either it is just compute, or if it wants the eco-system. [5]_
>> Designate is far from alone in this.
>> I am happy to talk to anyone about helping to fill in the needed resources -
>> Designate is a project that started in the very office I am writing this
>> blog
>> post in, and something I want to last.
>> For a visual this is Designate team in Atlanta, just before we got
>> incubated.
>> http://graham.hayes.ie/images/ATL.jpg
>> and this was our last mid cycle:
>> http://graham.hayes.ie/images/mid-cycle.jpg
>> and in Atlanta at the PTG, there will be two of us.
>> [1] In the `Oct-2016`_ User Survey Designate was deployed in 23% of clouds
>> [2] I have been lucky to have a management chain that is OK with me
>> spending some time on Designate, and have not asked me to take time off
>> for Summits or Gatherings, but my day job is working on a completely
>> different project.
>> [3] I do have other issues with the metrics - mainly that we existed
>> before becoming leaving stackforge, and some of the other stats are set
>> so high, that non "core" projects will probably never meet them.
>> [4] I recently went to an internal training talk, where they were
>> talking about new features in Newton. There was a whole slide about how
>> projects and improved, or gotten worse on these scores. A whole slide.
>> With tables of scores, and I think there may have even been a graph.
>> [5] Now, I am slightly biased, but I would argue that DNS is needed in
>> commodity compute, but again, that is my view.
>> .. _Oct-2016: https://www.openstack.org/analytics
>> .. _Big Tent:
>> https://governance.openstack.org/tc/resolutions/20141202-project-structure-reform-spec.html
>> .. _designate-tempest-plugin:
>> https://github.com/openstack/designate-tempest-plugin
>> .. _project navigator:
>> https://www.openstack.org/software/releases/newton/components/designate
>> .. _Leveller Playing Field:
>> http://graham.hayes.ie/posts/openstack-a-leveler-playing-field/
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Cheers & Best regards,
Feilong Wang (王飞龙)
Senior Cloud Software Engineer
Tel: +64-48032246
Email: flwang at catalyst.net.nz
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington

More information about the OpenStack-dev mailing list