<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Thu, Feb 9, 2017 at 7:22 PM Hayes, Graham <<a href="mailto:graham.hayes@hpe.com">graham.hayes@hpe.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The HTML version of this is here:<br class="gmail_msg">
<a href="http://graham.hayes.ie/posts/openstack-designate-where-we-are/" rel="noreferrer" class="gmail_msg" target="_blank">http://graham.hayes.ie/posts/openstack-designate-where-we-are/</a><br class="gmail_msg">
<br class="gmail_msg">
I have been asked a few times recently "What is the state of the Designate<br class="gmail_msg">
project?", "How is Designate getting on?", and by people who know what is<br class="gmail_msg">
happening "What are you going to do about Designate?".<br class="gmail_msg">
<br class="gmail_msg">
Needless to say, all of this is depressing to me, and the people that I have<br class="gmail_msg">
worked with for the last number of years to make Designate a truly useful,<br class="gmail_msg">
feature rich project.<br class="gmail_msg">
<br class="gmail_msg">
*TL;DR;* for this - Designate is not in a sustainable place.<br class="gmail_msg">
<br class="gmail_msg">
To start out - Designate has always been a small project. DNS does not have<br class="gmail_msg">
massive *cool* appeal - its not shiny, pretty, or something you see on the<br class="gmail_msg">
front page of HackerNews (unless it breaks - then oh boy do people<br class="gmail_msg">
become DNS<br class="gmail_msg">
experts).<br class="gmail_msg">
<br class="gmail_msg">
A line a previous PTL for the project used to use, and I have happily<br class="gmail_msg">
robbed is<br class="gmail_msg">
"DNS is like plumbing, no one cares about it until it breaks, and then<br class="gmail_msg">
you are<br class="gmail_msg">
standing knee deep in $expletive". (As an aside, that was the reason we<br class="gmail_msg">
chose<br class="gmail_msg">
the crocodile as our mascot - its basically a dinosaur, old as dirt, and<br class="gmail_msg">
when<br class="gmail_msg">
it bites it causes some serious complications).<br class="gmail_msg">
<br class="gmail_msg">
Unfortunately that comes over into the development of DNS products<br class="gmail_msg">
sometimes.<br class="gmail_msg">
DNSaaS is a check box on a tender response, an assumption.<br class="gmail_msg">
<br class="gmail_msg">
We were lucky in the beginning - we had 2 large(ish) public clouds that<br class="gmail_msg">
needed<br class="gmail_msg">
DNS services, and nothing currently existed in the eco-system, so we got<br class="gmail_msg">
funding for a team from a few sources.<br class="gmail_msg">
<br class="gmail_msg">
We got a ton done in that period - we moved from a v1 API which was<br class="gmail_msg">
synchronous to a new v2 async API, we massively increased the amount of DNS<br class="gmail_msg">
servers we supported, and added new features.<br class="gmail_msg">
<br class="gmail_msg">
Unfortunately, this didn't last. Internal priorities within companies<br class="gmail_msg">
sponsoring the development changed, and we started to shed contributors,<br class="gmail_msg">
which<br class="gmail_msg">
happens, however disappointing. Usually when this happens if a project is<br class="gmail_msg">
important enough the community will pick up where the previous group<br class="gmail_msg">
left off.<br class="gmail_msg">
<br class="gmail_msg">
We have yet to see many (meaningful) commits from the community though. We<br class="gmail_msg">
have some great deployers who will file bugs, and if they can put up patch<br class="gmail_msg">
sets - but they are (incredibly valuable and appreciated) tactical<br class="gmail_msg">
contributions. A project cannot survive on them, and we are no exception.<br class="gmail_msg">
<br class="gmail_msg">
So where does that leave us? Let have a look at how many actual commits we<br class="gmail_msg">
have had:<br class="gmail_msg">
<br class="gmail_msg">
Commits per cycle<br class="gmail_msg">
<br class="gmail_msg">
+--------------+---------+<br class="gmail_msg">
| Havana | 172 |<br class="gmail_msg">
+--------------+---------+<br class="gmail_msg">
| Icehouse | 165 |<br class="gmail_msg">
+--------------+---------+<br class="gmail_msg">
| Juno | 254 |<br class="gmail_msg">
+--------------+---------+<br class="gmail_msg">
| Kilo | 340 |<br class="gmail_msg">
+--------------+---------+<br class="gmail_msg">
| Liberty | 327 |<br class="gmail_msg">
+--------------+---------+<br class="gmail_msg">
| Mitaka | 246 |<br class="gmail_msg">
+--------------+---------+<br class="gmail_msg">
| Newton | 299 |<br class="gmail_msg">
+--------------+---------+<br class="gmail_msg">
| Ocata | 98 |<br class="gmail_msg">
+--------------+---------+<br class="gmail_msg">
<br class="gmail_msg">
Next cycle, we are going to have 2 community goals:<br class="gmail_msg">
<br class="gmail_msg">
* Control Plane API endpoints deployment via WSGI<br class="gmail_msg">
* Python 3.5 functional testing<br class="gmail_msg">
<br class="gmail_msg">
We would have been actually OK for the tempest one - we were one of the<br class="gmail_msg">
first<br class="gmail_msg">
external repo based plug-ins with `designate-tempest-plugin`_<br class="gmail_msg">
<br class="gmail_msg">
For WSGI based APIs, this will be a chunk of work - due to our internal code<br class="gmail_msg">
structure splitting out the API is going to be ... an issue. (and I think it<br class="gmail_msg">
will be harder than most people expect - anyone using olso.service has<br class="gmail_msg">
eventlet imported - I am not sure how that affects running in a WSGI server)<br class="gmail_msg">
<br class="gmail_msg">
Python 3.5 - I have no idea. We can't even run all our unit tests on python<br class="gmail_msg">
3.5, so I suspect getting functional testing may be an issue. And,<br class="gmail_msg">
convincing<br class="gmail_msg">
management that re-factoring parts of the code base due to "community goals"<br class="gmail_msg">
or a future potential pay-off can be more difficult than it should.<br class="gmail_msg">
<br class="gmail_msg">
<a href="http://graham.hayes.ie/images/oct-2016-projects-prod.jpg" rel="noreferrer" class="gmail_msg" target="_blank">http://graham.hayes.ie/images/oct-2016-projects-prod.jpg</a><br class="gmail_msg">
<br class="gmail_msg">
We now have a situation where the largest "non-core" project [1]_ in the<br class="gmail_msg">
tent<br class="gmail_msg">
has a tiny number of developers working on it. 42% of deployers are<br class="gmail_msg">
evaluating<br class="gmail_msg">
Designate, so we should see this start to increase.<br class="gmail_msg">
<br class="gmail_msg">
How did this happen?<br class="gmail_msg">
====================<br class="gmail_msg">
<br class="gmail_msg">
Like most situations, there is no single cause.<br class="gmail_msg">
<br class="gmail_msg">
Certainly there may have been fault on the side of the Designate leadership.<br class="gmail_msg">
We had started out as a small team, and had built a huge amount of trust and<br class="gmail_msg">
respect based on in person interactions over a few years, which meant that<br class="gmail_msg">
there was a fair bit of "tribal knowledge" in the heads of a few people, and<br class="gmail_msg">
that new people had a hard time becoming part of the group.<br class="gmail_msg">
<br class="gmail_msg">
Also, due to volume of work done by this small group, a lot of users /<br class="gmail_msg">
distros<br class="gmail_msg">
were OK leaving us work - some of us were also running a production<br class="gmail_msg">
designate<br class="gmail_msg">
service during this time, so we knew what we needed to develop, and we had<br class="gmail_msg">
pretty quick feedback when we made a mistake, or caused a bug. All of this<br class="gmail_msg">
resulted in the major development cost being funded by two companies, which<br class="gmail_msg">
left us vulnerable to changes in direction from those companies. Then that<br class="gmail_msg">
shoe dropped. We are now one corporate change of direction from having no<br class="gmail_msg">
cores on the project being paid to work on the project. [2]_<br class="gmail_msg">
<br class="gmail_msg">
Preceding this, the governance of OpenStack changed to the `Big Tent`_<br class="gmail_msg">
While this change was a good thing for the OpenStack project as a<br class="gmail_msg">
whole it had quite a bad impact on us.<br class="gmail_msg">
<br class="gmail_msg">
Pre Big Tent, you got integrated. This was at least a cycle, where you moved<br class="gmail_msg">
docs to <a href="http://docs.openstack.org" rel="noreferrer" class="gmail_msg" target="_blank">docs.openstack.org</a>, integrated with QA testing tooling, got packaged<br class="gmail_msg">
by Linux distros, and build cross project features.<br class="gmail_msg">
<br class="gmail_msg">
When this was a selective thing, there was teams available to help with<br class="gmail_msg">
that,<br class="gmail_msg">
docs teams would help with content (and tooling - docs was a mass of XML<br class="gmail_msg">
back<br class="gmail_msg">
then), QA would help with tempest and devstack, horizon would help with<br class="gmail_msg">
panels.<br class="gmail_msg">
<br class="gmail_msg">
In Big Tent, there just wasn't resources to do this - the scope of the<br class="gmail_msg">
project<br class="gmail_msg">
expansion was huge. However the big tent happened (in my opinion - I have<br class="gmail_msg">
written about this before) before the horizontal / cross project teams were<br class="gmail_msg">
ready. They stuck to covering the "integrated" projects, which was all<br class="gmail_msg">
they could do at the time.<br class="gmail_msg">
<br class="gmail_msg">
This left us in a position of having to reimplement tooling, figure out<br class="gmail_msg">
what tooling we did have access to, and migrate everything we had on our<br class="gmail_msg">
own. And, as a project that (at our peak level of contribution) only<br class="gmail_msg">
ever had<br class="gmail_msg">
5% of the number of contributors compared to a project like nova, this put<br class="gmail_msg">
quite a load on our developers. Things like grenade, tempest and horizon<br class="gmail_msg">
plug-ins, took weeks to figure out all of which took time from other vital<br class="gmail_msg">
things like docs, functional tests and getting designate into other tools.<br class="gmail_msg">
<br class="gmail_msg">
One of the companies who invested in designate had a QE engineer that<br class="gmail_msg">
used to<br class="gmail_msg">
contribute, and I can honestly say that the quality of our testing<br class="gmail_msg">
improved 10<br class="gmail_msg">
fold during the time he worked with us. Not just from in repo tests, but<br class="gmail_msg">
from<br class="gmail_msg">
standing up full deployment stacks, and trying to break them - we<br class="gmail_msg">
learned a lot<br class="gmail_msg">
about how we could improve things from his expertise.<br class="gmail_msg">
<br class="gmail_msg">
Which is kind of the point I think. Nobody is amazing at everything. You<br class="gmail_msg">
need<br class="gmail_msg">
people with domain knowledge to work on these areas. If you asked me to do a<br class="gmail_msg">
multi-node grenade job, I would either start drinking, throw my laptop<br class="gmail_msg">
at you<br class="gmail_msg">
or do both.<br class="gmail_msg">
<br class="gmail_msg">
We still have some of these problems to this day - most of our docs are<br class="gmail_msg">
in a messy pile in <a href="https://docs.openstack.org/developer/designate" rel="noreferrer" class="gmail_msg" target="_blank">https://docs.openstack.org/developer/designate</a><br class="gmail_msg">
while we still have a small amount of old functional tests that are not<br class="gmail_msg">
ported<br class="gmail_msg">
from our old non plug-in style.<br class="gmail_msg">
<br class="gmail_msg">
All of this adds up to make projects like Designate much less attractive<br class="gmail_msg">
to users - we just need to look at the `project navigator`_ to see what<br class="gmail_msg">
a bad<br class="gmail_msg">
image potential users get of us. [3]_ This is for a project that was ran<br class="gmail_msg">
as a<br class="gmail_msg">
full (non beta) service in a public cloud. [4]_<br class="gmail_msg">
<br class="gmail_msg">
Where too now then?<br class="gmail_msg">
===================<br class="gmail_msg">
<br class="gmail_msg">
Well, this is where I call out to people who actually use the project -<br class="gmail_msg">
don't<br class="gmail_msg">
jump ship and use something else because of the picture I have painted.<br class="gmail_msg">
We are<br class="gmail_msg">
a dedicated team, how cares about the project. We just need some help.<br class="gmail_msg">
<br class="gmail_msg">
I know there are large telcos who use Designate. I am sure there is tooling,<br class="gmail_msg">
or docs build up in these companies that could be very useful to the<br class="gmail_msg">
project.<br class="gmail_msg">
<br class="gmail_msg">
Nearly every commercial OpenStack distro has Designate. Some have had it<br class="gmail_msg">
since<br class="gmail_msg">
the beginning. Again, developers, docs, tooling, testers, anything and<br class="gmail_msg">
everything is welcome. We don't need a massive amount of resources - we<br class="gmail_msg">
are a<br class="gmail_msg">
small ish, stable, project.<br class="gmail_msg">
<br class="gmail_msg">
We need developers with upstream time allocated, and the budget to go to<br class="gmail_msg">
events<br class="gmail_msg">
like the PTG - for cross project work, and internal designate road map,<br class="gmail_msg">
these<br class="gmail_msg">
events form the core of how we work.<br class="gmail_msg">
<br class="gmail_msg">
We also need help from cross project teams - the work done by them is<br class="gmail_msg">
brilliant<br class="gmail_msg">
but it can be hard for smaller projects to consume. We have had a lot of<br class="gmail_msg">
progress since the `Leveller Playing Field`_ debate, but a lot of work is<br class="gmail_msg">
still optimised for the larger teams who get direct support, or well<br class="gmail_msg">
resourced<br class="gmail_msg">
teams who can dedicate people to the implementation of plugins / code.<br class="gmail_msg"></blockquote><div><br></div><div>The QA team is committed to serve every single project in big tent community.</div><div>Please reach out to us on IRC in #openstack-qa or add an item to the QA meeting agenda.</div><div>We are a rather small team ourselves, so we won't be able to implement things on project</div><div>team's behalf, but we'll do our best to help.</div><div><br></div><div>andrea</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="gmail_msg">
As someone I was talking to recently said - AWS is not winning public cloud<br class="gmail_msg">
because of commodity compute (that does help - a lot), but because of the<br class="gmail_msg">
added services that make using the cloud, well, cloud like. OpenStack<br class="gmail_msg">
needs to<br class="gmail_msg">
decide that either it is just compute, or if it wants the eco-system. [5]_<br class="gmail_msg">
Designate is far from alone in this.<br class="gmail_msg">
<br class="gmail_msg">
I am happy to talk to anyone about helping to fill in the needed resources -<br class="gmail_msg">
Designate is a project that started in the very office I am writing this<br class="gmail_msg">
blog<br class="gmail_msg">
post in, and something I want to last.<br class="gmail_msg">
<br class="gmail_msg">
For a visual this is Designate team in Atlanta, just before we got<br class="gmail_msg">
incubated.<br class="gmail_msg">
<br class="gmail_msg">
<a href="http://graham.hayes.ie/images/ATL.jpg" rel="noreferrer" class="gmail_msg" target="_blank">http://graham.hayes.ie/images/ATL.jpg</a><br class="gmail_msg">
<br class="gmail_msg">
and this was our last mid cycle:<br class="gmail_msg">
<br class="gmail_msg">
<a href="http://graham.hayes.ie/images/mid-cycle.jpg" rel="noreferrer" class="gmail_msg" target="_blank">http://graham.hayes.ie/images/mid-cycle.jpg</a><br class="gmail_msg">
<br class="gmail_msg">
and in Atlanta at the PTG, there will be two of us.<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
[1] In the `Oct-2016`_ User Survey Designate was deployed in 23% of clouds<br class="gmail_msg">
[2] I have been lucky to have a management chain that is OK with me<br class="gmail_msg">
spending some time on Designate, and have not asked me to take time off<br class="gmail_msg">
for Summits or Gatherings, but my day job is working on a completely<br class="gmail_msg">
different project.<br class="gmail_msg">
[3] I do have other issues with the metrics - mainly that we existed<br class="gmail_msg">
before becoming leaving stackforge, and some of the other stats are set<br class="gmail_msg">
so high, that non "core" projects will probably never meet them.<br class="gmail_msg">
[4] I recently went to an internal training talk, where they were<br class="gmail_msg">
talking about new features in Newton. There was a whole slide about how<br class="gmail_msg">
projects and improved, or gotten worse on these scores. A whole slide.<br class="gmail_msg">
With tables of scores, and I think there may have even been a graph.<br class="gmail_msg">
[5] Now, I am slightly biased, but I would argue that DNS is needed in<br class="gmail_msg">
commodity compute, but again, that is my view.<br class="gmail_msg">
<br class="gmail_msg">
.. _Oct-2016: <a href="https://www.openstack.org/analytics" rel="noreferrer" class="gmail_msg" target="_blank">https://www.openstack.org/analytics</a><br class="gmail_msg">
.. _Big Tent:<br class="gmail_msg">
<a href="https://governance.openstack.org/tc/resolutions/20141202-project-structure-reform-spec.html" rel="noreferrer" class="gmail_msg" target="_blank">https://governance.openstack.org/tc/resolutions/20141202-project-structure-reform-spec.html</a><br class="gmail_msg">
.. _designate-tempest-plugin:<br class="gmail_msg">
<a href="https://github.com/openstack/designate-tempest-plugin" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/openstack/designate-tempest-plugin</a><br class="gmail_msg">
.. _project navigator:<br class="gmail_msg">
<a href="https://www.openstack.org/software/releases/newton/components/designate" rel="noreferrer" class="gmail_msg" target="_blank">https://www.openstack.org/software/releases/newton/components/designate</a><br class="gmail_msg">
.. _Leveller Playing Field:<br class="gmail_msg">
<a href="http://graham.hayes.ie/posts/openstack-a-leveler-playing-field/" rel="noreferrer" class="gmail_msg" target="_blank">http://graham.hayes.ie/posts/openstack-a-leveler-playing-field/</a><br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
</blockquote></div></div>