Open Stack

Darragh Bailey <daragh.bailey at gmail.com> writes:

> On 13 December 2016 at 00:25, James E. Blair <corvus at inaugust.com> wrote:
>
>> Hi,
>>
>> I'm happy to report that we have made significant progress on adding
>> ZooKeeper support to Nodepool, in preparation for supporting Zuul v3.
>>
> I keep meaning to read up on what's planned for zuul v3, the recorded talk
> from the last summit was good in explaining some of the benefits for users
> of the new capabilities, but I'd also love to get up to speed on how some
> of the backend changes will improve things for operation. Is there some
> blog posts or something that goes into this?

The specs are probably the best source for that (they contain some
text about problem statements and solutions):

http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html
http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html

>> This branch contains an update to the nodepool-builder process which
>> uses ZooKeeper rather than MySQL and gearman for coordinating image
>> builds and storing information about them (the main nodepool process
>> still uses MySQL and gearman -- updating it to use ZooKeeper is the next
>> section of work).
>
> Will any of that break those still using zuul/nodepool with Jenkins? Or is
> that orthogonal? I still don't have a full handle on how
> Jenkins/Zuul/Nodepool interacts besides that it uses gearman.

Monty has a proposal (mentioned in IRC, but not otherwise written down
yet, I believe) to implement a shim between the zuulv3 style of Nodepool
allocation and zuulv2.  Basically, we would be able to run the new
zuulv3 style Nodepool daemon, and the shim would talk to zuulv2, make
requests of nodepool, then allocate nodes to zuulv2.  Strictly speaking
for OpenStack infra, all we need is for the shim to support the zuulv2.5
node allocation, but I think that it would probably not be difficult to
support Jenkins allocation as well.

Long story short: the next phase of work is designed to break backwards
compatibility with zuulv2 and Jenkins, however, with this shim I hope
that we will be able to use the new nodepool with both Zuul v2.5 as well
as Zuul v2.0 and Jenkins.

I think that would be great as it will enable everyone to perform the
stepwise transition to Zuul v3 we're doing for OpenStack infra: first
nodepool-builders, followed by nodepool-launchers, then finally zuul.

Note that the current phase of work, just completed, does remove support
for snapshot builds.  Our recommendation is to use diskimage builder.  I
think that if an installation doesn't need custom images, and can simply
use existing provider images, that should be easy to add to Nodepool,
but it has not been done yet.  If folks really want support for snapshot
builds, it would be possible to create a builder for them using the new
framework, but we would need a volunteer to add that support.

>> Tomorrow, I also plan on merging the zuulv3 branch into master.  If you
>> are interested in becoming an early adopter of this, that would be a
>> great time to start using it.  We don't have much documentation in
>> Nodepool itself about this, but the short version is: install and run
>> ZooKeeper (a one node cluster is fine) and point Nodepool at it.  There
>> is already (optional) support for it in puppet-openstackci and
>> puppet-nodepool.  You can take a look at the OpenStack infra operational
>> documentation here:
>>
>>   http://docs.openstack.org/infra/system-config/nodepool.html
>>
>> Once we tidy the master branch up a bit, we will release a new version,
>> 0.4.0 with the ZooKeeper based builder.  At that point, it would be a
>> good idea for anyone running Nodepool to begin using it.
>>
>> Thanks,
>>
>> Jim
>>
> Looking forward to checking it out, and hopefully getting involved around
> zuulv3 integration with GitHub!

Thanks, that would be great.  And you'll be in good company.  :)

-Jim