[openstack-dev] [all][infra] Zuul v3 rollout, the sequel

Gary Kotton gkotton at vmware.com
Tue Oct 10 12:05:05 UTC 2017


Hi,
At the moment the neutron, neutron-lib and many of the decomposed projects are still failing with the v3. Does this mean that we are broken from the 11th?
For the decomposed projects we have a work around to help address this in the short term – need to increase timeout and need a flag from Zuul3 that is not part of Jenkins - https://github.com/openstack/vmware-nsx/blob/master/tools/tox_install_project.sh#L37 (can we have ZUUL3_CLONER?)
Thanks
Gary

On 10/10/17, 2:59 AM, "Jeremy Stanley" <fungi at yuggoth.org> wrote:

    The tl;dr is that we're planning to roll forward out of our partial
    Zuul v3 rollback starting at 11:00 UTC on Wednesday October 11
    (a little over 35 hours from now), so expect some CI downtime and
    all of the benefits (though hopefully none of the drawbacks!) you
    witnessed when we tried the first time.
    
    It's been right at a week since we instituted a partial rollback of
    our initial v3 roll-out. That week has been filled by diagnosing and
    fixing all of the misbehaviors and performance degradation we
    identified, including some new issues we discovered while running
    under even heavier load and memory pressure artificially induced by
    trying to fire check jobs for everything v2 was running but with
    only a fraction of the node capacity. Numerous issues within the
    translated legacy job configs were also fixed, and a bunch more of
    them replaced by v3-native jobs.
    
    We anticipate an outage for the CI system of somewhere between 30-60
    minutes starting at 11:00 UTC on Wednesday October 11. Once
    complete, we'll send a follow-up announcement along with a link to
    where we're coordinating and triaging any newly observed issues. In
    the meantime, if you've been digging into recent legacy job failures
    for your projects you should consider trying to bring them to our
    attention as soon as possible (in #openstack-infra on the Freenode
    IRC network, or replying on-list to this announcement) and work on
    fixing them if you're familiar enough with the failures to do so
    yourself. We're assembling a sort of FAQ in the migration guide
    here:
    
        https://docs.openstack.org/infra/manual/zuulv3.html
    
    ...and we also have some more content in progress at:
    
        https://etherpad.openstack.org/p/zuulv3-migration-faq
    
    In summary, Zuul v3 is looking better than ever, and we hope you'll
    be as pleased with it as we are!
    -- 
    Jeremy Stanley
    



More information about the OpenStack-dev mailing list