Open Stack

On Fri, 2019-03-15 at 16:21 -0500, Ghanshyam Mann wrote:
> ---- On Thu, 14 Mar 2019 23:37:46 -0500 Takashi Yamamoto <yamamoto at midokura.com> wrote ----
>  > On Wed, Mar 13, 2019 at 2:25 PM Ghanshyam Mann <gmann at ghanshyammann.com> wrote: 
>  > > 
>  > > ---- On Mon, 11 Mar 2019 22:04:55 -0500 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ---- 
>  > >  >  ---- On Mon, 11 Mar 2019 01:03:51 -0500 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ---- 
>  > >  > Updates: 
>  > >  > 
>  > >  > I pushed few of the remaining projects testing patch today, below is the latest status. 
>  > >  > Request you to check your project testing patch and take appropriate action. March 13th 
>  > >  > is deadline to merge the base legacy jobs migration patch. 
>  > > [...] 
>  > > 
>  > > Updates: 
>  > > 
>  > > I checked the project which are passing their gate on Bionic and which are failed (have not checked the n-v or
> experimental jobs) 
>  > > - https://review.openstack.org/#/q/topic:legacy-job-bionic+(status:open+OR+status:merged) 
>  > > 
>  > > *FAILING PROJECTS: NEED ACTION* 
>  > > nova  - testing with tls-proxy disable - https://review.openstack.org/#/c/639017/ 
>  > > cinder - https://bugs.launchpad.net/cinder/+bug/1819819 
>  > > designate -  https://review.openstack.org/#/c/642319/ 
>  > > trove -https://review.openstack.org/#/c/642329/ 
>  > > zaqar - https://review.openstack.org/#/c/642349/ 
>  > > networking-midonet - https://review.openstack.org/#/c/639990/ 
>  >  
>  > is there an option to keep xenial jobs? 
>  > midonet (the backend networking-midonet talks to) doesn't support bionic yet. 
>  > https://midonet.atlassian.net/browse/MNA-1344 
> 
> We really want to avoid that. Main idea if doing this migration at community level
> is that all the OpenStack projects which are integrated with each other at some level
> can move and be tested on latest Distro. So that they do not break each other. 
> 
> In your case, neutron is integrated projects which is or going to be running/tested on
> Bionic and leaving networking-midonet on xenial can leads to integrated issues at some time. 
> As I have proposed the patch [1], we move the job to Bionic and make it
> n-v (which you already did) so that we finish this migration activity and you can work on 
> the failure.
> 
> As a summary, we want to avoid running anything on xenial in Stein.
im not that familar with midonet but perhaps if it does not have bionic support those jobs could be swapped
over to centos 7? i would personally avoid the fedora image but devstack and centos 7 are much happier then
they used to be and that would atleast allow project to avoid using xenial with stien. centos 8 will take a while
as to exist as RHEL 8 is not even released yet and centos typically trails major rhel release for a few months so
CentOS 7 will be the latest release for stien and likely train. in the past centos had more quirks with devstack then
ubuntu but for the last year or so i honestly have had equal sucess using bionic or centos 7 so that could be a good
path forward for project without bionic support for there dependencies and it effectively just needs the nodeset to be
chaged in the job definition.

by the way i may have accidentally broke python 2 support when i enabled python3 on centos last week... 
its being fix by https://review.openstack.org/#/c/643299/ but on the plus side python3.6 works well on centos too.
sorry about :) thats what i get for not testing with python 2 anymore but im honetst plesently suprised by how far
centos support has come over the last 18 months so if nothing else it proably worth a patch to just see if the jobs
will just run on centos. 


a little tangential to the topic the only other thing i will say is while there are  docs for how to use the new style
jobs i find are still a little confusing but once you figure out how to create non legacy jobs they are actully pretty
strait forworad. 

what you would have done in a devstackgaterc file in repo 
https://review.openstack.org/#/c/433491/4/devstack/gate/devstackgaterc
port pretty cleanly to a zuul job definition https://review.seanmooney.info/c/test/+/1063/23/.zuul.yaml
and a pre_test_hook.sh file https://review.openstack.org/#/c/433491/4/devstack/gate/pre_test_hook.sh
can eaisily be ported to an ansible script
https://review.seanmooney.info/c/test/+/1063/23/playbooks/dpdk.yaml

although if you like you can still leave the sript in the repo and just run it i only ported
the pre_test_hook to ansible because i had not merged the experimental ovs-dpdk job into the networking-ovs-dpdk
repo so it was simpler to port now then later.

there is a bit of a learning curve to porting jobs form legacy to non legacy but if you can set aside a day or
so you can learn enough ansible and read the required playbook to figure it out. the hardest part honestly was
just getting a couple of uninterrupted hours to sit down an do it. I honestly think it worth spending the time
to do it as it makes tweaking the jobs significantly simpler in the long run.


> 
> [1] https://review.openstack.org/#/c/639990/6
> 
> -gmann
> 
>  >  
>  > > freezer - not able to connect elasticsearch service 
>  > >  - https://review.openstack.org/#/c/642610/  https://review.openstack.org/#/c/642607/  https://review.openstack.o
> rg/#/c/642608/ https://review.openstack.org/#/c/642605/ 
>  > > 
>  > > As Next step for all the projects listed above, either check if you can fix the failure otherwise make the failed
> job as n-v so that 
>  > > this migration which is going to happen tomorrow  would not block your gate. 
>  > > 
>  > > *Testing not started*: 4 
>  > > Loci 
>  > > Openstack-Helm 
>  > > Rally 
>  > > 
>  > > -gmann 
>  > > 
>  > >  > 
>  > >  > 
>  > >  > *Testing Started *: 36 projects 
>  > >  > QA 
>  > >  > Nova 
>  > >  > Placement 
>  > >  > Cinder 
>  > >  > Glance 
>  > >  > Keystone 
>  > >  > Neutron 
>  > >  > Ironic 
>  > >  > Manila 
>  > >  > Designate 
>  > >  > Monasca 
>  > >  > Sahara 
>  > >  > Vitrage 
>  > >  > Congress 
>  > >  > Heat 
>  > >  > Kuryr 
>  > >  > Mistral: OK 
>  > >  > Murano 
>  > >  > Octavia: OK 
>  > >  > Trove 
>  > >  > Watcher 
>  > >  > Oslo 
>  > >  > Barbican 
>  > >  > Blazar 
>  > >  > Ec2-Api 
>  > >  > Cyborg 
>  > >  > Karbor 
>  > >  > Magnum 
>  > >  > Masakari 
>  > >  > Telemetry 
>  > >  > Zaqar 
>  > >  > Zun 
>  > >  > kolla 
>  > >  > Requirements 
>  > >  > Tricircle 
>  > >  > Freezer 
>  > >  > 
>  > >  > #OK. no legacy xenial jobs so no action required 
>  > >  > Qinling 
>  > >  > Searchlight 
>  > >  > Senlin 
>  > >  > Solum 
>  > >  > Storlets 
>  > >  > Winstackers 
>  > >  > Adjutant 
>  > >  > Release Management 
>  > >  > I18n 
>  > >  > Horizon 
>  > >  > Documentation 
>  > >  > Cloudkitty 
>  > >  > Packaging-Rpm 
>  > >  > Powervmstackers 
>  > >  > Chef Openstack 
>  > >  > Tripleo 
>  > >  > Openstack Charms 
>  > >  > Openstackclient 
>  > >  > Openstacksdk 
>  > >  > Puppet Openstack 
>  > >  > Openstackansible 
>  > >  > Swift 
>  > >  > 
>  > >  > *Testing not started*: 4 
>  > >  > Loci 
>  > >  > Openstack-Helm 
>  > >  > Rally 
>  > >  > Tacker 
>  > >  > 
>  > >  > - https://etherpad.openstack.org/p/legacy-job-bionic 
>  > >  > 
>  > >  > -gmann 
>  > >  > 
>  > >  > 
>  > >  >  > 
>  > >  >  >  ---- On Thu, 07 Mar 2019 20:34:57 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ---- 
>  > >  >  >  > Updates on planning and dadlines: Legacy jobs to bionic. 
>  > >  >  >  > 
>  > >  >  >  > I today TC ad hoc meeting on python3 transition & planning[1], we discussed about next steps on legacy
> jobs migration. 
>  > >  >  >  > Also you have seen the Zane's email about the agreement of moving the legacy jobs in Stein [2] which
> will help us to drop 
>  > >  >  >  > the py35 testing. 
>  > >  >  >  > 
>  > >  >  >  > NEW DEADLINE: March 13th, we will merge the base pacthes which move legacy base jobs nodeset to bionic. 
>  > >  >  >  >  ~70% of the project legacy jobs which depends on their parent job nodeset will start running on
> bionic. 
>  > >  >  >  > 
>  > >  >  >  > DO NOT BLOCK GATE: After deadline (when base nodeset is bionic) if any of the your legacy jobs are
> failing then, make those jobs non voting and keep working on fix. 
>  > >  >  >  > 
>  > >  >  >  > How to test your project's legacy jobs : 
>  > >  >  >  > I have updated the base patches which move the legacy base jobs to bionic and ready for projects to tets
> their gate. 
>  > >  >  >  > Detailed steps and testing pacthes example are in this etherpad-  
> https://etherpad.openstack.org/p/legacy-job-bionic 
>  > >  >  >  > 
>  > >  >  >  > For quick reference, I am writting the steps to test your legacy jobs on bionic: 
>  > >  >  >  > -------------------------------------------------- 
>  > >  >  >  > Step 1. Push the testing DNM patch on your project owned repos with: 
>  > >  >  >  > - Depends-On: https://review.openstack.org/#/c/641886/ 
>  > >  >  >  >  -Remove or change the nodeset to Bionic from your repo owned legacy jobs (if any of the legacy jobs has
> overridden the nodeset from parent job) 
>  > >  >  >  > Example: https://review.openstack.org/#/c/639017 
>  > >  >  >  > 
>  > >  >  >  > Step 2. If you have any legacy job not derived from base jobs then, you need to migrate their nodeset to
> bionic which are defined in - https://review.openstack.org/#/c/639018/ 
>  > >  >  >  > Example - https://review.openstack.org/#/c/639361/ 
>  > >  >  >  > 
>  > >  >  >  > Step3. if any of the job start failing on bionic then, either fix it before deadline of March 13th or
> make failing job as n-v and fix later. 
>  > >  >  >  > -------------------------------------------------- 
>  > >  >  >  > 
>  > >  >  >  > Feel free to ping me for any query or help. 
>  > >  >  >  > 
>  > >  >  >  > [1] http://eavesdrop.openstack.org/meetings/tc_python3/2019/tc_python3.2019-03-07-21.00.html 
>  > >  >  >  > [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003610.html 
>  > >  >  >  > 
>  > >  >  >  > -gmann 
>  > >  >  >  > 
>  > >  >  >  > 
>  > >  >  >  >  ---- On Thu, 07 Mar 2019 10:55:50 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ---- 
>  > >  >  >  >  > Thanks all who volunteer for their project and testing the jobs on bionic. 
>  > >  >  >  >  > 
>  > >  >  >  >  > Reminder for all other projects to plan for this activity. Only 5-6 projects have started the
> testing. 
>  > >  >  >  >  > I request PTL to plan someone for this activity. Feel free to ping me on #openstack-qa for any
> clarification. 
>  > >  >  >  >  > 
>  > >  >  >  >  > Thanks. 
>  > >  >  >  >  > 
>  > >  >  >  >  > -gmann 
>  > >  >  >  >  > 
>  > >  >  >  >  >  ---- On Mon, 25 Feb 2019 09:05:47 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ---- 
>  > >  >  >  >  >  > Hi Everyone, 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > During Dec/Jan month, we have migrated the devstack jobs (zuulv3 native jobs) from Xenial to
> Bionic. [1]. 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > But that did not move all gate job to Bionic as there are a large number of jobs are still the
> legacy job. All the legacy job still use Xenial as nodeset. 
>  > >  >  >  >  >  > As per the decided runtime for Stein, we need to test everything on OpenStack CI/CD on Bionic - 
> https://governance.openstack.org/tc/reference/runtimes/stein.html 
>  > >  >  >  >  >  > This is something we discussed TC meeting as one of the Stein items to finish in openstack
> community[2]. 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > I am starting this ML thread to coordinate the work to move all the legacy jobs to Bionic. I have
> created the 
>  > >  >  >  >  >  > Etherpad to include more details and  status check for each project gate- 
> https://etherpad.openstack.org/p/legacy-job-bionic 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > The approach is the same as we did for previous migration for devstack jobs. 
>  > >  >  >  >  >  > 1. Push patch which migrates the legacy base jobs to bionic - 1. 
> https://review.openstack.org/#/c/639096/ 
>  > >  >  >  >  >  > 2. Each Project team, add the testing patch with Depends-on on base patch (example 639096) and
> confirm the status 
>  > >  >  >  >  >  >      of their gate on etherpad - Example: https://review.openstack.org/#/c/639017 
>  > >  >  >  >  >  > 3. Project jobs not using base job as parent, need to migrate their legacy job to bionic by own. 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > Please add yourself as your project volunteer in etherpad and update the status accordingly. I am
> tagging [ptl] in this thread so that they can assign 
>  > >  >  >  >  >  > someone from their team if no volunteer. 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > Deadline: 1st April to merge the base legacy job to bionic.  That gives around 1 month to test the
> jobs which I feel enough for each project. 
>  > >  >  >  >  >  > let me know if more time is needed, we can adjust the same. The goal is to finish this activity
> before Stein release. 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > [1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000837.html 
>  > >  >  >  >  >  >        https://etherpad.openstack.org/p/devstack-bionic 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > [2] http://eavesdrop.openstack.org/meetings/tc/2019/tc.2019-02-07-14.00.html 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > -gmann 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > 
>  > >  >  >  >  >  > 
>  > >  >  >  >  > 
>  > >  >  >  >  > 
>  > >  >  >  > 
>  > >  >  >  > 
>  > >  >  > 
>  > >  >  > 
>  > >  >  > 
>  > >  >  > 
>  > >  > 
>  > >  > 
>  > >  > 
>  > >  > 
>  > > 
>  > > 
>  > > 
>  > 
> 
>