July 2020 - openstack-discuss - lists.openstack.org

[tc] weekly update
by Mohammed Naser 06 Jul '20

06 Jul '20

Hi everyone, Here’s an update for what happened in the OpenStack TC this week. You can get more information by checking for changes in openstack/governance repository. We've also included a few references to some important mailing list threads that you should check out. # Patches ## Open Review - Cleanup the remaining osf repos and their data https://review.opendev.org/739291 - Update goal selection docs to clarify the goal count https://review.opendev.org/739150 - Add legacy repository validation https://review.opendev.org/737559 - Add "tc:approved-release" tag to manila https://review.opendev.org/738105 - Create starter-kit:kubernetes-in-virt tag https://review.opendev.org/736369 - [draft] Add assert:supports-standalone https://review.opendev.org/722399 ## Project Updates - Add deprecated cycle for deprecated deliverables https://review.opendev.org/737590 - No longer track refstack repos in governance https://review.opendev.org/737962 - Add Neutron Arista plugin charm to OpenStack charms https://review.opendev.org/737734 ## General Changes - Add links to chosen release names https://review.opendev.org/738867 - Add storyboard link to migrate-to-focal goal https://review.opendev.org/738129 - TC Guide Follow Ups https://review.opendev.org/737650 # Email Threads - New Office Hours: http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015761.html - OSU Intern Work: http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015760.html - Summit Programming Committee Nominations Open: http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015756.html - Summit CFP Open: http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015730.html # Other Reminders - OpenStack's 10th anniversary community meeting should be happening July 16th: more info coming soon! - If you're an operator, make sure you fill out our user survey: https://www.openstack.org/user-survey/survey-2020/ Thanks for reading! Mohammed & Kendall -- Mohammed Naser VEXXHOST, Inc.

1 0

Re: [rdo-users] [rdo][ussuri][TripleO][nova][kvm] libvirt.libvirtError: internal error: process exited while connecting to monitor
by Arkady Shtempler 06 Jul '20

06 Jul '20

Hi all! I was able to analyze the attached log files and I hope that the results may help you understand what's going wrong with instance creation. You can find *Log_Tool's unique exported Error blocks* here: http://paste.openstack.org/show/795356/ *Some statistics and problematical messages:* ##### Statistics - Number of Errors/Warnings per Standard OSP log since: 2020-06-30 12:30:00 ##### Total_Number_Of_Errors --> 9 /home/ashtempl/Ruslanas/controller/neutron/server.log --> 1 /home/ashtempl/Ruslanas/compute/stdouts/ovn_controller.log --> 1 /home/ashtempl/Ruslanas/compute/nova/nova-compute.log --> 7 *nova-compute.log* *default default] Error launching a defined domain with XML: <domain type='kvm'>* 368-2020-06-30 12:30:10.815 7 *ERROR* nova.compute.manager [req-87bef18f-ad3d-4147-a1b3-196b5b64b688 7bdb8c3bf8004f98aae1b16d938ac09b 69134106b56941698e58c61... 70dc50f] Instance *failed* to spawn: *libvirt.libvirtError*: internal *error*: qemu unexpectedly closed the monitor: 2020-06-30T10:30:10.182675Z qemu-kvm: *error*: failed to set MSR 0... he monitor: 2020-06-30T10:30:10.182675Z *qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172* _msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' *failed*. [instance: 128f372c-cb2e-47d9-b1bf-ce17270dc50f] *Traceback* (most recent call last): 375-2020-06-30 12:30:10.815 7* ERROR* nova.compute.manager [instance: 128f372c-cb2e-47d9-b1bf-ce17270dc50f] File "/usr/lib/python3.6/site-packages/nova/vir... *server.log * 5821c815-d213-498d-9394-fe25c6849918', 'status': 'failed', *'code': 422} returned with failed status* *ovn_controller.log* 272-2020-06-30T12:30:10.126079625+02:00 stderr F 2020-06-30T10:30:10Z|00247|patch|WARN|*Bridge 'br-ex' not found for network 'datacentre'* Thanks! On Tue, Jun 30, 2020 at 2:13 PM Ruslanas Gžibovskis <ruslanas(a)lpic.lt> wrote: > attaching logs here. let's see if it will work. > > On Tue, 30 Jun 2020 at 12:55, Ruslanas Gžibovskis <ruslanas(a)lpic.lt> > wrote: > >> hi all, >> >> I am back, had some issues with MTU. >> Now looks good, at least deployment part. >> >> So I have installed back what I had, and still failing at same point as >> in first message. >> >> I have tried to use: LogTool, how to use it? well, I launched it, but it >> always say [0] detailed output: >> File "./PyTool.py", line 596, in <module> >> random_node=random.choice(overcloud_nodes) >> >> I do not get, how to make it work, should it get from stackrc ? as I see >> in >> overcloud_nodes = [] >> all_nodes = exec_command_line_command('source ' + source_rc_file_path >> + 'stackrc;openstack server list -f json')[ >> >> [0] http://paste.openstack.org/show/795345/ >> >> On Wed, 24 Jun 2020 at 20:02, Arkady Shtempler <ashtempl(a)redhat.com> >> wrote: >> >>> Hi Ruslanas! >>> >>> Is it possible to get all logs under /var/log/containers somehow? >>> >>> Thanks! >>> >>> On Wed, Jun 24, 2020 at 2:18 AM Ruslanas Gžibovskis <ruslanas(a)lpic.lt> >>> wrote: >>> >>>> Hi Alfredo, >>>> >>>> Compute nodes are baremetal or virtualized?, I've seen similar bug >>>>>> reports when using nested virtualization in other OSes. >>>>>> >>>>> baremetal. Dell R630 if to be VERY precise. >>>> >>>> >>>> >>>>>> When using podman, the recommended way to restart containers is using >>>>> systemd: >>>>> >>>>> >>>>> https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deploym… >>>>> >>>> >>>> Thank you, I will try. I also modified a file, and it looked like it >>>> relaunched podman container once config was changed. Either way, if I >>>> understand Linux config correctly, the default value for user and group is >>>> root, if commented out: >>>> #user = "root" >>>> #group = "root" >>>> >>>> also in some logs, I saw, that it detected, that it is not AMD CPU :) >>>> and it is really not AMD CPU. >>>> >>>> >>>> Just for fun, it might be important, here is how my node info looks. >>>> ComputeS01Parameters: >>>> NovaReservedHostMemory: 16384 >>>> KernelArgs: "crashkernel=no rhgb" >>>> ComputeS01ExtraConfig: >>>> nova::cpu_allocation_ratio: 4.0 >>>> nova::compute::libvirt::rx_queue_size: 1024 >>>> nova::compute::libvirt::tx_queue_size: 1024 >>>> nova::compute::resume_guests_state_on_host_boot: true >>>> _______________________________________________ >>>> users mailing list >>>> users(a)lists.rdoproject.org >>>> http://lists.rdoproject.org/mailman/listinfo/users >>>> >>>> To unsubscribe: users-unsubscribe(a)lists.rdoproject.org >>>> >>> >> >> -- >> Ruslanas Gžibovskis >> +370 6030 7030 >> > > > -- > Ruslanas Gžibovskis > +370 6030 7030 >

4 10

[python-openstackclient] microversion support
by jayaditya gupta 06 Jul '20

06 Jul '20

Hi , we discussed the microversion support in PTG meeting .I would like to get started with it. How can I help with this? Currently OpenStack CLI is defaulting to nova api V2, we want to change it so it takes latest version. for issue reference see this : https://storyboard.openstack.org/#!/story/2007727 Best Regards Jayaditya Gupta

1 0

[All][Neutron] Migrate old DB migration versions to init ops
by Rodolfo Alonso Hernandez 06 Jul '20

06 Jul '20

Hello all: Along this years we have increased the number of DB migrations each time we needed a new DB schema. This is good because that means the project is evolving and adding new features. Although this is not a problem per se, there are some inconvenients: - Every time a system is deployed (for example in the CI using devstack), the initial DB schema is created. Then, each migration is applied sequentially. - Some FT tests are still checking the sanity of some migrations [1] implemented a few releases ago. - We are still testing the contract DB migrations. Of course, this is something supported before and we still need to apply those revisions. - "TestWalkMigrationsMysql" and "TestModelsMigrationsMysql", both using MySQL backend, are still affected by LP#1687027. The proposal is to remove some DB migrations, starting from Liberty; of course, because all migrations must be applied in a specific order, we should begin from the initial revision, "kilo". The latest migration to be removed should be decided depending on the stable releases support. Apart from mitigating or solving some of the commented problems, we can "group" the DB model definition in one place. E.g.: "subnetpools" table is created in "other_extensions_init_ops". This file contains the first table. However is modified in at least two migrations: - 1b4c6e320f79_address_scope_support_in_subnetpool: added "address_scope_id" field - 13cfb89f881a_add_is_default_to_subnetpool: added "is_default" field Instead of having (at least) three places where the "subnetpools" DB schema is defined, we can remove the Mitaka migration and group this definition in just one place. One possible issue: some migrations add dependencies on other tables. That means the table the dependency is referring should be created in advance. That implies that, in some cases, the table creation order should be modified. That should never affect subsequent created tables or migrations. Do you see any inconvenience on this proposal? Am I missing something that I didn't consider? Thank you and regards. [1] https://github.com/openstack/neutron/blob/9fd60ffaac6b178de62dab169c826d52f…

4 6

[cyborg] Incomplete v2 API in Train
by Ildiko Vancsa 06 Jul '20

06 Jul '20

Hi Cyborg Team, I’m working with the CNTT community[1], they are working on building reference architecture for telecom workloads. Cyborg is important for their work to be able to utilize hardware acceleration resources. We are planning to use the Train version of OpenStack projects including Cyborg and it would be great to be able to switch to the v2 API as v1 is deprecated now. If my understanding is correct the v2 API implementation in Train is partial, but the documentation[2] doesn’t give accurate view about what is included. The CNTT team would like to be able to integrate and access the whole v2 API if that is possible. It would be great to discuss the options that we could use on the way forward. Would it be possible to bring this up and discuss on an upcoming Cyborg team meeting? Thanks, Ildikó [1] https://www.lfnetworking.org/about/cntt/ [2] https://docs.openstack.org/cyborg/train/api/api.html#v2-0

2 2

Re: [cyborg] Incomplete v2 API in Train
by yumeng bao 06 Jul '20

06 Jul '20

Hi Ildikó, > Hi Cyborg Team, > I’m working with the CNTT community[1], they are working on building reference architecture for telecom workloads. Cyborg is important for their work to be able to utilize hardware acceleration > resources. > We are planning to use the Train version of OpenStack projects including Cyborg and it would be great to be able to switch to the v2 API as v1 is deprecated now. If my understanding is correct > the v2 API implementation in Train is partial, but the documentation[2] doesn’t give accurate view about what is included. Yes,your understanding is correct,the v2 API implementation in Train is partial. I would update the documentation soon. I would recommend you to use the stable/ussuri version(instead of train release) of cyborg for two reasons: 1) API V2 in ussuri is complete while that of train is incomplete 2)the nova-cyborg integration[3] was not landed until Ussuri[4],so the integration in Train is also partial[5]. So if CNTT wants the complete accelerator management function,it would be better to use cyborg ussuri. > The CNTT team would like to be able to integrate and access the whole v2 API if that is possible. It would be great to discuss the options that we could use on the way forward. Would it be > possible to bring this up and discuss on an upcoming Cyborg team meeting? Yes, sure. We can bring this up on the next weekly meeting on this Thursday 03:00 UTC at #openstack-cyborg, I have added this to meeting agenda[6]. > Thanks, > Ildikó > [1] https://www.lfnetworking.org/about/cntt/ > [2] https://docs.openstack.org/cyborg/train/api/api.html#v2-0 [3]https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented… [4]https://releases.openstack.org/ussuri/highlights.html#cyborg [5]https://releases.openstack.org/train/highlights.html#cyborg [6]https://wiki.openstack.org/wiki/Meetings/CyborgTeamMeeting#Agenda Regards， Yumeng

2 1

[Neutron] Bug Deputy Report (June 29 - July 05)
by Hongbin Lu 05 Jul '20

05 Jul '20

Hi all, Below is the bug deputy report for last week. Critical: * https://bugs.launchpad.net/neutron/+bug/1885900 test_trunk_subport_lifecycle is failing in ovn based jobs * https://bugs.launchpad.net/neutron/+bug/1885899 test_qos_basic_and_update test is failing High: * https://bugs.launchpad.net/neutron/+bug/1886116 slaac no longer works on IPv6 tenant subnets * https://bugs.launchpad.net/neutron/+bug/1885898 test connectivity through 2 routers fails in neutron-ovn-tempest-full-multinode-ovs-master job * https://bugs.launchpad.net/neutron/+bug/1885897 Tempest test_create_router_set_gateway_with_fixed_ip test is failing often in dvr scenario job * https://bugs.launchpad.net/neutron/+bug/1885695 [OVS] "vsctl" implementation does not allow empty transactions Medium: * https://bugs.launchpad.net/neutron/+bug/1885891 DB exception when updating a "ml2_port_bindings" object * https://bugs.launchpad.net/neutron/+bug/1885758 RPCMessage timeouts when ovs agent is reporting status about many ports Low: * https://bugs.launchpad.net/neutron/+bug/1886216 keepalived-state-change does not format correctly the logs * https://bugs.launchpad.net/neutron/+bug/1885547 [fullstack] OVS interface events isolation error with more than one OVS agent RFE: * https://bugs.launchpad.net/neutron/+bug/1885921 [RFE][floatingip port_forwarding] Add port ranges

1 0

[all][tc][goals] Migrate CI/CD jobs to new Ubuntu LTS Focal: Week R-15 Update
by Ghanshyam Mann 05 Jul '20

05 Jul '20

Hello Everyone, Please find the week R-15 updates on 'Ubuntu Focal migration' community goal. Tracking: https://storyboard.openstack.org/#!/story/2007865 Progress: ======= * I have prepared the patched to migrate the unit/functional/doc/cover tox jobs to focal which are WIP till we finish the project side testing. This and its base patches - https://review.opendev.org/#/c/738328/ * devstack and tempest base patches are changed with Depends-On on 738328 tox job patch. This way we can test the complete gate (integration + unit +functional + doc + cover +pep8 + lower-constraint) jobs with a single testing patch by doing Depends-On: https://review.opendev.org/#/c/734700/ (or devstack base patch or tox one if you do not have tempest jobs to test) * I have started a few more project testing and found bugs on the incompatible deps versions for Focal. Please refer to the 'Bugs Report' section for details. Bugs Report: ========== Summary: Total 4 (1 fixed, 3 in-progress). 1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration. 2. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now. 3. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility. 4. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging. What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack. 1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): https://review.opendev.org/#/c/734700/ Example: https://review.opendev.org/#/c/738126/ 2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/ 3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches. Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches. References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-j… Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865 [1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032 -gmann

1 0

Re: [loci][helm][k8s] When do images on docker.io get updated
by Andrii Ostapenko 03 Jul '20

03 Jul '20

Hello Corne, OSH uses images built using gates in openstack/openstack-helm-images repository, not in loci itself. You may want to add a definition for watcher image similar to [0] and then refer to it in the corresponding release job, e.g. for Stein [1]. After your commit to openstack-helm-images is merged, new images will be published to docker.io/openstackhelm/watcher repository and can be used in the way you referenced them in your OSH commit [2]. [0] https://opendev.org/openstack/openstack-helm-images/src/branch/master/zuul.… [1] https://opendev.org/openstack/openstack-helm-images/src/branch/master/zuul.… [2] https://review.opendev.org/#/c/720140/

3 2

[TC] Monthly Meeting Summary
by Mohammed Naser 03 Jul '20

03 Jul '20

Hi everyone, Here’s a summary of what happened in our TC monthly meeting last Thursday, July 2nd. # ATTENDEES (LINES SAID) - mnaser (106) - gmann (43) - evrardjp (32) - diablo_rojo (32) - njohnston (11) - jungleboyj (8) - openstack (7) - ricolin (6) - fungi (4) - ttx (2) - clarkb (2) - belmoreira (1) - knikolla (1) - AJaeger (1) # MEETING SUMMARY - Rollcall (mnaser, 14:00:40) - Follow up on past action items (mnaser, 14:04:59) - OpenStack Foundation OSU Intern Project (diablo_rojo) (mnaser, 14:26:55) - W cycle goal selection start (mnaser, 14:36:02) - https://governance.openstack.org/tc/goals/#goal-selection-schedule (gmann, 14:37:37) - Completion of retirement cleanup (gmann) (mnaser, 14:48:36) - https://etherpad.opendev.org/p/tc-retirement-cleanup is a scratch pad; nothing pushed out towards the community (mnaser, 14:48:59) # ACTION ITEMS - evrardjp & njohnston to start writing resolution about how deconstructed PTL role - mnaser to find the owner to start using facing API pop-up team over ML - gmann update goal selection docs to clarify the goal count - gmann start discussion around reviewing currenet tags - mnaser propose change to implement weekly meetings - diablo_rojo start discussion on ML around potential items for OSF funded intern - njohnston and mugsie to work on getting goals groomed/proposed for W cycle - TC and community to help finish properly and cleanly retiring projects To read the full logs of the meeting, please refer to http://eavesdrop.openstack.org/meetings/tc/2020/tc.2020-07-02-14.00.log.html. -- Mohammed Naser VEXXHOST, Inc.

1 0