September 2020 - openstack-discuss

LVM misconfiguration after openstack stackpack server hang and reboot
by Alan Davis 24 Sep '20

24 Sep '20

This morning my CentOS 7.7 RDO packstack installation of Rocky hung. On reboot some of the VMs won't start. This is a primary system and I need to find the most expedient way to recover without losing data. I'm not using LVM thin volumes. Any help is appreciated. Looking at nova-compute.log I see errors trying to find LUN 0 during the sysfs stage. Several machines won't boot because their root disk entries in LVM are seen as PV and booting them doesn't see them in the DM subsystem. Other machines boot but there attached disks throw LVM errors about duplicate PV and preferring the cinder-volumes VG version. LVM is showing LVs that have both "bare" entries as well as entries in cinder-volumes and it's complaining about duplicate PVs, not using lvmetad and preferring some entries because they are in the dm subsystem. I've verified that, so far, I haven't lost any data. The "bare" LV not being used as part of the DM subsystem because it's server won't boot can be mounted on the openstack host and all data on it is accessible. This host has rebooted cleanly multiple times in the past. This is the first time it's shown any problems. Am I missing an LVM filter? (unlikely since it wasn't neede before) How can I reset the LVM configuration and convince it that it's not seeing duplicate PV? How do I ensure that openstack sees the right UUID and volume ID? Excerpts from error log and output of lvs : --- nova-compute.log --- during VM start 2020-09-24 11:15:27.091 13953 INFO os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d1c8 - default default] Trying to connect to iSCSI portal 172.10.0.40:3260 2020-09-24 11:15:29.721 13953 WARNING nova.compute.manager [req-fd32e16f-c879-402f-a32c-6be45a943c34 48af9a366301467d9fec912fd1c072c6 f9fc7b412a8446d083da1356aa370eb4 - default d efault] [instance: de7d740c-786a-4aa2-aa09-d447ae7e14b6] Received unexpected event network-vif-unplugged-79aff403-d2e4-4266-bd88-d7bd19d501a9 for instance with vm_state stopped a nd task_state powering-on. 2020-09-24 11:16:21.361 13953 WARNING os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d 1c8 - default default] LUN 0 on iSCSI portal 172.10.0.40:3260 not found on sysfs after logging in. 2020-09-24 11:16:23.482 13953 INFO os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d1c8 - default default] Trying to connect to iSCSI portal 172.10.0.40:3260 2020-09-24 11:17:17.741 13953 WARNING os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d 1c8 - default default] LUN 0 on iSCSI portal 172.10.0.40:3260 not found on sysfs after logging in.: VolumeDeviceNotFound: Volume device not found at . 2020-09-24 11:17:21.864 13953 INFO os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d1c8 - default default] Trying to connect to iSCSI portal 172.10.0.40:3260 2020-09-24 11:18:16.113 13953 WARNING os_brick.initiator.connectors.iscsi [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d 1c8 - default default] LUN 0 on iSCSI portal 172.10.0.40:3260 not found on sysfs after logging in.: VolumeDeviceNotFound: Volume device not found at . 2020-09-24 11:18:17.252 13953 INFO nova.compute.manager [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d1c8 - default defa ult] [instance: de7d740c-786a-4aa2-aa09-d447ae7e14b6] Successfully reverted task state from powering-on on failure for instance. 2020-09-24 11:18:17.279 13953 ERROR oslo_messaging.rpc.server [req-8d15fb6a-6324-471e-9497-587885eef8f6 396aeda6552f44fdac5f878b90325ee1 54af92f2bb494355b96024076184d1c8 - defaul t default] Exception during message handling: VolumeDeviceNotFound: Volume device not found at . 2020-09-24 11:18:17.279 13953 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-09-24 11:18:17.279 13953 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2020-09-24 11:18:17.279 13953 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) --- lvs output --- I've annotated 1 machine's disks to illustrate the relationship between the volume-*** cinder-volumes vg entries and the "bare" lv seen as directly accessible from the host. There are 3 servers that won't boot, they are the one's who's home/vg_home and encrypted_home/encrypted_vg entries are shown. WARNING: Not using lvmetad because duplicate PVs were found. WARNING: Use multipath or vgimportclone to resolve duplicate PVs? WARNING: After duplicates are resolved, run "pvscan --cache" to enable lvmetad. WARNING: Not using device /dev/sdu for PV yZy8Xk-foKT-ovjV-0EZv-VxEM-GqiP-WH7k53. == backup_lv/encrypted_vg WARNING: Not using device /dev/sdv for PV tHA9ui-eSIO-MDmI-RM3u-3Bf4-Dznb-Ha3XfP. == varoptgitlab/encrypted_vg WARNING: Not using device /dev/sdm for PV 5eoyCa-sMO4-b7O4-jIfh-byZE-L5pS-3lOu0D. WARNING: Not using device /dev/sdp for PV 3BI0nV-TP0k-rgPC-PrjH-FT7z-reMe-ec1spj. WARNING: Not using device /dev/sdt for PV ILdbcY-VFCm-fnH6-Y3jc-pdWZ-fnl8-PH3TPe. == storage_lv/encrypted_vg WARNING: Not using device /dev/sdr for PV zowU2N-oaBh-r4cO-cxgX-YYiq-Kf3q-mqlHfK. WARNING: PV yZy8Xk-foKT-ovjV-0EZv-VxEM-GqiP-WH7k53 prefers device /dev/cinder-volumes/volume-c8da1abf-7143-422c-9ee5-b2724a71c8ff because device is in dm subsystem. WARNING: PV tHA9ui-eSIO-MDmI-RM3u-3Bf4-Dznb-Ha3XfP prefers device /dev/cinder-volumes/volume-0a12012f-8c2e-41fb-aa0c-a7ae99c62487 because device is in dm subsystem. WARNING: PV 5eoyCa-sMO4-b7O4-jIfh-byZE-L5pS-3lOu0D prefers device /dev/cinder-volumes/volume-990a057c-46cc-4a81-ba02-28b72c34791d because device is in dm subsystem. WARNING: PV 3BI0nV-TP0k-rgPC-PrjH-FT7z-reMe-ec1spj prefers device /dev/cinder-volumes/volume-b6a9da6e-1958-46ea-90b4-ac1aebed8c04 because device is in dm subsystem. WARNING: PV ILdbcY-VFCm-fnH6-Y3jc-pdWZ-fnl8-PH3TPe prefers device /dev/cinder-volumes/volume-302dd53b-7d05-4f6d-9ada-8f2ed6e1d4c6 because device is in dm subsystem. WARNING: PV zowU2N-oaBh-r4cO-cxgX-YYiq-Kf3q-mqlHfK prefers device /dev/cinder-volumes/volume-df006472-be7a-4957-972a-1db4463f5d67 because device is in dm subsystem. LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home centos_stack3 -wi-ao---- 4.00g root centos_stack3 -wi-ao---- 50.00g swap centos_stack3 -wi-ao---- 4.00g _snapshot-05b1e46b-1ae3-4cd0-9117-3fb53a6d94b0 cinder-volumes swi-a-s--- 20.00g volume-1d0ff5d5-93a3-44e8-8bfa-a9290765c8c6 0.00 lv_filestore cinder-volumes -wi-ao---- 1.00t ... volume-c8da1abf-7143-422c-9ee5-b2724a71c8ff cinder-volumes -wi-ao---- 100.00g volume-0a12012f-8c2e-41fb-aa0c-a7ae99c62487 cinder-volumes -wi-ao---- 60.00g volume-990a057c-46cc-4a81-ba02-28b72c34791d cinder-volumes -wi-ao---- 200.00g volume-b6a9da6e-1958-46ea-90b4-ac1aebed8c04 cinder-volumes -wi-ao---- 30.00g volume-302dd53b-7d05-4f6d-9ada-8f2ed6e1d4c6 cinder-volumes -wi-ao---- 60.00g volume-df006472-be7a-4957-972a-1db4463f5d67 cinder-volumes -wi-ao---- 250.00g ... volume-f3250e15-bb9c-43d1-989d-8a8f6635a416 cinder-volumes -wi-ao---- 20.00g volume-fc1d5fcb-fda1-456b-a89d-582b7f94fb04 cinder-volumes -wi-ao---- 300.00g volume-fc50a717-0857-4da3-93cb-a55292f7ed6d cinder-volumes -wi-ao---- 20.00g volume-ff94e2d6-449b-495d-82e6-0debd694c1dd cinder-volumes -wi-ao---- 20.00g data2 data2_vg -wi-a----- <300.00g data data_vg -wi-a----- 1.79t backup_lv encrypted_vg -wi------- <100.00g == ...WH7k53 storage_lv encrypted_vg -wi------- <60.00g == ...PH3TPe varoptgitlab_lv encrypted_vg -wi------- <200.00g varoptgitlab_lv encrypted_vg -wi------- <30.00g varoptgitlab_lv encrypted_vg -wi------- <60.00g == ...Ha3XfP encrypted_home home_vg -wi-a----- <40.00g encrypted_home home_vg -wi------- <60.00g pub pub_vg -wi-a----- <40.00g pub_lv pub_vg -wi------- <250.00g rpms repo -wi-a----- 499.99g home vg_home -wi-a----- <40.00g gtri_pub vg_pub -wi-a----- 20.00g pub vg_pub -wi-a----- <40.00g -- Alan Davis Principal System Administrator Apogee Research LLC

1 1

[Glance] PTL on Vacation
by Abhishek Kekane 24 Sep '20

24 Sep '20

Hi All, I'm starting my vacations from 28th September and will be back on October 5th. Please direct any issues to the rest of the core team. Also there will be no glance weekly meeting on 01 October [1]. [1] https://etherpad.opendev.org/p/glance-team-meeting-agenda Thanks, Abhishek

1 0

[all][tc][goals] Migrate CI/CD jobs to new Ubuntu LTS Focal: Week R-4 Update
by Ghanshyam Mann 24 Sep '20

24 Sep '20

Hello Everyone, Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan. Tracking: https://storyboard.openstack.org/#!/story/2007865 Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan. * Part1: Migrating tox base job tomorrow (8th Sept): ** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ . **There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing. * Part2: Migrating devstack/tempest base job on 10th sept: * We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept. ** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003 Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+… * ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open * ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AV… Bugs Report: ========== 1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration. 2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003 3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now. 4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility. 5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging. What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack. 1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest: Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/ Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/ Example: https://review.opendev.org/#/c/738126/ 2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/ 3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2 4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches. Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches. References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-j… Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865 [1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032 -gmann

7 17

[tripleo] rdo infrastructure maintenance
by Wesley Hayutin 24 Sep '20

24 Sep '20

Greetings, There is some maintenance on the rdo infra that is causing check and gate jobs to fail upstream. The situation is actively being worked on at this time. You may find errors in your check / gate jobs.. like the following. TASK [oooci-build-images : Download TripleO source image] 2020-09-24 10:26:35.615996 | primary | ERROR 2020-09-24 10:26:35.617685 | primary | { 2020-09-24 10:26:35.617774 | primary | "attempts": 60, 2020-09-24 10:26:35.617850 | primary | "msg": "Failed to connect to images.rdoproject.org at port 443: [Errno 113] No route to host", 2020-09-24 10:26:35.617929 | primary | "status": -1, 2020-09-24 10:26:35.618001 | primary | "url": "https://images.rdoproject.org/CentOS-8-x86_64-GenericCloud.qcow2" 2020-09-24 10:26:35.618072 | primary | } https://5134c188955ee39fd51b-cd04c057bb6c1703504e41a1dbc31642.ssl.cf5.rackc… Sorry for the trouble, we'll update this email when we're clear. 0/

1 1

[manila] Doc-a-thon event coming up next Thursday (Aug 6th)
by Victoria Martínez de la Cruz 24 Sep '20

24 Sep '20

Hi folks, We will be organizing a doc-a-thon next Thursday, August 6th, with the main goal of improving our docs for the next release. We will be gathering on our Freenode channel #openstack-manila after our weekly meeting (3pm UTC) and also using a videoconference tool (exact details TBC) to go over a curated list of opened doc bugs we have here [0]. *Your* participation is truly valued, being you an already Manila contributor or if you are interested in contributing and you didn't know how, so looking forward to seeing you there :) Cheers, Victoria [0] https://ethercalc.openstack.org/ur17jprbprxx

2 3

[oslo] PTL Non-candidacy: I mean it this time :-P
by Ben Nemec 24 Sep '20

24 Sep '20

I'm eminently familiar with midwest goodbyes, but an entire cycle to actually leave is taking it a bit far, don't you think? :-) Seriously though, this is two cycles in a row that we've had to scramble late in a cycle to finish basic community goals, mostly because of my lack of attention. Now that we've all sort of adapted to the whole global pandemic thing I'm hoping my abdication will stick this time. I'll still be around and try to help out when I can, but it's past time for me to turn the leadership of Oslo over to someone else. We've got a great team and I'm sure any one of them will do a great job! Thanks. -Ben

5 4

[all][elections][tc] Stepping down from the TC
by Jean-Philippe Evrard 24 Sep '20

24 Sep '20

Hello everyone, This is probably not a surprise for most of you, but I think it's worth writing it down: I won't be a candidate for another term at the TC. It was a pleasure to work with all of you. I am not leaving because I don't find the TC interesting anymore, quite the opposite. I will probably still lurk and follow the channels and ML. I just have switched to (yet) another duty at my employer, keeping me away from OpenStack. Next to this, I believe it's good to have some fresh members in the TC, it's been a while I am part of this family now :) For those interested by running the TC: Don't hesitate to run! We need fresh ideas and motivated people. It's not by having always the same people at the helm that OpenStack will naturally or drastically evolve. If you want to change OpenStack, be the change! Thanks to all of you. It was nice. Regards, Jean-Philippe Evrard (evrardjp)

11 11

RE: [tacker] Propose Toshiaki Takahashi for tacker core
by TAKAHASHI TOSHIAKI(高橋　敏明) 24 Sep '20

24 Sep '20

Hi Yasufumi and Team, Thank you so much! I'm glad to join core team. I'll continue to contribute for Tacker community activation! Toshiaki > -----Original Message----- > From: Yasufumi Ogawa <yasufum.o(a)gmail.com> > Sent: Thursday, September 24, 2020 6:43 PM > To: openstack-discuss(a)lists.openstack.org; TAKAHASHI TOSHIAKI(高橋　敏明) > <ts-takahashi(a)nec.com> > Subject: ##freemail## Re: [tacker] Propose Toshiaki Takahashi for tacker core > > On 2020/09/17 4:06, yasufum wrote: > > Toshiaki Takahashi (takahashi-tsc) has been so active for reviewing, > > fixing bugs and answering questions in the recent releases [1][2] and > > had several sessions on summits for Tacker. In addition, he is now > > well distinguished as one of the responsibility from ETSI-NFV standard > > community as a contributor between the standard and implementation for > > the recent contributions for both of OpenStack and ETSI. > > > > I'd appreciate if we add Toshiaki to the core team. > > > > [1] https://www.stackalytics.com/?company=nec&module=tacker > > [2] > > https://www.stackalytics.com/?user_id=t-takahashi%40ig.jp.nec.com&metr > > ic=marks > > Thank you everyone! I've seen only positive responses, so I've added Toshiaki as a > new core of tacker team. > > Congratulations, Toshiaki! > > > > > > > Regards, > > Yasufumi > >

1 0

[tacker] Propose Toshiaki Takahashi for tacker core
by yasufum 24 Sep '20

24 Sep '20

Toshiaki Takahashi (takahashi-tsc) has been so active for reviewing, fixing bugs and answering questions in the recent releases [1][2] and had several sessions on summits for Tacker. In addition, he is now well distinguished as one of the responsibility from ETSI-NFV standard community as a contributor between the standard and implementation for the recent contributions for both of OpenStack and ETSI. I'd appreciate if we add Toshiaki to the core team. [1] https://www.stackalytics.com/?company=nec&module=tacker [2] https://www.stackalytics.com/?user_id=t-takahashi%40ig.jp.nec.com&metric=ma… Regards, Yasufumi

10 9

Error log
by Issac Chan 24 Sep '20

24 Sep '20

Hi all, May you have a look on this log file? I cannot start trove guest instance database normally. Best wishes, Issac

1 0