October 2020 - openstack-discuss - lists.openstack.org

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

[nova][dev] Revisiting qemu emulation where guest arch != host arch
by Apsey, Christopher 05 Jan '22

05 Jan '22

All, A few years ago I asked a question[1] about why nova, when given a hw_architecture property from glance for an image, would not end up using the correct qemu-system-xx binary when starting the guest process on a compute node if that compute nodes architecture did not match the proposed guest architecture. As an example, if we had all x86 hosts, but wanted to run an emulated ppc guest, we should be able to do that given that at least one compute node had qemu-system-ppc already installed and libvirt was successfully reporting that as a supported architecture to nova. It seemed like a heavy lift at the time, so it was put on the back burner. I am now in a position to fund a contract developer to make this happen, so the question is: would this be a useful blueprint that would potentially be accepted? Most of the time when people want to run an emulated guest they would just nest it inside of an already running guest of the native architecture, but that severely limits observability and the task of managing any more than a handful of instances in this manner quickly becomes a tangled nightmare of networking, etc. I see real benefit in allowing this scenario to run natively so all of the tooling that exists for fleet management 'just works'. This would also be a significant differentiator for OpenStack as a whole. Thoughts? [1] http://lists.openstack.org/pipermail/openstack-operators/2018-August/015653… Chris Apsey Director | Georgia Cyber Range GEORGIA CYBER CENTER 100 Grace Hopper Lane | Augusta, Georgia | 30901 https://www.gacybercenter.org

2 2

Help with eventlet 0.26.1 and dnspython >= 2
by Thomas Goirand 04 Oct '21

04 Oct '21

Hi, As you may know, eventlet is incompatible with dnspython >= 2.0.0.0rc1. See [1] for the details. However, Debian unstable has 2.0.0. Would there be some good soul willing to help me fix this situation? I would need a patch to fix this, but I'm really not sure how to start. Cheers, Thomas Goirand (zigo)

8 15

[simplification] Making ask.openstack.org read-only
by Thierry Carrez 27 May '21

27 May '21

Hi everyone, This has been discussed several times on this mailing list in the past, but we never got to actually pull the plug. Ask.openstack.org was launched in 2013. The reason for hosting our own setup was to be able to support multiple languages, while StackOverflow rejected our proposal to have our own openstack-branded StackExchange site. The Chinese ask.o.o side never really took off. The English side also never really worked perfectly (like email alerts are hopelessly broken), but we figured it would get better with time if a big community formed around it. Fast-forward to 2020 and the instance is lacking volunteers to help run it, while the code (and our customization of it) has become more complicated to maintain. It regularly fails one way or another, and questions there often go unanswered, making us look bad. Of the top 30 users, most have abandoned the platform since 2017, leaving only Bernd Bausch actively engaging and helping moderate questions lately. We have called for volunteers several times, but the offers for help never really materialized. At the same time, people are asking OpenStack questions on StackOverflow, and sometimes getting answers there[1]. The fragmentation of the "questions" space is not helping users getting good answers. I think it's time to pull the plug, make ask.openstack.org read-only (so that links to old answers are not lost) and redirect users to the mailing-list and the "OpenStack" tag on StackOverflow. I picked StackOverflow since it seems to have the most openstack questions (2,574 on SO, 76 on SuperUser and 430 on ServerFault). We discussed that option several times, but I now proposed a change to actually make it happen: https://review.opendev.org/#/c/746497/ It's always a difficult decision to make to kill a resource, but I feel like in this case, consolidation and simplification would help. Thoughts, comments? [1] https://stackoverflow.com/questions/tagged/openstack -- Thierry

10 18

[stein][neutron] gratuitous arp
by Ignazio Cassano 13 May '21

13 May '21

Hello, I have a problem on stein neutron. When a vm migrate from one node to another I cannot ping it for several minutes. If in the vm I put a script that ping the gateway continously, the live migration works fine and I can ping it. Why this happens ? I read something about gratuitous arp. Please, help me ? Any workaround , please ? Best Regards Ignazio

3 33

[ironic] Announcing deprecation of the iSCSI deploy interface
by Dmitry Tantsur 04 May '21

04 May '21

Hi all, Following up to the previous mailing list [1] and virtual meetup [2] discussions, I would like to announce the plans to deprecate the 'iscsi' deploy interface. This is the updated plan discussed on the virtual meetup: 1) In the Victoria cycle (i.e. right now): - Fill in the detected feature gaps [3]. - Switch off the iscsi deploy interface by default. - Change [agent]image_dowload_source to HTTP by default. - Give the direct deploy a higher priority, so that it's used by default unless disabled. - Mark it as deprecated in the code (causing warnings when enabled). - Release a major version of ironic to highlight the defaults changes. 2) In the W cycle: - Keep the iscsi deploy deprecated. - Listen to operators' feedback. 3) In the X cycle - Remove the iscsi deploy completely from ironic and IPA. - Remove support code from ironic-lib with a major version bump. Please let us know if you have any questions or concerns. Dmitry [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016681.h… [2] https://etherpad.opendev.org/p/Ironic-Victoria-midcycle [3] https://storyboard.openstack.org/#!/story/2008075 -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

1 1

Re: [openstack-dev] Are we ready to put stable/ocata into extended maintenance mode?
by Alex Schultz 03 Mar '21

03 Mar '21

On Tue, Sep 18, 2018 at 1:30 PM Alex Schultz <aschultz(a)redhat.com> wrote: > > On Tue, Sep 18, 2018 at 1:27 PM, Matt Riedemann <mriedemos(a)gmail.com> wrote: > > The release page says Ocata is planned to go into extended maintenance mode > > on Aug 27 [1]. There really isn't much to this except it means we don't do > > releases for Ocata anymore [2]. There is a caveat that project teams that do > > not wish to maintain stable/ocata after this point can immediately end of > > life the branch for their project [3]. We can still run CI using tags, e.g. > > if keystone goes ocata-eol, devstack on stable/ocata can still continue to > > install from stable/ocata for nova and the ocata-eol tag for keystone. > > Having said that, if there is no undue burden on the project team keeping > > the lights on for stable/ocata, I would recommend not tagging the > > stable/ocata branch end of life at this point. > > > > So, questions that need answering are: > > > > 1. Should we cut a final release for projects with stable/ocata branches > > before going into extended maintenance mode? I tend to think "yes" to flush > > the queue of backports. In fact, [3] doesn't mention it, but the resolution > > said we'd tag the branch [4] to indicate it has entered the EM phase. > > > > 2. Are there any projects that would want to skip EM and go directly to EOL > > (yes this feels like a Monopoly question)? > > > > I believe TripleO would like to EOL instead of EM for Ocata as > indicated by the thead > http://lists.openstack.org/pipermail/openstack-dev/2018-September/134671.ht… > Bringing this backup to see what we need to do to get the stable/ocata branches ended for the TripleO projects. I'm bringing this up because we have https://review.openstack.org/#/c/647009/ which is for the upcoming rename but CI is broken and we have no interest in continue to keep the stable/ocata branches alive (or fix ci for them). Thanks, -Alex > Thanks, > -Alex > > > [1] https://releases.openstack.org/ > > [2] > > https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… > > [3] > > https://docs.openstack.org/project-team-guide/stable-branches.html#extended… > > [4] > > https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.… > > > > -- > > > > Thanks, > > > > Matt > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

5 11

[largescale-sig] OpenStack DB Archiver
by Arnaud Morin 10 Feb '21

10 Feb '21

Hello large-scalers! TLDR: we opensource a tool to help reducing size of databases. See https://github.com/ovh/osarchiver/ Few months ago, we released a tool, name osarchiver, which we are using on our production environment (at OVH) to help reduce the size of our tables in mariadb (or mysql) In fact, some tables are well know to grow very quickly. We use it, for example, to clean the OpenStack mistral database from old tasks, actions and executions which are older than a year. Another use case could be to archive some data in another table (e.g. with _archived as suffix) if they are 6 months old, and delete this data after 1 year. The source code of this tool is available here: https://github.com/ovh/osarchiver/ We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance? Feel free to contact us and/or answer this thread. Cheers, -- Arnaud, Pierre-Samuel and OVH team

12 18

[qa][tempest] Update language in tempest code base
by Arx Cruz 05 Jan '21

05 Jan '21

Hello, I would like to start a discussion regarding the topic. At this moment in time we have an opportunity to be a more open and inclusive project by eliminating outdated naming conventions from tempest codebase, such as blacklist, whitelist. We should take the opportunity and do our best to replace outdated terms with their more inclusive alternatives. As you can see in [1] the TripleO project is already working on this initiative, and I would like to work on this as well on the tempest side. Any thoughts? Shall I start with a sepc, adding deprecation warnings? [1] https://review.opendev.org/#/c/740013/1/specs/victoria/renaming_rules.rst Kind regards, -- Arx Cruz Software Engineer Red Hat EMEA <https://www.redhat.com> arxcruz(a)redhat.com @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

6 9

Re: [placement] Train upgrade warning
by Seth Tunstall 21 Dec '20

21 Dec '20

Hello, As another person has mentioned (Chanu Romain), I am attempting an upgrade from Queens to Train. I have successfully upgraded Keystone and Glance, but while trying to separate Placement's databases out from Nova, I run into the the same issue with placement-status upgrade check: # placement-status upgrade check +-----------------------------------------------------------------+ | Upgrade Check Results | +-----------------------------------------------------------------+ | Check: Missing Root Provider IDs | | Result: Success | | Details: None | +-----------------------------------------------------------------+ | Check: Incomplete Consumers | | Result: Warning | | Details: There are -3511 incomplete consumers table records for | | existing allocations. Run the "placement-manage db | | online_data_migrations" command. | +-----------------------------------------------------------------+ Likewise, running placement-manage db online_data_migrations does nothing. I am running this placement migration on a dev cluster which has had the databases from a Prod cluster restored to it. This has worked well for me previously as a way to run through upgrades without touching the live system. Thanks very much, Seth

3 10

[ptl][release][stable][EM] Extended Maintenance - Stein
by Előd Illés 26 Nov '20

26 Nov '20

Hi, As Victoria was released yesterday and we are in a less busy period, it is a good opportunity to call your attention to the following: In less than a month Stein is planned to enter into Extended Maintenance phase [1] (planned date: 2020-11-11). I have generated the list of *open* and *unreleased* changes in *stable/stein* for the follows-policy tagged repositories [2]. These lists could help the teams, who are planning to do a *final* release on Stein before moving stable/stein branches to Extended Maintenance. Feel free to edit and extend these lists to track your progress! * At the transition date the Release Team will tag the *latest* (Stein) releases of repositories with *stein-em* tag. * After the transition stable/stein will be still open for bugfixes, but there won't be any official releases. NOTE: teams, please focus on wrapping up your libraries first if there is any concern about the changes, in order to avoid broken releases! Thanks, Előd [1] https://releases.openstack.org [2] https://etherpad.openstack.org/p/stein-final-release-before-em

2 2

[kolla][ptg] Kolla Wallaby priorities
by Mark Goddard 23 Nov '20

23 Nov '20

Hi, Thanks to everyone who attended the Kolla PTG sessions, we covered a lot of ground. The Etherpad is available at [1]. I wrote an email [2] summarising the discussions. I have moved candidate work items across to a second Etherpad [2] for voting on. Now is the time to vote! Please add your name against a maximum of 12 items across the 3 deliverables (kolla, kolla-ansible, kayobe) in the priorities [2] Etherpad. Anyone in the community is welcome to vote. [1] https://etherpad.opendev.org/p/kolla-wallaby-ptg [2] http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018445.… [3] https://etherpad.opendev.org/p/kolla-wallaby-priorities Thanks, Mark

1 3

[release] Meeting Time Poll
by Herve Beraud 19 Nov '20

19 Nov '20

Hello We have a few regular attendees of the Release Management meeting who have conflicts with the current meeting time. As a result, we would like to find a new time to hold the meeting. I've created a Doodle poll[1] for everyone to give their input on times. It's mostly limited to times that reasonably overlap the working day in the US and Europe since that's where most of our attendees are located. If you attend the Release Management meeting, please fill out the poll so we can hopefully find a time that works better for everyone. Thanks! [1] https://doodle.com/poll/7zgvfb8ne54bwhzw -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 1

[SDK][OSC] Meeting Times
by Kendall Nelson 16 Nov '20

16 Nov '20

Hello! In the PTG session today we talked about doing some sort of regular meetings to kind of jump start things after the more recent lack of activity. We discussed beginning with more frequent meetings (a couple across the next few weeks) and then shifting to a more relaxed monthly cadence. I propose 15:00 UTC on the third thursday of each month as our monthly meeting time? If people agree, I will propose a patch :) As for the more frequent ones to keep the momentum from discussion from this week/last week. We could use that time (15 UTC) and date (Thursdays) and maybe just do them on Nov 12 and 19th to keep us moving? That gives us next week to catch up/ take a breath from the last two weeks, but not waiting till after the US Thanksgiving holiday. Let me know what you think! -Kendall Nelson (diablo_rojo)

4 6

CloudKitty deployment on Train and/or Ussuri
by Thode Jocelyn 15 Nov '20

15 Nov '20

Hi, I was looking at the possibilities to have Rating-as-A-Service in our Openstack (currently train but planning to move to Ussuri) and I stumbled upon CloudKitty. I'm currently facing multiple issues with cloudkitty itself and with openstack-ansible. (We are using a containerized deployment with LXC) * I noticed that openstack-ansible has no playbook yet to install a service like CloudKitty even though starting from Ussuri it was added in the ansible-role-requirements.yml. I have created a small patch to be able to install CloudKitty via openstack-ansible and if this is of interest I could potentially submit a PR with this. * The os_cloudkitty role seems to kinda work and only with a relatively old version of CloudKitty. For example, here : https://opendev.org/openstack/openstack-ansible-os_cloudkitty/src/branch/st… since CloudKitty v8.0.0 the section should be named "[fetcher_keystone]", but I found no information as to which CloudKitty version should be used with this role. Does someone have any recommendation as to which version should be used and if there is any interest in improving the role to support more recent versions of CloudKitty? * Even when tweaking the configuration and installing v7.0.0 of Cloudkitty it only fixes cloudkitty-processor, I was never able to make cloudkitty-api work I always get an error like "Fatal Python error: Py_Initialize: Unable to get the locale encoding ModuleNotFoundError: No module named 'encodings'". Any input on this would be greatly appreciated. Cheers Jocelyn

4 5

[neutron][stable]Proposing Rodolfo Alonso Hernandez as a stable branches core reviewer
by Slawek Kaplonski 06 Nov '20

06 Nov '20

Hi, I would like to propose Rodolfo Alonso Hernandez (ralonsoh) to be new member of the Neutron stable core team. Rodolfo works in Neutron since very long time. He is core reviewer in master branch already. During last few cycles he also proved his ability to help with stable branches. He is proposing many backports from master to our stable branches as well as doing reviews of other backports. He has knowledge about stable branches policies. I think that he will be great addition to our (not too big) stable core team. I will open this nomination open for a week to get feedback about it. -- Slawek Kaplonski Principal Software Engineer Red Hat

6 6

[openstack][ptg][InteropWG] : Conclusion of InteropWG session 10/26 - next one will be 10/30
by prakash RAMCHANDRAN 06 Nov '20

06 Nov '20

Hi all My apology on the delayed start and here is the items we went over on first session. Refer : https://etherpad.opendev.org/p/interop-wallaby-ptg (Monday 10/26 recorded the meeting for everyone to see, its cloud recording and will know from PTG hosts where they have that?) Summary:1. Plan to Complete Victoria guidelines specs for interop in 2-6 weeks depending on adding new [OpenStack Powered File System Branding as Add-on Program] - last Ussuri guidelines are at https://opendev.org/osf/interop/src/branch/master/2020.06.json New one will be 2020.10.json to be added - For existing integrated modules [ Core OpenStack services including identity, compute, networking, block storage, and object storage ] - for two existing add-ons [Orchestration (heat) , DNS (designate) "Not a core capability, add-on program is available" - This is for Manila Filesystem add-on https://etherpad.opendev.org/p/wallaby-ptg-manila-interop "Not a core capability, add-on program to be made available" We have one more opportunity on 10/30 slot at the end of PTG if BMaaS (Ironic) if Julia has any proposals, besides (Open Infra Kata + Docker) Container has additional proposal community is working on. Look forward to seeing you all on October 30 as what the Road-map will look like for 2020 and 2021 going forward. ThanksPrakash On Monday, October 26, 2020, 06:09:00 AM PDT, Goutham Pacha Ravi <gouthampravi(a)gmail.com> wrote: Hi Prakash, We're here: https://zoom.us/j/92649902134?pwd=a01aMXl6ZlNEZDlsMjJMTGNMVUp1UT09 On Mon, Oct 26, 2020 at 6:06 AM prakash RAMCHANDRAN <pramchan(a)yahoo.com> wrote: https://meetpad.opendev.org/Interop-WG-weekly-meeting

3 5

Editing libvirt xml file
by vladimir franciz blando 04 Nov '20

04 Nov '20

Hi, I edited the libvirt xml (virsh edit...) file of a running Windows server instance because I changed the video model type from cirrus to vmvga so it can support a higher resolution on a console (cirrus supports up to 1280 only). After editing, i "soft-rebooted" the instance and the new configuration sticks. That's awesome. But when you do a "hard-reboot" it will revert back to the original config. I also tried to do a "shut-down" then "start" again, but it still reads the original config. Not sure where it is reading the config from... Regards Vlad

3 5

[ptl] Victoria Release Community Meeting
by helena＠openstack.org 04 Nov '20

04 Nov '20

Hi Everyone, We are looking to do a community meeting following The Open Infrastructure Summit and PTG to discuss the Victoria Release. If you’re a PTL, please let me know if you’re interested in doing a prerecorded explanation of your project’s key features for the release. We will show a compilation of these recordings at the community meeting and follow it with a live Q&A session. Post community meeting we will have this recording live in the project navigator. Cheers, Helena

8 10

[tripleo][ci] Monday Nov 2
by Wesley Hayutin 04 Nov '20

04 Nov '20

Greetings, The tripleo ci team has identified a handful of patches that we'd like to land prior to Nov 2, the day docker.io goes away. We've hit some new bugs and have also tuned a few things to try and make sure we can get patches to merge. Our current focus is across master, victoria, ussuri and centos-8 train, and queens while reducing coverage in rocky and stein. A list of the prioritized gerrit reviews can be found here: https://hackmd.io/MlbZ_izSTEuZsCWTJvu_Kg?view The entire topic can be found here: https://review.opendev.org/#/q/topic:new-ci-job Thanks all.

4 4

[placement][nova][cinder][neutron][blazar] Placement governance switch(back)
by Stephen Finucane 03 Nov '20

03 Nov '20

Placement has been a separate project with its own governance since Stein [1]. Since then, the main drivers behind the separation have moved onto pastures new and with Tetsuro sadly declaring his non- candidacy for the PTL position for Wallaby [2], we're left in the unenviable position of potentially not having a PTL for the Wallaby cycle. As such, it's probably time to discuss the future of Placement governance. Assuming no one steps forward for the Placement PTL role, it would appear to me that we have two options. Either we look at transitioning Placement to a PTL-less project, or we move it back under nova governance. To be honest, given how important placement is to nova and other projects now, I'm uncomfortable with the idea of not having a point person who is ultimately responsible for things like cutting a release (yes, delegation is encouraged but someone needs to herd the cats). At the same time, I do realize that placement is used by more that nova now so nova cores and what's left of the separate placement core team shouldn't be the only ones making this decision. So, assuming the worst happens and placement is left without a PTL for Victoria, what do we want to do? Stephen PS: Apologies if I missed other projects with an interest in placement. I did try to catch them all /o\ [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002575… [2] http://lists.openstack.org/pipermail/openstack-discuss/2020-September/01749…

8 10

[tripleo] please avoid creating ansible playbooks that call modules for trivial tasks
by Bogdan Dobrelya 02 Nov '20

02 Nov '20

Hi, In today TripleO client development we invoke ansible playbooks instead of Mistral workflows. I believe this may only be justified for non-trivial things. Otherwise please just import python modules into the client command actions without introducing additional moving parts in the form of ansible modules (that import the same python modules as well). Ansible brings declarative DSL that sometimes can better and simpler and faster describes things to do and shape inputs data, than if the same would be done in python, in the client. What should be the trivial thing and non-trivial then? Well, I think that example [0] illustrates such a trivial thing. A new playbook [1] would only install a few packages then "returns" back into python and calls a new ansible module [2]. And that new module only wraps [3] the same code being removed from the client: from tripleo_common.image import build ... manager = build.ImageBuildManager(<args>) I see no point of adding a new playbook and a new module for that trivial example. Those 4 packages could (and should) be as well installed from the client caller code without any ansible involved in the middle IMO. [0] https://review.opendev.org/#/c/759954/ [1] https://review.opendev.org/#/c/759953/3/tripleo_ansible/playbooks/cli-overc… [2] https://review.opendev.org/#/c/759953/3/tripleo_ansible/ansible_plugins/mod… -- Best regards, Bogdan Dobrelya, Irc #bogdando

6 9

[tripleo] stable/victoria for tripleo repos
by Marios Andreou 02 Nov '20

02 Nov '20

Hello all, I got some pings about tripleo stable/victoria so here is an update. The tripleo-ci team is busy preparing and we are almost ready. Basically we've blocked branching until we have adequate CI coverage. This means check & gate jobs but also promotion jobs so that check & gate aren't running with stale content. tl;dr We are aiming to branch victoria tripleoclient & common today and everything else next week - with more details below but please note: PLEASE AVOID MERGING to the stable/victoria tripleoclient/common that will appear after [4] until we are confident that our ci coverage is working well. Status: Most of our 'core' (standalone/scenarios/undercloud etc) check/gate jobs match on branch like [1] so we expect those to run OK once there is a stable/victoria available. There is ongoing work for branchful, upgrade, 3rd party and ovb jobs but we will not block branching on those. The victoria integration pipeline [2] (which produces current-tripleo) is in good shape. It was blocked on a nit in the component pipeline but it is resolved with [3]. Once we have a victoria promotion then we should be all clear to branch stable/victoria. Actions: A). today: make stable/victoria for python-tripleoclient & tripleo-common. The change at [4] will do this once merged. We will use this to see our check/gate jobs run against victoria & fix issues. B). next week: once we are happy with A). AND we've had a promotion, we release and make stable/victoria for everything else. Hopefully B). will happen end of next week but that depends on how well the above goes (plus inevitable delays cos PTG), I'll repeat this cos it's important: PLEASE AVOID MERGING to stable/victoria tripleoclient/common for now until we are confident that our jobs are doing the right thing! As always grateful for a sanity check or any thoughts about any of the above - I'm pretty new to this (weshay has been a great help so far thank you weshay++), thanks, marios [1] https://opendev.org/openstack/tripleo-ci/src/commit/d5514028452f9d427949f5a… [2] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integ… [3] https://review.rdoproject.org/r/#/c/30654/ [4] https://review.opendev.org/#/c/759449/

1 2

Python 3.9 is here in Debian Sid
by Thomas Goirand 02 Nov '20

02 Nov '20

Hi there! Matthias Klose decided that uploading Python 3.9 on the 14th of this month (yes, the same day of the OpenStack release) would be fun. I am now really having fun ... fixing bugs! :) What's not so funny for me, is that some key packages like Greenlet or lxml aren't built yet for Python 3.9, even though they should. As a consequence, I just need to wait to be able to upload to Unstable. So this will delay my usual announcement for the release GA in Debian (even though the unofficial Victoria repo for Buster is ready and I could spawn VMs on Buster + Victoria). Anyways, all of this will settle slowly. Though Python 3.9 is here to stay, and to soon reach Ubuntu as well (it's planned for the next release). It'd be nice if projects were starting to investigate gating on Python 3.9 as early as possible. As much as I can tell, there's not so many issues to fix. The first one I fixed is here: https://review.opendev.org/758237 but I don't expect much more, and hopefully, I'll be able to propose more patch as I find the issues. Must we wait until Python 3.9 reaches Ubuntu to enable gating in the OpenStack projects? FYI, I do expect Victoria to run over Python 3.9 in Bullseye. Please don't reply with the usual "Victoria doesn't support it", because that's irrelevant. I'm fine with backporting patches, as long as they also exist in Wallaby. Cheers, Thomas Goirand (zigo)

7 9

[magnum] hyperkube deprecation situation
by Ionut Biru 02 Nov '20

02 Nov '20

Hi guys, I was hoping that in Victoria there will be a decision regarding this topic. Currently, if I use vanilla magnum, I'm stuck on 1.18.6 being the latest version, even so 1.18.10 was released. We are kinda stuck on a version that maybe in the near future is pruned to security issues. How do other operators are using magnum (vanilla) and keeping their kubernetes up to date? Are operators using a forked version of https://review.opendev.org/#/c/752254/ ? -- Ionut Biru - https://fleio.com

2 2

Fwd: [MURANO] DevStack Murano "yaql_function" Error
by İzzettin Erdem 01 Nov '20

01 Nov '20

Thanks Andy. Regards. Forwarded Conversation Subject: [MURANO] DevStack Murano "yaql_function" Error ------------------------ Gönderen: Andy Botting <andy(a)andybotting.com> Date: 30 Nis 2020 Per, 14:13 To: İzzettin Erdem <root.mch(a)gmail.com> Cc: Rong Zhu <aaronzhu1121(a)gmail.com>, < openstack-discuss(a)lists.openstack.org> Hi İzzettin, Thanks for your interest. Complete log link is at below. It is happening > when i try to deploy or update an environment. > > http://paste.openstack.org/show/792444/ > I've encountered the error AttributeError: 'method' object has no attribute '__yaql_function__' in our environment recently. I believe it started after we upgraded to the Stein release. I haven't tracked it down yet, but calls to the YAQL function: std:Project.getEnvironmentOwner() in a Murano app seem to generate it. It's good to know that at least someone else has reproduced this. cheers, Andy ---------- Gönderen: İzzettin Erdem <root.mch(a)gmail.com> Date: 2 May 2020 Cmt, 16:08 To: Andy Botting <andy(a)andybotting.com> Hi Andy, I solved this problem, it is happening when you try to create or update an environment on existing network. If you change the network with "create new" parameter, it stuck on "creating -or updating- environment" state and after that throws connection error. So i searched Murano network errors and i found this: https://docs.oracle.com/cd/E73172_01/E73173/html/issue-21976631.html. I applied these steps and i created successfully an environment just only with "create new" network parameter. After all these, i realized my devstack environment has no internet connection and because of this murano-agent could not install. So i tried to install Murano on my OSA stable/train test environment and i came with success until last step. Murano-agent installed but this time agent could not connect to rabbitmq instance. I discussed this with OSA IRC channel and i think i have to struggle with this for a long time. Apologize for this long mail but i wanted to inform you. Thanks. Regards. ---------- Gönderen: Andy Botting <andy(a)andybotting.com> Date: 3 May 2020 Paz, 14:24 To: İzzettin Erdem <root.mch(a)gmail.com> Hi İzzettin, Glad you got it sorted. I think you've probably just worked around the issue. Using your new method, you're probably just avoiding the std:Project.getEnvironmentOwner YAQL call now. I've also worked around it in our environment, but I'd like to solve it properly. When I get some time to look into it properly, I'll let you know. cheers, Andy

3 3

[tc][all][ Wallaby Virtual PTG Summary
by Ghanshyam Mann 31 Oct '20

31 Oct '20

Hello Everyone, I am summarizing the TC PTG discussion and action items. Wallaby Leaderless assignment --------------------------------------- Four projects are still pending for leadership assignment. We discussed the next step. If you would like to help any of these or using in your cloud/distro, this is the right time to step up: 1. Karbor - Action item: diablo_rojo will start the retirement process. 2. Qinling - Action item: gmann to start the retirement process. 3. Searchlight -Action item: gmann to start the retirement process. 4. Placement - Action item: gibi will update TC about the final discussion and accordingly mnaser to propose the patch in governance. Victoria Retrospective: ----------------------------- There are a couple of things TC finished in last cycle, few of them are: * Reduced size of TC to speed up the things. * Updated policy to add projects faster. * Outlined Distributed Project Leadership Model as an alternate leadership solution of PTL. Oslo will be the first project to try this new model. * Normalized retirement processes. All the retired repos cleanup is finished. * Tag cleanups & additions This is not complete yet but started with tc-approved release tag in the Victoria cycle. * Merged UC with TC. This is one of the key things finished and the best strategy to make users-developers work more closely. TC Meeting time: ----------------- TC is going to try (re-try) the weekly meeting every Thursday 15:00 UTC. * Action Items: mnaser: propose patch. TC policy regarding OpenStack client: ----------------------------------------------- Forum Discussion: https://etherpad.opendev.org/p/vSummit2020_FeatureGap This is one of the important items and still no good progress on this. From TC perspective, we talked about how to ask or motivate projects to migrate to OSC. TC needs to be very clear on whether it's a strict policy or just guidelines. After long discussion on this we are going with the below strategy: These strategies will be documented as TC resolution. * All user docs should have `openstack` client commands. * Focus on user docs first and then ops docs later * All ci-tooling should have `openstack` client usage * Use the process to find what is missing so we can prioritize it. * Action items: - diablo_rojo to work on initial resolution proposal[1] Finalize the Wallaby cycle goal selection: --------------------------------------------------- There are two proposals for the Wallaby community-wide goals. First is 'oslo.rootwrap to oslo.privsep' which is already being discussed in the last cycle also and all set to be selected[2]. 2nd proposal came up during PTG itself. During policy-popup PTG[3], it came up that deprecating the JSON formatted of policy file will be a good advance step before the projects move to new RBAC policies. This will help operators to smoothly migrate to new policies. This does not involve much work and ok to select as 2nd goal for Wallaby cycle, TC agreed to have below goals for Wallaby cycle: * Migrate from oslo.rootwrap to oslo.privsep [2] * Migrate Default Policy Format from JSON to YAML [4] *Action: - TC to merge the patches which selects both goals for Wallaby cycle. TC tag cleanup: ------------------- In the Victoria cycle, we started the process to audit all the TC tags and start cleanup those. We removed the 'tc:approved-release' tag in the Victoria cycle[5]. In this PTG we discussed two more tags. 1. assert:supports-zero-downtime-upgrade: Currently, there is no project who has this tag and also no testing framework available. Testing for zero downtime is not so easy in upstream testing. We decided to remove this tag as it is advertising something we aren't doing. If anyone interested to spend time on this in the future then we can add it after projects start testing it and document it. 2. assert:supports-api-interoperability: This tag is whether project API is interoperable also or not and this is important from the interop trademark program also. We only have Nova having this tag. Our goal is to promote more projects to apply for this tag. During the discussion, we found that we need to clarify this tag more clearly. For example, this tag is not about implementing the microversion but any versioning schema which provides the feature (API changes) discoverability. And also it is about how we change the API not how our APIs are currently. As long as services have some versioning mechanism to discover the changes and follow the API SIG guidelines for interoperability, and test it via branchless testing way, that service is applicable to apply for this tag. TC will document this tag in a more clear way and encourage each project to start working on applying this tag. * Action: - Graham to put up a change to update the wording to allow for discovery mechanisms[6] - Should include the release in which they started to assert this tag. - also documents that it's about following the guidelines for future changes not existing one. - gmann - Should socialize this out to the projects after we get the documentation improvements landed. - gmann k8s Community Bridging : -------------------------------- This is one of the exciting discussion for everyone. We hosted this cross-community meeting with Kubernetes steering committee teams. Bob and dims from the Kubernetes steering committee joined us in this discussion. It was started with a quick introduction from both teams and then started the discussion on the below topics [11] * Sharing governance models: k8s governance hierarchy is LF->CNCF->k8s steering commitee->various SIG/working group. k8s Steering the committee consists of 7 elected members and doesn't actually influence the direction of the code instead leave it to SIG and arch committee. There is no chair in this committee and host biweekly private as well as public meetings. Each SIG (repos) team has approver and reviewer roles where reviewer review the code and approver are responsible for merging the code. Naser explained the OpenStack governance model. * What are your 3 biggest issues that are barriers to growth/sustainability as a community? as a project? - Going up the contribution ladder is hard. - Stale/emeritus membership in reviewer/approver level groups. - Mono repo makes it so that each SIG involved needs to review which can slow things down. * Area/Domain mapping vs service/repo centered teams - mono repo forces people to work together and it's a little more diverse who is reviewing - mono repo is bad because sometimes things need to be in multiple places like documentation - mono repo struggles with libraries especially on testing the external one. * Related Ecosystem projects comparisons The SIG lead has to sign off and accept that they are willing to take ownership + handle maintenance + releasing etc. It is general consensus that we try to distribute much work as possible to the subgroup and keep it out of k/k. There is general consensus across k8s leadership that work should be delegated out to subgroups. For API interoperability challenge in a distributed model, k8s has conformance tests which exercise the API performance and vendors try and upload conformance results every release or dot release. * Common FOSS topics release and CI part was discussed. and also on how COVID things are impacting community health. k8s community almost lost their independents or part-time contributors. also, the k8s community is doing 3 releases per year compared to 4. * Staying connected going forward To stay connected it will be a good idea if we extend an invite for K8s to join PTG. Upstream Investment Opportunities: ----------------------------------------------- We have three upstream opportunities defined for 2020[1] but there is no help on any of these, even in previous (2018, 2019) upstream opportunities also. We started the discussion if we need to continue this for 2021 years also or just stop defining it and decide the area to help when we have someone interested. Before deciding anything on this mnaser will discuss this with the board of directors and get their opinion. * Action Items: - mnaser to find ways to engage with members and talk with them. - mnaser to talk to board about infrastructure - mnaser to talk to board about platinum members contribution levels + what are they doing to drive engagement. Pop Up Team Check In: ----------------------------- Currently, we have two active popup teams 1. policy 2. Encryption. TC checked the progress on both teams. The policy team is very active and finished some work in Voctoria cycle (cyborg finished it and Barbican started) and also hosted forum[8], PTG sessions[9], and discussed Wallaby development plan. This team host biweekly meeting to discuss and review the progress. The encryption team is also active. Josephine explained the progress on this. Glance spec is merged. Both teams will continue in Wallaby cycle also. Completing the Gerrit breach audit : ---------------------------------------------- There are still 19 teams pending to finish this audit. Those are listed in etherpad - https://etherpad.opendev.org/p/code-audit-gerrit-breach-tracker We encourage all those pending teams to finish the audit, TC members will start following with those projects every week. * Action: - TC to follow up every week on progress Other topics: ----------------- We ran out of time and all the pending (below) topics will be discussed in TC regular meetings. TC will skip this month's meeting[10] but will have weekly meetings on 12th Nov onwards. *Better way to test defined testing runtime *Monitoring in OpenStack: Ceilometer + Telemetry + Gnocchi state *Stable core team maintainer and process to recruit the new members and adding core team members in stable core list. Detailed discussions are in this Etherpad: https://etherpad.opendev.org/p/tc-wallaby-ptg [1] https://review.opendev.org/759904 [2] https://review.opendev.org/#/c/755590/ [3] https://etherpad.opendev.org/p/policy-popup-wallaby-ptg [4] https://review.opendev.org/#/c/759881/ [5] http://lists.openstack.org/pipermail/openstack-discuss/2020-October/017786.… [6] https://review.opendev.org/760562 [7] https://governance.openstack.org/tc/reference/upstream-investment-opportuni… [8] https://etherpad.opendev.org/p/consistent-and-secure-default-policies-walla… [9] https://etherpad.opendev.org/p/policy-popup-wallaby-ptg [10] http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018439.… [11] https://etherpad.opendev.org/p/kubernetes-cross-community-topics -gmann

1 0

devstack installation error - [qa-octavia]
by Ratnopam Chakrabarti 31 Oct '20

31 Oct '20

I tried installing devstack in ubuntu 20.04 with no luck. Are the instructions > at https://docs.openstack.org/devstack/latest/ and > https://docs.openstack.org/devstack/latest/guides/devstack-with-lbaas-v2.ht… > still valid? Could anyone please confirm?

2 3

Weird iptables problem
by frommel＠gmail.com 31 Oct '20

31 Oct '20

Hi, so we deployed a test cluster with 8 total nodes. Openstack ussuri, Ubuntu 20.04 , opevswitch, octavia. Everything works more or less as it should , however when deploying a loadbalancer (which works perfectly), instances on the same node as the amphora deployment loose all access via public up after open switch logs a message : modified security group “long uuid”, which is a sec group belonging to octavia. What effect does that have with the rest? Questions: Should ufw be on and started? How can we troubleshoot this more in depth? Why does the lb work perfectly whole instance with floating up loos all access from the internet but the lb works fine passing thru the web services. One more thing. Does anyone have a recommended sysctl.conf Settings list for Ubuntu/ openstack ussuri? Thanks already and have a nice weekend, //f

1 0

[magnum] Keystone trust credentials failing authentication (after previously working)
by Paul Browne 30 Oct '20

30 Oct '20

Hello Magnum users and devs, I was wondering if any may be able to provide some clues on a mystifying problem we've been seeing with some Magnum-deployed k8s clusters, of older deployment date (v1.17, deployed ~6 months ago) and Keystone trusts used to manage OpenStack cloud resources. I've attached a copy of the Magnum template for the k8s cluster to give an idea of its initial environment. The issue we've been seeing is that the Keystone trust generated at Magnum cluster creation time seems no longer be usable to successfully authenticate to the OpenStack APIs, and so operations such as creation of Manila shares used as k8s Persistent Volume Claims then get stuck in Pending state, forever. The strange thing is that this doesn't happen for newly created Magnum clusters of the same template (in attached file, I use trust credentials for a newly deployed and old cluster; OpenStack API calls for new credentials succeed, whereas old cluster trust fails). Unfortunately following debug-level Keystone logging for rejected trust auth attempts hasn't led me much further to understanding the root issue; the Magnum generated trust exists and has no expiry, but it still seems to be rejected. Keystone debug logging doesn't seem to give much indication as to *why* it is being rejected or what policy violations may be involved, but perhaps there's details hidden in there I'm not seeing yet. Does anyone have any clues as to why this may be happening, or any advice on how we may be able to replace or refresh the trust that the Magnum-created k8s cluster is expecting to use in calling OpenStack APIs? Many thanks, Paul -- ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk Tel: 0044-1223-746548 *******************

1 0

[kolla] Wallaby PTG summary
by Mark Goddard 30 Oct '20

30 Oct '20

Hi, Thanks to everyone who joined the virtual PTG sessions this week. We had some great discussions covering many different topics. I will try to provide a summary here. # Victoria cycle retrospective The Ubuntu Focal migration went smoothly. Core reviewer bandwidth was hurt by the pandemic, but we did pick up towards the end of the cycle and landed a good number of features. As usual, lots of CI fires to extinguish. # Review Victoria PTG Actions We looked back at the actions from the last PTG. Quite a few are still incomplete - we should do a better job of reviewing these. We agreed that the docs-related actions are still the most important. Any help with improving our documentation is appreciated. # General topics ## Ability to run CI jobs locally (without Zuul, but possibly with Ansible) This depends on a Zuul feature called Zuul Runner [1], which should be ready soon. yoctozepto signed up to test it out and document how to use it with Kolla. ## Future of Kolla Klub and Kall The Kolla Klub started well, with lots of involvement, and some good sessions. Over time, attendance has fallen, and the agenda still seems to be largely driven by me. Keeping up productive sessions every two weeks is starting to feel like a bit of a chore, and I think we need to change things up or stop. We agreed to try rebooting at a time that better suits contributors in APAC, possibly recycling some earlier material. There was a proposal at the summit to have monthly ops meetups, and maybe we would do well to link these related communities. We had a few ideas for how to improve the productivity of the more development oriented Kolla Kall, including multiple smaller groups, pair/mob programming, and fewer longer sessions. ## Keep or drop lower-constraints jobs? We didn't find many reasons to keep the lower-constraints jobs, and they do seem to need a bit of maintenance. We voted 2 in favour to remove them, with everyone else not having strong feelings. ## How can we integrate better with our user & contributor base outside of EU/US timezones? Kolla has a large user base outside of EU/US timezones, but the core team and surrounding community is not always that effective at involving these users. We agreed to try to improve the situation by reaching out to known community members in other regions, and offering some limited mentorship. # Kolla ## Infra images This feature has been in progress for a few cycles, but is proving difficult to land. The main blocker now is in CI. We need to split our image build CI jobs into several pipelines. They need to share a local docker registry. This pattern should open up a few other interesting features. ## New Docker Hub limits Docker hub is changing its policy for anonymous and free users. This affects retention of unused images, and limits the number of pulls in a 6 hour period (100 for anonymous users, limited by IP, 200 for logged in free users). This is likely to affect users running without a local registry. Fl1nt agreed to add some information to our documentation to help users. This is likely also to impact our CI jobs, which need to pull images. There is an application process to be listed as an open source project, which we need to ask the opendev infra team to apply for. # Kolla Ansible ## More fine grained skipping of tasks, e.g. allow to skip service registration priteau proposed making it easier to skip certain tasks during deployment, such as bootstrapping or service registration. There are two obvious ways to do this - a new 'action' (e.g. deploy-without-register), or add tags to the relevant tasks, which may be skipped. A third approach is to combine two commands - genconfig & deploy-containers. This is related to work currently going to allow configuration to be generated separately from container deployment. In the end, we considered adding tags, as well as the two command approach. ## Deprecate reconfigure command? The reconfigure command is now essentially the same as deploy. This is a source of confusion for users, so we agreed to deprecate reconfigure. We will keep reconfigure as an alias for deploy at the CLI level. ## Modernise the old skool Swift role The Swift role is the last (except Bifrost, which is generally weird), to use the old format, prior to the better-reconfigure blueprint. This has some limitations, so we should bring it into line. Nothing controversial here, just needs someone to pick it up. ## DNS-based endpoint naming rafaelweingartne has proposed a mini spec [2] for DNS-based endpoint naming. I think we now have a reasonable idea of his solution (which is to use Consul for service discovery, and avoid HAProxy for internal API communication), however the hard part is defining the problem that is being solved. ## Support to identity federation (OpenID Connect) configurations in Keystone and Horizon via Kolla-ansible There is a patch [3] to support this which has been open for some time. It requires some fairly specific knowledge to review, and the challenge will be in making core reviewers confident that it works. We agreed that CI testing and/or documentation of a simple development environment would help. ## podman Podman keeps coming up as a replacement for Docker. There are several issues here, including support on Debian/Ubuntu - we do not want to maintain support for two container engines. We agreed that this is likely to be a large feature, and that we should get started with a proof of concept in Wallaby, possibly with a spec to tease out the details. Fl1nt may have some time early 2021 for this, but anyone is welcome to pick it up. ## Finish Masakari integration We don't have support for the host monitor, which is arguably the most useful part of Masakari. With yoctozepto as Masakari PTL, we expect to see good progress in Wallaby. ## Performance improvements We made some good progress with performance improvement in Victoria. There is always more that can be done, and the next phase will probably involve splitting roles that target both control and compute hosts. Mitogen [4] is also an interesting option. As always, changes should be based on benchmarks [5]. ## Let's Encrypt integration - container running certbot, triggers certificate distribution yoctozepto added initial support for the ACME protocol in Ussuri, and the next step is to support deployment of a certbot container, and unattended certificate rotation. There are lots of fiddly details here, including how to bootstrap it. headphoneJames offered to write up a high level description of how the pieces fit together. ## Support for HAProxy reload (e.g. for TLS) Currently any configuration changes to HAProxy require a restart to be applied. This causes connections to be dropped. We discussed adding support for hitless reloads of HAProxy [6]. ## Native fluentd logging We didn't discuss this as kevko was busy, but I thought I'd add it here anyway. The idea is to use use logging.conf to configure services for native fluentd logging. This avoids all the regexes in fluentd, but may change our document schema in Elasticsearch. Patch [7] is up for review. # Kayobe ## Support multiple environments from a single kayobe configuration This feature has been on the cards for some time, and is an important step for Kayobe. priteau has proposed a patch [8] which is a good starting point. We spent a good hour going over some of the outstanding issues, and made some good progress. We agreed on the need to keep the meaning of KAYOBE_CONFIG_PATH intact, and to allow a branch to define the default environment to use. We agreed to write a high level design document to pull out the remaining uncertainties, and aim to get it over the line in Wallaby. ## Switching Docker storage from devicemapper to overlay2 We switched the default Docker storage driver from devicemapper to overlay2 in Victoria. For existing users of devicemapper, the plan is to try to ensure they are migrated to overlay2 during CentOS 8 migration. Further, we will deprecate devicemapper on stable branches, and require users to take some action (set a variable) to allow its use. This should deter new users from using devicemapper, without completely breaking existing users. # Switch to NetworkManager? jovial proposed this discussion, as he likes the checkpoint/restore feature of NetworkManager. There are various ways to interface with NM, including nm-cli, an ansible module, templated config files + nmcli con reload. ## Strip out Grafana post configure functionality and move it to Kolla-Ansible dougszu proposed this topic, to improve the separation of concerns between Kayobe and Kolla Ansible. It may also apply to setup we do for ironic. ## Create a reference custom playbook repo We have a stackhpc repo [9] with various useful playbooks and scripts. dougszu proposed making something more official. I suggested that Ansible collections could be used to improve the user experience with custom playbooks in external repos. ## Config gen on ansible control host This is something we've talked about for a while - offline generation of configuration, and potentially validating it in a CI pipeline. jovial noted the importance of being able to redact passwords. This is mostly dependent on ongoing changes to Kolla Ansible, but could be wrapped up nicely by Kayobe. ## Hashivault integration While we have integrated with Hashicorp vault for secret retrieval, the user experience could be better. We also lack support for pushing passwords generated by kolla up to Vault. This is likely to be a Kolla Ansible feature, plus some Kayobe integration. [1] https://review.opendev.org/681277 [2] https://review.opendev.org/759706 [3] https://review.opendev.org/695432 [4] https://github.com/dw/mitogen [5] https://github.com/stackhpc/ansible-scaling [6] https://www.haproxy.com/blog/hitless-reloads-with-haproxy-howto/ [7] https://review.opendev.org/755775 [8] https://review.opendev.org/734867 [9] https://github.com/stackhpc/kayobe-ops Cheers, Mark

1 0

Devstack installation failure
by Ratnopam Chakrabarti 30 Oct '20

30 Oct '20

Hi, Has anyone successfully installed devstack in last 2 weeks with stack.sh script?It was working well for me before that. I am seeing failures when I try to install on Ubuntu 18.04. Here's the trace - [Call Trace] ./stack.sh:1304:start_glance /home/stack/devstack/lib/glance:480:die [ERROR] /home/stack/devstack/lib/glance:480 g-api did not start Error on exit nova-compute: no process found neutron-dhcp-agent: no process found neutron-l3-agent: no process found neutron-metadata-agent: no process found neutron-openvswitch-agent: no process found git repo i am using to download devstack is https://git.openstack.org/openstack-dev/devstackI tried with with stable/victoria branch also, same issue. Any idea on how to solve this? Thanks

2 1

[tc] monthly meeting skip
by Mohammed Naser 30 Oct '20

30 Oct '20

Hello everyone, Our monthly meeting was scheduled for next Thursday, Nov 5th. Since the PTG took place this week, we will be skipping this month’s meeting and consider the PTG as our monthly meeting instead. Thanks, Regards, Mohammed -- Mohammed Naser VEXXHOST, Inc.

1 0

[ansible][tripleo][openstack-ansible-modules] Openstack Ansible modules PTG wrap up
by Sagi Shnaidman 30 Oct '20

30 Oct '20

Openstack Ansible modules PTG wrap up 1. We continue to align new features in SDK, OSC and modules a) need to complete switch of cloud layer to use ONLY proxy layer and not reimplement the APIs b) in Ansible modules we'd better to use cloud layer when it's available and not proxy layer 2. For exposing SDK logs when running modules, we need to find a right place to print them. Either it would be one of module returns or special plugin printing logs with more verbosity. The discussion will be in the maillist. 3. We will start developing unit tests for modules and run them with ansible-test. 4. Also need to rework our current test system and make it more atomic and add coverage. 5. We decided to align all *_info modules to one standard - <resource>_info, some of them named as <resource>s_info. 6. We'll need to generate documentation for modules and place it on docs.openstack.org if it's possible. Current documentation in Ansible covers the release that is built-in in Ansible and doesn't reflect last changes. 7. Effort of moving modules to use a single OpenstackModule class will continue. 8. Transition and adding new baremetal modules will continue as well. Thanks -- Best regards Sagi Shnaidman

1 0

Re: [Openstack][cinder] dell unity iscsi faulty devices
by Gorka Eguileor 30 Oct '20

30 Oct '20

On 30/10, Ignazio Cassano wrote: > Please, se the last email where I upgraded to the last openstack nova on > queens. > [root@compute-0 nova]# rpm -qa|grep queens > centos-release-openstack-queens-1-2.el7.centos.noarch > [root@compute-0 nova]# rpm -qa|grep nova > openstack-nova-compute-17.0.13-1.el7.noarch > openstack-nova-common-17.0.13-1.el7.noarch > python-nova-17.0.13-1.el7.noarch > python2-novaclient-10.1.0-1.el7.noarch > > I sent you the logs on the update release. > I am not so skilled for reading fine logs output about this issue. > Sorry > Ignazio > Hi, I missed that email and the attachment, sorry. The logs you sent me were missing most of the connect_volume call, and only the end of the call was present, but I think it doesn't matter as I see what the problem is. The problem is that some of the nodes and sessions are duplicated. An example of a duplicated node: tcp: [3] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) tcp: [4] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) An example of that node's duplicated session: tcp: [3] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) tcp: [4] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) And os-brick is not prepared to handle that, because it is programmed to reuse the nodes and sesssions. So on disconnect it get's the first of each to look for the volumes provided by it. In the example of the duplicated node-session aboveit sees that it provides /dev/sdd, but that is not one of the disks that belong to the multipath that we are disconnecting, so it gets ignored. The volume we are looking for it's probably on the second session. So from this point forward (where we have duplicated node-sessions) it will not work again. I recommend you clean up that system so that you don't have duplicated nodes and sessions before trying to do a VM migration with a single volume attached. If that works, then try to attach 2 volumes on instances on the same host and see if the nodes and sessions are duplicated. Cheers, Gorka. > Il giorno ven 30 ott 2020 alle ore 09:27 Gorka Eguileor <geguileo(a)redhat.com> > ha scritto: > > > On 30/10, Ignazio Cassano wrote: > > > Hello, these are versions we are using: > > > [root@podto2-kvm02 ansible]# rpm -qa|grep queens > > > centos-release-openstack-queens-1-2.el7.centos.noarch > > > [root@podto2-kvm02 ansible]# rpm -qa|grep nova > > > openstack-nova-common-17.0.11-1.el7.noarch > > > python2-novaclient-10.1.0-1.el7.noarch > > > openstack-nova-compute-17.0.11-1.el7.noarch > > > python-nova-17.0.11-1.el7.noarch > > > > > > Cheers,Gorka. > > > > Hi, > > > > That release has the Nova bug fix, so Nova should not be calling Cinder > > to do an initialize connection on the source on the post-migration step > > anymore. > > > > I recommend comparing the connection info passed to connect_volume when > > the volume is attached on the source host and when it's disconnected on > > the post-migration step on the source host. > > > > Cheers, > > Gorka. > > > > > > > > > > > > > [1]: https://bugs.launchpad.net/nova/+bug/1754716 > > > > [2]: > > > > > > https://github.com/openstack/nova/commit/013f421bca4067bd430a9fac1e3b290cf1… > > > > > > > > > > > > > > > > > > > Il giorno gio 29 ott 2020 alle ore 09:12 Gorka Eguileor < > > > > geguileo(a)redhat.com> > > > > > ha scritto: > > > > > > > > > > > On 28/10, Ignazio Cassano wrote: > > > > > > > Hello Gorka, I would like to know if with unity iscsi driver, I > > must > > > > > > > configure iscsi initiator on both compute and controller nodes. > > > > > > > At this time I installed and cinfigured iscsi initiator only on > > > > compute > > > > > > > nodes and I got a lot of faulty devices when volumes ate > > detached. > > > > > > > Thanks > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > Both compute and controller nodes are in the data path. Computes > > when > > > > > > instances use the volumes, and controllers when we create volume > > from > > > > > > images, do generic volume migrations, create or restore backups, > > etc. > > > > > > > > > > > > Unless your deployment isn't doing any of the Cinder operations > > that > > > > > > involve the data plane, you'll have to configure iSCSI on the > > > > controller > > > > > > as well. > > > > > > > > > > > > Having said that, whether you configure the iSCSI initiator or not > > on > > > > > > the controller will have no effect on the paths used by the > > compute. > > > > > > > > > > > > I've seen the iSCSI initiator going crazy when the iscsid and the > > > > > > iscsiadm are from different versions. I've seen this in > > containerized > > > > > > environments. > > > > > > > > > > > > Faulty paths on multipathing is a tricky business, because there > > are > > > > > > different checkers, some generic (readsector0, tur, directio) and > > some > > > > > > vendor specific (emc_clarrion, hp_wd, rdac), and each one behaves > > in a > > > > > > different way. > > > > > > > > > > > > If you have a multipath device with faulty paths, that you think > > should > > > > > > not be faulty, you should look into what's going on with those > > paths: > > > > > > > > > > > > - Confirm that the device is still in the system under /dev/XYZ > > > > > > - Confirm in your storage array's console/webconsole that the > > volume is > > > > > > still mapped on that target-portal to that host's iscsi initiator > > > > > > name. > > > > > > - Confirm you can read the faulty devices with dd on the host > > > > > > - Confirm that the WWN of the device is the same in all the paths > > > > (using > > > > > > /lib/udev/scsi_id) > > > > > > - Finally look into what checker is multipath using for your device > > > > > > (sometimes checkers have bugs). > > > > > > > > > > > > Cheers, > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > Il Mar 20 Ott 2020, 19:58 Gorka Eguileor <geguileo(a)redhat.com> > > ha > > > > > > scritto: > > > > > > > > > > > > > > > On 20/10, Ignazio Cassano wrote: > > > > > > > > > This is the entre log from when the migration started: > > > > > > > > > > > > > > > > > > http://paste.openstack.org/show/799199/ > > > > > > > > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > There are no os-brick calls in there. :-( > > > > > > > > > > > > > > > > You should look for the call to connect_volume that should have > > > > > > > > something like: > > > > > > > > > > > > > > > > ==> disconnect_volume: call "{'args': > > > > > > > > (<os_brick.initiator.connectors.iscsi > > > > > > > > > > > > > > > > And the second parameter to that call is a dictionary where you > > > > can see > > > > > > > > the target_lun, target_luns, target_portals, target_portal, > > > > target_iqn, > > > > > > > > target_iqns... This will allow us to check if we are actually > > > > > > connected > > > > > > > > to those targets-portals > > > > > > > > > > > > > > > > The third parameter should contain two things that are > > relevant, > > > > the > > > > > > > > scsi_wwn and the path. You can check if the path exists and if > > > > that > > > > > > > > path actually has that wwn using /lib/udev/scsi_id --page 0x83 > > > > > > > > --whitelisted $path > > > > > > > > > > > > > > > > Those are the things I would check, because the only reason I > > can > > > > think > > > > > > > > that os-brick is not disconnecting any volumes are that the > > > > connection > > > > > > > > info is not right, or that the volume is no longer connected. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > Il giorno mar 20 ott 2020 alle ore 11:23 Gorka Eguileor < > > > > > > > > geguileo(a)redhat.com> > > > > > > > > > ha scritto: > > > > > > > > > > > > > > > > > > > On 20/10, Ignazio Cassano wrote: > > > > > > > > > > > Hello Gorka,this is what happens on nova compute with > > debug > > > > > > enabled, > > > > > > > > > > when I > > > > > > > > > > > migrate an instance with iscsi volumes ( note > > Disconnecting > > > > > > from[] > > > > > > > > should > > > > > > > > > > > be the issue): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > The disconnect from [] is the right clue, not necessarily > > the > > > > > > issue. > > > > > > > > > > > > > > > > > > > > OS-Brick is saying that for the connection information > > that has > > > > > > been > > > > > > > > > > passed in the "disconnect_volume" call (which is not > > present > > > > in the > > > > > > > > > > emailed logs) there are no volumes present in the system. > > > > > > > > > > > > > > > > > > > > You should check the connection info that Nova is passing > > to > > > > > > > > > > disconnect_volume and confirm if that data is correct. For > > > > example > > > > > > > > > > checking if the path present in the connection info > > dictionary > > > > is > > > > > > the > > > > > > > > > > same as the one in the instance's XML dump, or if the LUN > > from > > > > the > > > > > > > > > > connection info dict is actually present in the system. > > > > > > > > > > > > > > > > > > > > There are multiple reasons why Nova could be passing the > > wrong > > > > > > > > > > connection info to os-brick. The ones that come to mind > > are: > > > > > > > > > > > > > > > > > > > > - There was a failed migration at some point, and Nova > > didn't > > > > > > rollback > > > > > > > > > > the connection info on the BDM table. > > > > > > > > > > - Nova is calling multiple times initialize_connection on > > > > Cinder > > > > > > for > > > > > > > > the > > > > > > > > > > same host and the driver being used is not idempotent. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > stderr= _run_iscsiadm_bare > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1122 > > > > > > > > > > > 2020-10-20 09:52:33.066 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] iscsi session list stdout=tcp: [10] > > 10.138.209.48:3260,9 > > > > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a3 (non-flash) > > > > > > > > > > > tcp: [11] 10.138.215.17:3260,8 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [12] 10.138.215.17:3260,8 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [13] 10.138.215.18:3260,7 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [14] 10.138.215.18:3260,7 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [15] 10.138.209.47:3260,6 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b3 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [16] 10.138.209.47:3260,6 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b3 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [9] 10.138.209.48:3260,9 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a3 > > > > > > > > > > > (non-flash) > > > > > > > > > > > stderr= _run_iscsi_session > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1111 > > > > > > > > > > > 2020-10-20 09:52:33.078 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] Resulting device map defaultdict(<function <lambda> > > at > > > > > > > > > > 0x7f4f1b1f7cf8>, > > > > > > > > > > > {(u'10.138.215.17:3260', > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.a2'): > > > > > > > > > > > (set([]), set([u'sdg', u'sdi'])), (u'10.138.209.47:3260 > > ', > > > > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.b3'): (set([]), > > > > > > set([u'sdo', > > > > > > > > > > > u'sdq'])), (u'10.138.209.48:3260', > > > > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.a3'): (set([]), > > > > > > set([u'sdd', > > > > > > > > > > > u'sdb'])), (u'10.138.215.18:3260', > > > > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.b2'): (set([]), > > > > > > set([u'sdm', > > > > > > > > > > > u'sdk']))}) _get_connection_devices > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:844 > > > > > > > > > > > 2020-10-20 09:52:33.078 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] Disconnecting from: [] _disconnect_connection > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1099 > > > > > > > > > > > 2020-10-20 09:52:33.079 132171 DEBUG > > > > oslo_concurrency.lockutils > > > > > > [-] > > > > > > > > Lock > > > > > > > > > > > "connect_volume" released by > > > > > > > > > > > "os_brick.initiator.connectors.iscsi.disconnect_volume" > > :: > > > > held > > > > > > > > 1.058s > > > > > > > > > > > inner > > > > > > > > > > /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 > > > > > > > > > > > 2020-10-20 09:52:33.079 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] <== disconnect_volume: return (1057ms) None > > > > > > trace_logging_wrapper > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/utils.py:170 > > > > > > > > > > > 2020-10-20 09:52:33.079 132171 DEBUG > > > > > > nova.virt.libvirt.volume.iscsi > > > > > > > > [-] > > > > > > > > > > > [instance: 0c846f66-f194-40de-b31e-d53652570fa7] > > Disconnected > > > > > > iSCSI > > > > > > > > > > Volume > > > > > > > > > > > disconnect_volume > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:78 > > > > > > > > > > > 2020-10-20 09:52:33.080 132171 DEBUG os_brick.utils [-] > > ==> > > > > > > > > > > > get_connector_properties: call u"{'execute': None, > > 'my_ip': > > > > > > > > > > > '10.138.208.178', 'enforce_multipath': True, 'host': > > > > > > > > 'podiscsivc-kvm02', > > > > > > > > > > > 'root_helper': 'sudo nova-rootwrap > > /etc/nova/rootwrap.conf', > > > > > > > > 'multipath': > > > > > > > > > > > True}" trace_logging_wrapper > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/utils.py:146 > > > > > > > > > > > 2020-10-20 09:52:33.125 132171 DEBUG > > > > os_brick.initiator.linuxfc > > > > > > [-] > > > > > > > > No > > > > > > > > > > > Fibre Channel support detected on system. get_fc_hbas > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157 > > > > > > > > > > > 2020-10-20 09:52:33.126 132171 DEBUG > > > > os_brick.initiator.linuxfc > > > > > > [-] > > > > > > > > No > > > > > > > > > > > Fibre Channel support detected on system. get_fc_hbas > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157 > > > > > > > > > > > 2020-10-20 09:52:33.145 132171 DEBUG os_brick.utils [-] > > <== > > > > > > > > > > > get_connector_properties: return (61ms) {'initiator': > > > > > > > > > > > u'iqn.1994-05.com.redhat:fbfdc37eed4c', 'ip': > > > > u'10.138.208.178', > > > > > > > > 'system > > > > > > > > > > > uuid': u'4C4C4544-0051-4E10-8057-B6C04F425932', > > 'platform': > > > > > > > > u'x86_64', > > > > > > > > > > > 'host': u'podiscsivc-kvm02', 'do_local_attach': False, > > > > 'os_type': > > > > > > > > > > > u'linux2', 'multipath': True} trace_logging_wrapper > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/utils.py:170 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > > > Il giorno gio 15 ott 2020 alle ore 10:57 Gorka Eguileor < > > > > > > > > > > geguileo(a)redhat.com> > > > > > > > > > > > ha scritto: > > > > > > > > > > > > > > > > > > > > > > > On 14/10, Ignazio Cassano wrote: > > > > > > > > > > > > > Hello, thank you for the answer. > > > > > > > > > > > > > I am using os-brick 2.3.8 but I got same issues on > > stein > > > > with > > > > > > > > > > os.brick > > > > > > > > > > > > 2.8 > > > > > > > > > > > > > For explain better the situation I send you the > > output of > > > > > > > > multipath > > > > > > > > > > -ll > > > > > > > > > > > > on > > > > > > > > > > > > > a compute node: > > > > > > > > > > > > > root@podvc-kvm01 ansible]# multipath -ll > > > > > > > > > > > > > Oct 14 18:50:01 | sdbg: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbe: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbd: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbf: alua not supported > > > > > > > > > > > > > 360060160f0d049007ab7275f743d0286 dm-11 DGC > > ,VRAID > > > > > > > > > > > > > size=30G features='1 retain_attached_hw_handler' > > > > hwhandler='1 > > > > > > > > alua' > > > > > > > > > > wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > | |- 15:0:0:71 sdbg 67:160 failed faulty running > > > > > > > > > > > > > | `- 12:0:0:71 sdbe 67:128 failed faulty running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > |- 11:0:0:71 sdbd 67:112 failed faulty running > > > > > > > > > > > > > `- 13:0:0:71 sdbf 67:144 failed faulty running > > > > > > > > > > > > > 360060160f0d049004cdb615f52343fdb dm-8 DGC ,VRAID > > > > > > > > > > > > > size=80G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 15:0:0:210 sdau 66:224 active ready running > > > > > > > > > > > > > | `- 12:0:0:210 sdas 66:192 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 11:0:0:210 sdar 66:176 active ready running > > > > > > > > > > > > > `- 13:0:0:210 sdat 66:208 active ready running > > > > > > > > > > > > > 360060160f0d0490034aa645fe52265eb dm-12 DGC > > ,VRAID > > > > > > > > > > > > > size=100G features='2 queue_if_no_path > > > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 12:0:0:177 sdbi 67:192 active ready running > > > > > > > > > > > > > | `- 15:0:0:177 sdbk 67:224 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 11:0:0:177 sdbh 67:176 active ready running > > > > > > > > > > > > > `- 13:0:0:177 sdbj 67:208 active ready running > > > > > > > > > > > > > 360060160f0d04900159f225fd6126db9 dm-6 DGC ,VRAID > > > > > > > > > > > > > size=40G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 11:0:0:26 sdaf 65:240 active ready running > > > > > > > > > > > > > | `- 13:0:0:26 sdah 66:16 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 12:0:0:26 sdag 66:0 active ready running > > > > > > > > > > > > > `- 15:0:0:26 sdai 66:32 active ready running > > > > > > > > > > > > > Oct 14 18:50:01 | sdba: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbc: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdaz: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbb: alua not supported > > > > > > > > > > > > > 360060160f0d049007eb7275f93937511 dm-10 DGC > > ,VRAID > > > > > > > > > > > > > size=40G features='1 retain_attached_hw_handler' > > > > hwhandler='1 > > > > > > > > alua' > > > > > > > > > > wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > | |- 12:0:0:242 sdba 67:64 failed faulty running > > > > > > > > > > > > > | `- 15:0:0:242 sdbc 67:96 failed faulty running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > |- 11:0:0:242 sdaz 67:48 failed faulty running > > > > > > > > > > > > > `- 13:0:0:242 sdbb 67:80 failed faulty running > > > > > > > > > > > > > 360060160f0d049003a567c5fb72201e8 dm-7 DGC ,VRAID > > > > > > > > > > > > > size=40G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 12:0:0:57 sdbq 68:64 active ready running > > > > > > > > > > > > > | `- 15:0:0:57 sdbs 68:96 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 11:0:0:57 sdbp 68:48 active ready running > > > > > > > > > > > > > `- 13:0:0:57 sdbr 68:80 active ready running > > > > > > > > > > > > > 360060160f0d04900c120625f802ea1fa dm-9 DGC ,VRAID > > > > > > > > > > > > > size=25G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 11:0:0:234 sdav 66:240 active ready running > > > > > > > > > > > > > | `- 13:0:0:234 sdax 67:16 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 15:0:0:234 sday 67:32 active ready running > > > > > > > > > > > > > `- 12:0:0:234 sdaw 67:0 active ready running > > > > > > > > > > > > > 360060160f0d04900b8b0615fb14ef1bd dm-3 DGC ,VRAID > > > > > > > > > > > > > size=50G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 11:0:0:11 sdan 66:112 active ready running > > > > > > > > > > > > > | `- 13:0:0:11 sdap 66:144 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 12:0:0:11 sdao 66:128 active ready running > > > > > > > > > > > > > `- 15:0:0:11 sdaq 66:160 active ready running > > > > > > > > > > > > > > > > > > > > > > > > > > The active running are related to running virtual > > > > machines. > > > > > > > > > > > > > The faulty are related to virtual macnines migrated > > on > > > > other > > > > > > kvm > > > > > > > > > > nodes. > > > > > > > > > > > > > Every volume has 4 path because iscsi on unity needs > > two > > > > > > > > different > > > > > > > > > > vlans, > > > > > > > > > > > > > each one with 2 addresses. > > > > > > > > > > > > > I think this issue can be related to os-brick because > > > > when I > > > > > > > > migrate > > > > > > > > > > a > > > > > > > > > > > > > virtual machine from host A host B in the cova > > compute > > > > log on > > > > > > > > host A > > > > > > > > > > I > > > > > > > > > > > > read: > > > > > > > > > > > > > 2020-10-13 10:31:02.769 118727 DEBUG > > > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > > > [req-771ede8c-6e1b-4f3f-ad4a-1f6ed820a55c > > > > > > > > > > > > 66adb965bef64eaaab2af93ade87e2ca > > > > > > > > > > > > > 85cace94dcc7484c85ff9337eb1d0c4c - default default] > > > > > > > > *Disconnecting > > > > > > > > > > from: > > > > > > > > > > > > []* > > > > > > > > > > > > > > > > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > That's definitely the right clue!! Though I don't > > fully > > > > agree > > > > > > with > > > > > > > > > > this > > > > > > > > > > > > being an os-brick issue just yet. ;-) > > > > > > > > > > > > > > > > > > > > > > > > Like I mentioned before, RCA is usually non-trivial, > > and > > > > > > > > explaining how > > > > > > > > > > > > to debug these issues over email is close to > > impossible, > > > > but if > > > > > > > > this > > > > > > > > > > > > were my system, and assuming you have tested normal > > > > > > attach/detach > > > > > > > > > > > > procedure and is working fine, this is what I would do: > > > > > > > > > > > > > > > > > > > > > > > > - Enable DEBUG logs on Nova compute node (I believe you > > > > already > > > > > > > > have) > > > > > > > > > > > > - Attach a new device to an instance on that node with > > > > --debug > > > > > > to > > > > > > > > get > > > > > > > > > > > > the request id > > > > > > > > > > > > - Get the connection information dictionary that > > os-brick > > > > > > receives > > > > > > > > on > > > > > > > > > > > > the call to connect_volume for that request, and the > > data > > > > > > that > > > > > > > > > > > > os-brick returns to Nova on that method call > > completion. > > > > > > > > > > > > - Check if the returned data to Nova is a multipathed > > > > device or > > > > > > > > not (in > > > > > > > > > > > > 'path'), and whether we have the wwn or not (in > > > > > > 'scsi_wwn'). It > > > > > > > > > > > > should be a multipath device, and then I would check > > its > > > > > > status > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > multipath daemon. > > > > > > > > > > > > - Now do the live migration (with --debug to get the > > > > request > > > > > > id) > > > > > > > > and > > > > > > > > > > see > > > > > > > > > > > > what information Nova passes in that request to > > > > os-brick's > > > > > > > > > > > > disconnect_volume. > > > > > > > > > > > > - Is it the same? Then it's likely an os-brick issue, > > > > and I > > > > > > can > > > > > > > > have > > > > > > > > > > a > > > > > > > > > > > > look at the logs if you put the logs for that > > os-brick > > > > > > detach > > > > > > > > > > > > process in a pastebin [1]. > > > > > > > > > > > > - Is it different? Then it's either a Nova bug or a > > > > Cinder > > > > > > driver > > > > > > > > > > > > specific bug. > > > > > > > > > > > > - Is there a call from Nova to Cinder, in the > > migration > > > > > > > > request, > > > > > > > > > > for > > > > > > > > > > > > that same volume to initialize_connection > > passing the > > > > > > source > > > > > > > > host > > > > > > > > > > > > connector info (info from the host that is > > currently > > > > > > > > attached)? > > > > > > > > > > > > If there is a call, check if the returned data is > > > > > > different > > > > > > > > from > > > > > > > > > > > > the one we used to do the attach, if that's the > > case > > > > then > > > > > > > > it's a > > > > > > > > > > > > Nova and Cinder driver bug that was solved on the > > > > Nova > > > > > > side > > > > > > > > in > > > > > > > > > > > > 17.0.10 [2]. > > > > > > > > > > > > - If there's no call to Cinder's > > > > initialize_connection, the > > > > > > > > it's > > > > > > > > > > > > most likely a Nova bug. Try to find out if this > > > > > > connection > > > > > > > > info > > > > > > > > > > > > makes any sense for that host (LUN, target, > > etc.) or > > > > if > > > > > > this > > > > > > > > is > > > > > > > > > > > > the one from the destination volume. > > > > > > > > > > > > > > > > > > > > > > > > I hope this somehow helps. > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: http://paste.openstack.org/ > > > > > > > > > > > > [2]: https://review.opendev.org/#/c/637827/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Il giorno mer 14 ott 2020 alle ore 13:41 Gorka > > Eguileor < > > > > > > > > > > > > geguileo(a)redhat.com> > > > > > > > > > > > > > ha scritto: > > > > > > > > > > > > > > > > > > > > > > > > > > > On 09/10, Ignazio Cassano wrote: > > > > > > > > > > > > > > > Hello Stackers, I am using dell emc iscsi driver > > on > > > > my > > > > > > > > centos 7 > > > > > > > > > > > > queens > > > > > > > > > > > > > > > openstack. It works and instances work as well > > but on > > > > > > compute > > > > > > > > > > nodes I > > > > > > > > > > > > > > got a > > > > > > > > > > > > > > > lot a faulty device reported by multipath il > > comand. > > > > > > > > > > > > > > > I do know why this happens, probably attacching > > and > > > > > > detaching > > > > > > > > > > > > volumes and > > > > > > > > > > > > > > > live migrating instances do not close something > > well. > > > > > > > > > > > > > > > I read this can cause serious performances > > problems > > > > on > > > > > > > > compute > > > > > > > > > > nodes. > > > > > > > > > > > > > > > Please, any workaround and/or patch is suggested > > ? > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > There are many, many, many things that could be > > > > happening > > > > > > > > there, > > > > > > > > > > and > > > > > > > > > > > > > > it's not usually trivial doing the RCA, so the > > > > following > > > > > > > > questions > > > > > > > > > > are > > > > > > > > > > > > > > just me hoping this is something "easy" to find > > out. > > > > > > > > > > > > > > > > > > > > > > > > > > > > What os-brick version from Queens are you running? > > > > Latest > > > > > > > > > > (2.3.9), or > > > > > > > > > > > > > > maybe one older than 2.3.3? > > > > > > > > > > > > > > > > > > > > > > > > > > > > When you say you have faulty devices reported, are > > > > these > > > > > > faulty > > > > > > > > > > devices > > > > > > > > > > > > > > alone in the multipath DM? Or do you have some > > faulty > > > > ones > > > > > > with > > > > > > > > > > some > > > > > > > > > > > > > > that are ok? > > > > > > > > > > > > > > > > > > > > > > > > > > > > If there are some OK and some that aren't, are they > > > > > > consecutive > > > > > > > > > > > > devices? > > > > > > > > > > > > > > (as in /dev/sda /dev/sdb etc). > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

2 2

Re: [kolla] Setting up eth1 on VM for AIO
by Braden, Albert 30 Oct '20

30 Oct '20

From: Mark Goddard <mark(a)stackhpc.com> Sent: Friday, October 30, 2020 4:37 AM To: Radosław Piliszek <radoslaw.piliszek(a)gmail.com> Cc: Braden, Albert <C-Albert.Braden(a)charter.com>; OpenStack Discuss <openstack-discuss(a)lists.openstack.org> Subject: Re: [kolla] Setting up eth1 on VM for AIO On Thu, 29 Oct 2020 at 16:20, Radosław Piliszek <radoslaw.piliszek(a)gmail.com<mailto:radoslaw.piliszek@gmail.com>> wrote: Hi Albert! Sorry for top posting but I don't really see which part to quote would be best so they all lose. :D Indeed, it's tricky to handle this if you are not a fan of linux networking. I guess we should document an example way.... Anyhow, the way e.g. Kayobe handles it for users (as it deals with baremetal details) is to use bridges and veths. Basically, you create a bridge, attach it to the expected interface, set any networking you still want on that network for this host on this bridge. Then, you create a veth pair and connect one end to it and the second one is given to neutron. >Alternatively, if this is just for development, you can use a dummy interface. > >ip link add link eth1 type dummy > >It's not persistent, but good enough for testing. If you need external network access from it you could add a MASQUERADE iptables rule and enable the IP >forwarding sysctl. That seems to be working. After googling a bit I was able to make it persistent: [root@adjutant-poc ~]# cat /etc/modules-load.d/dummy.conf dummy [root@adjutant-poc ~]# cat /etc/modprobe.d/dummy.conf install dummy /sbin/modprobe --ignore-install dummy; /sbin/ip link set name eth1 dev dummy0 [root@adjutant-poc ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 NAME=eth1 BOOTPROTO=none DEVICE=eth1 ONBOOT=yes TYPE=Ethernet NM_CONTROLLED=no [root@adjutant-poc ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:22:c4:e2 brd ff:ff:ff:ff:ff:ff inet 192.168.1.83/22 brd 192.168.3.255 scope global dynamic eth0 valid_lft 85821sec preferred_lft 85821sec inet6 fe80::f816:3eff:fe22:c4e2/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 7e:e1:fe:31:c9:2e brd ff:ff:ff:ff:ff:ff inet6 fe80::7ce1:feff:fe31:c92e/64 scope link valid_lft forever preferred_lft forever -yoctozepto On Thu, Oct 29, 2020 at 12:49 PM Braden, Albert <C-Albert.Braden(a)charter.com<mailto:C-Albert.Braden@charter.com>> wrote: > > I’m trying to setup an allinone kolla-ansible Train POC on a VM, and I’m following the instructions here: > > > > https://docs.openstack.org/kolla-ansible/train/user/quickstart.html > > > > Under “Prepare initial configuration” where it talks about globals.yml, it says I need 2 network interfaces: > > > > network_interface: "eth0" > > neutron_external_interface: "eth1" > > > > My VM has a private IP and a floating IP that I use to ssh to it: > > > > | ID | Name | Status | Networks | Image | Flavor | > > | 41f6782a-8ca8-4ab4-8fdc-776584139ba0 | adjutant-poc | ACTIVE | private=<primary IP>, <floating IP> | | s1-standard-08 | > > > > I tried various options for eth1 but I can’t seem to get it to work: > > > > [root@adjutant-poc network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 > > BOOTPROTO=dhcp > > DEVICE=eth1 > > ONBOOT=yes > > TYPE=Ethernet > > NM_CONTROLLED=no > > > > I tried creating another port and adding it but that didn’t seem to make a difference. What am I doing wrong? Do I really need 2 network interfaces on the VM? If so, what’s the correct way to add one? > > > > [root@adjutant-poc network-scripts]# service network restart > > Restarting network (via systemctl): Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details. > > [FAILED] > > > > Log: > > > > http://paste.debian.net/1168982/ > > > > I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

1 0

[ironic] SPUC winter season
by Dmitry Tantsur 30 Oct '20

30 Oct '20

Boo! It's SPUC time again! Ironicers, their pet AIs, friendly stackers, as well as significant others and fluffy overladies and overlords, are welcome! For those who don't know, Sanity Preserving Un-Conference is a series of Friday calls for keeping (the remainings) of our sanity in these difficult times. It's goal is to let people virtually socialize and help brighten up lives for each other. Work achievements? A new hobby? A hilarious story? Funny things your kid did at 4am? The first words from your pizza dough that you forgot in the fridge during the previous lockdown? Anything that makes you and others happier is on-topic! Due to an increasing number of APAC contributors, I would like two pick slots: one in APAC evening/Europe morning, the other in Europe morning/US afternoon. Please vote in the doodle: https://doodle.com/poll/tw72utagidz2nac3 Cheers, Dmitry -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

4 6

[openstack][nova] urgent compute in evacuate state
by Ignazio Cassano 30 Oct '20

30 Oct '20

Hello Guys, I have a node in evacuate state....how exit from this state ? Thanks Ignazio

2 6

[kolla] Setting up eth1 on VM for AIO
by Braden, Albert 30 Oct '20

30 Oct '20

I'm trying to setup an allinone kolla-ansible Train POC on a VM, and I'm following the instructions here: https://docs.openstack.org/kolla-ansible/train/user/quickstart.html Under "Prepare initial configuration" where it talks about globals.yml, it says I need 2 network interfaces: network_interface: "eth0" neutron_external_interface: "eth1" My VM has a private IP and a floating IP that I use to ssh to it: | ID | Name | Status | Networks | Image | Flavor | | 41f6782a-8ca8-4ab4-8fdc-776584139ba0 | adjutant-poc | ACTIVE | private=<primary IP>, <floating IP> | | s1-standard-08 | I tried various options for eth1 but I can't seem to get it to work: [root@adjutant-poc network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 BOOTPROTO=dhcp DEVICE=eth1 ONBOOT=yes TYPE=Ethernet NM_CONTROLLED=no I tried creating another port and adding it but that didn't seem to make a difference. What am I doing wrong? Do I really need 2 network interfaces on the VM? If so, what's the correct way to add one? [root@adjutant-poc network-scripts]# service network restart Restarting network (via systemctl): Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details. [FAILED] Log: http://paste.debian.net/1168982/ I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

3 2

[Openstack][cinder] dell unity iscsi faulty devices
by Ignazio Cassano 30 Oct '20

30 Oct '20

Hello Stackers, I am using dell emc iscsi driver on my centos 7 queens openstack. It works and instances work as well but on compute nodes I got a lot a faulty device reported by multipath il comand. I do know why this happens, probably attacching and detaching volumes and live migrating instances do not close something well. I read this can cause serious performances problems on compute nodes. Please, any workaround and/or patch is suggested ? Regards Ignazio

2 18

cannot ping my guest's floating ip
by R Kendal 29 Oct '20

29 Oct '20

I am struggling with the fact that I cannot ping my guest's floating ip. I have one Controller node and one Compute node brought up with devstack. Any help would be appreciated! Kendal Details follow... Controller node... [1 picasso@controller ~]$ openstack server list +--------------------------------------+----------+--------+-------------------------------+----------------+----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+----------+--------+-------------------------------+----------------+----------+ | c5a7fd0a-61a1-4698-9c12-45b44cb20e54 | myserver | ACTIVE | private=10.0.1.12, 10.0.0.196 | picasso-server | m1.small | +--------------------------------------+----------+--------+-------------------------------+----------------+----------+ [1 picasso@controller ~]$ openstack floating ip list +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ | ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ | 055d825f-9398-4a11-bf5e-ff3d1a9fc8be | 10.0.0.196 | 10.0.1.12 | 9d036770-e1ba-4d5f-8f16-780a066416fd | 32631db5-7798-47df-8af6-7ee8fca011e4 | a0e078d4bac34db89105a2eb0ba4d896 | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ [0 picasso@controller ~]$ ping -c 1 -W 1 10.0.1.1 PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data. 64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=0.066 ms --- 10.0.1.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.066/0.066/0.066/0.000 ms [0 picasso@controller ~]$ ping -c 1 -W 1 10.0.1.12 PING 10.0.1.12 (10.0.1.12) 56(84) bytes of data. --- 10.0.1.12 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [1 picasso@controller ~]$ ping -c 1 -W 1 10.0.0.196 PING 10.0.0.196 (10.0.0.196) 56(84) bytes of data. --- 10.0.0.196 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [0 picasso@controller ~]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000 link/ether 08:00:27:72:b2:e5 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3 valid_lft 77664sec preferred_lft 77664sec inet6 fe80::a00:27ff:fe72:b2e5/64 scope link valid_lft forever preferred_lft forever 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000 link/ether 02:00:68:ed:7c:08 brd ff:ff:ff:ff:ff:ff inet 192.168.1.6/24 brd 192.168.1.255 scope global enp0s8 valid_lft forever preferred_lft forever inet6 fe80::68ff:feed:7c08/64 scope link valid_lft forever preferred_lft forever 4: enp0s9: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000 link/ether 02:00:b6:d5:fe:c5 brd ff:ff:ff:ff:ff:ff inet6 fe80::b6ff:fed5:fec5/64 scope link valid_lft forever preferred_lft forever 5: enp0s10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 02:00:72:74:73:d0 brd ff:ff:ff:ff:ff:ff inet 10.0.1.1/24 brd 10.0.1.255 scope global enp0s10 valid_lft forever preferred_lft forever inet6 fe80::72ff:fe74:73d0/64 scope link valid_lft forever preferred_lft forever 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether e6:fd:7a:97:a1:32 brd ff:ff:ff:ff:ff:ff 7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether f6:38:eb:d2:ce:41 brd ff:ff:ff:ff:ff:ff 8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether ba:d2:e4:c1:7f:48 brd ff:ff:ff:ff:ff:ff inet 10.0.0.8/24 brd 10.0.0.255 scope global br-ex valid_lft forever preferred_lft forever inet6 fe80::b8d2:e4ff:fec1:7f48/64 scope link valid_lft forever preferred_lft forever 9: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether ca:5b:f2:bd:65:40 brd ff:ff:ff:ff:ff:ff 13: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000 link/ether ca:7f:ae:3b:5a:14 brd ff:ff:ff:ff:ff:ff inet6 fe80::c87f:aeff:fe3b:5a14/64 scope link valid_lft forever preferred_lft forever [0 picasso@controller ~]$ ip r default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100 10.0.0.0/24 dev br-ex proto kernel scope link src 10.0.0.8 10.0.1.0/24 dev enp0s10 proto kernel scope link src 10.0.1.1 10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15 10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100 192.168.1.0/24 dev enp0s8 proto kernel scope link src 192.168.1.6 [0 picasso@controller ~]$ neutron net-show "public" +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | nova | | created_at | 2020-10-29T18:21:07Z | | description | | | id | 32631db5-7798-47df-8af6-7ee8fca011e4 | | ipv4_address_scope | | | ipv6_address_scope | | | is_default | False | | mtu | 1500 | | name | public | | port_security_enabled | True | | project_id | a0e078d4bac34db89105a2eb0ba4d896 | | provider:network_type | flat | | provider:physical_network | public | | provider:segmentation_id | | | revision_number | 2 | | router:external | True | | shared | True | | status | ACTIVE | | subnets | daab0afa-9adc-4978-9e00-65669b89fec5 | | tags | | | tenant_id | a0e078d4bac34db89105a2eb0ba4d896 | | updated_at | 2020-10-29T18:21:14Z | +---------------------------+--------------------------------------+ [0 picasso@controller ~]$ neutron subnet-show public-subnet +-------------------+----------------------------------------------+ | Field | Value | +-------------------+----------------------------------------------+ | allocation_pools | {"start": "10.0.0.128", "end": "10.0.0.254"} | | cidr | 10.0.0.0/24 | | created_at | 2020-10-29T18:21:14Z | | description | | | dns_nameservers | 10.0.0.1 | | enable_dhcp | False | | gateway_ip | 10.0.0.1 | | host_routes | | | id | daab0afa-9adc-4978-9e00-65669b89fec5 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | public-subnet | | network_id | 32631db5-7798-47df-8af6-7ee8fca011e4 | | project_id | a0e078d4bac34db89105a2eb0ba4d896 | | revision_number | 0 | | service_types | | | subnetpool_id | | | tags | | | tenant_id | a0e078d4bac34db89105a2eb0ba4d896 | | updated_at | 2020-10-29T18:21:14Z | +-------------------+----------------------------------------------+ [1 picasso@controller ~]$ neutron net-show private +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | nova | | created_at | 2020-10-29T18:20:56Z | | description | | | id | b440a536-a69c-42c8-b05d-5f8323f22cc0 | | ipv4_address_scope | | | ipv6_address_scope | | | mtu | 1450 | | name | private | | port_security_enabled | True | | project_id | 682549fcfe3b4645a21b59290e62e453 | | provider:network_type | vxlan | | provider:physical_network | | | provider:segmentation_id | 1 | | revision_number | 2 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | 35a88fb9-e1e6-4195-9c6b-95d96545d4c4 | | tags | | | tenant_id | 682549fcfe3b4645a21b59290e62e453 | | updated_at | 2020-10-29T18:21:02Z | +---------------------------+--------------------------------------+ [0 picasso@controller ~]$ neutron subnet-show private-subnet +-------------------+---------------------------------------------+ | Field | Value | +-------------------+---------------------------------------------+ | allocation_pools | {"start": "10.0.1.10", "end": "10.0.1.254"} | | cidr | 10.0.1.0/24 | | created_at | 2020-10-29T18:21:02Z | | description | | | dns_nameservers | 10.0.0.1 | | enable_dhcp | True | | gateway_ip | 10.0.1.1 | | host_routes | | | id | 35a88fb9-e1e6-4195-9c6b-95d96545d4c4 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | private-subnet | | network_id | b440a536-a69c-42c8-b05d-5f8323f22cc0 | | project_id | 682549fcfe3b4645a21b59290e62e453 | | revision_number | 0 | | service_types | | | subnetpool_id | | | tags | | | tenant_id | 682549fcfe3b4645a21b59290e62e453 | | updated_at | 2020-10-29T18:21:02Z | +-------------------+---------------------------------------------+ [0 picasso@controller ~]$ neutron router-show public +-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | nova | | created_at | 2020-10-29T18:21:20Z | | description | | | distributed | False | | external_gateway_info | {"network_id": "32631db5-7798-47df-8af6-7ee8fca011e4", "external_fixed_ips": [{"subnet_id": "daab0afa-9adc-4978-9e00-65669b89fec5", "ip_address": "10.0.0.169"}], "enable_snat": true} | | flavor_id | | | ha | False | | id | d424d63e-cc93-4d02-afad-5851bc7f96cd | | name | public | | project_id | a0e078d4bac34db89105a2eb0ba4d896 | | revision_number | 4 | | routes | | | status | ACTIVE | | tags | | | tenant_id | a0e078d4bac34db89105a2eb0ba4d896 | | updated_at | 2020-10-29T18:21:28Z | +-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [1 picasso@controller ~]$ sudo ovs-vsctl show f2c58de6-9a33-4222-94f5-da1e4a2cfdbb Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port br-tun Interface br-tun type: internal Port "vxlan-c0a801c8" Interface "vxlan-c0a801c8" type: vxlan options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="192.168.1.6", out_key=flow, remote_ip="192.168.1.200"} Bridge br-ex Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-ex Interface br-ex type: internal Port phy-br-ex Interface phy-br-ex type: patch options: {peer=int-br-ex} Port "enp0s9" Interface "enp0s9" Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port "tap456085c0-61" tag: 1 Interface "tap456085c0-61" type: internal Port "qr-0434594d-ee" tag: 1 Interface "qr-0434594d-ee" type: internal Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port int-br-ex Interface int-br-ex type: patch options: {peer=phy-br-ex} Port br-int Interface br-int type: internal Port "qg-77ed5bcd-25" tag: 2 Interface "qg-77ed5bcd-25" type: internal ovs_version: "2.12.1" router_name=public ROUTER_ID=$(openstack router show $router_name -f value -c id) QROUTER="qrouter-$ROUTER_ID" [0 picasso@controller ~]$ sudo ip netns exec $QROUTER ifconfig lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 45 bytes 4424 (4.4 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 45 bytes 4424 (4.4 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qg-77ed5bcd-25: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.0.0.169 netmask 255.255.255.0 broadcast 10.0.0.255 inet6 fe80::f816:3eff:febc:990c prefixlen 64 scopeid 0x20<link> ether fa:16:3e:bc:99:0c txqueuelen 1000 (Ethernet) RX packets 944 bytes 146787 (146.7 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 164 bytes 16468 (16.4 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qr-0434594d-ee: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 10.0.1.1 netmask 255.255.255.0 broadcast 10.0.1.255 inet6 fe80::f816:3eff:fe5c:9a7d prefixlen 64 scopeid 0x20<link> ether fa:16:3e:5c:9a:7d txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 169 bytes 7550 (7.5 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [0 picasso@controller ~]$ sudo ip netns exec $QROUTER ping -c 1 -W 1 10.0.1.1 PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data. 64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=0.034 ms --- 10.0.1.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.034/0.034/0.034/0.000 ms [0 picasso@controller ~]$ sudo ip netns exec $QROUTER ping -c 1 -W 1 10.0.1.196 PING 10.0.1.196 (10.0.1.196) 56(84) bytes of data. --- 10.0.1.196 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [1 picasso@controller ~]$ sudo ip netns exec $QROUTER ping -c 1 -W 1 10.0.1.169 PING 10.0.1.169 (10.0.1.169) 56(84) bytes of data. --- 10.0.1.169 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms ---------- Compute node... [1 picasso@compute ~]$ ping -c 1 -W 1 10.0.1.1 PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data. 64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=0.425 ms --- 10.0.1.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.425/0.425/0.425/0.000 ms [0 picasso@compute ~]$ ping -c 1 -W 1 10.0.1.12 PING 10.0.1.12 (10.0.1.12) 56(84) bytes of data. --- 10.0.1.12 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [1 picasso@compute ~]$ ping -c 1 -W 1 10.0.0.196 PING 10.0.0.196 (10.0.0.196) 56(84) bytes of data. --- 10.0.0.196 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [0 picasso@compute ~]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000 link/ether 08:00:27:72:b2:e5 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3 valid_lft 79425sec preferred_lft 79425sec inet6 fe80::a00:27ff:fe72:b2e5/64 scope link valid_lft forever preferred_lft forever 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000 link/ether 02:00:73:9d:41:7d brd ff:ff:ff:ff:ff:ff inet 192.168.1.200/24 brd 192.168.1.255 scope global enp0s8 valid_lft forever preferred_lft forever inet6 fe80::73ff:fe9d:417d/64 scope link valid_lft forever preferred_lft forever 4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 02:00:08:c1:84:db brd ff:ff:ff:ff:ff:ff inet 10.0.1.31/24 brd 10.0.1.255 scope global enp0s9 valid_lft forever preferred_lft forever inet6 fe80::8ff:fec1:84db/64 scope link valid_lft forever preferred_lft forever 5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:ee:20:86 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:ee:20:86 brd ff:ff:ff:ff:ff:ff 7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 9e:52:8b:6f:f1:47 brd ff:ff:ff:ff:ff:ff 8: br-int: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000 link/ether 76:e6:90:cc:01:48 brd ff:ff:ff:ff:ff:ff 9: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether aa:09:9f:d9:1d:4f brd ff:ff:ff:ff:ff:ff inet6 fe80::a809:9fff:fed9:1d4f/64 scope link valid_lft forever preferred_lft forever 10: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 26:76:22:af:06:40 brd ff:ff:ff:ff:ff:ff 11: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000 link/ether 8a:87:fb:68:ec:8c brd ff:ff:ff:ff:ff:ff inet6 fe80::8887:fbff:fe68:ec8c/64 scope link valid_lft forever preferred_lft forever 12: tap9d036770-e1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000 link/ether fe:16:3e:f5:f5:c0 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc16:3eff:fef5:f5c0/64 scope link valid_lft forever preferred_lft forever [0 picasso@compute ~]$ ip r default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100 10.0.1.0/24 dev enp0s9 proto kernel scope link src 10.0.1.31 10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15 10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100 192.168.1.0/24 dev enp0s8 proto kernel scope link src 192.168.1.200 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown [130 picasso@compute ~]$ sudo ovs-vsctl show a75bdb8e-9266-48f4-bab5-e119cf07c6af Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-tun Interface br-tun type: internal Port "vxlan-c0a80106" Interface "vxlan-c0a80106" type: vxlan options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="192.168.1.200", out_key=flow, remote_ip="192.168.1.6"} Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-int Interface br-int type: internal Port "tap9d036770-e1" tag: 1 Interface "tap9d036770-e1" Port int-br-ex Interface int-br-ex type: patch options: {peer=phy-br-ex} Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Bridge br-ex Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port phy-br-ex Interface phy-br-ex type: patch options: {peer=int-br-ex} Port br-ex Interface br-ex type: internal ovs_version: "2.12.1"

2 1

Nova missing instance
by Grant Morley 29 Oct '20

29 Oct '20

Hi all, A customer had contacted us stating their instance was not working and was unresponsive. They attempted to reboot it and it failed to reboot and went into an error status. I have had a look at the nova logs and nova seems to think the instance has "disappeared" and comes up with an error stating: "Instance cannot be found" I have attached the full logs in the pastebin link. Has anyone come across this before? https://pastebin.com/LWzvsc8u Regards, -- Grant Morley Cloud Engineer, Civo Ltd Unit H-K, Gateway 1000, Whittle Way Stevenage, Herts, SG1 2FP, UK Visit us at www.civo.com <https://www.civo.com/> Signup for an account now > <https://www.civo.com/signup>

2 1

[tripleo][ci] Docker.io Service Announcement: Pull Rate Limit Enforcement, Changes to Image Retention Policies Begin
by Wesley Hayutin 29 Oct '20

29 Oct '20

Greetings, FYI... docker.io will be unavailable to tripleo jobs on Monday Nov 2 2020. Documentation is available here: https://hackmd.io/YH9xtBmOQbSVqPu7fjgoog Reviews are welcome and appreciated https://review.opendev.org/#/q/topic:new-ci-job+(status:open+OR+status:merg… Thanks! ------ from docker.io ------ Hello: You are receiving this email because of a policy change to Docker products and services you use. On Monday, November 2, 2020 at 9am Pacific Standard Time, Docker will begin enforcing rate limits on container pulls for Anonymous and Free users. Anonymous (unauthenticated) users will be limited to 100 container image pulls every six hours, and Free (authenticated) users will be limited to 200 container image pulls every six hours, when enforcement is fully implemented. Docker Pro and Team subscribers can pull container images from Docker Hub without restriction, as long as the quantities are not excessive or abusive. In addition, we are pausing enforcement of the changes to our image-retention policies until mid-2021, when we anticipate incorporating them into usage-based pricing. Two months ago, we announced an update to Docker image-retention policies. As originally stated, this change, which was set to take effect on November 1, 2020, would result in the deletion of images for free Docker account users after six months of inactivity. Today’s announcement means Docker will not enforce image expiration on November 1, 2020. For more details, please read the blog post <http://email.notify.docker.com/c/eJxNjsFuhSAURL9Gdxi46EMWLNq89j-ucFVSBCMY49…> summarizing the changes or review the documentation for rate limits <http://email.notify.docker.com/c/eJxNjkEOgyAURE8jOwl8kOqCRRvbe3z8WEhRG8WY3r…> . Details about Docker subscription levels and differentiators are available on the Docker Pricing Page <http://email.notify.docker.com/c/eJxNjcsOgyAURL9GlgTu5VEWLNrY_gfBq5KqGKUh_n…> . Thank you for using Docker. Jean-Laurent de Morlhon VP, Engineering Docker

1 0

Windows client can't connect to internet
by liu 29 Oct '20

29 Oct '20

Dear OpenStack community, I just launch a windows client in my OpenStack platform, but it can't connect to the internet with the float IP. Before that use a ubuntu client I can access the internet. Any ideas about why this happened. Thank you for your help. Liu

2 2

[tripleo][ci] Fwd: Service Announcement: Pull Rate Limit Enforcement, Changes to Image Retention Policies Begin November 2, 2020
by Wes Hayutin 29 Oct '20

29 Oct '20

FYI... Reviews are welcome and appreciated https://review.opendev.org/#/q/topic:new-ci-job+(status:open+OR+status:merg… Thanks! ---------- Forwarded message --------- From: Docker <noreply(a)notify.docker.com> Date: Wed, Oct 28, 2020 at 10:08 PM Subject: Service Announcement: Pull Rate Limit Enforcement, Changes to Image Retention Policies Begin November 2, 2020 To: <weshayutin(a)gmail.com> [image: Docker] <http://email.notify.docker.com/c/eJxNjUEOgyAQAF-DR7LsosCBQ43tPxChkKo0imn8fe…> Hello: You are receiving this email because of a policy change to Docker products and services you use. On Monday, November 2, 2020 at 9am Pacific Standard Time, Docker will begin enforcing rate limits on container pulls for Anonymous and Free users. Anonymous (unauthenticated) users will be limited to 100 container image pulls every six hours, and Free (authenticated) users will be limited to 200 container image pulls every six hours, when enforcement is fully implemented. Docker Pro and Team subscribers can pull container images from Docker Hub without restriction, as long as the quantities are not excessive or abusive. In addition, we are pausing enforcement of the changes to our image-retention policies until mid-2021, when we anticipate incorporating them into usage-based pricing. Two months ago, we announced an update to Docker image-retention policies. As originally stated, this change, which was set to take effect on November 1, 2020, would result in the deletion of images for free Docker account users after six months of inactivity. Today’s announcement means Docker will not enforce image expiration on November 1, 2020. For more details, please read the blog post <http://email.notify.docker.com/c/eJxNjsFuhSAURL9Gdxi46EMWLNq89j-ucFVSBCMY49…> summarizing the changes or review the documentation for rate limits <http://email.notify.docker.com/c/eJxNjkEOgyAURE8jOwl8kOqCRRvbe3z8WEhRG8WY3r…>. Details about Docker subscription levels and differentiators are available on the Docker Pricing Page <http://email.notify.docker.com/c/eJxNjcsOgyAURL9GlgTu5VEWLNrY_gfBq5KqGKUh_n…>. Thank you for using Docker. Jean-Laurent de Morlhon VP, Engineering Docker © Docker, Inc., 2020 318 Cambridge Ave Palo Alto, CA 94306 Docker Privacy Policy <http://email.notify.docker.com/c/eJxNjssKgzAURL_G7AzxxiS6yKLF9j_y1ND4QG1D_r…>

1 0

Nova iSCSI oddness
by Grant Morley 29 Oct '20

29 Oct '20

Hi all, We are seeing some oddness on a couple of our compute hosts that seems to be related to iSCSI. On a couple of our hosts I am seeing this error in the nova compute logs: 2020-10-27 18:56:14.814 31490 WARNING os_brick.initiator.connectors.iscsi [req-8613ae69-1661-49cf-8bdc-6fec875d01ba - - - - -] Couldn't find iscsi sessions because iscsiadm err: iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session1707 That seems to also stop any instance on the compute host from being able to reboot. Reboots seem to get accepted but the instance never completes and gets stuck in the reboot state: 2020-10-27 19:11:58.891 48612 INFO nova.compute.manager [-] [instance: c8079e85-4777-4615-9d5a-3d1151e11984] During sync_power_state the instance has a pending task (reboot_started_hard). Skip. 2020-10-27 19:11:58.891 48612 INFO nova.compute.manager [-] [instance: 31128f26-910d-411f-98e0-c95dd36f4f0f] During sync_power_state the instance has a pending task (reboot_started_hard). Skip. Does anyone know of a way to resolve this without rebooting the entire compute host? I can't see any other issues other than the fact there is this iSCSI error which in turn seems to stop nova from processing anything for any instance. Any advice would be much appreciated. Regards, -- Grant Morley Cloud Engineer, Civo Ltd Unit H-K, Gateway 1000, Whittle Way Stevenage, Herts, SG1 2FP, UK Visit us at www.civo.com <https://www.civo.com/> Signup for an account now > <https://www.civo.com/signup>

4 9

[tripleo] TripleO at Wallaby PTG wrapup
by Marios Andreou 29 Oct '20

29 Oct '20

Hello all, thanks to everyone that participated and especially thanks to all who prepared and lead our PTG sessions. The number of participants was variable however the maximum attendees (that I noticed at least) per day were Monday:43, Tuesday:48, Wednesday:39. We had three full days with 18 sessions in total. I've included a condensed listing below [3] for convenience and you can find the full schedule at [1] with session descriptions. All sessions were recorded - hopefully we will get access to those soon (I don't have any info about that yet). If you have problems accessing any of the etherpads reach out to me I made backups. Our "team photo" from Monday is available at [2] :) Thanks again TripleO for a great PTG! regards, marios [1] https://etherpad.opendev.org/p/tripleo-ptg-wallaby [2] https://i.imgur.com/apqiYhk.png [3] DAY 1: retrospective & community topics Etherpad: https://etherpad.opendev.org/p/tripleo-ptg-retrospective Etherpad link: https://etherpad.opendev.org/p/tripleo-ptg-wallaby-community-process (Deployment) Composable Network v2 (sans Heat) Proposer: Emilien, Harald & Rabi Etherpad link: https://etherpad.opendev.org/p/composable-networks-v2 (Edge) Topic: State of DCN Proposer: James Slagle Etherpad link: https://etherpad.opendev.org/p/tripleo-ptg-wallaby-edge (PIDONE) Updates from PIDONE Proposer: dciabrin/bandini Etherpad link: https://etherpad.opendev.org/p/tripleo-ptg-wallaby-pidone (Ceph) Topic: review the cephadm integration specs https://review.opendev.org/#/c/757644/ and https://review.opendev.org/#/c/723108/ and https://review.opendev.org/759395 Etherpad link: https://etherpad.opendev.org/p/tripleo-ceph Presenter: [fmount, gfidente, fultonj] DAY 2: (Upgrades) Lessons learned upgrading from Queens to Train, then looking forward at Train to Wallaby Etherpad link: https://etherpad.opendev.org/p/tripleo-PTG-wallaby-upgrades Presenter: Jesse (odyssey4me), Lukas (lbezdick), Jose (jfrancoa), Sergii (holser) (CI) Topic: Dependency pipeline, CentOS Stream and CentOS 9 / Fedora 33 Slides: https://filedn.com/lHMSkrniM7gFSkf96rR711z/TripleODependencyPipeline.pdf Presenter: (rlandy) (Deployment) Topic: Drop heat-monolith local package requirement for undercloud Proposer: Alex Schultz (mwhahaha) Slides: https://docs.google.com/presentation/d/13ds5h9foCHEy3duElGs6mZ3mQwKcaggX6Vj… Blueprint: https://blueprints.launchpad.net/tripleo/+spec/switch-to-heat-all-container (Security) Topic: Updates for Security Etherpad Link: https://etherpad.opendev.org/p/tripleo-PTG-wallaby-security Presenter: Ade (alee) (Upgrades) Topic: Update should only be role inclusion, module and action plugins but we can't do the refactor without any upstream development framework or.... some broader coverage CI. Etherpad link: https://etherpad.opendev.org/p/tripleo-ptg-wallaby-refactor-update (CI) Topic: 0/ so long docker.io Etherpad link: https://hackmd.io/YH9xtBmOQbSVqPu7fjgoog?both Presenter: sshnaidm, chkumar, weshay, marios, DAY 3: (Deployment) Topic: Move to Ansible 2.10.x Proposer: Kevin Carter Etherpad link: https://etherpad.opendev.org/p/tripleo-ansible-210 (Upgrades) Topics: Run all roles in parallel during update by default for massive speed improvment. Etherpad link: https://etherpad.opendev.org/p/tripleo-ptg-wallaby-all-roles-in-parallel (Deployment) Topic: config-download Ansible Collection Etherpad link: https://etherpad.opendev.org/p/ptg-tripleo-config-download-collection Proposer: Luke Short (ekultails) (Deployment) Topic: Removal of Swift from the Undercloud Proposer: Kevin Carter (cloudnull) Etherpad link: https://etherpad.opendev.org/p/tripleo-swift-removal-undercloud (Ceph) Integration Discussion - Continued Etherpad link: https://etherpad.opendev.org/p/tripleo-ceph Presenter: [fmount, gfidente, fultonj] (Networking) Topic: BGP Integration https://review.opendev.org/#/c/758249/ "Spec for installing and configuring FRRouter" Proposer: Dan Sneddon (Cloudops) Topic: Moving deployment logic into ansible roles Proposer: efoley, all (discussion)

1 0

[operators][telemetry] Quick Telemetry survey
by Matthias Runge 29 Oct '20

29 Oct '20

Hi there, you may have seen, not much has been happening lately in OpenStack Telemetry. However, there have been interesting talks at the recent OpenInfra Summit. I would like to gather some data on current Telemetry usage in order to be able to focus a bit better[1]. There is not much value in keeping projects alive, if nobody uses them. At the same time, we may want to increase focus/contribution on often used projects. In order to help with the data, please fill in the form[1] and also feel free to forward this form to interested parties. Thank you, Matthias [1] https://forms.gle/EmncAxggRcG8r31b9

3 3

[devstack][cinder failure]:internal server error(HTTP 500)
by James 29 Oct '20

29 Oct '20

Hi:    I am new to openstack installing on one VM. The environment includes Win10,virtualbox,Ubuntu 18.04; The version of  devstack and openstack  is  Train;  the VM  includes 2vCPU,4G ram, 60G disk .    The process was interrupted by cinder error.[internal server error]        The stack.log shows as stack.log; the c-api log shows as c-api log. stack.log +[38;5;242mlib/cinder:create_volume_types:531 (B[m openstack --os-region-name=RegionOne volume type create --property volume_backend_name=lvmdriver-1 lvmdriver-1 2020-10-29 07:57:11.863 | Internal Server Error (HTTP 500) 2020-10-29 07:57:12.182 | +[38;5;242mlib/cinder:create_volume_types:1 (B[m exit_trap 2020-10-29 07:57:12.232 | +[38;5;242m./stack.sh:exit_trap:536 (B[m local r=1 2020-10-29 07:57:12.259 | ++[38;5;242m./stack.sh:exit_trap:537 (B[m jobs -p 2020-10-29 07:57:12.316 | +[38;5;242m./stack.sh:exit_trap:537 (B[m jobs= 2020-10-29 07:57:12.353 | +[38;5;242m./stack.sh:exit_trap:540 (B[m [[ -n '' ]] 2020-10-29 07:57:12.385 | +[38;5;242m./stack.sh:exit_trap:546 (B[m '[' -f /tmp/tmp.8Y7a6hpwLX ']' 2020-10-29 07:57:12.408 | +[38;5;242m./stack.sh:exit_trap:547 (B[m rm /tmp/tmp.8Y7a6hpwLX 2020-10-29 07:57:12.468 | +[38;5;242m./stack.sh:exit_trap:551 (B[m kill_spinner 2020-10-29 07:57:12.499 | +[38;5;242m./stack.sh:kill_spinner:446 (B[m '[' '!' -z '' ']' 2020-10-29 07:57:12.536 | +[38;5;242m./stack.sh:exit_trap:553 (B[m [[ 1 -ne 0 ]] 2020-10-29 07:57:12.581 | +[38;5;242m./stack.sh:exit_trap:554 (B[m echo 'Error on exit' 2020-10-29 07:57:12.581 | Error on exit 2020-10-29 07:57:12.615 | +[38;5;242m./stack.sh:exit_trap:556 (B[m type -p generate-subunit 2020-10-29 07:57:12.635 | +[38;5;242m./stack.sh:exit_trap:557 (B[m generate-subunit 1603957090 1142 fail 2020-10-29 07:57:14.084 | +[38;5;242m./stack.sh:exit_trap:559 (B[m [[ -z /opt/stack/logs ]] 2020-10-29 07:57:14.110 | +[38;5;242m./stack.sh:exit_trap:562 (B[m /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs 2020-10-29 07:57:17.223 | nova-compute: no process found 2020-10-29 07:57:17.264 | neutron-dhcp-agent: no process found 2020-10-29 07:57:17.338 | neutron-l3-agent: no process found 2020-10-29 07:57:17.409 | neutron-linuxbridge-agent: no process found 2020-10-29 07:57:17.471 | neutron-metadata-agent: no process found 2020-10-29 07:57:17.603 | +[38;5;242m./stack.sh:exit_trap:571 C-api logSTRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=12249) _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:307}} 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: ERROR cinder.api.middleware.fault [req-7d67a008-3478-4da0-8b29-259571c1fb94 req-d823c701-e76a-489b-8ec9-fad9f9504c7b admin admin] Caught error: <class 'oslo_db.exception.DBConnectionError'> (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: (Background on this error at: http://sqlalche.me/e/e3q8) DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: (Background on this error at: http://sqlalche.me/e/e3q8) 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: ERROR cinder.api.middleware.fault Traceback (most recent call last): 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: ERROR cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 85, in __call__ 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: ERROR cinder.api.middleware.fault return req.get_response(self.application)Thanks for your help,firtly.   发自我的iPhone

1 0

[infra][ci][all] Devstack error
by Slawek Kaplonski 29 Oct '20

29 Oct '20

Hi, Recently I see pretty often devstack based jobs failing with error "ERROR] / opt/stack/devstack/functions-common:230 Failure retrieving default route device". I reported it at [1] as Devstack bug but maybe it's something related to some changes in Ubuntu used on the gate. Did You saw something similar also? Maybe You already know why it's like that and have any ideas about how to fix it? Thx in advance for any info. [1] https://bugs.launchpad.net/devstack/+bug/1902002 -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0