October 2020 - openstack-discuss - lists.openstack.org

Fwd: [MURANO] DevStack Murano "yaql_function" Error
by İzzettin Erdem 01 Nov '20

01 Nov '20

Thanks Andy. Regards. Forwarded Conversation Subject: [MURANO] DevStack Murano "yaql_function" Error ------------------------ Gönderen: Andy Botting <andy(a)andybotting.com> Date: 30 Nis 2020 Per, 14:13 To: İzzettin Erdem <root.mch(a)gmail.com> Cc: Rong Zhu <aaronzhu1121(a)gmail.com>, < openstack-discuss(a)lists.openstack.org> Hi İzzettin, Thanks for your interest. Complete log link is at below. It is happening > when i try to deploy or update an environment. > > http://paste.openstack.org/show/792444/ > I've encountered the error AttributeError: 'method' object has no attribute '__yaql_function__' in our environment recently. I believe it started after we upgraded to the Stein release. I haven't tracked it down yet, but calls to the YAQL function: std:Project.getEnvironmentOwner() in a Murano app seem to generate it. It's good to know that at least someone else has reproduced this. cheers, Andy ---------- Gönderen: İzzettin Erdem <root.mch(a)gmail.com> Date: 2 May 2020 Cmt, 16:08 To: Andy Botting <andy(a)andybotting.com> Hi Andy, I solved this problem, it is happening when you try to create or update an environment on existing network. If you change the network with "create new" parameter, it stuck on "creating -or updating- environment" state and after that throws connection error. So i searched Murano network errors and i found this: https://docs.oracle.com/cd/E73172_01/E73173/html/issue-21976631.html. I applied these steps and i created successfully an environment just only with "create new" network parameter. After all these, i realized my devstack environment has no internet connection and because of this murano-agent could not install. So i tried to install Murano on my OSA stable/train test environment and i came with success until last step. Murano-agent installed but this time agent could not connect to rabbitmq instance. I discussed this with OSA IRC channel and i think i have to struggle with this for a long time. Apologize for this long mail but i wanted to inform you. Thanks. Regards. ---------- Gönderen: Andy Botting <andy(a)andybotting.com> Date: 3 May 2020 Paz, 14:24 To: İzzettin Erdem <root.mch(a)gmail.com> Hi İzzettin, Glad you got it sorted. I think you've probably just worked around the issue. Using your new method, you're probably just avoiding the std:Project.getEnvironmentOwner YAQL call now. I've also worked around it in our environment, but I'd like to solve it properly. When I get some time to look into it properly, I'll let you know. cheers, Andy

3 3

[tc][all][ Wallaby Virtual PTG Summary
by Ghanshyam Mann 31 Oct '20

31 Oct '20

Hello Everyone, I am summarizing the TC PTG discussion and action items. Wallaby Leaderless assignment --------------------------------------- Four projects are still pending for leadership assignment. We discussed the next step. If you would like to help any of these or using in your cloud/distro, this is the right time to step up: 1. Karbor - Action item: diablo_rojo will start the retirement process. 2. Qinling - Action item: gmann to start the retirement process. 3. Searchlight -Action item: gmann to start the retirement process. 4. Placement - Action item: gibi will update TC about the final discussion and accordingly mnaser to propose the patch in governance. Victoria Retrospective: ----------------------------- There are a couple of things TC finished in last cycle, few of them are: * Reduced size of TC to speed up the things. * Updated policy to add projects faster. * Outlined Distributed Project Leadership Model as an alternate leadership solution of PTL. Oslo will be the first project to try this new model. * Normalized retirement processes. All the retired repos cleanup is finished. * Tag cleanups & additions This is not complete yet but started with tc-approved release tag in the Victoria cycle. * Merged UC with TC. This is one of the key things finished and the best strategy to make users-developers work more closely. TC Meeting time: ----------------- TC is going to try (re-try) the weekly meeting every Thursday 15:00 UTC. * Action Items: mnaser: propose patch. TC policy regarding OpenStack client: ----------------------------------------------- Forum Discussion: https://etherpad.opendev.org/p/vSummit2020_FeatureGap This is one of the important items and still no good progress on this. From TC perspective, we talked about how to ask or motivate projects to migrate to OSC. TC needs to be very clear on whether it's a strict policy or just guidelines. After long discussion on this we are going with the below strategy: These strategies will be documented as TC resolution. * All user docs should have `openstack` client commands. * Focus on user docs first and then ops docs later * All ci-tooling should have `openstack` client usage * Use the process to find what is missing so we can prioritize it. * Action items: - diablo_rojo to work on initial resolution proposal[1] Finalize the Wallaby cycle goal selection: --------------------------------------------------- There are two proposals for the Wallaby community-wide goals. First is 'oslo.rootwrap to oslo.privsep' which is already being discussed in the last cycle also and all set to be selected[2]. 2nd proposal came up during PTG itself. During policy-popup PTG[3], it came up that deprecating the JSON formatted of policy file will be a good advance step before the projects move to new RBAC policies. This will help operators to smoothly migrate to new policies. This does not involve much work and ok to select as 2nd goal for Wallaby cycle, TC agreed to have below goals for Wallaby cycle: * Migrate from oslo.rootwrap to oslo.privsep [2] * Migrate Default Policy Format from JSON to YAML [4] *Action: - TC to merge the patches which selects both goals for Wallaby cycle. TC tag cleanup: ------------------- In the Victoria cycle, we started the process to audit all the TC tags and start cleanup those. We removed the 'tc:approved-release' tag in the Victoria cycle[5]. In this PTG we discussed two more tags. 1. assert:supports-zero-downtime-upgrade: Currently, there is no project who has this tag and also no testing framework available. Testing for zero downtime is not so easy in upstream testing. We decided to remove this tag as it is advertising something we aren't doing. If anyone interested to spend time on this in the future then we can add it after projects start testing it and document it. 2. assert:supports-api-interoperability: This tag is whether project API is interoperable also or not and this is important from the interop trademark program also. We only have Nova having this tag. Our goal is to promote more projects to apply for this tag. During the discussion, we found that we need to clarify this tag more clearly. For example, this tag is not about implementing the microversion but any versioning schema which provides the feature (API changes) discoverability. And also it is about how we change the API not how our APIs are currently. As long as services have some versioning mechanism to discover the changes and follow the API SIG guidelines for interoperability, and test it via branchless testing way, that service is applicable to apply for this tag. TC will document this tag in a more clear way and encourage each project to start working on applying this tag. * Action: - Graham to put up a change to update the wording to allow for discovery mechanisms[6] - Should include the release in which they started to assert this tag. - also documents that it's about following the guidelines for future changes not existing one. - gmann - Should socialize this out to the projects after we get the documentation improvements landed. - gmann k8s Community Bridging : -------------------------------- This is one of the exciting discussion for everyone. We hosted this cross-community meeting with Kubernetes steering committee teams. Bob and dims from the Kubernetes steering committee joined us in this discussion. It was started with a quick introduction from both teams and then started the discussion on the below topics [11] * Sharing governance models: k8s governance hierarchy is LF->CNCF->k8s steering commitee->various SIG/working group. k8s Steering the committee consists of 7 elected members and doesn't actually influence the direction of the code instead leave it to SIG and arch committee. There is no chair in this committee and host biweekly private as well as public meetings. Each SIG (repos) team has approver and reviewer roles where reviewer review the code and approver are responsible for merging the code. Naser explained the OpenStack governance model. * What are your 3 biggest issues that are barriers to growth/sustainability as a community? as a project? - Going up the contribution ladder is hard. - Stale/emeritus membership in reviewer/approver level groups. - Mono repo makes it so that each SIG involved needs to review which can slow things down. * Area/Domain mapping vs service/repo centered teams - mono repo forces people to work together and it's a little more diverse who is reviewing - mono repo is bad because sometimes things need to be in multiple places like documentation - mono repo struggles with libraries especially on testing the external one. * Related Ecosystem projects comparisons The SIG lead has to sign off and accept that they are willing to take ownership + handle maintenance + releasing etc. It is general consensus that we try to distribute much work as possible to the subgroup and keep it out of k/k. There is general consensus across k8s leadership that work should be delegated out to subgroups. For API interoperability challenge in a distributed model, k8s has conformance tests which exercise the API performance and vendors try and upload conformance results every release or dot release. * Common FOSS topics release and CI part was discussed. and also on how COVID things are impacting community health. k8s community almost lost their independents or part-time contributors. also, the k8s community is doing 3 releases per year compared to 4. * Staying connected going forward To stay connected it will be a good idea if we extend an invite for K8s to join PTG. Upstream Investment Opportunities: ----------------------------------------------- We have three upstream opportunities defined for 2020[1] but there is no help on any of these, even in previous (2018, 2019) upstream opportunities also. We started the discussion if we need to continue this for 2021 years also or just stop defining it and decide the area to help when we have someone interested. Before deciding anything on this mnaser will discuss this with the board of directors and get their opinion. * Action Items: - mnaser to find ways to engage with members and talk with them. - mnaser to talk to board about infrastructure - mnaser to talk to board about platinum members contribution levels + what are they doing to drive engagement. Pop Up Team Check In: ----------------------------- Currently, we have two active popup teams 1. policy 2. Encryption. TC checked the progress on both teams. The policy team is very active and finished some work in Voctoria cycle (cyborg finished it and Barbican started) and also hosted forum[8], PTG sessions[9], and discussed Wallaby development plan. This team host biweekly meeting to discuss and review the progress. The encryption team is also active. Josephine explained the progress on this. Glance spec is merged. Both teams will continue in Wallaby cycle also. Completing the Gerrit breach audit : ---------------------------------------------- There are still 19 teams pending to finish this audit. Those are listed in etherpad - https://etherpad.opendev.org/p/code-audit-gerrit-breach-tracker We encourage all those pending teams to finish the audit, TC members will start following with those projects every week. * Action: - TC to follow up every week on progress Other topics: ----------------- We ran out of time and all the pending (below) topics will be discussed in TC regular meetings. TC will skip this month's meeting[10] but will have weekly meetings on 12th Nov onwards. *Better way to test defined testing runtime *Monitoring in OpenStack: Ceilometer + Telemetry + Gnocchi state *Stable core team maintainer and process to recruit the new members and adding core team members in stable core list. Detailed discussions are in this Etherpad: https://etherpad.opendev.org/p/tc-wallaby-ptg [1] https://review.opendev.org/759904 [2] https://review.opendev.org/#/c/755590/ [3] https://etherpad.opendev.org/p/policy-popup-wallaby-ptg [4] https://review.opendev.org/#/c/759881/ [5] http://lists.openstack.org/pipermail/openstack-discuss/2020-October/017786.… [6] https://review.opendev.org/760562 [7] https://governance.openstack.org/tc/reference/upstream-investment-opportuni… [8] https://etherpad.opendev.org/p/consistent-and-secure-default-policies-walla… [9] https://etherpad.opendev.org/p/policy-popup-wallaby-ptg [10] http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018439.… [11] https://etherpad.opendev.org/p/kubernetes-cross-community-topics -gmann

1 0

devstack installation error - [qa-octavia]
by Ratnopam Chakrabarti 31 Oct '20

31 Oct '20

I tried installing devstack in ubuntu 20.04 with no luck. Are the instructions > at https://docs.openstack.org/devstack/latest/ and > https://docs.openstack.org/devstack/latest/guides/devstack-with-lbaas-v2.ht… > still valid? Could anyone please confirm?

2 3

Weird iptables problem
by frommel＠gmail.com 31 Oct '20

31 Oct '20

Hi, so we deployed a test cluster with 8 total nodes. Openstack ussuri, Ubuntu 20.04 , opevswitch, octavia. Everything works more or less as it should , however when deploying a loadbalancer (which works perfectly), instances on the same node as the amphora deployment loose all access via public up after open switch logs a message : modified security group “long uuid”, which is a sec group belonging to octavia. What effect does that have with the rest? Questions: Should ufw be on and started? How can we troubleshoot this more in depth? Why does the lb work perfectly whole instance with floating up loos all access from the internet but the lb works fine passing thru the web services. One more thing. Does anyone have a recommended sysctl.conf Settings list for Ubuntu/ openstack ussuri? Thanks already and have a nice weekend, //f

1 0

[magnum] Keystone trust credentials failing authentication (after previously working)
by Paul Browne 30 Oct '20

30 Oct '20

Hello Magnum users and devs, I was wondering if any may be able to provide some clues on a mystifying problem we've been seeing with some Magnum-deployed k8s clusters, of older deployment date (v1.17, deployed ~6 months ago) and Keystone trusts used to manage OpenStack cloud resources. I've attached a copy of the Magnum template for the k8s cluster to give an idea of its initial environment. The issue we've been seeing is that the Keystone trust generated at Magnum cluster creation time seems no longer be usable to successfully authenticate to the OpenStack APIs, and so operations such as creation of Manila shares used as k8s Persistent Volume Claims then get stuck in Pending state, forever. The strange thing is that this doesn't happen for newly created Magnum clusters of the same template (in attached file, I use trust credentials for a newly deployed and old cluster; OpenStack API calls for new credentials succeed, whereas old cluster trust fails). Unfortunately following debug-level Keystone logging for rejected trust auth attempts hasn't led me much further to understanding the root issue; the Magnum generated trust exists and has no expiry, but it still seems to be rejected. Keystone debug logging doesn't seem to give much indication as to *why* it is being rejected or what policy violations may be involved, but perhaps there's details hidden in there I'm not seeing yet. Does anyone have any clues as to why this may be happening, or any advice on how we may be able to replace or refresh the trust that the Magnum-created k8s cluster is expecting to use in calling OpenStack APIs? Many thanks, Paul -- ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk Tel: 0044-1223-746548 *******************

1 0

[kolla] Wallaby PTG summary
by Mark Goddard 30 Oct '20

30 Oct '20

Hi, Thanks to everyone who joined the virtual PTG sessions this week. We had some great discussions covering many different topics. I will try to provide a summary here. # Victoria cycle retrospective The Ubuntu Focal migration went smoothly. Core reviewer bandwidth was hurt by the pandemic, but we did pick up towards the end of the cycle and landed a good number of features. As usual, lots of CI fires to extinguish. # Review Victoria PTG Actions We looked back at the actions from the last PTG. Quite a few are still incomplete - we should do a better job of reviewing these. We agreed that the docs-related actions are still the most important. Any help with improving our documentation is appreciated. # General topics ## Ability to run CI jobs locally (without Zuul, but possibly with Ansible) This depends on a Zuul feature called Zuul Runner [1], which should be ready soon. yoctozepto signed up to test it out and document how to use it with Kolla. ## Future of Kolla Klub and Kall The Kolla Klub started well, with lots of involvement, and some good sessions. Over time, attendance has fallen, and the agenda still seems to be largely driven by me. Keeping up productive sessions every two weeks is starting to feel like a bit of a chore, and I think we need to change things up or stop. We agreed to try rebooting at a time that better suits contributors in APAC, possibly recycling some earlier material. There was a proposal at the summit to have monthly ops meetups, and maybe we would do well to link these related communities. We had a few ideas for how to improve the productivity of the more development oriented Kolla Kall, including multiple smaller groups, pair/mob programming, and fewer longer sessions. ## Keep or drop lower-constraints jobs? We didn't find many reasons to keep the lower-constraints jobs, and they do seem to need a bit of maintenance. We voted 2 in favour to remove them, with everyone else not having strong feelings. ## How can we integrate better with our user & contributor base outside of EU/US timezones? Kolla has a large user base outside of EU/US timezones, but the core team and surrounding community is not always that effective at involving these users. We agreed to try to improve the situation by reaching out to known community members in other regions, and offering some limited mentorship. # Kolla ## Infra images This feature has been in progress for a few cycles, but is proving difficult to land. The main blocker now is in CI. We need to split our image build CI jobs into several pipelines. They need to share a local docker registry. This pattern should open up a few other interesting features. ## New Docker Hub limits Docker hub is changing its policy for anonymous and free users. This affects retention of unused images, and limits the number of pulls in a 6 hour period (100 for anonymous users, limited by IP, 200 for logged in free users). This is likely to affect users running without a local registry. Fl1nt agreed to add some information to our documentation to help users. This is likely also to impact our CI jobs, which need to pull images. There is an application process to be listed as an open source project, which we need to ask the opendev infra team to apply for. # Kolla Ansible ## More fine grained skipping of tasks, e.g. allow to skip service registration priteau proposed making it easier to skip certain tasks during deployment, such as bootstrapping or service registration. There are two obvious ways to do this - a new 'action' (e.g. deploy-without-register), or add tags to the relevant tasks, which may be skipped. A third approach is to combine two commands - genconfig & deploy-containers. This is related to work currently going to allow configuration to be generated separately from container deployment. In the end, we considered adding tags, as well as the two command approach. ## Deprecate reconfigure command? The reconfigure command is now essentially the same as deploy. This is a source of confusion for users, so we agreed to deprecate reconfigure. We will keep reconfigure as an alias for deploy at the CLI level. ## Modernise the old skool Swift role The Swift role is the last (except Bifrost, which is generally weird), to use the old format, prior to the better-reconfigure blueprint. This has some limitations, so we should bring it into line. Nothing controversial here, just needs someone to pick it up. ## DNS-based endpoint naming rafaelweingartne has proposed a mini spec [2] for DNS-based endpoint naming. I think we now have a reasonable idea of his solution (which is to use Consul for service discovery, and avoid HAProxy for internal API communication), however the hard part is defining the problem that is being solved. ## Support to identity federation (OpenID Connect) configurations in Keystone and Horizon via Kolla-ansible There is a patch [3] to support this which has been open for some time. It requires some fairly specific knowledge to review, and the challenge will be in making core reviewers confident that it works. We agreed that CI testing and/or documentation of a simple development environment would help. ## podman Podman keeps coming up as a replacement for Docker. There are several issues here, including support on Debian/Ubuntu - we do not want to maintain support for two container engines. We agreed that this is likely to be a large feature, and that we should get started with a proof of concept in Wallaby, possibly with a spec to tease out the details. Fl1nt may have some time early 2021 for this, but anyone is welcome to pick it up. ## Finish Masakari integration We don't have support for the host monitor, which is arguably the most useful part of Masakari. With yoctozepto as Masakari PTL, we expect to see good progress in Wallaby. ## Performance improvements We made some good progress with performance improvement in Victoria. There is always more that can be done, and the next phase will probably involve splitting roles that target both control and compute hosts. Mitogen [4] is also an interesting option. As always, changes should be based on benchmarks [5]. ## Let's Encrypt integration - container running certbot, triggers certificate distribution yoctozepto added initial support for the ACME protocol in Ussuri, and the next step is to support deployment of a certbot container, and unattended certificate rotation. There are lots of fiddly details here, including how to bootstrap it. headphoneJames offered to write up a high level description of how the pieces fit together. ## Support for HAProxy reload (e.g. for TLS) Currently any configuration changes to HAProxy require a restart to be applied. This causes connections to be dropped. We discussed adding support for hitless reloads of HAProxy [6]. ## Native fluentd logging We didn't discuss this as kevko was busy, but I thought I'd add it here anyway. The idea is to use use logging.conf to configure services for native fluentd logging. This avoids all the regexes in fluentd, but may change our document schema in Elasticsearch. Patch [7] is up for review. # Kayobe ## Support multiple environments from a single kayobe configuration This feature has been on the cards for some time, and is an important step for Kayobe. priteau has proposed a patch [8] which is a good starting point. We spent a good hour going over some of the outstanding issues, and made some good progress. We agreed on the need to keep the meaning of KAYOBE_CONFIG_PATH intact, and to allow a branch to define the default environment to use. We agreed to write a high level design document to pull out the remaining uncertainties, and aim to get it over the line in Wallaby. ## Switching Docker storage from devicemapper to overlay2 We switched the default Docker storage driver from devicemapper to overlay2 in Victoria. For existing users of devicemapper, the plan is to try to ensure they are migrated to overlay2 during CentOS 8 migration. Further, we will deprecate devicemapper on stable branches, and require users to take some action (set a variable) to allow its use. This should deter new users from using devicemapper, without completely breaking existing users. # Switch to NetworkManager? jovial proposed this discussion, as he likes the checkpoint/restore feature of NetworkManager. There are various ways to interface with NM, including nm-cli, an ansible module, templated config files + nmcli con reload. ## Strip out Grafana post configure functionality and move it to Kolla-Ansible dougszu proposed this topic, to improve the separation of concerns between Kayobe and Kolla Ansible. It may also apply to setup we do for ironic. ## Create a reference custom playbook repo We have a stackhpc repo [9] with various useful playbooks and scripts. dougszu proposed making something more official. I suggested that Ansible collections could be used to improve the user experience with custom playbooks in external repos. ## Config gen on ansible control host This is something we've talked about for a while - offline generation of configuration, and potentially validating it in a CI pipeline. jovial noted the importance of being able to redact passwords. This is mostly dependent on ongoing changes to Kolla Ansible, but could be wrapped up nicely by Kayobe. ## Hashivault integration While we have integrated with Hashicorp vault for secret retrieval, the user experience could be better. We also lack support for pushing passwords generated by kolla up to Vault. This is likely to be a Kolla Ansible feature, plus some Kayobe integration. [1] https://review.opendev.org/681277 [2] https://review.opendev.org/759706 [3] https://review.opendev.org/695432 [4] https://github.com/dw/mitogen [5] https://github.com/stackhpc/ansible-scaling [6] https://www.haproxy.com/blog/hitless-reloads-with-haproxy-howto/ [7] https://review.opendev.org/755775 [8] https://review.opendev.org/734867 [9] https://github.com/stackhpc/kayobe-ops Cheers, Mark

1 0

Devstack installation failure
by Ratnopam Chakrabarti 30 Oct '20

30 Oct '20

Hi, Has anyone successfully installed devstack in last 2 weeks with stack.sh script?It was working well for me before that. I am seeing failures when I try to install on Ubuntu 18.04. Here's the trace - [Call Trace] ./stack.sh:1304:start_glance /home/stack/devstack/lib/glance:480:die [ERROR] /home/stack/devstack/lib/glance:480 g-api did not start Error on exit nova-compute: no process found neutron-dhcp-agent: no process found neutron-l3-agent: no process found neutron-metadata-agent: no process found neutron-openvswitch-agent: no process found git repo i am using to download devstack is https://git.openstack.org/openstack-dev/devstackI tried with with stable/victoria branch also, same issue. Any idea on how to solve this? Thanks

2 1

[tc] monthly meeting skip
by Mohammed Naser 30 Oct '20

30 Oct '20

Hello everyone, Our monthly meeting was scheduled for next Thursday, Nov 5th. Since the PTG took place this week, we will be skipping this month’s meeting and consider the PTG as our monthly meeting instead. Thanks, Regards, Mohammed -- Mohammed Naser VEXXHOST, Inc.

1 0

[ansible][tripleo][openstack-ansible-modules] Openstack Ansible modules PTG wrap up
by Sagi Shnaidman 30 Oct '20

30 Oct '20

Openstack Ansible modules PTG wrap up 1. We continue to align new features in SDK, OSC and modules a) need to complete switch of cloud layer to use ONLY proxy layer and not reimplement the APIs b) in Ansible modules we'd better to use cloud layer when it's available and not proxy layer 2. For exposing SDK logs when running modules, we need to find a right place to print them. Either it would be one of module returns or special plugin printing logs with more verbosity. The discussion will be in the maillist. 3. We will start developing unit tests for modules and run them with ansible-test. 4. Also need to rework our current test system and make it more atomic and add coverage. 5. We decided to align all *_info modules to one standard - <resource>_info, some of them named as <resource>s_info. 6. We'll need to generate documentation for modules and place it on docs.openstack.org if it's possible. Current documentation in Ansible covers the release that is built-in in Ansible and doesn't reflect last changes. 7. Effort of moving modules to use a single OpenstackModule class will continue. 8. Transition and adding new baremetal modules will continue as well. Thanks -- Best regards Sagi Shnaidman

1 0

Re: [Openstack][cinder] dell unity iscsi faulty devices
by Gorka Eguileor 30 Oct '20

30 Oct '20

On 30/10, Ignazio Cassano wrote: > Please, se the last email where I upgraded to the last openstack nova on > queens. > [root@compute-0 nova]# rpm -qa|grep queens > centos-release-openstack-queens-1-2.el7.centos.noarch > [root@compute-0 nova]# rpm -qa|grep nova > openstack-nova-compute-17.0.13-1.el7.noarch > openstack-nova-common-17.0.13-1.el7.noarch > python-nova-17.0.13-1.el7.noarch > python2-novaclient-10.1.0-1.el7.noarch > > I sent you the logs on the update release. > I am not so skilled for reading fine logs output about this issue. > Sorry > Ignazio > Hi, I missed that email and the attachment, sorry. The logs you sent me were missing most of the connect_volume call, and only the end of the call was present, but I think it doesn't matter as I see what the problem is. The problem is that some of the nodes and sessions are duplicated. An example of a duplicated node: tcp: [3] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) tcp: [4] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) An example of that node's duplicated session: tcp: [3] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) tcp: [4] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) And os-brick is not prepared to handle that, because it is programmed to reuse the nodes and sesssions. So on disconnect it get's the first of each to look for the volumes provided by it. In the example of the duplicated node-session aboveit sees that it provides /dev/sdd, but that is not one of the disks that belong to the multipath that we are disconnecting, so it gets ignored. The volume we are looking for it's probably on the second session. So from this point forward (where we have duplicated node-sessions) it will not work again. I recommend you clean up that system so that you don't have duplicated nodes and sessions before trying to do a VM migration with a single volume attached. If that works, then try to attach 2 volumes on instances on the same host and see if the nodes and sessions are duplicated. Cheers, Gorka. > Il giorno ven 30 ott 2020 alle ore 09:27 Gorka Eguileor <geguileo(a)redhat.com> > ha scritto: > > > On 30/10, Ignazio Cassano wrote: > > > Hello, these are versions we are using: > > > [root@podto2-kvm02 ansible]# rpm -qa|grep queens > > > centos-release-openstack-queens-1-2.el7.centos.noarch > > > [root@podto2-kvm02 ansible]# rpm -qa|grep nova > > > openstack-nova-common-17.0.11-1.el7.noarch > > > python2-novaclient-10.1.0-1.el7.noarch > > > openstack-nova-compute-17.0.11-1.el7.noarch > > > python-nova-17.0.11-1.el7.noarch > > > > > > Cheers,Gorka. > > > > Hi, > > > > That release has the Nova bug fix, so Nova should not be calling Cinder > > to do an initialize connection on the source on the post-migration step > > anymore. > > > > I recommend comparing the connection info passed to connect_volume when > > the volume is attached on the source host and when it's disconnected on > > the post-migration step on the source host. > > > > Cheers, > > Gorka. > > > > > > > > > > > > > [1]: https://bugs.launchpad.net/nova/+bug/1754716 > > > > [2]: > > > > > > https://github.com/openstack/nova/commit/013f421bca4067bd430a9fac1e3b290cf1… > > > > > > > > > > > > > > > > > > > Il giorno gio 29 ott 2020 alle ore 09:12 Gorka Eguileor < > > > > geguileo(a)redhat.com> > > > > > ha scritto: > > > > > > > > > > > On 28/10, Ignazio Cassano wrote: > > > > > > > Hello Gorka, I would like to know if with unity iscsi driver, I > > must > > > > > > > configure iscsi initiator on both compute and controller nodes. > > > > > > > At this time I installed and cinfigured iscsi initiator only on > > > > compute > > > > > > > nodes and I got a lot of faulty devices when volumes ate > > detached. > > > > > > > Thanks > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > Both compute and controller nodes are in the data path. Computes > > when > > > > > > instances use the volumes, and controllers when we create volume > > from > > > > > > images, do generic volume migrations, create or restore backups, > > etc. > > > > > > > > > > > > Unless your deployment isn't doing any of the Cinder operations > > that > > > > > > involve the data plane, you'll have to configure iSCSI on the > > > > controller > > > > > > as well. > > > > > > > > > > > > Having said that, whether you configure the iSCSI initiator or not > > on > > > > > > the controller will have no effect on the paths used by the > > compute. > > > > > > > > > > > > I've seen the iSCSI initiator going crazy when the iscsid and the > > > > > > iscsiadm are from different versions. I've seen this in > > containerized > > > > > > environments. > > > > > > > > > > > > Faulty paths on multipathing is a tricky business, because there > > are > > > > > > different checkers, some generic (readsector0, tur, directio) and > > some > > > > > > vendor specific (emc_clarrion, hp_wd, rdac), and each one behaves > > in a > > > > > > different way. > > > > > > > > > > > > If you have a multipath device with faulty paths, that you think > > should > > > > > > not be faulty, you should look into what's going on with those > > paths: > > > > > > > > > > > > - Confirm that the device is still in the system under /dev/XYZ > > > > > > - Confirm in your storage array's console/webconsole that the > > volume is > > > > > > still mapped on that target-portal to that host's iscsi initiator > > > > > > name. > > > > > > - Confirm you can read the faulty devices with dd on the host > > > > > > - Confirm that the WWN of the device is the same in all the paths > > > > (using > > > > > > /lib/udev/scsi_id) > > > > > > - Finally look into what checker is multipath using for your device > > > > > > (sometimes checkers have bugs). > > > > > > > > > > > > Cheers, > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > Il Mar 20 Ott 2020, 19:58 Gorka Eguileor <geguileo(a)redhat.com> > > ha > > > > > > scritto: > > > > > > > > > > > > > > > On 20/10, Ignazio Cassano wrote: > > > > > > > > > This is the entre log from when the migration started: > > > > > > > > > > > > > > > > > > http://paste.openstack.org/show/799199/ > > > > > > > > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > There are no os-brick calls in there. :-( > > > > > > > > > > > > > > > > You should look for the call to connect_volume that should have > > > > > > > > something like: > > > > > > > > > > > > > > > > ==> disconnect_volume: call "{'args': > > > > > > > > (<os_brick.initiator.connectors.iscsi > > > > > > > > > > > > > > > > And the second parameter to that call is a dictionary where you > > > > can see > > > > > > > > the target_lun, target_luns, target_portals, target_portal, > > > > target_iqn, > > > > > > > > target_iqns... This will allow us to check if we are actually > > > > > > connected > > > > > > > > to those targets-portals > > > > > > > > > > > > > > > > The third parameter should contain two things that are > > relevant, > > > > the > > > > > > > > scsi_wwn and the path. You can check if the path exists and if > > > > that > > > > > > > > path actually has that wwn using /lib/udev/scsi_id --page 0x83 > > > > > > > > --whitelisted $path > > > > > > > > > > > > > > > > Those are the things I would check, because the only reason I > > can > > > > think > > > > > > > > that os-brick is not disconnecting any volumes are that the > > > > connection > > > > > > > > info is not right, or that the volume is no longer connected. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > Il giorno mar 20 ott 2020 alle ore 11:23 Gorka Eguileor < > > > > > > > > geguileo(a)redhat.com> > > > > > > > > > ha scritto: > > > > > > > > > > > > > > > > > > > On 20/10, Ignazio Cassano wrote: > > > > > > > > > > > Hello Gorka,this is what happens on nova compute with > > debug > > > > > > enabled, > > > > > > > > > > when I > > > > > > > > > > > migrate an instance with iscsi volumes ( note > > Disconnecting > > > > > > from[] > > > > > > > > should > > > > > > > > > > > be the issue): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > The disconnect from [] is the right clue, not necessarily > > the > > > > > > issue. > > > > > > > > > > > > > > > > > > > > OS-Brick is saying that for the connection information > > that has > > > > > > been > > > > > > > > > > passed in the "disconnect_volume" call (which is not > > present > > > > in the > > > > > > > > > > emailed logs) there are no volumes present in the system. > > > > > > > > > > > > > > > > > > > > You should check the connection info that Nova is passing > > to > > > > > > > > > > disconnect_volume and confirm if that data is correct. For > > > > example > > > > > > > > > > checking if the path present in the connection info > > dictionary > > > > is > > > > > > the > > > > > > > > > > same as the one in the instance's XML dump, or if the LUN > > from > > > > the > > > > > > > > > > connection info dict is actually present in the system. > > > > > > > > > > > > > > > > > > > > There are multiple reasons why Nova could be passing the > > wrong > > > > > > > > > > connection info to os-brick. The ones that come to mind > > are: > > > > > > > > > > > > > > > > > > > > - There was a failed migration at some point, and Nova > > didn't > > > > > > rollback > > > > > > > > > > the connection info on the BDM table. > > > > > > > > > > - Nova is calling multiple times initialize_connection on > > > > Cinder > > > > > > for > > > > > > > > the > > > > > > > > > > same host and the driver being used is not idempotent. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > stderr= _run_iscsiadm_bare > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1122 > > > > > > > > > > > 2020-10-20 09:52:33.066 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] iscsi session list stdout=tcp: [10] > > 10.138.209.48:3260,9 > > > > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a3 (non-flash) > > > > > > > > > > > tcp: [11] 10.138.215.17:3260,8 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [12] 10.138.215.17:3260,8 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [13] 10.138.215.18:3260,7 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [14] 10.138.215.18:3260,7 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [15] 10.138.209.47:3260,6 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b3 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [16] 10.138.209.47:3260,6 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b3 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [9] 10.138.209.48:3260,9 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a3 > > > > > > > > > > > (non-flash) > > > > > > > > > > > stderr= _run_iscsi_session > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1111 > > > > > > > > > > > 2020-10-20 09:52:33.078 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] Resulting device map defaultdict(<function <lambda> > > at > > > > > > > > > > 0x7f4f1b1f7cf8>, > > > > > > > > > > > {(u'10.138.215.17:3260', > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.a2'): > > > > > > > > > > > (set([]), set([u'sdg', u'sdi'])), (u'10.138.209.47:3260 > > ', > > > > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.b3'): (set([]), > > > > > > set([u'sdo', > > > > > > > > > > > u'sdq'])), (u'10.138.209.48:3260', > > > > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.a3'): (set([]), > > > > > > set([u'sdd', > > > > > > > > > > > u'sdb'])), (u'10.138.215.18:3260', > > > > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.b2'): (set([]), > > > > > > set([u'sdm', > > > > > > > > > > > u'sdk']))}) _get_connection_devices > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:844 > > > > > > > > > > > 2020-10-20 09:52:33.078 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] Disconnecting from: [] _disconnect_connection > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1099 > > > > > > > > > > > 2020-10-20 09:52:33.079 132171 DEBUG > > > > oslo_concurrency.lockutils > > > > > > [-] > > > > > > > > Lock > > > > > > > > > > > "connect_volume" released by > > > > > > > > > > > "os_brick.initiator.connectors.iscsi.disconnect_volume" > > :: > > > > held > > > > > > > > 1.058s > > > > > > > > > > > inner > > > > > > > > > > /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 > > > > > > > > > > > 2020-10-20 09:52:33.079 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] <== disconnect_volume: return (1057ms) None > > > > > > trace_logging_wrapper > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/utils.py:170 > > > > > > > > > > > 2020-10-20 09:52:33.079 132171 DEBUG > > > > > > nova.virt.libvirt.volume.iscsi > > > > > > > > [-] > > > > > > > > > > > [instance: 0c846f66-f194-40de-b31e-d53652570fa7] > > Disconnected > > > > > > iSCSI > > > > > > > > > > Volume > > > > > > > > > > > disconnect_volume > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:78 > > > > > > > > > > > 2020-10-20 09:52:33.080 132171 DEBUG os_brick.utils [-] > > ==> > > > > > > > > > > > get_connector_properties: call u"{'execute': None, > > 'my_ip': > > > > > > > > > > > '10.138.208.178', 'enforce_multipath': True, 'host': > > > > > > > > 'podiscsivc-kvm02', > > > > > > > > > > > 'root_helper': 'sudo nova-rootwrap > > /etc/nova/rootwrap.conf', > > > > > > > > 'multipath': > > > > > > > > > > > True}" trace_logging_wrapper > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/utils.py:146 > > > > > > > > > > > 2020-10-20 09:52:33.125 132171 DEBUG > > > > os_brick.initiator.linuxfc > > > > > > [-] > > > > > > > > No > > > > > > > > > > > Fibre Channel support detected on system. get_fc_hbas > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157 > > > > > > > > > > > 2020-10-20 09:52:33.126 132171 DEBUG > > > > os_brick.initiator.linuxfc > > > > > > [-] > > > > > > > > No > > > > > > > > > > > Fibre Channel support detected on system. get_fc_hbas > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157 > > > > > > > > > > > 2020-10-20 09:52:33.145 132171 DEBUG os_brick.utils [-] > > <== > > > > > > > > > > > get_connector_properties: return (61ms) {'initiator': > > > > > > > > > > > u'iqn.1994-05.com.redhat:fbfdc37eed4c', 'ip': > > > > u'10.138.208.178', > > > > > > > > 'system > > > > > > > > > > > uuid': u'4C4C4544-0051-4E10-8057-B6C04F425932', > > 'platform': > > > > > > > > u'x86_64', > > > > > > > > > > > 'host': u'podiscsivc-kvm02', 'do_local_attach': False, > > > > 'os_type': > > > > > > > > > > > u'linux2', 'multipath': True} trace_logging_wrapper > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/utils.py:170 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > > > Il giorno gio 15 ott 2020 alle ore 10:57 Gorka Eguileor < > > > > > > > > > > geguileo(a)redhat.com> > > > > > > > > > > > ha scritto: > > > > > > > > > > > > > > > > > > > > > > > On 14/10, Ignazio Cassano wrote: > > > > > > > > > > > > > Hello, thank you for the answer. > > > > > > > > > > > > > I am using os-brick 2.3.8 but I got same issues on > > stein > > > > with > > > > > > > > > > os.brick > > > > > > > > > > > > 2.8 > > > > > > > > > > > > > For explain better the situation I send you the > > output of > > > > > > > > multipath > > > > > > > > > > -ll > > > > > > > > > > > > on > > > > > > > > > > > > > a compute node: > > > > > > > > > > > > > root@podvc-kvm01 ansible]# multipath -ll > > > > > > > > > > > > > Oct 14 18:50:01 | sdbg: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbe: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbd: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbf: alua not supported > > > > > > > > > > > > > 360060160f0d049007ab7275f743d0286 dm-11 DGC > > ,VRAID > > > > > > > > > > > > > size=30G features='1 retain_attached_hw_handler' > > > > hwhandler='1 > > > > > > > > alua' > > > > > > > > > > wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > | |- 15:0:0:71 sdbg 67:160 failed faulty running > > > > > > > > > > > > > | `- 12:0:0:71 sdbe 67:128 failed faulty running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > |- 11:0:0:71 sdbd 67:112 failed faulty running > > > > > > > > > > > > > `- 13:0:0:71 sdbf 67:144 failed faulty running > > > > > > > > > > > > > 360060160f0d049004cdb615f52343fdb dm-8 DGC ,VRAID > > > > > > > > > > > > > size=80G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 15:0:0:210 sdau 66:224 active ready running > > > > > > > > > > > > > | `- 12:0:0:210 sdas 66:192 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 11:0:0:210 sdar 66:176 active ready running > > > > > > > > > > > > > `- 13:0:0:210 sdat 66:208 active ready running > > > > > > > > > > > > > 360060160f0d0490034aa645fe52265eb dm-12 DGC > > ,VRAID > > > > > > > > > > > > > size=100G features='2 queue_if_no_path > > > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 12:0:0:177 sdbi 67:192 active ready running > > > > > > > > > > > > > | `- 15:0:0:177 sdbk 67:224 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 11:0:0:177 sdbh 67:176 active ready running > > > > > > > > > > > > > `- 13:0:0:177 sdbj 67:208 active ready running > > > > > > > > > > > > > 360060160f0d04900159f225fd6126db9 dm-6 DGC ,VRAID > > > > > > > > > > > > > size=40G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 11:0:0:26 sdaf 65:240 active ready running > > > > > > > > > > > > > | `- 13:0:0:26 sdah 66:16 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 12:0:0:26 sdag 66:0 active ready running > > > > > > > > > > > > > `- 15:0:0:26 sdai 66:32 active ready running > > > > > > > > > > > > > Oct 14 18:50:01 | sdba: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbc: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdaz: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbb: alua not supported > > > > > > > > > > > > > 360060160f0d049007eb7275f93937511 dm-10 DGC > > ,VRAID > > > > > > > > > > > > > size=40G features='1 retain_attached_hw_handler' > > > > hwhandler='1 > > > > > > > > alua' > > > > > > > > > > wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > | |- 12:0:0:242 sdba 67:64 failed faulty running > > > > > > > > > > > > > | `- 15:0:0:242 sdbc 67:96 failed faulty running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > |- 11:0:0:242 sdaz 67:48 failed faulty running > > > > > > > > > > > > > `- 13:0:0:242 sdbb 67:80 failed faulty running > > > > > > > > > > > > > 360060160f0d049003a567c5fb72201e8 dm-7 DGC ,VRAID > > > > > > > > > > > > > size=40G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 12:0:0:57 sdbq 68:64 active ready running > > > > > > > > > > > > > | `- 15:0:0:57 sdbs 68:96 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 11:0:0:57 sdbp 68:48 active ready running > > > > > > > > > > > > > `- 13:0:0:57 sdbr 68:80 active ready running > > > > > > > > > > > > > 360060160f0d04900c120625f802ea1fa dm-9 DGC ,VRAID > > > > > > > > > > > > > size=25G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 11:0:0:234 sdav 66:240 active ready running > > > > > > > > > > > > > | `- 13:0:0:234 sdax 67:16 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 15:0:0:234 sday 67:32 active ready running > > > > > > > > > > > > > `- 12:0:0:234 sdaw 67:0 active ready running > > > > > > > > > > > > > 360060160f0d04900b8b0615fb14ef1bd dm-3 DGC ,VRAID > > > > > > > > > > > > > size=50G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 11:0:0:11 sdan 66:112 active ready running > > > > > > > > > > > > > | `- 13:0:0:11 sdap 66:144 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 12:0:0:11 sdao 66:128 active ready running > > > > > > > > > > > > > `- 15:0:0:11 sdaq 66:160 active ready running > > > > > > > > > > > > > > > > > > > > > > > > > > The active running are related to running virtual > > > > machines. > > > > > > > > > > > > > The faulty are related to virtual macnines migrated > > on > > > > other > > > > > > kvm > > > > > > > > > > nodes. > > > > > > > > > > > > > Every volume has 4 path because iscsi on unity needs > > two > > > > > > > > different > > > > > > > > > > vlans, > > > > > > > > > > > > > each one with 2 addresses. > > > > > > > > > > > > > I think this issue can be related to os-brick because > > > > when I > > > > > > > > migrate > > > > > > > > > > a > > > > > > > > > > > > > virtual machine from host A host B in the cova > > compute > > > > log on > > > > > > > > host A > > > > > > > > > > I > > > > > > > > > > > > read: > > > > > > > > > > > > > 2020-10-13 10:31:02.769 118727 DEBUG > > > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > > > [req-771ede8c-6e1b-4f3f-ad4a-1f6ed820a55c > > > > > > > > > > > > 66adb965bef64eaaab2af93ade87e2ca > > > > > > > > > > > > > 85cace94dcc7484c85ff9337eb1d0c4c - default default] > > > > > > > > *Disconnecting > > > > > > > > > > from: > > > > > > > > > > > > []* > > > > > > > > > > > > > > > > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > That's definitely the right clue!! Though I don't > > fully > > > > agree > > > > > > with > > > > > > > > > > this > > > > > > > > > > > > being an os-brick issue just yet. ;-) > > > > > > > > > > > > > > > > > > > > > > > > Like I mentioned before, RCA is usually non-trivial, > > and > > > > > > > > explaining how > > > > > > > > > > > > to debug these issues over email is close to > > impossible, > > > > but if > > > > > > > > this > > > > > > > > > > > > were my system, and assuming you have tested normal > > > > > > attach/detach > > > > > > > > > > > > procedure and is working fine, this is what I would do: > > > > > > > > > > > > > > > > > > > > > > > > - Enable DEBUG logs on Nova compute node (I believe you > > > > already > > > > > > > > have) > > > > > > > > > > > > - Attach a new device to an instance on that node with > > > > --debug > > > > > > to > > > > > > > > get > > > > > > > > > > > > the request id > > > > > > > > > > > > - Get the connection information dictionary that > > os-brick > > > > > > receives > > > > > > > > on > > > > > > > > > > > > the call to connect_volume for that request, and the > > data > > > > > > that > > > > > > > > > > > > os-brick returns to Nova on that method call > > completion. > > > > > > > > > > > > - Check if the returned data to Nova is a multipathed > > > > device or > > > > > > > > not (in > > > > > > > > > > > > 'path'), and whether we have the wwn or not (in > > > > > > 'scsi_wwn'). It > > > > > > > > > > > > should be a multipath device, and then I would check > > its > > > > > > status > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > multipath daemon. > > > > > > > > > > > > - Now do the live migration (with --debug to get the > > > > request > > > > > > id) > > > > > > > > and > > > > > > > > > > see > > > > > > > > > > > > what information Nova passes in that request to > > > > os-brick's > > > > > > > > > > > > disconnect_volume. > > > > > > > > > > > > - Is it the same? Then it's likely an os-brick issue, > > > > and I > > > > > > can > > > > > > > > have > > > > > > > > > > a > > > > > > > > > > > > look at the logs if you put the logs for that > > os-brick > > > > > > detach > > > > > > > > > > > > process in a pastebin [1]. > > > > > > > > > > > > - Is it different? Then it's either a Nova bug or a > > > > Cinder > > > > > > driver > > > > > > > > > > > > specific bug. > > > > > > > > > > > > - Is there a call from Nova to Cinder, in the > > migration > > > > > > > > request, > > > > > > > > > > for > > > > > > > > > > > > that same volume to initialize_connection > > passing the > > > > > > source > > > > > > > > host > > > > > > > > > > > > connector info (info from the host that is > > currently > > > > > > > > attached)? > > > > > > > > > > > > If there is a call, check if the returned data is > > > > > > different > > > > > > > > from > > > > > > > > > > > > the one we used to do the attach, if that's the > > case > > > > then > > > > > > > > it's a > > > > > > > > > > > > Nova and Cinder driver bug that was solved on the > > > > Nova > > > > > > side > > > > > > > > in > > > > > > > > > > > > 17.0.10 [2]. > > > > > > > > > > > > - If there's no call to Cinder's > > > > initialize_connection, the > > > > > > > > it's > > > > > > > > > > > > most likely a Nova bug. Try to find out if this > > > > > > connection > > > > > > > > info > > > > > > > > > > > > makes any sense for that host (LUN, target, > > etc.) or > > > > if > > > > > > this > > > > > > > > is > > > > > > > > > > > > the one from the destination volume. > > > > > > > > > > > > > > > > > > > > > > > > I hope this somehow helps. > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: http://paste.openstack.org/ > > > > > > > > > > > > [2]: https://review.opendev.org/#/c/637827/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Il giorno mer 14 ott 2020 alle ore 13:41 Gorka > > Eguileor < > > > > > > > > > > > > geguileo(a)redhat.com> > > > > > > > > > > > > > ha scritto: > > > > > > > > > > > > > > > > > > > > > > > > > > > On 09/10, Ignazio Cassano wrote: > > > > > > > > > > > > > > > Hello Stackers, I am using dell emc iscsi driver > > on > > > > my > > > > > > > > centos 7 > > > > > > > > > > > > queens > > > > > > > > > > > > > > > openstack. It works and instances work as well > > but on > > > > > > compute > > > > > > > > > > nodes I > > > > > > > > > > > > > > got a > > > > > > > > > > > > > > > lot a faulty device reported by multipath il > > comand. > > > > > > > > > > > > > > > I do know why this happens, probably attacching > > and > > > > > > detaching > > > > > > > > > > > > volumes and > > > > > > > > > > > > > > > live migrating instances do not close something > > well. > > > > > > > > > > > > > > > I read this can cause serious performances > > problems > > > > on > > > > > > > > compute > > > > > > > > > > nodes. > > > > > > > > > > > > > > > Please, any workaround and/or patch is suggested > > ? > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > There are many, many, many things that could be > > > > happening > > > > > > > > there, > > > > > > > > > > and > > > > > > > > > > > > > > it's not usually trivial doing the RCA, so the > > > > following > > > > > > > > questions > > > > > > > > > > are > > > > > > > > > > > > > > just me hoping this is something "easy" to find > > out. > > > > > > > > > > > > > > > > > > > > > > > > > > > > What os-brick version from Queens are you running? > > > > Latest > > > > > > > > > > (2.3.9), or > > > > > > > > > > > > > > maybe one older than 2.3.3? > > > > > > > > > > > > > > > > > > > > > > > > > > > > When you say you have faulty devices reported, are > > > > these > > > > > > faulty > > > > > > > > > > devices > > > > > > > > > > > > > > alone in the multipath DM? Or do you have some > > faulty > > > > ones > > > > > > with > > > > > > > > > > some > > > > > > > > > > > > > > that are ok? > > > > > > > > > > > > > > > > > > > > > > > > > > > > If there are some OK and some that aren't, are they > > > > > > consecutive > > > > > > > > > > > > devices? > > > > > > > > > > > > > > (as in /dev/sda /dev/sdb etc). > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

2 2

Re: [kolla] Setting up eth1 on VM for AIO
by Braden, Albert 30 Oct '20

30 Oct '20

From: Mark Goddard <mark(a)stackhpc.com> Sent: Friday, October 30, 2020 4:37 AM To: Radosław Piliszek <radoslaw.piliszek(a)gmail.com> Cc: Braden, Albert <C-Albert.Braden(a)charter.com>; OpenStack Discuss <openstack-discuss(a)lists.openstack.org> Subject: Re: [kolla] Setting up eth1 on VM for AIO On Thu, 29 Oct 2020 at 16:20, Radosław Piliszek <radoslaw.piliszek(a)gmail.com<mailto:radoslaw.piliszek@gmail.com>> wrote: Hi Albert! Sorry for top posting but I don't really see which part to quote would be best so they all lose. :D Indeed, it's tricky to handle this if you are not a fan of linux networking. I guess we should document an example way.... Anyhow, the way e.g. Kayobe handles it for users (as it deals with baremetal details) is to use bridges and veths. Basically, you create a bridge, attach it to the expected interface, set any networking you still want on that network for this host on this bridge. Then, you create a veth pair and connect one end to it and the second one is given to neutron. >Alternatively, if this is just for development, you can use a dummy interface. > >ip link add link eth1 type dummy > >It's not persistent, but good enough for testing. If you need external network access from it you could add a MASQUERADE iptables rule and enable the IP >forwarding sysctl. That seems to be working. After googling a bit I was able to make it persistent: [root@adjutant-poc ~]# cat /etc/modules-load.d/dummy.conf dummy [root@adjutant-poc ~]# cat /etc/modprobe.d/dummy.conf install dummy /sbin/modprobe --ignore-install dummy; /sbin/ip link set name eth1 dev dummy0 [root@adjutant-poc ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 NAME=eth1 BOOTPROTO=none DEVICE=eth1 ONBOOT=yes TYPE=Ethernet NM_CONTROLLED=no [root@adjutant-poc ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:22:c4:e2 brd ff:ff:ff:ff:ff:ff inet 192.168.1.83/22 brd 192.168.3.255 scope global dynamic eth0 valid_lft 85821sec preferred_lft 85821sec inet6 fe80::f816:3eff:fe22:c4e2/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 7e:e1:fe:31:c9:2e brd ff:ff:ff:ff:ff:ff inet6 fe80::7ce1:feff:fe31:c92e/64 scope link valid_lft forever preferred_lft forever -yoctozepto On Thu, Oct 29, 2020 at 12:49 PM Braden, Albert <C-Albert.Braden(a)charter.com<mailto:C-Albert.Braden@charter.com>> wrote: > > I’m trying to setup an allinone kolla-ansible Train POC on a VM, and I’m following the instructions here: > > > > https://docs.openstack.org/kolla-ansible/train/user/quickstart.html > > > > Under “Prepare initial configuration” where it talks about globals.yml, it says I need 2 network interfaces: > > > > network_interface: "eth0" > > neutron_external_interface: "eth1" > > > > My VM has a private IP and a floating IP that I use to ssh to it: > > > > | ID | Name | Status | Networks | Image | Flavor | > > | 41f6782a-8ca8-4ab4-8fdc-776584139ba0 | adjutant-poc | ACTIVE | private=<primary IP>, <floating IP> | | s1-standard-08 | > > > > I tried various options for eth1 but I can’t seem to get it to work: > > > > [root@adjutant-poc network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 > > BOOTPROTO=dhcp > > DEVICE=eth1 > > ONBOOT=yes > > TYPE=Ethernet > > NM_CONTROLLED=no > > > > I tried creating another port and adding it but that didn’t seem to make a difference. What am I doing wrong? Do I really need 2 network interfaces on the VM? If so, what’s the correct way to add one? > > > > [root@adjutant-poc network-scripts]# service network restart > > Restarting network (via systemctl): Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details. > > [FAILED] > > > > Log: > > > > http://paste.debian.net/1168982/ > > > > I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

1 0

[ironic] SPUC winter season
by Dmitry Tantsur 30 Oct '20

30 Oct '20

Boo! It's SPUC time again! Ironicers, their pet AIs, friendly stackers, as well as significant others and fluffy overladies and overlords, are welcome! For those who don't know, Sanity Preserving Un-Conference is a series of Friday calls for keeping (the remainings) of our sanity in these difficult times. It's goal is to let people virtually socialize and help brighten up lives for each other. Work achievements? A new hobby? A hilarious story? Funny things your kid did at 4am? The first words from your pizza dough that you forgot in the fridge during the previous lockdown? Anything that makes you and others happier is on-topic! Due to an increasing number of APAC contributors, I would like two pick slots: one in APAC evening/Europe morning, the other in Europe morning/US afternoon. Please vote in the doodle: https://doodle.com/poll/tw72utagidz2nac3 Cheers, Dmitry -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

4 6

[openstack][nova] urgent compute in evacuate state
by Ignazio Cassano 30 Oct '20

30 Oct '20

Hello Guys, I have a node in evacuate state....how exit from this state ? Thanks Ignazio

2 6

[kolla] Setting up eth1 on VM for AIO
by Braden, Albert 30 Oct '20

30 Oct '20

I'm trying to setup an allinone kolla-ansible Train POC on a VM, and I'm following the instructions here: https://docs.openstack.org/kolla-ansible/train/user/quickstart.html Under "Prepare initial configuration" where it talks about globals.yml, it says I need 2 network interfaces: network_interface: "eth0" neutron_external_interface: "eth1" My VM has a private IP and a floating IP that I use to ssh to it: | ID | Name | Status | Networks | Image | Flavor | | 41f6782a-8ca8-4ab4-8fdc-776584139ba0 | adjutant-poc | ACTIVE | private=<primary IP>, <floating IP> | | s1-standard-08 | I tried various options for eth1 but I can't seem to get it to work: [root@adjutant-poc network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 BOOTPROTO=dhcp DEVICE=eth1 ONBOOT=yes TYPE=Ethernet NM_CONTROLLED=no I tried creating another port and adding it but that didn't seem to make a difference. What am I doing wrong? Do I really need 2 network interfaces on the VM? If so, what's the correct way to add one? [root@adjutant-poc network-scripts]# service network restart Restarting network (via systemctl): Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details. [FAILED] Log: http://paste.debian.net/1168982/ I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

3 2

[Openstack][cinder] dell unity iscsi faulty devices
by Ignazio Cassano 30 Oct '20

30 Oct '20

Hello Stackers, I am using dell emc iscsi driver on my centos 7 queens openstack. It works and instances work as well but on compute nodes I got a lot a faulty device reported by multipath il comand. I do know why this happens, probably attacching and detaching volumes and live migrating instances do not close something well. I read this can cause serious performances problems on compute nodes. Please, any workaround and/or patch is suggested ? Regards Ignazio

2 18

cannot ping my guest's floating ip
by R Kendal 29 Oct '20

29 Oct '20

I am struggling with the fact that I cannot ping my guest's floating ip. I have one Controller node and one Compute node brought up with devstack. Any help would be appreciated! Kendal Details follow... Controller node... [1 picasso@controller ~]$ openstack server list +--------------------------------------+----------+--------+-------------------------------+----------------+----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+----------+--------+-------------------------------+----------------+----------+ | c5a7fd0a-61a1-4698-9c12-45b44cb20e54 | myserver | ACTIVE | private=10.0.1.12, 10.0.0.196 | picasso-server | m1.small | +--------------------------------------+----------+--------+-------------------------------+----------------+----------+ [1 picasso@controller ~]$ openstack floating ip list +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ | ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ | 055d825f-9398-4a11-bf5e-ff3d1a9fc8be | 10.0.0.196 | 10.0.1.12 | 9d036770-e1ba-4d5f-8f16-780a066416fd | 32631db5-7798-47df-8af6-7ee8fca011e4 | a0e078d4bac34db89105a2eb0ba4d896 | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ [0 picasso@controller ~]$ ping -c 1 -W 1 10.0.1.1 PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data. 64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=0.066 ms --- 10.0.1.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.066/0.066/0.066/0.000 ms [0 picasso@controller ~]$ ping -c 1 -W 1 10.0.1.12 PING 10.0.1.12 (10.0.1.12) 56(84) bytes of data. --- 10.0.1.12 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [1 picasso@controller ~]$ ping -c 1 -W 1 10.0.0.196 PING 10.0.0.196 (10.0.0.196) 56(84) bytes of data. --- 10.0.0.196 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [0 picasso@controller ~]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000 link/ether 08:00:27:72:b2:e5 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3 valid_lft 77664sec preferred_lft 77664sec inet6 fe80::a00:27ff:fe72:b2e5/64 scope link valid_lft forever preferred_lft forever 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000 link/ether 02:00:68:ed:7c:08 brd ff:ff:ff:ff:ff:ff inet 192.168.1.6/24 brd 192.168.1.255 scope global enp0s8 valid_lft forever preferred_lft forever inet6 fe80::68ff:feed:7c08/64 scope link valid_lft forever preferred_lft forever 4: enp0s9: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000 link/ether 02:00:b6:d5:fe:c5 brd ff:ff:ff:ff:ff:ff inet6 fe80::b6ff:fed5:fec5/64 scope link valid_lft forever preferred_lft forever 5: enp0s10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 02:00:72:74:73:d0 brd ff:ff:ff:ff:ff:ff inet 10.0.1.1/24 brd 10.0.1.255 scope global enp0s10 valid_lft forever preferred_lft forever inet6 fe80::72ff:fe74:73d0/64 scope link valid_lft forever preferred_lft forever 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether e6:fd:7a:97:a1:32 brd ff:ff:ff:ff:ff:ff 7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether f6:38:eb:d2:ce:41 brd ff:ff:ff:ff:ff:ff 8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether ba:d2:e4:c1:7f:48 brd ff:ff:ff:ff:ff:ff inet 10.0.0.8/24 brd 10.0.0.255 scope global br-ex valid_lft forever preferred_lft forever inet6 fe80::b8d2:e4ff:fec1:7f48/64 scope link valid_lft forever preferred_lft forever 9: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether ca:5b:f2:bd:65:40 brd ff:ff:ff:ff:ff:ff 13: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000 link/ether ca:7f:ae:3b:5a:14 brd ff:ff:ff:ff:ff:ff inet6 fe80::c87f:aeff:fe3b:5a14/64 scope link valid_lft forever preferred_lft forever [0 picasso@controller ~]$ ip r default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100 10.0.0.0/24 dev br-ex proto kernel scope link src 10.0.0.8 10.0.1.0/24 dev enp0s10 proto kernel scope link src 10.0.1.1 10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15 10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100 192.168.1.0/24 dev enp0s8 proto kernel scope link src 192.168.1.6 [0 picasso@controller ~]$ neutron net-show "public" +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | nova | | created_at | 2020-10-29T18:21:07Z | | description | | | id | 32631db5-7798-47df-8af6-7ee8fca011e4 | | ipv4_address_scope | | | ipv6_address_scope | | | is_default | False | | mtu | 1500 | | name | public | | port_security_enabled | True | | project_id | a0e078d4bac34db89105a2eb0ba4d896 | | provider:network_type | flat | | provider:physical_network | public | | provider:segmentation_id | | | revision_number | 2 | | router:external | True | | shared | True | | status | ACTIVE | | subnets | daab0afa-9adc-4978-9e00-65669b89fec5 | | tags | | | tenant_id | a0e078d4bac34db89105a2eb0ba4d896 | | updated_at | 2020-10-29T18:21:14Z | +---------------------------+--------------------------------------+ [0 picasso@controller ~]$ neutron subnet-show public-subnet +-------------------+----------------------------------------------+ | Field | Value | +-------------------+----------------------------------------------+ | allocation_pools | {"start": "10.0.0.128", "end": "10.0.0.254"} | | cidr | 10.0.0.0/24 | | created_at | 2020-10-29T18:21:14Z | | description | | | dns_nameservers | 10.0.0.1 | | enable_dhcp | False | | gateway_ip | 10.0.0.1 | | host_routes | | | id | daab0afa-9adc-4978-9e00-65669b89fec5 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | public-subnet | | network_id | 32631db5-7798-47df-8af6-7ee8fca011e4 | | project_id | a0e078d4bac34db89105a2eb0ba4d896 | | revision_number | 0 | | service_types | | | subnetpool_id | | | tags | | | tenant_id | a0e078d4bac34db89105a2eb0ba4d896 | | updated_at | 2020-10-29T18:21:14Z | +-------------------+----------------------------------------------+ [1 picasso@controller ~]$ neutron net-show private +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | nova | | created_at | 2020-10-29T18:20:56Z | | description | | | id | b440a536-a69c-42c8-b05d-5f8323f22cc0 | | ipv4_address_scope | | | ipv6_address_scope | | | mtu | 1450 | | name | private | | port_security_enabled | True | | project_id | 682549fcfe3b4645a21b59290e62e453 | | provider:network_type | vxlan | | provider:physical_network | | | provider:segmentation_id | 1 | | revision_number | 2 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | 35a88fb9-e1e6-4195-9c6b-95d96545d4c4 | | tags | | | tenant_id | 682549fcfe3b4645a21b59290e62e453 | | updated_at | 2020-10-29T18:21:02Z | +---------------------------+--------------------------------------+ [0 picasso@controller ~]$ neutron subnet-show private-subnet +-------------------+---------------------------------------------+ | Field | Value | +-------------------+---------------------------------------------+ | allocation_pools | {"start": "10.0.1.10", "end": "10.0.1.254"} | | cidr | 10.0.1.0/24 | | created_at | 2020-10-29T18:21:02Z | | description | | | dns_nameservers | 10.0.0.1 | | enable_dhcp | True | | gateway_ip | 10.0.1.1 | | host_routes | | | id | 35a88fb9-e1e6-4195-9c6b-95d96545d4c4 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | private-subnet | | network_id | b440a536-a69c-42c8-b05d-5f8323f22cc0 | | project_id | 682549fcfe3b4645a21b59290e62e453 | | revision_number | 0 | | service_types | | | subnetpool_id | | | tags | | | tenant_id | 682549fcfe3b4645a21b59290e62e453 | | updated_at | 2020-10-29T18:21:02Z | +-------------------+---------------------------------------------+ [0 picasso@controller ~]$ neutron router-show public +-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | nova | | created_at | 2020-10-29T18:21:20Z | | description | | | distributed | False | | external_gateway_info | {"network_id": "32631db5-7798-47df-8af6-7ee8fca011e4", "external_fixed_ips": [{"subnet_id": "daab0afa-9adc-4978-9e00-65669b89fec5", "ip_address": "10.0.0.169"}], "enable_snat": true} | | flavor_id | | | ha | False | | id | d424d63e-cc93-4d02-afad-5851bc7f96cd | | name | public | | project_id | a0e078d4bac34db89105a2eb0ba4d896 | | revision_number | 4 | | routes | | | status | ACTIVE | | tags | | | tenant_id | a0e078d4bac34db89105a2eb0ba4d896 | | updated_at | 2020-10-29T18:21:28Z | +-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [1 picasso@controller ~]$ sudo ovs-vsctl show f2c58de6-9a33-4222-94f5-da1e4a2cfdbb Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port br-tun Interface br-tun type: internal Port "vxlan-c0a801c8" Interface "vxlan-c0a801c8" type: vxlan options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="192.168.1.6", out_key=flow, remote_ip="192.168.1.200"} Bridge br-ex Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-ex Interface br-ex type: internal Port phy-br-ex Interface phy-br-ex type: patch options: {peer=int-br-ex} Port "enp0s9" Interface "enp0s9" Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port "tap456085c0-61" tag: 1 Interface "tap456085c0-61" type: internal Port "qr-0434594d-ee" tag: 1 Interface "qr-0434594d-ee" type: internal Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port int-br-ex Interface int-br-ex type: patch options: {peer=phy-br-ex} Port br-int Interface br-int type: internal Port "qg-77ed5bcd-25" tag: 2 Interface "qg-77ed5bcd-25" type: internal ovs_version: "2.12.1" router_name=public ROUTER_ID=$(openstack router show $router_name -f value -c id) QROUTER="qrouter-$ROUTER_ID" [0 picasso@controller ~]$ sudo ip netns exec $QROUTER ifconfig lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 45 bytes 4424 (4.4 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 45 bytes 4424 (4.4 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qg-77ed5bcd-25: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.0.0.169 netmask 255.255.255.0 broadcast 10.0.0.255 inet6 fe80::f816:3eff:febc:990c prefixlen 64 scopeid 0x20<link> ether fa:16:3e:bc:99:0c txqueuelen 1000 (Ethernet) RX packets 944 bytes 146787 (146.7 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 164 bytes 16468 (16.4 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qr-0434594d-ee: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 10.0.1.1 netmask 255.255.255.0 broadcast 10.0.1.255 inet6 fe80::f816:3eff:fe5c:9a7d prefixlen 64 scopeid 0x20<link> ether fa:16:3e:5c:9a:7d txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 169 bytes 7550 (7.5 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [0 picasso@controller ~]$ sudo ip netns exec $QROUTER ping -c 1 -W 1 10.0.1.1 PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data. 64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=0.034 ms --- 10.0.1.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.034/0.034/0.034/0.000 ms [0 picasso@controller ~]$ sudo ip netns exec $QROUTER ping -c 1 -W 1 10.0.1.196 PING 10.0.1.196 (10.0.1.196) 56(84) bytes of data. --- 10.0.1.196 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [1 picasso@controller ~]$ sudo ip netns exec $QROUTER ping -c 1 -W 1 10.0.1.169 PING 10.0.1.169 (10.0.1.169) 56(84) bytes of data. --- 10.0.1.169 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms ---------- Compute node... [1 picasso@compute ~]$ ping -c 1 -W 1 10.0.1.1 PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data. 64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=0.425 ms --- 10.0.1.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.425/0.425/0.425/0.000 ms [0 picasso@compute ~]$ ping -c 1 -W 1 10.0.1.12 PING 10.0.1.12 (10.0.1.12) 56(84) bytes of data. --- 10.0.1.12 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [1 picasso@compute ~]$ ping -c 1 -W 1 10.0.0.196 PING 10.0.0.196 (10.0.0.196) 56(84) bytes of data. --- 10.0.0.196 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [0 picasso@compute ~]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000 link/ether 08:00:27:72:b2:e5 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3 valid_lft 79425sec preferred_lft 79425sec inet6 fe80::a00:27ff:fe72:b2e5/64 scope link valid_lft forever preferred_lft forever 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000 link/ether 02:00:73:9d:41:7d brd ff:ff:ff:ff:ff:ff inet 192.168.1.200/24 brd 192.168.1.255 scope global enp0s8 valid_lft forever preferred_lft forever inet6 fe80::73ff:fe9d:417d/64 scope link valid_lft forever preferred_lft forever 4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 02:00:08:c1:84:db brd ff:ff:ff:ff:ff:ff inet 10.0.1.31/24 brd 10.0.1.255 scope global enp0s9 valid_lft forever preferred_lft forever inet6 fe80::8ff:fec1:84db/64 scope link valid_lft forever preferred_lft forever 5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:ee:20:86 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:ee:20:86 brd ff:ff:ff:ff:ff:ff 7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 9e:52:8b:6f:f1:47 brd ff:ff:ff:ff:ff:ff 8: br-int: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000 link/ether 76:e6:90:cc:01:48 brd ff:ff:ff:ff:ff:ff 9: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether aa:09:9f:d9:1d:4f brd ff:ff:ff:ff:ff:ff inet6 fe80::a809:9fff:fed9:1d4f/64 scope link valid_lft forever preferred_lft forever 10: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 26:76:22:af:06:40 brd ff:ff:ff:ff:ff:ff 11: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000 link/ether 8a:87:fb:68:ec:8c brd ff:ff:ff:ff:ff:ff inet6 fe80::8887:fbff:fe68:ec8c/64 scope link valid_lft forever preferred_lft forever 12: tap9d036770-e1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000 link/ether fe:16:3e:f5:f5:c0 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc16:3eff:fef5:f5c0/64 scope link valid_lft forever preferred_lft forever [0 picasso@compute ~]$ ip r default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100 10.0.1.0/24 dev enp0s9 proto kernel scope link src 10.0.1.31 10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15 10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100 192.168.1.0/24 dev enp0s8 proto kernel scope link src 192.168.1.200 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown [130 picasso@compute ~]$ sudo ovs-vsctl show a75bdb8e-9266-48f4-bab5-e119cf07c6af Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-tun Interface br-tun type: internal Port "vxlan-c0a80106" Interface "vxlan-c0a80106" type: vxlan options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="192.168.1.200", out_key=flow, remote_ip="192.168.1.6"} Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-int Interface br-int type: internal Port "tap9d036770-e1" tag: 1 Interface "tap9d036770-e1" Port int-br-ex Interface int-br-ex type: patch options: {peer=phy-br-ex} Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Bridge br-ex Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port phy-br-ex Interface phy-br-ex type: patch options: {peer=int-br-ex} Port br-ex Interface br-ex type: internal ovs_version: "2.12.1"

2 1

Nova missing instance
by Grant Morley 29 Oct '20

29 Oct '20

Hi all, A customer had contacted us stating their instance was not working and was unresponsive. They attempted to reboot it and it failed to reboot and went into an error status. I have had a look at the nova logs and nova seems to think the instance has "disappeared" and comes up with an error stating: "Instance cannot be found" I have attached the full logs in the pastebin link. Has anyone come across this before? https://pastebin.com/LWzvsc8u Regards, -- Grant Morley Cloud Engineer, Civo Ltd Unit H-K, Gateway 1000, Whittle Way Stevenage, Herts, SG1 2FP, UK Visit us at www.civo.com <https://www.civo.com/> Signup for an account now > <https://www.civo.com/signup>

2 1

[tripleo][ci] Docker.io Service Announcement: Pull Rate Limit Enforcement, Changes to Image Retention Policies Begin
by Wesley Hayutin 29 Oct '20

29 Oct '20

Greetings, FYI... docker.io will be unavailable to tripleo jobs on Monday Nov 2 2020. Documentation is available here: https://hackmd.io/YH9xtBmOQbSVqPu7fjgoog Reviews are welcome and appreciated https://review.opendev.org/#/q/topic:new-ci-job+(status:open+OR+status:merg… Thanks! ------ from docker.io ------ Hello: You are receiving this email because of a policy change to Docker products and services you use. On Monday, November 2, 2020 at 9am Pacific Standard Time, Docker will begin enforcing rate limits on container pulls for Anonymous and Free users. Anonymous (unauthenticated) users will be limited to 100 container image pulls every six hours, and Free (authenticated) users will be limited to 200 container image pulls every six hours, when enforcement is fully implemented. Docker Pro and Team subscribers can pull container images from Docker Hub without restriction, as long as the quantities are not excessive or abusive. In addition, we are pausing enforcement of the changes to our image-retention policies until mid-2021, when we anticipate incorporating them into usage-based pricing. Two months ago, we announced an update to Docker image-retention policies. As originally stated, this change, which was set to take effect on November 1, 2020, would result in the deletion of images for free Docker account users after six months of inactivity. Today’s announcement means Docker will not enforce image expiration on November 1, 2020. For more details, please read the blog post <http://email.notify.docker.com/c/eJxNjsFuhSAURL9Gdxi46EMWLNq89j-ucFVSBCMY49…> summarizing the changes or review the documentation for rate limits <http://email.notify.docker.com/c/eJxNjkEOgyAURE8jOwl8kOqCRRvbe3z8WEhRG8WY3r…> . Details about Docker subscription levels and differentiators are available on the Docker Pricing Page <http://email.notify.docker.com/c/eJxNjcsOgyAURL9GlgTu5VEWLNrY_gfBq5KqGKUh_n…> . Thank you for using Docker. Jean-Laurent de Morlhon VP, Engineering Docker

1 0

Windows client can't connect to internet
by liu 29 Oct '20

29 Oct '20

Dear OpenStack community, I just launch a windows client in my OpenStack platform, but it can't connect to the internet with the float IP. Before that use a ubuntu client I can access the internet. Any ideas about why this happened. Thank you for your help. Liu

2 2

[tripleo][ci] Fwd: Service Announcement: Pull Rate Limit Enforcement, Changes to Image Retention Policies Begin November 2, 2020
by Wes Hayutin 29 Oct '20

29 Oct '20

FYI... Reviews are welcome and appreciated https://review.opendev.org/#/q/topic:new-ci-job+(status:open+OR+status:merg… Thanks! ---------- Forwarded message --------- From: Docker <noreply(a)notify.docker.com> Date: Wed, Oct 28, 2020 at 10:08 PM Subject: Service Announcement: Pull Rate Limit Enforcement, Changes to Image Retention Policies Begin November 2, 2020 To: <weshayutin(a)gmail.com> [image: Docker] <http://email.notify.docker.com/c/eJxNjUEOgyAQAF-DR7LsosCBQ43tPxChkKo0imn8fe…> Hello: You are receiving this email because of a policy change to Docker products and services you use. On Monday, November 2, 2020 at 9am Pacific Standard Time, Docker will begin enforcing rate limits on container pulls for Anonymous and Free users. Anonymous (unauthenticated) users will be limited to 100 container image pulls every six hours, and Free (authenticated) users will be limited to 200 container image pulls every six hours, when enforcement is fully implemented. Docker Pro and Team subscribers can pull container images from Docker Hub without restriction, as long as the quantities are not excessive or abusive. In addition, we are pausing enforcement of the changes to our image-retention policies until mid-2021, when we anticipate incorporating them into usage-based pricing. Two months ago, we announced an update to Docker image-retention policies. As originally stated, this change, which was set to take effect on November 1, 2020, would result in the deletion of images for free Docker account users after six months of inactivity. Today’s announcement means Docker will not enforce image expiration on November 1, 2020. For more details, please read the blog post <http://email.notify.docker.com/c/eJxNjsFuhSAURL9Gdxi46EMWLNq89j-ucFVSBCMY49…> summarizing the changes or review the documentation for rate limits <http://email.notify.docker.com/c/eJxNjkEOgyAURE8jOwl8kOqCRRvbe3z8WEhRG8WY3r…>. Details about Docker subscription levels and differentiators are available on the Docker Pricing Page <http://email.notify.docker.com/c/eJxNjcsOgyAURL9GlgTu5VEWLNrY_gfBq5KqGKUh_n…>. Thank you for using Docker. Jean-Laurent de Morlhon VP, Engineering Docker © Docker, Inc., 2020 318 Cambridge Ave Palo Alto, CA 94306 Docker Privacy Policy <http://email.notify.docker.com/c/eJxNjssKgzAURL_G7AzxxiS6yKLF9j_y1ND4QG1D_r…>

1 0

Nova iSCSI oddness
by Grant Morley 29 Oct '20

29 Oct '20

Hi all, We are seeing some oddness on a couple of our compute hosts that seems to be related to iSCSI. On a couple of our hosts I am seeing this error in the nova compute logs: 2020-10-27 18:56:14.814 31490 WARNING os_brick.initiator.connectors.iscsi [req-8613ae69-1661-49cf-8bdc-6fec875d01ba - - - - -] Couldn't find iscsi sessions because iscsiadm err: iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session1707 That seems to also stop any instance on the compute host from being able to reboot. Reboots seem to get accepted but the instance never completes and gets stuck in the reboot state: 2020-10-27 19:11:58.891 48612 INFO nova.compute.manager [-] [instance: c8079e85-4777-4615-9d5a-3d1151e11984] During sync_power_state the instance has a pending task (reboot_started_hard). Skip. 2020-10-27 19:11:58.891 48612 INFO nova.compute.manager [-] [instance: 31128f26-910d-411f-98e0-c95dd36f4f0f] During sync_power_state the instance has a pending task (reboot_started_hard). Skip. Does anyone know of a way to resolve this without rebooting the entire compute host? I can't see any other issues other than the fact there is this iSCSI error which in turn seems to stop nova from processing anything for any instance. Any advice would be much appreciated. Regards, -- Grant Morley Cloud Engineer, Civo Ltd Unit H-K, Gateway 1000, Whittle Way Stevenage, Herts, SG1 2FP, UK Visit us at www.civo.com <https://www.civo.com/> Signup for an account now > <https://www.civo.com/signup>

4 9

[tripleo] TripleO at Wallaby PTG wrapup
by Marios Andreou 29 Oct '20

29 Oct '20

Hello all, thanks to everyone that participated and especially thanks to all who prepared and lead our PTG sessions. The number of participants was variable however the maximum attendees (that I noticed at least) per day were Monday:43, Tuesday:48, Wednesday:39. We had three full days with 18 sessions in total. I've included a condensed listing below [3] for convenience and you can find the full schedule at [1] with session descriptions. All sessions were recorded - hopefully we will get access to those soon (I don't have any info about that yet). If you have problems accessing any of the etherpads reach out to me I made backups. Our "team photo" from Monday is available at [2] :) Thanks again TripleO for a great PTG! regards, marios [1] https://etherpad.opendev.org/p/tripleo-ptg-wallaby [2] https://i.imgur.com/apqiYhk.png [3] DAY 1: retrospective & community topics Etherpad: https://etherpad.opendev.org/p/tripleo-ptg-retrospective Etherpad link: https://etherpad.opendev.org/p/tripleo-ptg-wallaby-community-process (Deployment) Composable Network v2 (sans Heat) Proposer: Emilien, Harald & Rabi Etherpad link: https://etherpad.opendev.org/p/composable-networks-v2 (Edge) Topic: State of DCN Proposer: James Slagle Etherpad link: https://etherpad.opendev.org/p/tripleo-ptg-wallaby-edge (PIDONE) Updates from PIDONE Proposer: dciabrin/bandini Etherpad link: https://etherpad.opendev.org/p/tripleo-ptg-wallaby-pidone (Ceph) Topic: review the cephadm integration specs https://review.opendev.org/#/c/757644/ and https://review.opendev.org/#/c/723108/ and https://review.opendev.org/759395 Etherpad link: https://etherpad.opendev.org/p/tripleo-ceph Presenter: [fmount, gfidente, fultonj] DAY 2: (Upgrades) Lessons learned upgrading from Queens to Train, then looking forward at Train to Wallaby Etherpad link: https://etherpad.opendev.org/p/tripleo-PTG-wallaby-upgrades Presenter: Jesse (odyssey4me), Lukas (lbezdick), Jose (jfrancoa), Sergii (holser) (CI) Topic: Dependency pipeline, CentOS Stream and CentOS 9 / Fedora 33 Slides: https://filedn.com/lHMSkrniM7gFSkf96rR711z/TripleODependencyPipeline.pdf Presenter: (rlandy) (Deployment) Topic: Drop heat-monolith local package requirement for undercloud Proposer: Alex Schultz (mwhahaha) Slides: https://docs.google.com/presentation/d/13ds5h9foCHEy3duElGs6mZ3mQwKcaggX6Vj… Blueprint: https://blueprints.launchpad.net/tripleo/+spec/switch-to-heat-all-container (Security) Topic: Updates for Security Etherpad Link: https://etherpad.opendev.org/p/tripleo-PTG-wallaby-security Presenter: Ade (alee) (Upgrades) Topic: Update should only be role inclusion, module and action plugins but we can't do the refactor without any upstream development framework or.... some broader coverage CI. Etherpad link: https://etherpad.opendev.org/p/tripleo-ptg-wallaby-refactor-update (CI) Topic: 0/ so long docker.io Etherpad link: https://hackmd.io/YH9xtBmOQbSVqPu7fjgoog?both Presenter: sshnaidm, chkumar, weshay, marios, DAY 3: (Deployment) Topic: Move to Ansible 2.10.x Proposer: Kevin Carter Etherpad link: https://etherpad.opendev.org/p/tripleo-ansible-210 (Upgrades) Topics: Run all roles in parallel during update by default for massive speed improvment. Etherpad link: https://etherpad.opendev.org/p/tripleo-ptg-wallaby-all-roles-in-parallel (Deployment) Topic: config-download Ansible Collection Etherpad link: https://etherpad.opendev.org/p/ptg-tripleo-config-download-collection Proposer: Luke Short (ekultails) (Deployment) Topic: Removal of Swift from the Undercloud Proposer: Kevin Carter (cloudnull) Etherpad link: https://etherpad.opendev.org/p/tripleo-swift-removal-undercloud (Ceph) Integration Discussion - Continued Etherpad link: https://etherpad.opendev.org/p/tripleo-ceph Presenter: [fmount, gfidente, fultonj] (Networking) Topic: BGP Integration https://review.opendev.org/#/c/758249/ "Spec for installing and configuring FRRouter" Proposer: Dan Sneddon (Cloudops) Topic: Moving deployment logic into ansible roles Proposer: efoley, all (discussion)

1 0

[operators][telemetry] Quick Telemetry survey
by Matthias Runge 29 Oct '20

29 Oct '20

Hi there, you may have seen, not much has been happening lately in OpenStack Telemetry. However, there have been interesting talks at the recent OpenInfra Summit. I would like to gather some data on current Telemetry usage in order to be able to focus a bit better[1]. There is not much value in keeping projects alive, if nobody uses them. At the same time, we may want to increase focus/contribution on often used projects. In order to help with the data, please fill in the form[1] and also feel free to forward this form to interested parties. Thank you, Matthias [1] https://forms.gle/EmncAxggRcG8r31b9

3 3

[devstack][cinder failure]:internal server error(HTTP 500)
by James 29 Oct '20

29 Oct '20

Hi:    I am new to openstack installing on one VM. The environment includes Win10,virtualbox,Ubuntu 18.04; The version of  devstack and openstack  is  Train;  the VM  includes 2vCPU,4G ram, 60G disk .    The process was interrupted by cinder error.[internal server error]        The stack.log shows as stack.log; the c-api log shows as c-api log. stack.log +[38;5;242mlib/cinder:create_volume_types:531 (B[m openstack --os-region-name=RegionOne volume type create --property volume_backend_name=lvmdriver-1 lvmdriver-1 2020-10-29 07:57:11.863 | Internal Server Error (HTTP 500) 2020-10-29 07:57:12.182 | +[38;5;242mlib/cinder:create_volume_types:1 (B[m exit_trap 2020-10-29 07:57:12.232 | +[38;5;242m./stack.sh:exit_trap:536 (B[m local r=1 2020-10-29 07:57:12.259 | ++[38;5;242m./stack.sh:exit_trap:537 (B[m jobs -p 2020-10-29 07:57:12.316 | +[38;5;242m./stack.sh:exit_trap:537 (B[m jobs= 2020-10-29 07:57:12.353 | +[38;5;242m./stack.sh:exit_trap:540 (B[m [[ -n '' ]] 2020-10-29 07:57:12.385 | +[38;5;242m./stack.sh:exit_trap:546 (B[m '[' -f /tmp/tmp.8Y7a6hpwLX ']' 2020-10-29 07:57:12.408 | +[38;5;242m./stack.sh:exit_trap:547 (B[m rm /tmp/tmp.8Y7a6hpwLX 2020-10-29 07:57:12.468 | +[38;5;242m./stack.sh:exit_trap:551 (B[m kill_spinner 2020-10-29 07:57:12.499 | +[38;5;242m./stack.sh:kill_spinner:446 (B[m '[' '!' -z '' ']' 2020-10-29 07:57:12.536 | +[38;5;242m./stack.sh:exit_trap:553 (B[m [[ 1 -ne 0 ]] 2020-10-29 07:57:12.581 | +[38;5;242m./stack.sh:exit_trap:554 (B[m echo 'Error on exit' 2020-10-29 07:57:12.581 | Error on exit 2020-10-29 07:57:12.615 | +[38;5;242m./stack.sh:exit_trap:556 (B[m type -p generate-subunit 2020-10-29 07:57:12.635 | +[38;5;242m./stack.sh:exit_trap:557 (B[m generate-subunit 1603957090 1142 fail 2020-10-29 07:57:14.084 | +[38;5;242m./stack.sh:exit_trap:559 (B[m [[ -z /opt/stack/logs ]] 2020-10-29 07:57:14.110 | +[38;5;242m./stack.sh:exit_trap:562 (B[m /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs 2020-10-29 07:57:17.223 | nova-compute: no process found 2020-10-29 07:57:17.264 | neutron-dhcp-agent: no process found 2020-10-29 07:57:17.338 | neutron-l3-agent: no process found 2020-10-29 07:57:17.409 | neutron-linuxbridge-agent: no process found 2020-10-29 07:57:17.471 | neutron-metadata-agent: no process found 2020-10-29 07:57:17.603 | +[38;5;242m./stack.sh:exit_trap:571 C-api logSTRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=12249) _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:307}} 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: ERROR cinder.api.middleware.fault [req-7d67a008-3478-4da0-8b29-259571c1fb94 req-d823c701-e76a-489b-8ec9-fad9f9504c7b admin admin] Caught error: <class 'oslo_db.exception.DBConnectionError'> (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: (Background on this error at: http://sqlalche.me/e/e3q8) DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: (Background on this error at: http://sqlalche.me/e/e3q8) 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: ERROR cinder.api.middleware.fault Traceback (most recent call last): 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: ERROR cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 85, in __call__ 10月 29 10:56:07 controller devstack(a)c-api.service[12246]: ERROR cinder.api.middleware.fault return req.get_response(self.application)Thanks for your help,firtly.   发自我的iPhone

1 0

[infra][ci][all] Devstack error
by Slawek Kaplonski 29 Oct '20

29 Oct '20

Hi, Recently I see pretty often devstack based jobs failing with error "ERROR] / opt/stack/devstack/functions-common:230 Failure retrieving default route device". I reported it at [1] as Devstack bug but maybe it's something related to some changes in Ubuntu used on the gate. Did You saw something similar also? Maybe You already know why it's like that and have any ideas about how to fix it? Thx in advance for any info. [1] https://bugs.launchpad.net/devstack/+bug/1902002 -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0