From moreira.belmiro.email.lists at gmail.com Wed Sep 1 00:03:00 2021 From: moreira.belmiro.email.lists at gmail.com (Belmiro Moreira) Date: Wed, 1 Sep 2021 02:03:00 +0200 Subject: Subject: [all][elections][ptl] PTL Voting Kickoff - Cyborg In-Reply-To: References: Message-ID: (resending to fix the format) Polls for PTL elections are now open and will remain open for you to cast your vote until Sep 07, 2021 23:45 UTC. We are having PTL elections for Cyborg. Please rank all candidates in your order of preference. You are eligible to vote in a PTL election if you are a Foundation individual member[0] and had a commit in one of that team's deliverable repositories[1] over the Sep 25, 2020 00:00 UTC - Aug 24, 2021 00:00 UTC timeframe (Wallaby to Xena) or if you are in that team's list of extra-atcs[2]. If you are eligible to vote in an election, you should find your email with a link to the Condorcet page to cast your vote in the inbox of your Gerrit preferred email[3]. What to do if you don't see the email and have a commit in at least one of the projects having an election: * check the trash or spam folders of your gerrit Preferred Email address, in case it went into trash or spam * wait a bit and check again, in case your email server is a bit slow * find the sha of at least one commit from the project's deliverable repos[0] and email the election officials[4]. If we can confirm that you are entitled to vote, we will add you to the voters list for the appropriate election. Our democratic process is important to the health of OpenStack, please exercise your right to vote! Candidate statements/platforms can be found linked to Candidate names on this page: https://governance.openstack.org/election/ Happy voting, Belmiro, On behalf of all the Election Officials [0] https://www.openstack.org/community/members/ [1] The list of the repositories eligible for electoral status: https://opendev.org/openstack/governance/src/tag/0.10.0/reference/projects.yaml [2] Look for the extra-atcs element in [1] [3] Sign into review.openstack.org: Go to Settings > Contact Information. Look at the email listed as your preferred email. That is where the ballot has been sent. [4] https://governance.openstack.org/election/#election-officials On Wed, Sep 1, 2021 at 1:50 AM Belmiro Moreira < moreira.belmiro.email.lists at gmail.com> wrote: > Polls for PTL elections are now open and will remain open for > you to cast your vote until Sep 07, 2021 23:45 UTC. > We are having PTL elections for Cyborg. Please rank all candidates in your > order of preference. > You are eligible to vote in a PTL election if you are a > Foundationindividual member[0] and had a commit in one of that > team'sdeliverable repositories[1] over the Sep 25, 2020 00:00 UTC - Aug 24, > 2021 00:00 UTC timeframe (Wallaby to Xena) or if you are in that team's > list of extra-atcs[2]. > If you are eligible to vote in an election, you should find youremail with > a link to the Condorcet page to cast your vote in theinbox of your Gerrit > preferred email[3]. > What to do if you don't see the email and have a commit in atleast one of > the projects having an election:* check the trash or spam folders of your > gerrit Preferred Email address, in case it went into trash or spam* wait > a bit and check again, in case your email server is a bit slow* find the > sha of at least one commit from the project's deliverable repos[0] and > email the election officials[4]. > If we can confirm that you are entitled to vote, we will add youto the > voters list for the appropriate election. > Our democratic process is important to the health of OpenStack,please > exercise your right to vote! > Candidate statements/platforms can be found linked to Candidatenames on > this page:https://governance.openstack.org/election/ > Happy voting,Belmiro,On behalf of all the Election Officials > > [0] https://www.openstack.org/community/members/ > [1] The list of the repositories eligible for electoral status: > https://opendev.org/openstack/governance/src/tag/0.10.0/reference/projects.yaml > [2] Look for the extra-atcs element in [1] > [3] Sign into review.openstack.org: Go to Settings > Contact > Information. Look at the email listed as your preferred email. That is > where the ballot has been sent. > [4] https://governance.openstack.org/election/#election-officials > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhangbailin at inspur.com Wed Sep 1 01:55:18 2021 From: zhangbailin at inspur.com (=?gb2312?B?QnJpbiBaaGFuZyjVxbDZwdYp?=) Date: Wed, 1 Sep 2021 01:55:18 +0000 Subject: =?gb2312?B?tPC4tDogW25vdmFdW2N5Ym9yZ11bZ3B1XSBHUFUgcGVyZm9ybWFuY2UgdGVz?= =?gb2312?Q?ting?= In-Reply-To: References: Message-ID: <4e79976a887b41f4a04ecf788eb5e87a@inspur.com> > one with baremetal GPU, one with passthrough, one with virtual GPUs on Nova directly, one with Cyborg Agree with Sylvain Bauza, if you want to test the GPU performance you can use benchmarking, but performance may be related to the running business. Whether it is through nova or cyborg, the GPU is bound, and there should be little difference in performance. brinzhang Inspur Electronic Information Industry Co.,Ltd. 发件人: Sylvain Bauza [mailto:sbauza at redhat.com] 发送时间: 2021年8月31日 23:21 收件人: Ildiko Vancsa 抄送: OpenStack Discuss 主题: Re: [nova][cyborg][gpu] GPU performance testing On Tue, Aug 31, 2021 at 5:10 PM Sylvain Bauza > wrote: On Tue, Aug 31, 2021 at 4:34 PM Ildiko Vancsa > wrote: Hi, As we are approaching the end of the holiday season I wanted to surface back my question about GPU performance testing. Does anyone have any hints to find the best tools to do some benchmarking with? You made the point, Ildiko, I was on a long-running time-off so I didn't had time to look at your question yet. Good concern tho, I have no knowledge about this, but I can ping a few other folks to get you an answer. Btw, I can understand this can be a frustrating by short answer, so I'll develop. Technically, benchmarking is a huge word : depending on your usecase, performance can very differ for the same card (take the general example of CPU-bound vs. IO-bound tasks and you get the idea for GPUs) For this reason, I'd recommend you to first consider the metrics you'd like to stress on and only then identify the tools than can sustain your needs. For a standard test which can be errorprone but still a bit interesting, I'd propose you to run a couple of tensorflow examples against different environments (one with baremetal GPU, one with passthrough, one with virtual GPUs on Nova directly, one with Cyborg). This would give you the idea of the performance penalities but I suspect those to be less than minor. For real benchmarking cases, I can't answer, hence my call to other folks. By the way, I know CERN invested a bit into HPC testing with GPUs, maybe someone from their team or someone from the related Scientific WG could provide more insights ? -Sylvain -Sylvain Thanks, Ildikó > On Aug 9, 2021, at 08:46, Ildiko Vancsa > wrote: > > Hi, > > I got a question about tools and practices to check GPU performance in an OpenStack environment that I need some help to answer. > > The question is about recommended GPU performance testing/benchmarking tools if there are a few that people in the community are using and would recommend? The scope of the testing work is to check GPU performance in OpenStack VMs (both virtualized and passthrough). > > All the help and pointers are very much appreciated! > > Thanks, > Ildikó > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sz_cuitao at 163.com Wed Sep 1 03:16:47 2021 From: sz_cuitao at 163.com (Tommy Sway) Date: Wed, 1 Sep 2021 11:16:47 +0800 Subject: How to stop the whole system gracefully ? In-Reply-To: References: <00af01d79d61$1adb57d0$50920770$@163.com> <010b01d79e05$8719a970$954cfc50$@163.com> Message-ID: <004701d79edf$cce9f7e0$66bde7a0$@163.com> 1. Do I need to back up the current Mariadb before performing mariadB_recovery? Otherwise what backup is this command based on to restore the database? 2. After the stop command is run, the container stops, but is the configuration information deleted? Does re-running deploy-containers simply pull up the container? Will the configuration information also be overwritten? -----Original Message----- From: Sean Mooney Sent: Tuesday, August 31, 2021 6:08 PM To: Tommy Sway ; 'openstack-discuss' Subject: Re: How to stop the whole system gracefully ? On Tue, 2021-08-31 at 09:14 +0800, Tommy Sway wrote: > Anyone can help me ? > > > i belive what you are lookign for is "kolla-ansibel stop" then after the issue is resolved you can do kolla-ansible mariadb_recovery then kolla-ansible deploy-containers https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#kolla-ansible-cli > > > > > > > From: openstack-discuss-bounces+sz_cuitao=163.com at lists.openstack.org > On > Behalf Of Tommy Sway > Sent: Monday, August 30, 2021 1:37 PM > To: 'openstack-discuss' > Subject: How to stop the whole system gracefully ? > > > > My system is installed using kola-ansible. > > The database was inconsistent due to the last unexpected shutdown, and > the system could not start normally, which was very troublesome. > > I would like to ask what command should be used to shut down the > entire system gracefully in an emergency ? > > > > Thanks. > > > > > From rosmaita.fossdev at gmail.com Wed Sep 1 05:21:56 2021 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Wed, 1 Sep 2021 01:21:56 -0400 Subject: [cinder] final Xena cinderclient reviews Message-ID: Consistent with our goal to have the Xena cinderclient support Block Storage API v. 3.66, please focus on these patches: https://review.opendev.org/c/openstack/cinder/+/789564 https://review.opendev.org/c/openstack/python-cinderclient/+/806817/ cheers, brian From skaplons at redhat.com Wed Sep 1 06:46:50 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Wed, 01 Sep 2021 08:46:50 +0200 Subject: [release][heat][ironic][requirements][swift][OpenStackSDK][vitrage][barbican][heat][horizon][ironic][neutron][QA] Cycle With Intermediary Unreleased Deliverables In-Reply-To: References: Message-ID: <11196443.jP3EJilxpz@p1> Hi, On poniedziałek, 30 sierpnia 2021 16:23:36 CEST Herve Beraud wrote: > Hello teams > > Quick reminder that we'll need a release very soon for a number of > deliverables following a cycle-with-intermediary release model but which > have not done *any* release yet in the Xena cycle: > > ansible-role-atos-hsm > ansible-role-lunasa-hsm > ansible-role-thales-hsm > heat-agents > horizon > ironic-prometheus-exporter > ironic-python-agent-builder > ironic-ui > ovn-octavia-provider > patrole > python-openstackclient > requirements > swift > tempest > vitrage-dashboard > vitrage > > > Those should be released ASAP, and in all cases before the week of 13 > September, 2021, so > that we have a release to include in the final Xena release. > > Thanks for your attention. I already proposed new release of the ovn-octavia-provider (1.1.1) [1] and it's done. But TBH I'm a bit confused as we had already version 1.1.0 released in Xena cycle some time ago - see [2]. So I'm not really sure why this project is on that list now :) [1] https://review.opendev.org/c/openstack/releases/+/806716 [2] https://review.opendev.org/c/openstack/releases/+/803321 -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From hberaud at redhat.com Wed Sep 1 07:49:33 2021 From: hberaud at redhat.com (Herve Beraud) Date: Wed, 1 Sep 2021 09:49:33 +0200 Subject: [release][heat][ironic][requirements][swift][OpenStackSDK][vitrage][barbican][heat][horizon][ironic][neutron][QA] Cycle With Intermediary Unreleased Deliverables In-Reply-To: <11196443.jP3EJilxpz@p1> References: <11196443.jP3EJilxpz@p1> Message-ID: Le mer. 1 sept. 2021 à 08:47, Slawek Kaplonski a écrit : > Hi, > > On poniedziałek, 30 sierpnia 2021 16:23:36 CEST Herve Beraud wrote: > > Hello teams > > > > Quick reminder that we'll need a release very soon for a number of > > deliverables following a cycle-with-intermediary release model but which > > have not done *any* release yet in the Xena cycle: > > > > ansible-role-atos-hsm > > ansible-role-lunasa-hsm > > ansible-role-thales-hsm > > heat-agents > > horizon > > ironic-prometheus-exporter > > ironic-python-agent-builder > > ironic-ui > > ovn-octavia-provider > > patrole > > python-openstackclient > > requirements > > swift > > tempest > > vitrage-dashboard > > vitrage > > > > > > Those should be released ASAP, and in all cases before the week of 13 > > September, 2021, so > > that we have a release to include in the final Xena release. > > > > Thanks for your attention. > > I already proposed new release of the ovn-octavia-provider (1.1.1) [1] and > it's done. > But TBH I'm a bit confused as we had already version 1.1.0 released in > Xena > cycle some time ago - see [2]. So I'm not really sure why this project is > on > that list now :) > Indeed you are right our tooling used to generate this list missed this point. I'll try to see why it didn't ignored ovn-octavia-provider. Thanks for the heads up. > [1] https://review.opendev.org/c/openstack/releases/+/806716 > [2] https://review.opendev.org/c/openstack/releases/+/803321 > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Wed Sep 1 07:58:34 2021 From: hberaud at redhat.com (Herve Beraud) Date: Wed, 1 Sep 2021 09:58:34 +0200 Subject: [release][heat][ironic][requirements][OpenStackSDK][barbican][heat][ironic][neutron][QA] Cycle With Intermediary Unreleased Deliverables In-Reply-To: References: <11196443.jP3EJilxpz@p1> Message-ID: Well, the projects below could also ignore this thread, they are in the same case that the ovn-octavia-provider. For more details see the previous messages of this discussion. - horizon - swift - tempest - vitrage-dashboard - vitrage I remove these teams from the subject of this thread. Thanks for your attention. Le mer. 1 sept. 2021 à 09:49, Herve Beraud a écrit : > > > Le mer. 1 sept. 2021 à 08:47, Slawek Kaplonski a > écrit : > >> Hi, >> >> On poniedziałek, 30 sierpnia 2021 16:23:36 CEST Herve Beraud wrote: >> > Hello teams >> > >> > Quick reminder that we'll need a release very soon for a number of >> > deliverables following a cycle-with-intermediary release model but which >> > have not done *any* release yet in the Xena cycle: >> > >> > ansible-role-atos-hsm >> > ansible-role-lunasa-hsm >> > ansible-role-thales-hsm >> > heat-agents >> > horizon >> > ironic-prometheus-exporter >> > ironic-python-agent-builder >> > ironic-ui >> > ovn-octavia-provider >> > patrole >> > python-openstackclient >> > requirements >> > swift >> > tempest >> > vitrage-dashboard >> > vitrage >> > >> > >> > Those should be released ASAP, and in all cases before the week of 13 >> > September, 2021, so >> > that we have a release to include in the final Xena release. >> > >> > Thanks for your attention. >> >> I already proposed new release of the ovn-octavia-provider (1.1.1) [1] >> and >> it's done. >> But TBH I'm a bit confused as we had already version 1.1.0 released in >> Xena >> cycle some time ago - see [2]. So I'm not really sure why this project is >> on >> that list now :) >> > > Indeed you are right our tooling used to generate this list missed this > point. I'll try to see why it didn't ignored ovn-octavia-provider. > Thanks for the heads up. > > >> [1] https://review.opendev.org/c/openstack/releases/+/806716 >> [2] https://review.opendev.org/c/openstack/releases/+/803321 >> >> -- >> Slawek Kaplonski >> Principal Software Engineer >> Red Hat > > > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ > https://twitter.com/4383hberaud > > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From hjensas at redhat.com Wed Sep 1 08:13:49 2021 From: hjensas at redhat.com (Harald Jensas) Date: Wed, 1 Sep 2021 10:13:49 +0200 Subject: Need some help to understand "Baremetal Provision Configuration" file In-Reply-To: References: <41c7ae92-eee6-87c6-c232-6410579fe901@redhat.com> Message-ID: <50502e08-8635-7ef1-6c21-2a563c89a283@redhat.com> On 8/28/21 12:40 AM, wodel youchi wrote: > Hi again, > I redid my deployment from scratch, I reinstalled my undercloud and > prepared the network template. > Then I started the baremetal provisioning again, and I get this error: > > Deploy attempt failed on node computeHCI0 (UUID > 1d536020-50d6-43b6-9be4-1e5f60fa27d4), cleaning up\nTraceback (most > recent call last):\n  File \"/usr/lib/python3.6/site-pack > ages/metalsmith/_provisioner.py\", line 392, in provision_node\n >  nics.validate()\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_nics.py\", line 60, > in validate\n    result.append > (('network', self._get_network(nic)))\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_nics.py\", line 128, > in _get_network\n    '*Unexpected fields for a network: %s' % '*, > '.join(unex > pected))\nmetalsmith.exceptions.InvalidNIC: *Unexpected fields for a > network: subnet*\nDeploy attempt failed on node computeHCI2 (UUID > 0932aeac-e72c-4047-8c6b-c8fff674b0f3), cleaning up\nTrac > eback (most recent call last):\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_provisioner.py\", > line 392, in provision_node\n    nics.validate()\n  File > \"/usr/lib/python3.6/site-pa > ckages/metalsmith/_nics.py\", line 60, in validate\n >  result.append(('network', self._get_network(nic)))\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_nics.py\", line 128, > in _ge > t_network\n    'Unexpected fields for a network: %s' % ', > '.join(unexpected))\nmetalsmith.exceptions.InvalidNIC: Unexpected > fields for a network: subnet\nDeploy attempt failed on node contr > oller1 (UUID 8e4eb00b-6c34-4dcd-a9a5-aef3d9ec6c0a), cleaning > up\nTraceback (most recent call last):\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_provisioner.py\", > line 392, in pro > vision_node\n    nics.validate()\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_nics.py\", line 60, > in validate\n    result.append(('network', > self._get_network(nic)))\n  File \"/us > r/lib/python3.6/site-packages/metalsmith/_nics.py\", line 128, in > _get_network\n    'Unexpected fields for a network: %s' % ', > '.join(unexpected))\nmetalsmith.exceptions.InvalidNIC: Unexpec > ted fields for a network: subnet\nDeploy attempt failed on node > controller0 (UUID 8f794261-53c1-4516-aa51-11bee6d887e8), cleaning > up\nTraceback (most recent call last):\n  File \"/usr/lib/p > ython3.6/site-packages/metalsmith/_provisioner.py\", line 392, in > provision_node\n    nics.validate()\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_nics.py\", line 60, > in validate\ > n    result.append(('network', self._get_network(nic)))\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_nics.py\", line 128, > in _get_network\n    'Unexpected fields for a network: %s > ' % ', '.join(unexpected))\nmetalsmith.exceptions.InvalidNIC: > Unexpected fields for a network: subnet\nDeploy attempt failed on > node controller2 (UUID 32e591a3-d598-43aa-ac05-b646eecef073), >  cleaning up\nTraceback (most recent call last):\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_provisioner.py\", > line 392, in provision_node\n    nics.validate()\n  File \"/usr/lib > /python3.6/site-packages/metalsmith/_nics.py\", line 60, in > validate\n    result.append(('network', self._get_network(nic)))\n >  File \"/usr/lib/python3.6/site-packages/metalsmith/_nics.py\" > , line 128, in _get_network\n    'Unexpected fields for a network: > %s' % ', '.join(unexpected))\nmetalsmith.exceptions.InvalidNIC: > Unexpected fields for a network: subnet\nDeploy attempt fa > iled on node computeHCI1 (UUID > bf283f08-bef1-4f63-8bcd-d3c518c22e36), cleaning up\nTraceback (most > recent call last):\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_provisioner.py\" > , line 392, in provision_node\n    nics.validate()\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_nics.py\", line 60, > in validate\n    result.append(('network', self._get_network(ni > c)))\n  File > \"/usr/lib/python3.6/site-packages/metalsmith/_nics.py\", line 128, > in _get_network\n    'Unexpected fields for a network: %s' % ', > '.join(unexpected))\nmetalsmith.exceptions.I > nvalidNIC: Unexpected fields for a network: subnet\n", >     "msg": "Unexpected fields for a network: subnet" > > > Here is my modified bonds_vlan.j2 file > > --- > {% set mtu_ctlplane_list = [ctlplane_mtu] %} > {% set mtu_dataplane_list = [] %} > {% for network in role_networks %} > {% if network.startswith('Tenant') %} > {{ mtu_dataplane_list.append(lookup('vars', networks_lower[network] > ~ '_mtu')) }} > {% else %} > {{ mtu_ctlplane_list.append(lookup('vars', networks_lower[network] ~ > '_mtu')) }} > {%- endif %} > {%- endfor %} > {% set min_viable_mtu_ctlplane = mtu_ctlplane_list | max %} > {% set min_viable_mtu_dataplane = mtu_dataplane_list | max %} > network_config: > - type: interface >  name: nic1 >  mtu: {{ ctlplane_mtu }} >  use_dhcp: false >  addresses: >  - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_subnet_cidr }} >  routes: {{ ctlplane_host_routes }} > - type: ovs_bridge >  name: br1str >  dns_servers: {{ ctlplane_dns_nameservers }} >  domain: {{ dns_search_domains }} >  members: >  - type: ovs_bond >    name: bond-str >    mtu: {{ min_viable_mtu_dataplane }} >    bonding_options: {{ bond_interface_ovs_options }} >    members: >    - type: interface >      name: nic2 >      mtu: {{ min_viable_mtu_dataplane }} >      primary: true >    - type: interface >      name: nic3 >      mtu: {{ min_viable_mtu_dataplane }} > {% for network in role_networks if network*not in ["External", > "Tenant", "InternalApi"] %} * >  - type: vlan >    device: bond-str >    mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }} >    vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }} >    addresses: >    - ip_netmask: {{ lookup('vars', networks_lower[network] ~ '_ip') > }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }} >    routes: {{ lookup('vars', networks_lower[network] ~ > '_host_routes') }} > {% endfor %} > - type: ovs_bridge >  name: {{ neutron_physical_bridge_name }} >  dns_servers: {{ ctlplane_dns_nameservers }} >  members: >  - type: ovs_bond >    name: bond-data >    mtu: {{ min_viable_mtu_dataplane }} >    bonding_options: {{ bond_interface_ovs_options }} >    members: >    - type: interface >      name: nic4 >      mtu: {{ min_viable_mtu_dataplane }} >      primary: true >    - type: interface >      name: nic5 >      mtu: {{ min_viable_mtu_dataplane }} > {% for network in role_networks if network *in ["External", > "Tenant", "InternalApi"]  %}* >  - type: vlan >    device: bond-data >    mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }} >    vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }} >    addresses: >    - ip_netmask: {{ lookup('vars', networks_lower[network] ~ '_ip') > }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }} >    routes: {{ lookup('vars', networks_lower[network] ~ > '_host_routes') }} > {%- endfor %} > > > > Could someone help me understand where have I made mistakes?  In my > setup I use 5 nics : > - 1 nic for provisioning > - nic 2 and 3 : bonded for storage and storage management > - nic 4 and 5 : bonded for tenant, api and external > > Here is my baremetal file > > - name: Controller >  count: 3 >  defaults: >    networks: >    - network: ctlplane >      subnet: ctlplane_subnet Remove the subnet line from the entry with 'vif: true'. >      vif: true >    - network: external >      subnet: external_subnet >    - network: internal_api >      subnet: internal_api_subnet >    - network: storage >      subnet: storage_subnet >    - network: storage_mgmt >      subnet: storage_mgmt_subnet >    - network: tenant >      subnet: tenant_subnet >    network_config: >      template: /home/stack/templates/nic-configs/bonds_vlans.j2 >      bond_interface_ovs_options: bond_mode=active-backup with ^^ in place you should no longer see the error: "AnsibleUndefinedVariable: 'bond_interface_ovs_options' >      default_route_network: >      - external > - name: ComputeHCI >  count: 3 >  defaults: >    networks: >    - network: ctlplane >      subnet: ctlplane_subnet Remove the subnet line from the entry with 'vif: true'. >      vif: true >    - network: external >      subnet: external_subnet >    - network: internal_api >      subnet: internal_api_subnet >    - network: storage >      subnet: storage_subnet >    - network: storage_mgmt >      subnet: storage_mgmt_subnet >    - network: tenant >      subnet: tenant_subnet >    network_config: >      template: /home/stack/templates/nic-configs/bonds_vlans.j2 >      bond_interface_ovs_options: bond_mode=active-backup >      default_route_network: >      - external > > > And here is my network_data file > > - name: Storage >  name_lower: storage >  vip: true >  mtu: 1500 >  subnets: >    storage_subnet: >      ip_subnet: 10.100.8.0/24 >      allocation_pools: >        - start: 10.100.8.150 >          end: 10.100.8.250 >      vlan: 1108 > - name: StorageMgmt >  name_lower: storage_mgmt >  vip: true >  mtu: 1500 >  subnets: >    storage_mgmt_subnet: >      ip_subnet: 10.100.7.0/24 >      allocation_pools: >        - start: 10.100.7.150 >          end: 10.100.7.250 >      vlan: 1107 > - name: InternalApi >  name_lower: internal_api >  vip: true >  mtu: 1500 >  subnets: >    internal_api_subnet: >      ip_subnet: 10.100.5.0/24 >      allocation_pools: >        - start: 10.100.5.150 >          end: 10.100.5.250 >      vlan: 1105 > - name: Tenant >  vip: false  # Tenant network does not use VIPs >  mtu: 1500 >  name_lower: tenant >  subnets: >    tenant_subnet: >      ip_subnet: 10.100.6.0/24 >      allocation_pools: >        - start: 10.100.6.150 >          end: 10.100.6.250 >      vlan: 1106 > - name: External >  name_lower: external >  vip: true >  mtu: 1500 >  subnets: >    external_subnet: >      ip_subnet: 10.1.0.0/24 >      allocation_pools: >        - start: 10.1.0.10 >          end: 10.1.0.200 >      gateway_ip: 10.1.0.1 >      vlan: 2100 > > Thanks in advance > > Regards. > > Le mer. 25 août 2021 à 13:38, wodel youchi > a écrit : > > Hi and thanks > > I disabled the hide variable, and executed the command again, and > this is the result : > > 2021-08-25 13:27:21.834140 | > 52540075-9baf-e9c1-3396-0000000000a0 |      FATAL | Render > network_config from template | computehci-1 | error={ >     "changed": false, > *"msg": "AnsibleUndefinedVariable: 'bond_interface_ovs_options' > is undefined" * > } > > > I understand that this is a variable, but I don't see where it is > being loaded from. > I have a *network-environment-overrides.yaml* file which contains this : > > parameter_defaults: >   ControllerNetworkConfigTemplate: > '/home/stack/templates/nic-configs/bonds_vlans.j2' >   CephStorageNetworkConfigTemplate: > '/home/stack/templates/nic-configs/bonds_vlans.j2' >   ComputeNetworkConfigTemplate: > '/home/stack/templates/nic-configs/bonds_vlans.j2' > >   NeutronNetworkVLANRanges: 'datacentre:1:100,provider:400:1000' >   NeutronExternalNetworkBridge: "''" >   NeutronNetworkType: 'vlan,flat' > *BondInterfaceOvsOptions: "bond_mode=active-backup"* > > > But I don't see how to connect them? > > > Regards. > > Le mer. 25 août 2021 à 09:08, Harald Jensas > a écrit : > > On 8/24/21 7:07 PM, wodel youchi wrote: > >     *2021-08-24 18:01:18.371725 | > 52540075-9baf-8232-d4fa-0000000000a0 | > >           FATAL | Render network_config from template | > computehci-1 | > >     error={ > >          "censored": "the output has been hidden due to the > fact that > >     'no_log: true' was specified for this result", > >          "changed": false > >     } > > This looks like a problem with your nic config template, > /home/stack/templates/nic-configs/bonds_vlans.j2. To debug disable > "hideing" of sensitive logs. > > sudo sed -i 's/tripleo_network_config_hide_sensitive_logs: > true/tripleo_network_config_hide_sensitive_logs: false/g' > /usr/share/ansible/roles/tripleo-network-config/defaults/main.yml > > > -- > Harald > > From manchandavishal143 at gmail.com Wed Sep 1 08:19:41 2021 From: manchandavishal143 at gmail.com (vishal manchanda) Date: Wed, 1 Sep 2021 13:49:41 +0530 Subject: [release][heat][ironic][requirements][OpenStackSDK][barbican][heat][ironic][neutron][QA] Cycle With Intermediary Unreleased Deliverables In-Reply-To: References: <11196443.jP3EJilxpz@p1> Message-ID: Hi Herve, Same for the horizon as well like horizon already had version 20.0.0(first release for Xena) [1]. Do you still want me to cut a new release? [1] https://review.opendev.org/c/openstack/releases/+/802446 Thanks & Regards, Vishal Manchanda On Wed, Sep 1, 2021 at 1:29 PM Herve Beraud wrote: > Well, the projects below could also ignore this thread, they are in the > same case that the ovn-octavia-provider. For more details see the previous > messages of this discussion. > > - horizon > - swift > - tempest > - vitrage-dashboard > - vitrage > > I remove these teams from the subject of this thread. > Thanks for your attention. > > Le mer. 1 sept. 2021 à 09:49, Herve Beraud a écrit : > >> >> >> Le mer. 1 sept. 2021 à 08:47, Slawek Kaplonski a >> écrit : >> >>> Hi, >>> >>> On poniedziałek, 30 sierpnia 2021 16:23:36 CEST Herve Beraud wrote: >>> > Hello teams >>> > >>> > Quick reminder that we'll need a release very soon for a number of >>> > deliverables following a cycle-with-intermediary release model but >>> which >>> > have not done *any* release yet in the Xena cycle: >>> > >>> > ansible-role-atos-hsm >>> > ansible-role-lunasa-hsm >>> > ansible-role-thales-hsm >>> > heat-agents >>> > horizon >>> > ironic-prometheus-exporter >>> > ironic-python-agent-builder >>> > ironic-ui >>> > ovn-octavia-provider >>> > patrole >>> > python-openstackclient >>> > requirements >>> > swift >>> > tempest >>> > vitrage-dashboard >>> > vitrage >>> > >>> > >>> > Those should be released ASAP, and in all cases before the week of 13 >>> > September, 2021, so >>> > that we have a release to include in the final Xena release. >>> > >>> > Thanks for your attention. >>> >>> I already proposed new release of the ovn-octavia-provider (1.1.1) [1] >>> and >>> it's done. >>> But TBH I'm a bit confused as we had already version 1.1.0 released in >>> Xena >>> cycle some time ago - see [2]. So I'm not really sure why this project >>> is on >>> that list now :) >>> >> >> Indeed you are right our tooling used to generate this list missed this >> point. I'll try to see why it didn't ignored ovn-octavia-provider. >> Thanks for the heads up. >> >> >>> [1] https://review.opendev.org/c/openstack/releases/+/806716 >>> [2] https://review.opendev.org/c/openstack/releases/+/803321 >>> >>> -- >>> Slawek Kaplonski >>> Principal Software Engineer >>> Red Hat >> >> >> >> -- >> Hervé Beraud >> Senior Software Engineer at Red Hat >> irc: hberaud >> https://github.com/4383/ >> https://twitter.com/4383hberaud >> >> > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ > https://twitter.com/4383hberaud > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Wed Sep 1 08:36:49 2021 From: hberaud at redhat.com (Herve Beraud) Date: Wed, 1 Sep 2021 10:36:49 +0200 Subject: [release][heat][ironic][requirements][OpenStackSDK][barbican][heat][ironic][neutron][QA] Cycle With Intermediary Unreleased Deliverables In-Reply-To: References: <11196443.jP3EJilxpz@p1> Message-ID: Yes, thanks. This is why I removed horizon from the list in my previous message http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024583.html We will propose a last release round for the cycle-with-intermediary projects in 2 weeks (during R-3). Horizon will be in the lot. If you feel you need to release more early than R-3, then, fire. Le mer. 1 sept. 2021 à 10:19, vishal manchanda a écrit : > Hi Herve, > > Same for the horizon as well like horizon already had version 20.0.0(first > release for Xena) [1]. > Do you still want me to cut a new release? > > [1] https://review.opendev.org/c/openstack/releases/+/802446 > > Thanks & Regards, > Vishal Manchanda > > On Wed, Sep 1, 2021 at 1:29 PM Herve Beraud wrote: > >> Well, the projects below could also ignore this thread, they are in the >> same case that the ovn-octavia-provider. For more details see the previous >> messages of this discussion. >> >> - horizon >> - swift >> - tempest >> - vitrage-dashboard >> - vitrage >> >> I remove these teams from the subject of this thread. >> Thanks for your attention. >> >> Le mer. 1 sept. 2021 à 09:49, Herve Beraud a écrit : >> >>> >>> >>> Le mer. 1 sept. 2021 à 08:47, Slawek Kaplonski a >>> écrit : >>> >>>> Hi, >>>> >>>> On poniedziałek, 30 sierpnia 2021 16:23:36 CEST Herve Beraud wrote: >>>> > Hello teams >>>> > >>>> > Quick reminder that we'll need a release very soon for a number of >>>> > deliverables following a cycle-with-intermediary release model but >>>> which >>>> > have not done *any* release yet in the Xena cycle: >>>> > >>>> > ansible-role-atos-hsm >>>> > ansible-role-lunasa-hsm >>>> > ansible-role-thales-hsm >>>> > heat-agents >>>> > horizon >>>> > ironic-prometheus-exporter >>>> > ironic-python-agent-builder >>>> > ironic-ui >>>> > ovn-octavia-provider >>>> > patrole >>>> > python-openstackclient >>>> > requirements >>>> > swift >>>> > tempest >>>> > vitrage-dashboard >>>> > vitrage >>>> > >>>> > >>>> > Those should be released ASAP, and in all cases before the week of 13 >>>> > September, 2021, so >>>> > that we have a release to include in the final Xena release. >>>> > >>>> > Thanks for your attention. >>>> >>>> I already proposed new release of the ovn-octavia-provider (1.1.1) [1] >>>> and >>>> it's done. >>>> But TBH I'm a bit confused as we had already version 1.1.0 released in >>>> Xena >>>> cycle some time ago - see [2]. So I'm not really sure why this project >>>> is on >>>> that list now :) >>>> >>> >>> Indeed you are right our tooling used to generate this list missed this >>> point. I'll try to see why it didn't ignored ovn-octavia-provider. >>> Thanks for the heads up. >>> >>> >>>> [1] https://review.opendev.org/c/openstack/releases/+/806716 >>>> [2] https://review.opendev.org/c/openstack/releases/+/803321 >>>> >>>> -- >>>> Slawek Kaplonski >>>> Principal Software Engineer >>>> Red Hat >>> >>> >>> >>> -- >>> Hervé Beraud >>> Senior Software Engineer at Red Hat >>> irc: hberaud >>> https://github.com/4383/ >>> https://twitter.com/4383hberaud >>> >>> >> >> -- >> Hervé Beraud >> Senior Software Engineer at Red Hat >> irc: hberaud >> https://github.com/4383/ >> https://twitter.com/4383hberaud >> >> -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at stackhpc.com Wed Sep 1 08:52:29 2021 From: mark at stackhpc.com (Mark Goddard) Date: Wed, 1 Sep 2021 09:52:29 +0100 Subject: [nova][cinder][barbican] storing secrets in service project In-Reply-To: References: Message-ID: On Tue, 31 Aug 2021 at 14:50, Pavlo Shchelokovskyy wrote: > > Hi all, > > When Barbican is involved, and services like Nova and Cinder create secrets behind the scenes themselves, > this results in problems deleting or otherwise managing some resources by cloud admins. > > For example as described in https://bugs.launchpad.net/cinder/+bug/1775372, > admin can not delete an encrypted volume created by another user, > and needs to resort to adding herself to the project first with appropriate roles. > > This stems IMO from the difference in default resource access policies between Barbican and other services that consume it. > For example, while in Nova or Cinder, cloud admin by default can delete servers or volumes in other projects, > in Barbican by default only secret creator or admin "in the same project" can delete a secret. > > I came up with at least 3 solutions (already mentioned in the issue above) but would like > to discuss their validity, pros and cons, and especially how they affect the security model > and the use cases that this Nova-Cinder-Barbican integration aims to solve. > > 1. Just ignore failure to delete a secret and log appropriate warning that a secret may be left orphaned. > Obviously not the best solution, but rather simple one, also will protect from cases when for whatever reason > a secret will be deleted from Barbican directly (e.g. by mistake), rendering volume undeletable for anyone. > > 2. Change default barbican policies to allow secret deletion for cloud admin (system scope?). > Naturally has implications for the Barbican model for resources access, so may be not that appropriate. > > 3. Store the secrets in the service project instead of user's. > Currently those secrets are generated by Nova or Cinder themselves, and user quite probably is not even aware > that there was something created in Barbican on her behalf. The user also has no option to pass her own secrets > to any of the involved API calls, so this is something happening completely behind the scenes. > So do we really need to store these secrets in the user's project? > Wouldn't it be better to store them in the service project, have the same service user defined in Nova and Cinder, > so it can operate and access those secrets regardless of who manages the resource which depends on access to this secret? > This also may solve the problem of encrypted local instance images in Nova, where upon host restart, > Nova is not able to bring those instances up because it lacks the context capable of accessing those secrets. > > I think the third option is the best one, but I may be missing the actual use case meant behind this feature. > Is it "keep the data encrypted at rest and in transit" or more strictly "encrypt data of different users with different keys > and limit access to those keys" (for the case of breach etc) ? > In the first case it seems storing all the auto-generated secrets in a service project would suffice... > > Please help me understand the best approach to take. Some loosely-related context: There are certainly some operational issues when using encrypted volumes. In a recent project we used custom policy to meet the requirements to avoid some issues. We configured nova & cinder to send a service token to barbican, and configured the barbican policy to only allow secret decryption for certain combinations of user roles and service roles. This allows an operator with a particular role to migrate another user's instance, but prevents them from reading the secret directly since they wouldn't have the service role. Unfortunately it was not possible to do the same for barbican secrets since there is a hard coded project check in the barbican DB layer which bypasses policy. Some related patches: * https://review.opendev.org/c/openstack/castellan/+/769560 * https://review.opendev.org/c/openstack/nova/+/781079 Other patches are yet to make it out of the environment. > > Best regards, > Pavlo. From arthur.outhenin-chalandre at cern.ch Wed Sep 1 09:23:03 2021 From: arthur.outhenin-chalandre at cern.ch (Arthur Outhenin-Chalandre) Date: Wed, 1 Sep 2021 11:23:03 +0200 Subject: [cinder][nova] fsfreeze hooks issues with cinder snapshot/backup In-Reply-To: References: <853de0f0-e6ad-142c-4811-cab1fa921c17@cern.ch> Message-ID: Hello, On 8/31/21 5:19 PM, Sofia Enriquez wrote: > As far as I can see cinder hasn't implemented this. However, I'm not > sure about the status of this feature because the last update was on 2014[1] Oh ok, I was hoping it was a setup problem on our side. Thanks for letting me know, it has saved me from further testing/research! > This sounds like a good PTG discussion topic.  You can add it to the > planning etherpad here: > https://etherpad.opendev.org/p/yoga-ptg-cinder-planning > > > There's also info about the dates and times we'll be meeting on that > etherpad. I added the discussion topic and I will be around to discuss it. I should be able to participate in the dev if needed as well. Cheers, -- Arthur Outhenin-Chalandre From lyarwood at redhat.com Wed Sep 1 10:20:21 2021 From: lyarwood at redhat.com (Lee Yarwood) Date: Wed, 1 Sep 2021 11:20:21 +0100 Subject: [cinder][nova] fsfreeze hooks issues with cinder snapshot/backup In-Reply-To: References: <853de0f0-e6ad-142c-4811-cab1fa921c17@cern.ch> Message-ID: <20210901102021.7pdpozvq4u36nunl@lyarwood-laptop.usersys.redhat.com> On 31-08-21 12:19:54, Sofia Enriquez wrote: > Hello, > > As far as I can see cinder hasn't implemented this. However, I'm not sure > about the status of this feature because the last update was on 2014[1] Oddly enough imageCreate [1] in Nova does orchestrate this when creating image based snapshots of volume backed instances [2] when called directly. Nova doesn't provide an external API to quiesce an instance so outside of the imageCreate and os-assisted-volume-snapshots [3][4] (only used by the NFS c-vol backend) flows it isn't going to happen. [1] https://docs.openstack.org/api-ref/compute/?expanded=create-image-createimage-action-detail#create-image-createimage-action [2] https://github.com/openstack/nova/blob/e81211318a29c5f07ee2b9c753dbab041ee01c21/nova/compute/api.py#L3353-L3383 [3] https://docs.openstack.org/api-ref/compute/#assisted-volume-snapshots-os-assisted-volume-snapshots [4] https://github.com/openstack/nova/blob/e81211318a29c5f07ee2b9c753dbab041ee01c21/nova/virt/libvirt/driver.py#L3346-L3367 > I think it's important to mention that this would only affect the live > snapshots (handled by Nova) but for any other scenario every cinder driver > optimized the snapshot/backup creation in a different way. Well to be clear this specific problem is with Cinder orchestrated live attached volume snapshots of non-NFS backends. > This sounds like a good PTG discussion topic. You can add it to the > planning etherpad here: > https://etherpad.opendev.org/p/yoga-ptg-cinder-planning I'll add the above to the pad and will try to attend as this will require work on the Nova side to provide an external API or preferably an os-server-external-events event to quiesce the instance. Cheers, Lee > On Tue, Aug 31, 2021 at 11:52 AM Arthur Outhenin-Chalandre < > arthur.outhenin-chalandre at cern.ch> wrote: > > > Hello, > > > > We are trying to trigger an fsfreeze via a cinder backup or snapshot. We > > confirmed that the fsfreeze hooks are actually called with a nova > > snapshot with `/var/log/qga-fsfreeze-hook.log` in the VM, but we can't > > achieve the same thing with a cinder backup/snapshot attached to the > > same instance. We are using Wallaby, libvirt, RBD for cinder volumes and > > RBD as well for cinder-backup. > > > > According to this (old) spec [0], cinder should call the `quiesce()` > > method in nova during backup/snapshot. We looked in the cinder code and > > couldn't find any clear evidence that this method is actually called by > > cinder (but we may have missed something). We added some debug messages > > on quiesce/can_quiesce/require_quiesce/... in > > `nova/virt/libvirt/driver.py` and they are never called with a cinder > > backup/snapshot in our setup while they are (and succeed) if we do a > > nova snapshot. > > > > We are starting to suspect that something is missing in cinder, but it > > could very well be a problem with our setup as well... Does someone use > > this feature or know if it should be working/implemented? > > > > [0]: > > > > https://wiki.openstack.org/wiki/Cinder/QuiescedSnapshotWithQemuGuestAgent#Cinder > > > > Cheers, > > > > -- > > Arthur Outhenin-Chalandre > > > > > > -- > > L. Sofía Enriquez > > she/her > > Software Engineer > > Red Hat PnT > > IRC: @enriquetaso > @RedHat Red Hat > Red Hat > > -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From eblock at nde.ag Wed Sep 1 12:02:25 2021 From: eblock at nde.ag (Eugen Block) Date: Wed, 01 Sep 2021 12:02:25 +0000 Subject: Cinder API timeout on single-control node Message-ID: <20210901120225.Horde.Fyn_birethoRxxPJpwzbF3O@webmail.nde.ag> Hi *, since your last responses were quite helpful regarding rabbitmq I would like to ask a different question for the same environment. It's an older openstack version (Pike) running with only one control node. There already were lots of volumes (way more than 1000) in that cloud, but after adding a bunch more (not sure how many exactly) in one project the whole cinder api became extremely slow. Both horizon and CLI run into timeouts: [Wed Sep 01 13:18:52.109178 2021] [wsgi:error] [pid 60440] [client :58474] Timeout when reading response headers from daemon process 'horizon': /srv/www/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi, referer: http:///project/volumes/ [Wed Sep 01 13:18:53.664714 2021] [wsgi:error] [pid 13007] Not Found: /favicon.ico Sometimes the volume creation succeeds if you just retry, but it often fails. The dashboard shows a "504 gateway timeout" after two minutes (also after four minutes after I increased the timeout for the apache dashboard config). The timeout also shows even if I try to get into the volumes tab of an empty project. A couple of weeks ago I already noticed some performance issues with cinder api if there are lots of attached volumes, if there are many "available" volumes it doesn't seem to slow things down. But since then the total number of volumes has doubled. At the moment there are more than 960 attached volumes across all projects and more than 750 detached volumes. I searched the cinder.conf for any helpful setting but I'm not sure which would actually help. And since it's a production cloud I would like to avoid restarting services all the time just to try something. Maybe some of you can point me in the right direction? I would appreciate any help! If there's more information I can provide just let me know. Thanks! Eugen From joykechen at 163.com Wed Sep 1 12:37:47 2021 From: joykechen at 163.com (=?GBK?B?s8K/yw==?=) Date: Wed, 1 Sep 2021 20:37:47 +0800 (CST) Subject: [nova][cyborg][gpu] GPU performance testing In-Reply-To: References: Message-ID: <5b765247.6fd1.17ba15d8b43.Coremail.joykechen@163.com> Hi, I recommend a tool to test GPU performance, which depends on CUDA installation. link: https://github.com/GPUburn/gpuburn/tree/master/GPUBurn Thanks, Ke Chen 在 2021-08-31 23:10:03，"Sylvain Bauza" 写道： On Tue, Aug 31, 2021 at 4:34 PM Ildiko Vancsa wrote: Hi, As we are approaching the end of the holiday season I wanted to surface back my question about GPU performance testing. Does anyone have any hints to find the best tools to do some benchmarking with? You made the point, Ildiko, I was on a long-running time-off so I didn't had time to look at your question yet. Good concern tho, I have no knowledge about this, but I can ping a few other folks to get you an answer. -Sylvain Thanks, Ildikó > On Aug 9, 2021, at 08:46, Ildiko Vancsa wrote: > > Hi, > > I got a question about tools and practices to check GPU performance in an OpenStack environment that I need some help to answer. > > The question is about recommended GPU performance testing/benchmarking tools if there are a few that people in the community are using and would recommend? The scope of the testing work is to check GPU performance in OpenStack VMs (both virtualized and passthrough). > > All the help and pointers are very much appreciated! > > Thanks, > Ildikó > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joykechen at 163.com Wed Sep 1 12:28:14 2021 From: joykechen at 163.com (=?GBK?B?s8K/yw==?=) Date: Wed, 1 Sep 2021 20:28:14 +0800 (CST) Subject: [all][elections][ptl] PTL Voting - Cyborg - Ke Chen Message-ID: <2b3b57b0.6f63.17ba154cbc9.Coremail.joykechen@163.com> Hi, I would like to announce my candidacy for Cyborg PTL for the Yoga cycle, And hope to get your support. As an openstack enthusiast, I have been engaged in openstack for 4 years. I was lucky to make friends with openstack projects such as Cyborg and Watcher in 2018. I met many interesting partners, such as Matt Riedemann, Sundar Nadathur, Huangzhipeng, Liuli, Baoyumeng, Zhurong, Licanwei, Fengshaohe, Wangxinran, Shogo, Zhangbailin, Songwenping and others. You let me see the charm of the community and rigorous working attitude, thanks. In the past two years, except for the watcher project, I spent most of my time dealing with Cyborg. To be honest, I know cyborg very well. At present, Cyborg can manage FPGA, GPU, smartnic and other acceleration devices. Users can create virtual machines with acceleration resources as needed, which is a very cool thing. However, our Cyborg project still has many things to do. In the next y version, I think there are mainly the following five things to do: 1. Optimize the interaction between Nova and cyborg. The existing interface is synchronous call. On cyborg side, we can change synchronization to asynchronous to save the landing time of concurrent virtual machines. 2. Support for more drivers and more universal drivers. At present, vgpu driver is not supported in cyborg. I hope to implement it in the Yoga cycle. 3. Implementation of disable and enable acceleration devices. 4. More bugfix and documentation enhancements. 5. More virtual machine operation support, such as cold migration. I hope we can bring users a more exciting experience in the next Yoga version. Thanks, Ke Chen(chenke) -------------- next part -------------- An HTML attachment was scrubbed... URL: From senrique at redhat.com Wed Sep 1 13:21:09 2021 From: senrique at redhat.com (Sofia Enriquez) Date: Wed, 1 Sep 2021 10:21:09 -0300 Subject: [cinder] Bug deputy report for week of 2021-01-09 Message-ID: This is a bug report from 2021-18-08 to 2021-01-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/cinder/+bug/1942077 "Online database schema migrations to Xena always fail". - https://bugs.launchpad.net/cinder/+bug/1942218 "Huawei FC driver cannot co-locate cinder-volume service with computer". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1942095 "powerflex driver: no volume caching after 126 volumes". Assigned to Vladislav Belogrudov. - https://bugs.launchpad.net/cinder/+bug/1942090 "SolidFire retype fails due to volume status as retyping and not available". Assigned to Fábio Oliveira. - https://bugs.launchpad.net/cinder/+bug/1941694 "IBM SVF driver: Detaching from second instance fails for multi-attach volumes". Unassigned. Low - https://bugs.launchpad.net/cinder/+bug/1942301 "[synology] embed driver uses the insecure MD5 algorithm". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1941746 "JovianDSS: incorrect operation of ensure export function". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1930267 "ssh exec timeout". Unassigned. Incomplete - https://bugs.launchpad.net/cinder/+bug/1942154 " Backup availability_zone is None". Unassigned. Cheers, -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Wed Sep 1 13:32:52 2021 From: smooney at redhat.com (Sean Mooney) Date: Wed, 01 Sep 2021 14:32:52 +0100 Subject: [nova][cyborg][gpu] GPU performance testing In-Reply-To: <5b765247.6fd1.17ba15d8b43.Coremail.joykechen@163.com> References: <5b765247.6fd1.17ba15d8b43.Coremail.joykechen@163.com> Message-ID: On Wed, 2021-09-01 at 20:37 +0800, 陈克 wrote: > Hi, > I recommend a tool to test GPU performance, which depends on CUDA installation. > > > link: https://github.com/GPUburn/gpuburn/tree/master/GPUBurn > > > if youtube hard ware review have show us anything the best way to benchmark gpu hardware is actully to use repsentivte application rather then synteic benchmarks as gpu vendors have a habbit of tuninging the gpu driver to perfrom better in popular syntitich benchmarks. i would suggest selecting a representive set that cover a number of usecase from cad, gaming, to video transcoding to ml (inference and trianing) to compute vission. blender is a popular opensouce 3d modelely software and it provides some in build bench marks that can be used for evaulating the performace of 3d cad and rendering. opencv is a popular toolkit for compute vission and they have a numbner of benchmark examples such as https://docs.opencv.org/4.5.2/dc/d69/tutorial_dnn_superres_benchmark.html on the more pure ml side there are some syntetic benachmarks provided by tensorflow https://github.com/tensorflow/benchmarks/tree/master/perfzero but often a better approch is to use a common opensocue data set and model and messue the time it takes to train the same model with the same data set in differnet environemnts you also want to messure the infernce rate which involved takign a pre trained model and feeding it data form out side of its training set and mesurrign the performance. its important that that model and data used for the inferce is the same used across all your deployments. pytouch is also another popular ai/ml framework that can be used in a simialr way. looking quickly at https://github.com/GPUburn/gpuburn/tree/master/GPUBurn it looks liek its actully not a benchmark utility but instead a stress test tool for mesuring stablity which is a very differnt thing and i dont think it would be helpful in your cases. in fact using it in a public cloud for example could be considered a break of fair use since its really desiing to put the gpu under as much stress as possibel to detect fault hardware which will impact other users of the gpu/ most of the gaming/windows based tool are propritaty which makes them hard to use as benhcmarks in an opensouce project due to licensing so they likely shoudl be avoided. but ya my advice would be dont look for benchmarks in general but look for framworks/tools/applcation that peopel use to do actual useful work that can be sued to executate a repatable action like useing handbrake to transcode a video file form one format to another to messur the hardwarre encoder performance with fixed inputs then use that out in this case transcoding time as one aspect of the performance mesurment to establish your benchmark. > > > > > > Thanks, > Ke Chen > > > > > > > > 在 2021-08-31 23:10:03，"Sylvain Bauza" 写道： > > > > > > On Tue, Aug 31, 2021 at 4:34 PM Ildiko Vancsa wrote: > > Hi, > > As we are approaching the end of the holiday season I wanted to surface back my question about GPU performance testing. Does anyone have any hints to find the best tools to do some benchmarking with? > > > > > > > You made the point, Ildiko, I was on a long-running time-off so I didn't had time to look at your question yet. > > > Good concern tho, I have no knowledge about this, but I can ping a few other folks to get you an answer. > -Sylvain > > > Thanks, > Ildikó > > > > On Aug 9, 2021, at 08:46, Ildiko Vancsa wrote: > > > > Hi, > > > > I got a question about tools and practices to check GPU performance in an OpenStack environment that I need some help to answer. > > > > The question is about recommended GPU performance testing/benchmarking tools if there are a few that people in the community are using and would recommend? The scope of the testing work is to check GPU performance in OpenStack VMs (both virtualized and passthrough). > > > > All the help and pointers are very much appreciated! > > > > Thanks, > > Ildikó > > > > > > From jungleboyj at gmail.com Wed Sep 1 13:34:49 2021 From: jungleboyj at gmail.com (Jay Bryant) Date: Wed, 1 Sep 2021 08:34:49 -0500 Subject: [cinder][nova] fsfreeze hooks issues with cinder snapshot/backup In-Reply-To: References: <853de0f0-e6ad-142c-4811-cab1fa921c17@cern.ch> Message-ID: <99aef947-9660-fee5-50d3-a1f6de990580@gmail.com> On 8/31/2021 10:19 AM, Sofia Enriquez wrote: > Hello, > > As far as I can see cinder hasn't implemented this. However, I'm not > sure about the status of this feature because the last update was on > 2014[1] > I remember being a part of these discussions and the fact that there was interest in getting it working.  I am wondering if this was just a case where no one followed through on implementation. It might be that we encountered an unexpected challenge that I don't remember. Either way, I do agree that this would be a good topic for the PTG. > I think it's important to mention that this would only affect the live > snapshots (handled by Nova) but for any other scenario every cinder > driver optimized the snapshot/backup creation in a different way. > > This sounds like a good PTG discussion topic.  You can add it to the > planning etherpad here: > https://etherpad.opendev.org/p/yoga-ptg-cinder-planning > > > There's also info about the dates and times we'll be meeting on that > etherpad. > > Cheers, > Sofia > > [1] > https://blueprints.launchpad.net/cinder/+spec/quiesced-snapshots-with-qemu-guest-agent > > > On Tue, Aug 31, 2021 at 11:52 AM Arthur Outhenin-Chalandre > > wrote: > > Hello, > > We are trying to trigger an fsfreeze via a cinder backup or > snapshot. We > confirmed that the fsfreeze hooks are actually called with a nova > snapshot with `/var/log/qga-fsfreeze-hook.log` in the VM, but we can't > achieve the same thing with a cinder backup/snapshot attached to the > same instance. We are using Wallaby, libvirt, RBD for cinder > volumes and > RBD as well for cinder-backup. > > According to this (old) spec [0], cinder should call the `quiesce()` > method in nova during backup/snapshot. We looked in the cinder > code and > couldn't find any clear evidence that this method is actually > called by > cinder (but we may have missed something). We added some debug > messages > on quiesce/can_quiesce/require_quiesce/... in > `nova/virt/libvirt/driver.py` and they are never called with a cinder > backup/snapshot in our setup while they are (and succeed) if we do a > nova snapshot. > > We are starting to suspect that something is missing in cinder, but it > could very well be a problem with our setup as well... Does > someone use > this feature or know if it should be working/implemented? > > [0]: > https://wiki.openstack.org/wiki/Cinder/QuiescedSnapshotWithQemuGuestAgent#Cinder > > > Cheers, > > -- > Arthur Outhenin-Chalandre > > > > -- > > L. Sofía Enriquez > > she/her > > Software Engineer > > Red Hat PnT > > IRC: @enriquetaso > > @RedHat Red Hat > Red Hat > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnasiadka at gmail.com Wed Sep 1 13:57:29 2021 From: mnasiadka at gmail.com (=?utf-8?Q?Micha=C5=82_Nasiadka?=) Date: Wed, 1 Sep 2021 15:57:29 +0200 Subject: [kolla-ansible] [kuryr] Running Docker Swarm with Kuryr networking In-Reply-To: <7132cbfa-c8be-76c7-3f77-5027f74eff2e@celati.com> References: <7132cbfa-c8be-76c7-3f77-5027f74eff2e@celati.com> Message-ID: Hi Paolo, Would you like to use the Docker engine that is running on the OpenStack cluster hosts, or create Virtual Machines that will be used for a Docker Swarm cluster? I would propose the latter. About Kuryr - we don’t have CI coverage for testing Kuryr in Kolla-Ansible deployment, so the container images and Ansible deployment role are provided as-is currently. Maybe somebody from Kuryr project could help you out? Adding [kuryr] tag for visibility. Best regards, Michal On 1 Sep 2021, 00:32 +0200, Paolo Celati , wrote: > Hi, > long story short I have a 3 node Openstack cluster that I manage with kolla-ansible, and I'd like to run Docker Swarm on that as well. I am aware Magnum exists, but I'd first like to get my head around this simpler case. > Seeing as I'd like to connect Docker containers from swarm compose files to Neutron networks I'm trying to set up Kuryr together with a swarm configuration. However the documentation is a little scarce and I'd prefer running everything on these three hosts, including etcd. If I follow the guide and pass --cluster-store and --cluster-advertise arguments to dockerd then I can't run Docker in Swarm mode because I get an error saying Swarm is incompatible with those options, and at the same time it's not clear from documentation how you are expected to do Kuryr+Swarm. I did initialise the Swarm cluster before trying to add Kuryr, so I don't know if perhaps doing this the other way works? Do you have ideas or advice with this scenario? If worst comes to worst I can set up an external etcd cluster on a separate non-Openstack cluster but I'd rather avoid that. > > Thanks in advance, > Paolo -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.bell at cern.ch Wed Sep 1 14:46:16 2021 From: tim.bell at cern.ch (Tim Bell) Date: Wed, 1 Sep 2021 16:46:16 +0200 Subject: [nova][cyborg][gpu] GPU performance testing In-Reply-To: <5b765247.6fd1.17ba15d8b43.Coremail.joykechen@163.com> References: <5b765247.6fd1.17ba15d8b43.Coremail.joykechen@163.com> Message-ID: <8E5CAEE8-19DC-416F-939E-A7E7A0DF19FD@cern.ch> > On 1 Sep 2021, at 14:37, 陈克 wrote: > > Hi, > I recommend a tool to test GPU performance, which depends on CUDA installation. > > link: https://github.com/GPUburn/gpuburn/tree/master/GPUBurn > gpuburn is also the tool that CERN uses also when new hardware is delivered and we want to check the hardware is working under load. It’s also a good way to stress your PDUs and cooling if you have several cards (see https://indico.cern.ch/event/950196/contributions/3993259/attachments/2113752/3556174/GPUs_in_CERN_IT.pdf) We also have some specific High Energy Physics benchmarks - https://gitlab.cern.ch/hep-benchmarks/hep-workloads-gpu Cheers Tim > > Thanks, > Ke Chen > > > 在 2021-08-31 23:10:03，"Sylvain Bauza" 写道： > > > > On Tue, Aug 31, 2021 at 4:34 PM Ildiko Vancsa > wrote: > Hi, > > As we are approaching the end of the holiday season I wanted to surface back my question about GPU performance testing. Does anyone have any hints to find the best tools to do some benchmarking with? > > > > You made the point, Ildiko, I was on a long-running time-off so I didn't had time to look at your question yet. > > Good concern tho, I have no knowledge about this, but I can ping a few other folks to get you an answer. > -Sylvain > > Thanks, > Ildikó > > > > On Aug 9, 2021, at 08:46, Ildiko Vancsa > wrote: > > > > Hi, > > > > I got a question about tools and practices to check GPU performance in an OpenStack environment that I need some help to answer. > > > > The question is about recommended GPU performance testing/benchmarking tools if there are a few that people in the community are using and would recommend? The scope of the testing work is to check GPU performance in OpenStack VMs (both virtualized and passthrough). > > > > All the help and pointers are very much appreciated! > > > > Thanks, > > Ildikó > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdulko at redhat.com Wed Sep 1 15:23:54 2021 From: mdulko at redhat.com (=?UTF-8?Q?Micha=C5=82?= Dulko) Date: Wed, 01 Sep 2021 17:23:54 +0200 Subject: [kolla-ansible] [kuryr] Running Docker Swarm with Kuryr networking In-Reply-To: References: <7132cbfa-c8be-76c7-3f77-5027f74eff2e@celati.com> Message-ID: <92301d2d1b574d55ff9c5b45ceab80408efdd0b5.camel@redhat.com> On Wed, 2021-09-01 at 15:57 +0200, Michał Nasiadka wrote: > Hi Paolo, > > Would you like to use the Docker engine that is running on the > OpenStack cluster hosts, or create Virtual Machines that will be used > for a Docker Swarm cluster? > I would propose the latter. > > About Kuryr - we don’t have CI coverage for testing Kuryr in Kolla- > Ansible deployment, so the container images and Ansible deployment > role are provided as-is currently. > > Maybe somebody from Kuryr project could help you out? Adding [kuryr] > tag for visibility. I doubt anybody on the current team has any experience with running Docker's Kuryr. We're focused on kuryr-kubernetes and Kuryr is in de- facto unmaintained state. > Best regards, > > Michal > On 1 Sep 2021, 00:32 +0200, Paolo Celati , wrote: > > Hi, > > long story short I have a 3 node Openstack cluster that I manage > > with kolla-ansible, and I'd like to run Docker Swarm on that as > > well. I am aware Magnum exists, but I'd first like to get my head > > around this simpler case. > > Seeing as I'd like to connect Docker containers from swarm compose > > files to Neutron networks I'm trying to set up Kuryr together with > > a swarm configuration. However the documentation is a little scarce > > and I'd prefer running everything on these three hosts, including > > etcd. If I follow the guide and pass --cluster-store and --cluster- > > advertise arguments to dockerd then I can't run Docker in Swarm > > mode because I get an error saying Swarm is incompatible with those > > options, and at the same time it's not clear from documentation how > > you are expected to do Kuryr+Swarm. I did initialise the Swarm > > cluster before trying to add Kuryr, so I don't know if perhaps > > doing this the other way works? Do you have ideas or advice with > > this scenario? If worst comes to worst I can set up an external > > etcd cluster on a separate non-Openstack cluster but I'd rather > > avoid that. > > > > Thanks in advance, > > Paolo From yumeng_bao at yahoo.com Wed Sep 1 15:39:40 2021 From: yumeng_bao at yahoo.com (yumeng bao) Date: Wed, 1 Sep 2021 23:39:40 +0800 Subject: [all][elections][ptl] PTL Voting - Cyborg - Ke Chen References: Message-ID: Dear Ke， Thanks for your contributions in cyborg ever since 2018, it’s been more than five release cycles past. By reviewing the patches you’ve committed and suggestions you’ve given to the team, it’s obvious you know cyborg very well, from the bottom to up level including GPU,FPGA,SmartNIC drivers to api, as well as interactions with nova and placement. Although I’m not contributing in cyborg anymore due to my job change, I do agree the five things you listed are important for cyborg, and I would like to see them done in the future. So it would be a big vote +1 from my side. Good luck. Regards， Yumeng > On Sep 1, 2021, at 8:29 PM, 陈克 wrote: >  > Hi, > I would like to announce my candidacy for Cyborg PTL for the Yoga cycle, And hope to get your support. > > As an openstack enthusiast, I have been engaged in openstack for 4 years. > I was lucky to make friends with openstack projects such as Cyborg and Watcher in 2018. > > I met many interesting partners, such as Matt Riedemann, Sundar Nadathur, > Huangzhipeng, Liuli, Baoyumeng, Zhurong, Licanwei, Fengshaohe, Wangxinran, > Shogo, Zhangbailin, Songwenping and others. > > You let me see the charm of the community and rigorous working attitude, thanks. > > In the past two years, except for the watcher project, I spent most of my > time dealing with Cyborg. To be honest, I know cyborg very well. > > At present, Cyborg can manage FPGA, GPU, smartnic and other acceleration > devices. Users can create virtual machines with acceleration resources as > needed, which is a very cool thing. > > However, our Cyborg project still has many things to do. > In the next y version, I think there are mainly the following > five things to do: > 1. Optimize the interaction between Nova and cyborg. > The existing interface is synchronous call. On cyborg side, we can change > synchronization to asynchronous to save the landing time of concurrent > virtual machines. > > 2. Support for more drivers and more universal drivers. > At present, vgpu driver is not supported in cyborg. I hope to implement > it in the Yoga cycle. > > 3. Implementation of disable and enable acceleration devices. > > 4. More bugfix and documentation enhancements. > > 5. More virtual machine operation support, such as cold migration. > > I hope we can bring users a more exciting experience in the next Yoga version. > > Thanks, > Ke Chen(chenke) -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry at openstack.org Wed Sep 1 15:41:56 2021 From: thierry at openstack.org (Thierry Carrez) Date: Wed, 1 Sep 2021 17:41:56 +0200 Subject: [largescale-sig] Next meeting: Sept 1st, 15utc In-Reply-To: <9d0c7f7b-c8cb-9cce-3cb9-0aba854a665d@openstack.org> References: <9d0c7f7b-c8cb-9cce-3cb9-0aba854a665d@openstack.org> Message-ID: <0b18630e-785c-a850-9a7a-ee43617490c3@openstack.org> We held our meeting today. Small attendance, we discussed date andtopic for our next "Large Scale OpenStack" OpenInfra.Live episode. That one will be on Oct 14 and the working topic is around architecture for large scale deployments. You can read the meeting logs at: https://meetings.opendev.org/meetings/large_scale_sig/2021/large_scale_sig.2021-09-01-15.01.html Our next IRC meeting will be Sept 15, at 1500utc on #openstack-operators on OFTC. Regards, -- Thierry Carrez (ttx) From tonykarera at gmail.com Wed Sep 1 01:51:43 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 1 Sep 2021 03:51:43 +0200 Subject: Wallaby Magnum Issue In-Reply-To: References: Message-ID: Hey Feilong, Thanks a lot. The services are fine and indeed the log files are there in the directory [ /var/log/heat-config/heat-config-script] After checking, the master log is fine but the cluster log has this error below as I had mentioned earlier Starting to run kube-apiserver-to-kubelet-role + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ kubectl get --raw=/healthz The connection to the server localhost:8080 was refused - did you specify the right host or port? + '[' ok = '' ']' Regards Tony Karera On Tue, Aug 31, 2021 at 9:52 PM feilong wrote: > Hi Karea, > > Given you can see heat-container-agent container from podman which means > you should be able to see logs from below path: > > [root at k8s-100-eh6s5l6d73ie-master-0 heat-config-script]# ls > > c64e6ac2-db7e-4786-a387-1d45359812b8-k8s-100-eh6s5l6d73ie-kube_cluster_config-uxpsylgnayjy.log > > fa1f6247-51a8-4e70-befa-cbc61ee99e59-k8s-100-eh6s5l6d73ie-kube_masters-kmi423lgbjw3-0-oii7uzemq7aj-master_config-dhfam54i456j.log > [root at k8s-100-eh6s5l6d73ie-master-0 heat-config-script]# pwd > /var/log/heat-config/heat-config-script > > If you can not see the path and the log, then it means the > heat-container-agent didn't work well. You need to check the service status > by systemctl command and check the log by journalctl. From there, you > should be able to see why the cluster failed. > > > > On 1/09/21 1:41 am, Karera Tony wrote: > > Dear Ammad, > > Sorry to bother you again but I have failed to get the right command to > use to check. > > Every Kubectl command I run on either the master or worker. > The connection to the server localhost:8080 was refused - did you specify > the right host or port? > I get the error below. > > > Regards > > Tony Karera > > > > > On Fri, Aug 27, 2021 at 9:15 AM Ammad Syed wrote: > >> Your hyperkube services are not started. >> >> You need to check hyperkube services. >> >> Ammad >> >> On Fri, Aug 27, 2021 at 10:35 AM Karera Tony >> wrote: >> >>> Dear Ammad, >>> >>> Below is the output of podman ps >>> >>> CONTAINER ID IMAGE >>> COMMAND CREATED STATUS PORTS >>> NAMES >>> 319fbebc2f50 >>> docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1 >>> /usr/bin/start-he... 23 hours ago Up 23 hours ago >>> heat-container-agent >>> [root at k8s-cluster-2-4faiphvzsmzu-master-0 core]# >>> >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Thu, Aug 26, 2021 at 9:54 AM Ammad Syed >>> wrote: >>> >>>> The output in >>>> logfile 29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd >>>> is incomplete. >>>> >>>> There should be the installation and configuration of many other things >>>> that are missing. Also it looks that hyperkube is not installed. >>>> >>>> Can you check the response of "podman ps" command on master nodes. >>>> >>>> Ammad >>>> >>>> On Thu, Aug 26, 2021 at 11:30 AM Karera Tony >>>> wrote: >>>> >>>>> Here is the beginning of the Log >>>>> >>>>> Starting to run kube-apiserver-to-kubelet-role >>>>> + echo 'Waiting for Kubernetes API...' >>>>> Waiting for Kubernetes API... >>>>> ++ kubectl get --raw=/healthz >>>>> The connection to the server localhost:8080 was refused - did you >>>>> specify the right host or port? >>>>> + '[' ok = '' ']' >>>>> >>>>> >>>>> Regards >>>>> >>>>> Tony Karera >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Aug 26, 2021 at 7:53 AM Bharat Kunwar >>>>> wrote: >>>>> >>>>>> I assume these are from the master nodes? Can you share the logs >>>>>> shortly after creation rather than when it times out? I think there is some >>>>>> missing logs from the top. >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On 26 Aug 2021, at 06:14, Karera Tony wrote: >>>>>> >>>>>>  >>>>>> Hello Guys, >>>>>> >>>>>> Attached are the two logs from the >>>>>> /var/log/heat-config/heat-config-script directory >>>>>> Regards >>>>>> >>>>>> Tony Karera >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Aug 26, 2021 at 5:59 AM Karera Tony >>>>>> wrote: >>>>>> >>>>>>> Dear Sir, >>>>>>> >>>>>>> You are right. >>>>>>> >>>>>>> I am getting this error >>>>>>> >>>>>>> kubectl get --raw=/healthz >>>>>>> The connection to the server localhost:8080 was refused - did you >>>>>>> specify the right host or port? >>>>>>> >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> Tony Karera >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Aug 25, 2021 at 10:55 PM Bharat Kunwar >>>>>>> wrote: >>>>>>> >>>>>>>> I’d check the logs under /var/log/heat-config. >>>>>>>> >>>>>>>> Sent from my iPhone >>>>>>>> >>>>>>>> On 25 Aug 2021, at 19:39, Karera Tony wrote: >>>>>>>> >>>>>>>>  >>>>>>>> DeaR Ammad, >>>>>>>> >>>>>>>> I was able to make the communication work and the Worker nodes were >>>>>>>> created as well but the cluster failed. >>>>>>>> >>>>>>>> I logged in to the master node and there was no error but below are >>>>>>>> the error when I run systemctl status heat-container-agent on the worker >>>>>>>> noed. >>>>>>>> >>>>>>>> Aug 25 17:52:24 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>> Aug 25 17:52:55 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>> Aug 25 17:53:26 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>> Aug 25 17:53:57 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>> Aug 25 17:54:28 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>> Aug 25 17:54:59 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>> Aug 25 17:55:29 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>> Aug 25 17:56:00 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>> Aug 25 17:56:31 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>> Aug 25 17:57:02 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>> Regards >>>>>>>> >>>>>>>> Tony Karera >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Aug 25, 2021 at 10:38 AM Ammad Syed >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yes, keystone, Heat, Barbicane and magnum public endpoints must be >>>>>>>>> reachable from master and worker nodes. >>>>>>>>> >>>>>>>>> You can use below guide for the reference as well. >>>>>>>>> >>>>>>>>> >>>>>>>>> https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=11 >>>>>>>>> >>>>>>>>> Ammad >>>>>>>>> >>>>>>>>> On Wed, Aug 25, 2021 at 12:08 PM Karera Tony >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hello Ammad, >>>>>>>>>> >>>>>>>>>> I have deployed using the given image but I think there is an >>>>>>>>>> issue with keystone as per the screen shot below when I checked the master >>>>>>>>>> node's heat-container-agent status >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> >>>>>>>>>> Tony Karera >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Aug 25, 2021 at 8:28 AM Karera Tony >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hello Ammad, >>>>>>>>>>> >>>>>>>>>>> I actually first used that one and it was also getting stuck. >>>>>>>>>>> >>>>>>>>>>> I will try this one again and update you with the Logs though. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards >>>>>>>>>>> >>>>>>>>>>> Tony Karera >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Aug 25, 2021 at 8:25 AM Ammad Syed < >>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> It seems from the logs that you are using fedora atomic. Can >>>>>>>>>>>> you try with FCOS 32 image. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20201004.3.0/x86_64/fedora-coreos-32.20201004.3.0-openstack.x86_64.qcow2.xz >>>>>>>>>>>> >>>>>>>>>>>> Ammad >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Aug 25, 2021 at 11:20 AM Karera Tony < >>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello Sir, >>>>>>>>>>>>> >>>>>>>>>>>>> Attached is the Log file >>>>>>>>>>>>> >>>>>>>>>>>>> Regards >>>>>>>>>>>>> >>>>>>>>>>>>> Tony Karera >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 25, 2021 at 7:31 AM Ammad Syed < >>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Karera, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you share us the full log file. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ammad >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Aug 25, 2021 at 9:42 AM Karera Tony < >>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello Guys, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks a lot for the help but unfortunately I dont see much >>>>>>>>>>>>>>> information in the log file indicating a failure apart from the log that >>>>>>>>>>>>>>> keeps appearing. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Tony Karera >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Aug 24, 2021 at 8:12 PM Mohammed Naser < >>>>>>>>>>>>>>> mnaser at vexxhost.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Also check out /var/log/cloud-init.log :) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Aug 24, 2021 at 1:39 PM Ammad Syed < >>>>>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Then check journalctl -xe or status of heat agent service >>>>>>>>>>>>>>>> status. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Ammad >>>>>>>>>>>>>>>> > On Tue, Aug 24, 2021 at 10:36 PM Karera Tony < >>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> Hello Ammad, >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> There is no directory or log relevant to heat in the >>>>>>>>>>>>>>>> /var/log directory >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> Regards >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> Tony Karera >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> On Tue, Aug 24, 2021 at 12:43 PM Ammad Syed < >>>>>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> Hi Karera, >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> Login to master node and check the logs of heat agent >>>>>>>>>>>>>>>> in var log. There must be something the cluster is stucking somewhere in >>>>>>>>>>>>>>>> creating. >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> Ammad >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> On Tue, Aug 24, 2021 at 3:41 PM Karera Tony < >>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> Hello Ammad, >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> I had done as explained and it worked upto a certain >>>>>>>>>>>>>>>> point. The master node was created but the cluster remained in Creation in >>>>>>>>>>>>>>>> progress for over an hour and failed with error below >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> Stack Faults >>>>>>>>>>>>>>>> >>>> as follows: >>>>>>>>>>>>>>>> >>>> default-master >>>>>>>>>>>>>>>> >>>> Timed out >>>>>>>>>>>>>>>> >>>> default-worker >>>>>>>>>>>>>>>> >>>> Timed out >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> Regards >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> Tony Karera >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> On Tue, Aug 24, 2021 at 9:25 AM Ammad Syed < >>>>>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> Hi Tony, >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> You can try by creating your private vxlan network >>>>>>>>>>>>>>>> prior to deployment of cluster and explicitly create your cluster in vxlan >>>>>>>>>>>>>>>> network. >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> --fixed-network private --fixed-subnet private-subnet >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> You can specify above while creating a cluster. >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> Ammad >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> On Tue, Aug 24, 2021 at 11:59 AM Karera Tony < >>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>> >>>>>> Hello MOhamed, >>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>> >>>>>> I think the Kubernetes cluster is ok but it when I >>>>>>>>>>>>>>>> deploy it, It creates a fixed network using vlan which I am not using for >>>>>>>>>>>>>>>> internal networks. >>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>> >>>>>> When I create a a vxlan Network and use it in the >>>>>>>>>>>>>>>> cluster creation, It fails. Is there a trick around this ? >>>>>>>>>>>>>>>> >>>>>> Regards >>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>> >>>>>> Tony Karera >>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>> >>>>>> On Fri, Aug 20, 2021 at 9:00 AM feilong < >>>>>>>>>>>>>>>> feilong at catalyst.net.nz> wrote: >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> Oooh, are you using Swarm? I don't think that >>>>>>>>>>>>>>>> driver is well maintained. I didn't see any interest in the last 4 years >>>>>>>>>>>>>>>> since I involved in the Magnum project. If there is no specific reason, I >>>>>>>>>>>>>>>> would suggest go for k8s. >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> On 20/08/21 5:08 pm, Mohammed Naser wrote: >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> Please keep replies on list so others can help too. >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> I don’t know how well tested the Swarm driver is at >>>>>>>>>>>>>>>> this point. I believe most Magnum users are using it for Kubernetes only. >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> On Fri, Aug 20, 2021 at 1:05 AM Karera Tony < >>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> Hello Naser, >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> Please check below. >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> openstack coe cluster template create >>>>>>>>>>>>>>>> swarm-cluster-template1 \ >>>>>>>>>>>>>>>> >>>>>>>> --image fedora-atomic-latest \ >>>>>>>>>>>>>>>> >>>>>>>> --external-network >>>>>>>>>>>>>>>> External_1700\ >>>>>>>>>>>>>>>> >>>>>>>> --dns-nameserver 8.8.8.8 \ >>>>>>>>>>>>>>>> >>>>>>>> --master-flavor m1.small \ >>>>>>>>>>>>>>>> >>>>>>>> --flavor m1.small \ >>>>>>>>>>>>>>>> >>>>>>>> --coe swarm >>>>>>>>>>>>>>>> >>>>>>>> openstack coe cluster create swarm-cluster \ >>>>>>>>>>>>>>>> >>>>>>>> --cluster-template >>>>>>>>>>>>>>>> swarm-cluster-template \ >>>>>>>>>>>>>>>> >>>>>>>> --master-count 1 \ >>>>>>>>>>>>>>>> >>>>>>>> --node-count 2 \ >>>>>>>>>>>>>>>> >>>>>>>> --keypair Newkey >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> Regards >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> Tony Karera >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> On Fri, Aug 20, 2021 at 7:03 AM Mohammed Naser < >>>>>>>>>>>>>>>> mnaser at vexxhost.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>> What does your cluster template and cluster >>>>>>>>>>>>>>>> create command look like? >>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>> On Fri, Aug 20, 2021 at 12:59 AM Karera Tony < >>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> Hello Wang, >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> Thanks for the feedback. >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> Unfortunately Octavia is not deployed in my >>>>>>>>>>>>>>>> environment (at least not yet) and LB is not enabled on either the cluster >>>>>>>>>>>>>>>> template or the cluster itself. >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> Tony Karera >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> On Fri, Aug 20, 2021 at 6:52 AM feilong < >>>>>>>>>>>>>>>> feilong at catalyst.net.nz> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> Hi Karera, >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> It's probably a bug. If you do have Octavia >>>>>>>>>>>>>>>> deployed, can you try to not disable the LB and see how it goes? >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> On 20/08/21 4:18 pm, Karera Tony wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> Hello Team, >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> I deployed Openstack Wallby on Ubuntu20 and >>>>>>>>>>>>>>>> enabled Magum, however when I create a cluster I get the error below. >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> Status Reason >>>>>>>>>>>>>>>> >>>>>>>>>>> ERROR: Property error: : >>>>>>>>>>>>>>>> resources.api_lb.properties: : Property allowed_cidrs not assigned >>>>>>>>>>>>>>>> >>>>>>>>>>> Can someone advise on where I could be wrong. >>>>>>>>>>>>>>>> Btw, I disabled load balancer while creating the cluster. >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> Regards >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> Tony Karera >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>> Cheers & Best regards, >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>> >>>>>>>>>>> Feilong Wang (王飞龙) (he/him) >>>>>>>>>>>>>>>> >>>>>>>>>>> Head of Research & Development >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> Catalyst Cloud >>>>>>>>>>>>>>>> >>>>>>>>>>> Aotearoa's own >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> Mob: +64 21 0832 6348 | www.catalystcloud.nz >>>>>>>>>>>>>>>> >>>>>>>>>>> Level 6, 150 Willis Street, Wellington 6011, >>>>>>>>>>>>>>>> New Zealand >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>> CONFIDENTIALITY NOTICE: This email is intended >>>>>>>>>>>>>>>> for the named recipients only. >>>>>>>>>>>>>>>> >>>>>>>>>>> It may contain privileged, confidential or >>>>>>>>>>>>>>>> copyright information. If you are >>>>>>>>>>>>>>>> >>>>>>>>>>> not the named recipient, any use, reliance >>>>>>>>>>>>>>>> upon, disclosure or copying of this >>>>>>>>>>>>>>>> >>>>>>>>>>> email or its attachments is unauthorised. If >>>>>>>>>>>>>>>> you have received this email in >>>>>>>>>>>>>>>> >>>>>>>>>>> error, please reply via email or call +64 21 >>>>>>>>>>>>>>>> 0832 6348. >>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>> Mohammed Naser >>>>>>>>>>>>>>>> >>>>>>>>> VEXXHOST, Inc. >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>> Mohammed Naser >>>>>>>>>>>>>>>> >>>>>>> VEXXHOST, Inc. >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>> Cheers & Best regards, >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>> >>>>>>> Feilong Wang (王飞龙) (he/him) >>>>>>>>>>>>>>>> >>>>>>> Head of Research & Development >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> Catalyst Cloud >>>>>>>>>>>>>>>> >>>>>>> Aotearoa's own >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> Mob: +64 21 0832 6348 | www.catalystcloud.nz >>>>>>>>>>>>>>>> >>>>>>> Level 6, 150 Willis Street, Wellington 6011, New >>>>>>>>>>>>>>>> Zealand >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> CONFIDENTIALITY NOTICE: This email is intended for >>>>>>>>>>>>>>>> the named recipients only. >>>>>>>>>>>>>>>> >>>>>>> It may contain privileged, confidential or >>>>>>>>>>>>>>>> copyright information. If you are >>>>>>>>>>>>>>>> >>>>>>> not the named recipient, any use, reliance upon, >>>>>>>>>>>>>>>> disclosure or copying of this >>>>>>>>>>>>>>>> >>>>>>> email or its attachments is unauthorised. If you >>>>>>>>>>>>>>>> have received this email in >>>>>>>>>>>>>>>> >>>>>>> error, please reply via email or call +64 21 0832 >>>>>>>>>>>>>>>> 6348. >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> -- >>>>>>>>>>>>>>>> >>>>> Regards, >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> Syed Ammad Ali >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> -- >>>>>>>>>>>>>>>> >>> Regards, >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> Syed Ammad Ali >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > -- >>>>>>>>>>>>>>>> > Regards, >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Syed Ammad Ali >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Mohammed Naser >>>>>>>>>>>>>>>> VEXXHOST, Inc. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Syed Ammad Ali >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Syed Ammad Ali >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> >>>>>>>>> Syed Ammad Ali >>>>>>>>> >>>>>>>> >>>>>> <29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd.log> >>>>>> >>>>>> <6fca39b1-8eda-4786-8424-e5b04434cce7-k8s-cluster3-dcu52bgzpbuu-kube_cluster_config-aht4it6p7wfk.log> >>>>>> >>>>>> >>>> >>>> -- >>>> Regards, >>>> >>>> >>>> Syed Ammad Ali >>>> >>> -- >> Regards, >> >> >> Syed Ammad Ali >> > -- > Cheers & Best regards, > ------------------------------------------------------------------------------ > Feilong Wang (王飞龙) (he/him) > Head of Research & Development > > Catalyst Cloud > Aotearoa's own > > Mob: +64 21 0832 6348 | www.catalystcloud.nz > Level 6, 150 Willis Street, Wellington 6011, New Zealand > > CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. > It may contain privileged, confidential or copyright information. If you are > not the named recipient, any use, reliance upon, disclosure or copying of this > email or its attachments is unauthorised. If you have received this email in > error, please reply via email or call +64 21 0832 6348. > ------------------------------------------------------------------------------ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From feilong at catalyst.net.nz Wed Sep 1 02:04:40 2021 From: feilong at catalyst.net.nz (feilong) Date: Wed, 1 Sep 2021 14:04:40 +1200 Subject: Wallaby Magnum Issue In-Reply-To: References: Message-ID: Hi Karera, Can you please share all the log under /var/log/heat-config/heat-config-script ? Or you can jump in #openstack-containers channel on OFTC, I'm online now. On 1/09/21 1:51 pm, Karera Tony wrote: > Hey Feilong, > > Thanks a lot. > > The services are fine and indeed the log files are there in the > directory [/var/log/heat-config/heat-config-script] > > After checking, the master log is fine but the cluster log has this > error below as I had mentioned earlier > > Starting to run kube-apiserver-to-kubelet-role > + echo 'Waiting for Kubernetes API...' > Waiting for Kubernetes API... > ++ kubectl get --raw=/healthz > The connection to the server localhost:8080 was refused - did you > specify the right host or port? > + '[' ok = '' ']' > > > > Regards > > Tony Karera > > > > > On Tue, Aug 31, 2021 at 9:52 PM feilong > wrote: > > Hi Karea, > > Given you can see heat-container-agent container from podman which > means you should be able to see logs from below path: > > [root at k8s-100-eh6s5l6d73ie-master-0 heat-config-script]# ls > c64e6ac2-db7e-4786-a387-1d45359812b8-k8s-100-eh6s5l6d73ie-kube_cluster_config-uxpsylgnayjy.log > fa1f6247-51a8-4e70-befa-cbc61ee99e59-k8s-100-eh6s5l6d73ie-kube_masters-kmi423lgbjw3-0-oii7uzemq7aj-master_config-dhfam54i456j.log > [root at k8s-100-eh6s5l6d73ie-master-0 heat-config-script]# pwd > /var/log/heat-config/heat-config-script > > If you can not see the path and the log, then it means the > heat-container-agent didn't work well. You need to check the > service status by systemctl command and check the log by > journalctl. From there, you should be able to see why the cluster > failed. > > > > On 1/09/21 1:41 am, Karera Tony wrote: >> Dear Ammad, >> >> Sorry to bother you again but I have failed to get the right >> command to use to check. >> >> Every Kubectl command I run on either the master or worker. >> The connection to the server localhost:8080 was refused - did you >> specify the right host or port? >> I get the error below. >> >> >> Regards >> >> Tony Karera >> >> >> >> >> On Fri, Aug 27, 2021 at 9:15 AM Ammad Syed > > wrote: >> >> Your hyperkube services are not started. >> >> You need to check hyperkube services. >> >> Ammad >> >> On Fri, Aug 27, 2021 at 10:35 AM Karera Tony >> > wrote: >> >> Dear Ammad, >> >> Below is the output of podman ps >> >> CONTAINER ID  IMAGE                                       >>                      COMMAND               CREATED       >> STATUS           PORTS       NAMES >> 319fbebc2f50 >>  docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1 >> >>  /usr/bin/start-he...  23 hours ago  Up 23 hours ago     >>          heat-container-agent >> [root at k8s-cluster-2-4faiphvzsmzu-master-0 core]# >> >> >> Regards >> >> Tony Karera >> >> >> >> >> On Thu, Aug 26, 2021 at 9:54 AM Ammad Syed >> > wrote: >> >> The output in >> logfile 29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd >> is incomplete. >> >> There should be the installation and configuration of >> many other things that are missing. Also it looks >> that hyperkube is not installed. >> >> Can you check the response of "podman ps" command on >> master nodes.  >> >> Ammad  >> >> On Thu, Aug 26, 2021 at 11:30 AM Karera Tony >> > >> wrote: >> >> Here is the beginning of the Log >> >> Starting to run kube-apiserver-to-kubelet-role >> + echo 'Waiting for Kubernetes API...' >> Waiting for Kubernetes API... >> ++ kubectl get --raw=/healthz >> The connection to the server localhost:8080 was >> refused - did you specify the right host or port? >> + '[' ok = '' ']' >> >> >> Regards >> >> Tony Karera >> >> >> >> >> On Thu, Aug 26, 2021 at 7:53 AM Bharat Kunwar >> > > wrote: >> >> I assume these are from the master nodes? Can >> you share the logs shortly after creation >> rather than when it times out? I think there >> is some missing logs from the top. >> >> Sent from my iPhone >> >>> On 26 Aug 2021, at 06:14, Karera Tony >>> >> > wrote: >>> >>>  >>> Hello Guys, >>> >>> Attached are the two logs from the >>> /var/log/heat-config/heat-config-script >>> directory >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Thu, Aug 26, 2021 at 5:59 AM Karera Tony >>> >> > wrote: >>> >>> Dear Sir, >>> >>> You are right. >>> >>> I am getting this error >>> >>> kubectl get --raw=/healthz >>> The connection to the server >>> localhost:8080 was refused - did you >>> specify the right host or port? >>> >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Wed, Aug 25, 2021 at 10:55 PM Bharat >>> Kunwar >> > wrote: >>> >>> I’d check the logs under >>> /var/log/heat-config. >>> >>> Sent from my iPhone >>> >>>> On 25 Aug 2021, at 19:39, Karera >>>> Tony >>> > wrote: >>>> >>>>  >>>> DeaR Ammad, >>>> >>>> I was able to make the >>>> communication work and the Worker >>>> nodes were created as well but the >>>> cluster failed. >>>> >>>> I logged in to the master node and >>>> there was no error but below are >>>> the error when I run systemctl >>>> status heat-container-agent on the >>>> worker noed. >>>> >>>> Aug 25 17:52:24 >>>> cluster1-fmkpva3nozf7-node-0 >>>> podman[2268]: >>>> /var/lib/os-collect-config/local-data >>>> not found. Skipping >>>> Aug 25 17:52:55 >>>> cluster1-fmkpva3nozf7-node-0 >>>> podman[2268]: >>>> /var/lib/os-collect-config/local-data >>>> not found. Skipping >>>> Aug 25 17:53:26 >>>> cluster1-fmkpva3nozf7-node-0 >>>> podman[2268]: >>>> /var/lib/os-collect-config/local-data >>>> not found. Skipping >>>> Aug 25 17:53:57 >>>> cluster1-fmkpva3nozf7-node-0 >>>> podman[2268]: >>>> /var/lib/os-collect-config/local-data >>>> not found. Skipping >>>> Aug 25 17:54:28 >>>> cluster1-fmkpva3nozf7-node-0 >>>> podman[2268]: >>>> /var/lib/os-collect-config/local-data >>>> not found. Skipping >>>> Aug 25 17:54:59 >>>> cluster1-fmkpva3nozf7-node-0 >>>> podman[2268]: >>>> /var/lib/os-collect-config/local-data >>>> not found. Skipping >>>> Aug 25 17:55:29 >>>> cluster1-fmkpva3nozf7-node-0 >>>> podman[2268]: >>>> /var/lib/os-collect-config/local-data >>>> not found. Skipping >>>> Aug 25 17:56:00 >>>> cluster1-fmkpva3nozf7-node-0 >>>> podman[2268]: >>>> /var/lib/os-collect-config/local-data >>>> not found. Skipping >>>> Aug 25 17:56:31 >>>> cluster1-fmkpva3nozf7-node-0 >>>> podman[2268]: >>>> /var/lib/os-collect-config/local-data >>>> not found. Skipping >>>> Aug 25 17:57:02 >>>> cluster1-fmkpva3nozf7-node-0 >>>> podman[2268]: >>>> /var/lib/os-collect-config/local-data >>>> not found. Skipping >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Wed, Aug 25, 2021 at 10:38 AM >>>> Ammad Syed >>> > wrote: >>>> >>>> Yes, keystone, Heat, Barbicane >>>> and magnum public endpoints >>>> must be reachable from master >>>> and worker nodes. >>>> >>>> You can use below guide for the >>>> reference as well. >>>> >>>> https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=11 >>>> >>>> Ammad >>>> >>>> On Wed, Aug 25, 2021 at 12:08 >>>> PM Karera Tony >>>> >>> > >>>> wrote: >>>> >>>> Hello Ammad, >>>> >>>> I have deployed using the >>>> given image but I think >>>> there is an issue with >>>> keystone as per the screen >>>> shot below when I checked >>>> the master node's >>>> heat-container-agent status >>>> >>>> >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Wed, Aug 25, 2021 at >>>> 8:28 AM Karera Tony >>>> >>> > >>>> wrote: >>>> >>>> Hello Ammad, >>>> >>>> I actually first used >>>> that one and it was >>>> also getting stuck. >>>> >>>> I will try this one >>>> again and update you >>>> with the Logs though. >>>> >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Wed, Aug 25, 2021 at >>>> 8:25 AM Ammad Syed >>>> >>> > >>>> wrote: >>>> >>>> It seems from the >>>> logs that you are >>>> using fedora >>>> atomic. Can you try >>>> with FCOS 32 image. >>>> >>>> https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20201004.3.0/x86_64/fedora-coreos-32.20201004.3.0-openstack.x86_64.qcow2.xz >>>> >>>> Ammad >>>> >>>> On Wed, Aug 25, >>>> 2021 at 11:20 AM >>>> Karera Tony >>>> >>> > >>>> wrote: >>>> >>>> Hello Sir, >>>> >>>> Attached is the >>>> Log file >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Wed, Aug 25, >>>> 2021 at 7:31 AM >>>> Ammad Syed >>>> >>> > >>>> wrote: >>>> >>>> Hi Karera, >>>> >>>> Can you >>>> share us >>>> the full >>>> log file. >>>> >>>> Ammad >>>> >>>> On Wed, Aug >>>> 25, 2021 at >>>> 9:42 AM >>>> Karera Tony >>>> >>> > >>>> wrote: >>>> >>>> Hello Guys, >>>> >>>> Thanks >>>> a lot >>>> for the >>>> help >>>> but >>>> unfortunately >>>> I dont >>>> see >>>> much >>>> information >>>> in the >>>> log >>>> file >>>> indicating >>>> a >>>> failure >>>> apart >>>> from >>>> the log >>>> that >>>> keeps >>>> appearing. >>>> >>>> >>>> >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Tue, >>>> Aug 24, >>>> 2021 at >>>> 8:12 PM >>>> Mohammed >>>> Naser >>>> >>> > >>>> wrote: >>>> >>>> Also >>>> check >>>> out >>>> /var/log/cloud-init.log >>>> :) >>>> >>>> On >>>> Tue, >>>> Aug >>>> 24, >>>> 2021 >>>> at >>>> 1:39 >>>> PM >>>> Ammad >>>> Syed >>>> >>> > >>>> wrote: >>>> > >>>> > >>>> Then >>>> check >>>> journalctl >>>> -xe >>>> or >>>> status >>>> of >>>> heat >>>> agent >>>> service >>>> status. >>>> > >>>> > >>>> > Ammad >>>> > >>>> On >>>> Tue, >>>> Aug >>>> 24, >>>> 2021 >>>> at >>>> 10:36 >>>> PM >>>> Karera >>>> Tony >>>> >>> > >>>> wrote: >>>> >> >>>> >> >>>> Hello >>>> Ammad, >>>> >> >>>> >> >>>> There >>>> is >>>> no >>>> directory >>>> or >>>> log >>>> relevant >>>> to >>>> heat >>>> in >>>> the >>>> /var/log >>>> directory >>>> >> >>>> >> >>>> Regards >>>> >> >>>> >> >>>> Tony >>>> Karera >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On >>>> Tue, >>>> Aug >>>> 24, >>>> 2021 >>>> at >>>> 12:43 >>>> PM >>>> Ammad >>>> Syed >>>> >>> > >>>> wrote: >>>> >>> >>>> >>> >>>> Hi >>>> Karera, >>>> >>> >>>> >>> >>>> Login >>>> to >>>> master >>>> node >>>> and >>>> check >>>> the >>>> logs >>>> of >>>> heat >>>> agent >>>> in >>>> var >>>> log. >>>> There >>>> must >>>> be >>>> something >>>> the >>>> cluster >>>> is >>>> stucking >>>> somewhere >>>> in >>>> creating. >>>> >>> >>>> >>> >>>> Ammad >>>> >>> >>>> >>> >>>> On >>>> Tue, >>>> Aug >>>> 24, >>>> 2021 >>>> at >>>> 3:41 >>>> PM >>>> Karera >>>> Tony >>>> >>> > >>>> wrote: >>>> >>>> >>>> >>>> >>>> Hello >>>> Ammad, >>>> >>>> >>>> >>>> >>>> I >>>> had >>>> done >>>> as >>>> explained >>>> and >>>> it >>>> worked >>>> upto >>>> a >>>> certain >>>> point. >>>> The >>>> master >>>> node >>>> was >>>> created >>>> but >>>> the >>>> cluster >>>> remained >>>> in >>>> Creation >>>> in >>>> progress >>>> for >>>> over >>>> an >>>> hour >>>> and >>>> failed >>>> with >>>> error >>>> below >>>> >>>> >>>> >>>> >>>> Stack >>>> Faults >>>> >>>> >>>> as >>>> follows: >>>> >>>> >>>> default-master >>>> >>>> >>>> Timed >>>> out >>>> >>>> >>>> default-worker >>>> >>>> >>>> Timed >>>> out >>>> >>>> >>>> >>>> >>>> >>>> >>>> Regards >>>> >>>> >>>> >>>> >>>> Tony >>>> Karera >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On >>>> Tue, >>>> Aug >>>> 24, >>>> 2021 >>>> at >>>> 9:25 >>>> AM >>>> Ammad >>>> Syed >>>> >>> > >>>> wrote: >>>> >>>>> >>>> >>>>> >>>> Hi >>>> Tony, >>>> >>>>> >>>> >>>>> >>>> You >>>> can >>>> try >>>> by >>>> creating >>>> your >>>> private >>>> vxlan >>>> network >>>> prior >>>> to >>>> deployment >>>> of >>>> cluster >>>> and >>>> explicitly >>>> create >>>> your >>>> cluster >>>> in >>>> vxlan >>>> network. >>>> >>>>> >>>> >>>>> >>>> --fixed-network >>>> private >>>> --fixed-subnet >>>> private-subnet >>>> >>>>> >>>> >>>>> >>>> You >>>> can >>>> specify >>>> above >>>> while >>>> creating >>>> a >>>> cluster. >>>> >>>>> >>>> >>>>> >>>> Ammad >>>> >>>>> >>>> >>>>> >>>> On >>>> Tue, >>>> Aug >>>> 24, >>>> 2021 >>>> at >>>> 11:59 >>>> AM >>>> Karera >>>> Tony >>>> >>> > >>>> wrote: >>>> >>>>>> >>>> >>>>>> >>>> Hello >>>> MOhamed, >>>> >>>>>> >>>> >>>>>> >>>> I >>>> think >>>> the >>>> Kubernetes >>>> cluster >>>> is >>>> ok >>>> but >>>> it >>>> when >>>> I >>>> deploy >>>> it, >>>> It >>>> creates >>>> a >>>> fixed >>>> network >>>> using >>>> vlan >>>> which >>>> I >>>> am >>>> not >>>> using >>>> for >>>> internal >>>> networks. >>>> >>>>>> >>>> >>>>>> >>>> When >>>> I >>>> create >>>> a a >>>> vxlan >>>> Network >>>> and >>>> use >>>> it >>>> in >>>> the >>>> cluster >>>> creation, >>>> It >>>> fails. >>>> Is >>>> there >>>> a >>>> trick >>>> around >>>> this ? >>>> >>>>>> >>>> Regards >>>> >>>>>> >>>> >>>>>> >>>> Tony >>>> Karera >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> On >>>> Fri, >>>> Aug >>>> 20, >>>> 2021 >>>> at >>>> 9:00 >>>> AM >>>> feilong >>>> >>> > >>>> wrote: >>>> >>>>>>> >>>> >>>>>>> >>>> Oooh, >>>> are >>>> you >>>> using >>>> Swarm? >>>> I >>>> don't >>>> think >>>> that >>>> driver >>>> is >>>> well >>>> maintained. >>>> I >>>> didn't >>>> see >>>> any >>>> interest >>>> in >>>> the >>>> last >>>> 4 >>>> years >>>> since >>>> I >>>> involved >>>> in >>>> the >>>> Magnum >>>> project. >>>> If >>>> there >>>> is >>>> no >>>> specific >>>> reason, >>>> I >>>> would >>>> suggest >>>> go >>>> for >>>> k8s. >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> On >>>> 20/08/21 >>>> 5:08 >>>> pm, >>>> Mohammed >>>> Naser >>>> wrote: >>>> >>>>>>> >>>> >>>>>>> >>>> Please >>>> keep >>>> replies >>>> on >>>> list >>>> so >>>> others >>>> can >>>> help >>>> too. >>>> >>>>>>> >>>> >>>>>>> >>>> I >>>> don’t >>>> know >>>> how >>>> well >>>> tested >>>> the >>>> Swarm >>>> driver >>>> is >>>> at >>>> this >>>> point. >>>> I >>>> believe >>>> most >>>> Magnum >>>> users >>>> are >>>> using >>>> it >>>> for >>>> Kubernetes >>>> only. >>>> >>>>>>> >>>> >>>>>>> >>>> On >>>> Fri, >>>> Aug >>>> 20, >>>> 2021 >>>> at >>>> 1:05 >>>> AM >>>> Karera >>>> Tony >>>> >>> > >>>> wrote: >>>> >>>>>>>> >>>> >>>>>>>> >>>> Hello >>>> Naser, >>>> >>>>>>>> >>>> >>>>>>>> >>>> Please >>>> check >>>> below. >>>> >>>>>>>> >>>> >>>>>>>> >>>> openstack >>>> coe >>>> cluster >>>> template >>>> create >>>> swarm-cluster-template1 >>>> \ >>>> >>>>>>>>  >>>>     >>>>     >>>>     >>>>     >>>>     >>>> --image >>>> fedora-atomic-latest >>>> \ >>>> >>>>>>>>  >>>>     >>>>     >>>>     >>>>     >>>>     >>>> --external-network >>>> External_1700\ >>>> >>>>>>>>  >>>>     >>>>     >>>>     >>>>     >>>>     >>>> --dns-nameserver >>>> 8.8.8.8 >>>> \ >>>> >>>>>>>>  >>>>     >>>>     >>>>     >>>>     >>>>     >>>> --master-flavor >>>> m1.small >>>> \ >>>> >>>>>>>>  >>>>     >>>>     >>>>     >>>>     >>>>     >>>> --flavor >>>> m1.small >>>> \ >>>> >>>>>>>>  >>>>     >>>>     >>>>     >>>>     >>>>     >>>> --coe >>>> swarm >>>> >>>>>>>> >>>> openstack >>>> coe >>>> cluster >>>> create >>>> swarm-cluster >>>> \ >>>> >>>>>>>>  >>>>     >>>>     >>>>     >>>>     >>>>     >>>>   >>>>  --cluster-template >>>> swarm-cluster-template >>>> \ >>>> >>>>>>>>  >>>>     >>>>     >>>>     >>>>     >>>>     >>>>   >>>>  --master-count >>>> 1 \ >>>> >>>>>>>>  >>>>     >>>>     >>>>     >>>>     >>>>     >>>>   >>>>  --node-count >>>> 2 \ >>>> >>>>>>>>  >>>>     >>>>     >>>>     >>>>     >>>>     >>>>   >>>>  --keypair >>>> Newkey >>>> >>>>>>>> >>>> >>>>>>>> >>>> Regards >>>> >>>>>>>> >>>> >>>>>>>> >>>> Tony >>>> Karera >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> On >>>> Fri, >>>> Aug >>>> 20, >>>> 2021 >>>> at >>>> 7:03 >>>> AM >>>> Mohammed >>>> Naser >>>> >>> > >>>> wrote: >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> What >>>> does >>>> your >>>> cluster >>>> template >>>> and >>>> cluster >>>> create >>>> command >>>> look >>>> like? >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> On >>>> Fri, >>>> Aug >>>> 20, >>>> 2021 >>>> at >>>> 12:59 >>>> AM >>>> Karera >>>> Tony >>>> >>> > >>>> wrote: >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> Hello >>>> Wang, >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> Thanks >>>> for >>>> the >>>> feedback. >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> Unfortunately >>>> Octavia >>>> is >>>> not >>>> deployed >>>> in >>>> my >>>> environment >>>> (at >>>> least >>>> not >>>> yet) >>>> and >>>> LB >>>> is >>>> not >>>> enabled >>>> on >>>> either >>>> the >>>> cluster >>>> template >>>> or >>>> the >>>> cluster >>>> itself. >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> Regards >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> Tony >>>> Karera >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> On >>>> Fri, >>>> Aug >>>> 20, >>>> 2021 >>>> at >>>> 6:52 >>>> AM >>>> feilong >>>> >>> > >>>> wrote: >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Hi >>>> Karera, >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> It's >>>> probably >>>> a >>>> bug. >>>> If >>>> you >>>> do >>>> have >>>> Octavia >>>> deployed, >>>> can >>>> you >>>> try >>>> to >>>> not >>>> disable >>>> the >>>> LB >>>> and >>>> see >>>> how >>>> it >>>> goes? >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> On >>>> 20/08/21 >>>> 4:18 >>>> pm, >>>> Karera >>>> Tony >>>> wrote: >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Hello >>>> Team, >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> I >>>> deployed >>>> Openstack >>>> Wallby >>>> on >>>> Ubuntu20 >>>> and >>>> enabled >>>> Magum, >>>> however >>>> when >>>> I >>>> create >>>> a >>>> cluster >>>> I >>>> get >>>> the >>>> error >>>> below. >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Status >>>> Reason >>>> >>>>>>>>>>> >>>> ERROR: >>>> Property >>>> error: >>>> : >>>> resources.api_lb.properties: >>>> : >>>> Property >>>> allowed_cidrs >>>> not >>>> assigned >>>> >>>>>>>>>>> >>>> Can >>>> someone >>>> advise >>>> on >>>> where >>>> I >>>> could >>>> be >>>> wrong. >>>> Btw, >>>> I >>>> disabled >>>> load >>>> balancer >>>> while >>>> creating >>>> the >>>> cluster. >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Regards >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Tony >>>> Karera >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> -- >>>> >>>>>>>>>>> >>>> Cheers >>>> & >>>> Best >>>> regards, >>>> >>>>>>>>>>> >>>> ------------------------------------------------------------------------------ >>>> >>>>>>>>>>> >>>> Feilong >>>> Wang >>>> (王飞龙) >>>> (he/him) >>>> >>>>>>>>>>> >>>> Head >>>> of >>>> Research >>>> & >>>> Development >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Catalyst >>>> Cloud >>>> >>>>>>>>>>> >>>> Aotearoa's >>>> own >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Mob: >>>> +64 >>>> 21 >>>> 0832 >>>> 6348 >>>> | >>>> www.catalystcloud.nz >>>> >>>> >>>>>>>>>>> >>>> Level >>>> 6, >>>> 150 >>>> Willis >>>> Street, >>>> Wellington >>>> 6011, >>>> New >>>> Zealand >>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> CONFIDENTIALITY >>>> NOTICE: >>>> This >>>> email >>>> is >>>> intended >>>> for >>>> the >>>> named >>>> recipients >>>> only. >>>> >>>>>>>>>>> >>>> It >>>> may >>>> contain >>>> privileged, >>>> confidential >>>> or >>>> copyright >>>> information. >>>> If >>>> you are >>>> >>>>>>>>>>> >>>> not >>>> the >>>> named >>>> recipient, >>>> any >>>> use, >>>> reliance >>>> upon, >>>> disclosure >>>> or >>>> copying >>>> of this >>>> >>>>>>>>>>> >>>> email >>>> or >>>> its >>>> attachments >>>> is >>>> unauthorised. >>>> If >>>> you >>>> have >>>> received >>>> this >>>> email >>>> in >>>> >>>>>>>>>>> >>>> error, >>>> please >>>> reply >>>> via >>>> email >>>> or >>>> call >>>> +64 >>>> 21 >>>> 0832 >>>> 6348. >>>> >>>>>>>>>>> >>>> ------------------------------------------------------------------------------ >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> -- >>>> >>>>>>>>> >>>> Mohammed >>>> Naser >>>> >>>>>>>>> >>>> VEXXHOST, >>>> Inc. >>>> >>>>>>> >>>> >>>>>>> >>>> -- >>>> >>>>>>> >>>> Mohammed >>>> Naser >>>> >>>>>>> >>>> VEXXHOST, >>>> Inc. >>>> >>>>>>> >>>> >>>>>>> >>>> -- >>>> >>>>>>> >>>> Cheers >>>> & >>>> Best >>>> regards, >>>> >>>>>>> >>>> ------------------------------------------------------------------------------ >>>> >>>>>>> >>>> Feilong >>>> Wang >>>> (王飞龙) >>>> (he/him) >>>> >>>>>>> >>>> Head >>>> of >>>> Research >>>> & >>>> Development >>>> >>>>>>> >>>> >>>>>>> >>>> Catalyst >>>> Cloud >>>> >>>>>>> >>>> Aotearoa's >>>> own >>>> >>>>>>> >>>> >>>>>>> >>>> Mob: >>>> +64 >>>> 21 >>>> 0832 >>>> 6348 >>>> | >>>> www.catalystcloud.nz >>>> >>>> >>>>>>> >>>> Level >>>> 6, >>>> 150 >>>> Willis >>>> Street, >>>> Wellington >>>> 6011, >>>> New >>>> Zealand >>>> >>>> >>>>>>> >>>> >>>>>>> >>>> CONFIDENTIALITY >>>> NOTICE: >>>> This >>>> email >>>> is >>>> intended >>>> for >>>> the >>>> named >>>> recipients >>>> only. >>>> >>>>>>> >>>> It >>>> may >>>> contain >>>> privileged, >>>> confidential >>>> or >>>> copyright >>>> information. >>>> If >>>> you are >>>> >>>>>>> >>>> not >>>> the >>>> named >>>> recipient, >>>> any >>>> use, >>>> reliance >>>> upon, >>>> disclosure >>>> or >>>> copying >>>> of this >>>> >>>>>>> >>>> email >>>> or >>>> its >>>> attachments >>>> is >>>> unauthorised. >>>> If >>>> you >>>> have >>>> received >>>> this >>>> email >>>> in >>>> >>>>>>> >>>> error, >>>> please >>>> reply >>>> via >>>> email >>>> or >>>> call >>>> +64 >>>> 21 >>>> 0832 >>>> 6348. >>>> >>>>>>> >>>> ------------------------------------------------------------------------------ >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> -- >>>> >>>>> >>>> Regards, >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> Syed >>>> Ammad >>>> Ali >>>> >>> >>>> >>> >>>> >>> >>>> >>> -- >>>> >>> >>>> Regards, >>>> >>> >>>> >>> >>>> >>> >>>> Syed >>>> Ammad >>>> Ali >>>> > >>>> > -- >>>> > >>>> Regards, >>>> > >>>> > >>>> > >>>> Syed >>>> Ammad >>>> Ali >>>> >>>> >>>> >>>> -- >>>> Mohammed >>>> Naser >>>> VEXXHOST, >>>> Inc. >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> >>>> Syed Ammad Ali >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> >>>> Syed Ammad Ali >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> >>>> Syed Ammad Ali >>>> >>> <29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd.log> >>> <6fca39b1-8eda-4786-8424-e5b04434cce7-k8s-cluster3-dcu52bgzpbuu-kube_cluster_config-aht4it6p7wfk.log> >> >> >> >> -- >> Regards, >> >> >> Syed Ammad Ali >> >> -- >> Regards, >> >> >> Syed Ammad Ali >> > -- > Cheers & Best regards, > ------------------------------------------------------------------------------ > Feilong Wang (王飞龙) (he/him) > Head of Research & Development > > Catalyst Cloud > Aotearoa's own > > Mob: +64 21 0832 6348 | www.catalystcloud.nz > Level 6, 150 Willis Street, Wellington 6011, New Zealand > > CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. > It may contain privileged, confidential or copyright information. If you are > not the named recipient, any use, reliance upon, disclosure or copying of this > email or its attachments is unauthorised. If you have received this email in > error, please reply via email or call +64 21 0832 6348. > ------------------------------------------------------------------------------ > -- Cheers & Best regards, ------------------------------------------------------------------------------ Feilong Wang (王飞龙) (he/him) Head of Research & Development Catalyst Cloud Aotearoa's own Mob: +64 21 0832 6348 | www.catalystcloud.nz Level 6, 150 Willis Street, Wellington 6011, New Zealand CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipient, any use, reliance upon, disclosure or copying of this email or its attachments is unauthorised. If you have received this email in error, please reply via email or call +64 21 0832 6348. ------------------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 1 02:11:54 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 1 Sep 2021 04:11:54 +0200 Subject: Wallaby Magnum Issue In-Reply-To: References: Message-ID: Hello Feilong, I would be happy to join. Can you please assist with the link . Thanks a lot Regards Tony Karera On Wed, Sep 1, 2021 at 4:04 AM feilong wrote: > Hi Karera, > > Can you please share all the log under /var/log/heat-config/heat-config-script > ? Or you can jump in #openstack-containers channel on OFTC, I'm online now. > > > > On 1/09/21 1:51 pm, Karera Tony wrote: > > Hey Feilong, > > Thanks a lot. > > The services are fine and indeed the log files are there in the directory [ > /var/log/heat-config/heat-config-script] > > After checking, the master log is fine but the cluster log has this error > below as I had mentioned earlier > > Starting to run kube-apiserver-to-kubelet-role > + echo 'Waiting for Kubernetes API...' > Waiting for Kubernetes API... > ++ kubectl get --raw=/healthz > The connection to the server localhost:8080 was refused - did you specify > the right host or port? > + '[' ok = '' ']' > > > > Regards > > Tony Karera > > > > > On Tue, Aug 31, 2021 at 9:52 PM feilong wrote: > >> Hi Karea, >> >> Given you can see heat-container-agent container from podman which means >> you should be able to see logs from below path: >> >> [root at k8s-100-eh6s5l6d73ie-master-0 heat-config-script]# ls >> >> c64e6ac2-db7e-4786-a387-1d45359812b8-k8s-100-eh6s5l6d73ie-kube_cluster_config-uxpsylgnayjy.log >> >> fa1f6247-51a8-4e70-befa-cbc61ee99e59-k8s-100-eh6s5l6d73ie-kube_masters-kmi423lgbjw3-0-oii7uzemq7aj-master_config-dhfam54i456j.log >> [root at k8s-100-eh6s5l6d73ie-master-0 heat-config-script]# pwd >> /var/log/heat-config/heat-config-script >> >> If you can not see the path and the log, then it means the >> heat-container-agent didn't work well. You need to check the service status >> by systemctl command and check the log by journalctl. From there, you >> should be able to see why the cluster failed. >> >> >> >> On 1/09/21 1:41 am, Karera Tony wrote: >> >> Dear Ammad, >> >> Sorry to bother you again but I have failed to get the right command to >> use to check. >> >> Every Kubectl command I run on either the master or worker. >> The connection to the server localhost:8080 was refused - did you specify >> the right host or port? >> I get the error below. >> >> >> Regards >> >> Tony Karera >> >> >> >> >> On Fri, Aug 27, 2021 at 9:15 AM Ammad Syed wrote: >> >>> Your hyperkube services are not started. >>> >>> You need to check hyperkube services. >>> >>> Ammad >>> >>> On Fri, Aug 27, 2021 at 10:35 AM Karera Tony >>> wrote: >>> >>>> Dear Ammad, >>>> >>>> Below is the output of podman ps >>>> >>>> CONTAINER ID IMAGE >>>> COMMAND CREATED STATUS PORTS >>>> NAMES >>>> 319fbebc2f50 >>>> docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1 >>>> /usr/bin/start-he... 23 hours ago Up 23 hours ago >>>> heat-container-agent >>>> [root at k8s-cluster-2-4faiphvzsmzu-master-0 core]# >>>> >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Thu, Aug 26, 2021 at 9:54 AM Ammad Syed >>>> wrote: >>>> >>>>> The output in >>>>> logfile 29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd >>>>> is incomplete. >>>>> >>>>> There should be the installation and configuration of many other >>>>> things that are missing. Also it looks that hyperkube is not installed. >>>>> >>>>> Can you check the response of "podman ps" command on master nodes. >>>>> >>>>> Ammad >>>>> >>>>> On Thu, Aug 26, 2021 at 11:30 AM Karera Tony >>>>> wrote: >>>>> >>>>>> Here is the beginning of the Log >>>>>> >>>>>> Starting to run kube-apiserver-to-kubelet-role >>>>>> + echo 'Waiting for Kubernetes API...' >>>>>> Waiting for Kubernetes API... >>>>>> ++ kubectl get --raw=/healthz >>>>>> The connection to the server localhost:8080 was refused - did you >>>>>> specify the right host or port? >>>>>> + '[' ok = '' ']' >>>>>> >>>>>> >>>>>> Regards >>>>>> >>>>>> Tony Karera >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Aug 26, 2021 at 7:53 AM Bharat Kunwar >>>>>> wrote: >>>>>> >>>>>>> I assume these are from the master nodes? Can you share the logs >>>>>>> shortly after creation rather than when it times out? I think there is some >>>>>>> missing logs from the top. >>>>>>> >>>>>>> Sent from my iPhone >>>>>>> >>>>>>> On 26 Aug 2021, at 06:14, Karera Tony wrote: >>>>>>> >>>>>>>  >>>>>>> Hello Guys, >>>>>>> >>>>>>> Attached are the two logs from the >>>>>>> /var/log/heat-config/heat-config-script directory >>>>>>> Regards >>>>>>> >>>>>>> Tony Karera >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 26, 2021 at 5:59 AM Karera Tony >>>>>>> wrote: >>>>>>> >>>>>>>> Dear Sir, >>>>>>>> >>>>>>>> You are right. >>>>>>>> >>>>>>>> I am getting this error >>>>>>>> >>>>>>>> kubectl get --raw=/healthz >>>>>>>> The connection to the server localhost:8080 was refused - did you >>>>>>>> specify the right host or port? >>>>>>>> >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> Tony Karera >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Aug 25, 2021 at 10:55 PM Bharat Kunwar >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I’d check the logs under /var/log/heat-config. >>>>>>>>> >>>>>>>>> Sent from my iPhone >>>>>>>>> >>>>>>>>> On 25 Aug 2021, at 19:39, Karera Tony >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>  >>>>>>>>> DeaR Ammad, >>>>>>>>> >>>>>>>>> I was able to make the communication work and the Worker nodes >>>>>>>>> were created as well but the cluster failed. >>>>>>>>> >>>>>>>>> I logged in to the master node and there was no error but below >>>>>>>>> are the error when I run systemctl status heat-container-agent on the >>>>>>>>> worker noed. >>>>>>>>> >>>>>>>>> Aug 25 17:52:24 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>> Aug 25 17:52:55 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>> Aug 25 17:53:26 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>> Aug 25 17:53:57 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>> Aug 25 17:54:28 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>> Aug 25 17:54:59 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>> Aug 25 17:55:29 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>> Aug 25 17:56:00 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>> Aug 25 17:56:31 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>> Aug 25 17:57:02 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>> Regards >>>>>>>>> >>>>>>>>> Tony Karera >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Aug 25, 2021 at 10:38 AM Ammad Syed >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Yes, keystone, Heat, Barbicane and magnum public endpoints must >>>>>>>>>> be reachable from master and worker nodes. >>>>>>>>>> >>>>>>>>>> You can use below guide for the reference as well. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=11 >>>>>>>>>> >>>>>>>>>> Ammad >>>>>>>>>> >>>>>>>>>> On Wed, Aug 25, 2021 at 12:08 PM Karera Tony < >>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hello Ammad, >>>>>>>>>>> >>>>>>>>>>> I have deployed using the given image but I think there is an >>>>>>>>>>> issue with keystone as per the screen shot below when I checked the master >>>>>>>>>>> node's heat-container-agent status >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards >>>>>>>>>>> >>>>>>>>>>> Tony Karera >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Aug 25, 2021 at 8:28 AM Karera Tony < >>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello Ammad, >>>>>>>>>>>> >>>>>>>>>>>> I actually first used that one and it was also getting stuck. >>>>>>>>>>>> >>>>>>>>>>>> I will try this one again and update you with the Logs though. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards >>>>>>>>>>>> >>>>>>>>>>>> Tony Karera >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Aug 25, 2021 at 8:25 AM Ammad Syed < >>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> It seems from the logs that you are using fedora atomic. Can >>>>>>>>>>>>> you try with FCOS 32 image. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20201004.3.0/x86_64/fedora-coreos-32.20201004.3.0-openstack.x86_64.qcow2.xz >>>>>>>>>>>>> >>>>>>>>>>>>> Ammad >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 25, 2021 at 11:20 AM Karera Tony < >>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hello Sir, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Attached is the Log file >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards >>>>>>>>>>>>>> >>>>>>>>>>>>>> Tony Karera >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Aug 25, 2021 at 7:31 AM Ammad Syed < >>>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Karera, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can you share us the full log file. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ammad >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Aug 25, 2021 at 9:42 AM Karera Tony < >>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hello Guys, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks a lot for the help but unfortunately I dont see much >>>>>>>>>>>>>>>> information in the log file indicating a failure apart from the log that >>>>>>>>>>>>>>>> keeps appearing. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Tony Karera >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Aug 24, 2021 at 8:12 PM Mohammed Naser < >>>>>>>>>>>>>>>> mnaser at vexxhost.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Also check out /var/log/cloud-init.log :) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Aug 24, 2021 at 1:39 PM Ammad Syed < >>>>>>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Then check journalctl -xe or status of heat agent >>>>>>>>>>>>>>>>> service status. >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Ammad >>>>>>>>>>>>>>>>> > On Tue, Aug 24, 2021 at 10:36 PM Karera Tony < >>>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> Hello Ammad, >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> There is no directory or log relevant to heat in the >>>>>>>>>>>>>>>>> /var/log directory >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> Regards >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> Tony Karera >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> On Tue, Aug 24, 2021 at 12:43 PM Ammad Syed < >>>>>>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> Hi Karera, >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> Login to master node and check the logs of heat agent >>>>>>>>>>>>>>>>> in var log. There must be something the cluster is stucking somewhere in >>>>>>>>>>>>>>>>> creating. >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> Ammad >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> On Tue, Aug 24, 2021 at 3:41 PM Karera Tony < >>>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>> Hello Ammad, >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>> I had done as explained and it worked upto a certain >>>>>>>>>>>>>>>>> point. The master node was created but the cluster remained in Creation in >>>>>>>>>>>>>>>>> progress for over an hour and failed with error below >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>> Stack Faults >>>>>>>>>>>>>>>>> >>>> as follows: >>>>>>>>>>>>>>>>> >>>> default-master >>>>>>>>>>>>>>>>> >>>> Timed out >>>>>>>>>>>>>>>>> >>>> default-worker >>>>>>>>>>>>>>>>> >>>> Timed out >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>> Regards >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>> Tony Karera >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> >>>> On Tue, Aug 24, 2021 at 9:25 AM Ammad Syed < >>>>>>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> Hi Tony, >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> You can try by creating your private vxlan network >>>>>>>>>>>>>>>>> prior to deployment of cluster and explicitly create your cluster in vxlan >>>>>>>>>>>>>>>>> network. >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> --fixed-network private --fixed-subnet private-subnet >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> You can specify above while creating a cluster. >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> Ammad >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> On Tue, Aug 24, 2021 at 11:59 AM Karera Tony < >>>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> >>>>>> Hello MOhamed, >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> >>>>>> I think the Kubernetes cluster is ok but it when I >>>>>>>>>>>>>>>>> deploy it, It creates a fixed network using vlan which I am not using for >>>>>>>>>>>>>>>>> internal networks. >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> >>>>>> When I create a a vxlan Network and use it in the >>>>>>>>>>>>>>>>> cluster creation, It fails. Is there a trick around this ? >>>>>>>>>>>>>>>>> >>>>>> Regards >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> >>>>>> Tony Karera >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> >>>>>> On Fri, Aug 20, 2021 at 9:00 AM feilong < >>>>>>>>>>>>>>>>> feilong at catalyst.net.nz> wrote: >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> Oooh, are you using Swarm? I don't think that >>>>>>>>>>>>>>>>> driver is well maintained. I didn't see any interest in the last 4 years >>>>>>>>>>>>>>>>> since I involved in the Magnum project. If there is no specific reason, I >>>>>>>>>>>>>>>>> would suggest go for k8s. >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> On 20/08/21 5:08 pm, Mohammed Naser wrote: >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> Please keep replies on list so others can help too. >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> I don’t know how well tested the Swarm driver is >>>>>>>>>>>>>>>>> at this point. I believe most Magnum users are using it for Kubernetes only. >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> On Fri, Aug 20, 2021 at 1:05 AM Karera Tony < >>>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> Hello Naser, >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> Please check below. >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> openstack coe cluster template create >>>>>>>>>>>>>>>>> swarm-cluster-template1 \ >>>>>>>>>>>>>>>>> >>>>>>>> --image fedora-atomic-latest >>>>>>>>>>>>>>>>> \ >>>>>>>>>>>>>>>>> >>>>>>>> --external-network >>>>>>>>>>>>>>>>> External_1700\ >>>>>>>>>>>>>>>>> >>>>>>>> --dns-nameserver 8.8.8.8 \ >>>>>>>>>>>>>>>>> >>>>>>>> --master-flavor m1.small \ >>>>>>>>>>>>>>>>> >>>>>>>> --flavor m1.small \ >>>>>>>>>>>>>>>>> >>>>>>>> --coe swarm >>>>>>>>>>>>>>>>> >>>>>>>> openstack coe cluster create swarm-cluster \ >>>>>>>>>>>>>>>>> >>>>>>>> --cluster-template >>>>>>>>>>>>>>>>> swarm-cluster-template \ >>>>>>>>>>>>>>>>> >>>>>>>> --master-count 1 \ >>>>>>>>>>>>>>>>> >>>>>>>> --node-count 2 \ >>>>>>>>>>>>>>>>> >>>>>>>> --keypair Newkey >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> Regards >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> Tony Karera >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> On Fri, Aug 20, 2021 at 7:03 AM Mohammed Naser < >>>>>>>>>>>>>>>>> mnaser at vexxhost.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>> What does your cluster template and cluster >>>>>>>>>>>>>>>>> create command look like? >>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>> On Fri, Aug 20, 2021 at 12:59 AM Karera Tony < >>>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>> Hello Wang, >>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks for the feedback. >>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>> Unfortunately Octavia is not deployed in my >>>>>>>>>>>>>>>>> environment (at least not yet) and LB is not enabled on either the cluster >>>>>>>>>>>>>>>>> template or the cluster itself. >>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>> Tony Karera >>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>> On Fri, Aug 20, 2021 at 6:52 AM feilong < >>>>>>>>>>>>>>>>> feilong at catalyst.net.nz> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> Hi Karera, >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> It's probably a bug. If you do have Octavia >>>>>>>>>>>>>>>>> deployed, can you try to not disable the LB and see how it goes? >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> On 20/08/21 4:18 pm, Karera Tony wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> Hello Team, >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> I deployed Openstack Wallby on Ubuntu20 and >>>>>>>>>>>>>>>>> enabled Magum, however when I create a cluster I get the error below. >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> Status Reason >>>>>>>>>>>>>>>>> >>>>>>>>>>> ERROR: Property error: : >>>>>>>>>>>>>>>>> resources.api_lb.properties: : Property allowed_cidrs not assigned >>>>>>>>>>>>>>>>> >>>>>>>>>>> Can someone advise on where I could be wrong. >>>>>>>>>>>>>>>>> Btw, I disabled load balancer while creating the cluster. >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> Regards >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> Tony Karera >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>> Cheers & Best regards, >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>>> >>>>>>>>>>> Feilong Wang (王飞龙) (he/him) >>>>>>>>>>>>>>>>> >>>>>>>>>>> Head of Research & Development >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> Catalyst Cloud >>>>>>>>>>>>>>>>> >>>>>>>>>>> Aotearoa's own >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> Mob: +64 21 0832 6348 | www.catalystcloud.nz >>>>>>>>>>>>>>>>> >>>>>>>>>>> Level 6, 150 Willis Street, Wellington 6011, >>>>>>>>>>>>>>>>> New Zealand >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> CONFIDENTIALITY NOTICE: This email is intended >>>>>>>>>>>>>>>>> for the named recipients only. >>>>>>>>>>>>>>>>> >>>>>>>>>>> It may contain privileged, confidential or >>>>>>>>>>>>>>>>> copyright information. If you are >>>>>>>>>>>>>>>>> >>>>>>>>>>> not the named recipient, any use, reliance >>>>>>>>>>>>>>>>> upon, disclosure or copying of this >>>>>>>>>>>>>>>>> >>>>>>>>>>> email or its attachments is unauthorised. If >>>>>>>>>>>>>>>>> you have received this email in >>>>>>>>>>>>>>>>> >>>>>>>>>>> error, please reply via email or call +64 21 >>>>>>>>>>>>>>>>> 0832 6348. >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>> Mohammed Naser >>>>>>>>>>>>>>>>> >>>>>>>>> VEXXHOST, Inc. >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>> Mohammed Naser >>>>>>>>>>>>>>>>> >>>>>>> VEXXHOST, Inc. >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>> Cheers & Best regards, >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>>> >>>>>>> Feilong Wang (王飞龙) (he/him) >>>>>>>>>>>>>>>>> >>>>>>> Head of Research & Development >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> Catalyst Cloud >>>>>>>>>>>>>>>>> >>>>>>> Aotearoa's own >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> Mob: +64 21 0832 6348 | www.catalystcloud.nz >>>>>>>>>>>>>>>>> >>>>>>> Level 6, 150 Willis Street, Wellington 6011, New >>>>>>>>>>>>>>>>> Zealand >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> CONFIDENTIALITY NOTICE: This email is intended for >>>>>>>>>>>>>>>>> the named recipients only. >>>>>>>>>>>>>>>>> >>>>>>> It may contain privileged, confidential or >>>>>>>>>>>>>>>>> copyright information. If you are >>>>>>>>>>>>>>>>> >>>>>>> not the named recipient, any use, reliance upon, >>>>>>>>>>>>>>>>> disclosure or copying of this >>>>>>>>>>>>>>>>> >>>>>>> email or its attachments is unauthorised. If you >>>>>>>>>>>>>>>>> have received this email in >>>>>>>>>>>>>>>>> >>>>>>> error, please reply via email or call +64 21 0832 >>>>>>>>>>>>>>>>> 6348. >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> -- >>>>>>>>>>>>>>>>> >>>>> Regards, >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>> Syed Ammad Ali >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> -- >>>>>>>>>>>>>>>>> >>> Regards, >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> Syed Ammad Ali >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > -- >>>>>>>>>>>>>>>>> > Regards, >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Syed Ammad Ali >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Mohammed Naser >>>>>>>>>>>>>>>>> VEXXHOST, Inc. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Syed Ammad Ali >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Syed Ammad Ali >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Syed Ammad Ali >>>>>>>>>> >>>>>>>>> >>>>>>> <29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd.log> >>>>>>> >>>>>>> <6fca39b1-8eda-4786-8424-e5b04434cce7-k8s-cluster3-dcu52bgzpbuu-kube_cluster_config-aht4it6p7wfk.log> >>>>>>> >>>>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> >>>>> Syed Ammad Ali >>>>> >>>> -- >>> Regards, >>> >>> >>> Syed Ammad Ali >>> >> -- >> Cheers & Best regards, >> ------------------------------------------------------------------------------ >> Feilong Wang (王飞龙) (he/him) >> Head of Research & Development >> >> Catalyst Cloud >> Aotearoa's own >> >> Mob: +64 21 0832 6348 | www.catalystcloud.nz >> Level 6, 150 Willis Street, Wellington 6011, New Zealand >> >> CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. >> It may contain privileged, confidential or copyright information. If you are >> not the named recipient, any use, reliance upon, disclosure or copying of this >> email or its attachments is unauthorised. If you have received this email in >> error, please reply via email or call +64 21 0832 6348. >> ------------------------------------------------------------------------------ >> >> -- > Cheers & Best regards, > ------------------------------------------------------------------------------ > Feilong Wang (王飞龙) (he/him) > Head of Research & Development > > Catalyst Cloud > Aotearoa's own > > Mob: +64 21 0832 6348 | www.catalystcloud.nz > Level 6, 150 Willis Street, Wellington 6011, New Zealand > > CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. > It may contain privileged, confidential or copyright information. If you are > not the named recipient, any use, reliance upon, disclosure or copying of this > email or its attachments is unauthorised. If you have received this email in > error, please reply via email or call +64 21 0832 6348. > ------------------------------------------------------------------------------ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 1 02:27:11 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 1 Sep 2021 04:27:11 +0200 Subject: Wallaby Magnum Issue In-Reply-To: References: Message-ID: Attached are the logs Regards Tony Karera On Wed, Sep 1, 2021 at 4:11 AM Karera Tony wrote: > Hello Feilong, > > I would be happy to join. > > Can you please assist with the link . > > Thanks a lot > Regards > > Tony Karera > > > > > On Wed, Sep 1, 2021 at 4:04 AM feilong wrote: > >> Hi Karera, >> >> Can you please share all the log under /var/log/heat-config/heat-config-script >> ? Or you can jump in #openstack-containers channel on OFTC, I'm online now. >> >> >> >> On 1/09/21 1:51 pm, Karera Tony wrote: >> >> Hey Feilong, >> >> Thanks a lot. >> >> The services are fine and indeed the log files are there in the directory >> [/var/log/heat-config/heat-config-script] >> >> After checking, the master log is fine but the cluster log has this error >> below as I had mentioned earlier >> >> Starting to run kube-apiserver-to-kubelet-role >> + echo 'Waiting for Kubernetes API...' >> Waiting for Kubernetes API... >> ++ kubectl get --raw=/healthz >> The connection to the server localhost:8080 was refused - did you specify >> the right host or port? >> + '[' ok = '' ']' >> >> >> >> Regards >> >> Tony Karera >> >> >> >> >> On Tue, Aug 31, 2021 at 9:52 PM feilong wrote: >> >>> Hi Karea, >>> >>> Given you can see heat-container-agent container from podman which means >>> you should be able to see logs from below path: >>> >>> [root at k8s-100-eh6s5l6d73ie-master-0 heat-config-script]# ls >>> >>> c64e6ac2-db7e-4786-a387-1d45359812b8-k8s-100-eh6s5l6d73ie-kube_cluster_config-uxpsylgnayjy.log >>> >>> fa1f6247-51a8-4e70-befa-cbc61ee99e59-k8s-100-eh6s5l6d73ie-kube_masters-kmi423lgbjw3-0-oii7uzemq7aj-master_config-dhfam54i456j.log >>> [root at k8s-100-eh6s5l6d73ie-master-0 heat-config-script]# pwd >>> /var/log/heat-config/heat-config-script >>> >>> If you can not see the path and the log, then it means the >>> heat-container-agent didn't work well. You need to check the service status >>> by systemctl command and check the log by journalctl. From there, you >>> should be able to see why the cluster failed. >>> >>> >>> >>> On 1/09/21 1:41 am, Karera Tony wrote: >>> >>> Dear Ammad, >>> >>> Sorry to bother you again but I have failed to get the right command to >>> use to check. >>> >>> Every Kubectl command I run on either the master or worker. >>> The connection to the server localhost:8080 was refused - did you >>> specify the right host or port? >>> I get the error below. >>> >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Fri, Aug 27, 2021 at 9:15 AM Ammad Syed >>> wrote: >>> >>>> Your hyperkube services are not started. >>>> >>>> You need to check hyperkube services. >>>> >>>> Ammad >>>> >>>> On Fri, Aug 27, 2021 at 10:35 AM Karera Tony >>>> wrote: >>>> >>>>> Dear Ammad, >>>>> >>>>> Below is the output of podman ps >>>>> >>>>> CONTAINER ID IMAGE >>>>> COMMAND CREATED STATUS PORTS >>>>> NAMES >>>>> 319fbebc2f50 >>>>> docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1 >>>>> /usr/bin/start-he... 23 hours ago Up 23 hours ago >>>>> heat-container-agent >>>>> [root at k8s-cluster-2-4faiphvzsmzu-master-0 core]# >>>>> >>>>> >>>>> Regards >>>>> >>>>> Tony Karera >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Aug 26, 2021 at 9:54 AM Ammad Syed >>>>> wrote: >>>>> >>>>>> The output in >>>>>> logfile 29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd >>>>>> is incomplete. >>>>>> >>>>>> There should be the installation and configuration of many other >>>>>> things that are missing. Also it looks that hyperkube is not installed. >>>>>> >>>>>> Can you check the response of "podman ps" command on master nodes. >>>>>> >>>>>> Ammad >>>>>> >>>>>> On Thu, Aug 26, 2021 at 11:30 AM Karera Tony >>>>>> wrote: >>>>>> >>>>>>> Here is the beginning of the Log >>>>>>> >>>>>>> Starting to run kube-apiserver-to-kubelet-role >>>>>>> + echo 'Waiting for Kubernetes API...' >>>>>>> Waiting for Kubernetes API... >>>>>>> ++ kubectl get --raw=/healthz >>>>>>> The connection to the server localhost:8080 was refused - did you >>>>>>> specify the right host or port? >>>>>>> + '[' ok = '' ']' >>>>>>> >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> Tony Karera >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 26, 2021 at 7:53 AM Bharat Kunwar >>>>>>> wrote: >>>>>>> >>>>>>>> I assume these are from the master nodes? Can you share the logs >>>>>>>> shortly after creation rather than when it times out? I think there is some >>>>>>>> missing logs from the top. >>>>>>>> >>>>>>>> Sent from my iPhone >>>>>>>> >>>>>>>> On 26 Aug 2021, at 06:14, Karera Tony wrote: >>>>>>>> >>>>>>>>  >>>>>>>> Hello Guys, >>>>>>>> >>>>>>>> Attached are the two logs from the >>>>>>>> /var/log/heat-config/heat-config-script directory >>>>>>>> Regards >>>>>>>> >>>>>>>> Tony Karera >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Aug 26, 2021 at 5:59 AM Karera Tony >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Dear Sir, >>>>>>>>> >>>>>>>>> You are right. >>>>>>>>> >>>>>>>>> I am getting this error >>>>>>>>> >>>>>>>>> kubectl get --raw=/healthz >>>>>>>>> The connection to the server localhost:8080 was refused - did you >>>>>>>>> specify the right host or port? >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> >>>>>>>>> Tony Karera >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Aug 25, 2021 at 10:55 PM Bharat Kunwar < >>>>>>>>> bharat at stackhpc.com> wrote: >>>>>>>>> >>>>>>>>>> I’d check the logs under /var/log/heat-config. >>>>>>>>>> >>>>>>>>>> Sent from my iPhone >>>>>>>>>> >>>>>>>>>> On 25 Aug 2021, at 19:39, Karera Tony >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>  >>>>>>>>>> DeaR Ammad, >>>>>>>>>> >>>>>>>>>> I was able to make the communication work and the Worker nodes >>>>>>>>>> were created as well but the cluster failed. >>>>>>>>>> >>>>>>>>>> I logged in to the master node and there was no error but below >>>>>>>>>> are the error when I run systemctl status heat-container-agent on the >>>>>>>>>> worker noed. >>>>>>>>>> >>>>>>>>>> Aug 25 17:52:24 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>>> Aug 25 17:52:55 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>>> Aug 25 17:53:26 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>>> Aug 25 17:53:57 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>>> Aug 25 17:54:28 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>>> Aug 25 17:54:59 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>>> Aug 25 17:55:29 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>>> Aug 25 17:56:00 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>>> Aug 25 17:56:31 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>>> Aug 25 17:57:02 cluster1-fmkpva3nozf7-node-0 podman[2268]: >>>>>>>>>> /var/lib/os-collect-config/local-data not found. Skipping >>>>>>>>>> Regards >>>>>>>>>> >>>>>>>>>> Tony Karera >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Aug 25, 2021 at 10:38 AM Ammad Syed < >>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Yes, keystone, Heat, Barbicane and magnum public endpoints must >>>>>>>>>>> be reachable from master and worker nodes. >>>>>>>>>>> >>>>>>>>>>> You can use below guide for the reference as well. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=11 >>>>>>>>>>> >>>>>>>>>>> Ammad >>>>>>>>>>> >>>>>>>>>>> On Wed, Aug 25, 2021 at 12:08 PM Karera Tony < >>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello Ammad, >>>>>>>>>>>> >>>>>>>>>>>> I have deployed using the given image but I think there is an >>>>>>>>>>>> issue with keystone as per the screen shot below when I checked the master >>>>>>>>>>>> node's heat-container-agent status >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards >>>>>>>>>>>> >>>>>>>>>>>> Tony Karera >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Aug 25, 2021 at 8:28 AM Karera Tony < >>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello Ammad, >>>>>>>>>>>>> >>>>>>>>>>>>> I actually first used that one and it was also getting stuck. >>>>>>>>>>>>> >>>>>>>>>>>>> I will try this one again and update you with the Logs though. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards >>>>>>>>>>>>> >>>>>>>>>>>>> Tony Karera >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 25, 2021 at 8:25 AM Ammad Syed < >>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> It seems from the logs that you are using fedora atomic. Can >>>>>>>>>>>>>> you try with FCOS 32 image. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20201004.3.0/x86_64/fedora-coreos-32.20201004.3.0-openstack.x86_64.qcow2.xz >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ammad >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Aug 25, 2021 at 11:20 AM Karera Tony < >>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello Sir, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Attached is the Log file >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Tony Karera >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Aug 25, 2021 at 7:31 AM Ammad Syed < >>>>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Karera, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can you share us the full log file. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ammad >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Aug 25, 2021 at 9:42 AM Karera Tony < >>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hello Guys, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks a lot for the help but unfortunately I dont see >>>>>>>>>>>>>>>>> much information in the log file indicating a failure apart from the log >>>>>>>>>>>>>>>>> that keeps appearing. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Tony Karera >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Aug 24, 2021 at 8:12 PM Mohammed Naser < >>>>>>>>>>>>>>>>> mnaser at vexxhost.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Also check out /var/log/cloud-init.log :) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, Aug 24, 2021 at 1:39 PM Ammad Syed < >>>>>>>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > Then check journalctl -xe or status of heat agent >>>>>>>>>>>>>>>>>> service status. >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > Ammad >>>>>>>>>>>>>>>>>> > On Tue, Aug 24, 2021 at 10:36 PM Karera Tony < >>>>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> Hello Ammad, >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> There is no directory or log relevant to heat in the >>>>>>>>>>>>>>>>>> /var/log directory >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> Regards >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> Tony Karera >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> On Tue, Aug 24, 2021 at 12:43 PM Ammad Syed < >>>>>>>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> Hi Karera, >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> Login to master node and check the logs of heat agent >>>>>>>>>>>>>>>>>> in var log. There must be something the cluster is stucking somewhere in >>>>>>>>>>>>>>>>>> creating. >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> Ammad >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> On Tue, Aug 24, 2021 at 3:41 PM Karera Tony < >>>>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> Hello Ammad, >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> I had done as explained and it worked upto a certain >>>>>>>>>>>>>>>>>> point. The master node was created but the cluster remained in Creation in >>>>>>>>>>>>>>>>>> progress for over an hour and failed with error below >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> Stack Faults >>>>>>>>>>>>>>>>>> >>>> as follows: >>>>>>>>>>>>>>>>>> >>>> default-master >>>>>>>>>>>>>>>>>> >>>> Timed out >>>>>>>>>>>>>>>>>> >>>> default-worker >>>>>>>>>>>>>>>>>> >>>> Timed out >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> Regards >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> Tony Karera >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> On Tue, Aug 24, 2021 at 9:25 AM Ammad Syed < >>>>>>>>>>>>>>>>>> syedammad83 at gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> Hi Tony, >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> You can try by creating your private vxlan network >>>>>>>>>>>>>>>>>> prior to deployment of cluster and explicitly create your cluster in vxlan >>>>>>>>>>>>>>>>>> network. >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> --fixed-network private --fixed-subnet >>>>>>>>>>>>>>>>>> private-subnet >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> You can specify above while creating a cluster. >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> Ammad >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> On Tue, Aug 24, 2021 at 11:59 AM Karera Tony < >>>>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> Hello MOhamed, >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> I think the Kubernetes cluster is ok but it when I >>>>>>>>>>>>>>>>>> deploy it, It creates a fixed network using vlan which I am not using for >>>>>>>>>>>>>>>>>> internal networks. >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> When I create a a vxlan Network and use it in the >>>>>>>>>>>>>>>>>> cluster creation, It fails. Is there a trick around this ? >>>>>>>>>>>>>>>>>> >>>>>> Regards >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> Tony Karera >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> On Fri, Aug 20, 2021 at 9:00 AM feilong < >>>>>>>>>>>>>>>>>> feilong at catalyst.net.nz> wrote: >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> Oooh, are you using Swarm? I don't think that >>>>>>>>>>>>>>>>>> driver is well maintained. I didn't see any interest in the last 4 years >>>>>>>>>>>>>>>>>> since I involved in the Magnum project. If there is no specific reason, I >>>>>>>>>>>>>>>>>> would suggest go for k8s. >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> On 20/08/21 5:08 pm, Mohammed Naser wrote: >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> Please keep replies on list so others can help >>>>>>>>>>>>>>>>>> too. >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> I don’t know how well tested the Swarm driver is >>>>>>>>>>>>>>>>>> at this point. I believe most Magnum users are using it for Kubernetes only. >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> On Fri, Aug 20, 2021 at 1:05 AM Karera Tony < >>>>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> Hello Naser, >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> Please check below. >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> openstack coe cluster template create >>>>>>>>>>>>>>>>>> swarm-cluster-template1 \ >>>>>>>>>>>>>>>>>> >>>>>>>> --image >>>>>>>>>>>>>>>>>> fedora-atomic-latest \ >>>>>>>>>>>>>>>>>> >>>>>>>> --external-network >>>>>>>>>>>>>>>>>> External_1700\ >>>>>>>>>>>>>>>>>> >>>>>>>> --dns-nameserver 8.8.8.8 \ >>>>>>>>>>>>>>>>>> >>>>>>>> --master-flavor m1.small \ >>>>>>>>>>>>>>>>>> >>>>>>>> --flavor m1.small \ >>>>>>>>>>>>>>>>>> >>>>>>>> --coe swarm >>>>>>>>>>>>>>>>>> >>>>>>>> openstack coe cluster create swarm-cluster \ >>>>>>>>>>>>>>>>>> >>>>>>>> --cluster-template >>>>>>>>>>>>>>>>>> swarm-cluster-template \ >>>>>>>>>>>>>>>>>> >>>>>>>> --master-count 1 \ >>>>>>>>>>>>>>>>>> >>>>>>>> --node-count 2 \ >>>>>>>>>>>>>>>>>> >>>>>>>> --keypair Newkey >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> Regards >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> Tony Karera >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> On Fri, Aug 20, 2021 at 7:03 AM Mohammed Naser < >>>>>>>>>>>>>>>>>> mnaser at vexxhost.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>> What does your cluster template and cluster >>>>>>>>>>>>>>>>>> create command look like? >>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>> On Fri, Aug 20, 2021 at 12:59 AM Karera Tony < >>>>>>>>>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>> Hello Wang, >>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks for the feedback. >>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>> Unfortunately Octavia is not deployed in my >>>>>>>>>>>>>>>>>> environment (at least not yet) and LB is not enabled on either the cluster >>>>>>>>>>>>>>>>>> template or the cluster itself. >>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>> Tony Karera >>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>> On Fri, Aug 20, 2021 at 6:52 AM feilong < >>>>>>>>>>>>>>>>>> feilong at catalyst.net.nz> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Hi Karera, >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> It's probably a bug. If you do have Octavia >>>>>>>>>>>>>>>>>> deployed, can you try to not disable the LB and see how it goes? >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> On 20/08/21 4:18 pm, Karera Tony wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Hello Team, >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> I deployed Openstack Wallby on Ubuntu20 and >>>>>>>>>>>>>>>>>> enabled Magum, however when I create a cluster I get the error below. >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Status Reason >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ERROR: Property error: : >>>>>>>>>>>>>>>>>> resources.api_lb.properties: : Property allowed_cidrs not assigned >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Can someone advise on where I could be wrong. >>>>>>>>>>>>>>>>>> Btw, I disabled load balancer while creating the cluster. >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Tony Karera >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Cheers & Best regards, >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Feilong Wang (王飞龙) (he/him) >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Head of Research & Development >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Catalyst Cloud >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Aotearoa's own >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Mob: +64 21 0832 6348 | www.catalystcloud.nz >>>>>>>>>>>>>>>>>> >>>>>>>>>>> Level 6, 150 Willis Street, Wellington 6011, >>>>>>>>>>>>>>>>>> New Zealand >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> CONFIDENTIALITY NOTICE: This email is >>>>>>>>>>>>>>>>>> intended for the named recipients only. >>>>>>>>>>>>>>>>>> >>>>>>>>>>> It may contain privileged, confidential or >>>>>>>>>>>>>>>>>> copyright information. If you are >>>>>>>>>>>>>>>>>> >>>>>>>>>>> not the named recipient, any use, reliance >>>>>>>>>>>>>>>>>> upon, disclosure or copying of this >>>>>>>>>>>>>>>>>> >>>>>>>>>>> email or its attachments is unauthorised. If >>>>>>>>>>>>>>>>>> you have received this email in >>>>>>>>>>>>>>>>>> >>>>>>>>>>> error, please reply via email or call +64 21 >>>>>>>>>>>>>>>>>> 0832 6348. >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>>>> Mohammed Naser >>>>>>>>>>>>>>>>>> >>>>>>>>> VEXXHOST, Inc. >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>> Mohammed Naser >>>>>>>>>>>>>>>>>> >>>>>>> VEXXHOST, Inc. >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>> Cheers & Best regards, >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>>>> >>>>>>> Feilong Wang (王飞龙) (he/him) >>>>>>>>>>>>>>>>>> >>>>>>> Head of Research & Development >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> Catalyst Cloud >>>>>>>>>>>>>>>>>> >>>>>>> Aotearoa's own >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> Mob: +64 21 0832 6348 | www.catalystcloud.nz >>>>>>>>>>>>>>>>>> >>>>>>> Level 6, 150 Willis Street, Wellington 6011, New >>>>>>>>>>>>>>>>>> Zealand >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> CONFIDENTIALITY NOTICE: This email is intended >>>>>>>>>>>>>>>>>> for the named recipients only. >>>>>>>>>>>>>>>>>> >>>>>>> It may contain privileged, confidential or >>>>>>>>>>>>>>>>>> copyright information. If you are >>>>>>>>>>>>>>>>>> >>>>>>> not the named recipient, any use, reliance upon, >>>>>>>>>>>>>>>>>> disclosure or copying of this >>>>>>>>>>>>>>>>>> >>>>>>> email or its attachments is unauthorised. If you >>>>>>>>>>>>>>>>>> have received this email in >>>>>>>>>>>>>>>>>> >>>>>>> error, please reply via email or call +64 21 0832 >>>>>>>>>>>>>>>>>> 6348. >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> -- >>>>>>>>>>>>>>>>>> >>>>> Regards, >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> Syed Ammad Ali >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> -- >>>>>>>>>>>>>>>>>> >>> Regards, >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> Syed Ammad Ali >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > -- >>>>>>>>>>>>>>>>>> > Regards, >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > Syed Ammad Ali >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Mohammed Naser >>>>>>>>>>>>>>>>>> VEXXHOST, Inc. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Syed Ammad Ali >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Syed Ammad Ali >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Syed Ammad Ali >>>>>>>>>>> >>>>>>>>>> >>>>>>>> <29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd.log> >>>>>>>> >>>>>>>> <6fca39b1-8eda-4786-8424-e5b04434cce7-k8s-cluster3-dcu52bgzpbuu-kube_cluster_config-aht4it6p7wfk.log> >>>>>>>> >>>>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> >>>>>> >>>>>> Syed Ammad Ali >>>>>> >>>>> -- >>>> Regards, >>>> >>>> >>>> Syed Ammad Ali >>>> >>> -- >>> Cheers & Best regards, >>> ------------------------------------------------------------------------------ >>> Feilong Wang (王飞龙) (he/him) >>> Head of Research & Development >>> >>> Catalyst Cloud >>> Aotearoa's own >>> >>> Mob: +64 21 0832 6348 | www.catalystcloud.nz >>> Level 6, 150 Willis Street, Wellington 6011, New Zealand >>> >>> CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. >>> It may contain privileged, confidential or copyright information. If you are >>> not the named recipient, any use, reliance upon, disclosure or copying of this >>> email or its attachments is unauthorised. If you have received this email in >>> error, please reply via email or call +64 21 0832 6348. >>> ------------------------------------------------------------------------------ >>> >>> -- >> Cheers & Best regards, >> ------------------------------------------------------------------------------ >> Feilong Wang (王飞龙) (he/him) >> Head of Research & Development >> >> Catalyst Cloud >> Aotearoa's own >> >> Mob: +64 21 0832 6348 | www.catalystcloud.nz >> Level 6, 150 Willis Street, Wellington 6011, New Zealand >> >> CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. >> It may contain privileged, confidential or copyright information. If you are >> not the named recipient, any use, reliance upon, disclosure or copying of this >> email or its attachments is unauthorised. If you have received this email in >> error, please reply via email or call +64 21 0832 6348. >> ------------------------------------------------------------------------------ >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd.log Type: application/octet-stream Size: 11011 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6fca39b1-8eda-4786-8424-e5b04434cce7-k8s-cluster3-dcu52bgzpbuu-kube_cluster_config-aht4it6p7wfk.log Type: application/octet-stream Size: 1614278 bytes Desc: not available URL: From feilong at catalyst.net.nz Wed Sep 1 03:30:34 2021 From: feilong at catalyst.net.nz (feilong) Date: Wed, 1 Sep 2021 15:30:34 +1200 Subject: Wallaby Magnum Issue In-Reply-To: References: Message-ID: <40105897-286f-f38e-a91a-ffc32da4fc70@catalyst.net.nz> Your cluster failed because of https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/make-cert.sh#L24 So please just remove your label "tls_enabled=true" and try again. On 1/09/21 2:27 pm, Karera Tony wrote: > Attached are the logs > Regards > > Tony Karera > > > > > On Wed, Sep 1, 2021 at 4:11 AM Karera Tony > wrote: > > Hello Feilong, > > I would be happy to join. > > Can you please assist with the link . > > Thanks a lot > Regards > > Tony Karera > > > > > On Wed, Sep 1, 2021 at 4:04 AM feilong > wrote: > > Hi Karera, > > Can you please share all the log under > /var/log/heat-config/heat-config-script ? Or you can jump in > #openstack-containers channel on OFTC, I'm online now. > > > > On 1/09/21 1:51 pm, Karera Tony wrote: >> Hey Feilong, >> >> Thanks a lot. >> >> The services are fine and indeed the log files are there in >> the directory [/var/log/heat-config/heat-config-script] >> >> After checking, the master log is fine but the cluster log >> has this error below as I had mentioned earlier >> >> Starting to run kube-apiserver-to-kubelet-role >> + echo 'Waiting for Kubernetes API...' >> Waiting for Kubernetes API... >> ++ kubectl get --raw=/healthz >> The connection to the server localhost:8080 was refused - did >> you specify the right host or port? >> + '[' ok = '' ']' >> >> >> >> Regards >> >> Tony Karera >> >> >> >> >> On Tue, Aug 31, 2021 at 9:52 PM feilong >> > wrote: >> >> Hi Karea, >> >> Given you can see heat-container-agent container from >> podman which means you should be able to see logs from >> below path: >> >> [root at k8s-100-eh6s5l6d73ie-master-0 heat-config-script]# ls >> c64e6ac2-db7e-4786-a387-1d45359812b8-k8s-100-eh6s5l6d73ie-kube_cluster_config-uxpsylgnayjy.log >> fa1f6247-51a8-4e70-befa-cbc61ee99e59-k8s-100-eh6s5l6d73ie-kube_masters-kmi423lgbjw3-0-oii7uzemq7aj-master_config-dhfam54i456j.log >> [root at k8s-100-eh6s5l6d73ie-master-0 heat-config-script]# pwd >> /var/log/heat-config/heat-config-script >> >> If you can not see the path and the log, then it means >> the heat-container-agent didn't work well. You need to >> check the service status by systemctl command and check >> the log by journalctl. From there, you should be able to >> see why the cluster failed. >> >> >> >> On 1/09/21 1:41 am, Karera Tony wrote: >>> Dear Ammad, >>> >>> Sorry to bother you again but I have failed to get the >>> right command to use to check. >>> >>> Every Kubectl command I run on either the master or worker. >>> The connection to the server localhost:8080 was refused >>> - did you specify the right host or port? >>> I get the error below. >>> >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Fri, Aug 27, 2021 at 9:15 AM Ammad Syed >>> > >>> wrote: >>> >>> Your hyperkube services are not started. >>> >>> You need to check hyperkube services. >>> >>> Ammad >>> >>> On Fri, Aug 27, 2021 at 10:35 AM Karera Tony >>> > >>> wrote: >>> >>> Dear Ammad, >>> >>> Below is the output of podman ps >>> >>> CONTAINER ID  IMAGE                             >>>                                COMMAND           >>>     CREATED       STATUS           PORTS       NAMES >>> 319fbebc2f50 >>>  docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1 >>> >>>  /usr/bin/start-he...  23 hours ago  Up 23 hours >>> ago              heat-container-agent >>> [root at k8s-cluster-2-4faiphvzsmzu-master-0 core]# >>> >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Thu, Aug 26, 2021 at 9:54 AM Ammad Syed >>> >> > wrote: >>> >>> The output in >>> logfile 29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd >>> is incomplete. >>> >>> There should be the installation and >>> configuration of many other things that are >>> missing. Also it looks that hyperkube is not >>> installed. >>> >>> Can you check the response of "podman ps" >>> command on master nodes.  >>> >>> Ammad  >>> >>> On Thu, Aug 26, 2021 at 11:30 AM Karera Tony >>> >> > wrote: >>> >>> Here is the beginning of the Log >>> >>> Starting to run >>> kube-apiserver-to-kubelet-role >>> + echo 'Waiting for Kubernetes API...' >>> Waiting for Kubernetes API... >>> ++ kubectl get --raw=/healthz >>> The connection to the server >>> localhost:8080 was refused - did you >>> specify the right host or port? >>> + '[' ok = '' ']' >>> >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Thu, Aug 26, 2021 at 7:53 AM Bharat >>> Kunwar >> > wrote: >>> >>> I assume these are from the master >>> nodes? Can you share the logs >>> shortly after creation rather than >>> when it times out? I think there is >>> some missing logs from the top. >>> >>> Sent from my iPhone >>> >>>> On 26 Aug 2021, at 06:14, Karera >>>> Tony >>> > wrote: >>>> >>>>  >>>> Hello Guys, >>>> >>>> Attached are the two logs from the >>>> /var/log/heat-config/heat-config-script >>>> directory >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Thu, Aug 26, 2021 at 5:59 AM >>>> Karera Tony >>> > wrote: >>>> >>>> Dear Sir, >>>> >>>> You are right. >>>> >>>> I am getting this error >>>> >>>> kubectl get --raw=/healthz >>>> The connection to the server >>>> localhost:8080 was refused - >>>> did you specify the right host >>>> or port? >>>> >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Wed, Aug 25, 2021 at 10:55 >>>> PM Bharat Kunwar >>>> >>> > >>>> wrote: >>>> >>>> I’d check the logs under >>>> /var/log/heat-config. >>>> >>>> Sent from my iPhone >>>> >>>>> On 25 Aug 2021, at 19:39, >>>>> Karera Tony >>>>> >>>> > >>>>> wrote: >>>>> >>>>>  >>>>> DeaR Ammad, >>>>> >>>>> I was able to make the >>>>> communication work and the >>>>> Worker nodes were created >>>>> as well but the cluster >>>>> failed. >>>>> >>>>> I logged in to the master >>>>> node and there was no >>>>> error but below are the >>>>> error when I run systemctl >>>>> status >>>>> heat-container-agent on >>>>> the worker noed. >>>>> >>>>> Aug 25 17:52:24 >>>>> cluster1-fmkpva3nozf7-node-0 >>>>> podman[2268]: >>>>> /var/lib/os-collect-config/local-data >>>>> not found. Skipping >>>>> Aug 25 17:52:55 >>>>> cluster1-fmkpva3nozf7-node-0 >>>>> podman[2268]: >>>>> /var/lib/os-collect-config/local-data >>>>> not found. Skipping >>>>> Aug 25 17:53:26 >>>>> cluster1-fmkpva3nozf7-node-0 >>>>> podman[2268]: >>>>> /var/lib/os-collect-config/local-data >>>>> not found. Skipping >>>>> Aug 25 17:53:57 >>>>> cluster1-fmkpva3nozf7-node-0 >>>>> podman[2268]: >>>>> /var/lib/os-collect-config/local-data >>>>> not found. Skipping >>>>> Aug 25 17:54:28 >>>>> cluster1-fmkpva3nozf7-node-0 >>>>> podman[2268]: >>>>> /var/lib/os-collect-config/local-data >>>>> not found. Skipping >>>>> Aug 25 17:54:59 >>>>> cluster1-fmkpva3nozf7-node-0 >>>>> podman[2268]: >>>>> /var/lib/os-collect-config/local-data >>>>> not found. Skipping >>>>> Aug 25 17:55:29 >>>>> cluster1-fmkpva3nozf7-node-0 >>>>> podman[2268]: >>>>> /var/lib/os-collect-config/local-data >>>>> not found. Skipping >>>>> Aug 25 17:56:00 >>>>> cluster1-fmkpva3nozf7-node-0 >>>>> podman[2268]: >>>>> /var/lib/os-collect-config/local-data >>>>> not found. Skipping >>>>> Aug 25 17:56:31 >>>>> cluster1-fmkpva3nozf7-node-0 >>>>> podman[2268]: >>>>> /var/lib/os-collect-config/local-data >>>>> not found. Skipping >>>>> Aug 25 17:57:02 >>>>> cluster1-fmkpva3nozf7-node-0 >>>>> podman[2268]: >>>>> /var/lib/os-collect-config/local-data >>>>> not found. Skipping >>>>> Regards >>>>> >>>>> Tony Karera >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Aug 25, 2021 at >>>>> 10:38 AM Ammad Syed >>>>> >>>> > >>>>> wrote: >>>>> >>>>> Yes, keystone, Heat, >>>>> Barbicane and magnum >>>>> public endpoints must >>>>> be reachable from >>>>> master and worker nodes. >>>>> >>>>> You can use below >>>>> guide for the >>>>> reference as well. >>>>> >>>>> https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=11 >>>>> >>>>> Ammad >>>>> >>>>> On Wed, Aug 25, 2021 >>>>> at 12:08 PM Karera >>>>> Tony >>>>> >>>> > >>>>> wrote: >>>>> >>>>> Hello Ammad, >>>>> >>>>> I have deployed >>>>> using the given >>>>> image but I think >>>>> there is an issue >>>>> with keystone as >>>>> per the screen >>>>> shot below when I >>>>> checked the master >>>>> node's >>>>> heat-container-agent >>>>> status >>>>> >>>>> >>>>> >>>>> Regards >>>>> >>>>> Tony Karera >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Aug 25, >>>>> 2021 at 8:28 AM >>>>> Karera Tony >>>>> >>>> > >>>>> wrote: >>>>> >>>>> Hello Ammad, >>>>> >>>>> I actually >>>>> first used >>>>> that one and >>>>> it was also >>>>> getting stuck. >>>>> >>>>> I will try >>>>> this one again >>>>> and update you >>>>> with the Logs >>>>> though. >>>>> >>>>> >>>>> Regards >>>>> >>>>> Tony Karera >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Aug >>>>> 25, 2021 at >>>>> 8:25 AM Ammad >>>>> Syed >>>>> >>>> > >>>>> wrote: >>>>> >>>>> It seems >>>>> from the >>>>> logs that >>>>> you are >>>>> using >>>>> fedora >>>>> atomic. >>>>> Can you >>>>> try with >>>>> FCOS 32 >>>>> image. >>>>> >>>>> https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20201004.3.0/x86_64/fedora-coreos-32.20201004.3.0-openstack.x86_64.qcow2.xz >>>>> >>>>> Ammad >>>>> >>>>> On Wed, >>>>> Aug 25, >>>>> 2021 at >>>>> 11:20 AM >>>>> Karera >>>>> Tony >>>>> >>>> > >>>>> wrote: >>>>> >>>>> Hello Sir, >>>>> >>>>> Attached >>>>> is the >>>>> Log file >>>>> >>>>> Regards >>>>> >>>>> Tony >>>>> Karera >>>>> >>>>> >>>>> >>>>> >>>>> On >>>>> Wed, >>>>> Aug >>>>> 25, >>>>> 2021 >>>>> at >>>>> 7:31 >>>>> AM >>>>> Ammad >>>>> Syed >>>>> >>>> > >>>>> wrote: >>>>> >>>>> Hi Karera, >>>>> >>>>> >>>>> Can >>>>> you >>>>> share >>>>> us >>>>> the >>>>> full >>>>> log >>>>> file. >>>>> >>>>> Ammad >>>>> >>>>> On >>>>> Wed, >>>>> Aug >>>>> 25, >>>>> 2021 >>>>> at >>>>> 9:42 >>>>> AM >>>>> Karera >>>>> Tony >>>>> >>>> > >>>>> wrote: >>>>> >>>>> Hello >>>>> Guys, >>>>> >>>>> Thanks >>>>> a >>>>> lot >>>>> for >>>>> the >>>>> help >>>>> but >>>>> unfortunately >>>>> I >>>>> dont >>>>> see >>>>> much >>>>> information >>>>> in >>>>> the >>>>> log >>>>> file >>>>> indicating >>>>> a >>>>> failure >>>>> apart >>>>> from >>>>> the >>>>> log >>>>> that >>>>> keeps >>>>> appearing. >>>>> >>>>> >>>>> >>>>> >>>>> Regards >>>>> >>>>> Tony >>>>> Karera >>>>> >>>>> >>>>> >>>>> >>>>> On >>>>> Tue, >>>>> Aug >>>>> 24, >>>>> 2021 >>>>> at >>>>> 8:12 >>>>> PM >>>>> Mohammed >>>>> Naser >>>>> >>>> > >>>>> wrote: >>>>> >>>>> Also >>>>> check >>>>> out >>>>> /var/log/cloud-init.log >>>>> :) >>>>> >>>>> On >>>>> Tue, >>>>> Aug >>>>> 24, >>>>> 2021 >>>>> at >>>>> 1:39 >>>>> PM >>>>> Ammad >>>>> Syed >>>>> >>>> > >>>>> wrote: >>>>> > >>>>> > >>>>> Then >>>>> check >>>>> journalctl >>>>> -xe >>>>> or >>>>> status >>>>> of >>>>> heat >>>>> agent >>>>> service >>>>> status. >>>>> > >>>>> > >>>>> > >>>>> Ammad >>>>> > >>>>> On >>>>> Tue, >>>>> Aug >>>>> 24, >>>>> 2021 >>>>> at >>>>> 10:36 >>>>> PM >>>>> Karera >>>>> Tony >>>>> >>>> > >>>>> wrote: >>>>> >> >>>>> >> >>>>> Hello >>>>> Ammad, >>>>> >> >>>>> >> >>>>> There >>>>> is >>>>> no >>>>> directory >>>>> or >>>>> log >>>>> relevant >>>>> to >>>>> heat >>>>> in >>>>> the >>>>> /var/log >>>>> directory >>>>> >> >>>>> >> >>>>> Regards >>>>> >> >>>>> >> >>>>> Tony >>>>> Karera >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> On >>>>> Tue, >>>>> Aug >>>>> 24, >>>>> 2021 >>>>> at >>>>> 12:43 >>>>> PM >>>>> Ammad >>>>> Syed >>>>> >>>> > >>>>> wrote: >>>>> >>> >>>>> >>> >>>>> Hi >>>>> Karera, >>>>> >>> >>>>> >>> >>>>> Login >>>>> to >>>>> master >>>>> node >>>>> and >>>>> check >>>>> the >>>>> logs >>>>> of >>>>> heat >>>>> agent >>>>> in >>>>> var >>>>> log. >>>>> There >>>>> must >>>>> be >>>>> something >>>>> the >>>>> cluster >>>>> is >>>>> stucking >>>>> somewhere >>>>> in >>>>> creating. >>>>> >>> >>>>> >>> >>>>> Ammad >>>>> >>> >>>>> >>> >>>>> On >>>>> Tue, >>>>> Aug >>>>> 24, >>>>> 2021 >>>>> at >>>>> 3:41 >>>>> PM >>>>> Karera >>>>> Tony >>>>> >>>> > >>>>> wrote: >>>>> >>>> >>>>> >>>> >>>>> Hello >>>>> Ammad, >>>>> >>>> >>>>> >>>> >>>>> I >>>>> had >>>>> done >>>>> as >>>>> explained >>>>> and >>>>> it >>>>> worked >>>>> upto >>>>> a >>>>> certain >>>>> point. >>>>> The >>>>> master >>>>> node >>>>> was >>>>> created >>>>> but >>>>> the >>>>> cluster >>>>> remained >>>>> in >>>>> Creation >>>>> in >>>>> progress >>>>> for >>>>> over >>>>> an >>>>> hour >>>>> and >>>>> failed >>>>> with >>>>> error >>>>> below >>>>> >>>> >>>>> >>>> >>>>> Stack >>>>> Faults >>>>> >>>> >>>>> as >>>>> follows: >>>>> >>>> >>>>> default-master >>>>> >>>> >>>>> Timed >>>>> out >>>>> >>>> >>>>> default-worker >>>>> >>>> >>>>> Timed >>>>> out >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> Regards >>>>> >>>> >>>>> >>>> >>>>> Tony >>>>> Karera >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> On >>>>> Tue, >>>>> Aug >>>>> 24, >>>>> 2021 >>>>> at >>>>> 9:25 >>>>> AM >>>>> Ammad >>>>> Syed >>>>> >>>> > >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>> Hi >>>>> Tony, >>>>> >>>>> >>>>> >>>>> >>>>> You >>>>> can >>>>> try >>>>> by >>>>> creating >>>>> your >>>>> private >>>>> vxlan >>>>> network >>>>> prior >>>>> to >>>>> deployment >>>>> of >>>>> cluster >>>>> and >>>>> explicitly >>>>> create >>>>> your >>>>> cluster >>>>> in >>>>> vxlan >>>>> network. >>>>> >>>>> >>>>> >>>>> >>>>> --fixed-network >>>>> private >>>>> --fixed-subnet >>>>> private-subnet >>>>> >>>>> >>>>> >>>>> >>>>> You >>>>> can >>>>> specify >>>>> above >>>>> while >>>>> creating >>>>> a >>>>> cluster. >>>>> >>>>> >>>>> >>>>> >>>>> Ammad >>>>> >>>>> >>>>> >>>>> >>>>> On >>>>> Tue, >>>>> Aug >>>>> 24, >>>>> 2021 >>>>> at >>>>> 11:59 >>>>> AM >>>>> Karera >>>>> Tony >>>>> >>>> > >>>>> wrote: >>>>> >>>>>> >>>>> >>>>>> >>>>> Hello >>>>> MOhamed, >>>>> >>>>>> >>>>> >>>>>> >>>>> I >>>>> think >>>>> the >>>>> Kubernetes >>>>> cluster >>>>> is >>>>> ok >>>>> but >>>>> it >>>>> when >>>>> I >>>>> deploy >>>>> it, >>>>> It >>>>> creates >>>>> a >>>>> fixed >>>>> network >>>>> using >>>>> vlan >>>>> which >>>>> I >>>>> am >>>>> not >>>>> using >>>>> for >>>>> internal >>>>> networks. >>>>> >>>>>> >>>>> >>>>>> >>>>> When >>>>> I >>>>> create >>>>> a >>>>> a >>>>> vxlan >>>>> Network >>>>> and >>>>> use >>>>> it >>>>> in >>>>> the >>>>> cluster >>>>> creation, >>>>> It >>>>> fails. >>>>> Is >>>>> there >>>>> a >>>>> trick >>>>> around >>>>> this >>>>> ? >>>>> >>>>>> >>>>> Regards >>>>> >>>>>> >>>>> >>>>>> >>>>> Tony >>>>> Karera >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> On >>>>> Fri, >>>>> Aug >>>>> 20, >>>>> 2021 >>>>> at >>>>> 9:00 >>>>> AM >>>>> feilong >>>>> >>>> > >>>>> wrote: >>>>> >>>>>>> >>>>> >>>>>>> >>>>> Oooh, >>>>> are >>>>> you >>>>> using >>>>> Swarm? >>>>> I >>>>> don't >>>>> think >>>>> that >>>>> driver >>>>> is >>>>> well >>>>> maintained. >>>>> I >>>>> didn't >>>>> see >>>>> any >>>>> interest >>>>> in >>>>> the >>>>> last >>>>> 4 >>>>> years >>>>> since >>>>> I >>>>> involved >>>>> in >>>>> the >>>>> Magnum >>>>> project. >>>>> If >>>>> there >>>>> is >>>>> no >>>>> specific >>>>> reason, >>>>> I >>>>> would >>>>> suggest >>>>> go >>>>> for >>>>> k8s. >>>>> >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> On >>>>> 20/08/21 >>>>> 5:08 >>>>> pm, >>>>> Mohammed >>>>> Naser >>>>> wrote: >>>>> >>>>>>> >>>>> >>>>>>> >>>>> Please >>>>> keep >>>>> replies >>>>> on >>>>> list >>>>> so >>>>> others >>>>> can >>>>> help >>>>> too. >>>>> >>>>>>> >>>>> >>>>>>> >>>>> I >>>>> don’t >>>>> know >>>>> how >>>>> well >>>>> tested >>>>> the >>>>> Swarm >>>>> driver >>>>> is >>>>> at >>>>> this >>>>> point. >>>>> I >>>>> believe >>>>> most >>>>> Magnum >>>>> users >>>>> are >>>>> using >>>>> it >>>>> for >>>>> Kubernetes >>>>> only. >>>>> >>>>>>> >>>>> >>>>>>> >>>>> On >>>>> Fri, >>>>> Aug >>>>> 20, >>>>> 2021 >>>>> at >>>>> 1:05 >>>>> AM >>>>> Karera >>>>> Tony >>>>> >>>> > >>>>> wrote: >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> Hello >>>>> Naser, >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> Please >>>>> check >>>>> below. >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> openstack >>>>> coe >>>>> cluster >>>>> template >>>>> create >>>>> swarm-cluster-template1 >>>>> \ >>>>> >>>>>>>>  >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>> --image >>>>> fedora-atomic-latest >>>>> \ >>>>> >>>>>>>>  >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>> --external-network >>>>> External_1700\ >>>>> >>>>>>>>  >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>> --dns-nameserver >>>>> 8.8.8.8 >>>>> \ >>>>> >>>>>>>>  >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>> --master-flavor >>>>> m1.small >>>>> \ >>>>> >>>>>>>>  >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>> --flavor >>>>> m1.small >>>>> \ >>>>> >>>>>>>>  >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>> --coe >>>>> swarm >>>>> >>>>>>>> >>>>> openstack >>>>> coe >>>>> cluster >>>>> create >>>>> swarm-cluster >>>>> \ >>>>> >>>>>>>>  >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>  --cluster-template >>>>> swarm-cluster-template >>>>> \ >>>>> >>>>>>>>  >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>  --master-count >>>>> 1 >>>>> \ >>>>> >>>>>>>>  >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>  --node-count >>>>> 2 >>>>> \ >>>>> >>>>>>>>  >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>   >>>>>  --keypair >>>>> Newkey >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> Regards >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> Tony >>>>> Karera >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> On >>>>> Fri, >>>>> Aug >>>>> 20, >>>>> 2021 >>>>> at >>>>> 7:03 >>>>> AM >>>>> Mohammed >>>>> Naser >>>>> >>>> > >>>>> wrote: >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> What >>>>> does >>>>> your >>>>> cluster >>>>> template >>>>> and >>>>> cluster >>>>> create >>>>> command >>>>> look >>>>> like? >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> On >>>>> Fri, >>>>> Aug >>>>> 20, >>>>> 2021 >>>>> at >>>>> 12:59 >>>>> AM >>>>> Karera >>>>> Tony >>>>> >>>> > >>>>> wrote: >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> Hello >>>>> Wang, >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> Thanks >>>>> for >>>>> the >>>>> feedback. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> Unfortunately >>>>> Octavia >>>>> is >>>>> not >>>>> deployed >>>>> in >>>>> my >>>>> environment >>>>> (at >>>>> least >>>>> not >>>>> yet) >>>>> and >>>>> LB >>>>> is >>>>> not >>>>> enabled >>>>> on >>>>> either >>>>> the >>>>> cluster >>>>> template >>>>> or >>>>> the >>>>> cluster >>>>> itself. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> Regards >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> Tony >>>>> Karera >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> On >>>>> Fri, >>>>> Aug >>>>> 20, >>>>> 2021 >>>>> at >>>>> 6:52 >>>>> AM >>>>> feilong >>>>> >>>> > >>>>> wrote: >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Hi >>>>> Karera, >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> It's >>>>> probably >>>>> a >>>>> bug. >>>>> If >>>>> you >>>>> do >>>>> have >>>>> Octavia >>>>> deployed, >>>>> can >>>>> you >>>>> try >>>>> to >>>>> not >>>>> disable >>>>> the >>>>> LB >>>>> and >>>>> see >>>>> how >>>>> it >>>>> goes? >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> On >>>>> 20/08/21 >>>>> 4:18 >>>>> pm, >>>>> Karera >>>>> Tony >>>>> wrote: >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Hello >>>>> Team, >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> I >>>>> deployed >>>>> Openstack >>>>> Wallby >>>>> on >>>>> Ubuntu20 >>>>> and >>>>> enabled >>>>> Magum, >>>>> however >>>>> when >>>>> I >>>>> create >>>>> a >>>>> cluster >>>>> I >>>>> get >>>>> the >>>>> error >>>>> below. >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Status >>>>> Reason >>>>> >>>>>>>>>>> >>>>> ERROR: >>>>> Property >>>>> error: >>>>> : >>>>> resources.api_lb.properties: >>>>> : >>>>> Property >>>>> allowed_cidrs >>>>> not >>>>> assigned >>>>> >>>>>>>>>>> >>>>> Can >>>>> someone >>>>> advise >>>>> on >>>>> where >>>>> I >>>>> could >>>>> be >>>>> wrong. >>>>> Btw, >>>>> I >>>>> disabled >>>>> load >>>>> balancer >>>>> while >>>>> creating >>>>> the >>>>> cluster. >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Regards >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Tony >>>>> Karera >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> -- >>>>> >>>>>>>>>>> >>>>> Cheers >>>>> & >>>>> Best >>>>> regards, >>>>> >>>>>>>>>>> >>>>> ------------------------------------------------------------------------------ >>>>> >>>>>>>>>>> >>>>> Feilong >>>>> Wang >>>>> (王飞龙) >>>>> (he/him) >>>>> >>>>>>>>>>> >>>>> Head >>>>> of >>>>> Research >>>>> & >>>>> Development >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Catalyst >>>>> Cloud >>>>> >>>>>>>>>>> >>>>> Aotearoa's >>>>> own >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Mob: >>>>> +64 >>>>> 21 >>>>> 0832 >>>>> 6348 >>>>> | >>>>> www.catalystcloud.nz >>>>> >>>>> >>>>>>>>>>> >>>>> Level >>>>> 6, >>>>> 150 >>>>> Willis >>>>> Street, >>>>> Wellington >>>>> 6011, >>>>> New >>>>> Zealand >>>>> >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> CONFIDENTIALITY >>>>> NOTICE: >>>>> This >>>>> email >>>>> is >>>>> intended >>>>> for >>>>> the >>>>> named >>>>> recipients >>>>> only. >>>>> >>>>>>>>>>> >>>>> It >>>>> may >>>>> contain >>>>> privileged, >>>>> confidential >>>>> or >>>>> copyright >>>>> information. >>>>> If >>>>> you >>>>> are >>>>> >>>>>>>>>>> >>>>> not >>>>> the >>>>> named >>>>> recipient, >>>>> any >>>>> use, >>>>> reliance >>>>> upon, >>>>> disclosure >>>>> or >>>>> copying >>>>> of >>>>> this >>>>> >>>>>>>>>>> >>>>> email >>>>> or >>>>> its >>>>> attachments >>>>> is >>>>> unauthorised. >>>>> If >>>>> you >>>>> have >>>>> received >>>>> this >>>>> email >>>>> in >>>>> >>>>>>>>>>> >>>>> error, >>>>> please >>>>> reply >>>>> via >>>>> email >>>>> or >>>>> call >>>>> +64 >>>>> 21 >>>>> 0832 >>>>> 6348. >>>>> >>>>>>>>>>> >>>>> ------------------------------------------------------------------------------ >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> -- >>>>> >>>>>>>>> >>>>> Mohammed >>>>> Naser >>>>> >>>>>>>>> >>>>> VEXXHOST, >>>>> Inc. >>>>> >>>>>>> >>>>> >>>>>>> >>>>> -- >>>>> >>>>>>> >>>>> Mohammed >>>>> Naser >>>>> >>>>>>> >>>>> VEXXHOST, >>>>> Inc. >>>>> >>>>>>> >>>>> >>>>>>> >>>>> -- >>>>> >>>>>>> >>>>> Cheers >>>>> & >>>>> Best >>>>> regards, >>>>> >>>>>>> >>>>> ------------------------------------------------------------------------------ >>>>> >>>>>>> >>>>> Feilong >>>>> Wang >>>>> (王飞龙) >>>>> (he/him) >>>>> >>>>>>> >>>>> Head >>>>> of >>>>> Research >>>>> & >>>>> Development >>>>> >>>>>>> >>>>> >>>>>>> >>>>> Catalyst >>>>> Cloud >>>>> >>>>>>> >>>>> Aotearoa's >>>>> own >>>>> >>>>>>> >>>>> >>>>>>> >>>>> Mob: >>>>> +64 >>>>> 21 >>>>> 0832 >>>>> 6348 >>>>> | >>>>> www.catalystcloud.nz >>>>> >>>>> >>>>>>> >>>>> Level >>>>> 6, >>>>> 150 >>>>> Willis >>>>> Street, >>>>> Wellington >>>>> 6011, >>>>> New >>>>> Zealand >>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> CONFIDENTIALITY >>>>> NOTICE: >>>>> This >>>>> email >>>>> is >>>>> intended >>>>> for >>>>> the >>>>> named >>>>> recipients >>>>> only. >>>>> >>>>>>> >>>>> It >>>>> may >>>>> contain >>>>> privileged, >>>>> confidential >>>>> or >>>>> copyright >>>>> information. >>>>> If >>>>> you >>>>> are >>>>> >>>>>>> >>>>> not >>>>> the >>>>> named >>>>> recipient, >>>>> any >>>>> use, >>>>> reliance >>>>> upon, >>>>> disclosure >>>>> or >>>>> copying >>>>> of >>>>> this >>>>> >>>>>>> >>>>> email >>>>> or >>>>> its >>>>> attachments >>>>> is >>>>> unauthorised. >>>>> If >>>>> you >>>>> have >>>>> received >>>>> this >>>>> email >>>>> in >>>>> >>>>>>> >>>>> error, >>>>> please >>>>> reply >>>>> via >>>>> email >>>>> or >>>>> call >>>>> +64 >>>>> 21 >>>>> 0832 >>>>> 6348. >>>>> >>>>>>> >>>>> ------------------------------------------------------------------------------ >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> >>>>> Regards, >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Syed >>>>> Ammad >>>>> Ali >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>>>> >>> >>>>> Regards, >>>>> >>> >>>>> >>> >>>>> >>> >>>>> Syed >>>>> Ammad >>>>> Ali >>>>> > >>>>> > >>>>> -- >>>>> > >>>>> Regards, >>>>> > >>>>> > >>>>> > >>>>> Syed >>>>> Ammad >>>>> Ali >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Mohammed >>>>> Naser >>>>> VEXXHOST, >>>>> Inc. >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> >>>>> >>>>> Syed >>>>> Ammad >>>>> Ali >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> >>>>> Syed Ammad Ali >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> >>>>> Syed Ammad Ali >>>>> >>>> <29a37aff-f1f6-46b3-8541-887491c6cfe8-k8s-cluster3-dcu52bgzpbuu-kube_masters-ocfrn2ikpcgd-0-32tmkqgdq7wl-master_config-gihyfv3wlyzd.log> >>>> <6fca39b1-8eda-4786-8424-e5b04434cce7-k8s-cluster3-dcu52bgzpbuu-kube_cluster_config-aht4it6p7wfk.log> >>> >>> >>> >>> -- >>> Regards, >>> >>> >>> Syed Ammad Ali >>> >>> -- >>> Regards, >>> >>> >>> Syed Ammad Ali >>> >> -- >> Cheers & Best regards, >> ------------------------------------------------------------------------------ >> Feilong Wang (王飞龙) (he/him) >> Head of Research & Development >> >> Catalyst Cloud >> Aotearoa's own >> >> Mob: +64 21 0832 6348 | www.catalystcloud.nz >> Level 6, 150 Willis Street, Wellington 6011, New Zealand >> >> CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. >> It may contain privileged, confidential or copyright information. If you are >> not the named recipient, any use, reliance upon, disclosure or copying of this >> email or its attachments is unauthorised. If you have received this email in >> error, please reply via email or call +64 21 0832 6348. >> ------------------------------------------------------------------------------ >> > -- > Cheers & Best regards, > ------------------------------------------------------------------------------ > Feilong Wang (王飞龙) (he/him) > Head of Research & Development > > Catalyst Cloud > Aotearoa's own > > Mob: +64 21 0832 6348 | www.catalystcloud.nz > Level 6, 150 Willis Street, Wellington 6011, New Zealand > > CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. > It may contain privileged, confidential or copyright information. If you are > not the named recipient, any use, reliance upon, disclosure or copying of this > email or its attachments is unauthorised. If you have received this email in > error, please reply via email or call +64 21 0832 6348. > ------------------------------------------------------------------------------ > -- Cheers & Best regards, ------------------------------------------------------------------------------ Feilong Wang (王飞龙) (he/him) Head of Research & Development Catalyst Cloud Aotearoa's own Mob: +64 21 0832 6348 | www.catalystcloud.nz Level 6, 150 Willis Street, Wellington 6011, New Zealand CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipient, any use, reliance upon, disclosure or copying of this email or its attachments is unauthorised. If you have received this email in error, please reply via email or call +64 21 0832 6348. ------------------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkchn.in at gmail.com Wed Sep 1 12:32:27 2021 From: kkchn.in at gmail.com (KK CHN) Date: Wed, 1 Sep 2021 18:02:27 +0530 Subject: Windows2012 VM image importing and Instance Launch Failure In-Reply-To: <0670B960225633449A24709C291A525251C88F34@COM03.performair.local> References: <20210824080849.Horde.RiSp9yA46PcA-BV5WaLQkcq@webmail.nde.ag> <0670B960225633449A24709C291A525251C86EC8@COM03.performair.local> <0670B960225633449A24709C291A525251C88F34@COM03.performair.local> Message-ID: On Fri, Aug 27, 2021 at 11:00 AM wrote: > Kris; > > First, you can add properties during the create command, so steps 1 > through 3 can become: > # openstack image create “M_Windows_VM_NAME_Blaah” --file > Windows_VM_disk.qcow2 \ > --disk-format qcow2 --container-format bare --public --property > hw_firmware_type=uefi \ > --property os_secure_boot=required --property hw_disk_bios=ide > > I am able to import the WIndows VM, which was exported from (Rhvm oVirt) to my openstack(ussuri, qemu-kvm, glance, cinder-ceph ) booted the first bootable disk of this multi disk Windows 2012 RC VM. It has additionally a data disk which is of 9GB of data of 600 GB volume size originally. > Secondly, no. Only the boot image (C: drive) needs the properties. So > any additional disks for the same VM can use a command like this: > I executed this step for the second disk qcow2 image on my Controller node. # openstack image create “M_Windows_VM_NAME_DataDisk2” --file > Windows_VM_disk.qcow2 --disk-format qcow2 --public > > Then the image appears in the Horizon Image section. I tried to create Volume through Horizon GUI with 600 GB disk space allocated , it ran for 10 t0 20 minutes and went to Error state. multiple attempts I have made by deleting each error volume and creating new volume but same result. So I tried to do it through CLI First I listed the images uploaded : by #openstack image list c6695789-8524-446c-98c9-282b2880551c | CMDALocalDBWindowsdisk2 | active // This is the data disk image root at Meghctrl1:/home/cloud# #openstack volume create --image c6695789-8524-446c-98c9-282b2880551c --size 600 --availability-zone nova CMDA_DB_SecDisk_Vol root at Meghctrl1:/home/cloud# openstack volume list +--------------------------------------+----------------------+-----------+------+-------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------------------+-----------+------+-------------+ | 32f9452f-368f-4f26-b70d-c066c4c7e01c | CMDA_DB_SecDisk_Vol | available | 600 | | Now I have gone to the Horizon dashboard. I tried to attach this Volume CMDA_DB_SecDisk_Vol to My running Windows VM which is the first disk which was imported and booted successfully . I got the Dashboard Status alert while doing this : " Info: Attaching volume *CMDA_DB_SecDisk_Vol(32f9452f-368f-4f26-b70d-c066c4c7e01c) to instance 6630c9c8-135e-45b7-958b-c563337eb9c4 on /dev/hdb "* But This is not attached to the Running Windows VM. *You can see it is attached to none * root at Meghctrl1:/home/cloud# openstack volume list +--------------------------------------+----------------------+-----------+------+-------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------------------+-----------+------+-------------+ | 32f9452f-368f-4f26-b70d-c066c4c7e01c | CMDA_DB_SecDisk_Vol | available | 600 | | | 0c404d4c-1685-4a5a-84ae-1ee413f02f68 | cloudtest_DMS1 | error | 102 | | | e9949f4f-b23a-4a30-8620-5a8d234b523b | DMS_VM_APPLICATION-2 | available | 100 | Here you can see it attached to None. *Also in the horizon dashboard the running instance I clicked for more information : It shows no Volumes attached to this instance. * TestDB_CMDA_disk1 Create Snapshot - Overview - Interfaces - Log - Console - Action Log NameTestDB_CMDA_disk1ID6630c9c8-135e-45b7-958b-c563337eb9c4Description-Project IDb1bf4df41dc34e3ab5c3b0318158263dStatusActiveLockedFalseAvailability Zone novaCreatedSept. 1, 2021, 12:24 p.m.Age5 hours, 9 minutesHostMeghctrl1Specs ------------------------------ Flavor NameMediumFlavor ID5bb6bc50-b2f7-49b2-8f01-7a631af5ff8cRAM4GBVCPUs2 VCPUDisk150GBIP Addresses ------------------------------ Tenant_NW172.16.100.248Security Groups ------------------------------ default - ALLOW IPv6 from default - ALLOW IPv4 icmp to 0.0.0.0/0 - ALLOW IPv4 to 0.0.0.0/0 - ALLOW IPv4 tcp from 0.0.0.0/0 - ALLOW IPv4 icmp from 0.0.0.0/0 - ALLOW IPv4 from default - ALLOW IPv6 to ::/0 Metadata ------------------------------ Key NameTNSDCImage NameCMDALocalDBWindows1 Image ID6437fde7-db0c-4298-9d92-1f70aae3c894Volumes Attached ------------------------------ Volume*No volumes attached.* *Details of the CLI Created disk Volume details which is unable to attach to the running Windows 2012 RC instance is as follows from Horizon GUI* CMDA_DB_SecDisk_Vol Edit Volume - Overview - Snapshots NameCMDA_DB_SecDisk_VolID32f9452f-368f-4f26-b70d-c066c4c7e01cProject ID b1bf4df41dc34e3ab5c3b0318158263dStatusAvailableGroup-Specs ------------------------------ Size600 GiBType__DEFAULT__BootableNoEncryptedNoCreatedSept. 1, 2021, 4:07 p.m.Attachments ------------------------------ Attached To*Not attached*Volume Source ------------------------------ Image Metadata ------------------------------ None What am I doing wrong here ? : any hints or suggestions to add this data disk to the running WIndows instance are most welcome. Or do I need to add the volume to the instance through CLI, from the docs I am seeing this. Will this work for Running Windows VM instances also ? openstack server add volume 84c6e57d-a6b1-44b6-81eb-fcb36afd31b5 \ 573e024d-5235-49ce-8332-be1576d323f8 --device /dev/vdb *Will you advice to attaching the Volume through CLI like the following * #openstack server add volume 6630c9c8-135e-45b7-958b-c563337eb9c4 32f9452f-368f-4f26-b70d-c066c4c7e01c --device /dev/hdb * Will it solve this volume attachment issue ? or seriously am I missing any crucial parameters or steps for attaching the additional disk volumes for Windows VMs?* *Any hints most welcome: Any other information or specific log files I need to post, kindly tell me know the log file names I can post . I am doing it for the first time. * Thanks in advance for your answers. Kris You might consider making sure that these additional images aren't marked > as bootable. > > That said... Have you considered importing the disks directly into Cinder > instead? > > Thank you, > > Dominic L. Hilsbos, MBA > Vice President – Information Technology > Perform Air International Inc. > DHilsbos at PerformAir.com > www.PerformAir.com > > > From: KK CHN [mailto:kkchn.in at gmail.com] > Sent: Thursday, August 26, 2021 10:04 PM > To: Dominic Hilsbos > Subject: Re: Windows2012 VM image importing and Instance Launch Failure > > Hi Dominic, > For Importing Single disk Windows VM I performed the following steps on > the vhdx to qcow2 converted image > Step 1: > root at Meghctrl1:/home/cloud# openstack image create > “M_Windows_VM_NAME_Blaah” --file Windows_VM_disk.qcow2 > --disk-format qcow2 --container-format bare --public > Step 2: > root at Meghctrl1:/home/cloud# openstack image set --property > hw_firmware_type=uefi --property > os_secure_boot=required My_Windows_VM_NAME_Blaah > Step 3: > root at Meghctrl1:/home/cloud# openstack image set --property > hw_firmware_type=uefi > --property hw_disk_bios=ide MY_Windows_VM_NAME_Blaah > > And I am able to see the image in the Horizon dashboard "image" tab > and launch the instance from there . Everything went fine and I am able to > get the Windows Single Disk VM running. > > Query: > If I have a Multi Disc Windows VM , do I need to perform all the above > three steps for all the qcow2 converted images ? ( For eg: I have a > Database Server , A Windows VM already running in HyperV with two disks ( > 100GB disk and 600 GB another disk image) . So in these two disks > one will be WIndows OS and another will be attached disk without Windows > OS right ? > 1. For the first disk I will perform all the above three steps after > converting it from vhdx to qcow2. So that image will appear in the > horizon and I can launch an instance as I did early for Single Disk Windows > VM. > 2. For the 600 GB second disk (vhdx format) I will convert it to > qcow2 and then do I need to perform all the above three steps exactly ? > Or what changes I need to make in the above steps to import this disk as > this is an additional attached disk in hyperV. > Request your guidance or any suggestions in this regard. > > Thank You, > Kris > > > > On Wed, Aug 25, 2021 at 9:12 PM wrote: > Kris; > > This shouldn’t be all that much more difficult than Linux, nor that much > harder than a single-disk Windows VM. > > I would start by disabling the SQL service, as the OpenStack instance will > boot once before you can add the data disks (at least it does for me when > using Horizon). > > Export the disks, convert then, and import them, all as you normally > would. Create the instance, as you normally would. The instance will > start; shut it back down. Attach the additional volumes, and start the > instance back up. > > It would be nice to have an option to not start the instance after > creation. > > You could also investigate the “openstack server create” command; it has a > --volume argument that you may be able to pass more than once, though I > suspect not. > > Thank you, > > Dominic L. Hilsbos, MBA > Vice President – Information Technology > Perform Air International Inc. > DHilsbos at PerformAir.com > www.PerformAir.com > > > From: KK CHN [mailto:kkchn.in at gmail.com] > Sent: Tuesday, August 24, 2021 11:19 PM > To: openstack-discuss at lists.openstack.org > Subject: Re: Windows2012 VM image importing and Instance Launch Failure > > Thank you all. > > I am able to get the Single disk Windows VM exported from HyperV and > imported to OpenStack(Ussuri, Glance and Qemu KVM ) Instance launched and > its showing the status running in the PowerState Tab of Horizon dashboard. > > (need to check with the administrator user to login and confirm the VM is > working as expected . ) > > My second phase : I need to perform importing multiple disk > Windows VM to my OpenStack setup . > > ( For single disk Windows vhdx file, I 've converted it to qcow2 file and > imported to OpenStack as the steps I posted in my previous emails. ) > > Now how can I import Multiple disk Windows VM to OpenStack. > > First Step is to convert the vhdx files exported from HyperV to qcow2 > files. > > After this step , what steps need to be performed to bring a multi > disk Windows VM to OpeStack ? > > For Eg: I have a Windows VM > > 1. with Application server 1TB single disk image exported from hyperV. > This I can import to my OpenStack setup with the steps I followed in my > earlier emails as it is a single disk image. > > > 2. A Database server for the above application with 700 GB in multiple > disks. ( 100 and 600 GB disks images exported from HyperV and in vhdx > format for DB server) > > How to bring this server imported to my OpeStack setup (ussuri, glance > and QEMU KVM). As this is from Windows HyperV I am new to multidisk > environment from Windows. > ( for Linux VMs exported from oVirt(RHVM) I am able to import multidisk > VMs to OpenStack as they are in LVM and I have to mount it with VG name and > adding entries in /etc/fstab file ) > > But in Windows multi disks how to perform this ? what steps I have to > perform to import this to OpenStack. Any hints or suggestions most > welcome.. > > Kris > > > On Tue, Aug 24, 2021 at 1:40 PM Eugen Block wrote: > Of course you need a running nova-conductor, how did you launch VMs before? > > > Zitat von KK CHN : > > > yes. I too suspect this not the problem with image. I have seen the > logs: > > it throwing Errors which pasted here. > > > > Its waiting for nova-conductor service as in the logs. Is this the > issue? > > Do I need to explictly statrt conductor service or some other service ? > > nova-compute logs pasted below the lines. share your thoughts what may > > the issue. > > > > > > On Fri, Aug 20, 2021 at 8:11 AM Mohammed Naser > wrote: > > > >> I suggest that you have a look at the compute node logs when it fails to > >> spawn. I suspect the problem is not Your images but something inside > >> openstack > >> > >> > > tail: no files remaining > > cloud at Meghctrl1:~$ sudo tail -f /var/log/nova/nova-compute.log > > [sudo] password for cloud: > > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > > transport_options=transport_options) > > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > > File > > > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > > line 642, in _send > > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > > call_monitor_timeout) > > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > > File > > > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > > line 531, in wait > > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > > message = self.waiters.get(msg_id, timeout=timeout) > > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > > File > > > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > > line 409, in get > > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > > 'to messageID %s' % msg_id) > > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > > oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a > > reply to message ID 7a4e3a4acad3403a9966570cc259f7b7 > > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > > 2021-08-19 12:00:01.467 2847961 WARNING oslo.service.loopingcall [-] > > Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run > > outlasted interval by 50.03 sec > > 2021-08-19 12:00:34.012 2847812 WARNING nova.conductor.api > > [req-afca4d09-bf9c-4bbc-ae88-22008d12b978 - - - - -] Timed out waiting > > for nova-conductor. Is it running? Or did this service start before > > nova-conductor? Reattempting establishment of nova-conductor > > connection...: oslo_messaging.exceptions.MessagingTimeout: Timed out > > waiting for a reply to message ID 9f5b593361b34c8d91585c00e0514047 > > 2021-08-19 12:00:59.480 2847961 DEBUG oslo_service.periodic_task > > [req-58a0e182-6e56-4695-b4db-75624ca69c7b - - - - -] Running periodic > > task ComputeManager._check_instance_build_time run_periodic_tasks > > /usr/local/lib/python3.7/dist-packages/oslo_service/periodic_task.py:211 > > 2021-08-19 12:00:59.481 2847961 DEBUG oslo_service.periodic_task > > [req-58a0e182-6e56-4695-b4db-75624ca69c7b - - - - -] Running periodic > > task ComputeManager._sync_scheduler_instance_info run_periodic_tasks > > /usr/local/lib/python3.7/dist-packages/oslo_service/periodic_task.py:211 > > 2021-08-19 12:01:01.499 2847961 WARNING oslo.service.loopingcall [-] > > Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run > > outlasted interval by 50.03 sec > > 2021-08-19 12:01:34.040 2847812 WARNING nova.conductor.api > > [req-afca4d09-bf9c-4bbc-ae88-22008d12b978 - - - - -] Timed out waiting > > for nova-conductor. Is it running? Or did this service start before > > nova-conductor? Reattempting establishment of nova-conductor > > connection...: oslo_messaging.exceptions.MessagingTimeout: Timed out > > waiting for a reply to message ID e8e478be640e4ecab8a3d27c01960440 > > > > > > essage ID d620b1b3938e4428a0500c00df8d68ee > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > [req-58a0e182-6e56-4695-b4db-75624ca69c7b - - - - -] Error during > > ComputeManager._cleanup_expired_console_auth_tokens: > > oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a > > reply to message ID a493182a404242b79c8f11f8ec350e36 > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > Traceback (most recent call last): > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File > > > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > > line 405, in get > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > return self._queues[msg_id].get(block=True, timeout=timeout) > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File "/usr/local/lib/python3.7/dist-packages/eventlet/queue.py", line > > 322, in get > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > return waiter.wait() > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File "/usr/local/lib/python3.7/dist-packages/eventlet/queue.py", line > > 141, in wait > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > return get_hub().switch() > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File "/usr/local/lib/python3.7/dist-packages/eventlet/hubs/hub.py", > > line 298, in switch > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > return self.greenlet.switch() > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > _queue.Empty > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > During handling of the above exception, another exception occurred: > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > Traceback (most recent call last): > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File > "/usr/local/lib/python3.7/dist-packages/oslo_service/periodic_task.py", > > line 216, in run_periodic_tasks > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > task(self, context) > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File "/usr/local/lib/python3.7/dist-packages/nova/compute/manager.py", > > line 10450, in _cleanup_expired_console_auth_tokens > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > objects.ConsoleAuthToken.clean_expired_console_auths(context) > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File > "/usr/local/lib/python3.7/dist-packages/oslo_versionedobjects/base.py", > > line 177, in wrapper > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > args, kwargs) > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File "/usr/local/lib/python3.7/dist-packages/nova/conductor/rpcapi.py", > > line 243, in object_class_action_versions > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > args=args, kwargs=kwargs) > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/client.py", > > line 179, in call > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > transport_options=self.transport_options) > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/transport.py", > > line 128, in _send > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > transport_options=transport_options) > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File > > > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > > line 654, in send > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > transport_options=transport_options) > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File > > > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > > line 642, in _send > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > call_monitor_timeout) > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File > > > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > > line 531, in wait > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > message = self.waiters.get(msg_id, timeout=timeout) > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > File > > > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > > line 409, in get > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > 'to message ID %s' % msg_id) > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a > > reply to message ID a493182a404242b79c8f11f8ec350e36 > > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > > 2021-08-19 12:27:01.681 2847961 DEBUG oslo_service.periodic_task > > [req-58a0e182-6e56-4695-b4db-75624ca69c7b - - - - -] Running periodic > > task ComputeManager._check_instance_build_time run_periodic_tasks > > /usr/local/lib/python3.7/dist-packages/oslo_service/periodic_task.py:211 > > 2021-08-19 12:27:01.687 2847961 DEBUG oslo_service.periodic_task > > [req-58a0e182-6e56-4695-b4db-75624ca69c7b - - - - -] Running periodic > > task ComputeManager._sync_scheduler_instance_info run_periodic_tasks > > /usr/local/lib/python3.7/dist-packages/oslo_service/periodic_task.py:211 > > 2021-08-19 12:27:02.368 2847961 WARNING oslo.service.loopingcall [-] > > Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run > > outlasted interval by 50.03 sec > > > > > > > > Error-nov-log-file_while_sch ... e_in _horizaon_dashboard.txt > > Open with > > Displaying Error-nov-log-file_while_scheduling_Windows_VM_instance_in > > _horizaon_dashboard.txt. > > > > > > > > > > > >> If I was to guess it’s probably missing UEFI firmware packages :) > >> > >> On Wed, Aug 18, 2021 at 9:17 AM KK CHN wrote: > >> > >>> Error : failed to perform requested operation on instance "WindowsVM > "the > >>> instance has error status. Please try again later [Error exceeded > maximum > >>> number of retries. Exhausted all hosts available for retrying build > >>> failures for instance e3d5c095-7d26-4b1e-89d1-d1a6e20a45041 > >>> > >>> I am trying to import a WIndows2012 Single disk VM, to OpenStack > Ussuri, > >>> with glance and Qemu KVM. > >>> > >>> In bare machine KVM I am able to import and boot this Windows VM which > >>> exported from rhevm hypervisor as vhdx image. > >>> what I have done is > >>> > >>> 1. converted this windows image from vhdx to qcow2 > >>> 2. root at MeghCtrol1:/home/cloud/CMOBB_APP#cirt-install --name WINDOWS > >>> --ram=1048 --vcups=1 --cpu host --hvm --dick > >>> path=BackUP2_CMAPP_disk_1_Windows_qcow2_imagefile,device=disk, > >>> format=qcow2,bus=virtio --graphics vnc --boot uefi > >>> > >>> This uploaded the qcow2 image of WindowsVM to KVM hypervisor and its > >>> working. > >>> > >>> But when I do importing to openstack unable to launch instance from > >>> the image . > >>> > >>> These are the steps I performed.. > >>> > >>> 1. openstack image create "WindowsVM" --file CMAPP_disk_1.qcow2 > >>> --disk-format qcow2 --container-format bare --public > >>> > >>> 4.openstack image set --property hw_firmware_type=uefi --property > >>> os_secure_boot=required WindowsVM > >>> > >>> 5.openstack image set --property hw_firmware_type=uefi --property > >>> hw_disk_bus=ide WindowsVM > >>> > >>> 6.openstack image show WindowsVM > >>> > >>> 7. root at dmzcloud:/home/cloud# openstack image show WindowsVM|grep > >>> "properties" > >>> | properties | hw_disk_bus='ide', hw_firmware_type='uefi', > >>> os_hash_algo='sha512', > >>> > >>> > os_hash_value='753ee596980409e1e72d6d020c8219c56a6ada8b43f634fb575c594a245725a398e45982c0a1ad72b3fc3451cde62cceb9ff22be044863b31ecdd7893b049349', > >>> os_hidden='False', os_secure_boot='required', > >>> owner_specified.openstack.md5='', > >>> owner_specified.openstack.object='images/WindowsVM', > >>> owner_specified.openstack.sha256='' | > >>> root at dmzcloud:/home/cloud# > >>> > >>> > >>> Then I logged into horizon dashboard, from the images selected the > >>> imported image and try to launch the instance. With a Flavour of 550 > GB > >>> disk, 4 vcpus and 8GB Ram .. > >>> > >>> The instance spawning ran for 30 minutes and throws the error which I > >>> pasted first in the right top corner of horizon dashboard. > >>> > >>> How to solve this error and boot the Windows machine successfully .. > >>> > >>> > >>> """ > >>> Error : failed to perform requested operation on instance "WindowsVM > "the > >>> instance has error status. Please try again later [Error exceeded > maximum > >>> number of retries. Exhausted all hosts available for retrying build > >>> failures for instance e3d5c095-7d26-4b1e-89d1-d1a6e20a45041 > >>> > >>> > >>> """ > >>> > >>> Any help highly appreciated. > >>> > >>> Kris > >>> > >>> -- > >> Mohammed Naser > >> VEXXHOST, Inc. > >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anyrude10 at gmail.com Wed Sep 1 14:00:08 2021 From: anyrude10 at gmail.com (Anirudh Gupta) Date: Wed, 1 Sep 2021 19:30:08 +0530 Subject: [Kolla][Kolla-Ansible] Ironic Node Cleaning Failed In-Reply-To: References: Message-ID: Hi, I tried Creating a config override file at /etc/kolla/config/ironic.conf and specified the below items: [deploy] erase_devices_priority = 0 erase_devices_metadata_priority = 10 With this it took approx 15 mins to complete the process. Now I tried provisioning the Baremetal Node having 2 Hard disk with the above configuration. Below is my observations: On one of the Hard disk, OS was installed. On the other hard drive, we tried mounting it to a particular mount point. Initially, we got an error *"unknown fs type"*. So we formatted the sdb drive in XFS format using mkfs.xfs command (also tried with ext4) and it was successfully mounted then. After this we put some data onto that disk. Now when the baremetal server is recreated ( openstack server delete and then openstack server create), according to my understanding, the data on the 2nd hard drive sdb should remain intact. Only the data on OS should get deleted. But when I tried mounting the 2nd drive again, it gave us the same error *"unknown fs type"* That means without formatting sdb, I am not able to mount it which means I am not able to access my data stored on SDB which ideally should not be the case. Is there any additional setting that we need to do in an ironic in order to make it work? Regards Anirudh Gupta On Mon, Aug 23, 2021 at 1:20 PM Mark Goddard wrote: > > > On Fri, 20 Aug 2021 at 15:08, Dmitry Tantsur wrote: > >> Hi, >> >> On Fri, Aug 13, 2021 at 8:56 AM Anirudh Gupta >> wrote: >> >>> Hi All, >>> >>> I had a 900 GB hard disk on my Baremetal Node and it took approx *15 >>> hours *to make the baremetal node come in *available* state from >>> *clean_wait* state. >>> >>> Once the baremetal node came available, I was able to create a server >>> and provision it with a user image. >>> >>> Is taking 15 hours to erase_device in clean_wait normal for a 900 GB >>> hard disk in Ironic? >>> >> >> Unfortunately, yes. If your hardware does not support ATA secure erase, >> the only way we can remove data is to shred the disk (essentially, write >> all 900 GB several times). >> >> If you don't care about residual data on the disks, you can switch to >> metadata cleaning (only partition tables). This is fast but insecure. I >> don't know how the options are called in Kolla, but in Ironic you do >> something like this: >> >> [deploy] >> erase_devices_priority = 0 >> erase_devices_metadata_priority = 10 >> > > In kolla we do not try to own all options - simply create a config > override file at /etc/kolla/config/ironic.conf including the above. > > > https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html#openstack-service-configuration-in-kolla > > >> Dmitry >> >> >>> >>> Regards >>> Anirudh Gupta >>> >>> >>> On Mon, Aug 9, 2021 at 2:01 PM Anirudh Gupta >>> wrote: >>> >>>> Hi Mark, >>>> >>>> Earlier I was passing the boot_mode as uefi while creating the >>>> baremetal node. >>>> On Kolla-Ansible Launchpad, I found some issues related to UEFI mode, >>>> so I didn't pass the parameter. >>>> >>>> With IPXE and without passing UEFI boot mode parameter, my node started >>>> cleaning. It connected with the TFTP server. >>>> >>>> But from the last 2 hours, the state is still in *clean_wait* only. >>>> >>>> The ramdisk and kernel images I used were the ones mentioned in the >>>> link below >>>> >>>> >>>> - >>>> https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.kernel >>>> - >>>> https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.initramfs >>>> >>>> >>>> For this I followed the latest kolla ansible document:- >>>> >>>> - >>>> https://docs.openstack.org/kolla-ansible/latest/reference/bare-metal/ironic-guide.html >>>> >>>> >>>> All I can see in *ironic-conductor* logs is: >>>> >>>> 2021-08-09 13:49:51.159 7 DEBUG ironic.drivers.modules.agent_base [-] >>>> Heartbeat from node 8b1ec553-fbc9-4912-bd33-88afc41b8f81 heartbeat >>>> /var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/agent_base.py:641 >>>> 2021-08-09 13:49:51.178 7 DEBUG ironic.drivers.modules.agent_client [-] >>>> Fetching status of agent commands for node >>>> 8b1ec553-fbc9-4912-bd33-88afc41b8f81 get_commands_status >>>> /var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/agent_client.py:310 >>>> 2021-08-09 13:49:51.186 7 DEBUG ironic.drivers.modules.agent_client [-] >>>> Status of agent commands for node 8b1ec553-fbc9-4912-bd33-88afc41b8f81: >>>> get_clean_steps: result "{'clean_steps': {'GenericHardwareManager': >>>> [{'step': 'erase_devices', 'priority': 10, 'interface': 'deploy', >>>> 'reboot_requested': False, 'abortable': True}, {'step': >>>> 'erase_devices_metadata', 'priority': 99, 'interface': 'deploy', >>>> 'reboot_requested': False, 'abortable': True}, {'step': 'erase_pstore', >>>> 'priority': 0, 'interface': 'deploy', 'reboot_requested': False, >>>> 'abortable': True}, {'step': 'delete_configuration', 'priority': 0, >>>> 'interface': 'raid', 'reboot_requested': False, 'abortable': True}, >>>> {'step': 'create_configuration', 'priority': 0, 'interface': 'raid', >>>> 'reboot_requested': False, 'abortable': True}, {'step': 'burnin_cpu', >>>> 'priority': 0, 'interface': 'deploy', 'reboot_requested': False, >>>> 'abortable': True}, {'step': 'burnin_disk', 'priority': 0, 'interface': >>>> 'deploy', 'reboot_requested': False, 'abortable': True}, {'step': >>>> 'burnin_memory', 'priority': 0, 'interface': 'deploy', 'reboot_requested': >>>> False, 'abortable': True}, {'step': 'burnin_network', 'priority': 0, >>>> 'interface': 'deploy', 'reboot_requested': False, 'abortable': True}]}, >>>> 'hardware_manager_version': {'MellanoxDeviceHardwareManager': '1', >>>> 'generic_hardware_manager': '1.1'}}", error "None"; execute_clean_step: >>>> result "{'clean_result': None, 'clean_step': {'step': >>>> 'erase_devices_metadata', 'priority': 99, 'interface': 'deploy', >>>> 'reboot_requested': False, 'abortable': True, 'requires_ramdisk': True}}", >>>> error "None"; execute_clean_step: result "None", error "None" >>>> get_commands_status >>>> /var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/agent_client.py:342 >>>> 2021-08-09 13:49:51.186 7 DEBUG ironic.drivers.modules.agent_base [-] *Clean >>>> step still running for node 8b1ec553-fbc9-4912-bd33-88afc41b8f81:* >>>> None _get_completed_command >>>> /var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/agent_base.py:267 >>>> >>>> It would be a great help if you could suggest some pointers. >>>> >>>> Regards >>>> Anirudh Gupta >>>> >>>> >>>> >>>> >>>> I tried >>>> >>>> On Mon, Aug 9, 2021 at 1:43 PM Mark Goddard wrote: >>>> >>>>> >>>>> >>>>> On Fri, 6 Aug 2021 at 13:49, Anirudh Gupta >>>>> wrote: >>>>> >>>>>> Hi Dmitry, >>>>>> >>>>>> I tried taking TCPDUMP while the Baremetal Node was booting up and >>>>>> looked for tftp protocols and found there was some "*File Not Found" >>>>>> *traces for bootx64.efi >>>>>> >>>>>> [image: image.png] >>>>>> >>>>>> Then, I found a related post on openstack Discuss which suggested to >>>>>> enable IPXE >>>>>> >>>>>> http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010329.html >>>>>> >>>>>> After re-deploying the setup with IPXE enabled, i found similar >>>>>> traces now for *ipxe.efi file* >>>>>> >>>>>> [image: image.png] >>>>>> >>>>>> Can you please now suggest what possibly could be a miss in >>>>>> configuration and steps to resolve it. >>>>>> >>>>> >>>>> Hi Anirudh, >>>>> >>>>> I'd suggest installing a tftp client on your machine and making some >>>>> requests. The TFTP daemon runs in the ironic_pxe container, and TFTP files >>>>> are served from /tftpboot in that container. >>>>> >>>>> Mark >>>>> >>>>>> >>>>>> For your reference, I am attaching the complete tcpdump logs of both >>>>>> the Scenarios >>>>>> >>>>>> Looking forward to hearing from you. >>>>>> >>>>>> Regards >>>>>> Anirudh Gupta >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Aug 5, 2021 at 4:56 PM Anirudh Gupta >>>>>> wrote: >>>>>> >>>>>>> Hi Team, >>>>>>> >>>>>>> On further debugging, I found an error in neutron-server logs >>>>>>> >>>>>>> >>>>>>> Failed to bind port 476d8175-ffc2-49ba-bb12-0a77c1f07e5f on host >>>>>>> f4a43fa5-9c41-488e-a34d-714ae5a9d300 for vnic_type baremetal using segments >>>>>>> [{'id': '1a5bbe96-2488-4971-925f-7c9346ba3ef5', 'network_type': 'flat', >>>>>>> 'physical_network': 'physnet1', 'segmentation_id': None, 'network_id': >>>>>>> '5b6cccec-ad86-4ed9-8d3c-72a31ec3a0d4'}] >>>>>>> 2021-08-05 16:33:06.979 23 INFO neutron.plugins.ml2.plugin >>>>>>> [req-54d11d51-7319-43ea-b70c-fe39d8aafe8a 21d6a238438e4294912746bcdc895e31 >>>>>>> 3eca725754e1405eb178cc39bd0da3aa - default default] Attempt 9 to bind port >>>>>>> 476d8175-ffc2-49ba-bb12-0a77c1f07e5f >>>>>>> >>>>>>> where 476d8175-ffc2-49ba-bb12-0a77c1f07e5f is the uuid of Baremetal >>>>>>> Node >>>>>>> >>>>>>> However the port is created in openstack, but its state is down >>>>>>> >>>>>>> [ansible at localhost ~]$ openstack port list >>>>>>> >>>>>>> +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ >>>>>>> | ID | Name | MAC Address | >>>>>>> Fixed IP Addresses | >>>>>>> Status | >>>>>>> >>>>>>> +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ >>>>>>> | 07d6b83d-d83c-498f-8ba8-b4f21bef7249 | | fa:16:3e:38:05:9d | >>>>>>> ip_address='10.0.1.200', subnet_id='7b72c158-2146-4bd6-893b-bd76b4a3e869' | >>>>>>> ACTIVE | >>>>>>> | 476d8175-ffc2-49ba-bb12-0a77c1f07e5f | | *98:f2:b3:3f:72:d8* >>>>>>> | ip_address='10.0.1.202', subnet_id='7b72c158-2146-4bd6-893b-bd76b4a3e869' >>>>>>> | *DOWN * | >>>>>>> >>>>>>> +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ >>>>>>> >>>>>>> *98:f2:b3:3f:72:d8 *is the mac address of my Baremetal Node on >>>>>>> which PXE is enabled. >>>>>>> >>>>>>> Can someone please help in resolving this issue. >>>>>>> >>>>>>> *Issue:* >>>>>>> *Node goes in clean_failed from clean_wait.* >>>>>>> >>>>>>> Regards >>>>>>> Anirudh Gupta >>>>>>> >>>>>>> On Tue, Aug 3, 2021 at 8:32 PM Anirudh Gupta >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Dmitry, >>>>>>>> >>>>>>>> I might be wrong, but as per my understanding if there would be an >>>>>>>> issue in dnsmasq, then IP 20.20.20.10 would not have been assigned to the >>>>>>>> machine. >>>>>>>> >>>>>>>> TCPDUMP logs are as below: >>>>>>>> >>>>>>>> 20:16:58.938089 IP controller.bootps > 255.255.255.255.bootpc: >>>>>>>> BOOTP/DHCP, Reply, length 312 >>>>>>>> 20:17:02.765291 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: >>>>>>>> BOOTP/DHCP, Request from 98:f2:b3:3f:72:e5 (oui Unknown), length 359 >>>>>>>> 20:17:02.766303 IP controller.bootps > 255.255.255.255.bootpc: >>>>>>>> BOOTP/DHCP, Reply, length 312 >>>>>>>> 20:17:26.944378 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: >>>>>>>> BOOTP/DHCP, Request from 98:f2:b3:3f:72:e5 (oui Unknown), length 347 >>>>>>>> 20:17:26.944756 IP controller.bootps > 255.255.255.255.bootpc: >>>>>>>> BOOTP/DHCP, Reply, length 312 >>>>>>>> 20:17:30.763627 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: >>>>>>>> BOOTP/DHCP, Request from 98:f2:b3:3f:72:e5 (oui Unknown), length 359 >>>>>>>> 20:17:30.764620 IP controller.bootps > 255.255.255.255.bootpc: >>>>>>>> BOOTP/DHCP, Reply, length 312 >>>>>>>> 20:17:54.938791 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: >>>>>>>> BOOTP/DHCP, Request from 98:f2:b3:3f:72:e5 (oui Unknown), length 347 >>>>>>>> >>>>>>>> Also the neutron dnsmasq logs and ironic inspector logs are >>>>>>>> attached in the mail. >>>>>>>> >>>>>>>> Regards >>>>>>>> Anirudh Gupta >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Aug 3, 2021 at 7:29 PM Dmitry Tantsur >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> You need to check the dnsmasq logs (there are two dnsmasqs: from >>>>>>>>> neutron and from ironic-inspector). tcpdump may also help to determine >>>>>>>>> where the packages are lost. >>>>>>>>> >>>>>>>>> Dmitry >>>>>>>>> >>>>>>>>> On Fri, Jul 30, 2021 at 10:29 PM Anirudh Gupta < >>>>>>>>> anyrude10 at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Dmitry >>>>>>>>>> >>>>>>>>>> Thanks for your time. >>>>>>>>>> >>>>>>>>>> My system is getting IP 20.20.20.10 which is in the range defined >>>>>>>>>> in ironic_dnsmasq_dhcp_range field under globals.yml file. >>>>>>>>>> >>>>>>>>>> ironic_dnsmasq_dhcp_range: "20.20.20.10,20.20.20.100" >>>>>>>>>> >>>>>>>>>> And in the cleaning network (public1), the range defined is >>>>>>>>>> 20.20.20.150-20.20.20.200 >>>>>>>>>> >>>>>>>>>> As per my understanding, these 2 ranges should be mutually >>>>>>>>>> exclusive. >>>>>>>>>> >>>>>>>>>> Please suggest if my understanding is not correct. >>>>>>>>>> >>>>>>>>>> Any suggestions what should I do to resolve this issue? >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> Anirudh Gupta >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, 31 Jul, 2021, 12:06 am Dmitry Tantsur, < >>>>>>>>>> dtantsur at redhat.com> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Jul 29, 2021 at 6:05 PM Anirudh Gupta < >>>>>>>>>>> anyrude10 at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Team, >>>>>>>>>>>> >>>>>>>>>>>> In to the email below, I have some updated information:- >>>>>>>>>>>> >>>>>>>>>>>> Earlier the allocation range mentioned in " >>>>>>>>>>>> *ironic_dnsmasq_dhcp_range*" in globals.yml had an overlapping >>>>>>>>>>>> range with the cleaning network, due to which there was some issue in >>>>>>>>>>>> receiving the DHCP request >>>>>>>>>>>> >>>>>>>>>>>> After creating a cleaning network with a separate allocation >>>>>>>>>>>> range, I am successfully getting IP allocated to my Baremetal Node >>>>>>>>>>>> >>>>>>>>>>>> - openstack subnet create subnet1 --network public1 >>>>>>>>>>>> --subnet-range 20.20.20.0/24 --allocation-pool >>>>>>>>>>>> start=20.20.20.150,end=20.20.20.200 --ip-version=4 --gateway=20.20.20.1 >>>>>>>>>>>> --dhcp >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [image: image.png] >>>>>>>>>>>> >>>>>>>>>>>> After getting the IP, there is no further action on the node. >>>>>>>>>>>> From "*clean_wait*", it goes into "*clean_failed*" state after >>>>>>>>>>>> around half an hour. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The IP address is not from the cleaning range, it may come from >>>>>>>>>>> inspection. You probably need to investigate your network topology, maybe >>>>>>>>>>> use tcpdump. >>>>>>>>>>> >>>>>>>>>>> Unfortunately, I'm not fluent in Kolla to say if it can be a bug >>>>>>>>>>> or not. >>>>>>>>>>> >>>>>>>>>>> Dmitry >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On verifying the logs, I could see the below error messages >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> - In */var/log/kolla/ironic/ironic-conductor.log*, we >>>>>>>>>>>> observed the following error: >>>>>>>>>>>> >>>>>>>>>>>> ERROR ironic.conductor.utils [-] Cleaning for node >>>>>>>>>>>> 3a56748e-a8ca-4dec-a332-ace18e6d494e failed. *Timeout reached >>>>>>>>>>>> while cleaning the node. Please check if the ramdisk responsible for the >>>>>>>>>>>> cleaning is running on the node. Failed on step {}.* >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Note : For Cleaning the node, we have used the below images >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.kernel >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://tarballs.openstack.org/ironic-python-agent/dib/files/ipa-centos8-master.initramfs >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> - In /var/log/kolla/nova/nova-compute-ironic.log, we >>>>>>>>>>>> observed the error >>>>>>>>>>>> >>>>>>>>>>>> ERROR nova.compute.manager >>>>>>>>>>>> [req-810ffedf-3343-471c-94db-85411984e6cc - - - - -] No compute node record >>>>>>>>>>>> for host controller-ironic: >>>>>>>>>>>> nova.exception_Remote.ComputeHostNotFound_Remote: Compute host >>>>>>>>>>>> controller-ironic could not be found. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Can someone please help in this regard? >>>>>>>>>>>> >>>>>>>>>>>> Regards >>>>>>>>>>>> Anirudh Gupta >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 27, 2021 at 12:52 PM Anirudh Gupta < >>>>>>>>>>>> anyrude10 at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Team, >>>>>>>>>>>>> >>>>>>>>>>>>> We have deployed 2 node kolla ansible *12.0.0* in order to >>>>>>>>>>>>> deploy openstack *wallaby* release. We have also enabled >>>>>>>>>>>>> ironic in order to provision the bare metal nodes. >>>>>>>>>>>>> >>>>>>>>>>>>> On each server we have 3 nics >>>>>>>>>>>>> >>>>>>>>>>>>> - *eno1* - OAM for external connectivity and endpoint's >>>>>>>>>>>>> publicURL >>>>>>>>>>>>> - *eno2* - Mgmt for internal communication between various >>>>>>>>>>>>> openstack services. >>>>>>>>>>>>> - *ens2f0* - Data Interface >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Corresponding to this we have defined the following fields in >>>>>>>>>>>>> globals.yml >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> - kolla_base_distro: "centos" >>>>>>>>>>>>> - kolla_install_type: "source" >>>>>>>>>>>>> - openstack_release: "wallaby" >>>>>>>>>>>>> - network_interface: "eno2" >>>>>>>>>>>>> # MGMT interface >>>>>>>>>>>>> - kolla_external_vip_interface: "eno1" # OAM >>>>>>>>>>>>> Interface >>>>>>>>>>>>> - kolla_internal_vip_address: "192.168.10.3" # MGMT >>>>>>>>>>>>> Subnet free ip >>>>>>>>>>>>> - kolla_external_vip_address: "10.0.1.136" # OAM >>>>>>>>>>>>> subnet free IP >>>>>>>>>>>>> - neutron_external_interface: "ens2f0" # Data >>>>>>>>>>>>> Interface >>>>>>>>>>>>> - enable_neutron_provider_networks: "yes" >>>>>>>>>>>>> >>>>>>>>>>>>> Note: Only relevant fields are being shown in this query >>>>>>>>>>>>> >>>>>>>>>>>>> Also, for ironic following fields have been defined in >>>>>>>>>>>>> globals.yml >>>>>>>>>>>>> >>>>>>>>>>>>> - enable_ironic: "yes" >>>>>>>>>>>>> - enable_ironic_neutron_agent: "{{ enable_neutron | bool >>>>>>>>>>>>> and enable_ironic | bool }}" >>>>>>>>>>>>> - enable_horizon_ironic: "{{ enable_ironic | bool }}" >>>>>>>>>>>>> - ironic_dnsmasq_interface: "*ens2f0*" >>>>>>>>>>>>> # Data interface >>>>>>>>>>>>> - ironic_dnsmasq_dhcp_range: "20.20.20.10,20.20.20.100" >>>>>>>>>>>>> - ironic_dnsmasq_boot_file: "pxelinux.0" >>>>>>>>>>>>> - ironic_cleaning_network: "public1" >>>>>>>>>>>>> - ironic_dnsmasq_default_gateway: "20.20.20.1" >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> After successful deployment, a flat provider network with the >>>>>>>>>>>>> name public1 is being created in openstack using the below commands: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> - openstack network create public1 --provider-network-type >>>>>>>>>>>>> flat --provider-physical-network physnet1 >>>>>>>>>>>>> - openstack subnet create subnet1 --network public1 >>>>>>>>>>>>> --subnet-range 20.20.20.0/24 --allocation-pool >>>>>>>>>>>>> start=20.20.20.10,end=20.20.20.100 --ip-version=4 --gateway=20.20.20.1 >>>>>>>>>>>>> --dhcp >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Issue/Queries: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> - Is the configuration done in globals.yml correct or is >>>>>>>>>>>>> there anything else that needs to be done in order to separate control and >>>>>>>>>>>>> data plane traffic? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> - Also I have set automated_cleaning as "true" in >>>>>>>>>>>>> ironic-conductor conatiner settings.But after creating the baremetal node, >>>>>>>>>>>>> we run "node manage" command which runs successfully. Running "*openstack >>>>>>>>>>>>> baremetal node provide "* command powers on the >>>>>>>>>>>>> machine, sets the boot mode on Network Boot but no DHCP request for that >>>>>>>>>>>>> particular mac is obtained on the controller. Is there anything I am >>>>>>>>>>>>> missing that needs to be done in order to make ironic work? >>>>>>>>>>>>> >>>>>>>>>>>>> Note: I have also verified that the nic is PXE enabled in >>>>>>>>>>>>> system configuration setting >>>>>>>>>>>>> >>>>>>>>>>>>> Regards >>>>>>>>>>>>> Anirudh Gupta >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Red Hat GmbH, https://de.redhat.com/ , Registered seat: >>>>>>>>>>> Grasbrunn, >>>>>>>>>>> Commercial register: Amtsgericht Muenchen, HRB 153243, >>>>>>>>>>> Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, >>>>>>>>>>> Michael O'Neill >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Red Hat GmbH, https://de.redhat.com/ , Registered seat: >>>>>>>>> Grasbrunn, >>>>>>>>> Commercial register: Amtsgericht Muenchen, HRB 153243, >>>>>>>>> Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, >>>>>>>>> Michael O'Neill >>>>>>>>> >>>>>>>> >> >> -- >> Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, >> Commercial register: Amtsgericht Muenchen, HRB 153243, >> Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael >> O'Neill >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 38285 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 185546 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 200447 bytes Desc: not available URL: From paolo at celati.com Wed Sep 1 15:07:57 2021 From: paolo at celati.com (Paolo Celati) Date: Wed, 1 Sep 2021 17:07:57 +0200 Subject: [kolla-ansible] [kuryr] Running Docker Swarm with Kuryr networking In-Reply-To: References: <7132cbfa-c8be-76c7-3f77-5027f74eff2e@celati.com> Message-ID: Hi Michal,   Yes I would rather run all my Docker containers on the physical hosts.  At the moment I run Ceph and kolla images, but adding Swarm as well would be useful. I've had a fair number of problems with Magnum because it appears only Kubernetes is supported nowadays, and I have 0 experience with that.  I also prefer Swarm because it's a lot simpler for small scale. Administering my own VMs with Docker Swarm on top is also not exactly the best solution because that introduces extra machines to maintain.  And I don't have lots of spare RAM either as it's a homelab. Thanks for reminding me about tagging correctly, I forgot to put [kuryr].  Ok didn't know about the lack of CI but good to know. Kind regards, Paolo On 01/09/2021 15:57, Michał Nasiadka wrote: > Hi Paolo, > > Would you like to use the Docker engine that is running on the > OpenStack cluster hosts, or create Virtual Machines that will be used > for a Docker Swarm cluster? > I would propose the latter. > > About Kuryr - we don’t have CI coverage for testing Kuryr in > Kolla-Ansible deployment, so the container images and Ansible > deployment role are provided as-is currently. > > Maybe somebody from Kuryr project could help you out? Adding [kuryr] > tag for visibility. > > Best regards, > > Michal > On 1 Sep 2021, 00:32 +0200, Paolo Celati , wrote: >> >> Hi, >> >> long story short I have a 3 node Openstack cluster that I manage with >> kolla-ansible, and I'd like to run Docker Swarm on that as well. I am >> aware Magnum exists, but I'd first like to get my head around this >> simpler case. >> >> Seeing as I'd like to connect Docker containers from swarm compose >> files to Neutron networks I'm trying to set up Kuryr together with a >> swarm configuration. However the documentation is a little scarce >> >> and I'd prefer running everything on these three hosts, including >> etcd. If I follow the guide and pass --cluster-store and >> --cluster-advertise arguments to dockerd then I can't run Docker in >> Swarm mode because I get an error saying Swarm is incompatible with >> those options, and at the same time it's not clear from documentation >> how you are expected to do Kuryr+Swarm. I did initialise the Swarm >> cluster before trying to add Kuryr, so I don't know if perhaps doing >> this the other way works? Do you have ideas or advice with this >> scenario? If worst comes to worst I can set up an external etcd >> cluster on a separate non-Openstack cluster but I'd rather avoid that. >> >> >> Thanks in advance, >> >> Paolo >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_0x6A5811658B827BE4.asc Type: application/pgp-keys Size: 3123 bytes Desc: OpenPGP public key URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: From DHilsbos at performair.com Wed Sep 1 16:46:45 2021 From: DHilsbos at performair.com (DHilsbos at performair.com) Date: Wed, 1 Sep 2021 16:46:45 +0000 Subject: Windows2012 VM image importing and Instance Launch Failure In-Reply-To: References: <20210824080849.Horde.RiSp9yA46PcA-BV5WaLQkcq@webmail.nde.ag> <0670B960225633449A24709C291A525251C86EC8@COM03.performair.local> <0670B960225633449A24709C291A525251C88F34@COM03.performair.local> Message-ID: <0670B960225633449A24709C291A525251C8ECCC@COM03.performair.local> Kris; This doesn't look like an issue of Windows in OpenStack, it looks like a glance / nova / underlying storage issue. Do you have any images in this cluster that work correctly? Have you checked your glance / storage logs? What storage system do you use with glance? Have you tried working out how to import your disks directly into cinder as a volume? Thank you, Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos at PerformAir.com www.PerformAir.com From: KK CHN [mailto:kkchn.in at gmail.com] Sent: Wednesday, September 1, 2021 5:32 AM To: Dominic Hilsbos; openstack-discuss at lists.openstack.org Subject: Re: Windows2012 VM image importing and Instance Launch Failure On Fri, Aug 27, 2021 at 11:00 AM wrote: Kris; First, you can add properties during the create command, so steps 1 through 3 can become: # openstack image create  “M_Windows_VM_NAME_Blaah” --file Windows_VM_disk.qcow2 \ --disk-format qcow2 --container-format bare --public --property hw_firmware_type=uefi \ --property os_secure_boot=required --property hw_disk_bios=ide I am able to import  the  WIndows VM, which was exported from (Rhvm oVirt)  to  my openstack(ussuri, qemu-kvm, glance, cinder-ceph )  booted the first bootable disk  of this multi disk Windows 2012 RC VM.   It has additionally   a data disk which is of 9GB of data  of  600 GB volume size originally.    Secondly, no.  Only the boot image (C: drive) needs the properties.  So any additional disks for the same VM can use a command like this: I executed this step for the second disk qcow2 image  on my Controller node.  # openstack image create  “M_Windows_VM_NAME_DataDisk2” --file Windows_VM_disk.qcow2 --disk-format qcow2 --public Then the  image appears in the Horizon Image section.  I tried to create Volume through  Horizon GUI  with  600 GB disk space allocated ,  it ran for 10 t0 20 minutes and went to Error state.  multiple attempts I have made by deleting each error volume and creating new volume  but same result.  So I tried to do it through CLI First I listed the images uploaded : by #openstack image list   c6695789-8524-446c-98c9-282b2880551c | CMDALocalDBWindowsdisk2 | active           // This is  the  data disk image root at Meghctrl1:/home/cloud# #openstack volume create --image c6695789-8524-446c-98c9-282b2880551c --size 600  --availability-zone nova CMDA_DB_SecDisk_Vol root at Meghctrl1:/home/cloud# openstack volume list +--------------------------------------+----------------------+-----------+------+-------------+ | ID                                   | Name                 | Status    | Size | Attached to | +--------------------------------------+----------------------+-----------+------+-------------+ | 32f9452f-368f-4f26-b70d-c066c4c7e01c | CMDA_DB_SecDisk_Vol  | available |  600 |             |   Now I have gone to the Horizon dashboard. I tried to attach this Volume  CMDA_DB_SecDisk_Vol  to   My running  Windows VM  which is the first disk which was imported and booted successfully .  I got the Dashboard Status  alert  while doing this :   " Info: Attaching  volume  CMDA_DB_SecDisk_Vol(32f9452f-368f-4f26-b70d-c066c4c7e01c) to instance 6630c9c8-135e-45b7-958b-c563337eb9c4 on      /dev/hdb  "   But This is not attached to the Running Windows VM. You can see it is attached to none   root at Meghctrl1:/home/cloud# openstack volume list +--------------------------------------+----------------------+-----------+------+-------------+ | ID                                   | Name                 | Status    | Size | Attached to | +--------------------------------------+----------------------+-----------+------+-------------+ | 32f9452f-368f-4f26-b70d-c066c4c7e01c | CMDA_DB_SecDisk_Vol  | available |  600 |             | | 0c404d4c-1685-4a5a-84ae-1ee413f02f68 | cloudtest_DMS1       | error     |  102 |             | | e9949f4f-b23a-4a30-8620-5a8d234b523b | DMS_VM_APPLICATION-2 | available |  100 |              Here you can see it attached to None. Also in the horizon dashboard the running instance   I clicked for more information : It shows  no Volumes attached to this instance.  TestDB_CMDA_disk1 Create Snapshot • Overview • Interfaces • Log • Console • Action Log Name TestDB_CMDA_disk1 ID 6630c9c8-135e-45b7-958b-c563337eb9c4 Description - Project ID b1bf4df41dc34e3ab5c3b0318158263d Status Active Locked False Availability Zone nova Created Sept. 1, 2021, 12:24 p.m. Age 5 hours, 9 minutes Host Meghctrl1 Specs ________________________________________ Flavor Name Medium Flavor ID 5bb6bc50-b2f7-49b2-8f01-7a631af5ff8c RAM 4GB VCPUs 2 VCPU Disk 150GB IP Addresses ________________________________________ Tenant_NW 172.16.100.248 Security Groups ________________________________________ default • ALLOW IPv6 from default • ALLOW IPv4 icmp to 0.0.0.0/0 • ALLOW IPv4 to 0.0.0.0/0 • ALLOW IPv4 tcp from 0.0.0.0/0 • ALLOW IPv4 icmp from 0.0.0.0/0 • ALLOW IPv4 from default • ALLOW IPv6 to ::/0 Metadata ________________________________________ Key Name TNSDC Image Name CMDALocalDBWindows1 Image ID 6437fde7-db0c-4298-9d92-1f70aae3c894 Volumes Attached ________________________________________ Volume No volumes attached. Details of the  CLI Created disk Volume details  which is unable to attach to the running Windows 2012 RC instance is as follows from Horizon GUI CMDA_DB_SecDisk_Vol Edit Volume • Overview • Snapshots Name CMDA_DB_SecDisk_Vol ID 32f9452f-368f-4f26-b70d-c066c4c7e01c Project ID b1bf4df41dc34e3ab5c3b0318158263d Status Available Group - Specs ________________________________________ Size 600 GiB Type __DEFAULT__ Bootable No Encrypted No Created Sept. 1, 2021, 4:07 p.m. Attachments ________________________________________ Attached To Not attached Volume Source ________________________________________ Image Metadata ________________________________________ None What am I doing wrong here  ? :  any hints or suggestions  to add this data disk to the running WIndows instance are most welcome.  Or do I need to add the volume to the instance  through CLI,  from the docs  I am seeing this.   Will this  work for  Running Windows VM instances also  ? openstack server add volume 84c6e57d-a6b1-44b6-81eb-fcb36afd31b5 \ 573e024d-5235-49ce-8332-be1576d323f8 --device /dev/vdb Will you advice to  attaching the Volume  through CLI  like the following   #openstack server add volume 6630c9c8-135e-45b7-958b-c563337eb9c4        32f9452f-368f-4f26-b70d-c066c4c7e01c  --device  /dev/hdb    Will  it  solve this volume attachment issue ? or seriously  am I missing any crucial parameters or steps for attaching the additional disk volumes for Windows VMs? Any hints most welcome: Any other information  or  specific log files I need to post,  kindly tell me know  the log file names  I can post .  I am doing it for the first time.  Thanks in advance for your answers. Kris You might consider making sure that these additional images aren't marked as bootable. That said... Have you considered importing the disks directly into Cinder instead? Thank you, Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos at PerformAir.com www.PerformAir.com From: KK CHN [mailto:kkchn.in at gmail.com] Sent: Thursday, August 26, 2021 10:04 PM To: Dominic Hilsbos Subject: Re: Windows2012 VM image importing and Instance Launch Failure Hi Dominic, For Importing Single disk Windows VM   I performed the following steps on the  vhdx to qcow2 converted image Step 1: root at Meghctrl1:/home/cloud# openstack image create  “M_Windows_VM_NAME_Blaah”    --file   Windows_VM_disk.qcow2    --disk-format  qcow2   --container-format    bare       --public Step 2: root at Meghctrl1:/home/cloud# openstack image set    --property   hw_firmware_type=uefi   --property os_secure_boot=required    My_Windows_VM_NAME_Blaah  Step 3:  root at Meghctrl1:/home/cloud# openstack image set     --property    hw_firmware_type=uefi      --property    hw_disk_bios=ide     MY_Windows_VM_NAME_Blaah And I am able to see the image in the Horizon dashboard   "image"   tab  and  launch the instance from there . Everything went fine and I am able to get the Windows Single Disk VM running.  Query: If I have a Multi Disc  Windows VM ,   do I need to perform all the above three  steps for  all the   qcow2 converted  images ?  ( For eg:  I have a  Database Server ,  A Windows VM already running in HyperV with two disks  ( 100GB   disk   and 600 GB another disk image) .     So in these two  disks   one will be WIndows OS and another will be attached disk without Windows OS   right ?   1.  For the first disk I will perform all the  above three steps after converting it  from vhdx to  qcow2.  So that image will appear in the horizon and I can launch an instance as I did early for Single Disk Windows VM.        2.  For the 600 GB second disk (vhdx format) I will convert it to qcow2  and then  do I need to perform  all the above three steps exactly ?  Or what changes I need to make in the above steps  to import this disk as this is an additional attached disk in hyperV. Request your guidance or any suggestions in this regard.  Thank You, Kris On Wed, Aug 25, 2021 at 9:12 PM wrote: Kris;   This shouldn’t be all that much more difficult than Linux, nor that much harder than a single-disk Windows VM.   I would start by disabling the SQL service, as the OpenStack instance will boot once before you can add the data disks (at least it does for me when using Horizon).   Export the disks, convert then, and import them, all as you normally would.  Create the instance, as you normally would.  The instance will start; shut it back down.  Attach the additional volumes, and start the instance back up.   It would be nice to have an option to not start the instance after creation.   You could also investigate the “openstack server create” command; it has a --volume argument that you may be able to pass more than once, though I suspect not.   Thank you,   Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos at PerformAir.com www.PerformAir.com     From: KK CHN [mailto:kkchn.in at gmail.com] Sent: Tuesday, August 24, 2021 11:19 PM To: openstack-discuss at lists.openstack.org Subject: Re: Windows2012 VM image importing and Instance Launch Failure   Thank you all.    I am able to get the Single disk Windows VM  exported from HyperV  and  imported to OpenStack(Ussuri, Glance and Qemu KVM )  Instance launched and its showing the status running  in the PowerState Tab of Horizon dashboard.    (need to check with the administrator user to login and confirm the VM is working as expected . )     My second phase :    I need to perform  importing   multiple disk   Windows VM to   my OpenStack setup  .   ( For single disk  Windows vhdx file, I 've converted it to qcow2 file and imported to OpenStack as the steps  I posted in my  previous emails. )   Now how can I   import Multiple disk Windows VM  to  OpenStack.       First Step  is  to convert the  vhdx files exported from HyperV to  qcow2  files.    After this step ,   what steps need to be performed   to bring a multi disk Windows VM to  OpeStack ?   For Eg:  I have a Windows VM       1. with  Application server  1TB single disk image exported from hyperV.  This I can import to  my OpenStack setup with the steps I followed in my earlier emails as it is a single disk image.     2.  A Database server for the above application with   700 GB  in multiple disks.  ( 100 and 600 GB disks images exported from HyperV  and in vhdx format  for DB server)   How to bring this server imported to   my  OpeStack setup (ussuri, glance and QEMU KVM).   As this is from Windows  HyperV  I  am new to multidisk environment from Windows. ( for Linux VMs exported from oVirt(RHVM) I am able to import multidisk  VMs to OpenStack as they are in LVM and I have to mount it with VG name and adding entries in /etc/fstab file )   But in Windows multi disks how to perform this ?  what steps I have to perform to import this to OpenStack.    Any hints or suggestions most welcome..   Kris     On Tue, Aug 24, 2021 at 1:40 PM Eugen Block wrote: Of course you need a running nova-conductor, how did you launch VMs before? Zitat von KK CHN : > yes. I too suspect this not the problem with image.   I have seen the logs: > it throwing Errors which pasted here. > > Its waiting for nova-conductor service as in the logs.   Is this the issue? > Do I need to explictly statrt conductor service or some other service ? > nova-compute logs pasted below the lines.   share your thoughts what may > the issue. > > > On Fri, Aug 20, 2021 at 8:11 AM Mohammed Naser wrote: > >> I suggest that you have a look at the compute node logs when it fails to >> spawn. I suspect the problem is not Your images but something inside >> openstack >> >> > tail: no files remaining > cloud at Meghctrl1:~$ sudo tail -f /var/log/nova/nova-compute.log > [sudo] password for cloud: > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > transport_options=transport_options) > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > File  > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 642, in _send > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > call_monitor_timeout) > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > File  > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 531, in wait > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > message = self.waiters.get(msg_id, timeout=timeout) > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > File  > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 409, in get > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > 'to messageID %s' % msg_id) > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a > reply to message ID 7a4e3a4acad3403a9966570cc259f7b7 > 2021-08-19 11:59:59.477 2847961 ERROR oslo_service.periodic_task > 2021-08-19 12:00:01.467 2847961 WARNING oslo.service.loopingcall [-] > Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run > outlasted interval by 50.03 sec > 2021-08-19 12:00:34.012 2847812 WARNING nova.conductor.api > [req-afca4d09-bf9c-4bbc-ae88-22008d12b978 - - - - -] Timed out waiting > for nova-conductor.  Is it running? Or did this service start before > nova-conductor?  Reattempting establishment of nova-conductor > connection...: oslo_messaging.exceptions.MessagingTimeout: Timed out > waiting for a reply to message ID 9f5b593361b34c8d91585c00e0514047 > 2021-08-19 12:00:59.480 2847961 DEBUG oslo_service.periodic_task > [req-58a0e182-6e56-4695-b4db-75624ca69c7b - - - - -] Running periodic > task ComputeManager._check_instance_build_time run_periodic_tasks > /usr/local/lib/python3.7/dist-packages/oslo_service/periodic_task.py:211 > 2021-08-19 12:00:59.481 2847961 DEBUG oslo_service.periodic_task > [req-58a0e182-6e56-4695-b4db-75624ca69c7b - - - - -] Running periodic > task ComputeManager._sync_scheduler_instance_info run_periodic_tasks > /usr/local/lib/python3.7/dist-packages/oslo_service/periodic_task.py:211 > 2021-08-19 12:01:01.499 2847961 WARNING oslo.service.loopingcall [-] > Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run > outlasted interval by 50.03 sec > 2021-08-19 12:01:34.040 2847812 WARNING nova.conductor.api > [req-afca4d09-bf9c-4bbc-ae88-22008d12b978 - - - - -] Timed out waiting > for nova-conductor.  Is it running? Or did this service start before > nova-conductor?  Reattempting establishment of nova-conductor > connection...: oslo_messaging.exceptions.MessagingTimeout: Timed out > waiting for a reply to message ID e8e478be640e4ecab8a3d27c01960440 > > > essage ID d620b1b3938e4428a0500c00df8d68ee > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > [req-58a0e182-6e56-4695-b4db-75624ca69c7b - - - - -] Error during > ComputeManager._cleanup_expired_console_auth_tokens: > oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a > reply to message ID a493182a404242b79c8f11f8ec350e36 > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > Traceback (most recent call last): > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File  > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 405, in get > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > return self._queues[msg_id].get(block=True, timeout=timeout) > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File "/usr/local/lib/python3.7/dist-packages/eventlet/queue.py", line > 322, in get > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > return waiter.wait() > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File "/usr/local/lib/python3.7/dist-packages/eventlet/queue.py", line > 141, in wait > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > return get_hub().switch() > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File "/usr/local/lib/python3.7/dist-packages/eventlet/hubs/hub.py", > line 298, in switch > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > return self.greenlet.switch() > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task _queue.Empty > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > During handling of the above exception, another exception occurred: > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > Traceback (most recent call last): > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File "/usr/local/lib/python3.7/dist-packages/oslo_service/periodic_task.py", > line 216, in run_periodic_tasks > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > task(self, context) > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File "/usr/local/lib/python3.7/dist-packages/nova/compute/manager.py", > line 10450, in _cleanup_expired_console_auth_tokens > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > objects.ConsoleAuthToken.clean_expired_console_auths(context) > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File "/usr/local/lib/python3.7/dist-packages/oslo_versionedobjects/base.py", > line 177, in wrapper > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > args, kwargs) > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File "/usr/local/lib/python3.7/dist-packages/nova/conductor/rpcapi.py", > line 243, in object_class_action_versions > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > args=args, kwargs=kwargs) > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/client.py", > line 179, in call > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > transport_options=self.transport_options) > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/transport.py", > line 128, in _send > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > transport_options=transport_options) > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File  > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 654, in send > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > transport_options=transport_options) > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File  > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 642, in _send > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > call_monitor_timeout) > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File  > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 531, in wait > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > message = self.waiters.get(msg_id, timeout=timeout) > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > File  > "/usr/local/lib/python3.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", > line 409, in get > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > 'to message ID %s' % msg_id) > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a > reply to message ID a493182a404242b79c8f11f8ec350e36 > 2021-08-19 12:27:00.387 2847961 ERROR oslo_service.periodic_task > 2021-08-19 12:27:01.681 2847961 DEBUG oslo_service.periodic_task > [req-58a0e182-6e56-4695-b4db-75624ca69c7b - - - - -] Running periodic > task ComputeManager._check_instance_build_time run_periodic_tasks > /usr/local/lib/python3.7/dist-packages/oslo_service/periodic_task.py:211 > 2021-08-19 12:27:01.687 2847961 DEBUG oslo_service.periodic_task > [req-58a0e182-6e56-4695-b4db-75624ca69c7b - - - - -] Running periodic > task ComputeManager._sync_scheduler_instance_info run_periodic_tasks > /usr/local/lib/python3.7/dist-packages/oslo_service/periodic_task.py:211 > 2021-08-19 12:27:02.368 2847961 WARNING oslo.service.loopingcall [-] > Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run > outlasted interval by 50.03 sec > > > > Error-nov-log-file_while_sch ... e_in _horizaon_dashboard.txt > Open with > Displaying Error-nov-log-file_while_scheduling_Windows_VM_instance_in > _horizaon_dashboard.txt. > > > > > >> If I was to guess it’s probably missing UEFI firmware packages :) >> >> On Wed, Aug 18, 2021 at 9:17 AM KK CHN wrote: >> >>> Error : failed to perform requested operation on instance "WindowsVM "the >>> instance has error status. Please try again later [Error exceeded maximum >>> number of retries. Exhausted all hosts available for retrying build >>> failures for instance e3d5c095-7d26-4b1e-89d1-d1a6e20a45041 >>> >>> I am trying to import a WIndows2012 Single disk VM, to OpenStack Ussuri, >>> with glance and Qemu KVM. >>> >>> In bare machine KVM I am able to import and boot this Windows VM which >>> exported from rhevm hypervisor as  vhdx image. >>> what I have done is >>> >>> 1. converted this windows  image from vhdx to qcow2 >>> 2.  root at MeghCtrol1:/home/cloud/CMOBB_APP#cirt-install --name WINDOWS >>> --ram=1048 --vcups=1 --cpu host --hvm --dick >>> path=BackUP2_CMAPP_disk_1_Windows_qcow2_imagefile,device=disk, >>> format=qcow2,bus=virtio --graphics vnc --boot uefi >>> >>> This uploaded the qcow2 image of WindowsVM to  KVM  hypervisor and its >>> working. >>> >>> But when I do importing to openstack   unable to launch  instance from >>> the image . >>> >>> These are the steps I performed.. >>> >>> 1. openstack image create "WindowsVM" --file CMAPP_disk_1.qcow2 >>> --disk-format qcow2 --container-format bare --public >>> >>> 4.openstack image set --property hw_firmware_type=uefi --property >>> os_secure_boot=required WindowsVM >>> >>> 5.openstack image set --property hw_firmware_type=uefi --property >>> hw_disk_bus=ide WindowsVM >>> >>> 6.openstack image show WindowsVM >>> >>> 7. root at dmzcloud:/home/cloud# openstack image show WindowsVM|grep >>> "properties" >>> | properties       | hw_disk_bus='ide', hw_firmware_type='uefi', >>> os_hash_algo='sha512', >>> >>> os_hash_value='753ee596980409e1e72d6d020c8219c56a6ada8b43f634fb575c594a245725a398e45982c0a1ad72b3fc3451cde62cceb9ff22be044863b31ecdd7893b049349', >>> os_hidden='False', os_secure_boot='required', >>> owner_specified.openstack.md5='', >>> owner_specified.openstack.object='images/WindowsVM', >>> owner_specified.openstack.sha256='' | >>> root at dmzcloud:/home/cloud# >>> >>> >>> Then  I logged into horizon dashboard,  from the images  selected the >>> imported image and try to launch the instance.  With  a Flavour of 550 GB >>> disk, 4 vcpus and 8GB Ram .. >>> >>> The instance spawning ran for 30 minutes and throws the error which I >>> pasted first in the right top corner of horizon dashboard. >>> >>> How to solve this error and boot the Windows machine successfully .. >>> >>> >>> """ >>> Error : failed to perform requested operation on instance "WindowsVM "the >>> instance has error status. Please try again later [Error exceeded maximum >>> number of retries. Exhausted all hosts available for retrying build >>> failures for instance e3d5c095-7d26-4b1e-89d1-d1a6e20a45041 >>> >>> >>> """ >>> >>> Any help highly appreciated. >>> >>> Kris >>> >>> -- >> Mohammed Naser >> VEXXHOST, Inc. >> From radoslaw.piliszek at gmail.com Wed Sep 1 17:28:16 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Wed, 1 Sep 2021 19:28:16 +0200 Subject: [kolla-ansible] [kuryr] Running Docker Swarm with Kuryr networking In-Reply-To: References: <7132cbfa-c8be-76c7-3f77-5027f74eff2e@celati.com> Message-ID: On Wed, Sep 1, 2021 at 6:08 PM Paolo Celati wrote: > > Hi Michal, > > Yes I would rather run all my Docker containers on the physical hosts. At the moment I run Ceph and kolla images, but adding Swarm as well would be useful. > > I've had a fair number of problems with Magnum because it appears only Kubernetes is supported nowadays, and I have 0 experience with that. I also prefer Swarm because it's a lot simpler for small scale. > > Administering my own VMs with Docker Swarm on top is also not exactly the best solution because that introduces extra machines to maintain. And I don't have lots of spare RAM either as it's a homelab. > > Thanks for reminding me about tagging correctly, I forgot to put [kuryr]. Ok didn't know about the lack of CI but good to know. We actually *do* test Kuryr in CI - in the Zun scenario - and it works, at least on CentOS, it fails on Ubuntu for some reason but we don't have anyone to take care of that... Zun works using the on-host Docker as you want to do. *But* as Michał Dulko (hah, had to add the surname to differentiate ;-) ) said non-Kubernetes Kuryr is largely unmaintained and, actually, a similar statement applies to Zun... Your mileage may vary but going forward I advise you take the time to learn Kubernetes. As for mixing up Docker Swarm with Kolla Ansible, that's not really supported either. Kolla Ansible is designed to deploy to a bunch of standalone Docker daemons using only host networking. It could work but it's not tested. -yoctozepto > > Kind regards, > > Paolo > > > On 01/09/2021 15:57, Michał Nasiadka wrote: > > Hi Paolo, > > Would you like to use the Docker engine that is running on the OpenStack cluster hosts, or create Virtual Machines that will be used for a Docker Swarm cluster? > I would propose the latter. > > About Kuryr - we don’t have CI coverage for testing Kuryr in Kolla-Ansible deployment, so the container images and Ansible deployment role are provided as-is currently. > > Maybe somebody from Kuryr project could help you out? Adding [kuryr] tag for visibility. > > Best regards, > > Michal > On 1 Sep 2021, 00:32 +0200, Paolo Celati , wrote: > > Hi, > > long story short I have a 3 node Openstack cluster that I manage with kolla-ansible, and I'd like to run Docker Swarm on that as well. I am aware Magnum exists, but I'd first like to get my head around this simpler case. > > Seeing as I'd like to connect Docker containers from swarm compose files to Neutron networks I'm trying to set up Kuryr together with a swarm configuration. However the documentation is a little scarce and I'd prefer running everything on these three hosts, including etcd. If I follow the guide and pass --cluster-store and --cluster-advertise arguments to dockerd then I can't run Docker in Swarm mode because I get an error saying Swarm is incompatible with those options, and at the same time it's not clear from documentation how you are expected to do Kuryr+Swarm. I did initialise the Swarm cluster before trying to add Kuryr, so I don't know if perhaps doing this the other way works? Do you have ideas or advice with this scenario? If worst comes to worst I can set up an external etcd cluster on a separate non-Openstack cluster but I'd rather avoid that. > > > Thanks in advance, > > Paolo From gmann at ghanshyammann.com Wed Sep 1 20:41:53 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 01 Sep 2021 15:41:53 -0500 Subject: [all][tc] Technical Committee next weekly meeting on Sept 2nd at 1500 UTC In-Reply-To: <17b9d29598c.1049f1a9c135562.7894740217587346102@ghanshyammann.com> References: <17b9d29598c.1049f1a9c135562.7894740217587346102@ghanshyammann.com> Message-ID: <17ba318be97.cc9f4e3e210522.5438255807862395887@ghanshyammann.com> Hello Everyone, Below is the agenda for Tomorrow's TC meeting schedule at 1500 UTC in #openstack-tc IRC channel. NOTE: Tomorrow meeting will be a Video call at google meet, joining information are present in https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting == Agenda for tomorrow's TC meeting == * Roll call * Follow up on past action items * Gate health check (dansmith/yoctozepto) ** http://paste.openstack.org/show/jD6kAP9tHk7PZr2nhv8h/ * Leaderless projects ** https://etherpad.opendev.org/p/yoga-leaderless * New project application: 'Venus' ** http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019748.html ** https://review.opendev.org/c/openstack/governance/+/804824 * Next step on pain points from projects/SIGs/pop-up (ricolin) ** https://etherpad.opendev.org/p/pain-point-elimination * Propose OpenStack News: Newsletter ** https://etherpad.opendev.org/p/newsletter-openstack-news * Open Reviews ** https://review.opendev.org/q/projects:openstack/governance+is:open -gmann ---- On Tue, 31 Aug 2021 12:02:17 -0500 Ghanshyam Mann wrote ---- > Hello Everyone, > > Technical Committee's next weekly meeting is scheduled for Sept 2nd at 1500 UTC. > > If you would like to add topics for discussion, please add them to the below wiki page by > Wednesday, Sept 1st, at 2100 UTC. > > https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting > > -gmann > > From ildiko.vancsa at gmail.com Wed Sep 1 22:27:55 2021 From: ildiko.vancsa at gmail.com (Ildiko Vancsa) Date: Wed, 1 Sep 2021 15:27:55 -0700 Subject: [nova][cyborg][gpu] GPU performance testing In-Reply-To: <8E5CAEE8-19DC-416F-939E-A7E7A0DF19FD@cern.ch> References: <5b765247.6fd1.17ba15d8b43.Coremail.joykechen@163.com> <8E5CAEE8-19DC-416F-939E-A7E7A0DF19FD@cern.ch> Message-ID: <89CAE2E2-C19A-4EE6-8EF3-5373C75FFFB8@gmail.com> Hi, Thank you all for the recommendations and guidance. This is exactly the information I was looking for! :) Best Regards, Ildikó > On Sep 1, 2021, at 07:46, Tim Bell wrote: > > >> On 1 Sep 2021, at 14:37, 陈克 wrote: >> >> Hi, >> I recommend a tool to test GPU performance, which depends on CUDA installation. >> >> link: https://github.com/GPUburn/gpuburn/tree/master/GPUBurn >> > > gpuburn is also the tool that CERN uses also when new hardware is delivered and we want to check the hardware is working under load. > > It’s also a good way to stress your PDUs and cooling if you have several cards (see https://indico.cern.ch/event/950196/contributions/3993259/attachments/2113752/3556174/GPUs_in_CERN_IT.pdf) > > We also have some specific High Energy Physics benchmarks - https://gitlab.cern.ch/hep-benchmarks/hep-workloads-gpu > > Cheers > Tim > >> >> Thanks, >> Ke Chen >> >> >> 在 2021-08-31 23:10:03，"Sylvain Bauza" 写道： >> >> >> >> On Tue, Aug 31, 2021 at 4:34 PM Ildiko Vancsa wrote: >> Hi, >> >> As we are approaching the end of the holiday season I wanted to surface back my question about GPU performance testing. Does anyone have any hints to find the best tools to do some benchmarking with? >> >> >> >> You made the point, Ildiko, I was on a long-running time-off so I didn't had time to look at your question yet. >> >> Good concern tho, I have no knowledge about this, but I can ping a few other folks to get you an answer. >> -Sylvain >> >> Thanks, >> Ildikó >> >> >> > On Aug 9, 2021, at 08:46, Ildiko Vancsa wrote: >> > >> > Hi, >> > >> > I got a question about tools and practices to check GPU performance in an OpenStack environment that I need some help to answer. >> > >> > The question is about recommended GPU performance testing/benchmarking tools if there are a few that people in the community are using and would recommend? The scope of the testing work is to check GPU performance in OpenStack VMs (both virtualized and passthrough). >> > >> > All the help and pointers are very much appreciated! >> > >> > Thanks, >> > Ildikó >> > >> > >> >> >> >> >> > From zhangbailin at inspur.com Thu Sep 2 01:30:56 2021 From: zhangbailin at inspur.com (=?gb2312?B?QnJpbiBaaGFuZyjVxbDZwdYp?=) Date: Thu, 2 Sep 2021 01:30:56 +0000 Subject: [all][elections][ptl][cyborg] PTL voting for Cyborg- Bailin Zhang(brinzhang) Message-ID: <686a943fc784425ba3a6defff0fc49c9@inspur.com> Hi all. I was going to attempt to enumerate some sort of grand vision for Cyborg and our priorities for the next cycle. As I thought about it, though, I realized these remain largely the same as they always have been: provide a reliable, scalable, and efficient hardware-accelerated device management platform. In the past, we have added support for GPU, FPGA, NVMe SSD, and will add support for AEP, SPDK and other devices in the future. In order to increase the ability to interact with the VM, we were completed the feature of nova-cyborg interaction in Ussuri release. The exact details of how we do that may vary somewhat from cycle to cycle, and I was completed the support of nova rebuild/evacuate server and shelve/unshelve in the last two releases, which greatly improved the convenience of user operations. Such as the suspend and resume, the PoC code you can review https://review.opendev.org/c/openstack/nova/+/729945. And we also tried to support the code migration in the future, of course that need a spec and to talk with nova team. The log management project (Venus) team and I have completed the integration of Venus to Cyborg. We can use Venus to quickly retrieve Cyborg logs, shorten the problem location time, etc., so as to improve the efficiency of the Cyborg project. In the future, I will have in-depth exchanges with the Venus development team to continue to enhance this feature. In these two releases, I have promoted many newcomers to join the Cyborg team (such as Wenping Song, Eric Xie, fossnqiu, Zhiguang Wang, Arthur Dayne etc.), and they have made great contributions to the Cyborg project. In the future, I would like to help more people. I hope to get your support, thanks. Bailin Zhang (brinzhang) brinzhang Inspur Electronic Information Industry Co.,Ltd. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonyppe at gmail.com Thu Sep 2 03:50:44 2021 From: tonyppe at gmail.com (Tony Pearce) Date: Thu, 2 Sep 2021 11:50:44 +0800 Subject: [magnum] [kolla-ansible] [kayobe] [Victoria] Magnum Kubernetes cluster failure recovery In-Reply-To: <1701b95c-2cad-c593-e394-1825355e61ea@catalystcloud.nz> References: <85bd5e9e6b4c8e0a60f51f9fd32e170a1d4064ec.camel@mittwald.de> <1701b95c-2cad-c593-e394-1825355e61ea@catalystcloud.nz> Message-ID: Thanks Feilong and Sven. > If so, cluster resize should be able to bring the cluster back. And you can just resize the cluster to the current node number. For that case, magnum should be able to fix the heat stack. I thought this too. But when I try and run "check stack" under heat it fails. The log for this failure is that the resource is missing, ie one of the nodes is not there (which I know about). I tried the cluster resize from horizon, to resize the cluster to the valid size / current size (without the additional node failure which is not there) and horizon immediately fails this with a red error in the corner of the web page. There's no log printed within magnum or heat logs at all. And the horizon error is not really helpful with error "*Error: *Unable to resize given cluster id: 1a8e1ed9-64b3-41b1-ab11-0f01e66da1d7.". > Are you using the magnum auto healing feature by chance? The "repair unhealthy nodes" option was chosen for this I believe. But I didnt set up the cluster so I am not sure. Based on your replies, I discovered how to initiate the cluster resize using the CLI. After issuing the command, the missing node was rebuilt immediately. This then appears like some sort of issue with horizon only. I wanted to get the resized cluster operating successfully before I replied, but though it re-deployed the missing node, the cluster resize went timed out and failed. Aside from a quick 30 min investigation on this I've not been able to do much more with that and it's been abandoned. Thanks all the same for your help. Tony Pearce On Thu, 12 Aug 2021 at 05:06, feilong wrote: > Let me try to explain it from a design perspective: > > 1. Auto scaler: Now cluster auto scaler talks to Magnum resize API > directly to scale, see > > https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/magnum/magnum_manager_impl.go#L399 > > 2. Auto healer: As you know auto scaler only cares about the worker > node, it won't scale the master nodes. However, auto healer can repair > both master nodes and worker nodes. With worker nodes repairing, Magnum > auto healer uses magnum resize API. But because the magnum resize api > doesn't support master nodes resizing, so the master nodes repairing is > done by Heat stack update. magnum auto healer will mark some resources > of the master node as unhealthy, then call Heat stack update to rebuild > those resources. > > > On 11/08/21 10:25 pm, Sven Kieske wrote: > > On Mi, 2021-08-11 at 10:16 +0000, Sven Kieske wrote: > >> the problem is, that the kubernetes autoscaler directly talks to the > openstack api, e.g. > >> nova for creating and destroying instances. > > Nevermind I got that wrong. > > > > The autoscaler talks to heat, so there should no problem (but heat trips > itself up on some error conditions). > > I was in fact talking about the magnum auto healer ( > https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/magnum-auto-healer/using-magnum-auto-healer.md > ) > > which seems to circumvent heat and talks directly with nova. > > > > Are you using the magnum auto healing feature by chance? > > > > HTH > > > -- > Cheers & Best regards, > > ------------------------------------------------------------------------------ > Feilong Wang (王飞龙) (he/him) > Head of Research & Development > > Catalyst Cloud > Aotearoa's own > > Mob: +64 21 0832 6348 | www.catalystcloud.nz > Level 6, 150 Willis Street, Wellington 6011, New Zealand > > CONFIDENTIALITY NOTICE: This email is intended for the named recipients > only. > It may contain privileged, confidential or copyright information. If you > are > not the named recipient, any use, reliance upon, disclosure or copying of > this > email or its attachments is unauthorised. If you have received this email > in > error, please reply via email or call +64 21 0832 6348. > > ------------------------------------------------------------------------------ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From iurygregory at gmail.com Thu Sep 2 07:42:18 2021 From: iurygregory at gmail.com (Iury Gregory) Date: Thu, 2 Sep 2021 09:42:18 +0200 Subject: [Ironic] No weekly meeting on Sep 06 Message-ID: Hello Ironicers! During our weekly meeting this week (Aug 30) we decided to skip our next weekly meeting since it is a holiday in the US -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory at gmail.com * -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdulko at redhat.com Thu Sep 2 08:21:14 2021 From: mdulko at redhat.com (=?UTF-8?Q?Micha=C5=82?= Dulko) Date: Thu, 02 Sep 2021 10:21:14 +0200 Subject: [kolla-ansible] [kuryr] Running Docker Swarm with Kuryr networking In-Reply-To: References: <7132cbfa-c8be-76c7-3f77-5027f74eff2e@celati.com> Message-ID: On Wed, 2021-09-01 at 19:28 +0200, Radosław Piliszek wrote: > On Wed, Sep 1, 2021 at 6:08 PM Paolo Celati wrote: > > > > Hi Michal, > > > > Yes I would rather run all my Docker containers on the physical hosts. At the moment I run Ceph and kolla images, but adding Swarm as well would be useful. > > > > I've had a fair number of problems with Magnum because it appears only Kubernetes is supported nowadays, and I have 0 experience with that. I also prefer Swarm because it's a lot simpler for small scale. > > > > Administering my own VMs with Docker Swarm on top is also not exactly the best solution because that introduces extra machines to maintain. And I don't have lots of spare RAM either as it's a homelab. > > > > Thanks for reminding me about tagging correctly, I forgot to put [kuryr]. Ok didn't know about the lack of CI but good to know. > > We actually *do* test Kuryr in CI - in the Zun scenario - and it > works, at least on CentOS, it fails on Ubuntu for some reason but we > don't have anyone to take care of that... > Zun works using the on-host Docker as you want to do. I had a feeling that Zun is using kuryr-libnetwork, which is a different thing than kuryr itself. If the question if about kuryr- libnetwork, then it's maintained better and Hongbin Lu is an expert to try contacting with. Ping us on #openstack-kuryr and we'll try to get you his whereabouts. > *But* as Michał Dulko (hah, had to add the surname to differentiate > ;-) ) said non-Kubernetes Kuryr is largely unmaintained and, actually, > a similar statement applies to Zun... > Your mileage may vary but going forward I advise you take the time to > learn Kubernetes. > As for mixing up Docker Swarm with Kolla Ansible, that's not really > supported either. Kolla Ansible is designed to deploy to a bunch of > standalone Docker daemons using only host networking. > It could work but it's not tested. > > -yoctozepto > > > > > Kind regards, > > > > Paolo > > > > > > On 01/09/2021 15:57, Michał Nasiadka wrote: > > > > Hi Paolo, > > > > Would you like to use the Docker engine that is running on the OpenStack cluster hosts, or create Virtual Machines that will be used for a Docker Swarm cluster? > > I would propose the latter. > > > > About Kuryr - we don’t have CI coverage for testing Kuryr in Kolla-Ansible deployment, so the container images and Ansible deployment role are provided as-is currently. > > > > Maybe somebody from Kuryr project could help you out? Adding [kuryr] tag for visibility. > > > > Best regards, > > > > Michal > > On 1 Sep 2021, 00:32 +0200, Paolo Celati , wrote: > > > > Hi, > > > > long story short I have a 3 node Openstack cluster that I manage with kolla-ansible, and I'd like to run Docker Swarm on that as well. I am aware Magnum exists, but I'd first like to get my head around this simpler case. > > > > Seeing as I'd like to connect Docker containers from swarm compose files to Neutron networks I'm trying to set up Kuryr together with a swarm configuration. However the documentation is a little scarce and I'd prefer running everything on these three hosts, including etcd. If I follow the guide and pass --cluster-store and --cluster-advertise arguments to dockerd then I can't run Docker in Swarm mode because I get an error saying Swarm is incompatible with those options, and at the same time it's not clear from documentation how you are expected to do Kuryr+Swarm. I did initialise the Swarm cluster before trying to add Kuryr, so I don't know if perhaps doing this the other way works? Do you have ideas or advice with this scenario? If worst comes to worst I can set up an external etcd cluster on a separate non-Openstack cluster but I'd rather avoid that. > > > > > > Thanks in advance, > > > > Paolo > From radoslaw.piliszek at gmail.com Thu Sep 2 08:39:52 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Thu, 2 Sep 2021 10:39:52 +0200 Subject: [kolla-ansible] [kuryr] Running Docker Swarm with Kuryr networking In-Reply-To: References: <7132cbfa-c8be-76c7-3f77-5027f74eff2e@celati.com> Message-ID: On Thu, Sep 2, 2021 at 10:21 AM Michał Dulko wrote: > > On Wed, 2021-09-01 at 19:28 +0200, Radosław Piliszek wrote: > > On Wed, Sep 1, 2021 at 6:08 PM Paolo Celati wrote: > > > > > > Hi Michal, > > > > > > Yes I would rather run all my Docker containers on the physical hosts. At the moment I run Ceph and kolla images, but adding Swarm as well would be useful. > > > > > > I've had a fair number of problems with Magnum because it appears only Kubernetes is supported nowadays, and I have 0 experience with that. I also prefer Swarm because it's a lot simpler for small scale. > > > > > > Administering my own VMs with Docker Swarm on top is also not exactly the best solution because that introduces extra machines to maintain. And I don't have lots of spare RAM either as it's a homelab. > > > > > > Thanks for reminding me about tagging correctly, I forgot to put [kuryr]. Ok didn't know about the lack of CI but good to know. > > > > We actually *do* test Kuryr in CI - in the Zun scenario - and it > > works, at least on CentOS, it fails on Ubuntu for some reason but we > > don't have anyone to take care of that... > > Zun works using the on-host Docker as you want to do. > > I had a feeling that Zun is using kuryr-libnetwork, which is a > different thing than kuryr itself. If the question if about kuryr- > libnetwork, then it's maintained better and Hongbin Lu is an expert to > try contacting with. Ping us on #openstack-kuryr and we'll try to get > you his whereabouts. Ah, yeah. In Kolla we say "kuryr" but mean "kuryr-libnetwork". We actually install both kuryr and kuryr-libnetwork in the container and run kuryr-server. Perhaps mistakenly? I see you also have kuryr-lib for common code. The Kuryr integration was contributed by none of the current Kolla folks so we might not understand the relationships well... We will gladly accept improvements there. -yoctozepto > > *But* as Michał Dulko (hah, had to add the surname to differentiate > > ;-) ) said non-Kubernetes Kuryr is largely unmaintained and, actually, > > a similar statement applies to Zun... > > Your mileage may vary but going forward I advise you take the time to > > learn Kubernetes. > > As for mixing up Docker Swarm with Kolla Ansible, that's not really > > supported either. Kolla Ansible is designed to deploy to a bunch of > > standalone Docker daemons using only host networking. > > It could work but it's not tested. > > > > -yoctozepto > > > > > > > > Kind regards, > > > > > > Paolo > > > > > > > > > On 01/09/2021 15:57, Michał Nasiadka wrote: > > > > > > Hi Paolo, > > > > > > Would you like to use the Docker engine that is running on the OpenStack cluster hosts, or create Virtual Machines that will be used for a Docker Swarm cluster? > > > I would propose the latter. > > > > > > About Kuryr - we don’t have CI coverage for testing Kuryr in Kolla-Ansible deployment, so the container images and Ansible deployment role are provided as-is currently. > > > > > > Maybe somebody from Kuryr project could help you out? Adding [kuryr] tag for visibility. > > > > > > Best regards, > > > > > > Michal > > > On 1 Sep 2021, 00:32 +0200, Paolo Celati , wrote: > > > > > > Hi, > > > > > > long story short I have a 3 node Openstack cluster that I manage with kolla-ansible, and I'd like to run Docker Swarm on that as well. I am aware Magnum exists, but I'd first like to get my head around this simpler case. > > > > > > Seeing as I'd like to connect Docker containers from swarm compose files to Neutron networks I'm trying to set up Kuryr together with a swarm configuration. However the documentation is a little scarce and I'd prefer running everything on these three hosts, including etcd. If I follow the guide and pass --cluster-store and --cluster-advertise arguments to dockerd then I can't run Docker in Swarm mode because I get an error saying Swarm is incompatible with those options, and at the same time it's not clear from documentation how you are expected to do Kuryr+Swarm. I did initialise the Swarm cluster before trying to add Kuryr, so I don't know if perhaps doing this the other way works? Do you have ideas or advice with this scenario? If worst comes to worst I can set up an external etcd cluster on a separate non-Openstack cluster but I'd rather avoid that. > > > > > > > > > Thanks in advance, > > > > > > Paolo > > > > > From mdulko at redhat.com Thu Sep 2 08:58:01 2021 From: mdulko at redhat.com (=?UTF-8?Q?Micha=C5=82?= Dulko) Date: Thu, 02 Sep 2021 10:58:01 +0200 Subject: [kolla-ansible] [kuryr] Running Docker Swarm with Kuryr networking In-Reply-To: References: <7132cbfa-c8be-76c7-3f77-5027f74eff2e@celati.com> Message-ID: On Thu, 2021-09-02 at 10:39 +0200, Radosław Piliszek wrote: > On Thu, Sep 2, 2021 at 10:21 AM Michał Dulko wrote: > > > > On Wed, 2021-09-01 at 19:28 +0200, Radosław Piliszek wrote: > > > On Wed, Sep 1, 2021 at 6:08 PM Paolo Celati wrote: > > > > > > > > Hi Michal, > > > > > > > > Yes I would rather run all my Docker containers on the physical hosts. At the moment I run Ceph and kolla images, but adding Swarm as well would be useful. > > > > > > > > I've had a fair number of problems with Magnum because it appears only Kubernetes is supported nowadays, and I have 0 experience with that. I also prefer Swarm because it's a lot simpler for small scale. > > > > > > > > Administering my own VMs with Docker Swarm on top is also not exactly the best solution because that introduces extra machines to maintain. And I don't have lots of spare RAM either as it's a homelab. > > > > > > > > Thanks for reminding me about tagging correctly, I forgot to put [kuryr]. Ok didn't know about the lack of CI but good to know. > > > > > > We actually *do* test Kuryr in CI - in the Zun scenario - and it > > > works, at least on CentOS, it fails on Ubuntu for some reason but we > > > don't have anyone to take care of that... > > > Zun works using the on-host Docker as you want to do. > > > > I had a feeling that Zun is using kuryr-libnetwork, which is a > > different thing than kuryr itself. If the question if about kuryr- > > libnetwork, then it's maintained better and Hongbin Lu is an expert to > > try contacting with. Ping us on #openstack-kuryr and we'll try to get > > you his whereabouts. > > Ah, yeah. In Kolla we say "kuryr" but mean "kuryr-libnetwork". > We actually install both kuryr and kuryr-libnetwork in the container > and run kuryr-server. > Perhaps mistakenly? I see you also have kuryr-lib for common code. > The Kuryr integration was contributed by none of the current Kolla > folks so we might not understand the relationships well... > We will gladly accept improvements there. Yeah, it took me a while to get the relationships. :/ Here's how it works: * openstack/kuryr - Docker legacy network plugin *and* kuryr-lib source. * openstack/kuryr-libnetwork - Docker libnetwork implementation. * openstack/kuryr-kubernetes - (CNCF) CNI implementation, to be used in K8s. > -yoctozepto > > > > *But* as Michał Dulko (hah, had to add the surname to differentiate > > > ;-) ) said non-Kubernetes Kuryr is largely unmaintained and, actually, > > > a similar statement applies to Zun... > > > Your mileage may vary but going forward I advise you take the time to > > > learn Kubernetes. > > > As for mixing up Docker Swarm with Kolla Ansible, that's not really > > > supported either. Kolla Ansible is designed to deploy to a bunch of > > > standalone Docker daemons using only host networking. > > > It could work but it's not tested. > > > > > > -yoctozepto > > > > > > > > > > > Kind regards, > > > > > > > > Paolo > > > > > > > > > > > > On 01/09/2021 15:57, Michał Nasiadka wrote: > > > > > > > > Hi Paolo, > > > > > > > > Would you like to use the Docker engine that is running on the OpenStack cluster hosts, or create Virtual Machines that will be used for a Docker Swarm cluster? > > > > I would propose the latter. > > > > > > > > About Kuryr - we don’t have CI coverage for testing Kuryr in Kolla-Ansible deployment, so the container images and Ansible deployment role are provided as-is currently. > > > > > > > > Maybe somebody from Kuryr project could help you out? Adding [kuryr] tag for visibility. > > > > > > > > Best regards, > > > > > > > > Michal > > > > On 1 Sep 2021, 00:32 +0200, Paolo Celati , wrote: > > > > > > > > Hi, > > > > > > > > long story short I have a 3 node Openstack cluster that I manage with kolla-ansible, and I'd like to run Docker Swarm on that as well. I am aware Magnum exists, but I'd first like to get my head around this simpler case. > > > > > > > > Seeing as I'd like to connect Docker containers from swarm compose files to Neutron networks I'm trying to set up Kuryr together with a swarm configuration. However the documentation is a little scarce and I'd prefer running everything on these three hosts, including etcd. If I follow the guide and pass --cluster-store and --cluster-advertise arguments to dockerd then I can't run Docker in Swarm mode because I get an error saying Swarm is incompatible with those options, and at the same time it's not clear from documentation how you are expected to do Kuryr+Swarm. I did initialise the Swarm cluster before trying to add Kuryr, so I don't know if perhaps doing this the other way works? Do you have ideas or advice with this scenario? If worst comes to worst I can set up an external etcd cluster on a separate non-Openstack cluster but I'd rather avoid that. > > > > > > > > > > > > Thanks in advance, > > > > > > > > Paolo > > > > > > > > > > From ricolin at ricolky.com Thu Sep 2 09:05:27 2021 From: ricolin at ricolky.com (Rico Lin) Date: Thu, 2 Sep 2021 17:05:27 +0800 Subject: [tc][all]Please help to identify common pain points & video meeting scheduled next week In-Reply-To: References: Message-ID: Dear all For this pain point targeting, we will have this topic discussion in TC meeting today, Sept 2nd at 1500 UTC. Please join if you're interested. It will be a video call on google meet. See here for more details: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting *Rico Lin* OIF Individual Board of directors, OpenStack TC, Multi-arch SIG chair, Heat PTL, Senior Software Engineer at EasyStack On Fri, Aug 27, 2021 at 8:16 PM Rico Lin wrote: > Dear all > > First, I thank Kendall and everyone who help with the Pain point > elimination etherpad [1]. > > Two paths we plan to work on or would like to see how we can push it > forward: > > 1. To identify common pain points from OpenStack. And see if we can > target them in the following cycles. > 2. To see if we can have a community-wide goal to encourage project > teams to pick up their own pain point in the following cycles. > > *This ML is focused on the first path; To identify common pain points.* > > So I would like to encourage all to help to identify common pain points > from it and also help put your feedbacks on the etherpad [1]. > > To identify common pain points, you can help to leave comments to pain > points that you feel it is not an issue for a single project team but for a > larger scope. We kind of recognized rabbitmq (or said RPC backend) might be > one of the common pain points from that etherpad. And we wish to learn more > from all of you. And we need operators' feedback mostly. > > Also, *we plan to put `identify common pain points` on schedule for the > TC meeting next week (which is a video meeting on google meet). If you're > interested, please help with etherpad, and join the TC meeting [2] next > week.* > > [1] https://etherpad.opendev.org/p/pain-point-elimination > [2] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee > > *Rico Lin* > OIF Individual Board of directors, OpenStack TC, Multi-arch SIG chair, > Heat PTL, > Senior Software Engineer at EasyStack > -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Thu Sep 2 09:30:10 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Thu, 2 Sep 2021 11:30:10 +0200 Subject: [kolla-ansible] [kuryr] Running Docker Swarm with Kuryr networking In-Reply-To: References: <7132cbfa-c8be-76c7-3f77-5027f74eff2e@celati.com> Message-ID: On Thu, Sep 2, 2021 at 10:58 AM Michał Dulko wrote: > > On Thu, 2021-09-02 at 10:39 +0200, Radosław Piliszek wrote: > > On Thu, Sep 2, 2021 at 10:21 AM Michał Dulko wrote: > > > > > > On Wed, 2021-09-01 at 19:28 +0200, Radosław Piliszek wrote: > > > > On Wed, Sep 1, 2021 at 6:08 PM Paolo Celati wrote: > > > > > > > > > > Hi Michal, > > > > > > > > > > Yes I would rather run all my Docker containers on the physical hosts. At the moment I run Ceph and kolla images, but adding Swarm as well would be useful. > > > > > > > > > > I've had a fair number of problems with Magnum because it appears only Kubernetes is supported nowadays, and I have 0 experience with that. I also prefer Swarm because it's a lot simpler for small scale. > > > > > > > > > > Administering my own VMs with Docker Swarm on top is also not exactly the best solution because that introduces extra machines to maintain. And I don't have lots of spare RAM either as it's a homelab. > > > > > > > > > > Thanks for reminding me about tagging correctly, I forgot to put [kuryr]. Ok didn't know about the lack of CI but good to know. > > > > > > > > We actually *do* test Kuryr in CI - in the Zun scenario - and it > > > > works, at least on CentOS, it fails on Ubuntu for some reason but we > > > > don't have anyone to take care of that... > > > > Zun works using the on-host Docker as you want to do. > > > > > > I had a feeling that Zun is using kuryr-libnetwork, which is a > > > different thing than kuryr itself. If the question if about kuryr- > > > libnetwork, then it's maintained better and Hongbin Lu is an expert to > > > try contacting with. Ping us on #openstack-kuryr and we'll try to get > > > you his whereabouts. > > > > Ah, yeah. In Kolla we say "kuryr" but mean "kuryr-libnetwork". > > We actually install both kuryr and kuryr-libnetwork in the container > > and run kuryr-server. > > Perhaps mistakenly? I see you also have kuryr-lib for common code. > > The Kuryr integration was contributed by none of the current Kolla > > folks so we might not understand the relationships well... > > We will gladly accept improvements there. > > Yeah, it took me a while to get the relationships. :/ Here's how it > works: > > * openstack/kuryr - Docker legacy network plugin *and* kuryr-lib > source. > * openstack/kuryr-libnetwork - Docker libnetwork implementation. > * openstack/kuryr-kubernetes - (CNCF) CNI implementation, to be used > in K8s. Thank you! It seems we are doing the right thing then. ;-) -yoctozepto > > -yoctozepto > > > > > > *But* as Michał Dulko (hah, had to add the surname to differentiate > > > > ;-) ) said non-Kubernetes Kuryr is largely unmaintained and, actually, > > > > a similar statement applies to Zun... > > > > Your mileage may vary but going forward I advise you take the time to > > > > learn Kubernetes. > > > > As for mixing up Docker Swarm with Kolla Ansible, that's not really > > > > supported either. Kolla Ansible is designed to deploy to a bunch of > > > > standalone Docker daemons using only host networking. > > > > It could work but it's not tested. > > > > > > > > -yoctozepto > > > > > > > > > > > > > > Kind regards, > > > > > > > > > > Paolo > > > > > > > > > > > > > > > On 01/09/2021 15:57, Michał Nasiadka wrote: > > > > > > > > > > Hi Paolo, > > > > > > > > > > Would you like to use the Docker engine that is running on the OpenStack cluster hosts, or create Virtual Machines that will be used for a Docker Swarm cluster? > > > > > I would propose the latter. > > > > > > > > > > About Kuryr - we don’t have CI coverage for testing Kuryr in Kolla-Ansible deployment, so the container images and Ansible deployment role are provided as-is currently. > > > > > > > > > > Maybe somebody from Kuryr project could help you out? Adding [kuryr] tag for visibility. > > > > > > > > > > Best regards, > > > > > > > > > > Michal > > > > > On 1 Sep 2021, 00:32 +0200, Paolo Celati , wrote: > > > > > > > > > > Hi, > > > > > > > > > > long story short I have a 3 node Openstack cluster that I manage with kolla-ansible, and I'd like to run Docker Swarm on that as well. I am aware Magnum exists, but I'd first like to get my head around this simpler case. > > > > > > > > > > Seeing as I'd like to connect Docker containers from swarm compose files to Neutron networks I'm trying to set up Kuryr together with a swarm configuration. However the documentation is a little scarce and I'd prefer running everything on these three hosts, including etcd. If I follow the guide and pass --cluster-store and --cluster-advertise arguments to dockerd then I can't run Docker in Swarm mode because I get an error saying Swarm is incompatible with those options, and at the same time it's not clear from documentation how you are expected to do Kuryr+Swarm. I did initialise the Swarm cluster before trying to add Kuryr, so I don't know if perhaps doing this the other way works? Do you have ideas or advice with this scenario? If worst comes to worst I can set up an external etcd cluster on a separate non-Openstack cluster but I'd rather avoid that. > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > > > Paolo > > > > > > > > > > > > > > > > > From skaplons at redhat.com Thu Sep 2 10:52:40 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 02 Sep 2021 12:52:40 +0200 Subject: [neutron] Drivers meeting - Friday 3.09.2021 - cancelled Message-ID: <5137718.HvKhCC5Nb5@p1> Hi, Due to lack of agenda, let's cancel tomorrow's drivers meeting. See You on the meeting next week. -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From zigo at debian.org Thu Sep 2 11:25:49 2021 From: zigo at debian.org (Thomas Goirand) Date: Thu, 2 Sep 2021 13:25:49 +0200 Subject: [nova][cinder][barbican] storing secrets in service project In-Reply-To: <8ef51d37977fb2cd2d37fa6ccc69fa7864032598.camel@redhat.com> References: <8ef51d37977fb2cd2d37fa6ccc69fa7864032598.camel@redhat.com> Message-ID: <87057647-a349-082c-62a6-8f9b92f7319f@debian.org> On 8/31/21 4:17 PM, Sean Mooney wrote: > On Tue, 2021-08-31 at 16:46 +0300, Pavlo Shchelokovskyy wrote: >> Hi all, >> >> When Barbican is involved, and services like Nova and Cinder create secrets >> behind the scenes themselves, >> this results in problems deleting or otherwise managing some resources by >> cloud admins. > this is mostly by design. > cloud admin shoudl not be able to access tenat secret or teants cannot trust the > cloud. access the secret != delete a secret ... >> For example as described in https://bugs.launchpad.net/cinder/+bug/1775372, >> admin can not delete an encrypted volume created by another user, >> and needs to resort to adding herself to the project first with appropriate >> roles. > yep although i woudl argue that unless that is an abuse of admin privialges. > there are some case where the admin woudl have to intervient like this but > its generally agaisnt the self service model for admin to be doing this. It's perfectly legitimate for an admin to delete *ANY* resource if he feels he needs to. For example on a public cloud: after the grace period expired. Example for an internal cloud: when an employee leaves. >> I think the third option is the best one, but I may be missing the actual >> use case meant behind this feature. > i am not sure i agree. > > we intentionally do not allow admin to start instance that have encypted storage > as the admin should not be able to access the decrypted data. > yes they could always hack around this but to me this is intentioaly not supported > and not a bug. It is a bug that we can't live-migrate LUKS volumes. Please do recognize this as an OpenStack defect, because if you don't, you wont recognize we need to fix the situation, which is kind of super bad. >> Is it "keep the data encrypted at rest and in transit" or more strictly >> "encrypt data of different users with different keys >> and limit access to those keys" (for the case of breach etc) ? >> In the first case it seems storing all the auto-generated secrets in a >> service project would suffice... > its both the intent is to keep the data encyprted at rest and in transit and ensur that admin > cannot decypt it. deletetion of hte resouce is perhaps somethign that could be allowable with a sytem admin token > retrivial of the secreate e.g. to start the vm or do a snapthot for exmaple i think would be a regression in security > and not something we shoudl do. We you can't start the VM, it means you can't live-migrate. So there's a big issue to tackle here. I don't agree that it's a big issue that an admin can access secrets, because it's already the case (ie: by adding the creator role in that project). If you don't trust your admin: go away, it's game over anyways. As for LUKS devices, it'd be trivial for an admin to dump the VM's memory and access the decryption key anyways... So you're only being annoying to the admins, and not protecting anyone here. Cheers, Thomas Goirand (zigo) From smooney at redhat.com Thu Sep 2 11:45:39 2021 From: smooney at redhat.com (Sean Mooney) Date: Thu, 02 Sep 2021 12:45:39 +0100 Subject: [nova][cinder][barbican] storing secrets in service project In-Reply-To: <87057647-a349-082c-62a6-8f9b92f7319f@debian.org> References: <8ef51d37977fb2cd2d37fa6ccc69fa7864032598.camel@redhat.com> <87057647-a349-082c-62a6-8f9b92f7319f@debian.org> Message-ID: On Thu, 2021-09-02 at 13:25 +0200, Thomas Goirand wrote: > On 8/31/21 4:17 PM, Sean Mooney wrote: > > On Tue, 2021-08-31 at 16:46 +0300, Pavlo Shchelokovskyy wrote: > > > Hi all, > > > > > > When Barbican is involved, and services like Nova and Cinder create secrets > > > behind the scenes themselves, > > > this results in problems deleting or otherwise managing some resources by > > > cloud admins. > > this is mostly by design. > > cloud admin shoudl not be able to access tenat secret or teants cannot trust the > > cloud. > > access the secret != delete a secret ... > > > > For example as described in https://bugs.launchpad.net/cinder/+bug/1775372, > > > admin can not delete an encrypted volume created by another user, > > > and needs to resort to adding herself to the project first with appropriate > > > roles. > > yep although i woudl argue that unless that is an abuse of admin privialges. > > there are some case where the admin woudl have to intervient like this but > > its generally agaisnt the self service model for admin to be doing this. > > It's perfectly legitimate for an admin to delete *ANY* resource if he > feels he needs to. For example on a public cloud: after the grace period > expired. Example for an internal cloud: when an employee leaves. > > > > I think the third option is the best one, but I may be missing the actual > > > use case meant behind this feature. > > i am not sure i agree. > > > > we intentionally do not allow admin to start instance that have encypted storage > > as the admin should not be able to access the decrypted data. > > yes they could always hack around this but to me this is intentioaly not supported > > and not a bug. > > It is a bug that we can't live-migrate LUKS volumes. Please do recognize > this as an OpenStack defect, because if you don't, you wont recognize we > need to fix the situation, which is kind of super bad. live-migration should actully work i belive for ceph. i think its just if the volume is host mounted like we do with isci that it fails. cold migration will partly work you just wont be able to start the instance so you can cold migrate a shut off instance. same for evacuate. > > > > Is it "keep the data encrypted at rest and in transit" or more strictly > > > "encrypt data of different users with different keys > > > and limit access to those keys" (for the case of breach etc) ? > > > In the first case it seems storing all the auto-generated secrets in a > > > service project would suffice... > > its both the intent is to keep the data encyprted at rest and in transit and ensur that admin > > cannot decypt it. deletetion of hte resouce is perhaps somethign that could be allowable with a sytem admin token > > retrivial of the secreate e.g. to start the vm or do a snapthot for exmaple i think would be a regression in security > > and not something we shoudl do. > > We you can't start the VM, it means you can't live-migrate. > sure but if its not running you can cold migrate it. > So there's a > big issue to tackle here. > not really you should not be starting the instance in this case. if its off you should just cold migrate. that work in more cases the live migrate and is more effcinet since we wont have to copy any guest memory. > I don't agree that it's a big issue that an > admin can access secrets, because it's already the case (ie: by adding > the creator role in that project).  > > If you don't trust your admin: go > away, it's game over anyways. > this is hte how point of secure comptueing and the sev/image incryption work that peopel have been tryign to add to nova. the ablity to upload encypted images and use them to create encypted volumes that the admin cant access which can be verifyed using vtpm and other methods is becoming more and more a requirement for many government customers. > As for LUKS devices, it'd be trivial for > an admin to dump the VM's memory and access the decryption key > anyways... So you're only being annoying to the admins, and not > protecting anyone here. this is why SEV support was intoduced to prevent admin form doing that. its not about being annoying to admins its about ensureing we can actuly use openstack in secure enviroment so that we can continue to support our customer that need a miniuma or zero trust solution. > > Cheers, > > Thomas Goirand (zigo) > From paolo at celati.com Thu Sep 2 12:20:26 2021 From: paolo at celati.com (Paolo Celati) Date: Thu, 2 Sep 2021 14:20:26 +0200 Subject: [kolla-ansible] [kuryr] Running Docker Swarm with Kuryr networking In-Reply-To: References: <7132cbfa-c8be-76c7-3f77-5027f74eff2e@celati.com> Message-ID: On 01/09/2021 19:28, Radosław Piliszek wrote: > On Wed, Sep 1, 2021 at 6:08 PM Paolo Celati wrote: >> >> Hi Michal, >> >> Yes I would rather run all my Docker containers on the physical hosts. At the moment I run Ceph and kolla images, but adding Swarm as well would be useful. >> >> I've had a fair number of problems with Magnum because it appears only Kubernetes is supported nowadays, and I have 0 experience with that. I also prefer Swarm because it's a lot simpler for small scale. >> >> Administering my own VMs with Docker Swarm on top is also not exactly the best solution because that introduces extra machines to maintain. And I don't have lots of spare RAM either as it's a homelab. >> >> Thanks for reminding me about tagging correctly, I forgot to put [kuryr]. Ok didn't know about the lack of CI but good to know. > > We actually *do* test Kuryr in CI - in the Zun scenario - and it > works, at least on CentOS, it fails on Ubuntu for some reason but we > don't have anyone to take care of that... > Zun works using the on-host Docker as you want to do. > *But* as Michał Dulko (hah, had to add the surname to differentiate > ;-) ) said non-Kubernetes Kuryr is largely unmaintained and, actually, > a similar statement applies to Zun... > Your mileage may vary but going forward I advise you take the time to > learn Kubernetes. > As for mixing up Docker Swarm with Kolla Ansible, that's not really > supported either. Kolla Ansible is designed to deploy to a bunch of > standalone Docker daemons using only host networking. > It could work but it's not tested. > > -yoctozepto > Hi Radoslaw, Ok thanks I'll look into Kubernetes then, it looks like everything is going in that direction anyways so might as well. Well what I was intending to do with Docker Swarm is really just share the hosts running Kolla containers and Swarm containers. It makes for a lot less management work to have fewer Docker daemons. My idea was to have Kolla containers use host networking and Swarm uses overlay (vxlan) networks via Kuryr. But I'll probably try getting Magnum + Kubernetes up instead since that seems the best supported scenario. Paolo >> >> Kind regards, >> >> Paolo >> >> >> On 01/09/2021 15:57, Michał Nasiadka wrote: >> >> Hi Paolo, >> >> Would you like to use the Docker engine that is running on the OpenStack cluster hosts, or create Virtual Machines that will be used for a Docker Swarm cluster? >> I would propose the latter. >> >> About Kuryr - we don’t have CI coverage for testing Kuryr in Kolla-Ansible deployment, so the container images and Ansible deployment role are provided as-is currently. >> >> Maybe somebody from Kuryr project could help you out? Adding [kuryr] tag for visibility. >> >> Best regards, >> >> Michal >> On 1 Sep 2021, 00:32 +0200, Paolo Celati , wrote: >> >> Hi, >> >> long story short I have a 3 node Openstack cluster that I manage with kolla-ansible, and I'd like to run Docker Swarm on that as well. I am aware Magnum exists, but I'd first like to get my head around this simpler case. >> >> Seeing as I'd like to connect Docker containers from swarm compose files to Neutron networks I'm trying to set up Kuryr together with a swarm configuration. However the documentation is a little scarce and I'd prefer running everything on these three hosts, including etcd. If I follow the guide and pass --cluster-store and --cluster-advertise arguments to dockerd then I can't run Docker in Swarm mode because I get an error saying Swarm is incompatible with those options, and at the same time it's not clear from documentation how you are expected to do Kuryr+Swarm. I did initialise the Swarm cluster before trying to add Kuryr, so I don't know if perhaps doing this the other way works? Do you have ideas or advice with this scenario? If worst comes to worst I can set up an external etcd cluster on a separate non-Openstack cluster but I'd rather avoid that. >> >> >> Thanks in advance, >> >> Paolo -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_0x6A5811658B827BE4.asc Type: application/pgp-keys Size: 3123 bytes Desc: OpenPGP public key URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: From micou12 at gmail.com Thu Sep 2 13:33:05 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Thu, 2 Sep 2021 15:33:05 +0200 Subject: Fwd: [ceph-users] Re: Replacing swift with RGW In-Reply-To: References: <20210901122221.Horde.v0FeoCUfex4WYSaXReGzV1i@webmail.nde.ag> <20210901134023.Horde.ZhZ0JBty-nYNsBfGe6WOijH@webmail.nde.ag> <20210902071439.Horde.4cayqDV75oyYS75eyQZMSPD@webmail.nde.ag> <20210902081741.Horde.Xp06qPUkci05sKXFlo1F5ap@webmail.nde.ag> <20210902083925.Horde.a0LD9j43RkGPIl7UuWCRtyu@webmail.nde.ag> Message-ID: ---------- Forwarded message --------- From: Michel Niyoyita Date: Thu, Sep 2, 2021 at 12:17 PM Subject: Fwd: [ceph-users] Re: Replacing swift with RGW To: ---------- Forwarded message --------- From: Eugen Block Date: Thu, Sep 2, 2021 at 10:39 AM Subject: Re: [ceph-users] Re: Replacing swift with RGW To: Michel Niyoyita You should continue this thread on the openstack-discuss mailing list (openstack-discuss at lists.openstack.org) since this is not a ceph issue. I'm not familiar with kolla and I don't know the requirements to install openstack-swift, you'll need to ask the openstack community. Zitat von Michel Niyoyita : > Below are errors I am getting once I try to run swift commands . the second > one is the error I get once try to install python-swiftclient > > (kolla-open) [stack at kolla-open ~]$ swift -v stat > -bash: swift: command not found > (kolla-open) [stack at kolla-open ~]$ sudo yum -y install python-swiftclient > Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 09:21:53 AM > CAT. > No match for argument: python-swiftclient > Error: Unable to find a match: python-swiftclient > (kolla-open) [stack at kolla-open ~]$ > > Waiting for your help > > On Thu, Sep 2, 2021 at 10:17 AM Eugen Block wrote: > >> I can't tell for sure, but yes, I believe you need the openstack-swift >> package (with dependencies). What errors do you get? The more >> information you share the better people can help. >> >> >> Zitat von Michel Niyoyita : >> >> > I tried to install "sudo yum -y install python-swiftclient" on openstack >> > side but fails . are there openastack-shwift packages which are needed? >> > if are there please help me to get . may be also it is the cause I am >> > failing to run swift command on openstack cli side. >> > >> > thank you for your continued support. >> > >> > Micheal >> > >> > On Thu, Sep 2, 2021 at 9:14 AM Eugen Block wrote: >> > >> >> I only configured the endpoints for the clients to directly access the >> >> RGWs, but you'll probably need to install the openstack-swift package. >> >> Or have you done that already? >> >> >> >> >> >> Zitat von Michel Niyoyita : >> >> >> >> > Thank you Eugen for your prompt response. >> >> > >> >> > Now the commands provided work. but I am not finding the object >> storage >> >> on >> >> > the horizon dashboard , but it appears in the system information >> >> services. >> >> > [image: image.png] >> >> > so my question is how to configure it in order that it can appear in >> the >> >> > dashboard . >> >> > >> >> > Michel >> >> > >> >> > On Wed, Sep 1, 2021 at 3:49 PM Eugen Block wrote: >> >> > >> >> >> Sorry, one little detail slipped through, the '--region' flag has to >> >> >> be put before the 'service' name. The correct command would be: >> >> >> >> >> >> openstack endpoint create --region RegionOne swift admin >> >> >> http://ceph-osd3:8080/swift/v1 >> >> >> >> >> >> and respectively for the other interfaces. >> >> >> >> >> >> >> >> >> Zitat von Eugen Block : >> >> >> >> >> >> > Hi, >> >> >> > >> >> >> > this is not a ceph issue but your openstack cli command as the >> error >> >> >> > message states. >> >> >> > >> >> >> > Try one interface at a time: >> >> >> > >> >> >> > openstack endpoint create swift public >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >> >> >> > openstack endpoint create swift admin >> http://ceph-osd3:8080/swift/v1 >> >> >> > --region RegionOne swift >> >> >> > openstack endpoint create swift internal >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >> >> >> > >> >> >> > >> >> >> > Zitat von Michel Niyoyita : >> >> >> > >> >> >> >> Hello , >> >> >> >> >> >> >> >> Below are errors I am getting once I am trying to integrate swift >> >> with >> >> >> >> Radosgateway. >> >> >> >> >> >> >> >> From openstack side once i try to endpoint which will point to the >> >> >> >> radosgateway : >> >> >> >> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ openstack endpoint create >> >> >> >> --publicurl http://ceph-osd3:8080/swift/v1 --adminurl >> >> >> >> http://ceph-osd3:8080/swift/v1 --internal >> >> >> http://ceph-osd3:8080/swift/v1 >> >> >> >> --region RegionOne swift >> >> >> >> usage: openstack endpoint create [-h] [-f >> >> {json,shell,table,value,yaml}] >> >> >> >> [-c COLUMN] [--noindent] [--prefix >> >> >> PREFIX] >> >> >> >> [--max-width ] >> [--fit-width] >> >> >> >> [--print-empty] [--region >> >> ] >> >> >> >> [--enable | --disable] >> >> >> >> >> >> >> >> openstack endpoint create: error: argument : invalid >> >> choice: >> >> >> ' >> >> >> >> http://ceph-osd3:8080/swift/v1' (choose from 'admin', 'public', >> >> >> 'internal') >> >> >> >> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ >> >> >> >> >> >> >> >> below are my /etc/ceph/ceph.conf file : >> >> >> >> >> >> >> >> [client.rgw.ceph-osd3] >> >> >> >> rgw_dns_name = ceph-osd3 >> >> >> >> host = ceph-osd3 >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3/keyring >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.log >> >> >> >> rgw frontends = civetweb port=10.10.29.110:8080 num_threads=100 >> >> >> >> rgw_keystone_url=http://10.10.29.150:35357 >> >> >> >> rgw_keystone_admin_user=admin >> >> >> >> rgw_keystone_admin_password=admin >> >> >> >> rgw_keystone_admin_tenant=admin >> >> >> >> rgw_keystone_accepted_role=admin Member swiftoperator >> >> >> >> rgw_keystone_token_cache_size=200 >> >> >> >> rgw_keystone_revocation_interval=300 >> >> >> >> >> >> >> >> [client.rgw.ceph-osd3.rgw0] >> >> >> >> host = ceph-osd3 >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log >> >> >> >> rgw frontends = beast endpoint=10.10.29.110:8080 >> >> >> >> rgw thread pool size = 512 >> >> >> >> >> >> >> >> please note that my rgw_dns_name = ceph_osd3 with 10.10.29.110 as >> IP >> >> >> >> >> >> >> >> and 10.10.29.150 all-in-one IP >> >> >> >> >> >> >> >> >> >> >> >> Please crosscheck where I might make mistake and try to correct. >> >> >> >> >> >> >> >> Best regards >> >> >> >> >> >> >> >> Michel >> >> >> >> >> >> >> >> On Mon, Aug 30, 2021 at 11:25 AM Etienne Menguy < >> >> >> etienne.menguy at croit.io> >> >> >> >> wrote: >> >> >> >> >> >> >> >>> Hi, >> >> >> >>> >> >> >> >>> There are some information on Ceph documentation >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/ < >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/> . >> >> >> >>> - Use keystone as auth for RGW >> >> >> >>> - Create service and register your RGW as swift >> >> >> >>> >> >> >> >>> Étienne >> >> >> >>> >> >> >> >>>> On 27 Aug 2021, at 15:47, Michel Niyoyita >> >> wrote: >> >> >> >>>> >> >> >> >>>> Hello , >> >> >> >>>> >> >> >> >>>> I have configured RGW in my ceph cluster deployed using ceph >> >> ansible >> >> >> and >> >> >> >>>> create sub user to access the created containers and would like >> to >> >> >> >>> replace >> >> >> >>>> swift by RGW in the openstack side. Anyone can help on >> >> configuration >> >> >> to >> >> >> >>> be >> >> >> >>>> done in the OpenStack side in order to integrate those >> services. I >> >> >> have >> >> >> >>>> deployed OpenStack wallaby using Kolla-ansible on ubuntu 20.04. >> and >> >> >> ceph >> >> >> >>>> pacific 16.2.5 was deployed using ansible on ubuntu 20.04 >> >> >> >>>> >> >> >> >>>> Kindly help for the configuration or documentation. >> >> >> >>>> >> >> >> >>>> Best Regards >> >> >> >>>> >> >> >> >>>> Michel >> >> >> >>>> _______________________________________________ >> >> >> >>>> ceph-users mailing list -- ceph-users at ceph.io >> >> >> >>>> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >>> >> >> >> >>> _______________________________________________ >> >> >> >>> ceph-users mailing list -- ceph-users at ceph.io >> >> >> >>> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >>> >> >> >> >> _______________________________________________ >> >> >> >> ceph-users mailing list -- ceph-users at ceph.io >> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> ceph-users mailing list -- ceph-users at ceph.io >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From zaitcev at redhat.com Thu Sep 2 14:21:44 2021 From: zaitcev at redhat.com (Pete Zaitcev) Date: Thu, 2 Sep 2021 09:21:44 -0500 Subject: [Swift][Interop] PTG time In-Reply-To: References: Message-ID: <20210902092144.1a44225f@suzdal.zaitcev.lan> On Fri, 6 Aug 2021 11:32:22 -0500 Arkady Kanevsky wrote: > Interop team would like time on Yoga PTG Monday or Tu between 21-22 UTC > to discuss Interop guideline coverage for Swift. I suspect this fell through the cracks, it's not on Swift PTG Etherpad. I'll poke our PTL. The slots you're proposing aren't in conflict with existing PTG schedule, so this should work. -- Pete From aschultz at redhat.com Thu Sep 2 14:27:28 2021 From: aschultz at redhat.com (Alex Schultz) Date: Thu, 2 Sep 2021 08:27:28 -0600 Subject: [ceph-users] Re: Replacing swift with RGW In-Reply-To: References: <20210901122221.Horde.v0FeoCUfex4WYSaXReGzV1i@webmail.nde.ag> <20210901134023.Horde.ZhZ0JBty-nYNsBfGe6WOijH@webmail.nde.ag> <20210902071439.Horde.4cayqDV75oyYS75eyQZMSPD@webmail.nde.ag> <20210902081741.Horde.Xp06qPUkci05sKXFlo1F5ap@webmail.nde.ag> <20210902083925.Horde.a0LD9j43RkGPIl7UuWCRtyu@webmail.nde.ag> Message-ID: The swift docs are a bit out of date as they still reference python2 despite python3 being supported for some time now. Replace python- with python3- and try again. On Thu, Sep 2, 2021 at 7:35 AM Michel Niyoyita wrote: > > > ---------- Forwarded message --------- > From: Michel Niyoyita > Date: Thu, Sep 2, 2021 at 12:17 PM > Subject: Fwd: [ceph-users] Re: Replacing swift with RGW > To: > > > > > ---------- Forwarded message --------- > From: Eugen Block > Date: Thu, Sep 2, 2021 at 10:39 AM > Subject: Re: [ceph-users] Re: Replacing swift with RGW > To: Michel Niyoyita > > > You should continue this thread on the openstack-discuss mailing list > (openstack-discuss at lists.openstack.org) since this is not a ceph issue. > I'm not familiar with kolla and I don't know the requirements to > install openstack-swift, you'll need to ask the openstack community. > > > Zitat von Michel Niyoyita : > > > Below are errors I am getting once I try to run swift commands . the > second > > one is the error I get once try to install python-swiftclient > > > > (kolla-open) [stack at kolla-open ~]$ swift -v stat > > -bash: swift: command not found > > (kolla-open) [stack at kolla-open ~]$ sudo yum -y install > python-swiftclient > > Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 09:21:53 > AM > > CAT. > > No match for argument: python-swiftclient > > Error: Unable to find a match: python-swiftclient > > (kolla-open) [stack at kolla-open ~]$ > > > > Waiting for your help > > > > On Thu, Sep 2, 2021 at 10:17 AM Eugen Block wrote: > > > >> I can't tell for sure, but yes, I believe you need the openstack-swift > >> package (with dependencies). What errors do you get? The more > >> information you share the better people can help. > >> > >> > >> Zitat von Michel Niyoyita : > >> > >> > I tried to install "sudo yum -y install python-swiftclient" on > openstack > >> > side but fails . are there openastack-shwift packages which are > needed? > >> > if are there please help me to get . may be also it is the cause I am > >> > failing to run swift command on openstack cli side. > >> > > >> > thank you for your continued support. > >> > > >> > Micheal > >> > > >> > On Thu, Sep 2, 2021 at 9:14 AM Eugen Block wrote: > >> > > >> >> I only configured the endpoints for the clients to directly access > the > >> >> RGWs, but you'll probably need to install the openstack-swift > package. > >> >> Or have you done that already? > >> >> > >> >> > >> >> Zitat von Michel Niyoyita : > >> >> > >> >> > Thank you Eugen for your prompt response. > >> >> > > >> >> > Now the commands provided work. but I am not finding the object > >> storage > >> >> on > >> >> > the horizon dashboard , but it appears in the system information > >> >> services. > >> >> > [image: image.png] > >> >> > so my question is how to configure it in order that it can appear > in > >> the > >> >> > dashboard . > >> >> > > >> >> > Michel > >> >> > > >> >> > On Wed, Sep 1, 2021 at 3:49 PM Eugen Block wrote: > >> >> > > >> >> >> Sorry, one little detail slipped through, the '--region' flag has > to > >> >> >> be put before the 'service' name. The correct command would be: > >> >> >> > >> >> >> openstack endpoint create --region RegionOne swift admin > >> >> >> http://ceph-osd3:8080/swift/v1 > >> >> >> > >> >> >> and respectively for the other interfaces. > >> >> >> > >> >> >> > >> >> >> Zitat von Eugen Block : > >> >> >> > >> >> >> > Hi, > >> >> >> > > >> >> >> > this is not a ceph issue but your openstack cli command as the > >> error > >> >> >> > message states. > >> >> >> > > >> >> >> > Try one interface at a time: > >> >> >> > > >> >> >> > openstack endpoint create swift public > >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift > >> >> >> > openstack endpoint create swift admin > >> http://ceph-osd3:8080/swift/v1 > >> >> >> > --region RegionOne swift > >> >> >> > openstack endpoint create swift internal > >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift > >> >> >> > > >> >> >> > > >> >> >> > Zitat von Michel Niyoyita : > >> >> >> > > >> >> >> >> Hello , > >> >> >> >> > >> >> >> >> Below are errors I am getting once I am trying to integrate > swift > >> >> with > >> >> >> >> Radosgateway. > >> >> >> >> > >> >> >> >> From openstack side once i try to endpoint which will point to > the > >> >> >> >> radosgateway : > >> >> >> >> > >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ openstack endpoint > create > >> >> >> >> --publicurl http://ceph-osd3:8080/swift/v1 --adminurl > >> >> >> >> http://ceph-osd3:8080/swift/v1 --internal > >> >> >> http://ceph-osd3:8080/swift/v1 > >> >> >> >> --region RegionOne swift > >> >> >> >> usage: openstack endpoint create [-h] [-f > >> >> {json,shell,table,value,yaml}] > >> >> >> >> [-c COLUMN] [--noindent] > [--prefix > >> >> >> PREFIX] > >> >> >> >> [--max-width ] > >> [--fit-width] > >> >> >> >> [--print-empty] [--region > >> >> ] > >> >> >> >> [--enable | --disable] > >> >> >> >> > >> >> >> >> openstack endpoint create: error: argument : invalid > >> >> choice: > >> >> >> ' > >> >> >> >> http://ceph-osd3:8080/swift/v1' (choose from 'admin', > 'public', > >> >> >> 'internal') > >> >> >> >> > >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ > >> >> >> >> > >> >> >> >> below are my /etc/ceph/ceph.conf file : > >> >> >> >> > >> >> >> >> [client.rgw.ceph-osd3] > >> >> >> >> rgw_dns_name = ceph-osd3 > >> >> >> >> host = ceph-osd3 > >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3/keyring > >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.log > >> >> >> >> rgw frontends = civetweb port=10.10.29.110:8080 > num_threads=100 > >> >> >> >> rgw_keystone_url=http://10.10.29.150:35357 > >> >> >> >> rgw_keystone_admin_user=admin > >> >> >> >> rgw_keystone_admin_password=admin > >> >> >> >> rgw_keystone_admin_tenant=admin > >> >> >> >> rgw_keystone_accepted_role=admin Member swiftoperator > >> >> >> >> rgw_keystone_token_cache_size=200 > >> >> >> >> rgw_keystone_revocation_interval=300 > >> >> >> >> > >> >> >> >> [client.rgw.ceph-osd3.rgw0] > >> >> >> >> host = ceph-osd3 > >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring > >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log > >> >> >> >> rgw frontends = beast endpoint=10.10.29.110:8080 > >> >> >> >> rgw thread pool size = 512 > >> >> >> >> > >> >> >> >> please note that my rgw_dns_name = ceph_osd3 with 10.10.29.110 > as > >> IP > >> >> >> >> > >> >> >> >> and 10.10.29.150 all-in-one IP > >> >> >> >> > >> >> >> >> > >> >> >> >> Please crosscheck where I might make mistake and try to > correct. > >> >> >> >> > >> >> >> >> Best regards > >> >> >> >> > >> >> >> >> Michel > >> >> >> >> > >> >> >> >> On Mon, Aug 30, 2021 at 11:25 AM Etienne Menguy < > >> >> >> etienne.menguy at croit.io> > >> >> >> >> wrote: > >> >> >> >> > >> >> >> >>> Hi, > >> >> >> >>> > >> >> >> >>> There are some information on Ceph documentation > >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/ < > >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/> . > >> >> >> >>> - Use keystone as auth for RGW > >> >> >> >>> - Create service and register your RGW as swift > >> >> >> >>> > >> >> >> >>> Étienne > >> >> >> >>> > >> >> >> >>>> On 27 Aug 2021, at 15:47, Michel Niyoyita > > >> >> wrote: > >> >> >> >>>> > >> >> >> >>>> Hello , > >> >> >> >>>> > >> >> >> >>>> I have configured RGW in my ceph cluster deployed using ceph > >> >> ansible > >> >> >> and > >> >> >> >>>> create sub user to access the created containers and would > like > >> to > >> >> >> >>> replace > >> >> >> >>>> swift by RGW in the openstack side. Anyone can help on > >> >> configuration > >> >> >> to > >> >> >> >>> be > >> >> >> >>>> done in the OpenStack side in order to integrate those > >> services. I > >> >> >> have > >> >> >> >>>> deployed OpenStack wallaby using Kolla-ansible on ubuntu > 20.04. > >> and > >> >> >> ceph > >> >> >> >>>> pacific 16.2.5 was deployed using ansible on ubuntu 20.04 > >> >> >> >>>> > >> >> >> >>>> Kindly help for the configuration or documentation. > >> >> >> >>>> > >> >> >> >>>> Best Regards > >> >> >> >>>> > >> >> >> >>>> Michel > >> >> >> >>>> _______________________________________________ > >> >> >> >>>> ceph-users mailing list -- ceph-users at ceph.io > >> >> >> >>>> To unsubscribe send an email to ceph-users-leave at ceph.io > >> >> >> >>> > >> >> >> >>> _______________________________________________ > >> >> >> >>> ceph-users mailing list -- ceph-users at ceph.io > >> >> >> >>> To unsubscribe send an email to ceph-users-leave at ceph.io > >> >> >> >>> > >> >> >> >> _______________________________________________ > >> >> >> >> ceph-users mailing list -- ceph-users at ceph.io > >> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io > >> >> >> > >> >> >> > >> >> >> > >> >> >> _______________________________________________ > >> >> >> ceph-users mailing list -- ceph-users at ceph.io > >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io > >> >> >> > >> >> > >> >> > >> >> > >> >> > >> > >> > >> > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akanevsk at redhat.com Thu Sep 2 15:13:20 2021 From: akanevsk at redhat.com (Arkady Kanevsky) Date: Thu, 2 Sep 2021 10:13:20 -0500 Subject: [Swift][Interop] PTG time In-Reply-To: <20210902092144.1a44225f@suzdal.zaitcev.lan> References: <20210902092144.1a44225f@suzdal.zaitcev.lan> Message-ID: Thanks Pete. On Thu, Sep 2, 2021 at 9:21 AM Pete Zaitcev wrote: > On Fri, 6 Aug 2021 11:32:22 -0500 > Arkady Kanevsky wrote: > > > Interop team would like time on Yoga PTG Monday or Tu between 21-22 UTC > > to discuss Interop guideline coverage for Swift. > > I suspect this fell through the cracks, it's not on Swift PTG Etherpad. > I'll poke our PTL. The slots you're proposing aren't in conflict with > existing PTG schedule, so this should work. > > -- Pete > > -- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456 -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.Kieske at mittwald.de Thu Sep 2 16:48:32 2021 From: S.Kieske at mittwald.de (Sven Kieske) Date: Thu, 2 Sep 2021 16:48:32 +0000 Subject: [nova]is there support for discard/trim in virtio-blk? Message-ID: <056115b0d76175a5383bab5102c46e67f2375e42.camel@mittwald.de> Hi, according to our research discard/trim is commonly documented as "not supported by virtio-blk" in the wider community and in openstack nova[0]. I understand that this was not supported by the kernel or qemu when discard support was initially implemented[1] in openstack, but times have changed. Virtio-blk in upstream Kernel[2] and in qemu[3] does clearly support discard/trim, which we discovered thanks to StackExchange[4]. So my question is, has someone successfully used trim/discard with virtio-blk in nova provisioned vms? [1] makes me guess, that there is no codepath that works with virtio-blk, but I'm not sure I understood all the codepaths in nova yet. we are still working our way through it, though. We currently use the train release in conjunction with ceph rbd volumes for testing this feature but weren't yet able to use it successfully. Looking at nova's master branch not much seems to have changed regarding trim support. If this is no supported configuration currently, should I submit a blueprint to enable this feature? Would there be other devs/ops interested in this? Any help or pointers would be appreciated. [0]: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2015 (nova debug log saying discard is unsupported for virtio) [1]: https://blueprints.launchpad.net/nova/+spec/cinder-backend-report-discard (initial implementation of discard support in openstack) [2]: https://github.com/torvalds/linux/commit/1f23816b8eb8fdc39990abe166c10a18c16f6b21 (available since linux 5.0-rc1) [3]: https://github.com/qemu/qemu/commit/37b06f8d46fe602e630e4bdce24e80a3e0f70cc2 (available since qemu 4.0.0) [4]: https://unix.stackexchange.com/a/518223 -- Mit freundlichen Grüßen / Regards Sven Kieske Systementwickler / systems engineer     Mittwald CM Service GmbH & Co. KG Königsberger Straße 4-6 32339 Espelkamp   Tel.: 05772 / 293-900 Fax: 05772 / 293-333   https://www.mittwald.de   Geschäftsführer: Robert Meyer, Florian Jürgens   St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit  gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From smooney at redhat.com Thu Sep 2 17:44:42 2021 From: smooney at redhat.com (Sean Mooney) Date: Thu, 02 Sep 2021 18:44:42 +0100 Subject: [nova]is there support for discard/trim in virtio-blk? In-Reply-To: <056115b0d76175a5383bab5102c46e67f2375e42.camel@mittwald.de> References: <056115b0d76175a5383bab5102c46e67f2375e42.camel@mittwald.de> Message-ID: <81a5394854a9d72ca9f1d4e2254be785da042943.camel@redhat.com> On Thu, 2021-09-02 at 16:48 +0000, Sven Kieske wrote: > Hi, > > according to our research discard/trim is commonly documented as "not supported by virtio-blk" > in the wider community and in openstack nova[0]. > > I understand that this was not supported by the kernel or qemu when discard support was > initially implemented[1] in openstack, but times have changed. > > Virtio-blk in upstream Kernel[2] and in qemu[3] does clearly support > discard/trim, which we discovered thanks to StackExchange[4]. > > So my question is, has someone successfully used trim/discard with > virtio-blk in nova provisioned vms? that is a good question. if it works in upstrema qemu it should work with a nova provisioned vm unless there is something we explitly need to add in teh xml to make it work. i suspect it will jsut work and we > > [1] makes me guess, that there is no codepath that works with > virtio-blk, but I'm not sure I understood all the codepaths in nova yet. > we are still working our way through it, though. virtio-blk is our default storage backend for libvirt. if you dont specifiy otherwise its using virtio-blk, to request it explcitly you set hw_disk_bus=virtio virtio in the image properties mappes to virtio-blk, if you want virtio-scsi you have to request that explictly. > > We currently use the train release in conjunction with ceph rbd volumes for testing this feature > but weren't yet able to use it successfully. > Looking at nova's master branch not much seems to have changed regarding trim support. in the past to use trim you had to configure virtio-scsi in the past so that was the supported way to enable this when train was released. looking at libcvirt we might need to pas som addtional optional driver argument to make it work https://libvirt.org/formatdomain.html The optional discard attribute controls whether discard requests (also known as "trim" or "unmap") are ignored or passed to the filesystem. The value can be either "unmap" (allow the discard request to be passed) or "ignore" (ignore the discard request). Since 1.0.6 (QEMU and KVM only) The optional detect_zeroes attribute controls whether to detect zero write requests. The value can be "off", "on" or "unmap". First two values turn the detection off and on, respectively. The third value ("unmap") turns the detection on and additionally tries to discard such areas from the image based on the value of discard above (it will act as "on" if discard is set to "ignore"). NB enabling the detection is a compute intensive operation, but can save file space and/or time on slow media. Since 2.0.0 although its not clear that libvirt was updated to support the virtio-blk support added in qemu 4.0.0 sicne the libvirt docs were not updated to refernce that. > > If this is no supported configuration currently, should I submit a blueprint to enable this feature? yes although i think we need to confirm if this is support in libvirt kasyap perhaps you could ask our virt folk interneally and find out if we need to do anythin to enable trim support for virtio-blk? we do have support for seting the driver discard option https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/config.py#L1058 and there is an nova config option to enable it. https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.hw_disk_discard so perhaps you just need to set that o unmap e.g. /etc/nova/nova.conf: [libvirt] hw_disk_discard=unmap it is used in teh images backend https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/imagebackend.py#L104 and we also seam to have support in the volume driver https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/volume/volume.py#L94-L96 so i suspect you are just missing setting the config option to unmap and the debug log in https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2015 is proably outdated and can be removed if yoiu can validate the config option works then can you open a bug for the outdated message? > > Would there be other devs/ops interested in this? > > Any help or pointers would be appreciated. > > [0]: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2015 (nova debug log saying discard is unsupported for virtio) > [1]: https://blueprints.launchpad.net/nova/+spec/cinder-backend-report-discard (initial implementation of discard support in openstack) > [2]: https://github.com/torvalds/linux/commit/1f23816b8eb8fdc39990abe166c10a18c16f6b21 (available since linux 5.0-rc1) > [3]: https://github.com/qemu/qemu/commit/37b06f8d46fe602e630e4bdce24e80a3e0f70cc2 (available since qemu 4.0.0) > [4]: https://unix.stackexchange.com/a/518223 > From zhangbailin at inspur.com Thu Sep 2 01:50:17 2021 From: zhangbailin at inspur.com (=?utf-8?B?QnJpbiBaaGFuZyjlvKDnmb7mnpcp?=) Date: Thu, 2 Sep 2021 01:50:17 +0000 Subject: =?utf-8?B?562U5aSNOiBbbGlzdHMub3BlbnN0YWNrLm9yZ+S7o+WPkV1SZTogW2FsbF1b?= =?utf-8?B?dGNdIFRlY2huaWNhbCBDb21taXR0ZWUgbmV4dCB3ZWVrbHkgbWVldGluZyBv?= =?utf-8?Q?n_Sept_2nd_at_1500_UTC?= In-Reply-To: <17ba318be97.cc9f4e3e210522.5438255807862395887@ghanshyammann.com> References: <70ccb3ff212d51b5854e046a114f33f5@sslemail.net> <17ba318be97.cc9f4e3e210522.5438255807862395887@ghanshyammann.com> Message-ID: Hi TC team, Thanks. Hi all, regarding the venus project, if you have any questions, you can leave a comment at https://review.opendev.org/c/openstack/governance/+/804824 or through the mailing list. The attachment is the design framework and function introduction of Venus ,please check. brinzhang Inspur Electronic Information Industry Co.,Ltd. -----邮件原件----- 发件人: Ghanshyam Mann [mailto:gmann at ghanshyammann.com] 发送时间: 2021年9月2日 4:42 收件人: openstack-discuss 主题: [lists.openstack.org代发]Re: [all][tc] Technical Committee next weekly meeting on Sept 2nd at 1500 UTC Hello Everyone, Below is the agenda for Tomorrow's TC meeting schedule at 1500 UTC in #openstack-tc IRC channel. NOTE: Tomorrow meeting will be a Video call at google meet, joining information are present in https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting == Agenda for tomorrow's TC meeting == * Roll call * Follow up on past action items * Gate health check (dansmith/yoctozepto) ** http://paste.openstack.org/show/jD6kAP9tHk7PZr2nhv8h/ * Leaderless projects ** https://etherpad.opendev.org/p/yoga-leaderless * New project application: 'Venus' ** http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019748.html ** https://review.opendev.org/c/openstack/governance/+/804824 * Next step on pain points from projects/SIGs/pop-up (ricolin) ** https://etherpad.opendev.org/p/pain-point-elimination * Propose OpenStack News: Newsletter ** https://etherpad.opendev.org/p/newsletter-openstack-news * Open Reviews ** https://review.opendev.org/q/projects:openstack/governance+is:open -gmann ---- On Tue, 31 Aug 2021 12:02:17 -0500 Ghanshyam Mann wrote ---- > Hello Everyone, > > Technical Committee's next weekly meeting is scheduled for Sept 2nd at 1500 UTC. > > If you would like to add topics for discussion, please add them to the below wiki page by > Wednesday, Sept 1st, at 2100 UTC. > > https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting > > -gmann > > -------------- next part -------------- A non-text attachment was scrubbed... Name: venus works for OpenStack-v1.0.pdf Type: application/pdf Size: 628025 bytes Desc: venus works for OpenStack-v1.0.pdf URL: From micou12 at gmail.com Thu Sep 2 10:16:37 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Thu, 2 Sep 2021 12:16:37 +0200 Subject: Replacing swift with RGW Message-ID: Hello , I tried to replace Swift RGW but I face some errors 1) the object storage doesnot appear to the dashboard . but it appears in systems information and also it shows the rgw where it points . [image: image.png] the second error I try to run swift command through openstack cli but fails (kolla-open) [stack at kolla-open ~]$ swift -v stat -bash: swift: command not found (kolla-open) [stack at kolla-open ~]$ sudo yum -y install python-swiftclient Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 09:21:53 AM CAT. No match for argument: python-swiftclient Error: Unable to find a match: python-swiftclient (kolla-open) [stack at kolla-open ~]$ . Kindly advise Michel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 44626 bytes Desc: not available URL: From micou12 at gmail.com Thu Sep 2 10:17:35 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Thu, 2 Sep 2021 12:17:35 +0200 Subject: Fwd: [ceph-users] Re: Replacing swift with RGW In-Reply-To: <20210902083925.Horde.a0LD9j43RkGPIl7UuWCRtyu@webmail.nde.ag> References: <20210901122221.Horde.v0FeoCUfex4WYSaXReGzV1i@webmail.nde.ag> <20210901134023.Horde.ZhZ0JBty-nYNsBfGe6WOijH@webmail.nde.ag> <20210902071439.Horde.4cayqDV75oyYS75eyQZMSPD@webmail.nde.ag> <20210902081741.Horde.Xp06qPUkci05sKXFlo1F5ap@webmail.nde.ag> <20210902083925.Horde.a0LD9j43RkGPIl7UuWCRtyu@webmail.nde.ag> Message-ID: ---------- Forwarded message --------- From: Eugen Block Date: Thu, Sep 2, 2021 at 10:39 AM Subject: Re: [ceph-users] Re: Replacing swift with RGW To: Michel Niyoyita You should continue this thread on the openstack-discuss mailing list (openstack-discuss at lists.openstack.org) since this is not a ceph issue. I'm not familiar with kolla and I don't know the requirements to install openstack-swift, you'll need to ask the openstack community. Zitat von Michel Niyoyita : > Below are errors I am getting once I try to run swift commands . the second > one is the error I get once try to install python-swiftclient > > (kolla-open) [stack at kolla-open ~]$ swift -v stat > -bash: swift: command not found > (kolla-open) [stack at kolla-open ~]$ sudo yum -y install python-swiftclient > Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 09:21:53 AM > CAT. > No match for argument: python-swiftclient > Error: Unable to find a match: python-swiftclient > (kolla-open) [stack at kolla-open ~]$ > > Waiting for your help > > On Thu, Sep 2, 2021 at 10:17 AM Eugen Block wrote: > >> I can't tell for sure, but yes, I believe you need the openstack-swift >> package (with dependencies). What errors do you get? The more >> information you share the better people can help. >> >> >> Zitat von Michel Niyoyita : >> >> > I tried to install "sudo yum -y install python-swiftclient" on openstack >> > side but fails . are there openastack-shwift packages which are needed? >> > if are there please help me to get . may be also it is the cause I am >> > failing to run swift command on openstack cli side. >> > >> > thank you for your continued support. >> > >> > Micheal >> > >> > On Thu, Sep 2, 2021 at 9:14 AM Eugen Block wrote: >> > >> >> I only configured the endpoints for the clients to directly access the >> >> RGWs, but you'll probably need to install the openstack-swift package. >> >> Or have you done that already? >> >> >> >> >> >> Zitat von Michel Niyoyita : >> >> >> >> > Thank you Eugen for your prompt response. >> >> > >> >> > Now the commands provided work. but I am not finding the object >> storage >> >> on >> >> > the horizon dashboard , but it appears in the system information >> >> services. >> >> > [image: image.png] >> >> > so my question is how to configure it in order that it can appear in >> the >> >> > dashboard . >> >> > >> >> > Michel >> >> > >> >> > On Wed, Sep 1, 2021 at 3:49 PM Eugen Block wrote: >> >> > >> >> >> Sorry, one little detail slipped through, the '--region' flag has to >> >> >> be put before the 'service' name. The correct command would be: >> >> >> >> >> >> openstack endpoint create --region RegionOne swift admin >> >> >> http://ceph-osd3:8080/swift/v1 >> >> >> >> >> >> and respectively for the other interfaces. >> >> >> >> >> >> >> >> >> Zitat von Eugen Block : >> >> >> >> >> >> > Hi, >> >> >> > >> >> >> > this is not a ceph issue but your openstack cli command as the >> error >> >> >> > message states. >> >> >> > >> >> >> > Try one interface at a time: >> >> >> > >> >> >> > openstack endpoint create swift public >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >> >> >> > openstack endpoint create swift admin >> http://ceph-osd3:8080/swift/v1 >> >> >> > --region RegionOne swift >> >> >> > openstack endpoint create swift internal >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >> >> >> > >> >> >> > >> >> >> > Zitat von Michel Niyoyita : >> >> >> > >> >> >> >> Hello , >> >> >> >> >> >> >> >> Below are errors I am getting once I am trying to integrate swift >> >> with >> >> >> >> Radosgateway. >> >> >> >> >> >> >> >> From openstack side once i try to endpoint which will point to the >> >> >> >> radosgateway : >> >> >> >> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ openstack endpoint create >> >> >> >> --publicurl http://ceph-osd3:8080/swift/v1 --adminurl >> >> >> >> http://ceph-osd3:8080/swift/v1 --internal >> >> >> http://ceph-osd3:8080/swift/v1 >> >> >> >> --region RegionOne swift >> >> >> >> usage: openstack endpoint create [-h] [-f >> >> {json,shell,table,value,yaml}] >> >> >> >> [-c COLUMN] [--noindent] [--prefix >> >> >> PREFIX] >> >> >> >> [--max-width ] >> [--fit-width] >> >> >> >> [--print-empty] [--region >> >> ] >> >> >> >> [--enable | --disable] >> >> >> >> >> >> >> >> openstack endpoint create: error: argument : invalid >> >> choice: >> >> >> ' >> >> >> >> http://ceph-osd3:8080/swift/v1' (choose from 'admin', 'public', >> >> >> 'internal') >> >> >> >> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ >> >> >> >> >> >> >> >> below are my /etc/ceph/ceph.conf file : >> >> >> >> >> >> >> >> [client.rgw.ceph-osd3] >> >> >> >> rgw_dns_name = ceph-osd3 >> >> >> >> host = ceph-osd3 >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3/keyring >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.log >> >> >> >> rgw frontends = civetweb port=10.10.29.110:8080 num_threads=100 >> >> >> >> rgw_keystone_url=http://10.10.29.150:35357 >> >> >> >> rgw_keystone_admin_user=admin >> >> >> >> rgw_keystone_admin_password=admin >> >> >> >> rgw_keystone_admin_tenant=admin >> >> >> >> rgw_keystone_accepted_role=admin Member swiftoperator >> >> >> >> rgw_keystone_token_cache_size=200 >> >> >> >> rgw_keystone_revocation_interval=300 >> >> >> >> >> >> >> >> [client.rgw.ceph-osd3.rgw0] >> >> >> >> host = ceph-osd3 >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log >> >> >> >> rgw frontends = beast endpoint=10.10.29.110:8080 >> >> >> >> rgw thread pool size = 512 >> >> >> >> >> >> >> >> please note that my rgw_dns_name = ceph_osd3 with 10.10.29.110 as >> IP >> >> >> >> >> >> >> >> and 10.10.29.150 all-in-one IP >> >> >> >> >> >> >> >> >> >> >> >> Please crosscheck where I might make mistake and try to correct. >> >> >> >> >> >> >> >> Best regards >> >> >> >> >> >> >> >> Michel >> >> >> >> >> >> >> >> On Mon, Aug 30, 2021 at 11:25 AM Etienne Menguy < >> >> >> etienne.menguy at croit.io> >> >> >> >> wrote: >> >> >> >> >> >> >> >>> Hi, >> >> >> >>> >> >> >> >>> There are some information on Ceph documentation >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/ < >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/> . >> >> >> >>> - Use keystone as auth for RGW >> >> >> >>> - Create service and register your RGW as swift >> >> >> >>> >> >> >> >>> Étienne >> >> >> >>> >> >> >> >>>> On 27 Aug 2021, at 15:47, Michel Niyoyita >> >> wrote: >> >> >> >>>> >> >> >> >>>> Hello , >> >> >> >>>> >> >> >> >>>> I have configured RGW in my ceph cluster deployed using ceph >> >> ansible >> >> >> and >> >> >> >>>> create sub user to access the created containers and would like >> to >> >> >> >>> replace >> >> >> >>>> swift by RGW in the openstack side. Anyone can help on >> >> configuration >> >> >> to >> >> >> >>> be >> >> >> >>>> done in the OpenStack side in order to integrate those >> services. I >> >> >> have >> >> >> >>>> deployed OpenStack wallaby using Kolla-ansible on ubuntu 20.04. >> and >> >> >> ceph >> >> >> >>>> pacific 16.2.5 was deployed using ansible on ubuntu 20.04 >> >> >> >>>> >> >> >> >>>> Kindly help for the configuration or documentation. >> >> >> >>>> >> >> >> >>>> Best Regards >> >> >> >>>> >> >> >> >>>> Michel >> >> >> >>>> _______________________________________________ >> >> >> >>>> ceph-users mailing list -- ceph-users at ceph.io >> >> >> >>>> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >>> >> >> >> >>> _______________________________________________ >> >> >> >>> ceph-users mailing list -- ceph-users at ceph.io >> >> >> >>> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >>> >> >> >> >> _______________________________________________ >> >> >> >> ceph-users mailing list -- ceph-users at ceph.io >> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> ceph-users mailing list -- ceph-users at ceph.io >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From micou12 at gmail.com Thu Sep 2 13:33:57 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Thu, 2 Sep 2021 15:33:57 +0200 Subject: Fwd: Replacing swift with RGW In-Reply-To: References: Message-ID: Hello , I tried to replace Swift RGW but I face some errors 1) the object storage doesnot appear to the dashboard . but it appears in systems information and also it shows the rgw where it points . [image: image.png] the second error I try to run swift command through openstack cli but fails (kolla-open) [stack at kolla-open ~]$ swift -v stat -bash: swift: command not found (kolla-open) [stack at kolla-open ~]$ sudo yum -y install python-swiftclient Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 09:21:53 AM CAT. No match for argument: python-swiftclient Error: Unable to find a match: python-swiftclient (kolla-open) [stack at kolla-open ~]$ . Kindly advise Michel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 44626 bytes Desc: not available URL: From mark.mielke at gmail.com Thu Sep 2 22:47:42 2021 From: mark.mielke at gmail.com (Mark Mielke) Date: Thu, 2 Sep 2021 18:47:42 -0400 Subject: [nova]is there support for discard/trim in virtio-blk? In-Reply-To: <81a5394854a9d72ca9f1d4e2254be785da042943.camel@redhat.com> References: <056115b0d76175a5383bab5102c46e67f2375e42.camel@mittwald.de> <81a5394854a9d72ca9f1d4e2254be785da042943.camel@redhat.com> Message-ID: On Thu, Sep 2, 2021 at 1:47 PM Sean Mooney wrote: > On Thu, 2021-09-02 at 16:48 +0000, Sven Kieske wrote: > > Virtio-blk in upstream Kernel[2] and in qemu[3] does clearly support > > discard/trim, which we discovered thanks to StackExchange[4]. > > > > So my question is, has someone successfully used trim/discard with > > virtio-blk in nova provisioned vms? > and there is an nova config option to enable it. > > https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.hw_disk_discard > so perhaps you just need to set that o unmap > e.g. > /etc/nova/nova.conf: > [libvirt] > hw_disk_discard=unmap > With a recent enough qemu on hypervisor, and discard support on the block store, it can technically work. However, the guest was the problem for me. For example, I believe the RHEL/CentOS 8.x / 4.18 kernel does NOT provide support for this capability. But, Linux 5.4 works fine. Other than this, it works fine. I have used it for a while now, and it is just bringing the guests up-to-date that prevented it from being more useful. For older guests, I have a simple script that basically fills the guests file system up to 95% full with zero blocks, and the zero block has the same effect on my underlying storage. Finally, I would caveat that discard should not be a storage management solution. The storage should be the right size, and it should get naturally recycled. But, over time - perhaps every few months, or once a year, there is a consequence that guests don't pass through "free" information to the hypervisor on the regular, and it will capture dead blocks in the underlying storage that never get garbage collected, and this is an unfortunate waste. In our case, SolidFire stores the blocks 3X and it adds up. In five years, I've done the clean up with discard or zero only twice. -- Mark Mielke -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Thu Sep 2 23:22:10 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 2 Sep 2021 23:22:10 +0000 Subject: [all][infra] Mailing lists offline 2021-09-12 for server upgrade Message-ID: <20210902232210.jx2hrxpqjmjrn74z@yuggoth.org> n Sunday, September 12, the OpenDev sysadmins will be performing operating system upgrades for the server hosting the lists.airshipit.org, lists.opendev.org, lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org sites. Expect extended outages for message delivery and access to archives between 15:00 UTC and 21:00 UTC, though messages sent to the mailing lists at these sites should queue normally and be delivered automatically after maintenance activities have concluded. If you have any questions, feel free to reply to this message, or find us in the #opendev channel on the OFTC IRC network. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From kkchn.in at gmail.com Fri Sep 3 06:59:10 2021 From: kkchn.in at gmail.com (KK CHN) Date: Fri, 3 Sep 2021 12:29:10 +0530 Subject: Unable to attach a Volume to Openstack WIndows instance Message-ID: List, I am trying to import a multi disk Windows VM ( 100GB bootable disk and 600 GB data volume disk) exported from HyperV to Openstack(Ussuri, Qemu-KVM, Glance image store, Cinder volume with Ceph backend). I am able to import the First bootable disk(vhdx converted to qcow2 ) with Windows VM in the OpenStack and its up and running able to login. ONE BASIC queston I not perfomed the *"Virt IO Injection steps"* as I am not sure whether for existing Windows VMs exported from HyperV need this Injection or not . Some one can clarify or defend this statement * " That Any Windows VM with Volume disks attached to it running on Other HyperVisor( say HyperV in my case, oVirt) exported to openstack which uses qemu-kvm needed the VirtIO Injection must done before importing to OpenStack with KVM for these Windows VMs for up and running and for attaching volume disks which also exported from other hypervisors . " OR only performance improvement only VirtIO injection do, so that I can avoid VirtIO injection for importing WIndowsVMs with multi disk to OpenStack.* used this step to import the converted bootable disk to my openstack. # openstack image create “Your_Windows_VM_NAME_Blaah” --file Windows_VM_disk1.qcow2 --disk-format qcow2 --container-format bare --public --property hw_firmware_type=uefi --property os_secure_boot=required --property hw_disk_bus=ide I am trying to import the second disk # openstack image create “Your_Windows_VM_NAME_Blah” --file Windows_VM_disk2.qcow2 --disk-format qcow2 --public and tried to create Volume through horizon dashboard with this image and 600 GB space. But volume creation itself showing status creating for more than 1 hour for now . Is this usual behaviour or some this wrong ? Note: so this volume creation from image is taking long time. As trial and error method I tried to create a small plain volume and attach this to the running WIndows VM(Windows2012 RC). But failed. I tried to create a plain 20 GB volume from the Horizon GUI and its created with in 30 seconds.. And tried to attach this volume to the Windows VM disk1 which is running and not got attached. ( I shutdown the Windows VM and tried to attach) same result. not attached. A plain volume also unable to got attached to the Windows VM ? why ? But the large 600GB goes for long hours. ( yesterday also tried and it was in creating state for 5 to 10 hours and I was waiting with no use, then I used the cinder command to change the volume status to "available" and then removed the volume which had not yet changed the state from "creating" to ``available" . Why do these volume creations with large disk space go for hours ? Are there any short steps to make it fast ? Any hints most -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.Kieske at mittwald.de Fri Sep 3 08:15:51 2021 From: S.Kieske at mittwald.de (Sven Kieske) Date: Fri, 3 Sep 2021 08:15:51 +0000 Subject: [nova]is there support for discard/trim in virtio-blk? In-Reply-To: <81a5394854a9d72ca9f1d4e2254be785da042943.camel@redhat.com> References: <056115b0d76175a5383bab5102c46e67f2375e42.camel@mittwald.de> <81a5394854a9d72ca9f1d4e2254be785da042943.camel@redhat.com> Message-ID: On Do, 2021-09-02 at 18:44 +0100, Sean Mooney wrote: > although its not clear that libvirt was updated to support the virtio-blk support added in qemu 4.0.0 sicne the libvirt docs were not updated to refernce that. Thanks for all the feedback and pointers! FWIW we were already aware of the discard=unmap option in libvirt, I should have mentioned that. In the meantime I found this bugreport: https://bugzilla.redhat.com/show_bug.cgi?id=1672682 which seems to indicate that this indeed works in libvirt for virtio-blk. So I will continue to figure out why this doesn't work as intended in our setup. -- Mit freundlichen Grüßen / Regards Sven Kieske Systementwickler / systems engineer     Mittwald CM Service GmbH & Co. KG Königsberger Straße 4-6 32339 Espelkamp   Tel.: 05772 / 293-900 Fax: 05772 / 293-333   https://www.mittwald.de   Geschäftsführer: Robert Meyer, Florian Jürgens   St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit  gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From kchamart at redhat.com Fri Sep 3 08:32:55 2021 From: kchamart at redhat.com (Kashyap Chamarthy) Date: Fri, 3 Sep 2021 10:32:55 +0200 Subject: [nova]is there support for discard/trim in virtio-blk? In-Reply-To: References: <056115b0d76175a5383bab5102c46e67f2375e42.camel@mittwald.de> <81a5394854a9d72ca9f1d4e2254be785da042943.camel@redhat.com> Message-ID: On Fri, Sep 03, 2021 at 08:15:51AM +0000, Sven Kieske wrote: [...] > Thanks for all the feedback and pointers! FWIW we were already aware > of the discard=unmap option in libvirt, I should have mentioned that. > > In the meantime I found this bugreport: > https://bugzilla.redhat.com/show_bug.cgi?id=1672682 > > which seems to indicate that this indeed works in libvirt for > virtio-blk. > > So I will continue to figure out why this doesn't work as intended in > our setup. Yes, 'discard' support for 'virtio-blk' was added to Linux and QEMU in the following versions: - Linux: v5.0 onwards - QEMU: v4.0.0 So make sure you have those versions at a minimum. And as you've discovered in the above Red Hat bugzilla, libvirt already has the config option to enable 'discard'; and Nova has the config option too. * * * Some notes on 'discard' (also called as 'trim') that I learnt from my colleague Dan Berrangé: - To genuinely save storage space, you need to enable 'trim' at every single layer of the I/O stack -- guest, host, and storage server(s). - Even if the host storage doesn't support 'discard', it might still be useful to enable it for the guests -- it'll keep your qcow2 file size down by releasing unused clusters, so if you need to copy the qcow2 file to another host there will be less data needing copying. - If you're "thick-provisoning" your guests so that they will never trigger ENOSPC, then you don't want to enable 'discard'. [...] -- /kashyap From cjeanner at redhat.com Fri Sep 3 10:01:15 2021 From: cjeanner at redhat.com (=?UTF-8?Q?C=c3=a9dric_Jeanneret?=) Date: Fri, 3 Sep 2021 12:01:15 +0200 Subject: [TripleO] "cs9" hashtag for changes related to CenOS Stream 9 support Message-ID: <2581ae2f-77ab-bb84-24bd-ac658cbf0129@redhat.com> Hello there, We currently see a nice effort in order to integrate CentOS Stream 9 support in code base and CI. In order to make everything easier to track, it would be really nice if everyone working on that topic could set a simple "cs9" hashtag on their changes. This will allow to follow things with this link: https://review.opendev.org/q/hashtag:%22cs9%22+(status:open%20OR%20status:merged) Thank you in advance! Cheers, C. -- Cédric Jeanneret (He/Him/His) Sr. Software Engineer - OpenStack Platform Deployment Framework TC Red Hat EMEA https://www.redhat.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: From ignaziocassano at gmail.com Fri Sep 3 10:05:36 2021 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Fri, 3 Sep 2021 12:05:36 +0200 Subject: [kolla][wallaby][neutron] router ha not working Message-ID: Hello Everyone, I am facing an issue with router-ha in kolla wallaby. In my neutron.conf I inserted l3_ha = True max_l3_agents_per_router = 3 l3_ha_net_cidr = 169.254.192.0/18 but router ha does not work. The router namespace is present only on one of the three contrllers and if the controller goes down routing stop to work. I am using ovs and not ovn. Any help,please ? Ignazio -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Fri Sep 3 10:09:46 2021 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Fri, 3 Sep 2021 12:09:46 +0200 Subject: [kolla][wallaby][neutron] router ha not working In-Reply-To: References: Message-ID: Hello, further information: the l3 agent log on the controllers where router namespace is not present, report: https://paste.openstack.org/show/808573/ Il giorno ven 3 set 2021 alle ore 12:05 Ignazio Cassano < ignaziocassano at gmail.com> ha scritto: > Hello Everyone, > I am facing an issue with router-ha in kolla wallaby. In my neutron.conf I > inserted l3_ha = True > max_l3_agents_per_router = 3 > l3_ha_net_cidr = 169.254.192.0/18 > but router ha does not work. > The router namespace is present only on one of the three contrllers and if > the controller goes down routing stop to work. > I am using ovs and not ovn. > Any help,please ? > Ignazio > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Fri Sep 3 10:28:00 2021 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Fri, 3 Sep 2021 12:28:00 +0200 Subject: [kolla][wallaby][neutron] router ha not working In-Reply-To: References: Message-ID: Hello, further information. The controllers where router namespace is absent have issues on openvswitch agent, this is probably the cause of br-int error. The error in openvswitch agent log is reported here: https://paste.openstack.org/show/808576/ Thanks Il giorno ven 3 set 2021 alle ore 12:09 Ignazio Cassano < ignaziocassano at gmail.com> ha scritto: > Hello, further information: > the l3 agent log on the controllers where router namespace is not present, > report: > https://paste.openstack.org/show/808573/ > > Il giorno ven 3 set 2021 alle ore 12:05 Ignazio Cassano < > ignaziocassano at gmail.com> ha scritto: > >> Hello Everyone, >> I am facing an issue with router-ha in kolla wallaby. In my neutron.conf >> I inserted l3_ha = True >> max_l3_agents_per_router = 3 >> l3_ha_net_cidr = 169.254.192.0/18 >> but router ha does not work. >> The router namespace is present only on one of the three contrllers and >> if the controller goes down routing stop to work. >> I am using ovs and not ovn. >> Any help,please ? >> Ignazio >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Fri Sep 3 10:37:34 2021 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Fri, 3 Sep 2021 12:37:34 +0200 Subject: [kolla][wallaby][neutron] router ha not working In-Reply-To: References: Message-ID: Hello, Sorry for sending informations step by step. Seems kolla does not deploy openvswitch on all controllers :-( Ignazio Il giorno ven 3 set 2021 alle ore 12:28 Ignazio Cassano < ignaziocassano at gmail.com> ha scritto: > Hello, further information. > The controllers where router namespace is absent have issues on > openvswitch agent, this is probably the cause of br-int error. > The error in openvswitch agent log is reported here: > https://paste.openstack.org/show/808576/ > Thanks > > Il giorno ven 3 set 2021 alle ore 12:09 Ignazio Cassano < > ignaziocassano at gmail.com> ha scritto: > >> Hello, further information: >> the l3 agent log on the controllers where router namespace is not >> present, report: >> https://paste.openstack.org/show/808573/ >> >> Il giorno ven 3 set 2021 alle ore 12:05 Ignazio Cassano < >> ignaziocassano at gmail.com> ha scritto: >> >>> Hello Everyone, >>> I am facing an issue with router-ha in kolla wallaby. In my neutron.conf >>> I inserted l3_ha = True >>> max_l3_agents_per_router = 3 >>> l3_ha_net_cidr = 169.254.192.0/18 >>> but router ha does not work. >>> The router namespace is present only on one of the three contrllers and >>> if the controller goes down routing stop to work. >>> I am using ovs and not ovn. >>> Any help,please ? >>> Ignazio >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Fri Sep 3 10:49:47 2021 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Fri, 3 Sep 2021 12:49:47 +0200 Subject: [kolla][wallaby][neutron] router ha not working In-Reply-To: References: Message-ID: Hello, I solved running again: kolla-ansible -i /etc/kolla/multinode -e ansible_user=ansible -e ansible_become=true deploy -t openvswitch With the above it started openvswitch on all controllers. Stange it does not do it during deployment. Ignazio Il giorno ven 3 set 2021 alle ore 12:37 Ignazio Cassano < ignaziocassano at gmail.com> ha scritto: > Hello, Sorry for sending informations step by step. > Seems kolla does not deploy openvswitch on all controllers :-( > Ignazio > > Il giorno ven 3 set 2021 alle ore 12:28 Ignazio Cassano < > ignaziocassano at gmail.com> ha scritto: > >> Hello, further information. >> The controllers where router namespace is absent have issues on >> openvswitch agent, this is probably the cause of br-int error. >> The error in openvswitch agent log is reported here: >> https://paste.openstack.org/show/808576/ >> Thanks >> >> Il giorno ven 3 set 2021 alle ore 12:09 Ignazio Cassano < >> ignaziocassano at gmail.com> ha scritto: >> >>> Hello, further information: >>> the l3 agent log on the controllers where router namespace is not >>> present, report: >>> https://paste.openstack.org/show/808573/ >>> >>> Il giorno ven 3 set 2021 alle ore 12:05 Ignazio Cassano < >>> ignaziocassano at gmail.com> ha scritto: >>> >>>> Hello Everyone, >>>> I am facing an issue with router-ha in kolla wallaby. In my >>>> neutron.conf I inserted l3_ha = True >>>> max_l3_agents_per_router = 3 >>>> l3_ha_net_cidr = 169.254.192.0/18 >>>> but router ha does not work. >>>> The router namespace is present only on one of the three contrllers and >>>> if the controller goes down routing stop to work. >>>> I am using ovs and not ovn. >>>> Any help,please ? >>>> Ignazio >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhangbailin at inspur.com Fri Sep 3 11:20:52 2021 From: zhangbailin at inspur.com (=?utf-8?B?QnJpbiBaaGFuZyjlvKDnmb7mnpcp?=) Date: Fri, 3 Sep 2021 11:20:52 +0000 Subject: =?utf-8?B?562U5aSNOiBbbGlzdHMub3BlbnN0YWNrLm9yZ+S7o+WPkV1benVuXVtjeWJv?= =?utf-8?B?cmddW2t1cnlyXSBSZTogRWRnZSBmb3Igc2NpZW50aWZpYyByZXNlYXJjaCBz?= =?utf-8?Q?ession_on_September_13_Edge_WG_meeting?= In-Reply-To: References: Message-ID: Hi Ildiko, Currently Cyborg and nova support interaction, we are not very familiar with zun yet, if possible, please provide some relevant information. You can refer to [1] for Cyborg related materials, and [2] for the interactive content of nova and cyborg (not yet merged). [1]https://docs.openstack.org/cyborg/latest/user/architecture.html [2]https://review.opendev.org/c/openstack/cyborg/+/794549 https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/nova-cyborg-interaction.html Bailin Zhang (brinzhang) -----邮件原件----- 发件人: Ildiko Vancsa [mailto:ildiko.vancsa at gmail.com] 发送时间: 2021年8月31日 22:39 收件人: openstack-discuss 主题: [lists.openstack.org代发][zun][cyborg][kuryr] Re: Edge for scientific research session on September 13 Edge WG meeting Hi, It is a friendly reminder that the OpenInfra Edge Computing Group is having a joint session with the Chameleon project. During this session you can learn about the Chameleon project and how they are using OpenStack to build an edge platform for scientific research and academic purposes. They have also found some gaps during their journey. To mention an example the integration between Cyborg and Zun is missing to be able to use acceleration devices with containers orchestrated by OpenStack and they have been working to fill in this gap. The session will start with a presentation about the project and then we will use the remaining time to talk about solutions and next steps. I would like to invite everyone who is interested in learning about how OpenStack can be turned into an edge platform and what are the missing pieces of the puzzle that still need to be enhanced or implemented. The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. The meeting will be held on Zoom, for the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings Please let me know if you need a calendar invite and I can send you one to get reminded about the meeting. Thanks and Best Regards, Ildikó > On Aug 25, 2021, at 17:00, Ildiko Vancsa wrote: > > Hi, > > I’m very excited to invite you to a session with the Chameleon project to discuss how edge computing is powering education, scientific research and more! As always in the area of edge there are challenges and gaps that we need to solve together. In that sense the session will start with a presentation that we will then turn into a discussion to turn towards discussing solutions. > > Please also note that the Chameleon project is using OpenStack to build edge solutions. As they identified some gaps they would like to work with the community to address the challenges that they’ve been hitting. Please join us to get their feedback and find the best way to improve OpenStack to deliver a platform that can be used on the edge! > > The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. > For the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings > > Please find the details of the session below. > > Thanks and Best Regards, > Ildikó > > > Title: Chameleon: a cloud to edge infrastructure for science > > Abstract: The NSF-funded Chameleon project (www.chameleoncloud.org) provides an OpenStack-based bare metal reconfigurable cloud that supports systems research, education, and emergent application projects in science. The facility has been in existence for 6+ years, and has served ~6,000 users working on a variety of projects ranging from networking, operating systems design, and security, to innovative applications in astronomy, physics, and life sciences. Recent research trends in IoT and edge computing, as evidenced by increased numbers of project applications in those areas, made it clear that Chameleon needed to expand into the edge space. Remaining within the OpenStack framework allowed us to leverage end-user features that we developed for Chameleon, such as access via federated identity and Jupyter integration. Accordingly, we developed infrastructure called CHI at Edge that adapts Zun to provision containers on edge devices and significantly expands its capabilities to work in the edge space. The talk will describe our use case and the emergent product definition, the OpenStack adaptations we made to the current preview implementation, the directions we plan to take in our further work, as well as the ecosystem of projects and NSF infrastructure deployments leveraging this work and collaborating with us. > > From hberaud at redhat.com Fri Sep 3 14:37:36 2021 From: hberaud at redhat.com (Herve Beraud) Date: Fri, 3 Sep 2021 16:37:36 +0200 Subject: [release] Release countdown for week R-4, Sep 06 - Sep 10 Message-ID: Development Focus ----------------- We just passed feature freeze! Until release branches are cut, you should stop accepting featureful changes to deliverables following the cycle-with-rc release model, or to libraries. Exceptions should be discussed on separate threads on the mailing-list, and feature freeze exceptions approved by the team's PTL. Focus should be on finding and fixing release-critical bugs, so that release candidates and final versions of the Xena deliverables can be proposed, well ahead of the final Xena release date (06 October, 2021). General Information ------------------- We are still finishing up processing a few release requests, but the Xena release requirements are now frozen. If new library releases are needed to fix release-critical bugs in Xena, you must request a Requirements Freeze Exception (RFE) from the requirements team before we can do a new release to avoid having something released in Xena that is not actually usable. This is done by posting to the openstack-discuss mailing list with a subject line similar to: [$PROJECT][requirements] RFE requested for $PROJECT_LIB Include justification/reasoning for why a RFE is needed for this lib. If/when the requirements team OKs the post-freeze update, we can then process a new release. A soft String freeze is now in effect, in order to let the I18N team do the translation work in good conditions. In Horizon and the various dashboard plugins, you should stop accepting changes that modify user-visible strings. Exceptions should be discussed on the mailing-list. By 30 September this will become a hard string freeze, with no changes in user-visible strings allowed. Actions ------- stable/xena branches should be created soon for all not-already-branched libraries. You should expect 2-3 changes to be proposed for each: a .gitreview update, a reno update (skipped for projects not using reno), and a tox.ini constraints URL update. Please review those in priority so that the branch can be functional ASAP. The Prelude section of reno release notes is rendered as the top level overview for the release. Any important overall messaging for Xena changes should be added there to make sure the consumers of your release notes see them. Finally, if you haven't proposed Xena cycle-highlights yet, you are already late to the party. Please see http://lists.openstack.org/pipermail/openstack-discuss/2021-August/024105.html for details. Upcoming Deadlines & Dates -------------------------- RC1 deadline: 16 September (R-3 week) Final RC deadline: 30 September (R-1 week) Final Xena release: 6 October Thanks for your attention -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Fri Sep 3 16:48:16 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 03 Sep 2021 11:48:16 -0500 Subject: [all][tc] What's happening in Technical Committee: summary 3rd Sept, 21: Reading: 5 min Message-ID: <17bac8f950b.d351de0649259.4664258300272619635@ghanshyammann.com> Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC held this week's IRC meeting (Video call) on Sept 2nd Thursday. * Most of the meeting discussions are summarized in the below sections (Completed or in-progress activities section). To know more details, you can check the video recording@ (we will find a better place to put the recording next time.) - https://drive.google.com/drive/folders/16NiC60Gm8BXjZw6XCJrto6lMi6OigTiQ?usp=sharing * We will have next week's meeting on Sept 9th, Thursday 15:00 UTC on IRC, feel free the topic in agenda [1] by Sept 8th. 2. What we completed this week: ========================= * Closeout Yoga TC Election[2] * Added two more release liaisons for Keystone[3] * Retired puppet-monasca[4] 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ TC is using the etherpad[5] for Xena cycle working item. We will be checking and updating the status biweekly in the same etherpad. Open Reviews ----------------- * Five open reviews for ongoing activities[6]. Combined PTL/TC Election Season for Yoga ---------------------------------------------------- * TC election are closeout[2]. * TC chair and vice-chair nomination are also up for review/vote. * PTL election (Cyborg project having election) is going on. Leaderless projects ----------------------- * From the current nomination, we have six leaderless projects[7]. Out of them, we have five projects leaders volunteer (a few are late candidacy). One project 'Adjutant' still needs someone to step up to lead this project. Please add your name to etherpad if you are interested. Rico will also be reaching out to the previous PTL and team about it. New project application: 'Venus' -------------------------------------- * We discussed this application and all the queries are answered either on patch or ML. TC decided to start the voting in Gerrit patch[8]. Collecting Pain points from projects/SIGs/pop-up ----------------------------------------------------------- * We spend most of the meeting time on this topic. We have not gone through all the pain points mentioned in etherpad[9] but collecting two common pain points 1. rabbitMQ 2. OSC which we should continue or start discussing at the TC level. Other pain points are more project-specific. * We will continue discussing it in PTG, join us if you are interested to help/discuss this topic. Project updates ------------------- * Retiring js-openstack-lib [10] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [11]. * Current status: "Secure RBAC" is selected for Yoga cycle[12]. PTG planning ---------------- * We are collecting the PTG topics in etherpad[13], feel free to add any topic you would like to discuss. * We discussed the live stream of one of the TC PTG sessions like we did last time. Once we will have more topics in etherpad then we can select the appropriate one. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[14], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[15]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [16] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [17] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance/+/806270 [3] https://review.opendev.org/c/openstack/governance/+/806615 [4] https://review.opendev.org/c/openstack/governance/+/805105 [5] https://etherpad.opendev.org/p/tc-xena-tracker [6] https://review.opendev.org/q/project:openstack/governance+status:open [7] https://etherpad.opendev.org/p/yoga-leaderless [8] https://review.opendev.org/c/openstack/governance/+/804824 [9] https://etherpad.opendev.org/p/pain-point-elimination [10] https://review.opendev.org/c/openstack/governance/+/798540 [11] https://etherpad.opendev.org/p/y-series-goals [12] https://review.opendev.org/c/openstack/governance/+/803783 [13] https://etherpad.opendev.org/p/tc-yoga-ptg [14] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [15] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [16] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [17] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann From ces.eduardo98 at gmail.com Fri Sep 3 19:21:21 2021 From: ces.eduardo98 at gmail.com (Carlos Silva) Date: Fri, 3 Sep 2021 16:21:21 -0300 Subject: [manila][FFE] Feature freeze exception request Message-ID: Hello, everyone! I'd like to request feature freeze exceptions to some changes: [1] https://review.opendev.org/c/openstack/manila/+/803623 - Enhancements to Manila's share server migration mechanism. It will allow administrators to perform non-disruptive migrations of share servers in the cloud. [2] https://review.opendev.org/c/openstack/manila/+/803624 - New share server migration mechanism added to the NetApp ONTAP driver. With this, administrators will be able to migrate share servers nondisruptively. [3] https://review.opendev.org/c/openstack/manila/+/803622 Add support for FlexGroups in the NetApp ONTAP driver for Manila. With this feature, petabyte scale filesystems will be feasible. All changes I have mentioned were submitted in time in the feature proposal freeze for Manila, and there has been significant progress in the reviews until this point. We're working through the last of the minor enhancements reviewers have called for. These changes are isolated to the NetApp driver module, and have no impact to the manila database, REST/RPC API. Regards, carloss -------------- next part -------------- An HTML attachment was scrubbed... URL: From 22rahulmiragi at gmail.com Fri Sep 3 19:41:57 2021 From: 22rahulmiragi at gmail.com (Rahul Miragi) Date: Sat, 4 Sep 2021 01:11:57 +0530 Subject: [Neutron] [Kolla ansible ] external network setting for vlan Message-ID: Hi team, I am using kolla ansible to deploy wallaby release on single node. I have successfully deployed it and now I want to create external network and attach external IPs to openstack VMs. But somehow I’m unable to configure it. Could you please share some articles or config examples. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosmaita.fossdev at gmail.com Fri Sep 3 20:55:04 2021 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Fri, 3 Sep 2021 16:55:04 -0400 Subject: [cinder] xena feature freeze Message-ID: The Xena feature freeze for cinder is now in effect. Please do not approve patches proposing features for master until after the stable/xena branch has been cut the week of 13 September. The following blueprints are granted feature freeze exceptions: - https://blueprints.launchpad.net/cinder/+spec/s-rbac-ready-x - https://blueprints.launchpad.net/cinder/+spec/remove-sqlalchemy-migrate Any other feature patches (including driver features) that have not yet merged and are associated with an approved xena blueprint may apply for a Feature Freeze Exception by replying to this email before Tuesday, 7 September at 20:00 UTC. Note that application for an FFE is not a guarantee that an FFE will be granted. Grant of an FFE is not a guarantee that the code will be merged for Xena. It is expected that patches granted an FFE will be merged by 21:00 UTC on Monday 13 September. Bugfixes do not require an FFE. To be included in Xena, they must be merged before the first Release Candidate is cut on Thursday 16 September. After that point, only release-critical bugfixes will be allowed into the Xena release. Ping me in IRC if you have any questions. cheers, brian From fernandoperches at gmail.com Fri Sep 3 21:09:04 2021 From: fernandoperches at gmail.com (Fernando Ferraz) Date: Fri, 3 Sep 2021 18:09:04 -0300 Subject: [cinder] xena feature freeze In-Reply-To: References: Message-ID: Hi Team, I'd like to request FFE to the following changes: [1] Netapp ONTAP: Add support to revert to snapshot https://review.opendev.org/c/openstack/cinder/+/804093 - Patch adds support for the revert to snapshot feature to the NetApp Cinder drivers. [2] NetApp ONTAP: Add option to report storage provisioned capacity https://review.opendev.org/c/openstack/cinder/+/798198 - Patch adds an option to enable a new pool capability provisioned_capacity_gb that reports storage provisioned volume sizes directly from the storage system, instead of relying on the default scheduler's behavior based on internal states. Both changes and their blueprints were submitted in time and there has been significant progress in the reviews until this point. They are isolated to the NetApp driver module, and have no impact on the Cinder database, REST/RPC and APIs. Regards, Fernando Em sex., 3 de set. de 2021 às 18:01, Brian Rosmaita < rosmaita.fossdev at gmail.com> escreveu: > The Xena feature freeze for cinder is now in effect. Please do not > approve patches proposing features for master until after the > stable/xena branch has been cut the week of 13 September. > > The following blueprints are granted feature freeze exceptions: > - https://blueprints.launchpad.net/cinder/+spec/s-rbac-ready-x > - https://blueprints.launchpad.net/cinder/+spec/remove-sqlalchemy-migrate > > Any other feature patches (including driver features) that have not yet > merged and are associated with an approved xena blueprint may apply for > a Feature Freeze Exception by replying to this email before Tuesday, 7 > September at 20:00 UTC. Note that application for an FFE is not a > guarantee that an FFE will be granted. Grant of an FFE is not a > guarantee that the code will be merged for Xena. > > It is expected that patches granted an FFE will be merged by 21:00 UTC > on Monday 13 September. > > Bugfixes do not require an FFE. To be included in Xena, they must be > merged before the first Release Candidate is cut on Thursday 16 > September. After that point, only release-critical bugfixes will be > allowed into the Xena release. > > Ping me in IRC if you have any questions. > > cheers, > brian > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gouthampravi at gmail.com Fri Sep 3 23:04:11 2021 From: gouthampravi at gmail.com (Goutham Pacha Ravi) Date: Fri, 3 Sep 2021 16:04:11 -0700 Subject: [manila][FFE] Feature freeze exception request In-Reply-To: References: Message-ID: On Fri, Sep 3, 2021 at 12:27 PM Carlos Silva wrote: > Hello, everyone! > > I'd like to request feature freeze exceptions to some changes: > [1] https://review.opendev.org/c/openstack/manila/+/803623 > - Enhancements to Manila's share server migration mechanism. It will allow > administrators to perform non-disruptive migrations of share servers in the > cloud. > > [2] https://review.opendev.org/c/openstack/manila/+/803624 > - New share server migration mechanism added to the NetApp ONTAP driver. > With this, administrators will be able to migrate share servers > nondisruptively. > > [3] https://review.opendev.org/c/openstack/manila/+/803622 > Add support for FlexGroups in the NetApp ONTAP driver for Manila. With > this feature, petabyte scale filesystems will be feasible. > > All changes I have mentioned were submitted in time in the feature > proposal freeze for Manila, and there has been significant progress in the > reviews until this point. We're working through the last of the minor > enhancements reviewers have called for. These changes are isolated to the > NetApp driver module, and have no impact to the manila database, REST/RPC > API. > Thanks Carlos. Agreed; thanks for your hard work iterating through reviews. Just a friendly reminder - we’re at a long weekend holiday in many parts of the world, including where you are. I hope you enjoy the time off - we can take a little bit longer to get these in. Thanks, Goutham > Regards, > carloss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joykechen at 163.com Sat Sep 4 14:59:41 2021 From: joykechen at 163.com (=?GBK?B?s8K/yw==?=) Date: Sat, 4 Sep 2021 22:59:41 +0800 (CST) Subject: [zun][cyborg][kuryr] Re: Edge for scientific research session on September 13 Edge WG meeting In-Reply-To: References: <315A37E8-8928-4225-B26F-A15ED0788E24@gmail.com> Message-ID: <2ff99838.2b86.17bb1528b2a.Coremail.joykechen@163.com> Hi Ildiko, I'm a Cyborg core. I think it's cool to promote the integration of Cyborg and Zun. I'm happy to push the whole thing if I could. Zun's PTL Shengqin Feng is my colleague. I suggest you send her and me a reminder email before the meeting. Her Email: feng.shengqin at zte.com.cn Thanks, Ke Chen At 2021-08-31 22:39:01, "Ildiko Vancsa" wrote: >Hi, > >It is a friendly reminder that the OpenInfra Edge Computing Group is having a joint session with the Chameleon project. > >During this session you can learn about the Chameleon project and how they are using OpenStack to build an edge platform for scientific research and academic purposes. They have also found some gaps during their journey. To mention an example the integration between Cyborg and Zun is missing to be able to use acceleration devices with containers orchestrated by OpenStack and they have been working to fill in this gap. > >The session will start with a presentation about the project and then we will use the remaining time to talk about solutions and next steps. I would like to invite everyone who is interested in learning about how OpenStack can be turned into an edge platform and what are the missing pieces of the puzzle that still need to be enhanced or implemented. > >The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. >The meeting will be held on Zoom, for the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings > >Please let me know if you need a calendar invite and I can send you one to get reminded about the meeting. > >Thanks and Best Regards, >Ildikó > > >> On Aug 25, 2021, at 17:00, Ildiko Vancsa wrote: >> >> Hi, >> >> I’m very excited to invite you to a session with the Chameleon project to discuss how edge computing is powering education, scientific research and more! As always in the area of edge there are challenges and gaps that we need to solve together. In that sense the session will start with a presentation that we will then turn into a discussion to turn towards discussing solutions. >> >> Please also note that the Chameleon project is using OpenStack to build edge solutions. As they identified some gaps they would like to work with the community to address the challenges that they’ve been hitting. Please join us to get their feedback and find the best way to improve OpenStack to deliver a platform that can be used on the edge! >> >> The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. >> For the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings >> >> Please find the details of the session below. >> >> Thanks and Best Regards, >> Ildikó >> >> >> Title: Chameleon: a cloud to edge infrastructure for science >> >> Abstract: The NSF-funded Chameleon project (www.chameleoncloud.org) provides an OpenStack-based bare metal reconfigurable cloud that supports systems research, education, and emergent application projects in science. The facility has been in existence for 6+ years, and has served ~6,000 users working on a variety of projects ranging from networking, operating systems design, and security, to innovative applications in astronomy, physics, and life sciences. Recent research trends in IoT and edge computing, as evidenced by increased numbers of project applications in those areas, made it clear that Chameleon needed to expand into the edge space. Remaining within the OpenStack framework allowed us to leverage end-user features that we developed for Chameleon, such as access via federated identity and Jupyter integration. Accordingly, we developed infrastructure called CHI at Edge that adapts Zun to provision containers on edge devices and significantly expands its capabilities to work in the edge space. The talk will describe our use case and the emergent product definition, the OpenStack adaptations we made to the current preview implementation, the directions we plan to take in our further work, as well as the ecosystem of projects and NSF infrastructure deployments leveraging this work and collaborating with us. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yasufum.o at gmail.com Mon Sep 6 05:03:33 2021 From: yasufum.o at gmail.com (Yasufumi Ogawa) Date: Mon, 6 Sep 2021 14:03:33 +0900 Subject: [tacker] Ayumu Ueha for Tacker Core Message-ID: Hi everyone, I'd like to propose Ayumu Ueha as a core reviewer. He has made huge contributions not only for reviewing actively but also developing key features of NFV Container support and helping us for fixing critical problems [1][2]. Assuming that there are no objections, we will add Ayumu to tacker core team next week. [1] https://review.opendev.org/q/owner:ayumu [2] https://www.stackalytics.io/?module=opendev.org%2fopenstack%2ftacker&user_id=ueha From yamakawa.keiich at fujitsu.com Mon Sep 6 06:01:55 2021 From: yamakawa.keiich at fujitsu.com (yamakawa.keiich at fujitsu.com) Date: Mon, 6 Sep 2021 06:01:55 +0000 Subject: [tacker] Ayumu Ueha for Tacker Core In-Reply-To: References: Message-ID: +1 Best regards. -----Original Message----- From: Yasufumi Ogawa Sent: Monday, September 6, 2021 2:04 PM To: openstack-discuss Subject: [tacker] Ayumu Ueha for Tacker Core Hi everyone, I'd like to propose Ayumu Ueha as a core reviewer. He has made huge contributions not only for reviewing actively but also developing key features of NFV Container support and helping us for fixing critical problems [1][2]. Assuming that there are no objections, we will add Ayumu to tacker core team next week. [1] https://review.opendev.org/q/owner:ayumu [2] https://www.stackalytics.io/?module=opendev.org%2fopenstack%2ftacker&user_id=ueha From katada.yoshiyuk at fujitsu.com Mon Sep 6 06:08:13 2021 From: katada.yoshiyuk at fujitsu.com (katada.yoshiyuk at fujitsu.com) Date: Mon, 6 Sep 2021 06:08:13 +0000 Subject: [tacker] Ayumu Ueha for Tacker Core In-Reply-To: References: Message-ID: +1 Best Regards -----Original Message----- From: Yasufumi Ogawa [mailto:yasufum.o at gmail.com] Sent: Monday, September 6, 2021 2:04 PM To: openstack-discuss Subject: [tacker] Ayumu Ueha for Tacker Core Hi everyone, I'd like to propose Ayumu Ueha as a core reviewer. He has made huge contributions not only for reviewing actively but also developing key features of NFV Container support and helping us for fixing critical problems [1][2]. Assuming that there are no objections, we will add Ayumu to tacker core team next week. [1] https://review.opendev.org/q/owner:ayumu [2] https://www.stackalytics.io/?module=opendev.org%2fopenstack%2ftacker&user_id=ueha From kaurmanpreet2620 at gmail.com Mon Sep 6 06:13:50 2021 From: kaurmanpreet2620 at gmail.com (manpreet kaur) Date: Mon, 6 Sep 2021 11:43:50 +0530 Subject: [tacker] Ayumu Ueha for Tacker Core In-Reply-To: References: Message-ID: +1 Thanks & Regards, Manpreet Kaur On Mon, Sep 6, 2021 at 11:41 AM katada.yoshiyuk at fujitsu.com < katada.yoshiyuk at fujitsu.com> wrote: > +1 > Best Regards > -----Original Message----- > From: Yasufumi Ogawa [mailto:yasufum.o at gmail.com] > Sent: Monday, September 6, 2021 2:04 PM > To: openstack-discuss > Subject: [tacker] Ayumu Ueha for Tacker Core > > Hi everyone, > > I'd like to propose Ayumu Ueha as a core reviewer. He has made huge > contributions not only for reviewing actively but also developing key > features of NFV Container support and helping us for fixing critical > problems [1][2]. > > Assuming that there are no objections, we will add Ayumu to tacker core > team next week. > > [1] https://review.opendev.org/q/owner:ayumu > [2] > > https://www.stackalytics.io/?module=opendev.org%2fopenstack%2ftacker&user_id=ueha > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ts-takahashi at nec.com Mon Sep 6 06:37:01 2021 From: ts-takahashi at nec.com (=?utf-8?B?VEFLQUhBU0hJIFRPU0hJQUtJKOmrmOapi+OAgOaVj+aYjik=?=) Date: Mon, 6 Sep 2021 06:37:01 +0000 Subject: [tacker] Ayumu Ueha for Tacker Core In-Reply-To: References: Message-ID: +1. That's nice! Regards, Toshiaki > -----Original Message----- > From: Yasufumi Ogawa > Sent: Monday, September 6, 2021 2:04 PM > To: openstack-discuss > Subject: [tacker] Ayumu Ueha for Tacker Core > > Hi everyone, > > I'd like to propose Ayumu Ueha as a core reviewer. He has made huge > contributions not only for reviewing actively but also developing key features of > NFV Container support and helping us for fixing critical problems [1][2]. > > Assuming that there are no objections, we will add Ayumu to tacker core team > next week. > > [1] https://review.opendev.org/q/owner:ayumu > [2] > https://www.stackalytics.io/?module=opendev.org%2fopenstack%2ftacker&us > er_id=ueha From katonalala at gmail.com Mon Sep 6 07:06:17 2021 From: katonalala at gmail.com (Lajos Katona) Date: Mon, 6 Sep 2021 09:06:17 +0200 Subject: [Neutron] [Kolla ansible ] external network setting for vlan In-Reply-To: References: Message-ID: Hi Rahul, Not sure where You started your research so I add some basic documentation (but it has links to further docs): - Openstack Networking concepts: https://docs.openstack.org/neutron/latest/admin/intro-os-networking.html - Deployment with OVS (not sure which mechanism driver You use): https://docs.openstack.org/neutron/latest/admin/deploy-ovs.html - here You can find config and CLI examples if You open the links. Lajos (lajoskatona) Rahul Miragi <22rahulmiragi at gmail.com> ezt írta (időpont: 2021. szept. 3., P, 21:46): > Hi team, > > I am using kolla ansible to deploy wallaby release on single node. I have > successfully deployed it and now I want to create external network and > attach external IPs to openstack VMs. But somehow I’m unable to configure > it. Could you please share some articles or config examples. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkchn.in at gmail.com Mon Sep 6 07:22:58 2021 From: kkchn.in at gmail.com (KK CHN) Date: Mon, 6 Sep 2021 12:52:58 +0530 Subject: Error adding a Plain Volume created through GUI to Instance Message-ID: ExecutionError: Unexpected error while running command. Command: rbd map volume-4dcb78d5-e0ce-4bfc-95ae-36a4417c35d8 --pool volumes --id volumes --mon_host 10.236.223.131:6789 --mon_host 10.236.223.132:6789 Exit code: 6 Stdout: 'RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable volumes/volume-4dcb78d5-e0ce-4bfc-95ae-36a4417c35d8 journaling".\nIn some cases useful info is found in syslog - try "dmesg | tail".\n' Stderr: 'rbd: sysfs write failed\nrbd: map failed: (6) No such device or address\n' I am trying to add a 20 GB plain volume created through Horizon GUI to an existing Windows 2012 R VM running with VirtIO drivers installed. But this volume adding results in the error as shown above, trace tells to fix it by rbd feature disable .. what is the permanent fix for this issue ? and why it happens ? ( OpenStack Ussuri, Glance, Ceph , qemu-kvm) root at ctrl1:/home/cloud# ceph --version ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable) root at ctrl1:/home/cloud# -------------- next part -------------- An HTML attachment was scrubbed... URL: From zigo at debian.org Mon Sep 6 07:35:54 2021 From: zigo at debian.org (Thomas Goirand) Date: Mon, 6 Sep 2021 09:35:54 +0200 Subject: [horizon] Support for Django 3.x Message-ID: Hi, It's been a long time that Django 3.x is around, and therefore, it'd be nice if Horizon had some kind of support for it. Indeed, sooner or later, Django 3.x will be uploaded to Debian unstable. That day, I hope Horizon doesn't just break. So, what's the status? Is anyone is at least working on it? Cheers, Thomas Goirand (zigo) From stefan.hoffmann at cloudandheat.com Mon Sep 6 08:11:37 2021 From: stefan.hoffmann at cloudandheat.com (Stefan Hoffmann) Date: Mon, 06 Sep 2021 10:11:37 +0200 Subject: [cinder] discuss nas_secure options and root_squash (prohibiting root access to share) In-Reply-To: <9884c0fb2ffac6ef7ab9aeae2d1c692149a5a799.camel@cloudandheat.com> References: <9884c0fb2ffac6ef7ab9aeae2d1c692149a5a799.camel@cloudandheat.com> Message-ID: <83f3a77171aa27a92df4958f9c5d5514a2048e96.camel@cloudandheat.com> Hi cinder team, do you have any feedback, if this approach [1] follows the "right" way now? Will add this point to the meeting this week, would be nice, if you can have a look before, so we can discuss about it. Regards Stefan [1] https://review.opendev.org/c/openstack/cinder/+/802882 On Mon, 2021-08-16 at 18:05 +0200, Stefan Hoffmann wrote: > Hi cinder team, > > like discussed in the last meeting, I prepared a list [1] of > combinations of the nas_secure options and when to use them. > > If one want to prohibit root access to NFS share, only setting > nas_secure_file_operations and nas_secure_file_permissions to true is > a > useful option, I think. (Option 4) > > But also the nas_secure_file_operations is not useful to determine if > _qemu_img_info and fs access check at _connect_device should be done > with root user or cinder user. > So I will update the change [2] like proposed in the etherpad. > > Feel free to add other use cases and hints for the options to [1] and > discuss about the proposed change. > > Regards > Stefan > > > [1] https://etherpad.opendev.org/p/gSotXYAZ3JfJE8FEpMpS > [2] https://review.opendev.org/c/openstack/cinder/+/802882 > Initial Bug: > https://bugs.launchpad.net/cinder/+bug/1938196?comments=all > > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 878 bytes Desc: This is a digitally signed message part URL: From mdemaced at redhat.com Mon Sep 6 08:17:17 2021 From: mdemaced at redhat.com (Maysa De Macedo Souza) Date: Mon, 6 Sep 2021 10:17:17 +0200 Subject: Changing Magnum Heat template to create node network with vxlan instead of vlan In-Reply-To: References: Message-ID: Hello, Just some extra thought, did you check if there is some process running on port 8080? It's possible that the API failed to start, is Kuryr being used? Thanks, Maysa Macedo. On Tue, Aug 31, 2021 at 1:57 PM Karera Tony wrote: > Hello Guys, > > I was able to configure the cluster internal network with Vlans. The > communication was ok but I still get the same error > > + sleep 5 > ++ kubectl get --raw=/healthz > The connection to the server localhost:8080 was refused - did you specify > the right host or port? > + '[' ok = '' ']' > > Regards > > Tony Karera > > > > > On Sat, Aug 28, 2021 at 4:43 AM Mohammed Naser > wrote: > >> AFAIK, Magnum does not control what encapsulation type is being used. >> >> Is there a chance you have “vlan” inside tenant_network_types ? >> >> On Tue, Aug 24, 2021 at 12:05 AM Karera Tony >> wrote: >> >>> Hello Team, >>> >>> I was able to deploy Openstack with Magnum and Zun enabled. >>> >>> Unfortunately, When I create a cluster, Heat configure the fix/_network >>> for the nodes with Vlan and the nodes fail to get IPs. >>> >>> I was wondering if it is possible to trick the heat templates to create >>> vxlan network instead of vlan. >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> -- >> Mohammed Naser >> VEXXHOST, Inc. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From openinfradn at gmail.com Mon Sep 6 08:32:36 2021 From: openinfradn at gmail.com (open infra) Date: Mon, 6 Sep 2021 14:02:36 +0530 Subject: flavor with GPU Message-ID: Hi, How to create a flavor with GPU? The way we set vcpu and memory. Regards, Danishka -------------- next part -------------- An HTML attachment was scrubbed... URL: From balazs.gibizer at est.tech Mon Sep 6 08:51:37 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Mon, 06 Sep 2021 10:51:37 +0200 Subject: [nova][placement] nova-next job is broken Message-ID: <1M80ZQ.S70WTG0D8TSL1@est.tech> Hi, Since Friday the nova-next job fails with POST_FAILURE[1] due to the new osc-placement release. Please hold you rechecks. [1] https://bugs.launchpad.net/nova/+bug/1942740 Cheers, gibi From masazumi.oota.ds at hco.ntt.co.jp Mon Sep 6 09:27:09 2021 From: masazumi.oota.ds at hco.ntt.co.jp (=?UTF-8?B?5aSq55Sw5q2j57SU?=) Date: Mon, 06 Sep 2021 18:27:09 +0900 Subject: [tacker] Ayumu Ueha for Tacker Core In-Reply-To: References: Message-ID: <2764fd17-2dd5-59a4-dc9b-a28a44024847@hco.ntt.co.jp> +1 Best regards. On 2021/09/06 14:03, Yasufumi Ogawa wrote: > Hi everyone, > > I'd like to propose Ayumu Ueha as a core reviewer. He has made huge > contributions not only for reviewing actively but also developing key > features of NFV Container support and helping us for fixing critical > problems [1][2]. > > Assuming that there are no objections, we will add Ayumu to tacker core > team next week. > > [1] https://review.opendev.org/q/owner:ayumu > [2] > https://www.stackalytics.io/?module=opendev.org%2fopenstack%2ftacker&user_id=ueha > > -- --------------------------------- Masazumi OTA(太田 正純) NTT Network Innovation Center（NTT NIC) masazumi.oota.ds at hco.ntt.co.jp 0422-59-3396 From micou12 at gmail.com Mon Sep 6 09:34:08 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Mon, 6 Sep 2021 11:34:08 +0200 Subject: [ceph-users] Re: Replacing swift with RGW In-Reply-To: References: <20210901122221.Horde.v0FeoCUfex4WYSaXReGzV1i@webmail.nde.ag> <20210901134023.Horde.ZhZ0JBty-nYNsBfGe6WOijH@webmail.nde.ag> <20210902071439.Horde.4cayqDV75oyYS75eyQZMSPD@webmail.nde.ag> <20210902081741.Horde.Xp06qPUkci05sKXFlo1F5ap@webmail.nde.ag> <20210902083925.Horde.a0LD9j43RkGPIl7UuWCRtyu@webmail.nde.ag> Message-ID: Hello, I am trying to replace swift by RGW as backend storage but I failed once I try to post a container in the OpenStack side however, all interfaces are configured (admin, public and internal). but Once I post from RGW host it is created . Another issue is that object storage does not appear on the horizon dashboard . I have deployed openstack all-in-one using kolla-ansible and Os is ubuntu (kolla-open1) stack at kolla-open1:~$ swift -v post myswift Container POST failed: http://ceph-osd3:8080/swift/v1/myswift 401 Unauthorized b'AccessDenied' Failed Transaction ID: tx000000000000000000008-006135dcbd-87d63-default (kolla-open1) stack at kolla-open1:~$ swift list Account GET failed: http://ceph-osd3:8080/swift/v1?format=json 401 Unauthorized [first 60 chars of response] b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000c-' Failed Transaction ID: tx00000000000000000000c-006135de42-87d63-default Kindly help to solve the issue Michel On Thu, Sep 2, 2021 at 4:28 PM Alex Schultz wrote: > The swift docs are a bit out of date as they still reference python2 > despite python3 being supported for some time now. Replace python- with > python3- and try again. > > > On Thu, Sep 2, 2021 at 7:35 AM Michel Niyoyita wrote: > >> >> >> ---------- Forwarded message --------- >> From: Michel Niyoyita >> Date: Thu, Sep 2, 2021 at 12:17 PM >> Subject: Fwd: [ceph-users] Re: Replacing swift with RGW >> To: >> >> >> >> >> ---------- Forwarded message --------- >> From: Eugen Block >> Date: Thu, Sep 2, 2021 at 10:39 AM >> Subject: Re: [ceph-users] Re: Replacing swift with RGW >> To: Michel Niyoyita >> >> >> You should continue this thread on the openstack-discuss mailing list >> (openstack-discuss at lists.openstack.org) since this is not a ceph issue. >> I'm not familiar with kolla and I don't know the requirements to >> install openstack-swift, you'll need to ask the openstack community. >> >> >> Zitat von Michel Niyoyita : >> >> > Below are errors I am getting once I try to run swift commands . the >> second >> > one is the error I get once try to install python-swiftclient >> > >> > (kolla-open) [stack at kolla-open ~]$ swift -v stat >> > -bash: swift: command not found >> > (kolla-open) [stack at kolla-open ~]$ sudo yum -y install >> python-swiftclient >> > Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 09:21:53 >> AM >> > CAT. >> > No match for argument: python-swiftclient >> > Error: Unable to find a match: python-swiftclient >> > (kolla-open) [stack at kolla-open ~]$ >> > >> > Waiting for your help >> > >> > On Thu, Sep 2, 2021 at 10:17 AM Eugen Block wrote: >> > >> >> I can't tell for sure, but yes, I believe you need the openstack-swift >> >> package (with dependencies). What errors do you get? The more >> >> information you share the better people can help. >> >> >> >> >> >> Zitat von Michel Niyoyita : >> >> >> >> > I tried to install "sudo yum -y install python-swiftclient" on >> openstack >> >> > side but fails . are there openastack-shwift packages which are >> needed? >> >> > if are there please help me to get . may be also it is the cause I >> am >> >> > failing to run swift command on openstack cli side. >> >> > >> >> > thank you for your continued support. >> >> > >> >> > Micheal >> >> > >> >> > On Thu, Sep 2, 2021 at 9:14 AM Eugen Block wrote: >> >> > >> >> >> I only configured the endpoints for the clients to directly access >> the >> >> >> RGWs, but you'll probably need to install the openstack-swift >> package. >> >> >> Or have you done that already? >> >> >> >> >> >> >> >> >> Zitat von Michel Niyoyita : >> >> >> >> >> >> > Thank you Eugen for your prompt response. >> >> >> > >> >> >> > Now the commands provided work. but I am not finding the object >> >> storage >> >> >> on >> >> >> > the horizon dashboard , but it appears in the system information >> >> >> services. >> >> >> > [image: image.png] >> >> >> > so my question is how to configure it in order that it can appear >> in >> >> the >> >> >> > dashboard . >> >> >> > >> >> >> > Michel >> >> >> > >> >> >> > On Wed, Sep 1, 2021 at 3:49 PM Eugen Block wrote: >> >> >> > >> >> >> >> Sorry, one little detail slipped through, the '--region' flag >> has to >> >> >> >> be put before the 'service' name. The correct command would be: >> >> >> >> >> >> >> >> openstack endpoint create --region RegionOne swift admin >> >> >> >> http://ceph-osd3:8080/swift/v1 >> >> >> >> >> >> >> >> and respectively for the other interfaces. >> >> >> >> >> >> >> >> >> >> >> >> Zitat von Eugen Block : >> >> >> >> >> >> >> >> > Hi, >> >> >> >> > >> >> >> >> > this is not a ceph issue but your openstack cli command as the >> >> error >> >> >> >> > message states. >> >> >> >> > >> >> >> >> > Try one interface at a time: >> >> >> >> > >> >> >> >> > openstack endpoint create swift public >> >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >> >> >> >> > openstack endpoint create swift admin >> >> http://ceph-osd3:8080/swift/v1 >> >> >> >> > --region RegionOne swift >> >> >> >> > openstack endpoint create swift internal >> >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >> >> >> >> > >> >> >> >> > >> >> >> >> > Zitat von Michel Niyoyita : >> >> >> >> > >> >> >> >> >> Hello , >> >> >> >> >> >> >> >> >> >> Below are errors I am getting once I am trying to integrate >> swift >> >> >> with >> >> >> >> >> Radosgateway. >> >> >> >> >> >> >> >> >> >> From openstack side once i try to endpoint which will point >> to the >> >> >> >> >> radosgateway : >> >> >> >> >> >> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ openstack endpoint >> create >> >> >> >> >> --publicurl http://ceph-osd3:8080/swift/v1 --adminurl >> >> >> >> >> http://ceph-osd3:8080/swift/v1 --internal >> >> >> >> http://ceph-osd3:8080/swift/v1 >> >> >> >> >> --region RegionOne swift >> >> >> >> >> usage: openstack endpoint create [-h] [-f >> >> >> {json,shell,table,value,yaml}] >> >> >> >> >> [-c COLUMN] [--noindent] >> [--prefix >> >> >> >> PREFIX] >> >> >> >> >> [--max-width ] >> >> [--fit-width] >> >> >> >> >> [--print-empty] [--region >> >> >> ] >> >> >> >> >> [--enable | --disable] >> >> >> >> >> >> >> >> >> >> openstack endpoint create: error: argument : >> invalid >> >> >> choice: >> >> >> >> ' >> >> >> >> >> http://ceph-osd3:8080/swift/v1' (choose from 'admin', >> 'public', >> >> >> >> 'internal') >> >> >> >> >> >> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ >> >> >> >> >> >> >> >> >> >> below are my /etc/ceph/ceph.conf file : >> >> >> >> >> >> >> >> >> >> [client.rgw.ceph-osd3] >> >> >> >> >> rgw_dns_name = ceph-osd3 >> >> >> >> >> host = ceph-osd3 >> >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3/keyring >> >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.log >> >> >> >> >> rgw frontends = civetweb port=10.10.29.110:8080 >> num_threads=100 >> >> >> >> >> rgw_keystone_url=http://10.10.29.150:35357 >> >> >> >> >> rgw_keystone_admin_user=admin >> >> >> >> >> rgw_keystone_admin_password=admin >> >> >> >> >> rgw_keystone_admin_tenant=admin >> >> >> >> >> rgw_keystone_accepted_role=admin Member swiftoperator >> >> >> >> >> rgw_keystone_token_cache_size=200 >> >> >> >> >> rgw_keystone_revocation_interval=300 >> >> >> >> >> >> >> >> >> >> [client.rgw.ceph-osd3.rgw0] >> >> >> >> >> host = ceph-osd3 >> >> >> >> >> keyring = >> /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring >> >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log >> >> >> >> >> rgw frontends = beast endpoint=10.10.29.110:8080 >> >> >> >> >> rgw thread pool size = 512 >> >> >> >> >> >> >> >> >> >> please note that my rgw_dns_name = ceph_osd3 with >> 10.10.29.110 as >> >> IP >> >> >> >> >> >> >> >> >> >> and 10.10.29.150 all-in-one IP >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Please crosscheck where I might make mistake and try to >> correct. >> >> >> >> >> >> >> >> >> >> Best regards >> >> >> >> >> >> >> >> >> >> Michel >> >> >> >> >> >> >> >> >> >> On Mon, Aug 30, 2021 at 11:25 AM Etienne Menguy < >> >> >> >> etienne.menguy at croit.io> >> >> >> >> >> wrote: >> >> >> >> >> >> >> >> >> >>> Hi, >> >> >> >> >>> >> >> >> >> >>> There are some information on Ceph documentation >> >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/ < >> >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/> . >> >> >> >> >>> - Use keystone as auth for RGW >> >> >> >> >>> - Create service and register your RGW as swift >> >> >> >> >>> >> >> >> >> >>> Étienne >> >> >> >> >>> >> >> >> >> >>>> On 27 Aug 2021, at 15:47, Michel Niyoyita < >> micou12 at gmail.com> >> >> >> wrote: >> >> >> >> >>>> >> >> >> >> >>>> Hello , >> >> >> >> >>>> >> >> >> >> >>>> I have configured RGW in my ceph cluster deployed using ceph >> >> >> ansible >> >> >> >> and >> >> >> >> >>>> create sub user to access the created containers and would >> like >> >> to >> >> >> >> >>> replace >> >> >> >> >>>> swift by RGW in the openstack side. Anyone can help on >> >> >> configuration >> >> >> >> to >> >> >> >> >>> be >> >> >> >> >>>> done in the OpenStack side in order to integrate those >> >> services. I >> >> >> >> have >> >> >> >> >>>> deployed OpenStack wallaby using Kolla-ansible on ubuntu >> 20.04. >> >> and >> >> >> >> ceph >> >> >> >> >>>> pacific 16.2.5 was deployed using ansible on ubuntu 20.04 >> >> >> >> >>>> >> >> >> >> >>>> Kindly help for the configuration or documentation. >> >> >> >> >>>> >> >> >> >> >>>> Best Regards >> >> >> >> >>>> >> >> >> >> >>>> Michel >> >> >> >> >>>> _______________________________________________ >> >> >> >> >>>> ceph-users mailing list -- ceph-users at ceph.io >> >> >> >> >>>> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >> >>> >> >> >> >> >>> _______________________________________________ >> >> >> >> >>> ceph-users mailing list -- ceph-users at ceph.io >> >> >> >> >>> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >> >>> >> >> >> >> >> _______________________________________________ >> >> >> >> >> ceph-users mailing list -- ceph-users at ceph.io >> >> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> ceph-users mailing list -- ceph-users at ceph.io >> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From geguileo at redhat.com Mon Sep 6 09:41:03 2021 From: geguileo at redhat.com (Gorka Eguileor) Date: Mon, 6 Sep 2021 11:41:03 +0200 Subject: Error adding a Plain Volume created through GUI to Instance In-Reply-To: References: Message-ID: <20210906094103.sn4bn6qb4sab3p2m@localhost> On 06/09, KK CHN wrote: > ExecutionError: Unexpected error while running command. > Command: rbd map volume-4dcb78d5-e0ce-4bfc-95ae-36a4417c35d8 --pool volumes > --id volumes --mon_host 10.236.223.131:6789 --mon_host 10.236.223.132:6789 > Exit code: 6 > Stdout: 'RBD image feature set mismatch. You can disable features > unsupported by the kernel with "rbd feature disable > volumes/volume-4dcb78d5-e0ce-4bfc-95ae-36a4417c35d8 journaling".\nIn some > cases useful info is found in syslog - try "dmesg | tail".\n' > Stderr: 'rbd: sysfs write failed\nrbd: map failed: (6) No such device or > address\n' > > I am trying to add a 20 GB plain volume created through Horizon GUI to an > existing Windows 2012 R VM running with VirtIO drivers installed. But this > volume adding results in the error as shown above, > > trace tells to fix it by rbd feature disable .. what is the > permanent fix for this issue ? and why it happens ? > > ( OpenStack Ussuri, Glance, Ceph , qemu-kvm) > > root at ctrl1:/home/cloud# ceph --version > ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus > (stable) > root at ctrl1:/home/cloud# Hi, Usually RBD volumes are directly attached on the hypervisor, but there was an encryption performance issue [1] so nova and os-brick added support to attaching these volumes to the host and then use them on the VM. Attaching volumes on the host uses the krbd module, whereas attaching them to the hypervisor directly uses librbd, with the first one lagging in terms of Ceph features. The issue you are facing is that the images in RBD are being created with features that are not supported by krbd. Fixes are: - Disable those features in your ceph configuration file (usually /etc/ceph/ceph.conf) Under [global] section add: rbd default features = 3 - Change "[workarounds]rbd_volume_local_attach" to False in Nova's config. Cheers, Gorka. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1762765 From wodel.youchi at gmail.com Mon Sep 6 09:48:05 2021 From: wodel.youchi at gmail.com (wodel youchi) Date: Mon, 6 Sep 2021 10:48:05 +0100 Subject: Does ceph and openstack communicate over the openstack-API network? Message-ID: Hi, Does ceph and openstack communicate over the openstack-API network, for nova, cinder, ...etc to be able to use ceph pools? Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balazs.gibizer at est.tech Mon Sep 6 10:42:24 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Mon, 06 Sep 2021 12:42:24 +0200 Subject: flavor with GPU In-Reply-To: References: Message-ID: On Mon, Sep 6, 2021 at 14:02, open infra wrote: > Hi, > > How to create a flavor with GPU? This document helps with the vGPU support in nova: https://docs.openstack.org/nova/latest/admin/virtual-gpu.html If you need a whole physical GPU to be passed to the guest instead then you can pass that through as a PCI device. Documented here: https://docs.openstack.org/nova/latest/admin/pci-passthrough.html Cheers, gibi > The way we set vcpu and memory. > > Regards, > Danishka From balazs.gibizer at est.tech Mon Sep 6 10:51:33 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Mon, 06 Sep 2021 12:51:33 +0200 Subject: [nova][placement] Xena release In-Reply-To: References: Message-ID: Hi, On Thu, Aug 19, 2021 at 11:27, Balazs Gibizer wrote: > Hi, > > We are getting close to the Xena release. So I created a tracking > etherpad[1] with the schedule and TODOs. > > One thing that I want to highlight is that we will hit Feature Freeze > on 3rd of September. As the timeframe between FF and RC1 is short I'm > not > plannig with FFEs. Patches that are approved before 3rd of Sept EOB > can be > rechecked or rebased if needed and then re-approved. If you have a > patch that is really close but not approved before the deadline, and > you think there are two cores that willing to review it before RC1, > then please send a mail to the ML with [nova][FFE] subject prefix not > later than 7th of Sept EOB. As of last Friday we hit Feature Freeze. So still open features needs to be re-proposed to Yoga. The nova-spec repo and launchpad can be used to propose thing to yoga already. If you have a feature that is really close to land the please follow the above FFE process. If you have a feature patch that was approved before the freeze but not landed yet then those patches can be rechecked or rebased and re-approved if needed. Please let the core team know. Cheers, gibi > > Cheers, > gibi > > [1] https://etherpad.opendev.org/p/nova-xena-rc-potential > > > > From tpb at dyncloud.net Mon Sep 6 10:53:17 2021 From: tpb at dyncloud.net (Tom Barron) Date: Mon, 6 Sep 2021 06:53:17 -0400 Subject: Does ceph and openstack communicate over the openstack-API network? In-Reply-To: References: Message-ID: <20210906105317.xuldv354zq663hw4@barron.net> On 06/09/21 10:48 +0100, wodel youchi wrote: >Hi, > >Does ceph and openstack communicate over the openstack-API network, for >nova, cinder, ...etc to be able to use ceph pools? > >Regards. Generally it's not a good idea to combine the openstack API network and the Ceph "public" network since you want to reserve access to the latter to the cloud infrastructure itself (compute nodes, controllers) -- which is under control of the cloud administrators -- and prevent regular users (tenants) from accessing the storage infrastructure. From openinfradn at gmail.com Mon Sep 6 10:56:57 2021 From: openinfradn at gmail.com (open infra) Date: Mon, 6 Sep 2021 16:26:57 +0530 Subject: flavor with GPU In-Reply-To: References: Message-ID: Thanks Balazs. If I already create vgpu I just need to update the flavor using the following command? openstack flavor set vgpu_1 --property "resources:VGPU=1" On Mon, Sep 6, 2021 at 4:12 PM Balazs Gibizer wrote: > > > On Mon, Sep 6, 2021 at 14:02, open infra wrote: > > Hi, > > > > How to create a flavor with GPU? > This document helps with the vGPU support in nova: > https://docs.openstack.org/nova/latest/admin/virtual-gpu.html > > If you need a whole physical GPU to be passed to the guest instead then > you can pass that through as a PCI device. Documented here: > https://docs.openstack.org/nova/latest/admin/pci-passthrough.html > > Cheers, > gibi > > > The way we set vcpu and memory. > > > > Regards, > > Danishka > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.dibbo at stfc.ac.uk Mon Sep 6 10:57:43 2021 From: alexander.dibbo at stfc.ac.uk (Alexander Dibbo - STFC UKRI) Date: Mon, 6 Sep 2021 10:57:43 +0000 Subject: Adding another cell(v2) Message-ID: Hi, Could someone point me to the documentation on how to add a new cell to cellsv2? I have found various documentation that says to follow the procedure for Cells V1 but I cant find documentation for that. Below is the link to the doc I have found https://docs.openstack.org/nova/train/user/cells.html I am running train and have found myself needing to use more cells to deal with our latest hardware deployment. We use a home rolled deployment system. Thanks In Advance Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io To raise a support ticket with the cloud team please email cloud-support at gridpp.rl.ac.uk To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balazs.gibizer at est.tech Mon Sep 6 10:59:28 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Mon, 06 Sep 2021 12:59:28 +0200 Subject: flavor with GPU In-Reply-To: References: Message-ID: <4JE0ZQ.8R16JSDM1MXZ@est.tech> On Mon, Sep 6, 2021 at 16:26, open infra wrote: > Thanks Balazs. > > If I already create vgpu I just need to update the flavor using the > following command? > > openstack flavor set vgpu_1 --property "resources:VGPU=1" Yes, that is the way to request a VGPU via a flavor [1] [1] https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#configure-a-flavor-controller gibi > > On Mon, Sep 6, 2021 at 4:12 PM Balazs Gibizer > wrote: >> >> >> On Mon, Sep 6, 2021 at 14:02, open infra >> wrote: >> > Hi, >> > >> > How to create a flavor with GPU? >> This document helps with the vGPU support in nova: >> https://docs.openstack.org/nova/latest/admin/virtual-gpu.html >> >> If you need a whole physical GPU to be passed to the guest instead >> then >> you can pass that through as a PCI device. Documented here: >> https://docs.openstack.org/nova/latest/admin/pci-passthrough.html >> >> Cheers, >> gibi >> >> > The way we set vcpu and memory. >> > >> > Regards, >> > Danishka >> >> From manchandavishal143 at gmail.com Mon Sep 6 11:00:02 2021 From: manchandavishal143 at gmail.com (vishal manchanda) Date: Mon, 6 Sep 2021 16:30:02 +0530 Subject: [horizon] Support for Django 3.x In-Reply-To: References: Message-ID: Hi Thomas, Horizon team working on adding django3.x support in horizon. A series of patches to add django3.x support is already up [1] and We recently merged a non-voting job to test django3.x which is failing right now[2]. The main issue we are facing in migration to django3.x is horizon using django-pyscss which isn't compatible with django3.x although we have some workaround to fix it[3] but it is not the right way. So I tried to contact the django-pyscss maintainer, but they didn't respond. Then we plan to use an alternative of django-pyscss i.e. django-sass-processor. Update on django-sass-processor: Me and radomir from horizon team worked on adding django-sass-processor support in horizon[4]. As far our testing it works fine with horizon but horizon default theme doesn't work with it right now and fails with error[5]. I am working on this issue, any help is really appreciated. [1] https://review.opendev.org/q/topic:%22django3-support%22+(status:open%20OR%20status:merged) [2] https://review.opendev.org/c/openstack/horizon/+/777390 [3] https://review.opendev.org/c/openstack/horizon/+/777391/ [4] https://review.opendev.org/c/openstack/horizon/+/794809 [5] https://paste.opendev.org/show/808595/ Thanks & Regards, Vishal Manchanda On Mon, Sep 6, 2021 at 1:06 PM Thomas Goirand wrote: > Hi, > > It's been a long time that Django 3.x is around, and therefore, it'd be > nice if Horizon had some kind of support for it. Indeed, sooner or > later, Django 3.x will be uploaded to Debian unstable. That day, I hope > Horizon doesn't just break. > > So, what's the status? Is anyone is at least working on it? > > Cheers, > > Thomas Goirand (zigo) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fpantano at redhat.com Mon Sep 6 11:01:20 2021 From: fpantano at redhat.com (Francesco Pantano) Date: Mon, 6 Sep 2021 13:01:20 +0200 Subject: [ceph-users] Re: Replacing swift with RGW In-Reply-To: References: <20210901122221.Horde.v0FeoCUfex4WYSaXReGzV1i@webmail.nde.ag> <20210901134023.Horde.ZhZ0JBty-nYNsBfGe6WOijH@webmail.nde.ag> <20210902071439.Horde.4cayqDV75oyYS75eyQZMSPD@webmail.nde.ag> <20210902081741.Horde.Xp06qPUkci05sKXFlo1F5ap@webmail.nde.ag> <20210902083925.Horde.a0LD9j43RkGPIl7UuWCRtyu@webmail.nde.ag> Message-ID: Hello, not sure this helps for your configuration but we used to support this scenario with TripleO [1]. The scenario is related to an external Ceph cluster deployed via ceph-ansible (or any other Ceph related tool) and consumed by some OSP services. In particular you can configure an external Swift (or Ceph RadosGW) proxy, and you can probably refer to the doc [1] to double check the parameters (in terms of endpoints and other configs) on both OSP and ceph side. Also, make sure the openstack user you're using to issue the "swift stat" command has the right roles assigned (the same accepted by RGW and defined in the ceph config). [1] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/swift_external.html On Mon, Sep 6, 2021 at 11:40 AM Michel Niyoyita wrote: > Hello, > > I am trying to replace swift by RGW as backend storage but I failed once I > try to post a container in the OpenStack side however, all interfaces are > configured (admin, public and internal). but Once I post from RGW host it > is created . Another issue is that object storage does not appear on the > horizon dashboard . I have deployed openstack all-in-one using > kolla-ansible and Os is ubuntu > > (kolla-open1) stack at kolla-open1:~$ swift -v post myswift > Container POST failed: http://ceph-osd3:8080/swift/v1/myswift 401 > Unauthorized b'AccessDenied' > Failed Transaction ID: tx000000000000000000008-006135dcbd-87d63-default > > (kolla-open1) stack at kolla-open1:~$ swift list > Account GET failed: http://ceph-osd3:8080/swift/v1?format=json 401 > Unauthorized [first 60 chars of response] > b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000c-' > Failed Transaction ID: tx00000000000000000000c-006135de42-87d63-default > > Kindly help to solve the issue > > Michel > > On Thu, Sep 2, 2021 at 4:28 PM Alex Schultz wrote: > >> The swift docs are a bit out of date as they still reference python2 >> despite python3 being supported for some time now. Replace python- with >> python3- and try again. >> >> >> On Thu, Sep 2, 2021 at 7:35 AM Michel Niyoyita wrote: >> >>> >>> >>> ---------- Forwarded message --------- >>> From: Michel Niyoyita >>> Date: Thu, Sep 2, 2021 at 12:17 PM >>> Subject: Fwd: [ceph-users] Re: Replacing swift with RGW >>> To: >>> >>> >>> >>> >>> ---------- Forwarded message --------- >>> From: Eugen Block >>> Date: Thu, Sep 2, 2021 at 10:39 AM >>> Subject: Re: [ceph-users] Re: Replacing swift with RGW >>> To: Michel Niyoyita >>> >>> >>> You should continue this thread on the openstack-discuss mailing list >>> (openstack-discuss at lists.openstack.org) since this is not a ceph issue. >>> I'm not familiar with kolla and I don't know the requirements to >>> install openstack-swift, you'll need to ask the openstack community. >>> >>> >>> Zitat von Michel Niyoyita : >>> >>> > Below are errors I am getting once I try to run swift commands . the >>> second >>> > one is the error I get once try to install python-swiftclient >>> > >>> > (kolla-open) [stack at kolla-open ~]$ swift -v stat >>> > -bash: swift: command not found >>> > (kolla-open) [stack at kolla-open ~]$ sudo yum -y install >>> python-swiftclient >>> > Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 >>> 09:21:53 AM >>> > CAT. >>> > No match for argument: python-swiftclient >>> > Error: Unable to find a match: python-swiftclient >>> > (kolla-open) [stack at kolla-open ~]$ >>> > >>> > Waiting for your help >>> > >>> > On Thu, Sep 2, 2021 at 10:17 AM Eugen Block wrote: >>> > >>> >> I can't tell for sure, but yes, I believe you need the openstack-swift >>> >> package (with dependencies). What errors do you get? The more >>> >> information you share the better people can help. >>> >> >>> >> >>> >> Zitat von Michel Niyoyita : >>> >> >>> >> > I tried to install "sudo yum -y install python-swiftclient" on >>> openstack >>> >> > side but fails . are there openastack-shwift packages which are >>> needed? >>> >> > if are there please help me to get . may be also it is the cause I >>> am >>> >> > failing to run swift command on openstack cli side. >>> >> > >>> >> > thank you for your continued support. >>> >> > >>> >> > Micheal >>> >> > >>> >> > On Thu, Sep 2, 2021 at 9:14 AM Eugen Block wrote: >>> >> > >>> >> >> I only configured the endpoints for the clients to directly access >>> the >>> >> >> RGWs, but you'll probably need to install the openstack-swift >>> package. >>> >> >> Or have you done that already? >>> >> >> >>> >> >> >>> >> >> Zitat von Michel Niyoyita : >>> >> >> >>> >> >> > Thank you Eugen for your prompt response. >>> >> >> > >>> >> >> > Now the commands provided work. but I am not finding the object >>> >> storage >>> >> >> on >>> >> >> > the horizon dashboard , but it appears in the system information >>> >> >> services. >>> >> >> > [image: image.png] >>> >> >> > so my question is how to configure it in order that it can >>> appear in >>> >> the >>> >> >> > dashboard . >>> >> >> > >>> >> >> > Michel >>> >> >> > >>> >> >> > On Wed, Sep 1, 2021 at 3:49 PM Eugen Block >>> wrote: >>> >> >> > >>> >> >> >> Sorry, one little detail slipped through, the '--region' flag >>> has to >>> >> >> >> be put before the 'service' name. The correct command would be: >>> >> >> >> >>> >> >> >> openstack endpoint create --region RegionOne swift admin >>> >> >> >> http://ceph-osd3:8080/swift/v1 >>> >> >> >> >>> >> >> >> and respectively for the other interfaces. >>> >> >> >> >>> >> >> >> >>> >> >> >> Zitat von Eugen Block : >>> >> >> >> >>> >> >> >> > Hi, >>> >> >> >> > >>> >> >> >> > this is not a ceph issue but your openstack cli command as the >>> >> error >>> >> >> >> > message states. >>> >> >> >> > >>> >> >> >> > Try one interface at a time: >>> >> >> >> > >>> >> >> >> > openstack endpoint create swift public >>> >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >>> >> >> >> > openstack endpoint create swift admin >>> >> http://ceph-osd3:8080/swift/v1 >>> >> >> >> > --region RegionOne swift >>> >> >> >> > openstack endpoint create swift internal >>> >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > Zitat von Michel Niyoyita : >>> >> >> >> > >>> >> >> >> >> Hello , >>> >> >> >> >> >>> >> >> >> >> Below are errors I am getting once I am trying to integrate >>> swift >>> >> >> with >>> >> >> >> >> Radosgateway. >>> >> >> >> >> >>> >> >> >> >> From openstack side once i try to endpoint which will point >>> to the >>> >> >> >> >> radosgateway : >>> >> >> >> >> >>> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ openstack endpoint >>> create >>> >> >> >> >> --publicurl http://ceph-osd3:8080/swift/v1 --adminurl >>> >> >> >> >> http://ceph-osd3:8080/swift/v1 --internal >>> >> >> >> http://ceph-osd3:8080/swift/v1 >>> >> >> >> >> --region RegionOne swift >>> >> >> >> >> usage: openstack endpoint create [-h] [-f >>> >> >> {json,shell,table,value,yaml}] >>> >> >> >> >> [-c COLUMN] [--noindent] >>> [--prefix >>> >> >> >> PREFIX] >>> >> >> >> >> [--max-width ] >>> >> [--fit-width] >>> >> >> >> >> [--print-empty] [--region >>> >> >> ] >>> >> >> >> >> [--enable | --disable] >>> >> >> >> >> >>> >> >> >> >> openstack endpoint create: error: argument : >>> invalid >>> >> >> choice: >>> >> >> >> ' >>> >> >> >> >> http://ceph-osd3:8080/swift/v1' (choose from 'admin', >>> 'public', >>> >> >> >> 'internal') >>> >> >> >> >> >>> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ >>> >> >> >> >> >>> >> >> >> >> below are my /etc/ceph/ceph.conf file : >>> >> >> >> >> >>> >> >> >> >> [client.rgw.ceph-osd3] >>> >> >> >> >> rgw_dns_name = ceph-osd3 >>> >> >> >> >> host = ceph-osd3 >>> >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3/keyring >>> >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.log >>> >> >> >> >> rgw frontends = civetweb port=10.10.29.110:8080 >>> num_threads=100 >>> >> >> >> >> rgw_keystone_url=http://10.10.29.150:35357 >>> >> >> >> >> rgw_keystone_admin_user=admin >>> >> >> >> >> rgw_keystone_admin_password=admin >>> >> >> >> >> rgw_keystone_admin_tenant=admin >>> >> >> >> >> rgw_keystone_accepted_role=admin Member swiftoperator >>> >> >> >> >> rgw_keystone_token_cache_size=200 >>> >> >> >> >> rgw_keystone_revocation_interval=300 >>> >> >> >> >> >>> >> >> >> >> [client.rgw.ceph-osd3.rgw0] >>> >> >> >> >> host = ceph-osd3 >>> >> >> >> >> keyring = >>> /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring >>> >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log >>> >> >> >> >> rgw frontends = beast endpoint=10.10.29.110:8080 >>> >> >> >> >> rgw thread pool size = 512 >>> >> >> >> >> >>> >> >> >> >> please note that my rgw_dns_name = ceph_osd3 with >>> 10.10.29.110 as >>> >> IP >>> >> >> >> >> >>> >> >> >> >> and 10.10.29.150 all-in-one IP >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> Please crosscheck where I might make mistake and try to >>> correct. >>> >> >> >> >> >>> >> >> >> >> Best regards >>> >> >> >> >> >>> >> >> >> >> Michel >>> >> >> >> >> >>> >> >> >> >> On Mon, Aug 30, 2021 at 11:25 AM Etienne Menguy < >>> >> >> >> etienne.menguy at croit.io> >>> >> >> >> >> wrote: >>> >> >> >> >> >>> >> >> >> >>> Hi, >>> >> >> >> >>> >>> >> >> >> >>> There are some information on Ceph documentation >>> >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/ < >>> >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/> . >>> >> >> >> >>> - Use keystone as auth for RGW >>> >> >> >> >>> - Create service and register your RGW as swift >>> >> >> >> >>> >>> >> >> >> >>> Étienne >>> >> >> >> >>> >>> >> >> >> >>>> On 27 Aug 2021, at 15:47, Michel Niyoyita < >>> micou12 at gmail.com> >>> >> >> wrote: >>> >> >> >> >>>> >>> >> >> >> >>>> Hello , >>> >> >> >> >>>> >>> >> >> >> >>>> I have configured RGW in my ceph cluster deployed using >>> ceph >>> >> >> ansible >>> >> >> >> and >>> >> >> >> >>>> create sub user to access the created containers and would >>> like >>> >> to >>> >> >> >> >>> replace >>> >> >> >> >>>> swift by RGW in the openstack side. Anyone can help on >>> >> >> configuration >>> >> >> >> to >>> >> >> >> >>> be >>> >> >> >> >>>> done in the OpenStack side in order to integrate those >>> >> services. I >>> >> >> >> have >>> >> >> >> >>>> deployed OpenStack wallaby using Kolla-ansible on ubuntu >>> 20.04. >>> >> and >>> >> >> >> ceph >>> >> >> >> >>>> pacific 16.2.5 was deployed using ansible on ubuntu 20.04 >>> >> >> >> >>>> >>> >> >> >> >>>> Kindly help for the configuration or documentation. >>> >> >> >> >>>> >>> >> >> >> >>>> Best Regards >>> >> >> >> >>>> >>> >> >> >> >>>> Michel >>> >> >> >> >>>> _______________________________________________ >>> >> >> >> >>>> ceph-users mailing list -- ceph-users at ceph.io >>> >> >> >> >>>> To unsubscribe send an email to ceph-users-leave at ceph.io >>> >> >> >> >>> >>> >> >> >> >>> _______________________________________________ >>> >> >> >> >>> ceph-users mailing list -- ceph-users at ceph.io >>> >> >> >> >>> To unsubscribe send an email to ceph-users-leave at ceph.io >>> >> >> >> >>> >>> >> >> >> >> _______________________________________________ >>> >> >> >> >> ceph-users mailing list -- ceph-users at ceph.io >>> >> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> _______________________________________________ >>> >> >> >> ceph-users mailing list -- ceph-users at ceph.io >>> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >>> >> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >>> >> >>> >> >>> >> >>> >>> >>> >>> -- Francesco Pantano GPG KEY: F41BD75C -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Mon Sep 6 11:20:45 2021 From: tonykarera at gmail.com (Karera Tony) Date: Mon, 6 Sep 2021 13:20:45 +0200 Subject: Changing Magnum Heat template to create node network with vxlan instead of vlan In-Reply-To: References: Message-ID: Hello Maysa, There is no process running on port 8080 and yes I have configured kuryr Regards Tony Karera On Mon, Sep 6, 2021 at 10:17 AM Maysa De Macedo Souza wrote: > Hello, > > Just some extra thought, did you check if there is some process running on > port 8080? It's possible that the API failed to start, is Kuryr being used? > > Thanks, > Maysa Macedo. > > On Tue, Aug 31, 2021 at 1:57 PM Karera Tony wrote: > >> Hello Guys, >> >> I was able to configure the cluster internal network with Vlans. The >> communication was ok but I still get the same error >> >> + sleep 5 >> ++ kubectl get --raw=/healthz >> The connection to the server localhost:8080 was refused - did you specify >> the right host or port? >> + '[' ok = '' ']' >> >> Regards >> >> Tony Karera >> >> >> >> >> On Sat, Aug 28, 2021 at 4:43 AM Mohammed Naser >> wrote: >> >>> AFAIK, Magnum does not control what encapsulation type is being used. >>> >>> Is there a chance you have “vlan” inside tenant_network_types ? >>> >>> On Tue, Aug 24, 2021 at 12:05 AM Karera Tony >>> wrote: >>> >>>> Hello Team, >>>> >>>> I was able to deploy Openstack with Magnum and Zun enabled. >>>> >>>> Unfortunately, When I create a cluster, Heat configure the fix/_network >>>> for the nodes with Vlan and the nodes fail to get IPs. >>>> >>>> I was wondering if it is possible to trick the heat templates to create >>>> vxlan network instead of vlan. >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> -- >>> Mohammed Naser >>> VEXXHOST, Inc. >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephine.seifert at secustack.com Mon Sep 6 11:25:28 2021 From: josephine.seifert at secustack.com (Josephine Seifert) Date: Mon, 6 Sep 2021 13:25:28 +0200 Subject: [Image Encryption] Next Meeting on Monday 13 Message-ID: Hi everyone, as I am on vacation the next meeting will be next monday - the 13th of September. greetings Josephine (Luzi) From wilrichard00 at gmail.com Mon Sep 6 12:22:29 2021 From: wilrichard00 at gmail.com (Jean Richard) Date: Mon, 6 Sep 2021 08:22:29 -0400 Subject: [Neutron] [Kolla ansible ] external network setting for vlan In-Reply-To: References: Message-ID: Hi Rahul, I am new to Openstack. I have been trying to deploy Devstack (not much success) for practice and later Kolla Ansible All-in-one (again not much success). Can you please share your configuration file for deploying the Wallaby release? Thank you very much. Best regards, Jean Richard On Mon, Sep 6, 2021 at 3:09 AM Lajos Katona wrote: > Hi Rahul, > Not sure where You started your research so I add some basic documentation > (but it has links to further docs): > > - Openstack Networking concepts: > https://docs.openstack.org/neutron/latest/admin/intro-os-networking.html > - Deployment with OVS (not sure which mechanism driver You use): > https://docs.openstack.org/neutron/latest/admin/deploy-ovs.html > - here You can find config and CLI examples if You open the links. > > Lajos (lajoskatona) > > > Rahul Miragi <22rahulmiragi at gmail.com> ezt írta (időpont: 2021. szept. > 3., P, 21:46): > >> Hi team, >> >> I am using kolla ansible to deploy wallaby release on single node. I have >> successfully deployed it and now I want to create external network and >> attach external IPs to openstack VMs. But somehow I’m unable to configure >> it. Could you please share some articles or config examples. >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From geguileo at redhat.com Mon Sep 6 13:28:13 2021 From: geguileo at redhat.com (Gorka Eguileor) Date: Mon, 6 Sep 2021 15:28:13 +0200 Subject: [dev][cinder] Consultation about new cinder-backup features In-Reply-To: References: Message-ID: <20210906132813.xsaxbsyyvf4ey4vm@localhost> On 27/08, Daniel de Oliveira Pereira wrote: > Hello everyone, > > We have prototyped some new features on Cinder for our clients, and we > think that they are nice features and good candidates to be part of > upstream Cinder, so we would like to get feedback from OpenStack > community about these features and if you would be willing to accept > them in upstream OpenStack. Hi Daniel, Thank you very much for your willingness to give back!!! > > Our team implemented the following features for cinder-backup service: > > 1. A multi-backend backup driver, that allow OpenStack users to > choose, via API/CLI/Horizon, which backup driver (Ceph or NFS, in our > prototype) will be used during a backup operation to create a new volume > backup. This is a feature that has been discussed before, and e0ne already did some of the prerequisites for it. > 2. An improved NFS backup driver, that allow OpenStack users to back > up their volumes to private NFS servers, providing the NFS hostpath at > runtime via API/CLI/Horizon, while creating the volume backup. > What about the username and password? Can backups be restored from a remote location as well? This sounds like a very cool feature, but I'm not too comfortable with having it in Cinder. The idea is that Cinder provides an abstraction and doesn't let users know about implementation details. With that feature as it is a user could request a backup to an off-site location that could result in congestion in one of the outbound connections. I can only think of this being acceptable for admin users, and in that case I think it would be best to use the multi-backup destination feature instead. After all, how many times do we have to backup to a different location? Maybe I'm missing a use case. If the community thinks this as a desired feature I would encourage adding it with a policy that disables it by default. > Considering that cinder was configured to use the multi-backend backup > driver, this is how it works: > > During a volume backup operation, the user provides a "location" > parameter to indicate which backend will be used, and the backup > hostpath, if applicable (for NFS driver), to create the volume backup. > For instance: > > - Creating a backup using Ceph backend: > $ openstack volume backup create --name --location > ceph > > - Creating a backup using the improved NFS backend: > $ openstack volume backup create --name --location > nfs://my.nfs.server:/backups > > If the user chooses Ceph backend, the Ceph driver will be used to > create the backup. If the user chooses the NFS backend, the improved NFS > driver, previously mentioned, will be used to create the backup. > > The backup location, if provided, is stored on Cinder database, and > can be seen fetching the backup details: > $ openstack volume backup show > > Briefly, this is how the features were implemented: > > - Cinder API was updated to add an optional location parameter to > "create backup" method. Horizon, and OpenStack and Cinder CLIs were > updated accordingly, to handle the new parameter. > - Cinder backup controller was updated to handle the backup location > parameter, and a validator for the parameter was implemented using the > oslo config library. > - Cinder backup object model was updated to add a nullable location > property, so that the backup location could be stored on cinder database. > - a new backup driver base class, that extends BackupDriver and > accepts a backup context object, was implemented to handle the backup > configuration provided at runtime by the user. This new backup base > class requires that the concrete drivers implement a method to validate > the backup context (similar to BackupDriver.check_for_setup_error) > - the 2 new backup drivers, previously mentioned, were implemented > using these new backup base class. > - in BackupManager class, the "service" attribute, that on upstream > OpenStack holds the backup driver class name, was re-implemented as a > factory function that accepts a backup context object and return an > instance of a backup driver, according to the backup driver configured > on cinder.conf file and the backup context provided at runtime by the user. > - All the backup operations continue working as usual. > When this feature was discussed upstream we liked the idea of implementing this like we do multi-backends for the volume service, adding backup-types. In latest code backup creation operations have been modified to go through the scheduler, so that's a piece that is already implemented. > Could you please let us know your thoughts about these features and if > you would be open to adding them to upstream Cinder? If yes, we would be > willing to submit the specs and work on the upstream implementation, if > they are approved. > > Regards, > Daniel Pereira > I believe you will have the full community's support on the first idea (though probably not on the proposed implementation). I'm not so sure on the second one, iti will most likely depend on the use cases. Many times the reasons why features are dismissed upstream is because there are no clear use cases that justify the addition of the code. Looking forward to continuing this conversation at the PTG, IRC, in a spec, or through here. Cheers, Gorka. From zhangbailin at inspur.com Mon Sep 6 14:06:23 2021 From: zhangbailin at inspur.com (=?gb2312?B?QnJpbiBaaGFuZyjVxbDZwdYp?=) Date: Mon, 6 Sep 2021 14:06:23 +0000 Subject: =?gb2312?B?u9i4tKO6W2xpc3RzLm9wZW5zdGFjay5vcme0+reiXWZsYXZvciB3aXRoIEdQ?= =?gb2312?Q?U?= In-Reply-To: References: , Message-ID: If you deployed Cyborg on you environment, you can create a device_profile with GPU(openstack device profile create …)，then add the metadata of device_profile:GPU in the flavor，then you can boot an instance with this flavor，it will be done by nova and cyborg. Below [1] is the SSD supported test that you can refer, the GPU test report we will plan to add in Yoga release. [1]https://wiki.openstack.org/wiki/Cyborg/TestReport/InspurFPGA -------- 原始邮件 -------- 发件人： open infra 日期： 2021年9月6日周一 16:37 收件人： openstack-discuss 主 题： [lists.openstack.org代发]flavor with GPU Hi, How to create a flavor with GPU? The way we set vcpu and memory. Regards, Danishka From tonykarera at gmail.com Mon Sep 6 14:26:56 2021 From: tonykarera at gmail.com (Karera Tony) Date: Mon, 6 Sep 2021 16:26:56 +0200 Subject: Octavia Issues Message-ID: Dear Team, I have openstack wallaby running on Ubuntu deployed using kolla-ansible but whenever I create a load balancer I get the error below in the octavia-worker.log octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: Failed to retrieve 2021-09-06 14:14:49.392 20 ERROR oslo_messaging.rpc.server But I have an image with the correct tag as shown below. (kolla-openstack) stack at deployment:~$ openstack image show amphora1 | grep tag | tags | amphora Kindly advise if I am missing anything. Regards Tony Karera -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Mon Sep 6 14:44:01 2021 From: eblock at nde.ag (Eugen Block) Date: Mon, 06 Sep 2021 14:44:01 +0000 Subject: [ceph-users] Re: Replacing swift with RGW In-Reply-To: References: <20210901122221.Horde.v0FeoCUfex4WYSaXReGzV1i@webmail.nde.ag> <20210901134023.Horde.ZhZ0JBty-nYNsBfGe6WOijH@webmail.nde.ag> <20210902071439.Horde.4cayqDV75oyYS75eyQZMSPD@webmail.nde.ag> <20210902081741.Horde.Xp06qPUkci05sKXFlo1F5ap@webmail.nde.ag> <20210902083925.Horde.a0LD9j43RkGPIl7UuWCRtyu@webmail.nde.ag> Message-ID: <20210906144401.Horde.7j4UyxpB4wl0f5RgwX1Y0FD@webmail.nde.ag> It's hard to tell what is wrong with your setup, I don't really use mine but this was the last working config I had to be able to create swift containers directly in RGW: ---snip--- # ceph.conf: [client.rgw.ses6-mon1] rgw frontends = "beast port=80" rgw dns name = ses6-mon1.example.com rgw enable usage log = true rgw thread pool size = 512 rgw keystone api version = 3 rgw keystone url = http://control-node.example.com:5000 rgw keystone admin user = rgw rgw keystone admin password = **** rgw keystone admin domain = default rgw keystone admin project = service rgw keystone accepted roles = admin,Member,_member_,member rgw keystone verify ssl = false rgw s3 auth use keystone = true rgw keystone revocation interval = 0 # User role (I don't think admin is required) openstack role add --user rgw --project 9e8a67da237a4b26afb2819d2dea2219 admin # Create keystone endpoints openstack endpoint create --region RegionOne swift admin "http://ses6-mon1.example.com:80/swift/v1" openstack endpoint create --region RegionOne swift internal "http://ses6-mon1.example.com:80/swift/v1" openstack endpoint create --region RegionOne swift public "http://ses6-mon1.example.com:80/swift/v1" # Create container and files openstack container create swift1 +---------+-----------+---------------------------------------------------+ | account | container | x-trans-id | +---------+-----------+---------------------------------------------------+ | v1 | swift1 | tx000000000000000000001-0060b4ba48-d724dc-default | +---------+-----------+---------------------------------------------------+ openstack object create --name file1 swift1 chef-client.log +--------+-----------+----------------------------------+ | object | container | etag | +--------+-----------+----------------------------------+ | file1 | swift1 | 56a1ed3b201c1e753bcbe80c640349f7 | +--------+-----------+----------------------------------+ ---snip--- You are mixing dns names and IP addresses, I can't tell if that's a problem but it probably should work, I'm not sure. Compared to my ceph.conf these are the major differences: rgw keystone verify ssl = false rgw s3 auth use keystone = true rgw keystone revocation interval = 0 And I don't use rgw_keystone_token_cache_size. Maybe try again with the options I use. Zitat von Michel Niyoyita : > Hello, > > I am trying to replace swift by RGW as backend storage but I failed once I > try to post a container in the OpenStack side however, all interfaces are > configured (admin, public and internal). but Once I post from RGW host it > is created . Another issue is that object storage does not appear on the > horizon dashboard . I have deployed openstack all-in-one using > kolla-ansible and Os is ubuntu > > (kolla-open1) stack at kolla-open1:~$ swift -v post myswift > Container POST failed: http://ceph-osd3:8080/swift/v1/myswift 401 > Unauthorized b'AccessDenied' > Failed Transaction ID: tx000000000000000000008-006135dcbd-87d63-default > > (kolla-open1) stack at kolla-open1:~$ swift list > Account GET failed: http://ceph-osd3:8080/swift/v1?format=json 401 > Unauthorized [first 60 chars of response] > b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000c-' > Failed Transaction ID: tx00000000000000000000c-006135de42-87d63-default > > Kindly help to solve the issue > > Michel > > On Thu, Sep 2, 2021 at 4:28 PM Alex Schultz wrote: > >> The swift docs are a bit out of date as they still reference python2 >> despite python3 being supported for some time now. Replace python- with >> python3- and try again. >> >> >> On Thu, Sep 2, 2021 at 7:35 AM Michel Niyoyita wrote: >> >>> >>> >>> ---------- Forwarded message --------- >>> From: Michel Niyoyita >>> Date: Thu, Sep 2, 2021 at 12:17 PM >>> Subject: Fwd: [ceph-users] Re: Replacing swift with RGW >>> To: >>> >>> >>> >>> >>> ---------- Forwarded message --------- >>> From: Eugen Block >>> Date: Thu, Sep 2, 2021 at 10:39 AM >>> Subject: Re: [ceph-users] Re: Replacing swift with RGW >>> To: Michel Niyoyita >>> >>> >>> You should continue this thread on the openstack-discuss mailing list >>> (openstack-discuss at lists.openstack.org) since this is not a ceph issue. >>> I'm not familiar with kolla and I don't know the requirements to >>> install openstack-swift, you'll need to ask the openstack community. >>> >>> >>> Zitat von Michel Niyoyita : >>> >>> > Below are errors I am getting once I try to run swift commands . the >>> second >>> > one is the error I get once try to install python-swiftclient >>> > >>> > (kolla-open) [stack at kolla-open ~]$ swift -v stat >>> > -bash: swift: command not found >>> > (kolla-open) [stack at kolla-open ~]$ sudo yum -y install >>> python-swiftclient >>> > Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 09:21:53 >>> AM >>> > CAT. >>> > No match for argument: python-swiftclient >>> > Error: Unable to find a match: python-swiftclient >>> > (kolla-open) [stack at kolla-open ~]$ >>> > >>> > Waiting for your help >>> > >>> > On Thu, Sep 2, 2021 at 10:17 AM Eugen Block wrote: >>> > >>> >> I can't tell for sure, but yes, I believe you need the openstack-swift >>> >> package (with dependencies). What errors do you get? The more >>> >> information you share the better people can help. >>> >> >>> >> >>> >> Zitat von Michel Niyoyita : >>> >> >>> >> > I tried to install "sudo yum -y install python-swiftclient" on >>> openstack >>> >> > side but fails . are there openastack-shwift packages which are >>> needed? >>> >> > if are there please help me to get . may be also it is the cause I >>> am >>> >> > failing to run swift command on openstack cli side. >>> >> > >>> >> > thank you for your continued support. >>> >> > >>> >> > Micheal >>> >> > >>> >> > On Thu, Sep 2, 2021 at 9:14 AM Eugen Block wrote: >>> >> > >>> >> >> I only configured the endpoints for the clients to directly access >>> the >>> >> >> RGWs, but you'll probably need to install the openstack-swift >>> package. >>> >> >> Or have you done that already? >>> >> >> >>> >> >> >>> >> >> Zitat von Michel Niyoyita : >>> >> >> >>> >> >> > Thank you Eugen for your prompt response. >>> >> >> > >>> >> >> > Now the commands provided work. but I am not finding the object >>> >> storage >>> >> >> on >>> >> >> > the horizon dashboard , but it appears in the system information >>> >> >> services. >>> >> >> > [image: image.png] >>> >> >> > so my question is how to configure it in order that it can appear >>> in >>> >> the >>> >> >> > dashboard . >>> >> >> > >>> >> >> > Michel >>> >> >> > >>> >> >> > On Wed, Sep 1, 2021 at 3:49 PM Eugen Block wrote: >>> >> >> > >>> >> >> >> Sorry, one little detail slipped through, the '--region' flag >>> has to >>> >> >> >> be put before the 'service' name. The correct command would be: >>> >> >> >> >>> >> >> >> openstack endpoint create --region RegionOne swift admin >>> >> >> >> http://ceph-osd3:8080/swift/v1 >>> >> >> >> >>> >> >> >> and respectively for the other interfaces. >>> >> >> >> >>> >> >> >> >>> >> >> >> Zitat von Eugen Block : >>> >> >> >> >>> >> >> >> > Hi, >>> >> >> >> > >>> >> >> >> > this is not a ceph issue but your openstack cli command as the >>> >> error >>> >> >> >> > message states. >>> >> >> >> > >>> >> >> >> > Try one interface at a time: >>> >> >> >> > >>> >> >> >> > openstack endpoint create swift public >>> >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >>> >> >> >> > openstack endpoint create swift admin >>> >> http://ceph-osd3:8080/swift/v1 >>> >> >> >> > --region RegionOne swift >>> >> >> >> > openstack endpoint create swift internal >>> >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > Zitat von Michel Niyoyita : >>> >> >> >> > >>> >> >> >> >> Hello , >>> >> >> >> >> >>> >> >> >> >> Below are errors I am getting once I am trying to integrate >>> swift >>> >> >> with >>> >> >> >> >> Radosgateway. >>> >> >> >> >> >>> >> >> >> >> From openstack side once i try to endpoint which will point >>> to the >>> >> >> >> >> radosgateway : >>> >> >> >> >> >>> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ openstack endpoint >>> create >>> >> >> >> >> --publicurl http://ceph-osd3:8080/swift/v1 --adminurl >>> >> >> >> >> http://ceph-osd3:8080/swift/v1 --internal >>> >> >> >> http://ceph-osd3:8080/swift/v1 >>> >> >> >> >> --region RegionOne swift >>> >> >> >> >> usage: openstack endpoint create [-h] [-f >>> >> >> {json,shell,table,value,yaml}] >>> >> >> >> >> [-c COLUMN] [--noindent] >>> [--prefix >>> >> >> >> PREFIX] >>> >> >> >> >> [--max-width ] >>> >> [--fit-width] >>> >> >> >> >> [--print-empty] [--region >>> >> >> ] >>> >> >> >> >> [--enable | --disable] >>> >> >> >> >> >>> >> >> >> >> openstack endpoint create: error: argument : >>> invalid >>> >> >> choice: >>> >> >> >> ' >>> >> >> >> >> http://ceph-osd3:8080/swift/v1' (choose from 'admin', >>> 'public', >>> >> >> >> 'internal') >>> >> >> >> >> >>> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ >>> >> >> >> >> >>> >> >> >> >> below are my /etc/ceph/ceph.conf file : >>> >> >> >> >> >>> >> >> >> >> [client.rgw.ceph-osd3] >>> >> >> >> >> rgw_dns_name = ceph-osd3 >>> >> >> >> >> host = ceph-osd3 >>> >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3/keyring >>> >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.log >>> >> >> >> >> rgw frontends = civetweb port=10.10.29.110:8080 >>> num_threads=100 >>> >> >> >> >> rgw_keystone_url=http://10.10.29.150:35357 >>> >> >> >> >> rgw_keystone_admin_user=admin >>> >> >> >> >> rgw_keystone_admin_password=admin >>> >> >> >> >> rgw_keystone_admin_tenant=admin >>> >> >> >> >> rgw_keystone_accepted_role=admin Member swiftoperator >>> >> >> >> >> rgw_keystone_token_cache_size=200 >>> >> >> >> >> rgw_keystone_revocation_interval=300 >>> >> >> >> >> >>> >> >> >> >> [client.rgw.ceph-osd3.rgw0] >>> >> >> >> >> host = ceph-osd3 >>> >> >> >> >> keyring = >>> /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring >>> >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log >>> >> >> >> >> rgw frontends = beast endpoint=10.10.29.110:8080 >>> >> >> >> >> rgw thread pool size = 512 >>> >> >> >> >> >>> >> >> >> >> please note that my rgw_dns_name = ceph_osd3 with >>> 10.10.29.110 as >>> >> IP >>> >> >> >> >> >>> >> >> >> >> and 10.10.29.150 all-in-one IP >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> Please crosscheck where I might make mistake and try to >>> correct. >>> >> >> >> >> >>> >> >> >> >> Best regards >>> >> >> >> >> >>> >> >> >> >> Michel >>> >> >> >> >> >>> >> >> >> >> On Mon, Aug 30, 2021 at 11:25 AM Etienne Menguy < >>> >> >> >> etienne.menguy at croit.io> >>> >> >> >> >> wrote: >>> >> >> >> >> >>> >> >> >> >>> Hi, >>> >> >> >> >>> >>> >> >> >> >>> There are some information on Ceph documentation >>> >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/ < >>> >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/> . >>> >> >> >> >>> - Use keystone as auth for RGW >>> >> >> >> >>> - Create service and register your RGW as swift >>> >> >> >> >>> >>> >> >> >> >>> Étienne >>> >> >> >> >>> >>> >> >> >> >>>> On 27 Aug 2021, at 15:47, Michel Niyoyita < >>> micou12 at gmail.com> >>> >> >> wrote: >>> >> >> >> >>>> >>> >> >> >> >>>> Hello , >>> >> >> >> >>>> >>> >> >> >> >>>> I have configured RGW in my ceph cluster deployed using ceph >>> >> >> ansible >>> >> >> >> and >>> >> >> >> >>>> create sub user to access the created containers and would >>> like >>> >> to >>> >> >> >> >>> replace >>> >> >> >> >>>> swift by RGW in the openstack side. Anyone can help on >>> >> >> configuration >>> >> >> >> to >>> >> >> >> >>> be >>> >> >> >> >>>> done in the OpenStack side in order to integrate those >>> >> services. I >>> >> >> >> have >>> >> >> >> >>>> deployed OpenStack wallaby using Kolla-ansible on ubuntu >>> 20.04. >>> >> and >>> >> >> >> ceph >>> >> >> >> >>>> pacific 16.2.5 was deployed using ansible on ubuntu 20.04 >>> >> >> >> >>>> >>> >> >> >> >>>> Kindly help for the configuration or documentation. >>> >> >> >> >>>> >>> >> >> >> >>>> Best Regards >>> >> >> >> >>>> >>> >> >> >> >>>> Michel >>> >> >> >> >>>> _______________________________________________ >>> >> >> >> >>>> ceph-users mailing list -- ceph-users at ceph.io >>> >> >> >> >>>> To unsubscribe send an email to ceph-users-leave at ceph.io >>> >> >> >> >>> >>> >> >> >> >>> _______________________________________________ >>> >> >> >> >>> ceph-users mailing list -- ceph-users at ceph.io >>> >> >> >> >>> To unsubscribe send an email to ceph-users-leave at ceph.io >>> >> >> >> >>> >>> >> >> >> >> _______________________________________________ >>> >> >> >> >> ceph-users mailing list -- ceph-users at ceph.io >>> >> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> _______________________________________________ >>> >> >> >> ceph-users mailing list -- ceph-users at ceph.io >>> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >>> >> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >>> >> >>> >> >>> >> >>> >>> >>> >>> From balazs.gibizer at est.tech Mon Sep 6 14:52:09 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Mon, 06 Sep 2021 16:52:09 +0200 Subject: [placement][requirements] RFE requested for osc-placement lib Message-ID: Dear Release Team, The recently released osc-placement 3.1.0 has a serious bug[1] causing placement related openstack CLI commands to fail. The bugfix has been merged[2]. And a new bugfix release is proposed for osc-placement[3]. Please consider accepting this freeze exceptionm Thanks! gibi [1] https://bugs.launchpad.net/placement-osc-plugin/+bug/1942740 [2] https://review.opendev.org/c/openstack/osc-placement/+/807556 [3] https://review.opendev.org/c/openstack/releases/+/807591 From tkajinam at redhat.com Mon Sep 6 14:53:06 2021 From: tkajinam at redhat.com (Takashi Kajinami) Date: Mon, 6 Sep 2021 23:53:06 +0900 Subject: [puppet] Propose retiring puppet-freezer Message-ID: Hello, Today I looked into the current code of puppet-freezer and noticed this module is still very incomplete. Most of its implementations are inherited from the cookiecutter template, and the module doesn't support fundamental resources like packages, services and so on. Because we haven't seen any interest in fixing that feature gap for a long time, I'll propose retiring the module during this cycle. Please let me know if you have any concerns. If I don't hear any objections I'll submit a series of patches to retire the module. Thank you, Takashi Kajinami -------------- next part -------------- An HTML attachment was scrubbed... URL: From gthiemonge at redhat.com Mon Sep 6 15:14:08 2021 From: gthiemonge at redhat.com (Gregory Thiemonge) Date: Mon, 6 Sep 2021 17:14:08 +0200 Subject: Octavia Issues In-Reply-To: References: Message-ID: Hi Tony, I think the log line is truncated, do you have more information in the logs? If it is related to the amphora image, you also need to ensure that the owner of the image in glance is the same project as defined in the [controller_worker]/amp_image_owner_id config option in octavia.conf (it is the id of the admin project in devstack for instance) Greg On Mon, Sep 6, 2021 at 4:31 PM Karera Tony wrote: > Dear Team, > > I have openstack wallaby running on Ubuntu deployed using kolla-ansible > but whenever I create a load balancer I get the error below in the > octavia-worker.log > > octavia.common.exceptions.ComputeBuildException: Failed to build compute > instance due to: Failed to retrieve > 2021-09-06 14:14:49.392 20 ERROR oslo_messaging.rpc.server > > But I have an image with the correct tag as shown below. > > (kolla-openstack) stack at deployment:~$ openstack image show amphora1 | > grep tag > | tags | amphora > > Kindly advise if I am missing anything. > > Regards > > Tony Karera > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mthode at mthode.org Mon Sep 6 15:18:51 2021 From: mthode at mthode.org (Matthew Thode) Date: Mon, 6 Sep 2021 10:18:51 -0500 Subject: [placement][requirements] RFE requested for osc-placement lib In-Reply-To: References: Message-ID: <20210906151851.ky3clilvzcggk4zt@mthode.org> On 21-09-06 16:52:09, Balazs Gibizer wrote: > Dear Release Team, > > The recently released osc-placement 3.1.0 has a serious bug[1] causing > placement related openstack CLI commands to fail. The bugfix has been > merged[2]. And a new bugfix release is proposed for osc-placement[3]. Please > consider accepting this freeze exceptionm > > Thanks! > gibi > > [1] https://bugs.launchpad.net/placement-osc-plugin/+bug/1942740 > [2] https://review.opendev.org/c/openstack/osc-placement/+/807556 > [3] https://review.opendev.org/c/openstack/releases/+/807591 > > > Sounds good (freeze excepted) -- Matthew Thode -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From wodel.youchi at gmail.com Mon Sep 6 15:37:16 2021 From: wodel.youchi at gmail.com (wodel youchi) Date: Mon, 6 Sep 2021 16:37:16 +0100 Subject: Ceph dashboard HCI deployment Message-ID: Hi, I deployed Openstack Train using TripleO, I added ceph dashboard template to my deployment. But it is not reachable via the provisioning network, it is reachable via the storage network. Is this the default behavior? Did I miss something? I modified the haproxy configuration to point the dashboard to the ctplane vip address and it worked. How do I make this work from the deployment? Could an update of the overcloud, reset the configuration of the haproxy to the default? Thanks in advance. Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdemaced at redhat.com Mon Sep 6 15:52:16 2021 From: mdemaced at redhat.com (Maysa De Macedo Souza) Date: Mon, 6 Sep 2021 17:52:16 +0200 Subject: Changing Magnum Heat template to create node network with vxlan instead of vlan In-Reply-To: References: Message-ID: Hello, Is it a development environment using devstack? You can check the status of the load-balancer API and the Kuryr container/service logs. You can also reach us at #openstack-kuryr channel. Cheers, Maysa. On Mon, Sep 6, 2021 at 1:21 PM Karera Tony wrote: > Hello Maysa, > > There is no process running on port 8080 and yes I have configured kuryr > Regards > > Tony Karera > > > > > On Mon, Sep 6, 2021 at 10:17 AM Maysa De Macedo Souza > wrote: > >> Hello, >> >> Just some extra thought, did you check if there is some process running >> on port 8080? It's possible that the API failed to start, is Kuryr being >> used? >> >> Thanks, >> Maysa Macedo. >> >> On Tue, Aug 31, 2021 at 1:57 PM Karera Tony wrote: >> >>> Hello Guys, >>> >>> I was able to configure the cluster internal network with Vlans. The >>> communication was ok but I still get the same error >>> >>> + sleep 5 >>> ++ kubectl get --raw=/healthz >>> The connection to the server localhost:8080 was refused - did you >>> specify the right host or port? >>> + '[' ok = '' ']' >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Sat, Aug 28, 2021 at 4:43 AM Mohammed Naser >>> wrote: >>> >>>> AFAIK, Magnum does not control what encapsulation type is being used. >>>> >>>> Is there a chance you have “vlan” inside tenant_network_types ? >>>> >>>> On Tue, Aug 24, 2021 at 12:05 AM Karera Tony >>>> wrote: >>>> >>>>> Hello Team, >>>>> >>>>> I was able to deploy Openstack with Magnum and Zun enabled. >>>>> >>>>> Unfortunately, When I create a cluster, Heat configure the >>>>> fix/_network for the nodes with Vlan and the nodes fail to get IPs. >>>>> >>>>> I was wondering if it is possible to trick the heat templates to >>>>> create vxlan network instead of vlan. >>>>> >>>>> Regards >>>>> >>>>> Tony Karera >>>>> >>>>> >>>>> -- >>>> Mohammed Naser >>>> VEXXHOST, Inc. >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssbarnea at redhat.com Mon Sep 6 16:32:50 2021 From: ssbarnea at redhat.com (Sorin Sbarnea) Date: Mon, 6 Sep 2021 09:32:50 -0700 Subject: goodbye openstack, welcome ansible Message-ID: Today was my last day as a part of the OpenStack team at Red Hat. After almost 5 years, I've decided to use the opportunity to join the newly created Ansible DevTools team, a team that will improve and maintain development tools such as ansible-lint, molecule and the more recent Ansible extension for VS Code. If you are interested in finding out more, make sure to attend the next Ansible Contributor Summit which is happening at the end of this month. To those who will be missing me, don't worry I will not be far away. Cheers! Sorin Sbarnea -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlandy at redhat.com Mon Sep 6 16:47:22 2021 From: rlandy at redhat.com (Ronelle Landy) Date: Mon, 6 Sep 2021 12:47:22 -0400 Subject: goodbye openstack, welcome ansible In-Reply-To: References: Message-ID: Sorin, thank you for all you contributed. We will miss you - especially on the TripleO CI team. Best of luck in our new role! On Mon, Sep 6, 2021 at 12:37 PM Sorin Sbarnea wrote: > Today was my last day as a part of the OpenStack team at Red Hat. After > almost 5 years, I've decided to use the opportunity to join the newly > created Ansible DevTools team, a team that will improve and maintain > development tools such as ansible-lint, molecule and the more recent > Ansible extension for VS Code. > > If you are interested in finding out more, make sure to attend the next Ansible > Contributor Summit > which is > happening at the end of this month. > > To those who will be missing me, don't worry I will not be far away. > > Cheers! > Sorin Sbarnea > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fpantano at redhat.com Mon Sep 6 16:49:18 2021 From: fpantano at redhat.com (Francesco Pantano) Date: Mon, 6 Sep 2021 18:49:18 +0200 Subject: Ceph dashboard HCI deployment In-Reply-To: References: Message-ID: Hello, Thanks for looking into this. I think the problem you found on the haproxy config for the Ceph Dashboard is related to [1], where is given to the operator the flexibility of moving the ceph dashboard haproxy frontend vip on a network != ctlplane when no CephStorageDashboard composable network is defined [2]. On Mon, Sep 6, 2021 at 5:42 PM wodel youchi wrote: > Hi, > > I deployed Openstack Train using TripleO, I added ceph dashboard template > to my deployment. But it is not reachable via the provisioning network, it > is reachable via the storage network. Is this the default behavior? Did I > miss something? > > I modified the haproxy configuration to point the dashboard to the ctplane > vip address and it worked. > > How do I make this work from the deployment? > You can do that by overriding the CephDashboardNetwork parameter, which means you can add to your existing deployed *CephDashboardNetwork:ctlplane * By doing this, during the next stack update the haproxy config that points to the ctlplane network should be maintained. > > Could an update of the overcloud, reset the configuration of the haproxy to > the default? > Exactly, a stack update will result in applying the default values defined in your tht. For this reason, you should pass CephDashboardNetwork:ctlplane to your overcloud deploy. > > Thanks in advance. Regards. > I started a patch series [3] to make sure the Ceph Dashboard network uses the right networks, maintaining the ability to change it overriding the parameter described above. Thank you!! [1] https://review.opendev.org/c/openstack/tripleo-ansible/+/801757 [2] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/ceph_config.html#adding-ceph-dashboard-to-a-overcloud-deployment [3] https://review.opendev.org/q/topic:%22ceph_dashboard_network%22+(status:open%20OR%20status:merged) -- Francesco Pantano GPG KEY: F41BD75C -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Mon Sep 6 17:35:04 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Mon, 06 Sep 2021 12:35:04 -0500 Subject: [all][tc] Technical Committee next weekly meeting on Sept 9th at 1500 UTC Message-ID: <17bbc2d846e.11adf55c390492.8839891819324332226@ghanshyammann.com> Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 9th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 8th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann From kazumasa.nomura.rx at hitachi.com Mon Sep 6 18:48:09 2021 From: kazumasa.nomura.rx at hitachi.com (=?utf-8?B?6YeO5p2R5ZKM5q2jIC8gTk9NVVJB77yMS0FaVU1BU0E=?=) Date: Mon, 6 Sep 2021 18:48:09 +0000 Subject: [cinder] xena feature freeze In-Reply-To: References: Message-ID: Hi Team, I'd like to request FFE to the following change: [1] Hitachi: Add generic volume groups https://review.opendev.org/c/openstack/cinder/+/782910 - This patch adds consistency group capability as generic volume groups for the Hitachi driver. This change and blueprint were submitted in time and it has been good progress in the reviews until this point. This patch is isolated to the Hitachi driver module, and have no impact on the Cinder database, REST/RPC and APIs. Kind Regards, Kazumasa Nomura -----Original Message----- From: Brian Rosmaita Sent: Saturday, September 4, 2021 5:55 AM To: openstack-discuss at lists.openstack.org Subject: [!][cinder] xena feature freeze The Xena feature freeze for cinder is now in effect. Please do not approve patches proposing features for master until after the stable/xena branch has been cut the week of 13 September. The following blueprints are granted feature freeze exceptions: - https://secure-web.cisco.com/1esUHJyUlDOFXd10REcKVNw8NBLq3WM7zB53bQ2dG6d_4AKYrcnDvITVjZOXPlqoguipoly92XJj_W3xyJJXJkN9lP27RobrMUbliwyUrQYv2w7pRlBj-ikdJiRK5Tgh4tKsxdNufvDl39xyiVWpPnc4cD4GnXKMObQBa5UHG__8gMhk4v3biFveL4uaBjziC5fw9mzI1hOZyvFBhR36p5TVR4FnPdZ9DQ0jOoJ_YGcfdTZNzaadIlYA9HaMPhIo0wYdpjxBEBI9DYd_4inMliFshH-dLz4OWtvsu9bH21lOoU994XxUpyizz45wdPY_W/https%3A%2F%2Fblueprints.launchpad.net%2Fcinder%2F%2Bspec%2Fs-rbac-ready-x - https://secure-web.cisco.com/1wxRMdxHHb-gemG2KizQtfPjVmtpBfsSPKKdJXHmCPFs6L9o7JMH8uHQahyjUFbqwMGZz_SG5lNgP-DwS5dojrm-h3OhiOJFW1sf2QDBsshmMAhfM5srV3EYjsrtR49oFwMWw-fpQiM9T0DNSebWXVa_RvoS0BFix9oEasQxVe445BuRf23R3Z5WJ4EnyS0LB-kS00pI4Wng4mIg9216x_2hL1Te3d2GvmYCOdoYKVKxuDi019xG8lR5OTZGRqfRzj7yBXwWYSJihXxn1YRIUegKHGfi5mhG9_Jy-p-qVJrd8PWrrWrr3oLuT8Gy6-5u5/https%3A%2F%2Fblueprints.launchpad.net%2Fcinder%2F%2Bspec%2Fremove-sqlalchemy-migrate Any other feature patches (including driver features) that have not yet merged and are associated with an approved xena blueprint may apply for a Feature Freeze Exception by replying to this email before Tuesday, 7 September at 20:00 UTC. Note that application for an FFE is not a guarantee that an FFE will be granted. Grant of an FFE is not a guarantee that the code will be merged for Xena. It is expected that patches granted an FFE will be merged by 21:00 UTC on Monday 13 September. Bugfixes do not require an FFE. To be included in Xena, they must be merged before the first Release Candidate is cut on Thursday 16 September. After that point, only release-critical bugfixes will be allowed into the Xena release. Ping me in IRC if you have any questions. cheers, brian From kazumasa.nomura.rx at hitachi.com Mon Sep 6 18:53:33 2021 From: kazumasa.nomura.rx at hitachi.com (=?utf-8?B?6YeO5p2R5ZKM5q2jIC8gTk9NVVJB77yMS0FaVU1BU0E=?=) Date: Mon, 6 Sep 2021 18:53:33 +0000 Subject: [cinder] xena feature freeze In-Reply-To: References: Message-ID: Hi Team, I'd like to request FFE to the following change: [1] Hitachi: Add generic volume groups https://review.opendev.org/c/openstack/cinder/+/782910 - This patch adds consistency group capability as generic volume groups for the Hitachi driver. This change and blueprint were submitted in time and it has been good progress in the reviews until this point. This patch is isolated to the Hitachi driver module, and have no impact on the Cinder database, REST/RPC and APIs. Kind Regards, Kazumasa Nomura -----Original Message----- From: Brian Rosmaita Sent: Saturday, September 4, 2021 5:55 AM To: openstack-discuss at lists.openstack.org Subject: [!][cinder] xena feature freeze The Xena feature freeze for cinder is now in effect. Please do not approve patches proposing features for master until after the stable/xena branch has been cut the week of 13 September. The following blueprints are granted feature freeze exceptions: - https://secure-web.cisco.com/1esUHJyUlDOFXd10REcKVNw8NBLq3WM7zB53bQ2dG6d_4AKYrcnDvITVjZOXPlqoguipoly92XJj_W3xyJJXJkN9lP27RobrMUbliwyUrQYv2w7pRlBj-ikdJiRK5Tgh4tKsxdNufvDl39xyiVWpPnc4cD4GnXKMObQBa5UHG__8gMhk4v3biFveL4uaBjziC5fw9mzI1hOZyvFBhR36p5TVR4FnPdZ9DQ0jOoJ_YGcfdTZNzaadIlYA9HaMPhIo0wYdpjxBEBI9DYd_4inMliFshH-dLz4OWtvsu9bH21lOoU994XxUpyizz45wdPY_W/https%3A%2F%2Fblueprints.launchpad.net%2Fcinder%2F%2Bspec%2Fs-rbac-ready-x - https://secure-web.cisco.com/1wxRMdxHHb-gemG2KizQtfPjVmtpBfsSPKKdJXHmCPFs6L9o7JMH8uHQahyjUFbqwMGZz_SG5lNgP-DwS5dojrm-h3OhiOJFW1sf2QDBsshmMAhfM5srV3EYjsrtR49oFwMWw-fpQiM9T0DNSebWXVa_RvoS0BFix9oEasQxVe445BuRf23R3Z5WJ4EnyS0LB-kS00pI4Wng4mIg9216x_2hL1Te3d2GvmYCOdoYKVKxuDi019xG8lR5OTZGRqfRzj7yBXwWYSJihXxn1YRIUegKHGfi5mhG9_Jy-p-qVJrd8PWrrWrr3oLuT8Gy6-5u5/https%3A%2F%2Fblueprints.launchpad.net%2Fcinder%2F%2Bspec%2Fremove-sqlalchemy-migrate Any other feature patches (including driver features) that have not yet merged and are associated with an approved xena blueprint may apply for a Feature Freeze Exception by replying to this email before Tuesday, 7 September at 20:00 UTC. Note that application for an FFE is not a guarantee that an FFE will be granted. Grant of an FFE is not a guarantee that the code will be merged for Xena. It is expected that patches granted an FFE will be merged by 21:00 UTC on Monday 13 September. Bugfixes do not require an FFE. To be included in Xena, they must be merged before the first Release Candidate is cut on Thursday 16 September. After that point, only release-critical bugfixes will be allowed into the Xena release. Ping me in IRC if you have any questions. cheers, brian From zigo at debian.org Mon Sep 6 19:29:01 2021 From: zigo at debian.org (Thomas Goirand) Date: Mon, 6 Sep 2021 21:29:01 +0200 Subject: [horizon] Support for Django 3.x In-Reply-To: References: Message-ID: <1977ce4f-bf9b-9a03-672d-b85531ea1187@debian.org> On 9/6/21 1:00 PM, vishal manchanda wrote: > Hi Thomas, > > Horizon team working on adding django3.x support in horizon. > A series of patches to add django3.x support is already up [1] and We > recently merged a non-voting job to test > django3.x which is failing right now[2]. The main issue we are facing in > migration to django3.x is horizon > using django-pyscss which isn't compatible with django3.x although we > have some workaround to fix it[3] > but it is not the right way. So I tried to contact the django-pyscss > maintainer, but they didn't respond. > Then we plan to use an alternative of django-pyscss i.e. > django-sass-processor. > > Update on django-sass-processor: Me and radomir from horizon team worked > on adding django-sass-processor > support in horizon[4]. As far our testing it works fine with horizon but > horizon default theme doesn't work with it  > right now and fails with error[5]. I am working on this issue, any help > is really appreciated. Hi Vishal, Thanks a lot for giving a status update, I'm very happy to see there's work toward Django 3.x support. Thanks to you and all the team. FYI, I unfortunately don't have the skills to help. However, I can work for having new dependencies available in Debian, and so also in Ubuntu. Do you think I should package django-sass-processor right now? Or is it too early to tell it's going to be the adopted solution? Cheers, Thomas Goirand (zigo) From berndbausch at gmail.com Tue Sep 7 02:59:01 2021 From: berndbausch at gmail.com (Bernd Bausch) Date: Tue, 7 Sep 2021 11:59:01 +0900 Subject: Does ceph and openstack communicate over the openstack-API network? In-Reply-To: References: Message-ID: Neither OpenStack nor Ceph prescribe which networks are used for which purpose. They communicate over the network that you set up and configure. On 2021/09/06 6:48 PM, wodel youchi wrote: > Hi, > > Does ceph and openstack communicate over the openstack-API network, > for nova, cinder, ...etc to be able to use ceph pools? > > Regards. From marios at redhat.com Tue Sep 7 05:59:23 2021 From: marios at redhat.com (Marios Andreou) Date: Tue, 7 Sep 2021 08:59:23 +0300 Subject: goodbye openstack, welcome ansible In-Reply-To: References: Message-ID: good luck on the new role! also i will ping you in a bit too discuss the 'requests' dependency in get-hash for dstream use case ;) :D regards On Monday, September 6, 2021, Sorin Sbarnea wrote: > Today was my last day as a part of the OpenStack team at Red Hat. After > almost 5 years, I've decided to use the opportunity to join the newly > created Ansible DevTools team, a team that will improve and maintain > development tools such as ansible-lint, molecule and the more recent > Ansible extension for VS Code. > > If you are interested in finding out more, make sure to attend the next Ansible > Contributor Summit > which is > happening at the end of this month. > > To those who will be missing me, don't worry I will not be far away. > > Cheers! > Sorin Sbarnea > -- _sent from my mobile - sorry for spacing spelling etc_ -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrei.perepiolkin at open-e.com Tue Sep 7 07:03:25 2021 From: andrei.perepiolkin at open-e.com (Andrei Perapiolkin) Date: Tue, 7 Sep 2021 03:03:25 -0400 Subject: [cinder] xena feature freeze In-Reply-To: References: Message-ID: Hi Team, I'd like to request FFE to the following change: [1] JovianDSS: add multiattach and 16K block support https://review.opendev.org/c/openstack/cinder/+/806726 - Patch adds multiattach support and 16K block size support for Open-E JovianDSS driver. [2] JovianDSS: fix code style and naming https://review.opendev.org/c/openstack/cinder/+/806559 - Patch adds minor code style and naming changes Both changes were parts of a single patch: [3] https://review.opendev.org/c/openstack/cinder/+/806559 That was split in parts according to recommendations. Patches are merged by now. They are isolated to the Open-E JovianDSS driver module, and have no impact on the Cinder database, REST/RPC and APIs. Best regards, Andrei On 9/3/21 4:55 PM, Brian Rosmaita wrote: > The Xena feature freeze for cinder is now in effect.  Please do not > approve patches proposing features for master until after the > stable/xena branch has been cut the week of 13 September. > > The following blueprints are granted feature freeze exceptions: > - https://blueprints.launchpad.net/cinder/+spec/s-rbac-ready-x > - https://blueprints.launchpad.net/cinder/+spec/remove-sqlalchemy-migrate > > Any other feature patches (including driver features) that have not > yet merged and are associated with an approved xena blueprint may > apply for a Feature Freeze Exception by replying to this email before > Tuesday, 7 September at 20:00 UTC.  Note that application for an FFE > is not a guarantee that an FFE will be granted.  Grant of an FFE is > not a guarantee that the code will be merged for Xena. > > It is expected that patches granted an FFE will be merged by 21:00 UTC > on Monday 13 September. > > Bugfixes do not require an FFE.  To be included in Xena, they must be > merged before the first Release Candidate is cut on Thursday 16 > September.  After that point, only release-critical bugfixes will be > allowed into the Xena release. > > Ping me in IRC if you have any questions. > > cheers, > brian > From v at prokofev.me Tue Sep 7 07:40:34 2021 From: v at prokofev.me (Vladimir Prokofev) Date: Tue, 7 Sep 2021 10:40:34 +0300 Subject: [tempest] change output format Message-ID: Hello. Is there a way to change the output format of tempest tests? Preferably to JSON. So instead of test something... ok it produced the same result but in JSON format. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wodel.youchi at gmail.com Tue Sep 7 08:59:43 2021 From: wodel.youchi at gmail.com (wodel youchi) Date: Tue, 7 Sep 2021 09:59:43 +0100 Subject: Backup TripleO instances Message-ID: Hi, I found out that Freezer is not supported by TripleO. - Can Freezer be integrated with TripleO? - Is there any other free software that can be used to backup instances that can easily be integrated with TripleO. Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gchamoul at redhat.com Tue Sep 7 09:06:08 2021 From: gchamoul at redhat.com (=?utf-8?B?R2HDq2w=?= Chamoulaud) Date: Tue, 7 Sep 2021 11:06:08 +0200 Subject: goodbye openstack, welcome ansible In-Reply-To: References: Message-ID: <20210907090608.uvpk5m3lmyrpxxgv@gchamoul-mac> Very good luck on your new position, Sorin! You will definitely be missed. On 06/Sep/2021 09:32, Sorin Sbarnea wrote: > Today was my last day as a part of the OpenStack team at Red Hat. After almost > 5 years, I've decided to use the opportunity to join the newly created Ansible > DevTools team, a team that will improve and maintain development tools such as > ansible-lint, molecule and the more recent Ansible extension for VS Code. > > If you are interested in finding out more, make sure to attend the next Ansible > Contributor Summit which is happening at the end of this month. > > To those who will be missing me, don't worry I will not be far away. > > Cheers! > Sorin Sbarnea Best Regards, Gaël -- Gaël Chamoulaud - (He/Him/His) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From zigo at debian.org Tue Sep 7 09:09:45 2021 From: zigo at debian.org (Thomas Goirand) Date: Tue, 7 Sep 2021 11:09:45 +0200 Subject: [all] Dangerous trend toward too many release-independent components Message-ID: Hi, As I'm packaging Xena in Debian, I've seen more and more components switching to release-independent. This poses a number of problems which I'm really unsure how to address. Let me explain. First of all, once an OpenStack release reaches a distribution stable release, there will be no update of it (ie: no new upstream release), unless there's a security problems, or important bugfixes that cannot be backported individually. At least, that's how Debian works, and why we call our suite "stable" (ie: it doesn't change). However, if a component is declared release-independent, the OpenStack community expects it to be updated for all downward releases. But that's not what's going on in reality. Also, if there's a security fix, we're supposed to get this through an update of a new version, which may include changes not suitable for a distribution update. The danger is that the update may be refused by the Debian Stable release team, for very good reasons (which I agree on). I'm also questioning myself: since we have pinned global requirements, will these new updates be included in the tests of older OpenStack releases? Is this really the case right now? So, while I do understand why it's nicer to set release-independent tags on some components, it is my opinion that this doesn't help stability and poses numerous problems to downstream distributions. Could we limit the number of such release-independent released components to the strict minimum low-profile (ie: with the least amount of code) components, rather than constantly increasing the list? Your thoughts? Cheers, Thomas Goirand (zigo) From moreira.belmiro.email.lists at gmail.com Tue Sep 7 09:10:07 2021 From: moreira.belmiro.email.lists at gmail.com (Belmiro Moreira) Date: Tue, 7 Sep 2021 11:10:07 +0200 Subject: [all][elections][ptl] PTL Voting Last Day - Cyborg Message-ID: Hello Cyborg contributors, Just a quick reminder that elections are closing soon, if you haven't already you should use your right to vote and pick your favourite candidate! You have until Sep 07, 2021 23:45 UTC. Belmiro, On behalf of all the Election Officials -------------- next part -------------- An HTML attachment was scrubbed... URL: From balazs.gibizer at est.tech Tue Sep 7 09:19:17 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Tue, 07 Sep 2021 11:19:17 +0200 Subject: [nova][placement] nova-next job is broken In-Reply-To: <1M80ZQ.S70WTG0D8TSL1@est.tech> References: <1M80ZQ.S70WTG0D8TSL1@est.tech> Message-ID: <5K42ZQ.IAJ6MORGWXO32@est.tech> Hi, Yesterday we released a new osc-placement lib[1] and it seems it is resolved the problem with the nova-next job[2]. You can try to recheck failed builds. Cheers, gibi [1] https://review.opendev.org/c/openstack/requirements/+/807605 [2] https://zuul.opendev.org/t/openstack/builds?job_name=nova-next&branch=master On Mon, Sep 6, 2021 at 10:51, Balazs Gibizer wrote: > Hi, > > Since Friday the nova-next job fails with POST_FAILURE[1] due to the > new osc-placement release. Please hold you rechecks. > > [1] https://bugs.launchpad.net/nova/+bug/1942740 > > Cheers, > gibi > > > > From chkumar at redhat.com Tue Sep 7 09:20:36 2021 From: chkumar at redhat.com (Chandan Kumar) Date: Tue, 7 Sep 2021 14:50:36 +0530 Subject: goodbye openstack, welcome ansible In-Reply-To: References: Message-ID: On Mon, Sep 6, 2021 at 10:08 PM Sorin Sbarnea wrote: > > Today was my last day as a part of the OpenStack team at Red Hat. After almost 5 years, I've decided to use the opportunity to join the newly created Ansible DevTools team, a team that will improve and maintain development tools such as ansible-lint, molecule and the more recent Ansible extension for VS Code. > > If you are interested in finding out more, make sure to attend the next Ansible Contributor Summit which is happening at the end of this month. > > To those who will be missing me, don't worry I will not be far away. Thank you for all the contributions to OpenStack. We will miss you. All the best for a new adventure. :-) Thanks, Chandan Kumar From thierry at openstack.org Tue Sep 7 09:27:29 2021 From: thierry at openstack.org (Thierry Carrez) Date: Tue, 7 Sep 2021 11:27:29 +0200 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: References: Message-ID: <87682f00-8d19-36c2-5911-622fc3d0c988@openstack.org> Thomas Goirand wrote: > As I'm packaging Xena in Debian, I've seen more and more components > switching to release-independent. This poses a number of problems which > I'm really unsure how to address. Let me explain. > > First of all, once an OpenStack release reaches a distribution stable > release, there will be no update of it (ie: no new upstream release), > unless there's a security problems, or important bugfixes that cannot be > backported individually. At least, that's how Debian works, and why we > call our suite "stable" (ie: it doesn't change). > > However, if a component is declared release-independent, the OpenStack > community expects it to be updated for all downward releases. But that's > not what's going on in reality. Also, if there's a security fix, we're > supposed to get this through an update of a new version, which may > include changes not suitable for a distribution update. The danger is > that the update may be refused by the Debian Stable release team, for > very good reasons (which I agree on). > > I'm also questioning myself: since we have pinned global requirements, > will these new updates be included in the tests of older OpenStack > releases? Is this really the case right now? > > So, while I do understand why it's nicer to set release-independent tags > on some components, it is my opinion that this doesn't help stability > and poses numerous problems to downstream distributions. > > Could we limit the number of such release-independent released > components to the strict minimum low-profile (ie: with the least amount > of code) components, rather than constantly increasing the list? Hi Thomas, Those are all good points, and we will discuss them at the release team level. For context, recently the release team has encouraged a number of stale components (stable libraries etc.) to move to release-independent since they were not even seeing one release per cycle. There was no point in forcing releases every cycle that were just changing the branch name in .gitreview. Regards, -- Thierry Carrez (ttx) From jpodivin at redhat.com Tue Sep 7 11:06:47 2021 From: jpodivin at redhat.com (Jiri Podivin) Date: Tue, 7 Sep 2021 13:06:47 +0200 Subject: goodbye openstack, welcome ansible In-Reply-To: References: Message-ID: Good luck in the Ansible DevTools team Sorin! Regards, On Tue, Sep 7, 2021 at 11:23 AM Chandan Kumar wrote: > On Mon, Sep 6, 2021 at 10:08 PM Sorin Sbarnea wrote: > > > > Today was my last day as a part of the OpenStack team at Red Hat. After > almost 5 years, I've decided to use the opportunity to join the newly > created Ansible DevTools team, a team that will improve and maintain > development tools such as ansible-lint, molecule and the more recent > Ansible extension for VS Code. > > > > If you are interested in finding out more, make sure to attend the next > Ansible Contributor Summit which is happening at the end of this month. > > > > To those who will be missing me, don't worry I will not be far away. > > Thank you for all the contributions to OpenStack. > We will miss you. > All the best for a new adventure. :-) > > Thanks, > > Chandan Kumar > > > -- Jiri Podivin Associate Software Engineer, Openstack Red Hat Czech, s.r.o. Purkyňova 3080/97b 61200 Brno, Czech Republic jpodivin at redhat.com M: +420739108412 IRC: jpodivin -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Tue Sep 7 11:07:55 2021 From: smooney at redhat.com (Sean Mooney) Date: Tue, 07 Sep 2021 12:07:55 +0100 Subject: [all][tc] Minium python version for yoga and centos 9 stream. In-Reply-To: <33913002aa0ad4be63f6c36be310f4e1dd7bd008.camel@redhat.com> References: <33913002aa0ad4be63f6c36be310f4e1dd7bd008.camel@redhat.com> Message-ID: <4b261268e701d3c464f180e89a795841c06dc8d0.camel@redhat.com> resending to correct list On Tue, 2021-09-07 at 10:51 +0100, Sean Mooney wrote: > hi i was jsut looking at the yoga runtimes an notice that we have not really updated it > with regards to minium python versions. > > currently > > the runtime defintion is as follows https://github.com/openstack/governance/blob/master/reference/runtimes/yoga.rst > > It is the :doc:`policy <../../resolutions/20181024-python-update-process>` that each OpenStack release cycle will target the latest available version of Python; default Python runtimes on the > distributions listed above; and versions used in integration tests at the start of the cycle, at least until the point when all projects have migrated to a later version. > > Based on the criteria above, all Python-based projects must target and test against, at a minimum: > > Python 3.6 (default in CentOS 8 Stream) > Python 3.8 (latest available; default in Ubuntu 20.04) > > looking at to look at https://repology.org/project/python/versions what python version are shpping by default now in many distro > more and more of them are moving to python 3.9 which is currently not in the test list and wer are starting to fall quite far behind > > debian 11/12/stable are currently using 3.9.2 > centos 8 defaults to 3.6.8 but ships 3.8.6 and 3.9.2 as appstreams > centos 9 stream will default to python3-3.9.6-6.el9.x86_64.rpm https://composes.stream.centos.org/test/latest-CentOS-Stream/compose/BaseOS/x86_64/os/Packages/ > ubuntu 20.04 default to 3.8.2 but also ships 3.9.5 in universe > ubuntu 21.04 default to 3.9.5 so we can expect that when yoga ships on ubuntu 22.04 it will be based on at least python 3.9 and proably 3.10 > fedora well they basically ship everythin but i belive 3.9 is not the default based on complaints some of my fellow contiutors had with > runnign unit and functional tests. > > openSUSE Leap 15.3 main/oss python39 3.9.4 ship 3.9 but default to 3.6 > they aslo nolonger have an openstack product but it would still be nice to keep them in mind. > > > over all i think we shoudl consider raising or min tested/supported python version to python 3.8 for yoga and consider updating from centos 8 to centos 9. > i also think it worth including python 3.9 in the required testing versions for python projects. > > several distros will be releaseing openstack xena/wallaby with python 3.9 and also yoga likely with 3.10 in some cases. > if we only offically test with 3.6 and 3.8 we could have some hard probalems to solve going forward. the optional testing with unit and functional tests > that some project have for python 3.9 certenly helps but actully having an integrated tempest gate job that runs on 3.9 i think is going to be increasingly important. > > readring centos 9 stream, it was orginally planned of release at the end of q2. now that date has obviosly passed but i know that they are still hopeful to have > a release ready in the next month or so. cloud images are avaibel at https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/images/ although the repos are still only > partly complete https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/images/ and there are currently no active mirros so its still officaly > not released https://www.centos.org/centos-stream/ and i don not have any ETA beyond soonTM. as such im aware we cant offically declare centos 9 stream as part of the tested > runtimes at this point but it might be at least worht listing it as an experimtal runtime as i suspec there will be a desire to swtich to it during the yoga cycle for some proejct. > ooo/rdo come to mind as they discussed this for xena but obviously that was not possibel since it was not released in time. > > > regards > sean. > > ps removing python 3.6 and replacing it with 3.9 instead of just adding 3.9 as an addtional runtime is more to manage our testing > there isnt nessarly any python feature that i explcitly would like to be able to use that are added in 3.8 that would be enabled > by making that the new min python verion that is motivating this suggestion. there are some nice thingin 3.9/3.10 such as pattern > mataching that are commign but it will be some time (2-4 years)before we can actully use those as our min version so i was just being practial > and suggestion we raise to 3.8 and add 3.9 as required with 3.10 as optional/experimal for those that want it. > From rosmaita.fossdev at gmail.com Tue Sep 7 11:37:50 2021 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Tue, 7 Sep 2021 07:37:50 -0400 Subject: [cinder] xena feature freeze In-Reply-To: References: Message-ID: <07fd4237-481c-4300-d7aa-0d2dfc95bd2a@gmail.com> These patches have both had +2s at some point in the reviewing process, and it looks like the only changes have been merge conflict resolution, so these are granted FFEs subject to the caveats in original email. On 9/3/21 5:09 PM, Fernando Ferraz wrote: > Hi Team, > > I'd like to request FFE to the following changes: > > [1] Netapp ONTAP: Add support to revert to snapshot > https://review.opendev.org/c/openstack/cinder/+/804093 > > - Patch adds support for the revert to snapshot feature to the NetApp Cinder > drivers. > > [2] NetApp ONTAP: Add option to report storage provisioned capacity > https://review.opendev.org/c/openstack/cinder/+/798198 > > - Patch adds an option to enable a new pool capability > provisioned_capacity_gb > that reports storage provisioned volume sizes directly from the storage > system, > instead of relying on the default scheduler's behavior based on internal > states. > > Both changes and their blueprints were submitted in time and there has been > significant progress in the reviews until this point. They are isolated to > the NetApp driver module, and have no impact on the Cinder database, > REST/RPC and APIs. > > Regards, > Fernando > > > Em sex., 3 de set. de 2021 às 18:01, Brian Rosmaita > > escreveu: > > The Xena feature freeze for cinder is now in effect.  Please do not > approve patches proposing features for master until after the > stable/xena branch has been cut the week of 13 September. > > The following blueprints are granted feature freeze exceptions: > - https://blueprints.launchpad.net/cinder/+spec/s-rbac-ready-x > > - > https://blueprints.launchpad.net/cinder/+spec/remove-sqlalchemy-migrate > > > Any other feature patches (including driver features) that have not yet > merged and are associated with an approved xena blueprint may apply for > a Feature Freeze Exception by replying to this email before Tuesday, 7 > September at 20:00 UTC.  Note that application for an FFE is not a > guarantee that an FFE will be granted.  Grant of an FFE is not a > guarantee that the code will be merged for Xena. > > It is expected that patches granted an FFE will be merged by 21:00 UTC > on Monday 13 September. > > Bugfixes do not require an FFE.  To be included in Xena, they must be > merged before the first Release Candidate is cut on Thursday 16 > September.  After that point, only release-critical bugfixes will be > allowed into the Xena release. > > Ping me in IRC if you have any questions. > > cheers, > brian > From rosmaita.fossdev at gmail.com Tue Sep 7 12:17:31 2021 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Tue, 7 Sep 2021 08:17:31 -0400 Subject: [cinder] xena feature freeze In-Reply-To: References: Message-ID: <69ed5cf1-5858-96a1-871c-07ba47b36e94@gmail.com> This is a fairly large patch, but it is confined to a single driver and is unlikely to impact the main cinder code. I feel a lot better about its code coverage since you figured out the unit test discovery problem. So this patch is granted an FFE subject to the caveats in the original email. On 9/6/21 2:48 PM, 野村和正 / NOMURA，KAZUMASA wrote: > Hi Team, > > I'd like to request FFE to the following change: > > [1] Hitachi: Add generic volume groups > https://review.opendev.org/c/openstack/cinder/+/782910 > - This patch adds consistency group capability as generic volume groups > for the Hitachi driver. > > This change and blueprint were submitted in time and it has been > good progress in the reviews until this point. This patch is isolated to > the Hitachi driver module, and have no impact on the Cinder database, > REST/RPC and APIs. > > Kind Regards, > Kazumasa Nomura > > -----Original Message----- > From: Brian Rosmaita > Sent: Saturday, September 4, 2021 5:55 AM > To: openstack-discuss at lists.openstack.org > Subject: [!][cinder] xena feature freeze > > The Xena feature freeze for cinder is now in effect. Please do not approve patches proposing features for master until after the stable/xena branch has been cut the week of 13 September. > > The following blueprints are granted feature freeze exceptions: > - https://secure-web.cisco.com/1esUHJyUlDOFXd10REcKVNw8NBLq3WM7zB53bQ2dG6d_4AKYrcnDvITVjZOXPlqoguipoly92XJj_W3xyJJXJkN9lP27RobrMUbliwyUrQYv2w7pRlBj-ikdJiRK5Tgh4tKsxdNufvDl39xyiVWpPnc4cD4GnXKMObQBa5UHG__8gMhk4v3biFveL4uaBjziC5fw9mzI1hOZyvFBhR36p5TVR4FnPdZ9DQ0jOoJ_YGcfdTZNzaadIlYA9HaMPhIo0wYdpjxBEBI9DYd_4inMliFshH-dLz4OWtvsu9bH21lOoU994XxUpyizz45wdPY_W/https%3A%2F%2Fblueprints.launchpad.net%2Fcinder%2F%2Bspec%2Fs-rbac-ready-x > - https://secure-web.cisco.com/1wxRMdxHHb-gemG2KizQtfPjVmtpBfsSPKKdJXHmCPFs6L9o7JMH8uHQahyjUFbqwMGZz_SG5lNgP-DwS5dojrm-h3OhiOJFW1sf2QDBsshmMAhfM5srV3EYjsrtR49oFwMWw-fpQiM9T0DNSebWXVa_RvoS0BFix9oEasQxVe445BuRf23R3Z5WJ4EnyS0LB-kS00pI4Wng4mIg9216x_2hL1Te3d2GvmYCOdoYKVKxuDi019xG8lR5OTZGRqfRzj7yBXwWYSJihXxn1YRIUegKHGfi5mhG9_Jy-p-qVJrd8PWrrWrr3oLuT8Gy6-5u5/https%3A%2F%2Fblueprints.launchpad.net%2Fcinder%2F%2Bspec%2Fremove-sqlalchemy-migrate > > Any other feature patches (including driver features) that have not yet merged and are associated with an approved xena blueprint may apply for a Feature Freeze Exception by replying to this email before Tuesday, 7 September at 20:00 UTC. Note that application for an FFE is not a guarantee that an FFE will be granted. Grant of an FFE is not a guarantee that the code will be merged for Xena. > > It is expected that patches granted an FFE will be merged by 21:00 UTC on Monday 13 September. > > Bugfixes do not require an FFE. To be included in Xena, they must be merged before the first Release Candidate is cut on Thursday 16 September. After that point, only release-critical bugfixes will be allowed into the Xena release. > > Ping me in IRC if you have any questions. > > cheers, > brian > > From rosmaita.fossdev at gmail.com Tue Sep 7 12:46:16 2021 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Tue, 7 Sep 2021 08:46:16 -0400 Subject: [cinder] xena feature freeze In-Reply-To: References: Message-ID: <8a796afb-0a62-d964-20ee-51903e7e278e@gmail.com> Hi Andrei, [1] and [2] are granted FFEs; they've had +2s at various points and probably would have merged before the deadline except for some gate weirdness. I think you have a typo in [3], and are referring to: https://review.opendev.org/c/openstack/cinder/+/794962 Please go ahead and abandon [3] as it's no longer relevant. (As the reason, you can say "Superseded by https://review.opendev.org/c/openstack/cinder/+/806726 and https://review.opendev.org/c/openstack/cinder/+/806559".) On 9/7/21 3:03 AM, Andrei Perapiolkin wrote: > Hi Team, > > I'd like to request FFE to the following change: > > [1] JovianDSS: add multiattach and 16K block support > https://review.opendev.org/c/openstack/cinder/+/806726 > > - Patch adds multiattach support and 16K block size support > for Open-E JovianDSS driver. > > > [2] JovianDSS: fix code style and naming > https://review.opendev.org/c/openstack/cinder/+/806559 > > > - Patch adds minor code style and naming changes > > > Both changes were parts of a single patch: > [3] https://review.opendev.org/c/openstack/cinder/+/806559 > > That was split in parts according to recommendations. > Patches are merged by now. > They are isolated to the Open-E JovianDSS driver module, and have no > impact on the Cinder database, > REST/RPC and APIs. > > Best regards, > Andrei > > > On 9/3/21 4:55 PM, Brian Rosmaita wrote: >> The Xena feature freeze for cinder is now in effect.  Please do not >> approve patches proposing features for master until after the >> stable/xena branch has been cut the week of 13 September. >> >> The following blueprints are granted feature freeze exceptions: >> - https://blueprints.launchpad.net/cinder/+spec/s-rbac-ready-x >> - https://blueprints.launchpad.net/cinder/+spec/remove-sqlalchemy-migrate >> >> Any other feature patches (including driver features) that have not >> yet merged and are associated with an approved xena blueprint may >> apply for a Feature Freeze Exception by replying to this email before >> Tuesday, 7 September at 20:00 UTC.  Note that application for an FFE >> is not a guarantee that an FFE will be granted.  Grant of an FFE is >> not a guarantee that the code will be merged for Xena. >> >> It is expected that patches granted an FFE will be merged by 21:00 UTC >> on Monday 13 September. >> >> Bugfixes do not require an FFE.  To be included in Xena, they must be >> merged before the first Release Candidate is cut on Thursday 16 >> September.  After that point, only release-critical bugfixes will be >> allowed into the Xena release. >> >> Ping me in IRC if you have any questions. >> >> cheers, >> brian >> > From dtantsur at redhat.com Tue Sep 7 13:08:58 2021 From: dtantsur at redhat.com (Dmitry Tantsur) Date: Tue, 7 Sep 2021 15:08:58 +0200 Subject: [all][tc] Minium python version for yoga and centos 9 stream. In-Reply-To: <4b261268e701d3c464f180e89a795841c06dc8d0.camel@redhat.com> References: <33913002aa0ad4be63f6c36be310f4e1dd7bd008.camel@redhat.com> <4b261268e701d3c464f180e89a795841c06dc8d0.camel@redhat.com> Message-ID: On Tue, Sep 7, 2021 at 1:10 PM Sean Mooney wrote: > resending to correct list > On Tue, 2021-09-07 at 10:51 +0100, Sean Mooney wrote: > > hi i was jsut looking at the yoga runtimes an notice that we have not > really updated it > > with regards to minium python versions. > > > > currently > > > > the runtime defintion is as follows > https://github.com/openstack/governance/blob/master/reference/runtimes/yoga.rst > > > > It is the :doc:`policy > <../../resolutions/20181024-python-update-process>` that each OpenStack > release cycle will target the latest available version of Python; default > Python runtimes on the > > distributions listed above; and versions used in integration tests at > the start of the cycle, at least until the point when all projects have > migrated to a later version. > > > > Based on the criteria above, all Python-based projects must target > and test against, at a minimum: > > > > Python 3.6 (default in CentOS 8 Stream) > > Python 3.8 (latest available; default in Ubuntu 20.04) > > > > looking at to look at https://repology.org/project/python/versions what > python version are shpping by default now in many distro > > more and more of them are moving to python 3.9 which is currently not in > the test list and wer are starting to fall quite far behind > > > > debian 11/12/stable are currently using 3.9.2 > > centos 8 defaults to 3.6.8 but ships 3.8.6 and 3.9.2 as appstreams > We shouldn't rely on non-default versions. > > centos 9 stream will default to python3-3.9.6-6.el9.x86_64.rpm > https://composes.stream.centos.org/test/latest-CentOS-Stream/compose/BaseOS/x86_64/os/Packages/ > > ubuntu 20.04 default to 3.8.2 but also ships 3.9.5 in universe > Same. > > ubuntu 21.04 default to 3.9.5 so we can expect that when yoga ships on > ubuntu 22.04 it will be based on at least python 3.9 and proably 3.10 > > fedora well they basically ship everythin but i belive 3.9 is not the > default based on complaints some of my fellow contiutors had with > > runnign unit and functional tests. > > > > openSUSE Leap 15.3 main/oss python39 3.9.4 ship 3.9 but default > to 3.6 > > they aslo nolonger have an openstack product but it would still be nice > to keep them in mind. > Yep, some of us (namely Bifrost) support openSUSE Leap (and the transition to 15.3 hasn't happened yet) > > > > > > over all i think we shoudl consider raising or min tested/supported > python version to python 3.8 for yoga and consider updating from centos 8 > to centos 9. > Dropping CentOS 8 (and thus RHEL 8) support before RHEL 9 is released seems extremely premature to me. I understand that with CentOS Stream the world got some access to in-progress RHEL versions, but it does not change the fact that RHEL 9 does not exist yet, and there seems to be no public information on when it will appear. >From your analysis it seems that we should support 3.6 (CentOS/RHEL), 3.8 (Ubuntu 20.04) and 3.9 (Fedora, openSUSE, future RHEL). Dmitry > > i also think it worth including python 3.9 in the required testing > versions for python projects. > > > > several distros will be releaseing openstack xena/wallaby with python > 3.9 and also yoga likely with 3.10 in some cases. > > if we only offically test with 3.6 and 3.8 we could have some hard > probalems to solve going forward. the optional testing with unit and > functional tests > > that some project have for python 3.9 certenly helps but actully having > an integrated tempest gate job that runs on 3.9 i think is going to be > increasingly important. > > > > readring centos 9 stream, it was orginally planned of release at the end > of q2. now that date has obviosly passed but i know that they are still > hopeful to have > > a release ready in the next month or so. cloud images are avaibel at > https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/images/ > although the repos are still only > > partly complete > https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/images/ > and there are currently no active mirros so its still officaly > > not released https://www.centos.org/centos-stream/ and i don not have > any ETA beyond soonTM. as such im aware we cant offically declare centos 9 > stream as part of the tested > > runtimes at this point but it might be at least worht listing it as an > experimtal runtime as i suspec there will be a desire to swtich to it > during the yoga cycle for some proejct. > > ooo/rdo come to mind as they discussed this for xena but obviously that > was not possibel since it was not released in time. > > > > > > regards > > sean. > > > > ps removing python 3.6 and replacing it with 3.9 instead of just adding > 3.9 as an addtional runtime is more to manage our testing > > there isnt nessarly any python feature that i explcitly would like to be > able to use that are added in 3.8 that would be enabled > > by making that the new min python verion that is motivating this > suggestion. there are some nice thingin 3.9/3.10 such as pattern > > mataching that are commign but it will be some time (2-4 years)before we > can actully use those as our min version so i was just being practial > > and suggestion we raise to 3.8 and add 3.9 as required with 3.10 as > optional/experimal for those that want it. > > > > > > -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Tue Sep 7 13:29:53 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 7 Sep 2021 13:29:53 +0000 Subject: [tempest] change output format In-Reply-To: References: Message-ID: <20210907132953.wyjqzge5u6d2hk5n@yuggoth.org> On 2021-09-07 10:40:34 +0300 (+0300), Vladimir Prokofev wrote: > Is there a way to change the output format of tempest tests? > Preferably to JSON. So instead of test something... ok it produced > the same result but in JSON format. Tempest produces a subunit stream with details of all the tests performed. This is a standard data exchange format for test results, which can be readily parsed with a library like https://pypi.org/project/python-subunit/ after which serializing to JSON would presumably be a trivial exercise (or just consume the subunit stream directly if the desire was merely to use JSON for programmatic parsing, that's what the subunit is for anyway). -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From fungi at yuggoth.org Tue Sep 7 13:39:47 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 7 Sep 2021 13:39:47 +0000 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: References: Message-ID: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> On 2021-09-07 11:09:45 +0200 (+0200), Thomas Goirand wrote: [...] > since we have pinned global requirements, will these new updates > be included in the tests of older OpenStack releases? Is this > really the case right now? [...] They generally shouldn't, and most of the time can't realistically, have their entries altered in stable branches of the upper-constraints.txt file we use to pin dependency versions. To be able to do so would prevent them from using new features of their own dependencies in master branch development, since those would end up also needing updates in the pinned list of transitive stable branch constraints, which is exactly what we want to avoid. The workaround is to create a stable branch of the repository from the last version which was tagged during the cycle for the release which needs a fix to that independent deliverable backported. Those cases are rare, and basically what you would include as a stable distro package update for that coordinated release series anyway. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From haleyb.dev at gmail.com Tue Sep 7 13:44:35 2021 From: haleyb.dev at gmail.com (Brian Haley) Date: Tue, 7 Sep 2021 09:44:35 -0400 Subject: [neutron] Bug deputy report for week of August 30th Message-ID: <2322b0cb-a26a-8559-618f-8bd588343bd4@gmail.com> Hi, I was Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian Critical bugs ------------- * None High bugs --------- * https://bugs.launchpad.net/neutron/+bug/1942190 - [Fullstack] Timeout while waiting for port to be active - https://review.opendev.org/c/openstack/neutron/+/807134 Medium bugs ----------- * https://bugs.launchpad.net/neutron/+bug/1942033 - Mechanism driver 'ovn' failed in update_port_postcommit: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound - q-dhcp was also enabled in config causing the issue, removing fixed it * https://bugs.launchpad.net/neutron/+bug/1942251 - Routes has not added after router restart - https://review.opendev.org/c/openstack/neutron/+/721799 * https://bugs.launchpad.net/neutron/+bug/1942344 - ovs2ovn migration tool fails using VALIDATE_MIGRATION=False - https://review.opendev.org/c/openstack/neutron/+/806942 * https://bugs.launchpad.net/neutron/+bug/1942413 - subport status is not updated when trunk or subport notification is received - https://review.opendev.org/c/openstack/neutron/+/807076 * https://bugs.launchpad.net/neutron/+bug/1942615 - SG shared through RBAC mechanism can't be used to spawn instances - Slawek took ownership * https://bugs.launchpad.net/neutron/+bug/1942617 - SG rules from SG shared using RBAC aren't visible - Slawek took ownership Low bugs -------- * https://bugs.launchpad.net/neutron/+bug/1942234 - [OVS] Device "update" information could look the same between two different updates - https://review.opendev.org/c/openstack/neutron/+/806889 * https://bugs.launchpad.net/neutron/+bug/1942237 - Changed FIP event in Ie4c0f4a63a87c32026c49b03068e5f461deb38b6 - Won't fix - consumers have been fixed * https://bugs.launchpad.net/neutron/+bug/1942358 - Use "objects_exist" in port forwarding plugin - https://review.opendev.org/c/openstack/neutron/+/806962 * https://bugs.launchpad.net/neutron/+bug/1942448 - Neutron - “Update bandwidth limit rule” API on SUCCESS responds with 200 instead of Expected 202 - Looks like an api-ref typo - https://review.opendev.org/c/openstack/neutron-lib/+/807390 * https://bugs.launchpad.net/neutron/+bug/1942469 - Network delete notifications no longer contain segment info - Broken with https://review.opendev.org/c/openstack/neutron/+/786373 - https://review.opendev.org/c/openstack/neutron/+/807243 Misc bugs --------- * https://bugs.launchpad.net/neutron/+bug/1942458 - NeutronNetworks.create_and_delete_subnets fails NotFound - Asked for more information, could be container-related - Moved to neutron-charm for more investigation Wishlist bugs ------------- * https://bugs.launchpad.net/neutron/+bug/1942294 - Is allow_overlapping_ips config option still needed really? From dms at danplanet.com Tue Sep 7 14:07:14 2021 From: dms at danplanet.com (Dan Smith) Date: Tue, 07 Sep 2021 07:07:14 -0700 Subject: Adding another cell(v2) In-Reply-To: (Alexander Dibbo's message of "Mon, 6 Sep 2021 10:57:43 +0000") References: Message-ID: > Could someone point me to the documentation on how to add a new cell to cellsv2? I have found various documentation that says > to follow the procedure for Cells V1 but I cant find documentation for that. Below is the link to the doc I have found > > https://docs.openstack.org/nova/train/user/cells.html Yeah, this bullet in the doc you provided is what we have: https://docs.openstack.org/nova/latest/user/cells.html#adding-a-new-cell-to-an-existing-deployment It directs you to a different part of the page to run a single step of the v1 upgrade procedure. We obviously need to remove the v1 stuff and make the "add a new cell" thing stand on its own. There's a note in the latest version about that. But for the moment, you just need to create a new cell record with nova-manage (as described in that doc), pointing at the DB and MQ for the new cell. Then 'db sync' should hit that cell and blow in the schema just like it does for your main cell. Once that is done, start up conductors and computes pointing at the new cell MQ and DB and run the host discovery and make sure it finds the services registered in the new cell. --Dan From dmendiza at redhat.com Tue Sep 7 14:13:37 2021 From: dmendiza at redhat.com (Douglas Mendizabal) Date: Tue, 7 Sep 2021 09:13:37 -0500 Subject: [keystone] Weekly meetings In-Reply-To: References: Message-ID: On 8/18/21 10:39 AM, Douglas Mendizabal wrote: > Hi All, > > I'm helping the Keystone team get the weekly meetings back on track. [1] >  I'd like to propose a change to the meeting time slot from Tuesday at > 1700 UTC to Tuesday at 1500 UTC to make it a little earlier for our > contributors in EMEA. > > I'll be out next week, so I won't be available to chair the meeting, but > I should be back the following week for the August 31 meeting. > > If there are no objections to this time change, I'll go ahead and > propose a change to the meetings repo. > > Thanks, > - Douglas Mendizábal (redrobot) > > [1] https://meetings.opendev.org/#Keystone_Team_Meeting Hi All, Just a reminder that we'll be meeting at 1500 UTC today. Thanks, - Douglas Mendizábal (redrobot) From fungi at yuggoth.org Tue Sep 7 14:55:25 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 7 Sep 2021 14:55:25 +0000 Subject: [all][tc] Minium python version for yoga and centos 9 stream. In-Reply-To: <4b261268e701d3c464f180e89a795841c06dc8d0.camel@redhat.com> References: <33913002aa0ad4be63f6c36be310f4e1dd7bd008.camel@redhat.com> <4b261268e701d3c464f180e89a795841c06dc8d0.camel@redhat.com> Message-ID: <20210907145525.jhc4wfgsziyyabh5@yuggoth.org> On Tue, 2021-09-07 at 10:51 +0100, Sean Mooney wrote: > hi i was jsut looking at the yoga runtimes an notice that we have > not really updated it with regards to minium python versions. [...] That was introduced in July via https://review.opendev.org/799927 but should probably be considered a placeholder for now. We determine what versions of Python to target at or just prior to the start of the development cycle in order to use the latest information available to us, while avoiding changing team focus during the cycle. There's certainly still time to update it to include Python 3.9, and we do in fact already run (mostly non-voting) unit tests of many projects with 3.9. You may as well go ahead and push your proposed edit for review, it may not need much discussion. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From tonykarera at gmail.com Tue Sep 7 15:00:59 2021 From: tonykarera at gmail.com (Karera Tony) Date: Tue, 7 Sep 2021 17:00:59 +0200 Subject: Octavia Issue: Failed to build compute instance due to: Failed to retrieve image with amphora tag Message-ID: Hello Team, I have openstack wallaby running on Ubuntu deployed using kolla-ansible but whenever I create a load balancer I get the error below in the octavia-worker.log octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: Failed to retrieve image with amphora tag 2021-09-06 14:14:49.392 20 ERROR oslo_messaging.rpc.server But I have an image with the correct tag as shown below. (kolla-openstack) stack at deployment:~$ openstack image show amphora1 | grep tag | tags | amphora Kindly advise if I am missing anything. Regards Tony Karera -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Tue Sep 7 15:38:12 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 07 Sep 2021 10:38:12 -0500 Subject: [all][tc] Minium python version for yoga and centos 9 stream. In-Reply-To: <4b261268e701d3c464f180e89a795841c06dc8d0.camel@redhat.com> References: <33913002aa0ad4be63f6c36be310f4e1dd7bd008.camel@redhat.com> <4b261268e701d3c464f180e89a795841c06dc8d0.camel@redhat.com> Message-ID: <17bc0e8dea4.fc87ca61142047.29866411617360676@ghanshyammann.com> ---- On Tue, 07 Sep 2021 06:07:55 -0500 Sean Mooney wrote ---- > resending to correct list > On Tue, 2021-09-07 at 10:51 +0100, Sean Mooney wrote: > > hi i was jsut looking at the yoga runtimes an notice that we have not really updated it > > with regards to minium python versions. > > > > currently > > > > the runtime defintion is as follows https://github.com/openstack/governance/blob/master/reference/runtimes/yoga.rst > > > > It is the :doc:`policy <../../resolutions/20181024-python-update-process>` that each OpenStack release cycle will target the latest available version of Python; default Python runtimes on the > > distributions listed above; and versions used in integration tests at the start of the cycle, at least until the point when all projects have migrated to a later version. > > > > Based on the criteria above, all Python-based projects must target and test against, at a minimum: > > > > Python 3.6 (default in CentOS 8 Stream) > > Python 3.8 (latest available; default in Ubuntu 20.04) > > > > looking at to look at https://repology.org/project/python/versions what python version are shpping by default now in many distro > > more and more of them are moving to python 3.9 which is currently not in the test list and wer are starting to fall quite far behind > > > > debian 11/12/stable are currently using 3.9.2 > > centos 8 defaults to 3.6.8 but ships 3.8.6 and 3.9.2 as appstreams > > centos 9 stream will default to python3-3.9.6-6.el9.x86_64.rpm https://composes.stream.centos.org/test/latest-CentOS-Stream/compose/BaseOS/x86_64/os/Packages/ > > ubuntu 20.04 default to 3.8.2 but also ships 3.9.5 in universe > > ubuntu 21.04 default to 3.9.5 so we can expect that when yoga ships on ubuntu 22.04 it will be based on at least python 3.9 and proably 3.10 > > fedora well they basically ship everythin but i belive 3.9 is not the default based on complaints some of my fellow contiutors had with > > runnign unit and functional tests. > > > > openSUSE Leap 15.3 main/oss python39 3.9.4 ship 3.9 but default to 3.6 > > they aslo nolonger have an openstack product but it would still be nice to keep them in mind. > > Definitely we can move to 3.9 as official testing runtime if any of the currently supporting distro support it as default. openSuse distro is not supported anymore due to no/less volunteer to maintain it. centos9 or Ubuntu 22.04 LTS is not yet released so we have not picked them yet but that is something we discussed during the review of defining Yoga testing runtime. If any of these distro is released before Yoga development cycle we can propose a update to the current testing runtime - https://review.opendev.org/c/openstack/governance/+/799927/1/reference/runtimes/yoga.rst#9 Also one thing to make a note here, py3.9 is being tested as non-voting in all the project unit tests and hopefully projects has made it green. There is no restriction to run unit or functional tests on py3.9 as voting testing. Testing runtime is just a minimum testing we except projects to do and it can be extended at any distro/versions at project level. -gmann > > > > over all i think we shoudl consider raising or min tested/supported python version to python 3.8 for yoga and consider updating from centos 8 to centos 9. > > i also think it worth including python 3.9 in the required testing versions for python projects. > > > > several distros will be releaseing openstack xena/wallaby with python 3.9 and also yoga likely with 3.10 in some cases. > > if we only offically test with 3.6 and 3.8 we could have some hard probalems to solve going forward. the optional testing with unit and functional tests > > that some project have for python 3.9 certenly helps but actully having an integrated tempest gate job that runs on 3.9 i think is going to be increasingly important. > > > > readring centos 9 stream, it was orginally planned of release at the end of q2. now that date has obviosly passed but i know that they are still hopeful to have > > a release ready in the next month or so. cloud images are avaibel at https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/images/ although the repos are still only > > partly complete https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/images/ and there are currently no active mirros so its still officaly > > not released https://www.centos.org/centos-stream/ and i don not have any ETA beyond soonTM. as such im aware we cant offically declare centos 9 stream as part of the tested > > runtimes at this point but it might be at least worht listing it as an experimtal runtime as i suspec there will be a desire to swtich to it during the yoga cycle for some proejct. > > ooo/rdo come to mind as they discussed this for xena but obviously that was not possibel since it was not released in time. > > > > > > regards > > sean. > > > > ps removing python 3.6 and replacing it with 3.9 instead of just adding 3.9 as an addtional runtime is more to manage our testing > > there isnt nessarly any python feature that i explcitly would like to be able to use that are added in 3.8 that would be enabled > > by making that the new min python verion that is motivating this suggestion. there are some nice thingin 3.9/3.10 such as pattern > > mataching that are commign but it will be some time (2-4 years)before we can actully use those as our min version so i was just being practial > > and suggestion we raise to 3.8 and add 3.9 as required with 3.10 as optional/experimal for those that want it. > > > > > > From johnsomor at gmail.com Tue Sep 7 15:56:26 2021 From: johnsomor at gmail.com (Michael Johnson) Date: Tue, 7 Sep 2021 08:56:26 -0700 Subject: Octavia Issue: Failed to build compute instance due to: Failed to retrieve image with amphora tag In-Reply-To: References: Message-ID: Hi Tony, As Greg mentioned in his reply to your previous email, that log line is truncated and doesn't tell us what issue you are seeing. Please see his response: http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024688.html Michael On Tue, Sep 7, 2021 at 8:07 AM Karera Tony wrote: > > Hello Team, > > I have openstack wallaby running on Ubuntu deployed using kolla-ansible > but whenever I create a load balancer I get the error below in the > octavia-worker.log > > octavia.common.exceptions.ComputeBuildException: Failed to build compute > instance due to: Failed to retrieve image with amphora tag > 2021-09-06 14:14:49.392 20 ERROR oslo_messaging.rpc.server > > But I have an image with the correct tag as shown below. > > (kolla-openstack) stack at deployment:~$ openstack image show amphora1 | > grep tag > | tags | amphora > > Kindly advise if I am missing anything. > Regards > > Tony Karera > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Tue Sep 7 16:03:09 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 7 Sep 2021 16:03:09 +0000 Subject: [all][tc] Minium python version for yoga and centos 9 stream. In-Reply-To: <17bc0e8dea4.fc87ca61142047.29866411617360676@ghanshyammann.com> References: <33913002aa0ad4be63f6c36be310f4e1dd7bd008.camel@redhat.com> <4b261268e701d3c464f180e89a795841c06dc8d0.camel@redhat.com> <17bc0e8dea4.fc87ca61142047.29866411617360676@ghanshyammann.com> Message-ID: <20210907160308.5qcih7675nbjwghu@yuggoth.org> On 2021-09-07 10:38:12 -0500 (-0500), Ghanshyam Mann wrote: [...] > Definitely we can move to 3.9 as official testing runtime if any > of the currently supporting distro support it as default. [...] I don't think it needs to be the default Python in a release so long as it's available. To quote from https://governance.openstack.org/tc/resolutions/20181024-python-update-process.html#unit-tests "Prior to the beginning of each release cycle, the TC will designate the minor versions of Python 3 that will be tested in that release using the following criteria: [...] The latest released version of Python 3 that is available in any distribution we can feasibly use for testing. It need not be a long-term-supported release, but could be a non-LTS version of Ubuntu, Fedora, openSUSE Leap, or even a rolling release distribution (such as Debian Testing or openSUSE Tumbleweed) if necessary." I recall clearly that this was added so that we would attempt to test the latest available version of Python in order to help avoid having software which is clearly non-functional on distributions which are working in parallel to update their defaults (and so might reasonably want to ship OpenStack with an interpreter version which was not their default at the time we started development for that coordinated release). -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From gmann at ghanshyammann.com Tue Sep 7 16:44:02 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 07 Sep 2021 11:44:02 -0500 Subject: [all][tc] Minium python version for yoga and centos 9 stream. In-Reply-To: <20210907160308.5qcih7675nbjwghu@yuggoth.org> References: <33913002aa0ad4be63f6c36be310f4e1dd7bd008.camel@redhat.com> <4b261268e701d3c464f180e89a795841c06dc8d0.camel@redhat.com> <17bc0e8dea4.fc87ca61142047.29866411617360676@ghanshyammann.com> <20210907160308.5qcih7675nbjwghu@yuggoth.org> Message-ID: <17bc125274c.114f9c117145230.1407351724601243402@ghanshyammann.com> ---- On Tue, 07 Sep 2021 11:03:09 -0500 Jeremy Stanley wrote ---- > On 2021-09-07 10:38:12 -0500 (-0500), Ghanshyam Mann wrote: > [...] > > Definitely we can move to 3.9 as official testing runtime if any > > of the currently supporting distro support it as default. > [...] > > I don't think it needs to be the default Python in a release so long > as it's available. To quote from > https://governance.openstack.org/tc/resolutions/20181024-python-update-process.html#unit-tests > > "Prior to the beginning of each release cycle, the TC will > designate the minor versions of Python 3 that will be tested in > that release using the following criteria: > [...] > The latest released version of Python 3 that is available in any > distribution we can feasibly use for testing. It need not be a > long-term-supported release, but could be a non-LTS version of > Ubuntu, Fedora, openSUSE Leap, or even a rolling release > distribution (such as Debian Testing or openSUSE Tumbleweed) if > necessary." > > I recall clearly that this was added so that we would attempt to > test the latest available version of Python in order to help avoid > having software which is clearly non-functional on distributions > which are working in parallel to update their defaults (and so might > reasonably want to ship OpenStack with an interpreter version which > was not their default at the time we started development for that > coordinated release). Sure, and by seeing the current py39 job it is mostly passing - https://zuul.openstack.org/builds?job_name=openstack-tox-py39 Let's discuss it in PTG and hopefully by then we might see centos9 release or closure plan to release dates and as discussed during defining the Yoga tetsing runtime, we can update it once if centos9 is there then update centos versionb as well as the py version otherwise we can disucss only making py39 jobs voting which are running on ubuntu focal. -gmann > -- > Jeremy Stanley > From smooney at redhat.com Tue Sep 7 17:39:40 2021 From: smooney at redhat.com (Sean Mooney) Date: Tue, 07 Sep 2021 18:39:40 +0100 Subject: [all][tc] Minium python version for yoga and centos 9 stream. In-Reply-To: <17bc125274c.114f9c117145230.1407351724601243402@ghanshyammann.com> References: <33913002aa0ad4be63f6c36be310f4e1dd7bd008.camel@redhat.com> <4b261268e701d3c464f180e89a795841c06dc8d0.camel@redhat.com> <17bc0e8dea4.fc87ca61142047.29866411617360676@ghanshyammann.com> <20210907160308.5qcih7675nbjwghu@yuggoth.org> <17bc125274c.114f9c117145230.1407351724601243402@ghanshyammann.com> Message-ID: <50423dee4b245d18befa21bd7ef1666653f7bf2e.camel@redhat.com> On Tue, 2021-09-07 at 11:44 -0500, Ghanshyam Mann wrote: > ---- On Tue, 07 Sep 2021 11:03:09 -0500 Jeremy Stanley wrote ---- > > On 2021-09-07 10:38:12 -0500 (-0500), Ghanshyam Mann wrote: > > [...] > > > Definitely we can move to 3.9 as official testing runtime if any > > > of the currently supporting distro support it as default. > > [...] > > > > I don't think it needs to be the default Python in a release so long > > as it's available. To quote from > > https://governance.openstack.org/tc/resolutions/20181024-python-update-process.html#unit-tests > > > > "Prior to the beginning of each release cycle, the TC will > > designate the minor versions of Python 3 that will be tested in > > that release using the following criteria: > > [...] > > The latest released version of Python 3 that is available in any > > distribution we can feasibly use for testing. It need not be a > > long-term-supported release, but could be a non-LTS version of > > Ubuntu, Fedora, openSUSE Leap, or even a rolling release > > distribution (such as Debian Testing or openSUSE Tumbleweed) if > > necessary." > > > > I recall clearly that this was added so that we would attempt to > > test the latest available version of Python in order to help avoid > > having software which is clearly non-functional on distributions > > which are working in parallel to update their defaults (and so might > > reasonably want to ship OpenStack with an interpreter version which > > was not their default at the time we started development for that > > coordinated release). > > Sure, and by seeing the current py39 job it is mostly passing > - https://zuul.openstack.org/builds?job_name=openstack-tox-py39 > > Let's discuss it in PTG and hopefully by then we might see centos9 > release or closure plan to release dates and as discussed during > defining the Yoga tetsing runtime, we can update it once if > centos9 is there then update centos versionb as well as the py version > otherwise we can disucss only making py39 jobs voting which are running > on ubuntu focal. not sure if centos 9 will be avaiable by ptg but i hope so. in anycaes ubuntu 21.04 default to python 3.9 and debian 11 "bullseye" which is the latest __stable__ debian release also default to python 3.9 as does fedora 34 but that a much bigger moving target. in general 3.9 is avaiable if not default in several distro which is why i would like to include it for required testing e.g. move the current non voting jobs to voting for unit and functest at least. it woudl be nice if we could have at least on tempst job on py39 too but sure lets pick this up at the ptg. im not agaisnt keeping support for phython 3.6 and centos 8 stream for yoga but i would like to be abel to do at least some smoke tests with centos 9 stream if its avaiable. the none stream version fo centos 8 i think is end of life at the end of this year? but the centos 8 stream release will continue untile rhel 8 is eol so that shoudl still be supproted to some degree for the full yoga cycle. the none stream version however wont be supported when yoga releases. > > -gmann > > > -- > > Jeremy Stanley > > > From mkopec at redhat.com Tue Sep 7 19:15:04 2021 From: mkopec at redhat.com (Martin Kopec) Date: Tue, 7 Sep 2021 21:15:04 +0200 Subject: [patrole][qa] Proposing Soniya Vyas to Patrole core Message-ID: Hi all, Soniya Vyas (IRC: soniya29) is very enthusiastic about Patrole project and since she has been very helpful, I'd like to propose her for Patrole core. You can vote/feedback on this email. If no objections within a week, I'll add her to the core list. Regards, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Tue Sep 7 19:29:49 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 07 Sep 2021 14:29:49 -0500 Subject: [patrole][qa] Proposing Soniya Vyas to Patrole core In-Reply-To: References: Message-ID: <17bc1bcebec.d7cc6895149770.7494147722877953142@ghanshyammann.com> ---- On Tue, 07 Sep 2021 14:15:04 -0500 Martin Kopec wrote ---- > Hi all, > > Soniya Vyas (IRC: soniya29) is very enthusiastic about Patrole project and since she has been very helpful, I'd like to propose her for Patrole core. > > You can vote/feedback on this email. If no objections within a week, I'll add her to the core list. +1, she has been very helpful in Patrole project. -gmann > > Regards, > -- > Martin Kopec > Senior Software Quality Engineer > Red Hat EMEA > > > From dmeng at uvic.ca Tue Sep 7 19:20:39 2021 From: dmeng at uvic.ca (dmeng) Date: Tue, 07 Sep 2021 12:20:39 -0700 Subject: [sdk]: Block storage service update volume status from reserved to available Message-ID: <11f602803604a655e1be5d8cc05383b4@uvic.ca> Hello there, Hope everything is going well. I would like to know if there is any method that could update the volume status or reset its state. While we creating volumes use the block storage service or creating vms with volumes use the compute service, sometimes we got volumes stuck in creating/deleteing/attaching/reserved state. And we couldn't remove them or reset their state without an admin account even we own the project. The way we create volumes and boot vms is like the following: # create the volume cv = cinder.create_volume(name=bv_name, size=size, image_id=imageobj.id, volume_type=volume_type, is_bootable=True) # format mapping data bdm = [{'uuid': cv.id, 'source_type': 'volume', 'destination_type': 'volume', 'delete_on_termination': True, 'boot_index': 0}] # create the vm nova.create_server(name=hostname, image_id=imageobj.id, flavor_id=flavorl.id, key_name=key_name, block_device_mapping=bdm, user_data=format_userdata, networks=netid, security_groups=sec_groups) In the normal situation, everything works. But in the case of any network issue, the volume might stuck in the process of creating, deleting, or attaching. And if the volume got created successfully, when it tries to create the vm, the volume will be reserved for it, but if anything wrong happen when creating the vm or get timed out, the volume will be in the reserved stage forever. We would like to know if we could update the state or delete them without the admin account? We tried the command line tools 'openstack volume set --state available ' and 'cinder reset-state --state available ', but both got: Policy doesn't allow volume_extension:volume_admin_actions:reset_status to be performed. (HTTP 403) (Request-ID: req-b50236c3-5662-4cc0-8602-19e79e80219d) ERROR: Unable to reset the state for the specified entity(s). We are using the chameleon openstack cloud but we are not the admin user there, so wondering if there is any method that we could update the volume in our own project? Thanks and have a great day! Catherine -------------- next part -------------- An HTML attachment was scrubbed... URL: From ildiko.vancsa at gmail.com Tue Sep 7 22:08:15 2021 From: ildiko.vancsa at gmail.com (Ildiko Vancsa) Date: Tue, 7 Sep 2021 15:08:15 -0700 Subject: =?utf-8?Q?Re=3A_=5Blists=2Eopenstack=2Eorg=E4=BB=A3=E5=8F=91=5D?= =?utf-8?Q?=5Bzun=5D=5Bcyborg=5D=5Bkuryr=5D_Edge_for_scientific_research_s?= =?utf-8?Q?ession_on_September_13_Edge_WG_meeting?= In-Reply-To: References: Message-ID: <5DB3E1B1-1D6A-4C0B-AB3D-FE33F0B88961@gmail.com> Hi Bailin, Thank you for sharing the links. Zun is the project in OpenStack that is supporting to run and manage containers directly from OpenStack without the need of Kubernetes. As the Chameleon project is using it to run their containerized workload and they also need hardware accelerators they integrated it with Cyborg. Many of their activities fall under the edge computing area and they are starting their collaboration with the OpenInfra Edge Computing group for this reason. As part of that work we are helping them to reach relevant communities and project teams where their requirements identified some gaps and I believe the Cyborg-Zun integration is one of these. They would like to upstream their work if the community finds value in it as it may or may not fall into the current scope of the respective projects. I hope that we can start to discuss some of this on the meeting next Monday and take next steps from there. I hope this helps a bit with the context and we can have a fruitful discussion next week. I will share the recording of the meeting as soon as it is available in case someone would not be able to make it and would like to check out the presentation and the discussion that follows. Thanks, Ildikó > On Sep 3, 2021, at 04:20, Brin Zhang(张百林) wrote: > > Hi Ildiko, > > Currently Cyborg and nova support interaction, we are not very familiar with zun yet, if possible, please provide some relevant information. You can refer to [1] for Cyborg related materials, and [2] for the interactive content of nova and cyborg (not yet merged). > > [1]https://docs.openstack.org/cyborg/latest/user/architecture.html > [2]https://review.opendev.org/c/openstack/cyborg/+/794549 > https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/nova-cyborg-interaction.html > > > Bailin Zhang (brinzhang) > > > -----邮件原件----- > 发件人: Ildiko Vancsa [mailto:ildiko.vancsa at gmail.com] > 发送时间: 2021年8月31日 22:39 > 收件人: openstack-discuss > 主题: [lists.openstack.org代发][zun][cyborg][kuryr] Re: Edge for scientific research session on September 13 Edge WG meeting > > Hi, > > It is a friendly reminder that the OpenInfra Edge Computing Group is having a joint session with the Chameleon project. > > During this session you can learn about the Chameleon project and how they are using OpenStack to build an edge platform for scientific research and academic purposes. They have also found some gaps during their journey. To mention an example the integration between Cyborg and Zun is missing to be able to use acceleration devices with containers orchestrated by OpenStack and they have been working to fill in this gap. > > The session will start with a presentation about the project and then we will use the remaining time to talk about solutions and next steps. I would like to invite everyone who is interested in learning about how OpenStack can be turned into an edge platform and what are the missing pieces of the puzzle that still need to be enhanced or implemented. > > The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. > The meeting will be held on Zoom, for the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings > > Please let me know if you need a calendar invite and I can send you one to get reminded about the meeting. > > Thanks and Best Regards, > Ildikó > > >> On Aug 25, 2021, at 17:00, Ildiko Vancsa wrote: >> >> Hi, >> >> I’m very excited to invite you to a session with the Chameleon project to discuss how edge computing is powering education, scientific research and more! As always in the area of edge there are challenges and gaps that we need to solve together. In that sense the session will start with a presentation that we will then turn into a discussion to turn towards discussing solutions. >> >> Please also note that the Chameleon project is using OpenStack to build edge solutions. As they identified some gaps they would like to work with the community to address the challenges that they’ve been hitting. Please join us to get their feedback and find the best way to improve OpenStack to deliver a platform that can be used on the edge! >> >> The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. >> For the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings >> >> Please find the details of the session below. >> >> Thanks and Best Regards, >> Ildikó >> >> >> Title: Chameleon: a cloud to edge infrastructure for science >> >> Abstract: The NSF-funded Chameleon project (www.chameleoncloud.org) provides an OpenStack-based bare metal reconfigurable cloud that supports systems research, education, and emergent application projects in science. The facility has been in existence for 6+ years, and has served ~6,000 users working on a variety of projects ranging from networking, operating systems design, and security, to innovative applications in astronomy, physics, and life sciences. Recent research trends in IoT and edge computing, as evidenced by increased numbers of project applications in those areas, made it clear that Chameleon needed to expand into the edge space. Remaining within the OpenStack framework allowed us to leverage end-user features that we developed for Chameleon, such as access via federated identity and Jupyter integration. Accordingly, we developed infrastructure called CHI at Edge that adapts Zun to provision containers on edge devices and significantly expands its capabilities to work in the edge space. The talk will describe our use case and the emergent product definition, the OpenStack adaptations we made to the current preview implementation, the directions we plan to take in our further work, as well as the ecosystem of projects and NSF infrastructure deployments leveraging this work and collaborating with us. >> >> > > From ildiko.vancsa at gmail.com Tue Sep 7 22:11:20 2021 From: ildiko.vancsa at gmail.com (Ildiko Vancsa) Date: Tue, 7 Sep 2021 15:11:20 -0700 Subject: [zun][cyborg][kuryr] Edge for scientific research session on September 13 Edge WG meeting In-Reply-To: <2ff99838.2b86.17bb1528b2a.Coremail.joykechen@163.com> References: <315A37E8-8928-4225-B26F-A15ED0788E24@gmail.com> <2ff99838.2b86.17bb1528b2a.Coremail.joykechen@163.com> Message-ID: <9B16F1E4-A983-4C69-B890-1AED785AFF69@gmail.com> Dear Ke Chen, Thank you for the nice feedback and support. I hope the outcome of the discussion and activities that we will start next week will be to upstream this work item so that everyone can benefit from it. I will send you a reminder email about the meeting. See you next Monday! Best Regards, Ildikó > On Sep 4, 2021, at 07:59, 陈克 wrote: > > Hi Ildiko, > I'm a Cyborg core. I think it's cool to promote the integration of Cyborg and Zun. I'm happy to push > the whole thing if I could. Zun's PTL Shengqin Feng is my colleague. I suggest you send her and me a > reminder email before the meeting. > > Her Email: > feng.shengqin at zte.com.cn > > > Thanks, > Ke Chen > > > > > > > At 2021-08-31 22:39:01, "Ildiko Vancsa" wrote: > >Hi, > > > >It is a friendly reminder that the OpenInfra Edge Computing Group is having a joint session with the Chameleon project. > > > >During this session you can learn about the Chameleon project and how they are using OpenStack to build an edge platform for scientific research and academic purposes. They have also found some gaps during their journey. To mention an example the integration between Cyborg and Zun is missing to be able to use acceleration devices with containers orchestrated by OpenStack and they have been working to fill in this gap. > > > >The session will start with a presentation about the project and then we will use the remaining time to talk about solutions and next steps. I would like to invite everyone who is interested in learning about how OpenStack can be turned into an edge platform and what are the missing pieces of the puzzle that still need to be enhanced or implemented. > > > >The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. > >The meeting will be held on Zoom, for the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings > > > >Please let me know if you need a calendar invite and I can send you one to get reminded about the meeting. > > > >Thanks and Best Regards, > >Ildikó > > > > > >> On Aug 25, 2021, at 17:00, Ildiko Vancsa wrote: > >> > >> Hi, > >> > >> I’m very excited to invite you to a session with the Chameleon project to discuss how edge computing is powering education, scientific research and more! As always in the area of edge there are challenges and gaps that we need to solve together. In that sense the session will start with a presentation that we will then turn into a discussion to turn towards discussing solutions. > >> > >> Please also note that the Chameleon project is using OpenStack to build edge solutions. As they identified some gaps they would like to work with the community to address the challenges that they’ve been hitting. Please join us to get their feedback and find the best way to improve OpenStack to deliver a platform that can be used on the edge! > >> > >> The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. > >> For the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings > >> > >> Please find the details of the session below. > >> > >> Thanks and Best Regards, > >> Ildikó > >> > >> > >> Title: Chameleon: a cloud to edge infrastructure for science > >> > >> Abstract: The NSF-funded Chameleon project (www.chameleoncloud.org) provides an OpenStack-based bare metal reconfigurable cloud that supports systems research, education, and emergent application projects in science. The facility has been in existence for 6+ years, and has served ~6,000 users working on a variety of projects ranging from networking, operating systems design, and security, to innovative applications in astronomy, physics, and life sciences. Recent research trends in IoT and edge computing, as evidenced by increased numbers of project applications in those areas, made it clear that Chameleon needed to expand into the edge space. Remaining within the OpenStack framework allowed us to leverage end-user features that we developed for Chameleon, such as access via federated identity and Jupyter integration. Accordingly, we developed infrastructure called CHI at Edge that adapts Zun to provision containers on edge devices and significantly expands its capabilities to work in the edge space. The talk will describe our use case and the emergent product definition, the OpenStack adaptations we made to the current preview implementation, the directions we plan to take in our further work, as well as the ecosystem of projects and NSF infrastructure deployments leveraging this work and collaborating with us. > >> > >> > > > > > > From ildiko.vancsa at gmail.com Tue Sep 7 22:20:11 2021 From: ildiko.vancsa at gmail.com (Ildiko Vancsa) Date: Tue, 7 Sep 2021 15:20:11 -0700 Subject: [zun][cyborg][kuryr][all] Edge for scientific research session on September 13 Edge WG meeting In-Reply-To: References: <315A37E8-8928-4225-B26F-A15ED0788E24@gmail.com> Message-ID: <24F010F0-5DE7-4455-8038-E23B9793FACD@gmail.com> Hi, It is a friendly reminder that the OpenInfra Edge Computing Group is having a joint session with the Chameleon project next Monday (September 13). During this session you can learn about the Chameleon project and how they are using OpenStack to build an edge platform for scientific research and academic purposes and the gaps they identified on the way along with the work they started to fill them. The session will start with a presentation about the project and then we will use the remaining time to talk about solutions and next steps. I would like to invite everyone who is interested in learning about how OpenStack can be turned into an edge platform and what are the missing pieces of the puzzle that still need to be enhanced or implemented. The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. The meeting will be held on Zoom, for the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings Please let me know if you need a calendar invite and I can send you one to get reminded about the meeting. Thanks and Best Regards, Ildikó From zigo at debian.org Tue Sep 7 22:46:30 2021 From: zigo at debian.org (Thomas Goirand) Date: Wed, 8 Sep 2021 00:46:30 +0200 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> References: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> Message-ID: On 9/7/21 3:39 PM, Jeremy Stanley wrote: > On 2021-09-07 11:09:45 +0200 (+0200), Thomas Goirand wrote: > [...] >> since we have pinned global requirements, will these new updates >> be included in the tests of older OpenStack releases? Is this >> really the case right now? > [...] > > They generally shouldn't, and most of the time can't realistically, > have their entries altered in stable branches of the > upper-constraints.txt file we use to pin dependency versions. To be > able to do so would prevent them from using new features of their > own dependencies in master branch development, since those would end > up also needing updates in the pinned list of transitive stable > branch constraints, which is exactly what we want to avoid. > > The workaround is to create a stable branch of the repository from > the last version which was tagged during the cycle for the release > which needs a fix to that independent deliverable backported. Those > cases are rare, and basically what you would include as a stable > distro package update for that coordinated release series anyway. > Hi, Thanks for your insights. Basically, what you're writing above confirm what I thought: the release-independent components are in fact not really release-independent, and are bound to a specific release of OpenStack. Most of the time, they aren't updated in the stable release CI. Therefore, it would probably be wise to try to minimize the amount of release-independent components to reflect this reality. In fact, I was particularly confuse to realize that something like taskflow was declared release-independent since Wallaby. Now, if I'm told that it is loosely coupled with OpenStack, knowing that it's a key component, I'm not sure what I should do. Should I get taskflow updated to 4.6.2 in Wallaby, so I get bugfixes? Or is it safer to just upgrade it for Xena? I took Taskflow as an example (you can reply to me about Taskflow in particular, but that's not the point I'm trying to make). But it could have been something else (deptcollector? oslo.i18n?). What I'm trying to say is that as a downstream consumer, it's from hard to impossible to know what to do for the past OpenStack releases regarding these release-independent components: should I update them or not? It is unrealistic to hope that one can track what component has been updated to a newer version in the stable release CI. Therefore, if possible, I would prefer clearer, easier to track, release-bound components with point release updates (though I understand that it may not be possible if that's too much work and OpenStack is getting understaffed...). Cheers, Thomas Goirand (zigo) From fungi at yuggoth.org Tue Sep 7 23:21:44 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 7 Sep 2021 23:21:44 +0000 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: References: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> Message-ID: <20210907232144.n25ei2cyrmkr2xaz@yuggoth.org> On 2021-09-08 00:46:30 +0200 (+0200), Thomas Goirand wrote: [...] > In fact, I was particularly confuse to realize that something like > taskflow was declared release-independent since Wallaby. Now, if > I'm told that it is loosely coupled with OpenStack, knowing that > it's a key component, I'm not sure what I should do. Should I get > taskflow updated to 4.6.2 in Wallaby, so I get bugfixes? Or is it > safer to just upgrade it for Xena? > > I took Taskflow as an example (you can reply to me about Taskflow > in particular, but that's not the point I'm trying to make). But > it could have been something else (deptcollector? oslo.i18n?). > What I'm trying to say is that as a downstream consumer, it's from > hard to impossible to know what to do for the past OpenStack > releases regarding these release-independent components: should I > update them or not? It is unrealistic to hope that one can track > what component has been updated to a newer version in the stable > release CI. Specific examples are great actually, because it at least lets us try to formulate a specific answer and then reason about whether a similar answer can be generally applied to other related scenarios. So let's take taskflow in Wallaby and Xena: The closest thing we have to a recommendation for versions of "outside" dependencies (which is essentially what we treat release-independent deliverables as) is the upper-constraints.txt file on each branch of the openstack/requirements repository. In the case of taskflow, at the moment the stable/wallaby branch upper constraint for taskflow is 4.6.0, which is the version we indicate all OpenStack projects should test their stable/wallaby changes against. That entry was last updated by https://review.opendev.org/773620 which merged to the master branch of openstack/requirements in February, well before the coordinated Wallaby release. Even though the master (upcoming Xena coordinated release) upper-constraints.txt file lists taskflow===4.6.2, we don't test stable branch changes of OpenStack projects with that, only what will eventually become stable/xena once the Xena cycle concludes. > Therefore, if possible, I would prefer clearer, easier to track, > release-bound components with point release updates (though I > understand that it may not be possible if that's too much work and > OpenStack is getting understaffed...). It seems like what you're asking for is this: https://opendev.org/openstack/requirements/src/branch/stable/wallaby/upper-constraints.txt Or rather, that's the closest thing we have to what I think you're requesting. If you follow the commit history of that file, you'll see recent updates for it on the stable/wallaby branch do consist almost entirely of stable point releases of cycle-oriented projects: https://opendev.org/openstack/requirements/commits/branch/stable/wallaby/upper-constraints.txt Does this help narrow the scope of concern to the point where we can better reason about the problems or challenges you're facing? -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From ildiko.vancsa at gmail.com Tue Sep 7 23:22:42 2021 From: ildiko.vancsa at gmail.com (Ildiko Vancsa) Date: Tue, 7 Sep 2021 16:22:42 -0700 Subject: Edge sessions at the upcoming PTG Message-ID: Hi, I’m reaching out to you to share the agenda of the OpenInfra Edge Computing Group that we put together for the upcoming PTG. I would like to invite everyone who is interested in discussing edge challenges and finding solutions! We summarized our plans for the event in a short blog post to give some context to each of the topic that we picked to discuss in details. We picked key areas like security, networking, automation and tools, containers and more: https://superuser.openstack.org/articles/the-project-teams-gathering-is-coming-lets-talk-edge/ Our etherpad for the sessions: https://etherpad.opendev.org/p/ecg-ptg-october-2021 Please let me know if you have any questions about the agenda or topics. Thanks and Best Regards, Ildikó From zhangbailin at inspur.com Wed Sep 8 00:15:36 2021 From: zhangbailin at inspur.com (=?utf-8?B?QnJpbiBaaGFuZyjlvKDnmb7mnpcp?=) Date: Wed, 8 Sep 2021 00:15:36 +0000 Subject: =?utf-8?B?562U5aSNOiBbbGlzdHMub3BlbnN0YWNrLm9yZ+S7o+WPkV1benVuXVtjeWJv?= =?utf-8?B?cmddW2t1cnlyXSBFZGdlIGZvciBzY2llbnRpZmljIHJlc2VhcmNoIHNlc3Np?= =?utf-8?Q?on_on_September_13_Edge_WG_meeting?= In-Reply-To: <5DB3E1B1-1D6A-4C0B-AB3D-FE33F0B88961@gmail.com> References: <5DB3E1B1-1D6A-4C0B-AB3D-FE33F0B88961@gmail.com> Message-ID: <360c63346cfd44fca42eaa744c84ef8d@inspur.com> > I will share the recording of the meeting as soon as it is available in case someone would not be able to make it and would like to check out the presentation and the discussion that follows. Thanks for you will share of the record, at 1300UTC I have another meeting (start 1100UTC), if I can end up that meeting earlier, I will join this meeting ^^ brinzhang -----邮件原件----- 发件人: Ildiko Vancsa [mailto:ildiko.vancsa at gmail.com] 发送时间: 2021年9月8日 6:08 收件人: Brin Zhang(张百林) 抄送: openstack-discuss at lists.openstack.org; xin-ran.wang at intel.com 主题: Re: [lists.openstack.org代发][zun][cyborg][kuryr] Edge for scientific research session on September 13 Edge WG meeting Hi Bailin, Thank you for sharing the links. Zun is the project in OpenStack that is supporting to run and manage containers directly from OpenStack without the need of Kubernetes. As the Chameleon project is using it to run their containerized workload and they also need hardware accelerators they integrated it with Cyborg. Many of their activities fall under the edge computing area and they are starting their collaboration with the OpenInfra Edge Computing group for this reason. As part of that work we are helping them to reach relevant communities and project teams where their requirements identified some gaps and I believe the Cyborg-Zun integration is one of these. They would like to upstream their work if the community finds value in it as it may or may not fall into the current scope of the respective projects. I hope that we can start to discuss some of this on the meeting next Monday and take next steps from there. I hope this helps a bit with the context and we can have a fruitful discussion next week. I will share the recording of the meeting as soon as it is available in case someone would not be able to make it and would like to check out the presentation and the discussion that follows. Thanks, Ildikó > On Sep 3, 2021, at 04:20, Brin Zhang(张百林) wrote: > > Hi Ildiko, > > Currently Cyborg and nova support interaction, we are not very familiar with zun yet, if possible, please provide some relevant information. You can refer to [1] for Cyborg related materials, and [2] for the interactive content of nova and cyborg (not yet merged). > > [1]https://docs.openstack.org/cyborg/latest/user/architecture.html > [2]https://review.opendev.org/c/openstack/cyborg/+/794549 > https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/nova-cyborg-interaction.html > > > Bailin Zhang (brinzhang) > > > -----邮件原件----- > 发件人: Ildiko Vancsa [mailto:ildiko.vancsa at gmail.com] > 发送时间: 2021年8月31日 22:39 > 收件人: openstack-discuss > 主题: [lists.openstack.org代发][zun][cyborg][kuryr] Re: Edge for scientific research session on September 13 Edge WG meeting > > Hi, > > It is a friendly reminder that the OpenInfra Edge Computing Group is having a joint session with the Chameleon project. > > During this session you can learn about the Chameleon project and how they are using OpenStack to build an edge platform for scientific research and academic purposes. They have also found some gaps during their journey. To mention an example the integration between Cyborg and Zun is missing to be able to use acceleration devices with containers orchestrated by OpenStack and they have been working to fill in this gap. > > The session will start with a presentation about the project and then we will use the remaining time to talk about solutions and next steps. I would like to invite everyone who is interested in learning about how OpenStack can be turned into an edge platform and what are the missing pieces of the puzzle that still need to be enhanced or implemented. > > The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. > The meeting will be held on Zoom, for the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings > > Please let me know if you need a calendar invite and I can send you one to get reminded about the meeting. > > Thanks and Best Regards, > Ildikó > > >> On Aug 25, 2021, at 17:00, Ildiko Vancsa wrote: >> >> Hi, >> >> I’m very excited to invite you to a session with the Chameleon project to discuss how edge computing is powering education, scientific research and more! As always in the area of edge there are challenges and gaps that we need to solve together. In that sense the session will start with a presentation that we will then turn into a discussion to turn towards discussing solutions. >> >> Please also note that the Chameleon project is using OpenStack to build edge solutions. As they identified some gaps they would like to work with the community to address the challenges that they’ve been hitting. Please join us to get their feedback and find the best way to improve OpenStack to deliver a platform that can be used on the edge! >> >> The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. >> For the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings >> >> Please find the details of the session below. >> >> Thanks and Best Regards, >> Ildikó >> >> >> Title: Chameleon: a cloud to edge infrastructure for science >> >> Abstract: The NSF-funded Chameleon project (www.chameleoncloud.org) provides an OpenStack-based bare metal reconfigurable cloud that supports systems research, education, and emergent application projects in science. The facility has been in existence for 6+ years, and has served ~6,000 users working on a variety of projects ranging from networking, operating systems design, and security, to innovative applications in astronomy, physics, and life sciences. Recent research trends in IoT and edge computing, as evidenced by increased numbers of project applications in those areas, made it clear that Chameleon needed to expand into the edge space. Remaining within the OpenStack framework allowed us to leverage end-user features that we developed for Chameleon, such as access via federated identity and Jupyter integration. Accordingly, we developed infrastructure called CHI at Edge that adapts Zun to provision containers on edge devices and significantly expands its capabilities to work in the edge space. The talk will describe our use case and the emergent product definition, the OpenStack adaptations we made to the current preview implementation, the directions we plan to take in our further work, as well as the ecosystem of projects and NSF infrastructure deployments leveraging this work and collaborating with us. >> >> > > From joshua.hesketh at gmail.com Wed Sep 8 04:39:38 2021 From: joshua.hesketh at gmail.com (Joshua Hesketh) Date: Wed, 8 Sep 2021 14:39:38 +1000 Subject: Fwd: linux.conf.au 2022 - Call for Sessions now open! In-Reply-To: References: Message-ID: Hello all, Firstly sorry if this isn't the right list for this sort of content. While I've continued to lurk in the OpenStack community I haven't been engaged in a number of years sadly (and missing ya'll!). Nonetheless, linux.conf.au 2022 is coming soon as an online event. This is your unique opportunity to speak at a conference that would otherwise require some 24+ hours of travel for a lot of folk. The Call For Proposals is currently open until Sunday the 12th (it was extended) and it would be great to see some open infrastructure talks (Ansible, OpenStack, Kubernetes, zuul, Airship, etc etc). Read on for more information or let me know if you have any questions. Regardless if you submit a session it is also a great opportunity to attend a world class open source conference remotely! Cheers, Josh -------- Forwarded Message -------- Subject: linux.conf.au 2022 - Call for Sessions now open! Date: Tue, 10 Aug 2021 08:07:01 +1000 From: linux.conf.au Announcements Reply-To: lca-announce at lists.linux.org.au To: LCA Announce , Linux Aus The linux.conf.au 2022 Call for Sessions is now open. It will stay open until 11:59pm 5 September 2021 Anywhere on Earth (AoE). The theme for 2022 is 'community'. After the challenges of the past year, our community has explored ways to rebuild the connections we were used to having face-to-face. Many Open Source communities already had their roots online, so how can this be applied to other areas, and how can we keep people interested as they shift to living even more of their lives online? How can we keep in contact with connections in other countries in a virtual way? If you have ideas or developments you'd like to share with the open source community at linux.conf.au, we'd love to hear from you. ## Call for Sessions The main conference runs on Saturday 15 and Sunday 16 January, with multiple streams catering for a wide range of interest areas. We invite you to submit a session proposal on a topic you are familiar with via our proposals portal at https://lca2022.linux.org.au/programme/proposals/. Talks are 45 minute presentations on a single topic presented in lecture format. Each accepted talk will receive one ticket to attend the conference. ## Call for Miniconfs We are pleased to announce we will again have four Miniconfs on the first day, Friday 14 January 2022. These are: * GO GLAM meets Community * Kernel * Open Hardware * System Administration Based on feedback over a few years, we will be introducing two major changes for Miniconfs in 2022: all presentations will be 30 minutes long, and each accepted presentation will receive one ticket to the conference. The Call for Miniconf Sessions is now open on our website, so we encourage you to submit your proposal today. Check out our Miniconfs page at https://lca2022.linux.org.au/programme/miniconfs/ for more information. ## No need to book flights or hotels Don't forget: the conference will be an online, virtual experience. This means our speakers will be beaming in from their own homes or workplaces. The organising team will be able to help speakers with their tech set-up. Each accepted presenter will have a tech check prior to the event to smooth out any difficulties and there will be an option to pre-record presentations. ## Introducing LCA Local We know many of you have missed the experience of a face-to-face conference and in 2022 we are launching LCA Local. While our conference will be online, we are inviting people to join others in their local area and participate in the conference together. More information and an expression of interest form for LCA Local will be released soon. ## Have you got an idea? You can find out how to submit your session proposals at https://lca2022.linux.org.au/programme/proposals/. If you have any other questions, you can contact us via email at contact at lca2022.linux.org.au. The session selection committee is looking forward to reading your submissions. We would also like to thank them for coming together and volunteering their time to help put this conference together. linux.conf.au 2022 Organising Team ---- Read this online at https://lca2022.linux.org.au/news/call-for-sessions/ _______________________________________________ lca-announce mailing list lca-announce at lists.linux.org.au http://lists.linux.org.au/mailman/listinfo/lca-announce -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdhasman at redhat.com Wed Sep 8 05:05:48 2021 From: rdhasman at redhat.com (Rajat Dhasmana) Date: Wed, 8 Sep 2021 10:35:48 +0530 Subject: [sdk]: Block storage service update volume status from reserved to available In-Reply-To: <11f602803604a655e1be5d8cc05383b4@uvic.ca> References: <11f602803604a655e1be5d8cc05383b4@uvic.ca> Message-ID: Hi, On Wed, Sep 8, 2021 at 1:02 AM dmeng wrote: > Hello there, > > Hope everything is going well. > > I would like to know if there is any method that could update > the volume status or reset its state. While we creating volumes use the > block storage service or creating vms with volumes use the compute service, > sometimes we got volumes stuck in creating/deleteing/attaching/reserved > state. And we couldn't remove them or reset their state without an admin > account even we own the project. The way we create volumes and boot vms is > like the following: > > # create the volume > cv = cinder.create_volume(name=bv_name, size=size, image_id=imageobj.id, volume_type=volume_type, is_bootable=True) > # format mapping data > bdm = [{'uuid': cv.id, 'source_type': 'volume', 'destination_type': 'volume', 'delete_on_termination': True, 'boot_index': 0}] > # create the vm > nova.create_server(name=hostname, image_id=imageobj.id, flavor_id=flavorl.id, key_name=key_name, block_device_mapping=bdm, user_data=format_userdata, networks=netid, security_groups=sec_groups) > > In the normal situation, everything works. But in the case of any network issue, the volume might stuck in the process of creating, deleting, or attaching. And if the volume got created successfully, when it tries to create the vm, the volume will be reserved for it, but if anything wrong happen when creating the vm or get timed out, the volume will be in the reserved stage forever. We would like to know if we could update the state or delete them without the admin account? > > We tried the command line tools 'openstack volume set --state available ' and 'cinder reset-state --state available ', but both got: > Policy doesn't allow volume_extension:volume_admin_actions:reset_status to be performed. (HTTP 403) (Request-ID: req-b50236c3-5662-4cc0-8602-19e79e80219d) > ERROR: Unable to reset the state for the specified entity(s). > > For the specific case where a volume is stuck in reserved state and the above code uses cinder's new attachments API, you can use* ``cinder --os-volume-api-version 3.27 attachment-list`` *to list all attachments for volumes and if there exists attachment(s) for your volume, you can delete them via the command *``cinder --os-volume-api-version 3.27 attachment-delete ``* > We are using the chameleon openstack cloud but we are not the admin user there, so wondering if there is any method that we could update the volume in our own project? > > Thanks and have a great day! > Catherine > > Thanks and regards Rajat Dhasmana -------------- next part -------------- An HTML attachment was scrubbed... URL: From zigo at debian.org Wed Sep 8 08:27:50 2021 From: zigo at debian.org (Thomas Goirand) Date: Wed, 8 Sep 2021 10:27:50 +0200 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: <20210907232144.n25ei2cyrmkr2xaz@yuggoth.org> References: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> <20210907232144.n25ei2cyrmkr2xaz@yuggoth.org> Message-ID: <1aaf9274-f669-1e7b-f2f2-4cf795f60f0c@debian.org> On 9/8/21 1:21 AM, Jeremy Stanley wrote: > On 2021-09-08 00:46:30 +0200 (+0200), Thomas Goirand wrote: > [...] >> In fact, I was particularly confuse to realize that something like >> taskflow was declared release-independent since Wallaby. Now, if >> I'm told that it is loosely coupled with OpenStack, knowing that >> it's a key component, I'm not sure what I should do. Should I get >> taskflow updated to 4.6.2 in Wallaby, so I get bugfixes? Or is it >> safer to just upgrade it for Xena? >> >> I took Taskflow as an example (you can reply to me about Taskflow >> in particular, but that's not the point I'm trying to make). But >> it could have been something else (deptcollector? oslo.i18n?). >> What I'm trying to say is that as a downstream consumer, it's from >> hard to impossible to know what to do for the past OpenStack >> releases regarding these release-independent components: should I >> update them or not? It is unrealistic to hope that one can track >> what component has been updated to a newer version in the stable >> release CI. > > Specific examples are great actually, because it at least lets us > try to formulate a specific answer and then reason about whether a > similar answer can be generally applied to other related scenarios. > So let's take taskflow in Wallaby and Xena: > > The closest thing we have to a recommendation for versions of > "outside" dependencies (which is essentially what we treat > release-independent deliverables as) is the upper-constraints.txt > file on each branch of the openstack/requirements repository. In > the case of taskflow, at the moment the stable/wallaby branch upper > constraint for taskflow is 4.6.0, which is the version we indicate > all OpenStack projects should test their stable/wallaby changes > against. That entry was last updated by > https://review.opendev.org/773620 which merged to the master branch > of openstack/requirements in February, well before the coordinated > Wallaby release. Ok, so your recommendation is that, for each and every component that is marked release-independent, I should be checking the global requirements. I just wrote that this is unrealistic that a human does such a check. If you didn't know, in Debian, Michal Arbet (who co-maintains OpenStack with me together in Debian) wrote this: https://salsa.debian.org/openstack-team/debian/os-version-checker This VersionStatus.py checks for what's in https://releases.openstack.org/, for example in https://releases.openstack.org/xena/index.html for the Xena upcoming release. The result output of the os-version-checker is this page: http://osbpo.debian.net/deb-status/ Thanks to this page, anyone can know the status of OpenStack packaging on all the different branches we maintain. If you look at it right now, you'll notice that we're up-to-date, as we packaged all non-clients and all clients library, but we didn't package new versions of the puppet manifests yet (I'm waiting for the RCs to do that). If you click on the buster-victoria button, and look for Taskflow, you'll see that it's up-to-date. If you click on bullseye-xena, and search for Taskflow, you will ... find nothing! That's because taskflow doesn't appear in https://releases.openstack.org/xena/index.html since it is release-independent. As a result, I'm left with no other choice than being very careful and manually check the global-requirements. Every check that is supposed to be done by a human isn't reliable. It'd be nice if I had a way to automatically check for this. The thing is, the global-requirements is full of information, some of it not relevant to my own use case, and it's hard to check automatically. First of all, because the Python module names as in PyPi, do not necessarily match the package names. Also, because some artifacts are not relevant to Debian. For example, I have no use of many TripleO stuff, and some services are simply not packaged in Debian. In an ideal world, we'd have an API to check for all of this, rather than digging through the release HTML pages. > It seems like what you're asking for is this: > > https://opendev.org/openstack/requirements/src/branch/stable/wallaby/upper-constraints.txt > > Or rather, that's the closest thing we have to what I think you're > requesting. If you follow the commit history of that file, you'll > see recent updates for it on the stable/wallaby branch do consist > almost entirely of stable point releases of cycle-oriented projects: > > https://opendev.org/openstack/requirements/commits/branch/stable/wallaby/upper-constraints.txt > > Does this help narrow the scope of concern to the point where we can > better reason about the problems or challenges you're facing? Unfortunately, while it is part of the answer, due to what I explained just above, it is very hard to exploit the available data without a real, long effort on our side to fix this. :/ One solution could probably be for *US* (downstream distros) to write some kind of API as I just wrote, and make it available for everyone on releases.openstack.org. Are Red Hat people also interested by such an effort? Any pointer where we should attempt to propose patches? Cheers, Thomas Goirand (zigo) From hberaud at redhat.com Wed Sep 8 09:31:31 2021 From: hberaud at redhat.com (Herve Beraud) Date: Wed, 8 Sep 2021 11:31:31 +0200 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: <1aaf9274-f669-1e7b-f2f2-4cf795f60f0c@debian.org> References: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> <20210907232144.n25ei2cyrmkr2xaz@yuggoth.org> <1aaf9274-f669-1e7b-f2f2-4cf795f60f0c@debian.org> Message-ID: Le mer. 8 sept. 2021 à 10:29, Thomas Goirand a écrit : > On 9/8/21 1:21 AM, Jeremy Stanley wrote: > > On 2021-09-08 00:46:30 +0200 (+0200), Thomas Goirand wrote: > > [...] > >> In fact, I was particularly confuse to realize that something like > >> taskflow was declared release-independent since Wallaby. Now, if > >> I'm told that it is loosely coupled with OpenStack, knowing that > >> it's a key component, I'm not sure what I should do. Should I get > >> taskflow updated to 4.6.2 in Wallaby, so I get bugfixes? Or is it > >> safer to just upgrade it for Xena? > >> > >> I took Taskflow as an example (you can reply to me about Taskflow > >> in particular, but that's not the point I'm trying to make). But > >> it could have been something else (deptcollector? oslo.i18n?). > >> What I'm trying to say is that as a downstream consumer, it's from > >> hard to impossible to know what to do for the past OpenStack > >> releases regarding these release-independent components: should I > >> update them or not? It is unrealistic to hope that one can track > >> what component has been updated to a newer version in the stable > >> release CI. > > > > Specific examples are great actually, because it at least lets us > > try to formulate a specific answer and then reason about whether a > > similar answer can be generally applied to other related scenarios. > > So let's take taskflow in Wallaby and Xena: > > > > The closest thing we have to a recommendation for versions of > > "outside" dependencies (which is essentially what we treat > > release-independent deliverables as) is the upper-constraints.txt > > file on each branch of the openstack/requirements repository. In > > the case of taskflow, at the moment the stable/wallaby branch upper > > constraint for taskflow is 4.6.0, which is the version we indicate > > all OpenStack projects should test their stable/wallaby changes > > against. That entry was last updated by > > https://review.opendev.org/773620 which merged to the master branch > > of openstack/requirements in February, well before the coordinated > > Wallaby release. > > Ok, so your recommendation is that, for each and every component that is > marked release-independent, I should be checking the global > requirements. I just wrote that this is unrealistic that a human does > such a check. > > If you didn't know, in Debian, Michal Arbet (who co-maintains OpenStack > with me together in Debian) wrote this: > > https://salsa.debian.org/openstack-team/debian/os-version-checker > > This VersionStatus.py checks for what's in > https://releases.openstack.org/, for example in > https://releases.openstack.org/xena/index.html for the Xena upcoming > release. > > The result output of the os-version-checker is this page: > > http://osbpo.debian.net/deb-status/ > > Thanks to this page, anyone can know the status of OpenStack packaging > on all the different branches we maintain. If you look at it right now, > you'll notice that we're up-to-date, as we packaged all non-clients and > all clients library, but we didn't package new versions of the puppet > manifests yet (I'm waiting for the RCs to do that). > > If you click on the buster-victoria button, and look for Taskflow, > you'll see that it's up-to-date. If you click on bullseye-xena, and > search for Taskflow, you will ... find nothing! That's because taskflow > doesn't appear in https://releases.openstack.org/xena/index.html since > it is release-independent. As a result, I'm left with no other choice > than being very careful and manually check the global-requirements. > > Every check that is supposed to be done by a human isn't reliable. It'd > be nice if I had a way to automatically check for this. The thing is, > the global-requirements is full of information, some of it not relevant > to my own use case, and it's hard to check automatically. First of all, > because the Python module names as in PyPi, do not necessarily match the > package names. Also, because some artifacts are not relevant to Debian. > For example, I have no use of many TripleO stuff, and some services are > simply not packaged in Debian. > > In an ideal world, we'd have an API to check for all of this, rather > than digging through the release HTML pages. > > > It seems like what you're asking for is this: > > > > > https://opendev.org/openstack/requirements/src/branch/stable/wallaby/upper-constraints.txt > > > > Or rather, that's the closest thing we have to what I think you're > > requesting. If you follow the commit history of that file, you'll > > see recent updates for it on the stable/wallaby branch do consist > > almost entirely of stable point releases of cycle-oriented projects: > > > > > https://opendev.org/openstack/requirements/commits/branch/stable/wallaby/upper-constraints.txt > > > > Does this help narrow the scope of concern to the point where we can > > better reason about the problems or challenges you're facing? > > Unfortunately, while it is part of the answer, due to what I explained > just above, it is very hard to exploit the available data without a > real, long effort on our side to fix this. :/ > > One solution could probably be for *US* (downstream distros) to write > some kind of API as I just wrote, and make it available for everyone on > releases.openstack.org. Are Red Hat people also interested by such an > effort? Any pointer where we should attempt to propose patches? > An API seems quite a good idea I'm ok to give help on this point with my upstream cap and my Red Hat cap. As release manager I added your proposition to our "things to changes" on our release process and tools. I think that it should be discussed first to see how to implement it and how to host it. I think that the technical details would be a transversal decision between the infra team and the release team and the volunteers on this topic. However you should notice that we are near Xena's deadline, and, so, for now, I don't think that we will have the time to work on this topic. Also, although I'm volunteering to help, you should notice that the release team will have another PTL for Y, so, the lead of this topic is at the discretion of my replacant. I think a good starting point could be to design some specs or propose some related goals, so, Thomas, maybe you could start by creating something like that with your expectations. It will allow us to start this topic, share our different feedback and our expectations on it (our different viewpoints, at least from the release team, infra team, and downstream/distros teams). Hopefully it will help us. [1] https://etherpad.opendev.org/p/xena-relmgt-tracking [2] https://governance.openstack.org/tc/goals/ > Cheers, > > Thomas Goirand (zigo) > > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From lyarwood at redhat.com Wed Sep 8 09:56:45 2021 From: lyarwood at redhat.com (Lee Yarwood) Date: Wed, 8 Sep 2021 10:56:45 +0100 Subject: Error adding a Plain Volume created through GUI to Instance In-Reply-To: <20210906094103.sn4bn6qb4sab3p2m@localhost> References: <20210906094103.sn4bn6qb4sab3p2m@localhost> Message-ID: <20210908095645.droxjr6bmlof4tfd@lyarwood-laptop.usersys.redhat.com> On 06-09-21 11:41:03, Gorka Eguileor wrote: > On 06/09, KK CHN wrote: > > ExecutionError: Unexpected error while running command. > > Command: rbd map volume-4dcb78d5-e0ce-4bfc-95ae-36a4417c35d8 --pool volumes > > --id volumes --mon_host 10.236.223.131:6789 --mon_host 10.236.223.132:6789 > > Exit code: 6 > > Stdout: 'RBD image feature set mismatch. You can disable features > > unsupported by the kernel with "rbd feature disable > > volumes/volume-4dcb78d5-e0ce-4bfc-95ae-36a4417c35d8 journaling".\nIn some > > cases useful info is found in syslog - try "dmesg | tail".\n' > > Stderr: 'rbd: sysfs write failed\nrbd: map failed: (6) No such device or > > address\n' > > > > I am trying to add a 20 GB plain volume created through Horizon GUI to an > > existing Windows 2012 R VM running with VirtIO drivers installed. But this > > volume adding results in the error as shown above, > > > > trace tells to fix it by rbd feature disable .. what is the > > permanent fix for this issue ? and why it happens ? > > > > ( OpenStack Ussuri, Glance, Ceph , qemu-kvm) > > > > root at ctrl1:/home/cloud# ceph --version > > ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus > > (stable) > > root at ctrl1:/home/cloud# > > Hi, > > Usually RBD volumes are directly attached on the hypervisor, but there > was an encryption performance issue [1] so nova and os-brick added > support to attaching these volumes to the host and then use them on the > VM. > > Attaching volumes on the host uses the krbd module, whereas attaching > them to the hypervisor directly uses librbd, with the first one lagging > in terms of Ceph features. > > The issue you are facing is that the images in RBD are being created > with features that are not supported by krbd. > > Fixes are: > > - Disable those features in your ceph configuration file (usually > /etc/ceph/ceph.conf) > > Under [global] section add: > rbd default features = 3 > > - Change "[workarounds]rbd_volume_local_attach" to False in Nova's > config. > > Cheers, > Gorka. > > [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1762765 Yup thanks Gorka and just for awareness this workaround and the associated disable_native_luksv1 workaround have been removed in the Xena release as the underlying performance issues have been resolved AFAIK: workarounds: Remove rbd_volume_local_attach https://review.opendev.org/c/openstack/nova/+/805648 workarounds: Remove disable_native_luksv1 https://review.opendev.org/c/openstack/nova/+/805647/ Cheers, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From chkumar at redhat.com Wed Sep 8 10:00:14 2021 From: chkumar at redhat.com (Chandan Kumar) Date: Wed, 8 Sep 2021 15:30:14 +0530 Subject: [patrole][qa] Proposing Soniya Vyas to Patrole core In-Reply-To: <17bc1bcebec.d7cc6895149770.7494147722877953142@ghanshyammann.com> References: <17bc1bcebec.d7cc6895149770.7494147722877953142@ghanshyammann.com> Message-ID: On Wed, Sep 8, 2021 at 1:03 AM Ghanshyam Mann wrote: > > ---- On Tue, 07 Sep 2021 14:15:04 -0500 Martin Kopec wrote ---- > > Hi all, > > > > Soniya Vyas (IRC: soniya29) is very enthusiastic about Patrole project and since she has been very helpful, I'd like to propose her for Patrole core. > > > > You can vote/feedback on this email. If no objections within a week, I'll add her to the core list. > > +1, she has been very helpful in Patrole project. +1, Thank you Sonia for all the great work. Thanks, Chandan Kumar From lyarwood at redhat.com Wed Sep 8 10:57:00 2021 From: lyarwood at redhat.com (Lee Yarwood) Date: Wed, 8 Sep 2021 11:57:00 +0100 Subject: [sdk]: Block storage service update volume status from reserved to available In-Reply-To: References: <11f602803604a655e1be5d8cc05383b4@uvic.ca> Message-ID: <20210908105700.qfw43cd2o757rpv3@lyarwood-laptop.usersys.redhat.com> On 08-09-21 10:35:48, Rajat Dhasmana wrote: > Hi, > > On Wed, Sep 8, 2021 at 1:02 AM dmeng wrote: > > > Hello there, > > > > Hope everything is going well. > > > > I would like to know if there is any method that could update > > the volume status or reset its state. While we creating volumes use the > > block storage service or creating vms with volumes use the compute service, > > sometimes we got volumes stuck in creating/deleteing/attaching/reserved > > state. And we couldn't remove them or reset their state without an admin > > account even we own the project. The way we create volumes and boot vms is > > like the following: > > > > # create the volume > > cv = cinder.create_volume(name=bv_name, size=size, image_id=imageobj.id, volume_type=volume_type, is_bootable=True) > > # format mapping data > > bdm = [{'uuid': cv.id, 'source_type': 'volume', 'destination_type': 'volume', 'delete_on_termination': True, 'boot_index': 0}] > > # create the vm > > nova.create_server(name=hostname, image_id=imageobj.id, flavor_id=flavorl.id, key_name=key_name, block_device_mapping=bdm, user_data=format_userdata, networks=netid, security_groups=sec_groups) > > > > In the normal situation, everything works. But in the case of any > > network issue, the volume might stuck in the process of creating, > > deleting, or attaching. And if the volume got created successfully, > > when it tries to create the vm, the volume will be reserved for it, > > but if anything wrong happen when creating the vm or get timed out, > > the volume will be in the reserved stage forever. We would like to > > know if we could update the state or delete them without the admin > > account? You need to delete the ERROR'd out instance, this will delete the volume attachment in Cinder moving the volume to available again. Nova does this to ensure another instance doesn't take this volume as the user might request that the instance is rebuilt to workaround whatever issue they encountered during the initial build. > > We tried the command line tools 'openstack volume set --state > > available ' and 'cinder reset-state --state available > > ', but both got: Policy doesn't allow > > volume_extension:volume_admin_actions:reset_status to be performed. > > (HTTP 403) (Request-ID: req-b50236c3-5662-4cc0-8602-19e79e80219d) > > ERROR: Unable to reset the state for the specified entity(s). > > > > For the specific case where a volume is stuck in reserved state and the > above code uses cinder's new attachments API, you can use* ``cinder > --os-volume-api-version 3.27 attachment-list`` *to list all attachments for > volumes and if there exists attachment(s) for your volume, you can delete > them via the command *``cinder --os-volume-api-version 3.27 > attachment-delete ``* As above we retain the volume attachment on build failures so either rebuild the instance or delete it to free up the volume again. Cheers, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From eblock at nde.ag Wed Sep 8 11:02:25 2021 From: eblock at nde.ag (Eugen Block) Date: Wed, 08 Sep 2021 11:02:25 +0000 Subject: [ceph-users] Re: Replacing swift with RGW In-Reply-To: References: <20210901122221.Horde.v0FeoCUfex4WYSaXReGzV1i@webmail.nde.ag> <20210901134023.Horde.ZhZ0JBty-nYNsBfGe6WOijH@webmail.nde.ag> <20210902071439.Horde.4cayqDV75oyYS75eyQZMSPD@webmail.nde.ag> <20210902081741.Horde.Xp06qPUkci05sKXFlo1F5ap@webmail.nde.ag> <20210902083925.Horde.a0LD9j43RkGPIl7UuWCRtyu@webmail.nde.ag> <20210906144401.Horde.7j4UyxpB4wl0f5RgwX1Y0FD@webmail.nde.ag> Message-ID: <20210908110225.Horde.VCbq8x0RD-cMYvSMHmMvJOr@webmail.nde.ag> Great that you got it working. The dashboard logs should tell you want went wrong, e.g. in /var/log/apache2/openstack-dashboard-error_log. It's probably some role missing for your user, but you'll find details in the logs. Zitat von Michel Niyoyita : > Hello team, > > Now I am able to create containers and upload objects in the openstack CLI > side using the following commands: > (kolla-open1) stack at kolla-open1:~$ swift -V 1.0 -A > http://ceph-mon2:8080/auth/v1 -U operator:swift -K opswift post > my-container3 > (kolla-open1) stack at kolla-open1:~$ swift -V 1.0 -A > http://ceph-mon2:8080/auth/v1 -U operator:swift -K opswift upload > my-container3 michel.txt > > above commands are successful and containers and their objects are stored > in ceph RGW default pool which replace swift . > > The issues I remain with is that once I try to access the object store on > horizon and click on containers it immediately disconnect hoizon to the > login status. > [image: image.png] > [image: image.png] > > Kindly help and advise > > Michel > > > > On Mon, Sep 6, 2021 at 4:50 PM Eugen Block wrote: > >> It's hard to tell what is wrong with your setup, I don't really use >> mine but this was the last working config I had to be able to create >> swift containers directly in RGW: >> >> ---snip--- >> # ceph.conf: >> >> [client.rgw.ses6-mon1] >> rgw frontends = "beast port=80" >> rgw dns name = ses6-mon1.example.com >> rgw enable usage log = true >> >> rgw thread pool size = 512 >> rgw keystone api version = 3 >> rgw keystone url = http://control-node.example.com:5000 >> >> rgw keystone admin user = rgw >> rgw keystone admin password = **** >> rgw keystone admin domain = default >> rgw keystone admin project = service >> rgw keystone accepted roles = admin,Member,_member_,member >> rgw keystone verify ssl = false >> rgw s3 auth use keystone = true >> rgw keystone revocation interval = 0 >> >> >> # User role (I don't think admin is required) >> >> openstack role add --user rgw --project 9e8a67da237a4b26afb2819d2dea2219 >> admin >> >> >> # Create keystone endpoints >> >> openstack endpoint create --region RegionOne swift admin >> "http://ses6-mon1.example.com:80/swift/v1" >> openstack endpoint create --region RegionOne swift internal >> "http://ses6-mon1.example.com:80/swift/v1" >> openstack endpoint create --region RegionOne swift public >> "http://ses6-mon1.example.com:80/swift/v1" >> >> >> # Create container and files >> >> openstack container create swift1 >> +---------+-----------+---------------------------------------------------+ >> | account | container | x-trans-id | >> +---------+-----------+---------------------------------------------------+ >> | v1 | swift1 | tx000000000000000000001-0060b4ba48-d724dc-default | >> +---------+-----------+---------------------------------------------------+ >> >> openstack object create --name file1 swift1 chef-client.log >> +--------+-----------+----------------------------------+ >> | object | container | etag | >> +--------+-----------+----------------------------------+ >> | file1 | swift1 | 56a1ed3b201c1e753bcbe80c640349f7 | >> +--------+-----------+----------------------------------+ >> ---snip--- >> >> >> You are mixing dns names and IP addresses, I can't tell if that's a >> problem but it probably should work, I'm not sure. Compared to my >> ceph.conf these are the major differences: >> >> rgw keystone verify ssl = false >> rgw s3 auth use keystone = true >> rgw keystone revocation interval = 0 >> >> And I don't use rgw_keystone_token_cache_size. Maybe try again with >> the options I use. >> >> >> Zitat von Michel Niyoyita : >> >> > Hello, >> > >> > I am trying to replace swift by RGW as backend storage but I failed once >> I >> > try to post a container in the OpenStack side however, all interfaces are >> > configured (admin, public and internal). but Once I post from RGW host it >> > is created . Another issue is that object storage does not appear on the >> > horizon dashboard . I have deployed openstack all-in-one using >> > kolla-ansible and Os is ubuntu >> > >> > (kolla-open1) stack at kolla-open1:~$ swift -v post myswift >> > Container POST failed: http://ceph-osd3:8080/swift/v1/myswift 401 >> > Unauthorized b'AccessDenied' >> > Failed Transaction ID: tx000000000000000000008-006135dcbd-87d63-default >> > >> > (kolla-open1) stack at kolla-open1:~$ swift list >> > Account GET failed: http://ceph-osd3:8080/swift/v1?format=json 401 >> > Unauthorized [first 60 chars of response] >> > b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000c-' >> > Failed Transaction ID: tx00000000000000000000c-006135de42-87d63-default >> > >> > Kindly help to solve the issue >> > >> > Michel >> > >> > On Thu, Sep 2, 2021 at 4:28 PM Alex Schultz wrote: >> > >> >> The swift docs are a bit out of date as they still reference python2 >> >> despite python3 being supported for some time now. Replace python- with >> >> python3- and try again. >> >> >> >> >> >> On Thu, Sep 2, 2021 at 7:35 AM Michel Niyoyita >> wrote: >> >> >> >>> >> >>> >> >>> ---------- Forwarded message --------- >> >>> From: Michel Niyoyita >> >>> Date: Thu, Sep 2, 2021 at 12:17 PM >> >>> Subject: Fwd: [ceph-users] Re: Replacing swift with RGW >> >>> To: >> >>> >> >>> >> >>> >> >>> >> >>> ---------- Forwarded message --------- >> >>> From: Eugen Block >> >>> Date: Thu, Sep 2, 2021 at 10:39 AM >> >>> Subject: Re: [ceph-users] Re: Replacing swift with RGW >> >>> To: Michel Niyoyita >> >>> >> >>> >> >>> You should continue this thread on the openstack-discuss mailing list >> >>> (openstack-discuss at lists.openstack.org) since this is not a ceph >> issue. >> >>> I'm not familiar with kolla and I don't know the requirements to >> >>> install openstack-swift, you'll need to ask the openstack community. >> >>> >> >>> >> >>> Zitat von Michel Niyoyita : >> >>> >> >>> > Below are errors I am getting once I try to run swift commands . the >> >>> second >> >>> > one is the error I get once try to install python-swiftclient >> >>> > >> >>> > (kolla-open) [stack at kolla-open ~]$ swift -v stat >> >>> > -bash: swift: command not found >> >>> > (kolla-open) [stack at kolla-open ~]$ sudo yum -y install >> >>> python-swiftclient >> >>> > Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 >> 09:21:53 >> >>> AM >> >>> > CAT. >> >>> > No match for argument: python-swiftclient >> >>> > Error: Unable to find a match: python-swiftclient >> >>> > (kolla-open) [stack at kolla-open ~]$ >> >>> > >> >>> > Waiting for your help >> >>> > >> >>> > On Thu, Sep 2, 2021 at 10:17 AM Eugen Block wrote: >> >>> > >> >>> >> I can't tell for sure, but yes, I believe you need the >> openstack-swift >> >>> >> package (with dependencies). What errors do you get? The more >> >>> >> information you share the better people can help. >> >>> >> >> >>> >> >> >>> >> Zitat von Michel Niyoyita : >> >>> >> >> >>> >> > I tried to install "sudo yum -y install python-swiftclient" on >> >>> openstack >> >>> >> > side but fails . are there openastack-shwift packages which are >> >>> needed? >> >>> >> > if are there please help me to get . may be also it is the cause >> I >> >>> am >> >>> >> > failing to run swift command on openstack cli side. >> >>> >> > >> >>> >> > thank you for your continued support. >> >>> >> > >> >>> >> > Micheal >> >>> >> > >> >>> >> > On Thu, Sep 2, 2021 at 9:14 AM Eugen Block wrote: >> >>> >> > >> >>> >> >> I only configured the endpoints for the clients to directly >> access >> >>> the >> >>> >> >> RGWs, but you'll probably need to install the openstack-swift >> >>> package. >> >>> >> >> Or have you done that already? >> >>> >> >> >> >>> >> >> >> >>> >> >> Zitat von Michel Niyoyita : >> >>> >> >> >> >>> >> >> > Thank you Eugen for your prompt response. >> >>> >> >> > >> >>> >> >> > Now the commands provided work. but I am not finding the object >> >>> >> storage >> >>> >> >> on >> >>> >> >> > the horizon dashboard , but it appears in the system >> information >> >>> >> >> services. >> >>> >> >> > [image: image.png] >> >>> >> >> > so my question is how to configure it in order that it can >> appear >> >>> in >> >>> >> the >> >>> >> >> > dashboard . >> >>> >> >> > >> >>> >> >> > Michel >> >>> >> >> > >> >>> >> >> > On Wed, Sep 1, 2021 at 3:49 PM Eugen Block >> wrote: >> >>> >> >> > >> >>> >> >> >> Sorry, one little detail slipped through, the '--region' flag >> >>> has to >> >>> >> >> >> be put before the 'service' name. The correct command would >> be: >> >>> >> >> >> >> >>> >> >> >> openstack endpoint create --region RegionOne swift admin >> >>> >> >> >> http://ceph-osd3:8080/swift/v1 >> >>> >> >> >> >> >>> >> >> >> and respectively for the other interfaces. >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> Zitat von Eugen Block : >> >>> >> >> >> >> >>> >> >> >> > Hi, >> >>> >> >> >> > >> >>> >> >> >> > this is not a ceph issue but your openstack cli command as >> the >> >>> >> error >> >>> >> >> >> > message states. >> >>> >> >> >> > >> >>> >> >> >> > Try one interface at a time: >> >>> >> >> >> > >> >>> >> >> >> > openstack endpoint create swift public >> >>> >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >> >>> >> >> >> > openstack endpoint create swift admin >> >>> >> http://ceph-osd3:8080/swift/v1 >> >>> >> >> >> > --region RegionOne swift >> >>> >> >> >> > openstack endpoint create swift internal >> >>> >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift >> >>> >> >> >> > >> >>> >> >> >> > >> >>> >> >> >> > Zitat von Michel Niyoyita : >> >>> >> >> >> > >> >>> >> >> >> >> Hello , >> >>> >> >> >> >> >> >>> >> >> >> >> Below are errors I am getting once I am trying to integrate >> >>> swift >> >>> >> >> with >> >>> >> >> >> >> Radosgateway. >> >>> >> >> >> >> >> >>> >> >> >> >> From openstack side once i try to endpoint which will point >> >>> to the >> >>> >> >> >> >> radosgateway : >> >>> >> >> >> >> >> >>> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ openstack endpoint >> >>> create >> >>> >> >> >> >> --publicurl http://ceph-osd3:8080/swift/v1 --adminurl >> >>> >> >> >> >> http://ceph-osd3:8080/swift/v1 --internal >> >>> >> >> >> http://ceph-osd3:8080/swift/v1 >> >>> >> >> >> >> --region RegionOne swift >> >>> >> >> >> >> usage: openstack endpoint create [-h] [-f >> >>> >> >> {json,shell,table,value,yaml}] >> >>> >> >> >> >> [-c COLUMN] [--noindent] >> >>> [--prefix >> >>> >> >> >> PREFIX] >> >>> >> >> >> >> [--max-width ] >> >>> >> [--fit-width] >> >>> >> >> >> >> [--print-empty] [--region >> >>> >> >> ] >> >>> >> >> >> >> [--enable | --disable] >> >>> >> >> >> >> >> >>> >> >> >> >> openstack endpoint create: error: argument : >> >>> invalid >> >>> >> >> choice: >> >>> >> >> >> ' >> >>> >> >> >> >> http://ceph-osd3:8080/swift/v1' (choose from 'admin', >> >>> 'public', >> >>> >> >> >> 'internal') >> >>> >> >> >> >> >> >>> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ >> >>> >> >> >> >> >> >>> >> >> >> >> below are my /etc/ceph/ceph.conf file : >> >>> >> >> >> >> >> >>> >> >> >> >> [client.rgw.ceph-osd3] >> >>> >> >> >> >> rgw_dns_name = ceph-osd3 >> >>> >> >> >> >> host = ceph-osd3 >> >>> >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3/keyring >> >>> >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.log >> >>> >> >> >> >> rgw frontends = civetweb port=10.10.29.110:8080 >> >>> num_threads=100 >> >>> >> >> >> >> rgw_keystone_url=http://10.10.29.150:35357 >> >>> >> >> >> >> rgw_keystone_admin_user=admin >> >>> >> >> >> >> rgw_keystone_admin_password=admin >> >>> >> >> >> >> rgw_keystone_admin_tenant=admin >> >>> >> >> >> >> rgw_keystone_accepted_role=admin Member swiftoperator >> >>> >> >> >> >> rgw_keystone_token_cache_size=200 >> >>> >> >> >> >> rgw_keystone_revocation_interval=300 >> >>> >> >> >> >> >> >>> >> >> >> >> [client.rgw.ceph-osd3.rgw0] >> >>> >> >> >> >> host = ceph-osd3 >> >>> >> >> >> >> keyring = >> >>> /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring >> >>> >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log >> >>> >> >> >> >> rgw frontends = beast endpoint=10.10.29.110:8080 >> >>> >> >> >> >> rgw thread pool size = 512 >> >>> >> >> >> >> >> >>> >> >> >> >> please note that my rgw_dns_name = ceph_osd3 with >> >>> 10.10.29.110 as >> >>> >> IP >> >>> >> >> >> >> >> >>> >> >> >> >> and 10.10.29.150 all-in-one IP >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> Please crosscheck where I might make mistake and try to >> >>> correct. >> >>> >> >> >> >> >> >>> >> >> >> >> Best regards >> >>> >> >> >> >> >> >>> >> >> >> >> Michel >> >>> >> >> >> >> >> >>> >> >> >> >> On Mon, Aug 30, 2021 at 11:25 AM Etienne Menguy < >> >>> >> >> >> etienne.menguy at croit.io> >> >>> >> >> >> >> wrote: >> >>> >> >> >> >> >> >>> >> >> >> >>> Hi, >> >>> >> >> >> >>> >> >>> >> >> >> >>> There are some information on Ceph documentation >> >>> >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/ < >> >>> >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/> . >> >>> >> >> >> >>> - Use keystone as auth for RGW >> >>> >> >> >> >>> - Create service and register your RGW as swift >> >>> >> >> >> >>> >> >>> >> >> >> >>> Étienne >> >>> >> >> >> >>> >> >>> >> >> >> >>>> On 27 Aug 2021, at 15:47, Michel Niyoyita < >> >>> micou12 at gmail.com> >> >>> >> >> wrote: >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> Hello , >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> I have configured RGW in my ceph cluster deployed using >> ceph >> >>> >> >> ansible >> >>> >> >> >> and >> >>> >> >> >> >>>> create sub user to access the created containers and >> would >> >>> like >> >>> >> to >> >>> >> >> >> >>> replace >> >>> >> >> >> >>>> swift by RGW in the openstack side. Anyone can help on >> >>> >> >> configuration >> >>> >> >> >> to >> >>> >> >> >> >>> be >> >>> >> >> >> >>>> done in the OpenStack side in order to integrate those >> >>> >> services. I >> >>> >> >> >> have >> >>> >> >> >> >>>> deployed OpenStack wallaby using Kolla-ansible on ubuntu >> >>> 20.04. >> >>> >> and >> >>> >> >> >> ceph >> >>> >> >> >> >>>> pacific 16.2.5 was deployed using ansible on ubuntu 20.04 >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> Kindly help for the configuration or documentation. >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> Best Regards >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> Michel >> >>> >> >> >> >>>> _______________________________________________ >> >>> >> >> >> >>>> ceph-users mailing list -- ceph-users at ceph.io >> >>> >> >> >> >>>> To unsubscribe send an email to ceph-users-leave at ceph.io >> >>> >> >> >> >>> >> >>> >> >> >> >>> _______________________________________________ >> >>> >> >> >> >>> ceph-users mailing list -- ceph-users at ceph.io >> >>> >> >> >> >>> To unsubscribe send an email to ceph-users-leave at ceph.io >> >>> >> >> >> >>> >> >>> >> >> >> >> _______________________________________________ >> >>> >> >> >> >> ceph-users mailing list -- ceph-users at ceph.io >> >>> >> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> _______________________________________________ >> >>> >> >> >> ceph-users mailing list -- ceph-users at ceph.io >> >>> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >>> >> >>> >> >>> >> >> >> >> >> From zigo at debian.org Wed Sep 8 11:27:08 2021 From: zigo at debian.org (Thomas Goirand) Date: Wed, 8 Sep 2021 13:27:08 +0200 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: References: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> <20210907232144.n25ei2cyrmkr2xaz@yuggoth.org> <1aaf9274-f669-1e7b-f2f2-4cf795f60f0c@debian.org> Message-ID: On 9/8/21 11:31 AM, Herve Beraud wrote: > An API seems quite a good idea > > I'm ok to give help on this point with my upstream cap and my Red Hat cap. > > As release manager I added your proposition to our "things to changes" > on our release process and tools. > I think that it should be discussed first to see how to implement it and > how to host it. > I think that the technical details would be a transversal decision > between the infra team and the release team and the volunteers on this > topic. > > However you should notice that we are near Xena's deadline, and, so, for > now, I don't think that we will have the time to work on this topic. > Also,  although I'm volunteering to help, you should notice that the > release team will have another PTL for Y, so, the lead of this topic is > at the discretion of my replacant. > > I think a good starting point could be to design some specs or propose > some related goals, so, Thomas, maybe you could start by creating > something like that with your expectations. It will allow us to start > this topic, share our different feedback and our expectations on it (our > different viewpoints, at least from the release team, infra team, and > downstream/distros teams). > > Hopefully it will help us. Hi Herve, Thanks a lot for your enthusiasm, volunteering, and all. I very much agree that it's too late for Xena, though I wasn't even hoping for it that early. This was merely a suggestion. I will try to get my co-maintainer (Michal Arbet) in the loop so we can at least design the blue-print together. I also have to give this some thoughts myself before I can start. Cheers, Thomas Goirand (zigo) From ianyrchoi at gmail.com Wed Sep 8 11:56:50 2021 From: ianyrchoi at gmail.com (Ian Y. Choi) Date: Wed, 8 Sep 2021 20:56:50 +0900 Subject: [all][elections][ptl] Project Team Lead Election Conclusion and Results Message-ID: Thank you to the electorate, to all those who voted and to all candidates who put their name forward for Project Team Lead (PTL) in this election. A healthy, open process breeds trust in our decision making capability thank you to all those who make this process possible. Now for the results of the PTL election process, please join me in extending congratulations to the following PTLs: * Barbican : Douglas Mendizábal * Blazar : Pierre Riteau * Cinder : Brian Rosmaita * Cloudkitty : Rafael Weingartner * Cyborg : Bailin Zhang * Designate : Michael Johnson * Ec2 Api : Andrey Pavlov * Freezer : ge cong * Glance : Abhishek Kekane * Horizon : Vishal Manchanda * Ironic : Iury Gregory Melo Ferreira * Kolla : Michal Nasiadka * Kuryr : Maysa de Macedo Souza * Magnum : Feilong Wang * Manila : Goutham Pacha Ravi * Masakari : Radosław Piliszek * Murano : Rong Zhu * Neutron : Lajos Katona * Nova : Sylvain Bauza * Octavia : Gregory Thiemonge * OpenStack Charms : Alex Kavanagh * Openstack Chef : Lance Albertson * OpenStack Helm : Gage Hugo * OpenStackAnsible : Dmitriy Rabotyagov * OpenStackSDK : Artem Goncharov * Quality Assurance : Martin Kopec * Rally : Andrey Kurilin * Release Management : Elod Illes * Requirements : Matthew Thode * Senlin : XueFeng Liu * Solum : Rong Zhu * Storlets : Takashi Kajinami * Swift : Tim Burke * Tacker : Yasufumi Ogawa * Telemetry : Matthias Runge * Tripleo : James Slagle * Trove : Lingxian Kong * Vitrage : Eyal Bar-Ilan * Watcher : chen ke * Winstackers : Lucian Petrut * Zaqar : wang hao Elections: * Cyborg: https://civs1.civs.us/cgi-bin/results.pl?id=E_807146df7d6a1d3b Election process details and results are also available here: https://governance.openstack.org/election/ Thank you to all involved in the PTL election process, Ian Y. Choi, on behalf of the OpenStack Technical Election Officials -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Wed Sep 8 12:07:25 2021 From: hberaud at redhat.com (Herve Beraud) Date: Wed, 8 Sep 2021 14:07:25 +0200 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: References: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> <20210907232144.n25ei2cyrmkr2xaz@yuggoth.org> <1aaf9274-f669-1e7b-f2f2-4cf795f60f0c@debian.org> Message-ID: Openstack enter more and more in maintenance mode, with decreasing staffs, and reduced project activity which de facto lead us to this kind of situation, so this is a rational proposition. Automation and tooling is a big part of the solution to this problem. Great, thanks for the future blue-print. Do not hesitate to ping us either on IRC or by replying to this thread. Le mer. 8 sept. 2021 à 13:28, Thomas Goirand a écrit : > On 9/8/21 11:31 AM, Herve Beraud wrote: > > An API seems quite a good idea > > > > I'm ok to give help on this point with my upstream cap and my Red Hat > cap. > > > > As release manager I added your proposition to our "things to changes" > > on our release process and tools. > > I think that it should be discussed first to see how to implement it and > > how to host it. > > I think that the technical details would be a transversal decision > > between the infra team and the release team and the volunteers on this > > topic. > > > > However you should notice that we are near Xena's deadline, and, so, for > > now, I don't think that we will have the time to work on this topic. > > Also, although I'm volunteering to help, you should notice that the > > release team will have another PTL for Y, so, the lead of this topic is > > at the discretion of my replacant. > > > > I think a good starting point could be to design some specs or propose > > some related goals, so, Thomas, maybe you could start by creating > > something like that with your expectations. It will allow us to start > > this topic, share our different feedback and our expectations on it (our > > different viewpoints, at least from the release team, infra team, and > > downstream/distros teams). > > > > Hopefully it will help us. > > Hi Herve, > > Thanks a lot for your enthusiasm, volunteering, and all. > > I very much agree that it's too late for Xena, though I wasn't even > hoping for it that early. This was merely a suggestion. > > I will try to get my co-maintainer (Michal Arbet) in the loop so we can > at least design the blue-print together. I also have to give this some > thoughts myself before I can start. > > Cheers, > > Thomas Goirand (zigo) > > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From arxcruz at redhat.com Wed Sep 8 12:10:52 2021 From: arxcruz at redhat.com (Arx Cruz) Date: Wed, 8 Sep 2021 14:10:52 +0200 Subject: [patrole][qa] Proposing Soniya Vyas to Patrole core In-Reply-To: References: <17bc1bcebec.d7cc6895149770.7494147722877953142@ghanshyammann.com> Message-ID: +1 Congratulations!!! On Wed, Sep 8, 2021 at 12:02 PM Chandan Kumar wrote: > On Wed, Sep 8, 2021 at 1:03 AM Ghanshyam Mann > wrote: > > > > ---- On Tue, 07 Sep 2021 14:15:04 -0500 Martin Kopec > wrote ---- > > > Hi all, > > > > > > Soniya Vyas (IRC: soniya29) is very enthusiastic about Patrole > project and since she has been very helpful, I'd like to propose her for > Patrole core. > > > > > > You can vote/feedback on this email. If no objections within a week, > I'll add her to the core list. > > > > +1, she has been very helpful in Patrole project. > > +1, Thank you Sonia for all the great work. > > Thanks, > > Chandan Kumar > > > -- Arx Cruz Software Engineer Red Hat EMEA arxcruz at redhat.com @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From senrique at redhat.com Wed Sep 8 12:46:11 2021 From: senrique at redhat.com (Sofia Enriquez) Date: Wed, 8 Sep 2021 09:46:11 -0300 Subject: [cinder] Bug deputy report for week of 2021-08-09 Message-ID: This is a bug report from 2021-01-09 to 2021-08-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- High - https://bugs.launchpad.net/cinder/+bug/1942696 "RateLimitingMiddleware still belongs to v2 API code" Medium - https://bugs.launchpad.net/cinder/+bug/1942953 "Upload volume to image will terminate the vm's volume attachment". Assigned to xuanyandong - https://bugs.launchpad.net/cinder/+bug/1942342 "Huawei FC driver fails to create a volume from a snapshot with Dorado V6 - Bad or unexpected response from the storage volume backend API: Create luncopy error". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1934036 "Undefined CONF.quota_metadata_items is used in unit tests". Assigned to Takashi Kajinami. Cheers, -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From patryk.jakuszew at gmail.com Wed Sep 8 12:48:50 2021 From: patryk.jakuszew at gmail.com (Patryk Jakuszew) Date: Wed, 8 Sep 2021 14:48:50 +0200 Subject: [neutron] Slow router provisioning during full resync of L3 Agent Message-ID: Hello, I have a long-standing issue with L3 Agent which I would like to finally solve - *very* slow router provisioning in L3 Agent. We are operating a Rocky-based OpenStack deployment with three bare-metal L3 Agent nodes running in legacy mode. After restarting the L3 node, it takes a really long time for the L3 agent to become fully operational. There are two parts of resync which take much time: getting a list of routers from neutron-server and actually recreate them in the L3 node. While the long running time of router list retrieval is somewhat understandable, the router provisioning process itself proves to be very troublesome in our operations tasks. In our production deployment with around 250 routers, it takes around 2 hours (!) to recreate the router namespaces and have the L3 node fully functional again. Two hours of router re-provisioning is actually an optimistic scenario, this proved to be much longer during the outages we encountered (sometimes the sync took nearly 6-8 hours). This effectively prolongs any maintenance upgrades, configuration changes and OpenStack release upgrades. Another thing is, on that same production environment the first 100 routers usually get provisioned fast (around 30 minutes), after that it slows down with each router - this kind of non deterministic behavior makes it hard to communicate the maintenance finish ETA for our users. We also have a test environment with Stein already installed, where this problem is also present - full resync of 150 routers, with only one external gateway ports, takes around an hour to complete. Are there any operators here who also encountered that issue? Does anyone have any experience with similar situation and are willing to share their observations and optimizations? -- Regards, Patryk Jakuszew From fungi at yuggoth.org Wed Sep 8 13:14:09 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 8 Sep 2021 13:14:09 +0000 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: <1aaf9274-f669-1e7b-f2f2-4cf795f60f0c@debian.org> References: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> <20210907232144.n25ei2cyrmkr2xaz@yuggoth.org> <1aaf9274-f669-1e7b-f2f2-4cf795f60f0c@debian.org> Message-ID: <20210908131409.273rtnfdw57rhweo@yuggoth.org> On 2021-09-08 10:27:50 +0200 (+0200), Thomas Goirand wrote: [...] > Ok, so your recommendation is that, for each and every component that is > marked release-independent, I should be checking the global > requirements. I just wrote that this is unrealistic that a human does > such a check. I'm not saying that a human needs to check it, in fact that file is machine-readable (it's used as input to pip but it's quite trivial to parse). What I said is that it's what we have now, and probably the closest thing we have to what you're asking for. It's a list of what exact versions of dependencies we test the software with, nothing more. We effectively freeze and branch that list whenever a new coordinated release of OpenStack happens, and then only adjust it to reflect stable point releases of our cycle-based deliverables. Any independent release deliverables in there are still basically frozen, treated just like any other "external" dependency of OpenStack. If it helps, we also maintain a redirector service which routes URLs like https://releases.openstack.org/constraints/upper/wallaby to a raw download of the correct file. We also make sure the codename for the cycle currently in progress is supported, so that you can begin using it in preparation for the next coordinated release. > If you didn't know, in Debian, Michal Arbet (who co-maintains OpenStack > with me together in Debian) wrote this: [...] > Thanks to this page, anyone can know the status of OpenStack packaging > on all the different branches we maintain. If you look at it right now, > you'll notice that we're up-to-date, as we packaged all non-clients and > all clients library, but we didn't package new versions of the puppet > manifests yet (I'm waiting for the RCs to do that). That's very cool! > If you click on the buster-victoria button, and look for Taskflow, > you'll see that it's up-to-date. If you click on bullseye-xena, and > search for Taskflow, you will ... find nothing! That's because taskflow > doesn't appear in https://releases.openstack.org/xena/index.html since > it is release-independent. As a result, I'm left with no other choice > than being very careful and manually check the global-requirements. [...] wget -qO- https://releases.openstack.org/constraints/upper/wallaby \ | grep -i ^taskflow= wget -qO- https://releases.openstack.org/constraints/upper/xena \ | grep -i ^taskflow= Its not ideal, no, but it's fairly straightforward (not that I would do it in shell script, but parsing that with the re module in Python's stdlib is trivial, or you could use something like the packaging.requirements parser from python3-packaging). As an aside, parsing HTML is very fiddly, you might consider directly consuming the structured data maintained in the openstack/releases repository as it's far easier to script (it's YAML). I believe the openstack_releases Python package even provides modules for parsing those. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From tonykarera at gmail.com Wed Sep 8 13:16:57 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 8 Sep 2021 15:16:57 +0200 Subject: Tenant vlan Network Issue In-Reply-To: <5528888.crQR7OUkfV@p1> References: <5528888.crQR7OUkfV@p1> Message-ID: Hello Slawek, I really need your help. I have tried all I can but still the k8s-cluster gets stuck with this log here root at k8s-cluster-1-vo5s5jvtjoa7-master-0 core]# cd /var/log/heat-config/heat-config-script/ [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# ls a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7-kube_cluster_config-asfqq352zosg.log c42dcbf1-7812-49a2-b312-96c24225b407-k8s-cluster-1-vo5s5jvtjoa7-kube_masters-waugw6vksw5p-0-vrt2gkfugedg-master_config-xv3vlmr27r46.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# cat a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7-kube_cluster_config-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# cat a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7-kube_cluster_config-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# cat a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7-kube_cluster_config-asfqq352zosg.log Starting to run kube-apiserver-to-kubelet-role + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ kubectl get --raw=/healthz The connection to the server localhost:8080 was refused - did you specify the right host or port? + '[' ok = '' ']' + sleep 5 ++ kubectl get --raw=/healthz The connection to the server localhost:8080 was refused - did you specify the right host or port? + '[' ok = '' ']' + sleep 5 Regards Tony Karera On Mon, Aug 30, 2021 at 8:50 AM Slawek Kaplonski wrote: > Hi, > > You can use vlan as tenant network too. But You have to remember that vlan > is > provider network, which means that You need to have configured vlans on > Your > network infrastructure too. And in case of tenant network, Neutron will > automatically allocate one of the vlan ids to the network as user don't > have > possibility to choose vlan id for network (at least that's default policy > in > Neutron). > So, You should check what vlan id is used by such tenant network, check if > this vlan is properly confiugred on Your switches and then, if all is > good, > check where exactly such dhcp requests are dropped (are they going from > compute node? are packets comming to the node with dhcp agent?). Base on > that > You can narrow down where the issue can be. > > On sobota, 28 sierpnia 2021 18:07:07 CEST Karera Tony wrote: > > How can i make them both vxlan and vlan > > > > On Sat, 28 Aug 2021, 17:43 Mohammed Naser, wrote: > > > Your issue is that tenant_network_types should be vxlan. > > > > > > On Sat, Aug 28, 2021 at 4:29 AM Karera Tony > wrote: > > >> Hello Team, > > >> > > >> Whenever I create an internal network with type vlan. > > >> > > >> The Instances don't get IPs but for external Network it's fine. > > >> > > >> > > >> Below is the etc/kolla/config/neutron/ml2_conf.ini file > > >> > > >> > > >> [ml2] > > >> type_drivers = flat,vlan,vxlan > > >> tenant_network_types = vlan,flat > > >> mechanism_drivers = openvswitch,l2population > > >> > > >> [ml2_type_vlan] > > >> network_vlan_ranges = physnet1 > > >> [ml2_type_flat] > > >> flat_networks = physnet1 > > >> > > >> Is there something I need to change or I need to configure the > interface > > >> differently > > >> Regards > > >> > > >> Tony Karera > > >> > > >> > > >> -- > > > > > > Mohammed Naser > > > VEXXHOST, Inc. > > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Wed Sep 8 13:31:18 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Wed, 08 Sep 2021 15:31:18 +0200 Subject: Tenant vlan Network Issue In-Reply-To: References: <5528888.crQR7OUkfV@p1> Message-ID: <3773315.Oo1np0eLaN@p1> Hi, On środa, 8 września 2021 15:16:57 CEST Karera Tony wrote: > Hello Slawek, > > I really need your help. > > I have tried all I can but still the k8s-cluster gets stuck with this log > here > > root at k8s-cluster-1-vo5s5jvtjoa7-master-0 core]# cd > /var/log/heat-config/heat-config-script/ > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# ls > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- kube_cluster_c > onfig-asfqq352zosg.log > c42dcbf1-7812-49a2-b312-96c24225b407-k8s-cluster-1-vo5s5jvtjoa7- kube_masters- > waugw6vksw5p-0-vrt2gkfugedg-master_config-xv3vlmr27r46.log > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# cat > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- kube_cluster_c > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 > heat-config-script]# cat > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- kube_cluster_c > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 > heat-config-script]# cat > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- kube_cluster_c > onfig-asfqq352zosg.log Starting to run kube-apiserver-to-kubelet-role > + echo 'Waiting for Kubernetes API...' > Waiting for Kubernetes API... > ++ kubectl get --raw=/healthz > The connection to the server localhost:8080 was refused - did you specify > the right host or port? > + '[' ok = '' ']' > + sleep 5 > ++ kubectl get --raw=/healthz > The connection to the server localhost:8080 was refused - did you specify > the right host or port? But why Your script is trying to connect to localhost? It's not the VM address I guess :) > + '[' ok = '' ']' > + sleep 5 > > > Regards > > Tony Karera > > > > > On Mon, Aug 30, 2021 at 8:50 AM Slawek Kaplonski > > wrote: > > Hi, > > > > You can use vlan as tenant network too. But You have to remember that vlan > > is > > provider network, which means that You need to have configured vlans on > > Your > > network infrastructure too. And in case of tenant network, Neutron will > > automatically allocate one of the vlan ids to the network as user don't > > have > > possibility to choose vlan id for network (at least that's default policy > > in > > Neutron). > > So, You should check what vlan id is used by such tenant network, check if > > this vlan is properly confiugred on Your switches and then, if all is > > good, > > check where exactly such dhcp requests are dropped (are they going from > > compute node? are packets comming to the node with dhcp agent?). Base on > > that > > You can narrow down where the issue can be. > > > > On sobota, 28 sierpnia 2021 18:07:07 CEST Karera Tony wrote: > > > How can i make them both vxlan and vlan > > > > > > On Sat, 28 Aug 2021, 17:43 Mohammed Naser, wrote: > > > > Your issue is that tenant_network_types should be vxlan. > > > > > > > > On Sat, Aug 28, 2021 at 4:29 AM Karera Tony > > > > wrote: > > > >> Hello Team, > > > >> > > > >> Whenever I create an internal network with type vlan. > > > >> > > > >> The Instances don't get IPs but for external Network it's fine. > > > >> > > > >> > > > >> Below is the etc/kolla/config/neutron/ml2_conf.ini file > > > >> > > > >> > > > >> [ml2] > > > >> type_drivers = flat,vlan,vxlan > > > >> tenant_network_types = vlan,flat > > > >> mechanism_drivers = openvswitch,l2population > > > >> > > > >> [ml2_type_vlan] > > > >> network_vlan_ranges = physnet1 > > > >> [ml2_type_flat] > > > >> flat_networks = physnet1 > > > >> > > > >> Is there something I need to change or I need to configure the > > > > interface > > > > > >> differently > > > >> Regards > > > >> > > > >> Tony Karera > > > >> > > > >> > > > >> -- > > > > > > > > Mohammed Naser > > > > VEXXHOST, Inc. > > > > -- > > Slawek Kaplonski > > Principal Software Engineer > > Red Hat -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From radoslaw.piliszek at gmail.com Wed Sep 8 13:44:29 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Wed, 8 Sep 2021 15:44:29 +0200 Subject: [all][tc] Minium python version for yoga and centos 9 stream. In-Reply-To: <50423dee4b245d18befa21bd7ef1666653f7bf2e.camel@redhat.com> References: <33913002aa0ad4be63f6c36be310f4e1dd7bd008.camel@redhat.com> <4b261268e701d3c464f180e89a795841c06dc8d0.camel@redhat.com> <17bc0e8dea4.fc87ca61142047.29866411617360676@ghanshyammann.com> <20210907160308.5qcih7675nbjwghu@yuggoth.org> <17bc125274c.114f9c117145230.1407351724601243402@ghanshyammann.com> <50423dee4b245d18befa21bd7ef1666653f7bf2e.camel@redhat.com> Message-ID: Sorry for top posting but I don't have a precise quote to reply to. What I wish to say is that I am trying to get Debian tested in CI but it looks like python3.9 might be playing tricks. I would appreciate it if someone more knowledgeable in Nova and Neutron internals could take a look at failures in https://review.opendev.org/c/openstack/devstack/+/789083 Do note slightly different tests fail each run. -yoctozepto On Tue, Sep 7, 2021 at 7:40 PM Sean Mooney wrote: > > On Tue, 2021-09-07 at 11:44 -0500, Ghanshyam Mann wrote: > > ---- On Tue, 07 Sep 2021 11:03:09 -0500 Jeremy Stanley wrote ---- > > > On 2021-09-07 10:38:12 -0500 (-0500), Ghanshyam Mann wrote: > > > [...] > > > > Definitely we can move to 3.9 as official testing runtime if any > > > > of the currently supporting distro support it as default. > > > [...] > > > > > > I don't think it needs to be the default Python in a release so long > > > as it's available. To quote from > > > https://governance.openstack.org/tc/resolutions/20181024-python-update-process.html#unit-tests > > > > > > "Prior to the beginning of each release cycle, the TC will > > > designate the minor versions of Python 3 that will be tested in > > > that release using the following criteria: > > > [...] > > > The latest released version of Python 3 that is available in any > > > distribution we can feasibly use for testing. It need not be a > > > long-term-supported release, but could be a non-LTS version of > > > Ubuntu, Fedora, openSUSE Leap, or even a rolling release > > > distribution (such as Debian Testing or openSUSE Tumbleweed) if > > > necessary." > > > > > > I recall clearly that this was added so that we would attempt to > > > test the latest available version of Python in order to help avoid > > > having software which is clearly non-functional on distributions > > > which are working in parallel to update their defaults (and so might > > > reasonably want to ship OpenStack with an interpreter version which > > > was not their default at the time we started development for that > > > coordinated release). > > > > Sure, and by seeing the current py39 job it is mostly passing > > - https://zuul.openstack.org/builds?job_name=openstack-tox-py39 > > > > Let's discuss it in PTG and hopefully by then we might see centos9 > > release or closure plan to release dates and as discussed during > > defining the Yoga tetsing runtime, we can update it once if > > centos9 is there then update centos versionb as well as the py version > > otherwise we can disucss only making py39 jobs voting which are running > > on ubuntu focal. > not sure if centos 9 will be avaiable by ptg but i hope so. > in anycaes ubuntu 21.04 default to python 3.9 and debian 11 "bullseye" which is the latest > __stable__ debian release also default to python 3.9 as does fedora 34 but that a much bigger moving target. > > in general 3.9 is avaiable if not default in several distro which is why i would like to include it for > required testing e.g. move the current non voting jobs to voting for unit and functest at least. > it woudl be nice if we could have at least on tempst job on py39 too but sure lets pick this up at the ptg. > > im not agaisnt keeping support for phython 3.6 and centos 8 stream for yoga but i would like to be abel to > do at least some smoke tests with centos 9 stream if its avaiable. > > the none stream version fo centos 8 i think is end of life at the end of this year? but the centos 8 stream > release will continue untile rhel 8 is eol so that shoudl still be supproted to some degree for the full yoga cycle. > the none stream version however wont be supported when yoga releases. > > > > > > -gmann > > > > > -- > > > Jeremy Stanley > > > > > > > > From micou12 at gmail.com Wed Sep 8 09:49:03 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Wed, 8 Sep 2021 11:49:03 +0200 Subject: [ceph-users] Re: Replacing swift with RGW In-Reply-To: <20210906144401.Horde.7j4UyxpB4wl0f5RgwX1Y0FD@webmail.nde.ag> References: <20210901122221.Horde.v0FeoCUfex4WYSaXReGzV1i@webmail.nde.ag> <20210901134023.Horde.ZhZ0JBty-nYNsBfGe6WOijH@webmail.nde.ag> <20210902071439.Horde.4cayqDV75oyYS75eyQZMSPD@webmail.nde.ag> <20210902081741.Horde.Xp06qPUkci05sKXFlo1F5ap@webmail.nde.ag> <20210902083925.Horde.a0LD9j43RkGPIl7UuWCRtyu@webmail.nde.ag> <20210906144401.Horde.7j4UyxpB4wl0f5RgwX1Y0FD@webmail.nde.ag> Message-ID: Hello team, Now I am able to create containers and upload objects in the openstack CLI side using the following commands: (kolla-open1) stack at kolla-open1:~$ swift -V 1.0 -A http://ceph-mon2:8080/auth/v1 -U operator:swift -K opswift post my-container3 (kolla-open1) stack at kolla-open1:~$ swift -V 1.0 -A http://ceph-mon2:8080/auth/v1 -U operator:swift -K opswift upload my-container3 michel.txt above commands are successful and containers and their objects are stored in ceph RGW default pool which replace swift . The issues I remain with is that once I try to access the object store on horizon and click on containers it immediately disconnect hoizon to the login status. [image: image.png] [image: image.png] Kindly help and advise Michel On Mon, Sep 6, 2021 at 4:50 PM Eugen Block wrote: > It's hard to tell what is wrong with your setup, I don't really use > mine but this was the last working config I had to be able to create > swift containers directly in RGW: > > ---snip--- > # ceph.conf: > > [client.rgw.ses6-mon1] > rgw frontends = "beast port=80" > rgw dns name = ses6-mon1.example.com > rgw enable usage log = true > > rgw thread pool size = 512 > rgw keystone api version = 3 > rgw keystone url = http://control-node.example.com:5000 > > rgw keystone admin user = rgw > rgw keystone admin password = **** > rgw keystone admin domain = default > rgw keystone admin project = service > rgw keystone accepted roles = admin,Member,_member_,member > rgw keystone verify ssl = false > rgw s3 auth use keystone = true > rgw keystone revocation interval = 0 > > > # User role (I don't think admin is required) > > openstack role add --user rgw --project 9e8a67da237a4b26afb2819d2dea2219 > admin > > > # Create keystone endpoints > > openstack endpoint create --region RegionOne swift admin > "http://ses6-mon1.example.com:80/swift/v1" > openstack endpoint create --region RegionOne swift internal > "http://ses6-mon1.example.com:80/swift/v1" > openstack endpoint create --region RegionOne swift public > "http://ses6-mon1.example.com:80/swift/v1" > > > # Create container and files > > openstack container create swift1 > +---------+-----------+---------------------------------------------------+ > | account | container | x-trans-id | > +---------+-----------+---------------------------------------------------+ > | v1 | swift1 | tx000000000000000000001-0060b4ba48-d724dc-default | > +---------+-----------+---------------------------------------------------+ > > openstack object create --name file1 swift1 chef-client.log > +--------+-----------+----------------------------------+ > | object | container | etag | > +--------+-----------+----------------------------------+ > | file1 | swift1 | 56a1ed3b201c1e753bcbe80c640349f7 | > +--------+-----------+----------------------------------+ > ---snip--- > > > You are mixing dns names and IP addresses, I can't tell if that's a > problem but it probably should work, I'm not sure. Compared to my > ceph.conf these are the major differences: > > rgw keystone verify ssl = false > rgw s3 auth use keystone = true > rgw keystone revocation interval = 0 > > And I don't use rgw_keystone_token_cache_size. Maybe try again with > the options I use. > > > Zitat von Michel Niyoyita : > > > Hello, > > > > I am trying to replace swift by RGW as backend storage but I failed once > I > > try to post a container in the OpenStack side however, all interfaces are > > configured (admin, public and internal). but Once I post from RGW host it > > is created . Another issue is that object storage does not appear on the > > horizon dashboard . I have deployed openstack all-in-one using > > kolla-ansible and Os is ubuntu > > > > (kolla-open1) stack at kolla-open1:~$ swift -v post myswift > > Container POST failed: http://ceph-osd3:8080/swift/v1/myswift 401 > > Unauthorized b'AccessDenied' > > Failed Transaction ID: tx000000000000000000008-006135dcbd-87d63-default > > > > (kolla-open1) stack at kolla-open1:~$ swift list > > Account GET failed: http://ceph-osd3:8080/swift/v1?format=json 401 > > Unauthorized [first 60 chars of response] > > b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000c-' > > Failed Transaction ID: tx00000000000000000000c-006135de42-87d63-default > > > > Kindly help to solve the issue > > > > Michel > > > > On Thu, Sep 2, 2021 at 4:28 PM Alex Schultz wrote: > > > >> The swift docs are a bit out of date as they still reference python2 > >> despite python3 being supported for some time now. Replace python- with > >> python3- and try again. > >> > >> > >> On Thu, Sep 2, 2021 at 7:35 AM Michel Niyoyita > wrote: > >> > >>> > >>> > >>> ---------- Forwarded message --------- > >>> From: Michel Niyoyita > >>> Date: Thu, Sep 2, 2021 at 12:17 PM > >>> Subject: Fwd: [ceph-users] Re: Replacing swift with RGW > >>> To: > >>> > >>> > >>> > >>> > >>> ---------- Forwarded message --------- > >>> From: Eugen Block > >>> Date: Thu, Sep 2, 2021 at 10:39 AM > >>> Subject: Re: [ceph-users] Re: Replacing swift with RGW > >>> To: Michel Niyoyita > >>> > >>> > >>> You should continue this thread on the openstack-discuss mailing list > >>> (openstack-discuss at lists.openstack.org) since this is not a ceph > issue. > >>> I'm not familiar with kolla and I don't know the requirements to > >>> install openstack-swift, you'll need to ask the openstack community. > >>> > >>> > >>> Zitat von Michel Niyoyita : > >>> > >>> > Below are errors I am getting once I try to run swift commands . the > >>> second > >>> > one is the error I get once try to install python-swiftclient > >>> > > >>> > (kolla-open) [stack at kolla-open ~]$ swift -v stat > >>> > -bash: swift: command not found > >>> > (kolla-open) [stack at kolla-open ~]$ sudo yum -y install > >>> python-swiftclient > >>> > Last metadata expiration check: 0:59:21 ago on Thu 02 Sep 2021 > 09:21:53 > >>> AM > >>> > CAT. > >>> > No match for argument: python-swiftclient > >>> > Error: Unable to find a match: python-swiftclient > >>> > (kolla-open) [stack at kolla-open ~]$ > >>> > > >>> > Waiting for your help > >>> > > >>> > On Thu, Sep 2, 2021 at 10:17 AM Eugen Block wrote: > >>> > > >>> >> I can't tell for sure, but yes, I believe you need the > openstack-swift > >>> >> package (with dependencies). What errors do you get? The more > >>> >> information you share the better people can help. > >>> >> > >>> >> > >>> >> Zitat von Michel Niyoyita : > >>> >> > >>> >> > I tried to install "sudo yum -y install python-swiftclient" on > >>> openstack > >>> >> > side but fails . are there openastack-shwift packages which are > >>> needed? > >>> >> > if are there please help me to get . may be also it is the cause > I > >>> am > >>> >> > failing to run swift command on openstack cli side. > >>> >> > > >>> >> > thank you for your continued support. > >>> >> > > >>> >> > Micheal > >>> >> > > >>> >> > On Thu, Sep 2, 2021 at 9:14 AM Eugen Block wrote: > >>> >> > > >>> >> >> I only configured the endpoints for the clients to directly > access > >>> the > >>> >> >> RGWs, but you'll probably need to install the openstack-swift > >>> package. > >>> >> >> Or have you done that already? > >>> >> >> > >>> >> >> > >>> >> >> Zitat von Michel Niyoyita : > >>> >> >> > >>> >> >> > Thank you Eugen for your prompt response. > >>> >> >> > > >>> >> >> > Now the commands provided work. but I am not finding the object > >>> >> storage > >>> >> >> on > >>> >> >> > the horizon dashboard , but it appears in the system > information > >>> >> >> services. > >>> >> >> > [image: image.png] > >>> >> >> > so my question is how to configure it in order that it can > appear > >>> in > >>> >> the > >>> >> >> > dashboard . > >>> >> >> > > >>> >> >> > Michel > >>> >> >> > > >>> >> >> > On Wed, Sep 1, 2021 at 3:49 PM Eugen Block > wrote: > >>> >> >> > > >>> >> >> >> Sorry, one little detail slipped through, the '--region' flag > >>> has to > >>> >> >> >> be put before the 'service' name. The correct command would > be: > >>> >> >> >> > >>> >> >> >> openstack endpoint create --region RegionOne swift admin > >>> >> >> >> http://ceph-osd3:8080/swift/v1 > >>> >> >> >> > >>> >> >> >> and respectively for the other interfaces. > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> Zitat von Eugen Block : > >>> >> >> >> > >>> >> >> >> > Hi, > >>> >> >> >> > > >>> >> >> >> > this is not a ceph issue but your openstack cli command as > the > >>> >> error > >>> >> >> >> > message states. > >>> >> >> >> > > >>> >> >> >> > Try one interface at a time: > >>> >> >> >> > > >>> >> >> >> > openstack endpoint create swift public > >>> >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift > >>> >> >> >> > openstack endpoint create swift admin > >>> >> http://ceph-osd3:8080/swift/v1 > >>> >> >> >> > --region RegionOne swift > >>> >> >> >> > openstack endpoint create swift internal > >>> >> >> >> > http://ceph-osd3:8080/swift/v1 --region RegionOne swift > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> > Zitat von Michel Niyoyita : > >>> >> >> >> > > >>> >> >> >> >> Hello , > >>> >> >> >> >> > >>> >> >> >> >> Below are errors I am getting once I am trying to integrate > >>> swift > >>> >> >> with > >>> >> >> >> >> Radosgateway. > >>> >> >> >> >> > >>> >> >> >> >> From openstack side once i try to endpoint which will point > >>> to the > >>> >> >> >> >> radosgateway : > >>> >> >> >> >> > >>> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ openstack endpoint > >>> create > >>> >> >> >> >> --publicurl http://ceph-osd3:8080/swift/v1 --adminurl > >>> >> >> >> >> http://ceph-osd3:8080/swift/v1 --internal > >>> >> >> >> http://ceph-osd3:8080/swift/v1 > >>> >> >> >> >> --region RegionOne swift > >>> >> >> >> >> usage: openstack endpoint create [-h] [-f > >>> >> >> {json,shell,table,value,yaml}] > >>> >> >> >> >> [-c COLUMN] [--noindent] > >>> [--prefix > >>> >> >> >> PREFIX] > >>> >> >> >> >> [--max-width ] > >>> >> [--fit-width] > >>> >> >> >> >> [--print-empty] [--region > >>> >> >> ] > >>> >> >> >> >> [--enable | --disable] > >>> >> >> >> >> > >>> >> >> >> >> openstack endpoint create: error: argument : > >>> invalid > >>> >> >> choice: > >>> >> >> >> ' > >>> >> >> >> >> http://ceph-osd3:8080/swift/v1' (choose from 'admin', > >>> 'public', > >>> >> >> >> 'internal') > >>> >> >> >> >> > >>> >> >> >> >> (kolla-open) [stack at kolla-open kolla]$ > >>> >> >> >> >> > >>> >> >> >> >> below are my /etc/ceph/ceph.conf file : > >>> >> >> >> >> > >>> >> >> >> >> [client.rgw.ceph-osd3] > >>> >> >> >> >> rgw_dns_name = ceph-osd3 > >>> >> >> >> >> host = ceph-osd3 > >>> >> >> >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3/keyring > >>> >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.log > >>> >> >> >> >> rgw frontends = civetweb port=10.10.29.110:8080 > >>> num_threads=100 > >>> >> >> >> >> rgw_keystone_url=http://10.10.29.150:35357 > >>> >> >> >> >> rgw_keystone_admin_user=admin > >>> >> >> >> >> rgw_keystone_admin_password=admin > >>> >> >> >> >> rgw_keystone_admin_tenant=admin > >>> >> >> >> >> rgw_keystone_accepted_role=admin Member swiftoperator > >>> >> >> >> >> rgw_keystone_token_cache_size=200 > >>> >> >> >> >> rgw_keystone_revocation_interval=300 > >>> >> >> >> >> > >>> >> >> >> >> [client.rgw.ceph-osd3.rgw0] > >>> >> >> >> >> host = ceph-osd3 > >>> >> >> >> >> keyring = > >>> /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring > >>> >> >> >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log > >>> >> >> >> >> rgw frontends = beast endpoint=10.10.29.110:8080 > >>> >> >> >> >> rgw thread pool size = 512 > >>> >> >> >> >> > >>> >> >> >> >> please note that my rgw_dns_name = ceph_osd3 with > >>> 10.10.29.110 as > >>> >> IP > >>> >> >> >> >> > >>> >> >> >> >> and 10.10.29.150 all-in-one IP > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> Please crosscheck where I might make mistake and try to > >>> correct. > >>> >> >> >> >> > >>> >> >> >> >> Best regards > >>> >> >> >> >> > >>> >> >> >> >> Michel > >>> >> >> >> >> > >>> >> >> >> >> On Mon, Aug 30, 2021 at 11:25 AM Etienne Menguy < > >>> >> >> >> etienne.menguy at croit.io> > >>> >> >> >> >> wrote: > >>> >> >> >> >> > >>> >> >> >> >>> Hi, > >>> >> >> >> >>> > >>> >> >> >> >>> There are some information on Ceph documentation > >>> >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/ < > >>> >> >> >> >>> https://docs.ceph.com/en/latest/radosgw/keystone/> . > >>> >> >> >> >>> - Use keystone as auth for RGW > >>> >> >> >> >>> - Create service and register your RGW as swift > >>> >> >> >> >>> > >>> >> >> >> >>> Étienne > >>> >> >> >> >>> > >>> >> >> >> >>>> On 27 Aug 2021, at 15:47, Michel Niyoyita < > >>> micou12 at gmail.com> > >>> >> >> wrote: > >>> >> >> >> >>>> > >>> >> >> >> >>>> Hello , > >>> >> >> >> >>>> > >>> >> >> >> >>>> I have configured RGW in my ceph cluster deployed using > ceph > >>> >> >> ansible > >>> >> >> >> and > >>> >> >> >> >>>> create sub user to access the created containers and > would > >>> like > >>> >> to > >>> >> >> >> >>> replace > >>> >> >> >> >>>> swift by RGW in the openstack side. Anyone can help on > >>> >> >> configuration > >>> >> >> >> to > >>> >> >> >> >>> be > >>> >> >> >> >>>> done in the OpenStack side in order to integrate those > >>> >> services. I > >>> >> >> >> have > >>> >> >> >> >>>> deployed OpenStack wallaby using Kolla-ansible on ubuntu > >>> 20.04. > >>> >> and > >>> >> >> >> ceph > >>> >> >> >> >>>> pacific 16.2.5 was deployed using ansible on ubuntu 20.04 > >>> >> >> >> >>>> > >>> >> >> >> >>>> Kindly help for the configuration or documentation. > >>> >> >> >> >>>> > >>> >> >> >> >>>> Best Regards > >>> >> >> >> >>>> > >>> >> >> >> >>>> Michel > >>> >> >> >> >>>> _______________________________________________ > >>> >> >> >> >>>> ceph-users mailing list -- ceph-users at ceph.io > >>> >> >> >> >>>> To unsubscribe send an email to ceph-users-leave at ceph.io > >>> >> >> >> >>> > >>> >> >> >> >>> _______________________________________________ > >>> >> >> >> >>> ceph-users mailing list -- ceph-users at ceph.io > >>> >> >> >> >>> To unsubscribe send an email to ceph-users-leave at ceph.io > >>> >> >> >> >>> > >>> >> >> >> >> _______________________________________________ > >>> >> >> >> >> ceph-users mailing list -- ceph-users at ceph.io > >>> >> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> _______________________________________________ > >>> >> >> >> ceph-users mailing list -- ceph-users at ceph.io > >>> >> >> >> To unsubscribe send an email to ceph-users-leave at ceph.io > >>> >> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> > >>> > >>> > >>> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 11853 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 11591 bytes Desc: not available URL: From gmann at ghanshyammann.com Wed Sep 8 14:21:38 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 08 Sep 2021 09:21:38 -0500 Subject: [all][elections][ptl] Project Team Lead Election Conclusion and Results In-Reply-To: References: Message-ID: <17bc5c92365.1202dce8f194142.5550056093873382816@ghanshyammann.com> Thanks, Ian, Amy, Helena, Andy, and Belmiro for doing an awesome job as election officials and for the smooth running of both elections. -gmann ---- On Wed, 08 Sep 2021 06:56:50 -0500 Ian Y. Choi wrote ---- > Thank you to the electorate, to all those who voted and to all candidates who put their name forward for Project Team Lead (PTL) in this election. A healthy, open process breeds trust in our decision making capability thank you to all those who make this process possible. > > Now for the results of the PTL election process, please join me in extending congratulations to the following PTLs: > > * Barbican : Douglas Mendizábal > * Blazar : Pierre Riteau > * Cinder : Brian Rosmaita > * Cloudkitty : Rafael Weingartner > * Cyborg : Bailin Zhang > * Designate : Michael Johnson > * Ec2 Api : Andrey Pavlov > * Freezer : ge cong > * Glance : Abhishek Kekane > * Horizon : Vishal Manchanda > * Ironic : Iury Gregory Melo Ferreira > * Kolla : Michal Nasiadka > * Kuryr : Maysa de Macedo Souza > * Magnum : Feilong Wang > * Manila : Goutham Pacha Ravi > * Masakari : Radosław Piliszek > * Murano : Rong Zhu > * Neutron : Lajos Katona > * Nova : Sylvain Bauza > * Octavia : Gregory Thiemonge > * OpenStack Charms : Alex Kavanagh > * Openstack Chef : Lance Albertson > * OpenStack Helm : Gage Hugo > * OpenStackAnsible : Dmitriy Rabotyagov > * OpenStackSDK : Artem Goncharov > * Quality Assurance : Martin Kopec > * Rally : Andrey Kurilin > * Release Management : Elod Illes > * Requirements : Matthew Thode > * Senlin : XueFeng Liu > * Solum : Rong Zhu > * Storlets : Takashi Kajinami > * Swift : Tim Burke > * Tacker : Yasufumi Ogawa > * Telemetry : Matthias Runge > * Tripleo : James Slagle > * Trove : Lingxian Kong > * Vitrage : Eyal Bar-Ilan > * Watcher : chen ke > * Winstackers : Lucian Petrut > * Zaqar : wang hao > > Elections: > * Cyborg: https://civs1.civs.us/cgi-bin/results.pl?id=E_807146df7d6a1d3b > > Election process details and results are also available here: https://governance.openstack.org/election/ > > Thank you to all involved in the PTL election process, > > Ian Y. Choi, on behalf of the OpenStack Technical Election Officials > From mkopec at redhat.com Wed Sep 8 14:24:08 2021 From: mkopec at redhat.com (Martin Kopec) Date: Wed, 8 Sep 2021 16:24:08 +0200 Subject: [designate][interop] test_create_all_recordset_types ddt issue Message-ID: Hi designate team, I'm reaching out because we have encountered a problem with one of the tests [1]. We track the following name [2]: designate_tempest_plugin.tests.api.v2.test_recordset.RecordsetsTest.test_create_all_recordset_types That test name is actually incorrect due to the ddt and the real name of the test is created dynamically based on the test data passed via ddt. We have a consistency check which is supposed to verify that the test names tracked in our guidelines match the ones in Tempest/tempest plugins, however, there was a bug - we are fixing that by [3]. The question is, should interop track all the permutations of the test? See [4]. [1] https://opendev.org/openstack/designate-tempest-plugin/src/commit/da27a70ae2b39695ef6f03bbefb55afeacf1cdf3/designate_tempest_plugin/tests/api/v2/test_recordset.py#L87 [2] https://opendev.org/osf/interop/src/commit/d8eac71fa507c68e9c1f83a19dd77af8483d01b6/add-ons/guidelines/dns.2020.11.json#L116-L117 [3] https://review.opendev.org/c/osf/interop/+/806598 [4] https://review.opendev.org/c/osf/interop/+/807586 Thank you, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 8 15:24:57 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 8 Sep 2021 17:24:57 +0200 Subject: Tenant vlan Network Issue In-Reply-To: <3773315.Oo1np0eLaN@p1> References: <5528888.crQR7OUkfV@p1> <3773315.Oo1np0eLaN@p1> Message-ID: Hello Slawek, Unfortunately I am not sure because all I did is configure the cluster. Nothing else. Regards Tony Karera On Wed, Sep 8, 2021 at 3:31 PM Slawek Kaplonski wrote: > Hi, > > On środa, 8 września 2021 15:16:57 CEST Karera Tony wrote: > > Hello Slawek, > > > > I really need your help. > > > > I have tried all I can but still the k8s-cluster gets stuck with this log > > here > > > > root at k8s-cluster-1-vo5s5jvtjoa7-master-0 core]# cd > > /var/log/heat-config/heat-config-script/ > > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# ls > > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- > kube_cluster_c > > onfig-asfqq352zosg.log > > c42dcbf1-7812-49a2-b312-96c24225b407-k8s-cluster-1-vo5s5jvtjoa7- > kube_masters- > > waugw6vksw5p-0-vrt2gkfugedg-master_config-xv3vlmr27r46.log > > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# cat > > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- > kube_cluster_c > > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 > > heat-config-script]# cat > > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- > kube_cluster_c > > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 > > heat-config-script]# cat > > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- > kube_cluster_c > > onfig-asfqq352zosg.log Starting to run kube-apiserver-to-kubelet-role > > + echo 'Waiting for Kubernetes API...' > > Waiting for Kubernetes API... > > ++ kubectl get --raw=/healthz > > The connection to the server localhost:8080 was refused - did you specify > > the right host or port? > > + '[' ok = '' ']' > > + sleep 5 > > ++ kubectl get --raw=/healthz > > The connection to the server localhost:8080 was refused - did you specify > > the right host or port? > > But why Your script is trying to connect to localhost? It's not the VM > address > I guess :) > > > + '[' ok = '' ']' > > + sleep 5 > > > > > > Regards > > > > Tony Karera > > > > > > > > > > On Mon, Aug 30, 2021 at 8:50 AM Slawek Kaplonski > > > > wrote: > > > Hi, > > > > > > You can use vlan as tenant network too. But You have to remember that > vlan > > > is > > > provider network, which means that You need to have configured vlans on > > > Your > > > network infrastructure too. And in case of tenant network, Neutron will > > > automatically allocate one of the vlan ids to the network as user don't > > > have > > > possibility to choose vlan id for network (at least that's default > policy > > > in > > > Neutron). > > > So, You should check what vlan id is used by such tenant network, > check if > > > this vlan is properly confiugred on Your switches and then, if all is > > > good, > > > check where exactly such dhcp requests are dropped (are they going from > > > compute node? are packets comming to the node with dhcp agent?). Base > on > > > that > > > You can narrow down where the issue can be. > > > > > > On sobota, 28 sierpnia 2021 18:07:07 CEST Karera Tony wrote: > > > > How can i make them both vxlan and vlan > > > > > > > > On Sat, 28 Aug 2021, 17:43 Mohammed Naser, > wrote: > > > > > Your issue is that tenant_network_types should be vxlan. > > > > > > > > > > On Sat, Aug 28, 2021 at 4:29 AM Karera Tony > > > > > > wrote: > > > > >> Hello Team, > > > > >> > > > > >> Whenever I create an internal network with type vlan. > > > > >> > > > > >> The Instances don't get IPs but for external Network it's fine. > > > > >> > > > > >> > > > > >> Below is the etc/kolla/config/neutron/ml2_conf.ini file > > > > >> > > > > >> > > > > >> [ml2] > > > > >> type_drivers = flat,vlan,vxlan > > > > >> tenant_network_types = vlan,flat > > > > >> mechanism_drivers = openvswitch,l2population > > > > >> > > > > >> [ml2_type_vlan] > > > > >> network_vlan_ranges = physnet1 > > > > >> [ml2_type_flat] > > > > >> flat_networks = physnet1 > > > > >> > > > > >> Is there something I need to change or I need to configure the > > > > > > interface > > > > > > > >> differently > > > > >> Regards > > > > >> > > > > >> Tony Karera > > > > >> > > > > >> > > > > >> -- > > > > > > > > > > Mohammed Naser > > > > > VEXXHOST, Inc. > > > > > > -- > > > Slawek Kaplonski > > > Principal Software Engineer > > > Red Hat > > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From syedammad83 at gmail.com Wed Sep 8 15:36:54 2021 From: syedammad83 at gmail.com (Ammad Syed) Date: Wed, 8 Sep 2021 20:36:54 +0500 Subject: Tenant vlan Network Issue In-Reply-To: References: <5528888.crQR7OUkfV@p1> <3773315.Oo1np0eLaN@p1> Message-ID: Hi karera, I don’t think your the error is related to network side. Its because kublet is not installed properly in your kubernetes master node. You need to check logs that why hyperkube services are not started. You can check it in heat logs on mater node and you can check the status of kubelet container via podman ps commad. Ammad On Wed, Sep 8, 2021 at 8:28 PM Karera Tony wrote: > Hello Slawek, > > Unfortunately I am not sure because all I did is configure the cluster. > Nothing else. > > > Regards > > Tony Karera > > > > > On Wed, Sep 8, 2021 at 3:31 PM Slawek Kaplonski > wrote: > >> Hi, >> >> On środa, 8 września 2021 15:16:57 CEST Karera Tony wrote: >> > Hello Slawek, >> > >> > I really need your help. >> > >> > I have tried all I can but still the k8s-cluster gets stuck with this >> log >> > here >> > >> > root at k8s-cluster-1-vo5s5jvtjoa7-master-0 core]# cd >> > /var/log/heat-config/heat-config-script/ >> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# ls >> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >> kube_cluster_c >> > onfig-asfqq352zosg.log >> > c42dcbf1-7812-49a2-b312-96c24225b407-k8s-cluster-1-vo5s5jvtjoa7- >> kube_masters- >> > waugw6vksw5p-0-vrt2gkfugedg-master_config-xv3vlmr27r46.log >> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# cat >> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >> kube_cluster_c >> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >> > heat-config-script]# cat >> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >> kube_cluster_c >> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >> > heat-config-script]# cat >> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >> kube_cluster_c >> > onfig-asfqq352zosg.log Starting to run kube-apiserver-to-kubelet-role >> > + echo 'Waiting for Kubernetes API...' >> > Waiting for Kubernetes API... >> > ++ kubectl get --raw=/healthz >> > The connection to the server localhost:8080 was refused - did you >> specify >> > the right host or port? >> > + '[' ok = '' ']' >> > + sleep 5 >> > ++ kubectl get --raw=/healthz >> > The connection to the server localhost:8080 was refused - did you >> specify >> > the right host or port? >> >> But why Your script is trying to connect to localhost? It's not the VM >> address >> I guess :) >> >> > + '[' ok = '' ']' >> > + sleep 5 >> > >> > >> > Regards >> > >> > Tony Karera >> > >> > >> > >> > >> > On Mon, Aug 30, 2021 at 8:50 AM Slawek Kaplonski >> > >> > wrote: >> > > Hi, >> > > >> > > You can use vlan as tenant network too. But You have to remember that >> vlan >> > > is >> > > provider network, which means that You need to have configured vlans >> on >> > > Your >> > > network infrastructure too. And in case of tenant network, Neutron >> will >> > > automatically allocate one of the vlan ids to the network as user >> don't >> > > have >> > > possibility to choose vlan id for network (at least that's default >> policy >> > > in >> > > Neutron). >> > > So, You should check what vlan id is used by such tenant network, >> check if >> > > this vlan is properly confiugred on Your switches and then, if all is >> > > good, >> > > check where exactly such dhcp requests are dropped (are they going >> from >> > > compute node? are packets comming to the node with dhcp agent?). Base >> on >> > > that >> > > You can narrow down where the issue can be. >> > > >> > > On sobota, 28 sierpnia 2021 18:07:07 CEST Karera Tony wrote: >> > > > How can i make them both vxlan and vlan >> > > > >> > > > On Sat, 28 Aug 2021, 17:43 Mohammed Naser, >> wrote: >> > > > > Your issue is that tenant_network_types should be vxlan. >> > > > > >> > > > > On Sat, Aug 28, 2021 at 4:29 AM Karera Tony > > >> > > >> > > wrote: >> > > > >> Hello Team, >> > > > >> >> > > > >> Whenever I create an internal network with type vlan. >> > > > >> >> > > > >> The Instances don't get IPs but for external Network it's fine. >> > > > >> >> > > > >> >> > > > >> Below is the etc/kolla/config/neutron/ml2_conf.ini file >> > > > >> >> > > > >> >> > > > >> [ml2] >> > > > >> type_drivers = flat,vlan,vxlan >> > > > >> tenant_network_types = vlan,flat >> > > > >> mechanism_drivers = openvswitch,l2population >> > > > >> >> > > > >> [ml2_type_vlan] >> > > > >> network_vlan_ranges = physnet1 >> > > > >> [ml2_type_flat] >> > > > >> flat_networks = physnet1 >> > > > >> >> > > > >> Is there something I need to change or I need to configure the >> > > >> > > interface >> > > >> > > > >> differently >> > > > >> Regards >> > > > >> >> > > > >> Tony Karera >> > > > >> >> > > > >> >> > > > >> -- >> > > > > >> > > > > Mohammed Naser >> > > > > VEXXHOST, Inc. >> > > >> > > -- >> > > Slawek Kaplonski >> > > Principal Software Engineer >> > > Red Hat >> >> >> -- >> Slawek Kaplonski >> Principal Software Engineer >> Red Hat > > -- Regards, Syed Ammad Ali -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnsomor at gmail.com Wed Sep 8 15:48:30 2021 From: johnsomor at gmail.com (Michael Johnson) Date: Wed, 8 Sep 2021 08:48:30 -0700 Subject: [designate][interop] test_create_all_recordset_types ddt issue In-Reply-To: References: Message-ID: Hi Martin, This is a really good question, thanks for bringing it up. Normally I would not include "ddt" tests in the interop test suite. I have two reasons for that opinion: 1. All DDT tests will end up with the same idempotent_id (UUID for the test), which makes them hard to track. 2. The list of tests generated from that "ddt" data will change over time as we add new recordset types [1]. We will also need to be very careful that the dataset order does not change. That said, those are important tests for the functionality of the Designate service. Users will expect to be able to create records of those types if they are using a "OpenStack Powered Platform with DNS" cloud. So, yes, I think we should keep them. I would expect the list to expand over time, with [1] being a candidate for that in the near future. If the use of "ddt" is a big problem for the compliance testing, we can consider breaking these out into individual tests (likely to be duplicates to the existing "ddt" tests to maintain the legacy test UUIDs). Michael [1] https://review.opendev.org/c/openstack/designate/+/802190 On Wed, Sep 8, 2021 at 7:28 AM Martin Kopec wrote: > > Hi designate team, > > I'm reaching out because we have encountered a problem with one of the tests [1]. We track the following name [2]: > designate_tempest_plugin.tests.api.v2.test_recordset.RecordsetsTest.test_create_all_recordset_types > > That test name is actually incorrect due to the ddt and the real name of the test is created dynamically based on the test data passed via ddt. > > We have a consistency check which is supposed to verify that the test names tracked in our guidelines match the ones in Tempest/tempest plugins, however, there was a bug - we are fixing that by [3]. > > The question is, should interop track all the permutations of the test? See [4]. > > [1] https://opendev.org/openstack/designate-tempest-plugin/src/commit/da27a70ae2b39695ef6f03bbefb55afeacf1cdf3/designate_tempest_plugin/tests/api/v2/test_recordset.py#L87 > [2] https://opendev.org/osf/interop/src/commit/d8eac71fa507c68e9c1f83a19dd77af8483d01b6/add-ons/guidelines/dns.2020.11.json#L116-L117 > [3] https://review.opendev.org/c/osf/interop/+/806598 > [4] https://review.opendev.org/c/osf/interop/+/807586 > > Thank you, > -- > Martin Kopec > Senior Software Quality Engineer > Red Hat EMEA > > > From rosmaita.fossdev at gmail.com Wed Sep 8 16:00:17 2021 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Wed, 8 Sep 2021 12:00:17 -0400 Subject: [ops][cinder] is anyone using RateLimitingMiddleware ? Message-ID: Hello operators, Takashi Kajinami noticed that cinder's RateLimitingMiddleware class lives among the legacy v2 classes and should be moved to live with the v3 classes, but then it occurred to him that maybe it should simply be deprecated and removed [0]. There are a few reasons to believe that nobody is actually using this functionality, for example, it's not included in the default api-paste.ini file, this middleware no longer exists in Nova, it gives you inexact behavior with multiple API nodes, etc. We discussed this topic at today's cinder meeting and decided to ask whether anyone is using the RateLimitingMiddleware with cinder before deprecating it for removal. If it's going to be deprecated, we'd like to do it in Xena, so please reply to this email or leave a comment on the deprecation patch [1] (or preferably both) before 23:59 UTC on Tuesday 14 September 2021. Thank you! [0] https://bugs.launchpad.net/cinder/+bug/1942696 [1] https://review.opendev.org/c/openstack/cinder/+/807469 From wodel.youchi at gmail.com Wed Sep 8 16:13:30 2021 From: wodel.youchi at gmail.com (wodel youchi) Date: Wed, 8 Sep 2021 17:13:30 +0100 Subject: Instances cannot ping each other and cannot ping virtual-router Message-ID: Hi, I deployed OpenStack Train using TripleO using this tutorial : https://kdjlab.com/deploying-rdo-in-a-cohesive-manner/ and the documentation of TripleO. I deployed it with DVR. In my deployment I am using virtual machines with nested-kvm. The deployment went well, I am using network isolation like this : - nic1 : provisioning - nic2 and nic3 (bond0) storage and storage mgmt networks, each one in it's VLAN - nic3 and nic5 (bond1) tenant, api and *external* (10.0.2.0/24 VLAN2100) networks, each one in it's VLAN In my physical host (the bare metal KVM) I created a bridge which handles the provisioning, tenant, api and external networks. I created a private tenant network (172.16.100.0/24). openstack network create private neutron subnet-create private 172.16.100.0/24 --name private-sub --dns-nameserver 172.16.0.252 I created a public network and I attached it to the external network using the same VLAN tag (10.0.2.0/24 VLAN 2100, pool: 10.0.2.100-10.0.2.120) : *openstack network create --provider-network-type vlan --provider-physical-network datacentre --provider-segment 2100 --external public* neutron subnet-create public 10.0.2.0/24 --name public-sub --disable-dhcp --allocation-pool=start=10.0.2.100,end=10.0.2.120 --gateway=10.0.2.1 --dns-nameserver 172.16.0.252 I created a vrouter, one port in the public network and the other in the private network. I created two cirrus instances, each one got it's ip address from the private network. I found : cirrus-1 : 172.16.100.81 cirrus-2 : 172.16.100.103 vrouter : 172.16.100.1 private : 10.0.2.101 external neutron:dhcp : 172.16.100.2 The problems : - The instances cannot ping each other. - The instances cannot ping the vrouter. - I cannot ping the public vrouter interface. But both instances can ping neutron:dhcp Could someone help me dig into this. Thanks in advance, Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ltoscano at redhat.com Wed Sep 8 16:15:18 2021 From: ltoscano at redhat.com (Luigi Toscano) Date: Wed, 08 Sep 2021 18:15:18 +0200 Subject: [designate][interop] test_create_all_recordset_types ddt issue In-Reply-To: References: Message-ID: <3229108.usfYGdeWWP@whitebase.usersys.redhat.com> On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote: > If the use of "ddt" is a big problem for the compliance testing, we > can consider breaking these out into individual tests (likely to be > duplicates to the existing "ddt" tests to maintain the legacy test > UUIDs). I was wondering: wouldn't it be possible to expand or use ddt somehow to inject different UUIDs for each generated test? You know in advance how many tests are going to be generated. -- Luigi From syedammad83 at gmail.com Wed Sep 8 16:20:53 2021 From: syedammad83 at gmail.com (Ammad Syed) Date: Wed, 8 Sep 2021 21:20:53 +0500 Subject: Tenant vlan Network Issue In-Reply-To: References: <5528888.crQR7OUkfV@p1> <3773315.Oo1np0eLaN@p1> Message-ID: You can see here that kubelet container is not started. You need to check kubelet logs. Can you confirm which fcos version you are using? Ammad On Wed, Sep 8, 2021 at 9:17 PM Karera Tony wrote: > Hello Team, > > I tried to create another cluster with tls enabled and below are the > results for podman ps . I have also attached the heat logs > > podman ps > CONTAINER ID IMAGE > COMMAND CREATED STATUS PORTS > NAMES > e9c028723166 > docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1 > /usr/bin/start-he... 34 minutes ago Up 34 minutes ago > heat-container-agent > 4128a9e951de k8s.gcr.io/hyperkube:v1.18.16 > kube-apiserver --... 30 minutes ago Up 30 minutes ago > kube-apiserver > 5430aa72e7f0 k8s.gcr.io/hyperkube:v1.18.16 > kube-controller-m... 30 minutes ago Up 30 minutes ago > kube-controller-manager > 8f2c76272916 k8s.gcr.io/hyperkube:v1.18.16 > kube-scheduler --... 30 minutes ago Up 30 minutes ago > kube-scheduler > af5f1a661c65 quay.io/coreos/etcd:v3.4.6 > /usr/local/bin/et... 30 minutes ago Up 30 minutes ago > etcd > > Regards > > Tony Karera > > > > > On Wed, Sep 8, 2021 at 5:37 PM Ammad Syed wrote: > >> Hi karera, >> >> I don’t think your the error is related to network side. Its because >> kublet is not installed properly in your kubernetes master node. You need >> to check logs that why hyperkube services are not started. >> >> You can check it in heat logs on mater node and you can check the status >> of kubelet container via podman ps commad. >> >> Ammad >> On Wed, Sep 8, 2021 at 8:28 PM Karera Tony wrote: >> >>> Hello Slawek, >>> >>> Unfortunately I am not sure because all I did is configure the cluster. >>> Nothing else. >>> >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Wed, Sep 8, 2021 at 3:31 PM Slawek Kaplonski >>> wrote: >>> >>>> Hi, >>>> >>>> On środa, 8 września 2021 15:16:57 CEST Karera Tony wrote: >>>> > Hello Slawek, >>>> > >>>> > I really need your help. >>>> > >>>> > I have tried all I can but still the k8s-cluster gets stuck with this >>>> log >>>> > here >>>> > >>>> > root at k8s-cluster-1-vo5s5jvtjoa7-master-0 core]# cd >>>> > /var/log/heat-config/heat-config-script/ >>>> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# ls >>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>> kube_cluster_c >>>> > onfig-asfqq352zosg.log >>>> > c42dcbf1-7812-49a2-b312-96c24225b407-k8s-cluster-1-vo5s5jvtjoa7- >>>> kube_masters- >>>> > waugw6vksw5p-0-vrt2gkfugedg-master_config-xv3vlmr27r46.log >>>> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# cat >>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>> kube_cluster_c >>>> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >>>> > heat-config-script]# cat >>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>> kube_cluster_c >>>> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >>>> > heat-config-script]# cat >>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>> kube_cluster_c >>>> > onfig-asfqq352zosg.log Starting to run kube-apiserver-to-kubelet-role >>>> > + echo 'Waiting for Kubernetes API...' >>>> > Waiting for Kubernetes API... >>>> > ++ kubectl get --raw=/healthz >>>> > The connection to the server localhost:8080 was refused - did you >>>> specify >>>> > the right host or port? >>>> > + '[' ok = '' ']' >>>> > + sleep 5 >>>> > ++ kubectl get --raw=/healthz >>>> > The connection to the server localhost:8080 was refused - did you >>>> specify >>>> > the right host or port? >>>> >>>> But why Your script is trying to connect to localhost? It's not the VM >>>> address >>>> I guess :) >>>> >>>> > + '[' ok = '' ']' >>>> > + sleep 5 >>>> > >>>> > >>>> > Regards >>>> > >>>> > Tony Karera >>>> > >>>> > >>>> > >>>> > >>>> > On Mon, Aug 30, 2021 at 8:50 AM Slawek Kaplonski >>> > >>>> > >>>> > wrote: >>>> > > Hi, >>>> > > >>>> > > You can use vlan as tenant network too. But You have to remember >>>> that vlan >>>> > > is >>>> > > provider network, which means that You need to have configured >>>> vlans on >>>> > > Your >>>> > > network infrastructure too. And in case of tenant network, Neutron >>>> will >>>> > > automatically allocate one of the vlan ids to the network as user >>>> don't >>>> > > have >>>> > > possibility to choose vlan id for network (at least that's default >>>> policy >>>> > > in >>>> > > Neutron). >>>> > > So, You should check what vlan id is used by such tenant network, >>>> check if >>>> > > this vlan is properly confiugred on Your switches and then, if all >>>> is >>>> > > good, >>>> > > check where exactly such dhcp requests are dropped (are they going >>>> from >>>> > > compute node? are packets comming to the node with dhcp agent?). >>>> Base on >>>> > > that >>>> > > You can narrow down where the issue can be. >>>> > > >>>> > > On sobota, 28 sierpnia 2021 18:07:07 CEST Karera Tony wrote: >>>> > > > How can i make them both vxlan and vlan >>>> > > > >>>> > > > On Sat, 28 Aug 2021, 17:43 Mohammed Naser, >>>> wrote: >>>> > > > > Your issue is that tenant_network_types should be vxlan. >>>> > > > > >>>> > > > > On Sat, Aug 28, 2021 at 4:29 AM Karera Tony < >>>> tonykarera at gmail.com> >>>> > > >>>> > > wrote: >>>> > > > >> Hello Team, >>>> > > > >> >>>> > > > >> Whenever I create an internal network with type vlan. >>>> > > > >> >>>> > > > >> The Instances don't get IPs but for external Network it's fine. >>>> > > > >> >>>> > > > >> >>>> > > > >> Below is the etc/kolla/config/neutron/ml2_conf.ini file >>>> > > > >> >>>> > > > >> >>>> > > > >> [ml2] >>>> > > > >> type_drivers = flat,vlan,vxlan >>>> > > > >> tenant_network_types = vlan,flat >>>> > > > >> mechanism_drivers = openvswitch,l2population >>>> > > > >> >>>> > > > >> [ml2_type_vlan] >>>> > > > >> network_vlan_ranges = physnet1 >>>> > > > >> [ml2_type_flat] >>>> > > > >> flat_networks = physnet1 >>>> > > > >> >>>> > > > >> Is there something I need to change or I need to configure the >>>> > > >>>> > > interface >>>> > > >>>> > > > >> differently >>>> > > > >> Regards >>>> > > > >> >>>> > > > >> Tony Karera >>>> > > > >> >>>> > > > >> >>>> > > > >> -- >>>> > > > > >>>> > > > > Mohammed Naser >>>> > > > > VEXXHOST, Inc. >>>> > > >>>> > > -- >>>> > > Slawek Kaplonski >>>> > > Principal Software Engineer >>>> > > Red Hat >>>> >>>> >>>> -- >>>> Slawek Kaplonski >>>> Principal Software Engineer >>>> Red Hat >>> >>> -- >> Regards, >> >> >> Syed Ammad Ali >> > -- Regards, Syed Ammad Ali -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 8 16:33:07 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 8 Sep 2021 18:33:07 +0200 Subject: Tenant vlan Network Issue In-Reply-To: References: <5528888.crQR7OUkfV@p1> <3773315.Oo1np0eLaN@p1> Message-ID: Hello Ammad, Yes I can see. I cant see the kubelet logs. Please advise on how to check the fcos Logs. It is new to me unfortunately. Regards Tony Karera On Wed, Sep 8, 2021 at 6:21 PM Ammad Syed wrote: > You can see here that kubelet container is not started. You need to check > kubelet logs. > > Can you confirm which fcos version you are using? > > Ammad > On Wed, Sep 8, 2021 at 9:17 PM Karera Tony wrote: > >> Hello Team, >> >> I tried to create another cluster with tls enabled and below are the >> results for podman ps . I have also attached the heat logs >> >> podman ps >> CONTAINER ID IMAGE >> COMMAND CREATED STATUS PORTS >> NAMES >> e9c028723166 >> docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1 >> /usr/bin/start-he... 34 minutes ago Up 34 minutes ago >> heat-container-agent >> 4128a9e951de k8s.gcr.io/hyperkube:v1.18.16 >> kube-apiserver --... 30 minutes ago Up 30 minutes ago >> kube-apiserver >> 5430aa72e7f0 k8s.gcr.io/hyperkube:v1.18.16 >> kube-controller-m... 30 minutes ago Up 30 minutes ago >> kube-controller-manager >> 8f2c76272916 k8s.gcr.io/hyperkube:v1.18.16 >> kube-scheduler --... 30 minutes ago Up 30 minutes ago >> kube-scheduler >> af5f1a661c65 quay.io/coreos/etcd:v3.4.6 >> /usr/local/bin/et... 30 minutes ago Up 30 minutes ago >> etcd >> >> Regards >> >> Tony Karera >> >> >> >> >> On Wed, Sep 8, 2021 at 5:37 PM Ammad Syed wrote: >> >>> Hi karera, >>> >>> I don’t think your the error is related to network side. Its because >>> kublet is not installed properly in your kubernetes master node. You need >>> to check logs that why hyperkube services are not started. >>> >>> You can check it in heat logs on mater node and you can check the status >>> of kubelet container via podman ps commad. >>> >>> Ammad >>> On Wed, Sep 8, 2021 at 8:28 PM Karera Tony wrote: >>> >>>> Hello Slawek, >>>> >>>> Unfortunately I am not sure because all I did is configure the cluster. >>>> Nothing else. >>>> >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Wed, Sep 8, 2021 at 3:31 PM Slawek Kaplonski >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> On środa, 8 września 2021 15:16:57 CEST Karera Tony wrote: >>>>> > Hello Slawek, >>>>> > >>>>> > I really need your help. >>>>> > >>>>> > I have tried all I can but still the k8s-cluster gets stuck with >>>>> this log >>>>> > here >>>>> > >>>>> > root at k8s-cluster-1-vo5s5jvtjoa7-master-0 core]# cd >>>>> > /var/log/heat-config/heat-config-script/ >>>>> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# ls >>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>> kube_cluster_c >>>>> > onfig-asfqq352zosg.log >>>>> > c42dcbf1-7812-49a2-b312-96c24225b407-k8s-cluster-1-vo5s5jvtjoa7- >>>>> kube_masters- >>>>> > waugw6vksw5p-0-vrt2gkfugedg-master_config-xv3vlmr27r46.log >>>>> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# cat >>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>> kube_cluster_c >>>>> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >>>>> > heat-config-script]# cat >>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>> kube_cluster_c >>>>> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >>>>> > heat-config-script]# cat >>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>> kube_cluster_c >>>>> > onfig-asfqq352zosg.log Starting to run kube-apiserver-to-kubelet-role >>>>> > + echo 'Waiting for Kubernetes API...' >>>>> > Waiting for Kubernetes API... >>>>> > ++ kubectl get --raw=/healthz >>>>> > The connection to the server localhost:8080 was refused - did you >>>>> specify >>>>> > the right host or port? >>>>> > + '[' ok = '' ']' >>>>> > + sleep 5 >>>>> > ++ kubectl get --raw=/healthz >>>>> > The connection to the server localhost:8080 was refused - did you >>>>> specify >>>>> > the right host or port? >>>>> >>>>> But why Your script is trying to connect to localhost? It's not the VM >>>>> address >>>>> I guess :) >>>>> >>>>> > + '[' ok = '' ']' >>>>> > + sleep 5 >>>>> > >>>>> > >>>>> > Regards >>>>> > >>>>> > Tony Karera >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > On Mon, Aug 30, 2021 at 8:50 AM Slawek Kaplonski < >>>>> skaplons at redhat.com> >>>>> > >>>>> > wrote: >>>>> > > Hi, >>>>> > > >>>>> > > You can use vlan as tenant network too. But You have to remember >>>>> that vlan >>>>> > > is >>>>> > > provider network, which means that You need to have configured >>>>> vlans on >>>>> > > Your >>>>> > > network infrastructure too. And in case of tenant network, Neutron >>>>> will >>>>> > > automatically allocate one of the vlan ids to the network as user >>>>> don't >>>>> > > have >>>>> > > possibility to choose vlan id for network (at least that's default >>>>> policy >>>>> > > in >>>>> > > Neutron). >>>>> > > So, You should check what vlan id is used by such tenant network, >>>>> check if >>>>> > > this vlan is properly confiugred on Your switches and then, if all >>>>> is >>>>> > > good, >>>>> > > check where exactly such dhcp requests are dropped (are they going >>>>> from >>>>> > > compute node? are packets comming to the node with dhcp agent?). >>>>> Base on >>>>> > > that >>>>> > > You can narrow down where the issue can be. >>>>> > > >>>>> > > On sobota, 28 sierpnia 2021 18:07:07 CEST Karera Tony wrote: >>>>> > > > How can i make them both vxlan and vlan >>>>> > > > >>>>> > > > On Sat, 28 Aug 2021, 17:43 Mohammed Naser, >>>>> wrote: >>>>> > > > > Your issue is that tenant_network_types should be vxlan. >>>>> > > > > >>>>> > > > > On Sat, Aug 28, 2021 at 4:29 AM Karera Tony < >>>>> tonykarera at gmail.com> >>>>> > > >>>>> > > wrote: >>>>> > > > >> Hello Team, >>>>> > > > >> >>>>> > > > >> Whenever I create an internal network with type vlan. >>>>> > > > >> >>>>> > > > >> The Instances don't get IPs but for external Network it's >>>>> fine. >>>>> > > > >> >>>>> > > > >> >>>>> > > > >> Below is the etc/kolla/config/neutron/ml2_conf.ini file >>>>> > > > >> >>>>> > > > >> >>>>> > > > >> [ml2] >>>>> > > > >> type_drivers = flat,vlan,vxlan >>>>> > > > >> tenant_network_types = vlan,flat >>>>> > > > >> mechanism_drivers = openvswitch,l2population >>>>> > > > >> >>>>> > > > >> [ml2_type_vlan] >>>>> > > > >> network_vlan_ranges = physnet1 >>>>> > > > >> [ml2_type_flat] >>>>> > > > >> flat_networks = physnet1 >>>>> > > > >> >>>>> > > > >> Is there something I need to change or I need to configure the >>>>> > > >>>>> > > interface >>>>> > > >>>>> > > > >> differently >>>>> > > > >> Regards >>>>> > > > >> >>>>> > > > >> Tony Karera >>>>> > > > >> >>>>> > > > >> >>>>> > > > >> -- >>>>> > > > > >>>>> > > > > Mohammed Naser >>>>> > > > > VEXXHOST, Inc. >>>>> > > >>>>> > > -- >>>>> > > Slawek Kaplonski >>>>> > > Principal Software Engineer >>>>> > > Red Hat >>>>> >>>>> >>>>> -- >>>>> Slawek Kaplonski >>>>> Principal Software Engineer >>>>> Red Hat >>>> >>>> -- >>> Regards, >>> >>> >>> Syed Ammad Ali >>> >> -- > Regards, > > > Syed Ammad Ali > -------------- next part -------------- An HTML attachment was scrubbed... URL: From syedammad83 at gmail.com Wed Sep 8 16:59:44 2021 From: syedammad83 at gmail.com (Ammad Syed) Date: Wed, 8 Sep 2021 21:59:44 +0500 Subject: Tenant vlan Network Issue In-Reply-To: References: <5528888.crQR7OUkfV@p1> <3773315.Oo1np0eLaN@p1> Message-ID: Can you confirm fedora coreOS version image you are using for magnum clusters ? I suspect you are using 34. Which have trouble with kubelet service. You should use version 33 or 32. You can check systemctl status kubelet service and its logs. On Wed, Sep 8, 2021 at 9:33 PM Karera Tony wrote: > Hello Ammad, > > Yes I can see. > > I cant see the kubelet logs. Please advise on how to check the fcos Logs. > > It is new to me unfortunately. > > Regards > > Tony Karera > > > > > On Wed, Sep 8, 2021 at 6:21 PM Ammad Syed wrote: > >> You can see here that kubelet container is not started. You need to check >> kubelet logs. >> >> Can you confirm which fcos version you are using? >> >> Ammad >> On Wed, Sep 8, 2021 at 9:17 PM Karera Tony wrote: >> >>> Hello Team, >>> >>> I tried to create another cluster with tls enabled and below are the >>> results for podman ps . I have also attached the heat logs >>> >>> podman ps >>> CONTAINER ID IMAGE >>> COMMAND CREATED STATUS PORTS >>> NAMES >>> e9c028723166 >>> docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1 >>> /usr/bin/start-he... 34 minutes ago Up 34 minutes ago >>> heat-container-agent >>> 4128a9e951de k8s.gcr.io/hyperkube:v1.18.16 >>> kube-apiserver --... 30 minutes ago Up 30 minutes ago >>> kube-apiserver >>> 5430aa72e7f0 k8s.gcr.io/hyperkube:v1.18.16 >>> kube-controller-m... 30 minutes ago Up 30 minutes ago >>> kube-controller-manager >>> 8f2c76272916 k8s.gcr.io/hyperkube:v1.18.16 >>> kube-scheduler --... 30 minutes ago Up 30 minutes ago >>> kube-scheduler >>> af5f1a661c65 quay.io/coreos/etcd:v3.4.6 >>> /usr/local/bin/et... 30 minutes ago Up 30 minutes ago >>> etcd >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Wed, Sep 8, 2021 at 5:37 PM Ammad Syed wrote: >>> >>>> Hi karera, >>>> >>>> I don’t think your the error is related to network side. Its because >>>> kublet is not installed properly in your kubernetes master node. You need >>>> to check logs that why hyperkube services are not started. >>>> >>>> You can check it in heat logs on mater node and you can check the >>>> status of kubelet container via podman ps commad. >>>> >>>> Ammad >>>> On Wed, Sep 8, 2021 at 8:28 PM Karera Tony >>>> wrote: >>>> >>>>> Hello Slawek, >>>>> >>>>> Unfortunately I am not sure because all I did is configure the >>>>> cluster. Nothing else. >>>>> >>>>> >>>>> Regards >>>>> >>>>> Tony Karera >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Sep 8, 2021 at 3:31 PM Slawek Kaplonski >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> On środa, 8 września 2021 15:16:57 CEST Karera Tony wrote: >>>>>> > Hello Slawek, >>>>>> > >>>>>> > I really need your help. >>>>>> > >>>>>> > I have tried all I can but still the k8s-cluster gets stuck with >>>>>> this log >>>>>> > here >>>>>> > >>>>>> > root at k8s-cluster-1-vo5s5jvtjoa7-master-0 core]# cd >>>>>> > /var/log/heat-config/heat-config-script/ >>>>>> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# ls >>>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>>> kube_cluster_c >>>>>> > onfig-asfqq352zosg.log >>>>>> > c42dcbf1-7812-49a2-b312-96c24225b407-k8s-cluster-1-vo5s5jvtjoa7- >>>>>> kube_masters- >>>>>> > waugw6vksw5p-0-vrt2gkfugedg-master_config-xv3vlmr27r46.log >>>>>> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# cat >>>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>>> kube_cluster_c >>>>>> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >>>>>> > heat-config-script]# cat >>>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>>> kube_cluster_c >>>>>> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >>>>>> > heat-config-script]# cat >>>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>>> kube_cluster_c >>>>>> > onfig-asfqq352zosg.log Starting to run >>>>>> kube-apiserver-to-kubelet-role >>>>>> > + echo 'Waiting for Kubernetes API...' >>>>>> > Waiting for Kubernetes API... >>>>>> > ++ kubectl get --raw=/healthz >>>>>> > The connection to the server localhost:8080 was refused - did you >>>>>> specify >>>>>> > the right host or port? >>>>>> > + '[' ok = '' ']' >>>>>> > + sleep 5 >>>>>> > ++ kubectl get --raw=/healthz >>>>>> > The connection to the server localhost:8080 was refused - did you >>>>>> specify >>>>>> > the right host or port? >>>>>> >>>>>> But why Your script is trying to connect to localhost? It's not the >>>>>> VM address >>>>>> I guess :) >>>>>> >>>>>> > + '[' ok = '' ']' >>>>>> > + sleep 5 >>>>>> > >>>>>> > >>>>>> > Regards >>>>>> > >>>>>> > Tony Karera >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > On Mon, Aug 30, 2021 at 8:50 AM Slawek Kaplonski < >>>>>> skaplons at redhat.com> >>>>>> > >>>>>> > wrote: >>>>>> > > Hi, >>>>>> > > >>>>>> > > You can use vlan as tenant network too. But You have to remember >>>>>> that vlan >>>>>> > > is >>>>>> > > provider network, which means that You need to have configured >>>>>> vlans on >>>>>> > > Your >>>>>> > > network infrastructure too. And in case of tenant network, >>>>>> Neutron will >>>>>> > > automatically allocate one of the vlan ids to the network as user >>>>>> don't >>>>>> > > have >>>>>> > > possibility to choose vlan id for network (at least that's >>>>>> default policy >>>>>> > > in >>>>>> > > Neutron). >>>>>> > > So, You should check what vlan id is used by such tenant network, >>>>>> check if >>>>>> > > this vlan is properly confiugred on Your switches and then, if >>>>>> all is >>>>>> > > good, >>>>>> > > check where exactly such dhcp requests are dropped (are they >>>>>> going from >>>>>> > > compute node? are packets comming to the node with dhcp agent?). >>>>>> Base on >>>>>> > > that >>>>>> > > You can narrow down where the issue can be. >>>>>> > > >>>>>> > > On sobota, 28 sierpnia 2021 18:07:07 CEST Karera Tony wrote: >>>>>> > > > How can i make them both vxlan and vlan >>>>>> > > > >>>>>> > > > On Sat, 28 Aug 2021, 17:43 Mohammed Naser, >>>>>> wrote: >>>>>> > > > > Your issue is that tenant_network_types should be vxlan. >>>>>> > > > > >>>>>> > > > > On Sat, Aug 28, 2021 at 4:29 AM Karera Tony < >>>>>> tonykarera at gmail.com> >>>>>> > > >>>>>> > > wrote: >>>>>> > > > >> Hello Team, >>>>>> > > > >> >>>>>> > > > >> Whenever I create an internal network with type vlan. >>>>>> > > > >> >>>>>> > > > >> The Instances don't get IPs but for external Network it's >>>>>> fine. >>>>>> > > > >> >>>>>> > > > >> >>>>>> > > > >> Below is the etc/kolla/config/neutron/ml2_conf.ini file >>>>>> > > > >> >>>>>> > > > >> >>>>>> > > > >> [ml2] >>>>>> > > > >> type_drivers = flat,vlan,vxlan >>>>>> > > > >> tenant_network_types = vlan,flat >>>>>> > > > >> mechanism_drivers = openvswitch,l2population >>>>>> > > > >> >>>>>> > > > >> [ml2_type_vlan] >>>>>> > > > >> network_vlan_ranges = physnet1 >>>>>> > > > >> [ml2_type_flat] >>>>>> > > > >> flat_networks = physnet1 >>>>>> > > > >> >>>>>> > > > >> Is there something I need to change or I need to configure >>>>>> the >>>>>> > > >>>>>> > > interface >>>>>> > > >>>>>> > > > >> differently >>>>>> > > > >> Regards >>>>>> > > > >> >>>>>> > > > >> Tony Karera >>>>>> > > > >> >>>>>> > > > >> >>>>>> > > > >> -- >>>>>> > > > > >>>>>> > > > > Mohammed Naser >>>>>> > > > > VEXXHOST, Inc. >>>>>> > > >>>>>> > > -- >>>>>> > > Slawek Kaplonski >>>>>> > > Principal Software Engineer >>>>>> > > Red Hat >>>>>> >>>>>> >>>>>> -- >>>>>> Slawek Kaplonski >>>>>> Principal Software Engineer >>>>>> Red Hat >>>>> >>>>> -- >>>> Regards, >>>> >>>> >>>> Syed Ammad Ali >>>> >>> -- >> Regards, >> >> >> Syed Ammad Ali >> > -- Regards, Syed Ammad Ali -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 8 16:17:42 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 8 Sep 2021 18:17:42 +0200 Subject: Tenant vlan Network Issue In-Reply-To: References: <5528888.crQR7OUkfV@p1> <3773315.Oo1np0eLaN@p1> Message-ID: Hello Team, I tried to create another cluster with tls enabled and below are the results for podman ps . I have also attached the heat logs podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e9c028723166 docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1 /usr/bin/start-he... 34 minutes ago Up 34 minutes ago heat-container-agent 4128a9e951de k8s.gcr.io/hyperkube:v1.18.16 kube-apiserver --... 30 minutes ago Up 30 minutes ago kube-apiserver 5430aa72e7f0 k8s.gcr.io/hyperkube:v1.18.16 kube-controller-m... 30 minutes ago Up 30 minutes ago kube-controller-manager 8f2c76272916 k8s.gcr.io/hyperkube:v1.18.16 kube-scheduler --... 30 minutes ago Up 30 minutes ago kube-scheduler af5f1a661c65 quay.io/coreos/etcd:v3.4.6 /usr/local/bin/et... 30 minutes ago Up 30 minutes ago etcd Regards Tony Karera On Wed, Sep 8, 2021 at 5:37 PM Ammad Syed wrote: > Hi karera, > > I don’t think your the error is related to network side. Its because > kublet is not installed properly in your kubernetes master node. You need > to check logs that why hyperkube services are not started. > > You can check it in heat logs on mater node and you can check the status > of kubelet container via podman ps commad. > > Ammad > On Wed, Sep 8, 2021 at 8:28 PM Karera Tony wrote: > >> Hello Slawek, >> >> Unfortunately I am not sure because all I did is configure the cluster. >> Nothing else. >> >> >> Regards >> >> Tony Karera >> >> >> >> >> On Wed, Sep 8, 2021 at 3:31 PM Slawek Kaplonski >> wrote: >> >>> Hi, >>> >>> On środa, 8 września 2021 15:16:57 CEST Karera Tony wrote: >>> > Hello Slawek, >>> > >>> > I really need your help. >>> > >>> > I have tried all I can but still the k8s-cluster gets stuck with this >>> log >>> > here >>> > >>> > root at k8s-cluster-1-vo5s5jvtjoa7-master-0 core]# cd >>> > /var/log/heat-config/heat-config-script/ >>> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# ls >>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>> kube_cluster_c >>> > onfig-asfqq352zosg.log >>> > c42dcbf1-7812-49a2-b312-96c24225b407-k8s-cluster-1-vo5s5jvtjoa7- >>> kube_masters- >>> > waugw6vksw5p-0-vrt2gkfugedg-master_config-xv3vlmr27r46.log >>> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# cat >>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>> kube_cluster_c >>> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >>> > heat-config-script]# cat >>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>> kube_cluster_c >>> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >>> > heat-config-script]# cat >>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>> kube_cluster_c >>> > onfig-asfqq352zosg.log Starting to run kube-apiserver-to-kubelet-role >>> > + echo 'Waiting for Kubernetes API...' >>> > Waiting for Kubernetes API... >>> > ++ kubectl get --raw=/healthz >>> > The connection to the server localhost:8080 was refused - did you >>> specify >>> > the right host or port? >>> > + '[' ok = '' ']' >>> > + sleep 5 >>> > ++ kubectl get --raw=/healthz >>> > The connection to the server localhost:8080 was refused - did you >>> specify >>> > the right host or port? >>> >>> But why Your script is trying to connect to localhost? It's not the VM >>> address >>> I guess :) >>> >>> > + '[' ok = '' ']' >>> > + sleep 5 >>> > >>> > >>> > Regards >>> > >>> > Tony Karera >>> > >>> > >>> > >>> > >>> > On Mon, Aug 30, 2021 at 8:50 AM Slawek Kaplonski >>> > >>> > wrote: >>> > > Hi, >>> > > >>> > > You can use vlan as tenant network too. But You have to remember >>> that vlan >>> > > is >>> > > provider network, which means that You need to have configured vlans >>> on >>> > > Your >>> > > network infrastructure too. And in case of tenant network, Neutron >>> will >>> > > automatically allocate one of the vlan ids to the network as user >>> don't >>> > > have >>> > > possibility to choose vlan id for network (at least that's default >>> policy >>> > > in >>> > > Neutron). >>> > > So, You should check what vlan id is used by such tenant network, >>> check if >>> > > this vlan is properly confiugred on Your switches and then, if all is >>> > > good, >>> > > check where exactly such dhcp requests are dropped (are they going >>> from >>> > > compute node? are packets comming to the node with dhcp agent?). >>> Base on >>> > > that >>> > > You can narrow down where the issue can be. >>> > > >>> > > On sobota, 28 sierpnia 2021 18:07:07 CEST Karera Tony wrote: >>> > > > How can i make them both vxlan and vlan >>> > > > >>> > > > On Sat, 28 Aug 2021, 17:43 Mohammed Naser, >>> wrote: >>> > > > > Your issue is that tenant_network_types should be vxlan. >>> > > > > >>> > > > > On Sat, Aug 28, 2021 at 4:29 AM Karera Tony < >>> tonykarera at gmail.com> >>> > > >>> > > wrote: >>> > > > >> Hello Team, >>> > > > >> >>> > > > >> Whenever I create an internal network with type vlan. >>> > > > >> >>> > > > >> The Instances don't get IPs but for external Network it's fine. >>> > > > >> >>> > > > >> >>> > > > >> Below is the etc/kolla/config/neutron/ml2_conf.ini file >>> > > > >> >>> > > > >> >>> > > > >> [ml2] >>> > > > >> type_drivers = flat,vlan,vxlan >>> > > > >> tenant_network_types = vlan,flat >>> > > > >> mechanism_drivers = openvswitch,l2population >>> > > > >> >>> > > > >> [ml2_type_vlan] >>> > > > >> network_vlan_ranges = physnet1 >>> > > > >> [ml2_type_flat] >>> > > > >> flat_networks = physnet1 >>> > > > >> >>> > > > >> Is there something I need to change or I need to configure the >>> > > >>> > > interface >>> > > >>> > > > >> differently >>> > > > >> Regards >>> > > > >> >>> > > > >> Tony Karera >>> > > > >> >>> > > > >> >>> > > > >> -- >>> > > > > >>> > > > > Mohammed Naser >>> > > > > VEXXHOST, Inc. >>> > > >>> > > -- >>> > > Slawek Kaplonski >>> > > Principal Software Engineer >>> > > Red Hat >>> >>> >>> -- >>> Slawek Kaplonski >>> Principal Software Engineer >>> Red Hat >> >> -- > Regards, > > > Syed Ammad Ali > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 965d744e-2a63-4aec-9a65-ab83c92a47b4-k8s-cluster1-vnfx5pwx57gt-kube_masters-ikh4or2pa3o6-0-vawscctajrms-master_config-dchrxrgxwlou.log Type: application/octet-stream Size: 238556 bytes Desc: not available URL: From james.slagle at gmail.com Wed Sep 8 18:07:18 2021 From: james.slagle at gmail.com (James Slagle) Date: Wed, 8 Sep 2021 14:07:18 -0400 Subject: [TripleO] Yoga PTG topics etherpad Message-ID: I've started an etherpad where we can collect topics for discussion at the upcoming PTG: https://etherpad.opendev.org/p/tripleo-yoga-topics Please add topics over the next couple of weeks, and then we'll work on a schedule. Thanks. -- -- James Slagle -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From adivya1.singh at gmail.com Wed Sep 8 18:31:36 2021 From: adivya1.singh at gmail.com (Adivya Singh) Date: Thu, 9 Sep 2021 00:01:36 +0530 Subject: Steps for Controller Node Removal in Victoria Version Openstack using ansible Message-ID: Hello Team, Is there any way , We can have a link or guide to decommission a controller node We have 3 controller nodes and 4 compute nodes, and we are planning to Scale down the Infrastructure by 1 controller and 2 compute nodes Regards Adivya Singh -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 8 18:37:28 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 8 Sep 2021 20:37:28 +0200 Subject: Tenant vlan Network Issue In-Reply-To: References: <5528888.crQR7OUkfV@p1> <3773315.Oo1np0eLaN@p1> Message-ID: Hello Ammad, Thanks a lot Bro. I will really really appreciate it. The issue was on the Fedore-coreos I was using. I changed to 33 and everything worked OK. Thanks a lot everyone. Regards Regards Tony Karera On Wed, Sep 8, 2021 at 6:59 PM Ammad Syed wrote: > Can you confirm fedora coreOS version image you are using for magnum > clusters ? > > I suspect you are using 34. Which have trouble with kubelet service. You > should use version 33 or 32. > > You can check systemctl status kubelet service and its logs. > > On Wed, Sep 8, 2021 at 9:33 PM Karera Tony wrote: > >> Hello Ammad, >> >> Yes I can see. >> >> I cant see the kubelet logs. Please advise on how to check the fcos Logs. >> >> It is new to me unfortunately. >> >> Regards >> >> Tony Karera >> >> >> >> >> On Wed, Sep 8, 2021 at 6:21 PM Ammad Syed wrote: >> >>> You can see here that kubelet container is not started. You need to >>> check kubelet logs. >>> >>> Can you confirm which fcos version you are using? >>> >>> Ammad >>> On Wed, Sep 8, 2021 at 9:17 PM Karera Tony wrote: >>> >>>> Hello Team, >>>> >>>> I tried to create another cluster with tls enabled and below are the >>>> results for podman ps . I have also attached the heat logs >>>> >>>> podman ps >>>> CONTAINER ID IMAGE >>>> COMMAND CREATED STATUS PORTS >>>> NAMES >>>> e9c028723166 >>>> docker.io/openstackmagnum/heat-container-agent:wallaby-stable-1 >>>> /usr/bin/start-he... 34 minutes ago Up 34 minutes ago >>>> heat-container-agent >>>> 4128a9e951de k8s.gcr.io/hyperkube:v1.18.16 >>>> kube-apiserver --... 30 minutes ago Up 30 minutes ago >>>> kube-apiserver >>>> 5430aa72e7f0 k8s.gcr.io/hyperkube:v1.18.16 >>>> kube-controller-m... 30 minutes ago Up 30 minutes ago >>>> kube-controller-manager >>>> 8f2c76272916 k8s.gcr.io/hyperkube:v1.18.16 >>>> kube-scheduler --... 30 minutes ago Up 30 minutes ago >>>> kube-scheduler >>>> af5f1a661c65 quay.io/coreos/etcd:v3.4.6 >>>> /usr/local/bin/et... 30 minutes ago Up 30 minutes ago >>>> etcd >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Wed, Sep 8, 2021 at 5:37 PM Ammad Syed >>>> wrote: >>>> >>>>> Hi karera, >>>>> >>>>> I don’t think your the error is related to network side. Its because >>>>> kublet is not installed properly in your kubernetes master node. You need >>>>> to check logs that why hyperkube services are not started. >>>>> >>>>> You can check it in heat logs on mater node and you can check the >>>>> status of kubelet container via podman ps commad. >>>>> >>>>> Ammad >>>>> On Wed, Sep 8, 2021 at 8:28 PM Karera Tony >>>>> wrote: >>>>> >>>>>> Hello Slawek, >>>>>> >>>>>> Unfortunately I am not sure because all I did is configure the >>>>>> cluster. Nothing else. >>>>>> >>>>>> >>>>>> Regards >>>>>> >>>>>> Tony Karera >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Sep 8, 2021 at 3:31 PM Slawek Kaplonski >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> On środa, 8 września 2021 15:16:57 CEST Karera Tony wrote: >>>>>>> > Hello Slawek, >>>>>>> > >>>>>>> > I really need your help. >>>>>>> > >>>>>>> > I have tried all I can but still the k8s-cluster gets stuck with >>>>>>> this log >>>>>>> > here >>>>>>> > >>>>>>> > root at k8s-cluster-1-vo5s5jvtjoa7-master-0 core]# cd >>>>>>> > /var/log/heat-config/heat-config-script/ >>>>>>> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# ls >>>>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>>>> kube_cluster_c >>>>>>> > onfig-asfqq352zosg.log >>>>>>> > c42dcbf1-7812-49a2-b312-96c24225b407-k8s-cluster-1-vo5s5jvtjoa7- >>>>>>> kube_masters- >>>>>>> > waugw6vksw5p-0-vrt2gkfugedg-master_config-xv3vlmr27r46.log >>>>>>> > [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 heat-config-script]# cat >>>>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>>>> kube_cluster_c >>>>>>> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >>>>>>> > heat-config-script]# cat >>>>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>>>> kube_cluster_c >>>>>>> > onfig-asfqq352zosg.log [root at k8s-cluster-1-vo5s5jvtjoa7-master-0 >>>>>>> > heat-config-script]# cat >>>>>>> > a70c65af-34e2-4071-be18-2e4d66f85cf6-k8s-cluster-1-vo5s5jvtjoa7- >>>>>>> kube_cluster_c >>>>>>> > onfig-asfqq352zosg.log Starting to run >>>>>>> kube-apiserver-to-kubelet-role >>>>>>> > + echo 'Waiting for Kubernetes API...' >>>>>>> > Waiting for Kubernetes API... >>>>>>> > ++ kubectl get --raw=/healthz >>>>>>> > The connection to the server localhost:8080 was refused - did you >>>>>>> specify >>>>>>> > the right host or port? >>>>>>> > + '[' ok = '' ']' >>>>>>> > + sleep 5 >>>>>>> > ++ kubectl get --raw=/healthz >>>>>>> > The connection to the server localhost:8080 was refused - did you >>>>>>> specify >>>>>>> > the right host or port? >>>>>>> >>>>>>> But why Your script is trying to connect to localhost? It's not the >>>>>>> VM address >>>>>>> I guess :) >>>>>>> >>>>>>> > + '[' ok = '' ']' >>>>>>> > + sleep 5 >>>>>>> > >>>>>>> > >>>>>>> > Regards >>>>>>> > >>>>>>> > Tony Karera >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > On Mon, Aug 30, 2021 at 8:50 AM Slawek Kaplonski < >>>>>>> skaplons at redhat.com> >>>>>>> > >>>>>>> > wrote: >>>>>>> > > Hi, >>>>>>> > > >>>>>>> > > You can use vlan as tenant network too. But You have to remember >>>>>>> that vlan >>>>>>> > > is >>>>>>> > > provider network, which means that You need to have configured >>>>>>> vlans on >>>>>>> > > Your >>>>>>> > > network infrastructure too. And in case of tenant network, >>>>>>> Neutron will >>>>>>> > > automatically allocate one of the vlan ids to the network as >>>>>>> user don't >>>>>>> > > have >>>>>>> > > possibility to choose vlan id for network (at least that's >>>>>>> default policy >>>>>>> > > in >>>>>>> > > Neutron). >>>>>>> > > So, You should check what vlan id is used by such tenant >>>>>>> network, check if >>>>>>> > > this vlan is properly confiugred on Your switches and then, if >>>>>>> all is >>>>>>> > > good, >>>>>>> > > check where exactly such dhcp requests are dropped (are they >>>>>>> going from >>>>>>> > > compute node? are packets comming to the node with dhcp agent?). >>>>>>> Base on >>>>>>> > > that >>>>>>> > > You can narrow down where the issue can be. >>>>>>> > > >>>>>>> > > On sobota, 28 sierpnia 2021 18:07:07 CEST Karera Tony wrote: >>>>>>> > > > How can i make them both vxlan and vlan >>>>>>> > > > >>>>>>> > > > On Sat, 28 Aug 2021, 17:43 Mohammed Naser, < >>>>>>> mnaser at vexxhost.com> wrote: >>>>>>> > > > > Your issue is that tenant_network_types should be vxlan. >>>>>>> > > > > >>>>>>> > > > > On Sat, Aug 28, 2021 at 4:29 AM Karera Tony < >>>>>>> tonykarera at gmail.com> >>>>>>> > > >>>>>>> > > wrote: >>>>>>> > > > >> Hello Team, >>>>>>> > > > >> >>>>>>> > > > >> Whenever I create an internal network with type vlan. >>>>>>> > > > >> >>>>>>> > > > >> The Instances don't get IPs but for external Network it's >>>>>>> fine. >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> Below is the etc/kolla/config/neutron/ml2_conf.ini file >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> [ml2] >>>>>>> > > > >> type_drivers = flat,vlan,vxlan >>>>>>> > > > >> tenant_network_types = vlan,flat >>>>>>> > > > >> mechanism_drivers = openvswitch,l2population >>>>>>> > > > >> >>>>>>> > > > >> [ml2_type_vlan] >>>>>>> > > > >> network_vlan_ranges = physnet1 >>>>>>> > > > >> [ml2_type_flat] >>>>>>> > > > >> flat_networks = physnet1 >>>>>>> > > > >> >>>>>>> > > > >> Is there something I need to change or I need to configure >>>>>>> the >>>>>>> > > >>>>>>> > > interface >>>>>>> > > >>>>>>> > > > >> differently >>>>>>> > > > >> Regards >>>>>>> > > > >> >>>>>>> > > > >> Tony Karera >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> -- >>>>>>> > > > > >>>>>>> > > > > Mohammed Naser >>>>>>> > > > > VEXXHOST, Inc. >>>>>>> > > >>>>>>> > > -- >>>>>>> > > Slawek Kaplonski >>>>>>> > > Principal Software Engineer >>>>>>> > > Red Hat >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Slawek Kaplonski >>>>>>> Principal Software Engineer >>>>>>> Red Hat >>>>>> >>>>>> -- >>>>> Regards, >>>>> >>>>> >>>>> Syed Ammad Ali >>>>> >>>> -- >>> Regards, >>> >>> >>> Syed Ammad Ali >>> >> -- > Regards, > > > Syed Ammad Ali > -------------- next part -------------- An HTML attachment was scrubbed... URL: From haleyb.dev at gmail.com Wed Sep 8 19:10:55 2021 From: haleyb.dev at gmail.com (Brian Haley) Date: Wed, 8 Sep 2021 15:10:55 -0400 Subject: [neutron] Slow router provisioning during full resync of L3 Agent In-Reply-To: References: Message-ID: Hi Patryk, Yes, the re-synchronization of the l3-agent can sometimes be time consuming. A number of things have been added over the years to help speed this up, some are in Rocky, some are in later releases. On 9/8/21 8:48 AM, Patryk Jakuszew wrote: > Hello, > > I have a long-standing issue with L3 Agent which I would like to > finally solve - *very* slow router provisioning in L3 Agent. > > We are operating a Rocky-based OpenStack deployment with three > bare-metal L3 Agent nodes running in legacy mode. After restarting the > L3 node, it takes a really long time for the L3 agent to become fully > operational. There are two parts of resync which take much time: > getting a list of routers from neutron-server and actually recreate > them in the L3 node. The first of these, getting info from neutron-server, was initially fixed in 2015 and has been enhanced over the years - retrieving routers in 'chunks' to reduce the load on neutron-server, since trying to get info on 250 routers is a large response to construct. When this happens do you see neutron-server under heavy load? It might be you need to tune the number of RPC workers in this instance to help. The second has also been slowly improved on the l3-agent side in a number of ways, for example, by dynamically increasing worker threads when long backlogs occur (not in Rocky). Other changes like using privsep instead of rootwrap has brought the times down slightly as well. There are probably others I'm not thinking of... > While the long running time of router list retrieval is somewhat > understandable, the router provisioning process itself proves to be > very troublesome in our operations tasks. In our production deployment > with around 250 routers, it takes around 2 hours (!) to recreate the > router namespaces and have the L3 node fully functional again. Two > hours of router re-provisioning is actually an optimistic scenario, > this proved to be much longer during the outages we encountered > (sometimes the sync took nearly 6-8 hours). This effectively prolongs > any maintenance upgrades, configuration changes and OpenStack release > upgrades. > > Another thing is, on that same production environment the first 100 > routers usually get provisioned fast (around 30 minutes), after that > it slows down with each router - this kind of non deterministic > behavior makes it hard to communicate the maintenance finish ETA for > our users. > > We also have a test environment with Stein already installed, where > this problem is also present - full resync of 150 routers, with only > one external gateway ports, takes around an hour to complete. > > Are there any operators here who also encountered that issue? Does > anyone have any experience with similar situation and are willing to > share their observations and optimizations? Yes, I know of other operators that have encountered this issue, and the community has tried to address it over the years. It seems you might have some of the fixes, but not all of them, and some tuning of worker threads might help. That said, I've never seen sync times in the 6-8 hour range, I wonder if the systems in question are under any CPU or memory pressure? Are there any other failures in the logs that show things timing out, like RPC failure/retries? Some other thoughts: Last year (2020) there were a number of debug messages added to the l3-agent that might help pinpoint where time is being spent for each router being processed, but that will not be in either of the later releases you mentioned. Maybe if you could install your test environment with something much newer it would help resolve or debug the issue better? Using the OVN mechanism driver totally eliminates the l3-agent, but I believe you'd need to jump to Victoria (?) in order to use that. -Brian From patryk.jakuszew at gmail.com Wed Sep 8 21:34:46 2021 From: patryk.jakuszew at gmail.com (Patryk Jakuszew) Date: Wed, 8 Sep 2021 23:34:46 +0200 Subject: [neutron] Slow router provisioning during full resync of L3 Agent In-Reply-To: References: Message-ID: Hi Brian, On Wed, 8 Sept 2021 at 21:10, Brian Haley wrote: > The first of these, getting info from neutron-server, was initially > fixed in 2015 and has been enhanced over the years - retrieving routers > in 'chunks' to reduce the load on neutron-server, since trying to get > info on 250 routers is a large response to construct. When this happens > do you see neutron-server under heavy load? It might be you need to > tune the number of RPC workers in this instance to help. > > The second has also been slowly improved on the l3-agent side in a > number of ways, for example, by dynamically increasing worker threads > when long backlogs occur (not in Rocky). Other changes like using > privsep instead of rootwrap has brought the times down slightly as well. > There are probably others I'm not thinking of... In our test environment I noticed that indeed there was a higher CPU load on neutron-server. I will take a look at both of the options that you mentioned - recently I've seen some mentions of adjusting RPC workers to CPU count in order to improve inter-service communication, but I didn't know about the possibility of switching between privsep and rootwrap. > That said, I've never seen sync times in the 6-8 hour range, I wonder if > the systems in question are under any CPU or memory pressure? Are there > any other failures in the logs that show things timing out, like RPC > failure/retries? This indeed happened during a full resync caused by a major outage of the entire RabbitMQ cluster. (Upgrade from 3.6.x to 3.9.x went wrong) Our control plane runs mostly on VMs, with exception of Neutron services which run on dedicated physical nodes. During the upgrade we actually wanted to add more vCPUs to RabbitMQ machines, but after noticing the control plane instability we rolled back that change. I will conduct more tests to see how much load is generated during the resync. > Some other thoughts: > > Last year (2020) there were a number of debug messages added to the > l3-agent that might help pinpoint where time is being spent for each > router being processed, but that will not be in either of the later > releases you mentioned. Maybe if you could install your test > environment with something much newer it would help resolve or debug the > issue better? > > Using the OVN mechanism driver totally eliminates the l3-agent, but I > believe you'd need to jump to Victoria (?) in order to use that. > > -Brian If newer releases have much more debug information available, then it is definitely worth checking out - I tried gathering some initial information about duration of certain operations by attaching py-spy into neutron-l3-agent (https://github.com/benfred/py-spy), but it didn't actually say how long it took for particular operations to complete. As for OVN... I have evaluated it a bit on my private environment (packstack all-in-one) and while it does have many welcome improvements like the elimination of separate agent processes, it also misses a feature that makes it a no-go for our production environment - neutron-vpnaas support. We have *lots* of users that would not be happy if we took away neutron-vpnaas. :/ Thank you very much for all the information - now I have some additional directions to look at. -- Best regards, Patryk Jakuszew On Wed, 8 Sept 2021 at 21:10, Brian Haley wrote: > > Hi Patryk, > > Yes, the re-synchronization of the l3-agent can sometimes be time > consuming. A number of things have been added over the years to help > speed this up, some are in Rocky, some are in later releases. > > On 9/8/21 8:48 AM, Patryk Jakuszew wrote: > > Hello, > > > > I have a long-standing issue with L3 Agent which I would like to > > finally solve - *very* slow router provisioning in L3 Agent. > > > > We are operating a Rocky-based OpenStack deployment with three > > bare-metal L3 Agent nodes running in legacy mode. After restarting the > > L3 node, it takes a really long time for the L3 agent to become fully > > operational. There are two parts of resync which take much time: > > getting a list of routers from neutron-server and actually recreate > > them in the L3 node. > > The first of these, getting info from neutron-server, was initially > fixed in 2015 and has been enhanced over the years - retrieving routers > in 'chunks' to reduce the load on neutron-server, since trying to get > info on 250 routers is a large response to construct. When this happens > do you see neutron-server under heavy load? It might be you need to > tune the number of RPC workers in this instance to help. > > The second has also been slowly improved on the l3-agent side in a > number of ways, for example, by dynamically increasing worker threads > when long backlogs occur (not in Rocky). Other changes like using > privsep instead of rootwrap has brought the times down slightly as well. > There are probably others I'm not thinking of... > > > While the long running time of router list retrieval is somewhat > > understandable, the router provisioning process itself proves to be > > very troublesome in our operations tasks. In our production deployment > > with around 250 routers, it takes around 2 hours (!) to recreate the > > router namespaces and have the L3 node fully functional again. Two > > hours of router re-provisioning is actually an optimistic scenario, > > this proved to be much longer during the outages we encountered > > (sometimes the sync took nearly 6-8 hours). This effectively prolongs > > any maintenance upgrades, configuration changes and OpenStack release > > upgrades. > > > > Another thing is, on that same production environment the first 100 > > routers usually get provisioned fast (around 30 minutes), after that > > it slows down with each router - this kind of non deterministic > > behavior makes it hard to communicate the maintenance finish ETA for > > our users. > > > > We also have a test environment with Stein already installed, where > > this problem is also present - full resync of 150 routers, with only > > one external gateway ports, takes around an hour to complete. > > > > Are there any operators here who also encountered that issue? Does > > anyone have any experience with similar situation and are willing to > > share their observations and optimizations? > > Yes, I know of other operators that have encountered this issue, and the > community has tried to address it over the years. It seems you might > have some of the fixes, but not all of them, and some tuning of worker > threads might help. > > That said, I've never seen sync times in the 6-8 hour range, I wonder if > the systems in question are under any CPU or memory pressure? Are there > any other failures in the logs that show things timing out, like RPC > failure/retries? > > Some other thoughts: > > Last year (2020) there were a number of debug messages added to the > l3-agent that might help pinpoint where time is being spent for each > router being processed, but that will not be in either of the later > releases you mentioned. Maybe if you could install your test > environment with something much newer it would help resolve or debug the > issue better? > > Using the OVN mechanism driver totally eliminates the l3-agent, but I > believe you'd need to jump to Victoria (?) in order to use that. > > -Brian From os-user157 at protonmail.com Wed Sep 8 21:34:15 2021 From: os-user157 at protonmail.com (os-user157) Date: Wed, 08 Sep 2021 21:34:15 +0000 Subject: [devstack][openstack-qa] DevStack on VirtualBox with Host-only Adapter Message-ID: Hi all, I'm trying to deploy DevStack inside an Ubuntu Server 20.04.3 LTS virtual machine inside VirtualBox, but I'm struggling a lot, I got a lot of different errors. My use case: I want to test Zuul and DevStack, so I'll create a virtual machine for DevStack and one for Zuul, both on VirtualBox. I'm using a Host-only adapter with CIDR 192.168.200.0/24, and I need DevStack to be accessible on this network, and to use this network as the floating range for giving floating IP addresses to instances inside DevStack, so that Zuul can also access them. My local.conf: https://paste.openstack.org/show/809182/ Screenshots of the network configuration of my VirtualBox VM: First adapter, NAT (used for internet connectivity): https://i.imgur.com/C7XTiLJ.png Second adapter, host-only, used for communication with my Windows host and other VMs in the same network: https://i.imgur.com/Qeciuqx.png DevStack is erroring with the arping command on lib/neutron-legacy, arping gets 0 responses and exits the script with exit code 1 Does anyone have experience in dpeloying DevStack inside VirtualBox in such manner? I really need some help with this, if anyone could give me some ideas of waht could be wrong, it'd very much appreciated! Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From os-user157 at protonmail.com Wed Sep 8 21:37:30 2021 From: os-user157 at protonmail.com (os-user157) Date: Wed, 08 Sep 2021 21:37:30 +0000 Subject: [devstack][openstack-qa] DevStack on VirtualBox with Host-only Adapter Message-ID: <1CDGGTuntimwQAUEHVBdZvwxrY5s1aYy2ccWSYtqPUl9JbKg4Y6cMBPv1TKCCDCPmNLQ6SWfZNe76BnDmpQUGWtfzZAEdtXJdz4fsCrn3Yo=@protonmail.com> Hi all, I'm trying to deploy DevStack inside an Ubuntu Server 20.04.3 LTS virtual machine inside VirtualBox, but I'm struggling a lot, I got a lot of different errors. My use case: I want to test Zuul and DevStack, so I'll create a virtual machine for DevStack and one for Zuul, both on VirtualBox. I'm using a Host-only adapter with CIDR 192.168.200.0/24, and I need DevStack to be accessible on this network, and to use this network as the floating range for giving floating IP addresses to instances inside DevStack, so that Zuul can also access them. My local.conf: https://paste.openstack.org/show/809182/ Screenshots of the network configuration of my VirtualBox VM: First adapter, NAT (used for internet connectivity): https://i.imgur.com/C7XTiLJ.png Second adapter, host-only, used for communication with my Windows host and other VMs in the same network: https://i.imgur.com/Qeciuqx.png DevStack is erroring with the arping command on lib/neutron-legacy, arping gets 0 responses and exits the script with exit code 1 Does anyone have experience in dpeloying DevStack inside VirtualBox in such manner? I really need some help with this, if anyone could give me some ideas of waht could be wrong, it'd very much appreciated! Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Wed Sep 8 23:01:28 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Wed, 08 Sep 2021 16:01:28 -0700 Subject: =?UTF-8?Q?Re:_[devstack][openstack-qa]_DevStack_on_VirtualBox_with_Host-?= =?UTF-8?Q?only_Adapter?= In-Reply-To: <1CDGGTuntimwQAUEHVBdZvwxrY5s1aYy2ccWSYtqPUl9JbKg4Y6cMBPv1TKCCDCPmNLQ6SWfZNe76BnDmpQUGWtfzZAEdtXJdz4fsCrn3Yo=@protonmail.com> References: <1CDGGTuntimwQAUEHVBdZvwxrY5s1aYy2ccWSYtqPUl9JbKg4Y6cMBPv1TKCCDCPmNLQ6SWfZNe76BnDmpQUGWtfzZAEdtXJdz4fsCrn3Yo=@protonmail.com> Message-ID: <05c67c42-4527-4031-b291-14bcaa0aa7e6@www.fastmail.com> On Wed, Sep 8, 2021, at 2:37 PM, os-user157 wrote: > Hi all, > > I'm trying to deploy DevStack inside an Ubuntu Server 20.04.3 LTS > virtual machine inside VirtualBox, but I'm struggling a lot, I got a > lot of different errors. > > My use case: I want to test Zuul and DevStack, so I'll create a virtual > machine for DevStack and one for Zuul, both on VirtualBox. I'm using a > Host-only adapter with CIDR 192.168.200.0/24, and I need DevStack to be > accessible on this network, and to use this network as the floating > range for giving floating IP addresses to instances inside DevStack, so > that Zuul can also access them. > > My local.conf: https://paste.openstack.org/show/809182/ > Screenshots of the network configuration of my VirtualBox VM: > First adapter, NAT (used for internet connectivity): > https://i.imgur.com/C7XTiLJ.png > Second adapter, host-only, used for communication with my Windows host > and other VMs in the same network: https://i.imgur.com/Qeciuqx.png > > DevStack is erroring with the arping command on lib/neutron-legacy, > arping gets 0 responses and exits the script with exit code 1 > > Does anyone have experience in dpeloying DevStack inside VirtualBox in > such manner? I really need some help with this, if anyone could give me > some ideas of waht could be wrong, it'd very much appreciated! Thanks. I've not done this before, but the multinode devstack + tempest CI jobs do something similar. In those jobs we create a layer2 network using vxlan between the instances then set up layer three such that 172.24.4.0/24 is used for the openstack services and 172.24.5.0/24 is used as the floating IP range. Then we have routes set up for 172.24.4.0/23 as on link connections allowing both /24 ranges to be routed properly. The disjoint sets are important here and I notice that you are overlapping your service/infrastructure IPs and the floating IP range. The first thing I would do is change that. Maybe use 192.168.200.128/25 as the floating IP range and then avoid that for the services. Also when you get errors from devstack it helps tremendously if you can share pastes of them so that others can see context and maybe understand why things broke. Another thing to note: running instances without nested virt is slow and may not produce a great Zuul test environment. When using nested virt we have noticed random crashes are relatively common. Something to be aware of. Are you trying to set this up for Zuul and Nodepool development? If so the Zuul and Nodepool test suites are actually pretty good at mocking out the cloud interactions and then upstream CI does do final verification against a devstack cloud. All that to say it might be easier to just rely on the test suite. Hope this helps, Clark From gmann at ghanshyammann.com Thu Sep 9 00:11:14 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 08 Sep 2021 19:11:14 -0500 Subject: [all][tc] Technical Committee next weekly meeting on Sept 9th at 1500 UTC In-Reply-To: <17bbc2d846e.11adf55c390492.8839891819324332226@ghanshyammann.com> References: <17bbc2d846e.11adf55c390492.8839891819324332226@ghanshyammann.com> Message-ID: <17bc7e4eff5.1161f6233210248.5053739727733279750@ghanshyammann.com> Hello Everyone, Below is the agenda for Tomorrow's TC meeting schedule at 1500 UTC in #openstack-tc IRC channel. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting == Agenda for tomorrow's TC meeting == * Roll call * Follow up on past action items * Gate health check (dansmith/yoctozepto) ** http://paste.openstack.org/show/jD6kAP9tHk7PZr2nhv8h/ * Xena Tracker ** https://etherpad.opendev.org/p/tc-xena-tracker * Leaderless projects ** https://etherpad.opendev.org/p/yoga-leaderless * New project application: 'Venus' ** http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019748.html ** https://review.opendev.org/c/openstack/governance/+/804824 * Next step on pain points from projects/SIGs/pop-up (ricolin) ** https://etherpad.opendev.org/p/pain-point-elimination * TC tags analysis ** https://docs.google.com/spreadsheets/d/18GXibtdQnSkIwA7DsBvX9dPwbw9JH76AhhRUknzBb3Q/edit * TC Video call feedback & plan: ** https://etherpad.opendev.org/p/tc-video-meeting-feedback * Open Reviews ** https://review.opendev.org/q/projects:openstack/governance+is:open -gmann ---- On Mon, 06 Sep 2021 12:35:04 -0500 Ghanshyam Mann wrote ---- > Hello Everyone, > > Technical Committee's next weekly meeting is scheduled for Sept 9th at 1500 UTC. > > If you would like to add topics for discussion, please add them to the below wiki page by > Wednesday, Sept 8th, at 2100 UTC. > > https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting > > -gmann > > From skaplons at redhat.com Thu Sep 9 07:19:14 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 09 Sep 2021 09:19:14 +0200 Subject: [neutron] Slow router provisioning during full resync of L3 Agent In-Reply-To: References: Message-ID: <31788380.zD9AKE7zTE@p1> Hi, On środa, 8 września 2021 23:34:46 CEST Patryk Jakuszew wrote: > Hi Brian, > > On Wed, 8 Sept 2021 at 21:10, Brian Haley wrote: > > The first of these, getting info from neutron-server, was initially > > fixed in 2015 and has been enhanced over the years - retrieving routers > > in 'chunks' to reduce the load on neutron-server, since trying to get > > info on 250 routers is a large response to construct. When this happens > > do you see neutron-server under heavy load? It might be you need to > > tune the number of RPC workers in this instance to help. > > > > The second has also been slowly improved on the l3-agent side in a > > number of ways, for example, by dynamically increasing worker threads > > when long backlogs occur (not in Rocky). Other changes like using > > privsep instead of rootwrap has brought the times down slightly as well. > > > > There are probably others I'm not thinking of... > > In our test environment I noticed that indeed there was a higher CPU > load on neutron-server. I will take a look at both of the options that > you mentioned - recently I've seen some mentions of adjusting RPC > workers to CPU count in order to improve inter-service communication, > but I didn't know about the possibility of switching between privsep > and rootwrap. > > > That said, I've never seen sync times in the 6-8 hour range, I wonder if > > the systems in question are under any CPU or memory pressure? Are there > > any other failures in the logs that show things timing out, like RPC > > failure/retries? > > This indeed happened during a full resync caused by a major outage of > the entire RabbitMQ cluster. (Upgrade from 3.6.x to 3.9.x went wrong) > > Our control plane runs mostly on VMs, with exception of Neutron > services which run on dedicated physical nodes. During the upgrade we > actually wanted to add more vCPUs to RabbitMQ machines, but after > noticing the control plane instability we rolled back that change. I > will conduct more tests to see how much load is generated during the > resync. > > > Some other thoughts: > > > > Last year (2020) there were a number of debug messages added to the > > l3-agent that might help pinpoint where time is being spent for each > > router being processed, but that will not be in either of the later > > releases you mentioned. Maybe if you could install your test > > environment with something much newer it would help resolve or debug the > > issue better? > > > > Using the OVN mechanism driver totally eliminates the l3-agent, but I > > believe you'd need to jump to Victoria (?) in order to use that. > > > > -Brian > > If newer releases have much more debug information available, then it > is definitely worth checking out - I tried gathering some initial > information about duration of certain operations by attaching py-spy > into neutron-l3-agent (https://github.com/benfred/py-spy), but it > didn't actually say how long it took for particular operations to > complete. > > As for OVN... I have evaluated it a bit on my private environment > (packstack all-in-one) and while it does have many welcome > improvements like the elimination of separate agent processes, it also > misses a feature that makes it a no-go for our production environment > - neutron-vpnaas support. We have *lots* of users that would not be > happy if we took away neutron-vpnaas. :/ Support for vpnaas in OVN backend is already reported as RFE: https:// bugs.launchpad.net/neutron/+bug/1905391 - unfortunatelly that work stopped some time ago and there is no progress now. But maybe You would have time and want to help with it - any help is welcome :) > > Thank you very much for all the information - now I have some > additional directions to look at. > > -- > Best regards, > Patryk Jakuszew > > On Wed, 8 Sept 2021 at 21:10, Brian Haley wrote: > > Hi Patryk, > > > > Yes, the re-synchronization of the l3-agent can sometimes be time > > consuming. A number of things have been added over the years to help > > speed this up, some are in Rocky, some are in later releases. > > > > On 9/8/21 8:48 AM, Patryk Jakuszew wrote: > > > Hello, > > > > > > I have a long-standing issue with L3 Agent which I would like to > > > finally solve - *very* slow router provisioning in L3 Agent. > > > > > > We are operating a Rocky-based OpenStack deployment with three > > > bare-metal L3 Agent nodes running in legacy mode. After restarting the > > > L3 node, it takes a really long time for the L3 agent to become fully > > > operational. There are two parts of resync which take much time: > > > getting a list of routers from neutron-server and actually recreate > > > them in the L3 node. > > > > The first of these, getting info from neutron-server, was initially > > fixed in 2015 and has been enhanced over the years - retrieving routers > > in 'chunks' to reduce the load on neutron-server, since trying to get > > info on 250 routers is a large response to construct. When this happens > > do you see neutron-server under heavy load? It might be you need to > > tune the number of RPC workers in this instance to help. > > > > The second has also been slowly improved on the l3-agent side in a > > number of ways, for example, by dynamically increasing worker threads > > when long backlogs occur (not in Rocky). Other changes like using > > privsep instead of rootwrap has brought the times down slightly as well. > > > > There are probably others I'm not thinking of... > > > > > While the long running time of router list retrieval is somewhat > > > understandable, the router provisioning process itself proves to be > > > very troublesome in our operations tasks. In our production deployment > > > with around 250 routers, it takes around 2 hours (!) to recreate the > > > router namespaces and have the L3 node fully functional again. Two > > > hours of router re-provisioning is actually an optimistic scenario, > > > this proved to be much longer during the outages we encountered > > > (sometimes the sync took nearly 6-8 hours). This effectively prolongs > > > any maintenance upgrades, configuration changes and OpenStack release > > > upgrades. > > > > > > Another thing is, on that same production environment the first 100 > > > routers usually get provisioned fast (around 30 minutes), after that > > > it slows down with each router - this kind of non deterministic > > > behavior makes it hard to communicate the maintenance finish ETA for > > > our users. > > > > > > We also have a test environment with Stein already installed, where > > > this problem is also present - full resync of 150 routers, with only > > > one external gateway ports, takes around an hour to complete. > > > > > > Are there any operators here who also encountered that issue? Does > > > anyone have any experience with similar situation and are willing to > > > share their observations and optimizations? > > > > Yes, I know of other operators that have encountered this issue, and the > > community has tried to address it over the years. It seems you might > > have some of the fixes, but not all of them, and some tuning of worker > > threads might help. > > > > That said, I've never seen sync times in the 6-8 hour range, I wonder if > > the systems in question are under any CPU or memory pressure? Are there > > any other failures in the logs that show things timing out, like RPC > > failure/retries? > > > > Some other thoughts: > > > > Last year (2020) there were a number of debug messages added to the > > l3-agent that might help pinpoint where time is being spent for each > > router being processed, but that will not be in either of the later > > releases you mentioned. Maybe if you could install your test > > environment with something much newer it would help resolve or debug the > > issue better? > > > > Using the OVN mechanism driver totally eliminates the l3-agent, but I > > believe you'd need to jump to Victoria (?) in order to use that. > > > > -Brian -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From skaplons at redhat.com Thu Sep 9 07:32:21 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 09 Sep 2021 09:32:21 +0200 Subject: [neutron][neutron-vpnaas] Propose to EOL stable/queens, stable/rocky and stable/stein branches or neutron-vpnaas Message-ID: <127605535.TiF7tupeXR@p1> Hi, On our last meeting of the CI team we discussed the problem with broken stable branches (stein and older) in the neutron-vpnaas project [1]. I checked that last patches which were merged in those branches are more than year ago: * Queens - last merged patch in April 2019 * Rocky - last merged patch in February 2020 * Stein - last merged patch in September 2019 Giving the above and the current status of the CI in those branches, I propose to make them End Of Life. I will wait until end of next week for anyone who would like to maybe step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in neutron- vpnaas. [1] https://meetings.opendev.org/meetings/neutron_ci/2021/neutron_ci. 2021-09-07-15.00.log.html#l-64 -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From zhang.lei.fly+os-discuss at gmail.com Thu Sep 9 07:47:57 2021 From: zhang.lei.fly+os-discuss at gmail.com (Jeffrey Zhang) Date: Thu, 9 Sep 2021 15:47:57 +0800 Subject: Question about vDPA support for live migration Message-ID: Hey everyone, I am testing vDPA on OpenStack Wallaby. I have set it up successfully. But I found there are several features that are not implemented, including live/code migration[0], which I wanted. Anyone know what's the latest progress about this. when will the live migration feature land? I found no talk about this. [0] https://review.opendev.org/c/openstack/nova/+/780866/3/ releasenotes/notes/vdpa-cc2300d2c46c150b.yaml Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkopec at redhat.com Thu Sep 9 07:52:17 2021 From: mkopec at redhat.com (Martin Kopec) Date: Thu, 9 Sep 2021 09:52:17 +0200 Subject: [designate][interop] test_create_all_recordset_types ddt issue In-Reply-To: <3229108.usfYGdeWWP@whitebase.usersys.redhat.com> References: <3229108.usfYGdeWWP@whitebase.usersys.redhat.com> Message-ID: >From interop perspective it's also better not to have multiple tests with the same id. We encountered one more problem with ddt - the test names seem not to be generated consistently, see this: https://paste.opendev.org/show/809187/ The test can have either _00009_TXT suffix or _9_TXT one. Until we figure this out, I think we will need to flag the test in interop - so that a skip of the test (because of the name mismatch in this case) won't make the whole guideline fail. Luigi's idea is great. Every test should be identified by a unique id and it shouldn't matter that the test is generated (ddt). Different input data -> different test -> different name -> different id. Let's try to explore whether having a unique id per ddt entry is possible. On Wed, 8 Sept 2021 at 18:15, Luigi Toscano wrote: > On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote: > > If the use of "ddt" is a big problem for the compliance testing, we > > can consider breaking these out into individual tests (likely to be > > duplicates to the existing "ddt" tests to maintain the legacy test > > UUIDs). > > I was wondering: wouldn't it be possible to expand or use ddt somehow to > inject different UUIDs for each generated test? You know in advance how > many > tests are going to be generated. > > -- > Luigi > > > -- Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From micou12 at gmail.com Thu Sep 9 08:10:27 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Thu, 9 Sep 2021 10:10:27 +0200 Subject: Horizon connection errors from object store Message-ID: Hello team , I am facing an issue when I am trying to connect to the object store containers on the horizon dashboad . Once click on containers it automatically disconnect. please find below logs I am getting and help for further analysis. [Thu Sep 09 06:35:22.185771 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55130] Attempted scope to domain Default failed, will attempt to scope to another domain. [Thu Sep 09 06:35:22.572522 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55130] Login successful for user "admin" using domain "Default", remote address 10.10.29.150. [Thu Sep 09 06:35:51.494815 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" [Thu Sep 09 06:35:51.495140 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized [Thu Sep 09 06:35:51.495541 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx00000000000000000000f-006139ab44-9fc1a-default', 'X-Openstack-Request-Id': 'tx00000000000000000000f-006139ab44-9fc1a-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, 09 Sep 2021 06:35:51 GMT', 'Connection': 'Keep-Alive'} [Thu Sep 09 06:35:51.495792 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000f-006139ab44-9fc1a-default","HostId":"9fc1a-default-default"}' [Thu Sep 09 06:35:51.498743 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Unauthorized: /api/swift/containers/ [Thu Sep 09 06:35:52.924169 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" [Thu Sep 09 06:35:52.924520 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized [Thu Sep 09 06:35:52.924789 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx000000000000000000010-006139ab48-9fc1a-default', 'X-Openstack-Request-Id': 'tx000000000000000000010-006139ab48-9fc1a-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, 09 Sep 2021 06:35:52 GMT', 'Connection': 'Keep-Alive'} [Thu Sep 09 06:35:52.925034 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx000000000000000000010-006139ab48-9fc1a-default","HostId":"9fc1a-default-default"}' [Thu Sep 09 06:35:52.929398 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Unauthorized: /api/swift/containers/ [Thu Sep 09 06:35:52.935799 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:56016] Logging out user "admin". [Thu Sep 09 06:35:53.061489 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Logging out user "". [Thu Sep 09 06:35:54.541593 2021] [wsgi:error] [pid 165:tid 139887608641280] [remote 10.10.29.150:55852] The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example. [Thu Sep 09 06:35:54.542896 2021] [wsgi:error] [pid 165:tid 139887608641280] [remote 10.10.29.150:55852] Bad Request: /api/swift/policies/ [Thu Sep 09 06:35:54.566055 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55860] The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example. [Thu Sep 09 06:35:54.567130 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55860] Bad Request: /api/swift/info/ (kolla-open1) stack at kolla-open1 :/var/lib/docker/volumes/kolla_logs/_data/horizon$ Michel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ueha.ayumu at fujitsu.com Thu Sep 9 09:23:16 2021 From: ueha.ayumu at fujitsu.com (ueha.ayumu at fujitsu.com) Date: Thu, 9 Sep 2021 09:23:16 +0000 Subject: [heat] Issue happend during server_group creation with policies Message-ID: Hi heat team. I reported the following bug report [1] on Sep 2. [1] https://storyboard.openstack.org/#!/story/2009164 Have you confirmed it? This error seems to appear to have occurred after patch [2] was merged. [2] https://review.opendev.org/c/openstack/heat/+/783971 Please kindly check it, thanks. Regards, Ayumu Ueha -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Thu Sep 9 10:11:06 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 09 Sep 2021 12:11:06 +0200 Subject: [neutron] Drivers meeting agenda - 10.09.2021 Message-ID: <4687421.kKWKmoin3I@p1> Hi, Agenda for our tomorrow's drivers meeting is at [1]. We have 3 RFEs to discuss: * https://bugs.launchpad.net/neutron/+bug/1830014 - [RFE] add API for neutron debug tool "probe" - there is also spec proposed for that one [2] * https://bugs.launchpad.net/neutron/+bug/1916428 - dibbler tool for dhcpv6 is concluded * https://bugs.launchpad.net/neutron/+bug/1870319 - [RFE] Network cascade deletion API call All of them are pretty old but we need to get back to them and hopefully decide something :) See You all on the meeting tomorrow. [1] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers#Agenda [2] https://review.opendev.org/c/openstack/neutron-specs/+/662541 -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From nikitakalyanov at gmail.com Thu Sep 9 10:43:52 2021 From: nikitakalyanov at gmail.com (Nikita Kalyanov) Date: Thu, 9 Sep 2021 13:43:52 +0300 Subject: [oslo] RFE requested for oslo-messaging Message-ID: Dear Release Team, We experienced strange freezes and timeouts for a long time on our OpenStack deployment [1] and some duplicate messages [2]. After applying the fixes [3], [4] both bugs are gone. Is a freeze exception possible for these? Thanks! nkalyanov [1] https://bugs.launchpad.net/oslo.messaging/+bug/1935864 [2] https://bugs.launchpad.net/oslo.messaging/+bug/1935883 [3] https://review.opendev.org/c/openstack/oslo.messaging/+/800564 [4] https://review.opendev.org/c/openstack/oslo.messaging/+/800575 From radoslaw.piliszek at gmail.com Thu Sep 9 11:00:26 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Thu, 9 Sep 2021 13:00:26 +0200 Subject: [oslo] RFE requested for oslo-messaging In-Reply-To: References: Message-ID: On Thu, Sep 9, 2021 at 12:44 PM Nikita Kalyanov wrote: > > Dear Release Team, > > We experienced strange freezes and timeouts for a long time on our > OpenStack deployment [1] and some duplicate messages [2]. After > applying the fixes [3], [4] both bugs are gone. Hmm, I have read the bug reports out of curiosity and I think we might be hitting these issues randomly in Kolla CI (due to load or bad network). (I am obviously unsure whether the proposed fixes are correct or not.) As these are bugfixes, they should not need a freeze exception per se. I hope Oslo messaging cores respond soon. -yoctozepto > > Is a freeze exception possible for these? > > Thanks! > nkalyanov > > [1] https://bugs.launchpad.net/oslo.messaging/+bug/1935864 > [2] https://bugs.launchpad.net/oslo.messaging/+bug/1935883 > [3] https://review.opendev.org/c/openstack/oslo.messaging/+/800564 > [4] https://review.opendev.org/c/openstack/oslo.messaging/+/800575 > From smooney at redhat.com Thu Sep 9 11:25:21 2021 From: smooney at redhat.com (Sean Mooney) Date: Thu, 09 Sep 2021 12:25:21 +0100 Subject: Question about vDPA support for live migration In-Reply-To: References: Message-ID: <8e698e1a0adbaabe00db9732fc90f7eba167d22f.camel@redhat.com> On Thu, 2021-09-09 at 15:47 +0800, Jeffrey Zhang wrote: > Hey everyone, > > I am testing vDPA on OpenStack Wallaby. I have set it up successfully. But > I found there are several features that are not implemented, including > live/code migration[0], which I wanted. > > Anyone know what's the latest progress about this. when will the live > migration feature land? > I found no talk about this. high i had planned on anding some more feature in xena but i was not able to work on it this cycle live migration is not a feature we can enable currently as qemu does not support it so it will likely be the Z or A cycle before live migration will be supported. currently we really only support boot, start/stop/reboot (hard and soft)/rebuild and vm delete in termes of lifecycle events. i had hoped to add shelve, evacuate, resize and cold migration in xena but i will try to find time to work on it in yoga instead. resize to same host works but we blocked it sicne resize to different host was untested. shelve works but unshelve failed and i did not have time to understand why during wallaby suspend fails because i have not extended it to unplug the vdpa devices like pci devices and libvirt managed_save requires that. if resize to different host works cold migrate will too they go hand in hand. evacuate is just rebuild to adifferent host i suspect it will mostly work but like cold migrate it was untested and we will need to ensure the devices are claimed correctly. interface attach works but it blocked because qemu raised an error if we tried to detach im hopintg that with a newer version of qemu this will just work. there are todos in the code that note most of this if you or other have time to work on this then please do. i currently have issue getting the hardware to work on this deploy as a result its not easy for me to work on this currently. while i do have 2 phsyical nics that support vdpa the servier i intended to use fot finsih this work are too old to be able to boot with them installed. i dont have access to the server i used for the inital developemt as that was a loan that is normaly used for ci and QE testing so i dont really have an option of uing it for protracted period of time to do developement. > > [0] > > https://review.opendev.org/c/openstack/nova/+/780866/3/ > > releasenotes/notes/vdpa-cc2300d2c46c150b.yaml > > Thanks. From eblock at nde.ag Thu Sep 9 12:11:24 2021 From: eblock at nde.ag (Eugen Block) Date: Thu, 09 Sep 2021 12:11:24 +0000 Subject: Horizon connection errors from object store In-Reply-To: Message-ID: <20210909121124.Horde.coxF4B2TZ6FfifJpjt6CbRs@webmail.nde.ag> Hi, I could reproduce this in my lab environment. The issue must be either in your ceph.conf on the RGW host(s) or your openstack role assigments. I have a dedicated user for my setup as you can see in my previous response. The user "rgw" gets then assigned the "member" role to the "service" project. If I login to Horizon dashboard with this user I can see the object-storage panel and see existing containers for that user. If I login as admin and try to see the container panel I get logged out, too. If I replace "rgw" with "admin" in the ceph.conf and restart the RGW it works. But note that in this case the admin user has to have the proper role assignment, too. So to achieve this you need to add a matching role (from "rgw keystone accepted roles") for your admin user in the respective project, like this: # replace rgw with admin in your case, PROJECT_ID is "service" in my case openstack role add --user rgw --project member # check with openstack role assignment list --names To make it easier to follow, please share your current ceph.conf and the openstack role assignment output. Regards, Eugen Zitat von Michel Niyoyita : > Hello team , > > I am facing an issue when I am trying to connect to the object store > containers on the horizon dashboad . Once click on containers it > automatically disconnect. please find below logs I am getting and help for > further analysis. > > [Thu Sep 09 06:35:22.185771 2021] [wsgi:error] [pid 167:tid > 139887608641280] [remote 10.10.29.150:55130] Attempted scope to domain > Default failed, will attempt to scope to another domain. > [Thu Sep 09 06:35:22.572522 2021] [wsgi:error] [pid 167:tid > 139887608641280] [remote 10.10.29.150:55130] Login successful for user > "admin" using domain "Default", remote address 10.10.29.150. > [Thu Sep 09 06:35:51.494815 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" > [Thu Sep 09 06:35:51.495140 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized > [Thu Sep 09 06:35:51.495541 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: > {'Content-Length': '119', 'X-Trans-Id': > 'tx00000000000000000000f-006139ab44-9fc1a-default', > 'X-Openstack-Request-Id': > 'tx00000000000000000000f-006139ab44-9fc1a-default', 'Accept-Ranges': > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, > 09 Sep 2021 06:35:51 GMT', 'Connection': 'Keep-Alive'} > [Thu Sep 09 06:35:51.495792 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: > b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000f-006139ab44-9fc1a-default","HostId":"9fc1a-default-default"}' > [Thu Sep 09 06:35:51.498743 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: > /api/swift/containers/ > [Thu Sep 09 06:35:52.924169 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" > [Thu Sep 09 06:35:52.924520 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized > [Thu Sep 09 06:35:52.924789 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: > {'Content-Length': '119', 'X-Trans-Id': > 'tx000000000000000000010-006139ab48-9fc1a-default', > 'X-Openstack-Request-Id': > 'tx000000000000000000010-006139ab48-9fc1a-default', 'Accept-Ranges': > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, > 09 Sep 2021 06:35:52 GMT', 'Connection': 'Keep-Alive'} > [Thu Sep 09 06:35:52.925034 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: > b'{"Code":"AccessDenied","RequestId":"tx000000000000000000010-006139ab48-9fc1a-default","HostId":"9fc1a-default-default"}' > [Thu Sep 09 06:35:52.929398 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: > /api/swift/containers/ > [Thu Sep 09 06:35:52.935799 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:56016] Logging out user "admin". > [Thu Sep 09 06:35:53.061489 2021] [wsgi:error] [pid 166:tid > 139887608641280] [remote 10.10.29.150:55806] Logging out user "". > [Thu Sep 09 06:35:54.541593 2021] [wsgi:error] [pid 165:tid > 139887608641280] [remote 10.10.29.150:55852] The request's session was > deleted before the request completed. The user may have logged out in a > concurrent request, for example. > [Thu Sep 09 06:35:54.542896 2021] [wsgi:error] [pid 165:tid > 139887608641280] [remote 10.10.29.150:55852] Bad Request: > /api/swift/policies/ > [Thu Sep 09 06:35:54.566055 2021] [wsgi:error] [pid 167:tid > 139887608641280] [remote 10.10.29.150:55860] The request's session was > deleted before the request completed. The user may have logged out in a > concurrent request, for example. > [Thu Sep 09 06:35:54.567130 2021] [wsgi:error] [pid 167:tid > 139887608641280] [remote 10.10.29.150:55860] Bad Request: /api/swift/info/ > (kolla-open1) stack at kolla-open1 > :/var/lib/docker/volumes/kolla_logs/_data/horizon$ > > Michel From fungi at yuggoth.org Thu Sep 9 13:09:03 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 9 Sep 2021 13:09:03 +0000 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: References: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> <20210907232144.n25ei2cyrmkr2xaz@yuggoth.org> <1aaf9274-f669-1e7b-f2f2-4cf795f60f0c@debian.org> Message-ID: <20210909130902.xgvked65l2ikt5rh@yuggoth.org> On 2021-09-08 11:31:31 +0200 (+0200), Herve Beraud wrote: [...] > An API seems quite a good idea > > I'm ok to give help on this point with my upstream cap and my Red > Hat cap. > > As release manager I added your proposition to our "things to > changes" on our release process and tools. I think that it should > be discussed first to see how to implement it and how to host it. > I think that the technical details would be a transversal decision > between the infra team and the release team and the volunteers on > this topic. [...] Note that an API in this sense can just be a consumable data structure. The amount of release metadata associated with a coordinated release of OpenStack isn't exactly tiny, but it's not so large it couldn't just be serialized into some sort of structured data file which tools can consume. The upper-constraints.txt redirector on the releases site provides one such example, though its not rich enough in detail for the described use case as far as I can tell. The series deliverable files in the releases repo are another such example, but they probably have a little too much detail and at the same time, as pointed out, don't associate independent deliverable releases with release cycles (the crux of the issue which is raised here). > I think a good starting point could be to design some specs or > propose some related goals, so, Thomas, maybe you could start by > creating something like that with your expectations. It will allow > us to start this topic, share our different feedback and our > expectations on it (our different viewpoints, at least from the > release team, infra team, and downstream/distros teams). [...] Which then begs the question, where should such a design specification document be proposed/reviewed? The release team doesn't have a specs repo, but also it might be sufficient to attempt to use this mailing list to hammer out a clearer explanation of the data people are looking for and the methods they would be comfortable employing to obtain it. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From hberaud at redhat.com Thu Sep 9 13:38:59 2021 From: hberaud at redhat.com (Herve Beraud) Date: Thu, 9 Sep 2021 15:38:59 +0200 Subject: [oslo] RFE requested for oslo-messaging In-Reply-To: References: Message-ID: Yes simply need a RFE (requirement freeze exception) as we past the requirement freeze date. Le jeu. 9 sept. 2021 à 13:01, Radosław Piliszek a écrit : > On Thu, Sep 9, 2021 at 12:44 PM Nikita Kalyanov > wrote: > > > > Dear Release Team, > > > > We experienced strange freezes and timeouts for a long time on our > > OpenStack deployment [1] and some duplicate messages [2]. After > > applying the fixes [3], [4] both bugs are gone. > > Hmm, I have read the bug reports out of curiosity and I think we might > be hitting these issues randomly in Kolla CI (due to load or bad > network). > (I am obviously unsure whether the proposed fixes are correct or not.) > > As these are bugfixes, they should not need a freeze exception per se. > I hope Oslo messaging cores respond soon. > > -yoctozepto > > > > > Is a freeze exception possible for these? > > > > Thanks! > > nkalyanov > > > > [1] https://bugs.launchpad.net/oslo.messaging/+bug/1935864 > > [2] https://bugs.launchpad.net/oslo.messaging/+bug/1935883 > > [3] https://review.opendev.org/c/openstack/oslo.messaging/+/800564 > > [4] https://review.opendev.org/c/openstack/oslo.messaging/+/800575 > > > > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Thu Sep 9 14:04:36 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 9 Sep 2021 14:04:36 +0000 Subject: [OSSA-2021-006] Neutron: Routes middleware memory leak for nonexistent controllers (CVE-2021-40797) Message-ID: <20210909140435.6u5kg2reejf6ctse@yuggoth.org> ======================================================================== OSSA-2021-006: Routes middleware memory leak for nonexistent controllers ======================================================================== :Date: September 09, 2021 :CVE: CVE-2021-40797 Affects ~~~~~~~ - Neutron: <16.4.1, >=17.0.0 <17.2.1, >=18.0.0 <18.1.1 Description ~~~~~~~~~~~ Slawek Kaplonski with Red Hat reported a vulnerability in Neutron's routes middleware. By making API requests involving nonexistent controllers, an authenticated user may cause the API worker to consume increasing amounts of memory, resulting in API performance degradation or denial of service. All Neutron deployments are affected. Patches ~~~~~~~ - https://review.opendev.org/807638 (Queens) - https://review.opendev.org/807637 (Rocky) - https://review.opendev.org/807636 (Stein) - https://review.opendev.org/807635 (Train) - https://review.opendev.org/807634 (Ussuri) - https://review.opendev.org/807633 (Victoria) - https://review.opendev.org/807632 (Wallaby) - https://review.opendev.org/807335 (Xena) Credits ~~~~~~~ - Slawek Kaplonski from Red Hat (CVE-2021-40797) References ~~~~~~~~~~ - https://launchpad.net/bugs/1942179 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-40797 Notes ~~~~~ - The stable/train, stable/stein, stable/rocky, and stable/queens branches are under extended maintenance and will receive no new point releases, but patches for them are provided as a courtesy. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From zhang.lei.fly at gmail.com Thu Sep 9 07:31:19 2021 From: zhang.lei.fly at gmail.com (Jeffrey Zhang) Date: Thu, 9 Sep 2021 15:31:19 +0800 Subject: Question about vDPA support for live migration Message-ID: Hey everyone, I am testing vDPA on OpenStack Wallaby. I have set it up successfully. But I found there are several features that are not implemented, including live/code migration[0], which I wanted. Anyone know what's the latest progress about this? I found no talk about this. [0] https://review.opendev.org/c/openstack/nova/+/780866/3/releasenotes/notes/vdpa-cc2300d2c46c150b.yaml --- Regards, Jeffrey Zhang Blog: http://xcodest.me -------------- next part -------------- An HTML attachment was scrubbed... URL: From micou12 at gmail.com Thu Sep 9 07:38:16 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Thu, 9 Sep 2021 09:38:16 +0200 Subject: Horizon connection errors from object store Message-ID: Hello team , I am facing an issue when I am trying to connect to the object store containers on the horizon dashboad . Once click on containers it automatically disconnect. please find below logs I am getting and help for further analysis. [Thu Sep 09 06:35:22.185771 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55130] Attempted scope to domain Default failed, will attempt to scope to another domain. [Thu Sep 09 06:35:22.572522 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55130] Login successful for user "admin" using domain "Default", remote address 10.10.29.150. [Thu Sep 09 06:35:51.494815 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" [Thu Sep 09 06:35:51.495140 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized [Thu Sep 09 06:35:51.495541 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx00000000000000000000f-006139ab44-9fc1a-default', 'X-Openstack-Request-Id': 'tx00000000000000000000f-006139ab44-9fc1a-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, 09 Sep 2021 06:35:51 GMT', 'Connection': 'Keep-Alive'} [Thu Sep 09 06:35:51.495792 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000f-006139ab44-9fc1a-default","HostId":"9fc1a-default-default"}' [Thu Sep 09 06:35:51.498743 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Unauthorized: /api/swift/containers/ [Thu Sep 09 06:35:52.924169 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" [Thu Sep 09 06:35:52.924520 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 Unauthorized [Thu Sep 09 06:35:52.924789 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx000000000000000000010-006139ab48-9fc1a-default', 'X-Openstack-Request-Id': 'tx000000000000000000010-006139ab48-9fc1a-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, 09 Sep 2021 06:35:52 GMT', 'Connection': 'Keep-Alive'} [Thu Sep 09 06:35:52.925034 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx000000000000000000010-006139ab48-9fc1a-default","HostId":"9fc1a-default-default"}' [Thu Sep 09 06:35:52.929398 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Unauthorized: /api/swift/containers/ [Thu Sep 09 06:35:52.935799 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:56016] Logging out user "admin". [Thu Sep 09 06:35:53.061489 2021] [wsgi:error] [pid 166:tid 139887608641280] [remote 10.10.29.150:55806] Logging out user "". [Thu Sep 09 06:35:54.541593 2021] [wsgi:error] [pid 165:tid 139887608641280] [remote 10.10.29.150:55852] The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example. [Thu Sep 09 06:35:54.542896 2021] [wsgi:error] [pid 165:tid 139887608641280] [remote 10.10.29.150:55852] Bad Request: /api/swift/policies/ [Thu Sep 09 06:35:54.566055 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55860] The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example. [Thu Sep 09 06:35:54.567130 2021] [wsgi:error] [pid 167:tid 139887608641280] [remote 10.10.29.150:55860] Bad Request: /api/swift/info/ (kolla-open1) stack at kolla-open1 :/var/lib/docker/volumes/kolla_logs/_data/horizon$ [image: image.png] [image: image.png] Michel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 18056 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 13884 bytes Desc: not available URL: From zigo at debian.org Thu Sep 9 15:12:01 2021 From: zigo at debian.org (Thomas Goirand) Date: Thu, 9 Sep 2021 17:12:01 +0200 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: <20210909130902.xgvked65l2ikt5rh@yuggoth.org> References: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> <20210907232144.n25ei2cyrmkr2xaz@yuggoth.org> <1aaf9274-f669-1e7b-f2f2-4cf795f60f0c@debian.org> <20210909130902.xgvked65l2ikt5rh@yuggoth.org> Message-ID: On 9/9/21 3:09 PM, Jeremy Stanley wrote: > On 2021-09-08 11:31:31 +0200 (+0200), Herve Beraud wrote: > [...] >> An API seems quite a good idea >> >> I'm ok to give help on this point with my upstream cap and my Red >> Hat cap. >> >> As release manager I added your proposition to our "things to >> changes" on our release process and tools. I think that it should >> be discussed first to see how to implement it and how to host it. >> I think that the technical details would be a transversal decision >> between the infra team and the release team and the volunteers on >> this topic. > [...] > > Note that an API in this sense can just be a consumable data > structure. The amount of release metadata associated with a > coordinated release of OpenStack isn't exactly tiny, but it's not so > large it couldn't just be serialized into some sort of structured > data file which tools can consume. Yes, that's more or less what I was thinking. Something like: curl -X GET http://releases.openstack.org/api/release/xena { "release-name": "xena", "components": { "oslo.config": { "version": "8.7.1", "release-date": "Thu Jul 15 10:19:24 2021 +0000", }, "oslo.db": { "version": "11.0.0", "release-date": "Mon Aug 23 10:23:24 2021 +0000", }, ... } } That'd be a way more reliable to parse than the HTML page like we do right now... > The series deliverable files in the releases repo are > another such example, but they probably have a little too much > detail and at the same time, as pointed out, don't associate > independent deliverable releases with release cycles (the crux of > the issue which is raised here). Where's the "series deliverable files" that you're talking about? > Which then begs the question, where should such a design > specification document be proposed/reviewed? The release team > doesn't have a specs repo, but also it might be sufficient to > attempt to use this mailing list to hammer out a clearer explanation > of the data people are looking for and the methods they would be > comfortable employing to obtain it. Same over here: where should we propose such specs? Cheers, Thomas Goirand (zigo) From fungi at yuggoth.org Thu Sep 9 15:21:45 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 9 Sep 2021 15:21:45 +0000 Subject: [all] Dangerous trend toward too many release-independent components In-Reply-To: References: <20210907133947.wmosbwxdjkh6cren@yuggoth.org> <20210907232144.n25ei2cyrmkr2xaz@yuggoth.org> <1aaf9274-f669-1e7b-f2f2-4cf795f60f0c@debian.org> <20210909130902.xgvked65l2ikt5rh@yuggoth.org> Message-ID: <20210909152144.47wphduzd76dz67m@yuggoth.org> On 2021-09-09 17:12:01 +0200 (+0200), Thomas Goirand wrote: > On 9/9/21 3:09 PM, Jeremy Stanley wrote: [...] > > The series deliverable files in the releases repo are > > another such example, but they probably have a little too much > > detail and at the same time, as pointed out, don't associate > > independent deliverable releases with release cycles (the crux of > > the issue which is raised here). > > Where's the "series deliverable files" that you're talking about? [...] https://opendev.org/openstack/releases/src/branch/master/deliverables Perhaps unsurprisingly, those pages of HTML you're parsing from the releases site are built from structured data (YAML) in the openstack/releases Git repository using a Python library also maintained in that same repo, and continuously published with Zuul jobs similar to how we publish other documentation. This Sphinx extension is a convenient window into most of it: https://opendev.org/openstack/releases/src/branch/master/doc/source/_exts/deliverables.py -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From tonyliu0592 at hotmail.com Thu Sep 9 16:39:14 2021 From: tonyliu0592 at hotmail.com (Tony Liu) Date: Thu, 9 Sep 2021 16:39:14 +0000 Subject: Cinder API timeout on single-control node In-Reply-To: <20210901120225.Horde.Fyn_birethoRxxPJpwzbF3O@webmail.nde.ag> References: <20210901120225.Horde.Fyn_birethoRxxPJpwzbF3O@webmail.nde.ag> Message-ID: The first issue is 504 timeout, update timeout in haproxy helps on that. The next is the timeout from cinder-api, [1] helps on that. Then the next is rbd client timeout. I started "debug RBD timeout issue" thread for that. It seems that the root cause of this series timeout is from Ceph. I followed comments from Konstantin to use msgr2 only. Hopefully that will fix the whole timeout issue. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1930806 Thanks! Tony ________________________________________ From: Eugen Block Sent: September 1, 2021 05:02 AM To: openstack-discuss at lists.openstack.org Subject: Cinder API timeout on single-control node Hi *, since your last responses were quite helpful regarding rabbitmq I would like to ask a different question for the same environment. It's an older openstack version (Pike) running with only one control node. There already were lots of volumes (way more than 1000) in that cloud, but after adding a bunch more (not sure how many exactly) in one project the whole cinder api became extremely slow. Both horizon and CLI run into timeouts: [Wed Sep 01 13:18:52.109178 2021] [wsgi:error] [pid 60440] [client :58474] Timeout when reading response headers from daemon process 'horizon': /srv/www/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi, referer: http:///project/volumes/ [Wed Sep 01 13:18:53.664714 2021] [wsgi:error] [pid 13007] Not Found: /favicon.ico Sometimes the volume creation succeeds if you just retry, but it often fails. The dashboard shows a "504 gateway timeout" after two minutes (also after four minutes after I increased the timeout for the apache dashboard config). The timeout also shows even if I try to get into the volumes tab of an empty project. A couple of weeks ago I already noticed some performance issues with cinder api if there are lots of attached volumes, if there are many "available" volumes it doesn't seem to slow things down. But since then the total number of volumes has doubled. At the moment there are more than 960 attached volumes across all projects and more than 750 detached volumes. I searched the cinder.conf for any helpful setting but I'm not sure which would actually help. And since it's a production cloud I would like to avoid restarting services all the time just to try something. Maybe some of you can point me in the right direction? I would appreciate any help! If there's more information I can provide just let me know. Thanks! Eugen From gmann at ghanshyammann.com Thu Sep 9 16:47:00 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Thu, 09 Sep 2021 11:47:00 -0500 Subject: [all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community In-Reply-To: <17b4f7fc2ab.be9841f87523.859289224419125442@ghanshyammann.com> References: <17b4f7fc2ab.be9841f87523.859289224419125442@ghanshyammann.com> Message-ID: <17bcb74964e.101c130b3259268.4181971006334121517@ghanshyammann.com> ---- On Mon, 16 Aug 2021 10:06:18 -0500 Ghanshyam Mann wrote ---- > ---- On Wed, 13 Jan 2021 04:59:47 -0600 Thierry Carrez wrote ---- > > Laurent Dumont wrote: > > > This seems really interesting. Tracing events with request-ids is > > > something that is quite useful. > > > > > > What is the current state? Can it be deployed by a third party? > > > > I see code up at https://opendev.org/inspur/ but I haven't tried > > deploying it. > > > > If it gathers momentum, I suspect it will be proposed as a new official > > OpenStack project, and if the Technical Committee approves it, it will > > be moved under the openstack/ namespace on opendev.org. It already > > follows our usual repository structure (venus, python-venusclient, > > venus-tempest-plugin...) > > +1, I agree that this is ready to apply as an official project and then we can start > the discussion in TC about checking the requirement. > > Please propose the patch to governance and I am adding it to the next TC meeting agenda too. > > - https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestions Thanks to the 'Venus' project team for all the discussion and welcome to the OpenStack governance projects. TC has merged the project application, you can start the process of moving it under openstack/ namespace: https://review.opendev.org/c/openstack/governance/+/804824 -gmann > > -gmann > > > > > -- > > Thierry > > > > > > From gouthampravi at gmail.com Thu Sep 9 17:07:15 2021 From: gouthampravi at gmail.com (Goutham Pacha Ravi) Date: Thu, 9 Sep 2021 10:07:15 -0700 Subject: [designate][interop] test_create_all_recordset_types ddt issue In-Reply-To: References: <3229108.usfYGdeWWP@whitebase.usersys.redhat.com> Message-ID: On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec wrote: > From interop perspective it's also better not to have multiple tests with > the same id. > We encountered one more problem with ddt - the test names seem not to be > generated consistently, see this: > https://paste.opendev.org/show/809187/ > The test can have either _00009_TXT suffix or _9_TXT one. > > Until we figure this out, I think we will need to flag the test in interop > - so that a skip of the test (because of the name mismatch in this case) > won't make the whole guideline fail. > > Luigi's idea is great. Every test should be identified by a unique id and > it shouldn't matter that the test is generated (ddt). Different input data > -> different test -> different name -> different id. > Let's try to explore whether having a unique id per ddt entry is possible. > I think we could annotate the test inputs and include a unique UUID included in the test name/doc per test input. Something along these lines: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114 However, adding the UUID as a test attr (a functionality of "testtools" [1]) will require a hook within ddt likely. However, as we've discovered, there's no way to avoid adding an ordinal into the generated test name when using DDT, and so we'll have to either set a PYTHONHASHSEED and disable hash randomization, or ensure that values appear in the same order all the time (example [2]) [1] https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc9d68320330aa/testtools/testcase.py#L892 [2] https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859 > > > On Wed, 8 Sept 2021 at 18:15, Luigi Toscano wrote: > >> On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote: >> > If the use of "ddt" is a big problem for the compliance testing, we >> > can consider breaking these out into individual tests (likely to be >> > duplicates to the existing "ddt" tests to maintain the legacy test >> > UUIDs). >> >> I was wondering: wouldn't it be possible to expand or use ddt somehow to >> inject different UUIDs for each generated test? You know in advance how >> many >> tests are going to be generated. >> >> -- >> Luigi >> >> >> > > -- > Martin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennelson11 at gmail.com Thu Sep 9 18:12:00 2021 From: kennelson11 at gmail.com (Kendall Nelson) Date: Thu, 9 Sep 2021 11:12:00 -0700 Subject: [all][elections][ptl] Project Team Lead Election Conclusion and Results In-Reply-To: References: Message-ID: Thank you to the election officials for their hard work! -Kendall On Wed, Sep 8, 2021 at 4:57 AM Ian Y. Choi wrote: > Thank you to the electorate, to all those who voted and to all candidates > who put their name forward for Project Team Lead (PTL) in this election. A > healthy, open process breeds trust in our decision making capability thank > you to all those who make this process possible. > > Now for the results of the PTL election process, please join me in > extending congratulations to the following PTLs: > > * Barbican : Douglas Mendizábal > * Blazar : Pierre Riteau > * Cinder : Brian Rosmaita > * Cloudkitty : Rafael Weingartner > * Cyborg : Bailin Zhang > * Designate : Michael Johnson > * Ec2 Api : Andrey Pavlov > * Freezer : ge cong > * Glance : Abhishek Kekane > * Horizon : Vishal Manchanda > * Ironic : Iury Gregory Melo Ferreira > * Kolla : Michal Nasiadka > * Kuryr : Maysa de Macedo Souza > * Magnum : Feilong Wang > * Manila : Goutham Pacha Ravi > * Masakari : Radosław Piliszek > * Murano : Rong Zhu > * Neutron : Lajos Katona > * Nova : Sylvain Bauza > * Octavia : Gregory Thiemonge > * OpenStack Charms : Alex Kavanagh > * Openstack Chef : Lance Albertson > * OpenStack Helm : Gage Hugo > * OpenStackAnsible : Dmitriy Rabotyagov > * OpenStackSDK : Artem Goncharov > * Quality Assurance : Martin Kopec > * Rally : Andrey Kurilin > * Release Management : Elod Illes > * Requirements : Matthew Thode > * Senlin : XueFeng Liu > * Solum : Rong Zhu > * Storlets : Takashi Kajinami > * Swift : Tim Burke > * Tacker : Yasufumi Ogawa > * Telemetry : Matthias Runge > * Tripleo : James Slagle > * Trove : Lingxian Kong > * Vitrage : Eyal Bar-Ilan > * Watcher : chen ke > * Winstackers : Lucian Petrut > * Zaqar : wang hao > > Elections: > * Cyborg: https://civs1.civs.us/cgi-bin/results.pl?id=E_807146df7d6a1d3b > > Election process details and results are also available here: > https://governance.openstack.org/election/ > > Thank you to all involved in the PTL election process, > > Ian Y. Choi, on behalf of the OpenStack Technical Election Officials > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosmaita.fossdev at gmail.com Thu Sep 9 19:53:02 2021 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Thu, 9 Sep 2021 15:53:02 -0400 Subject: [cinder] review patches with FFE first Message-ID: <27e5cce7-6f91-b5c8-e244-6ca0e195e2f3@gmail.com> For anyone who missed yesterday's cinder meeting, we are conveniently tracking the patches that have FFEs on this etherpad: https://etherpad.opendev.org/p/cinder-xena-ffe Please give them the highest review priority. cheers, brian From zhangbailin at inspur.com Fri Sep 10 05:35:34 2021 From: zhangbailin at inspur.com (=?utf-8?B?QnJpbiBaaGFuZyjlvKDnmb7mnpcp?=) Date: Fri, 10 Sep 2021 05:35:34 +0000 Subject: =?utf-8?B?562U5aSNOiBbbGlzdHMub3BlbnN0YWNrLm9yZ+S7o+WPkV1SZTogW2FsbF1J?= =?utf-8?B?bnRyb2R1Y3Rpb24gdG8gdmVudXMgd2hpY2ggaXMgdGhlIHByb2plY3Qgb2Yg?= =?utf-8?B?bG9nIG1hbmFnZW1lbnQgYW5kIGhhcyBiZWVuIGNvbnRyaWJ1dGVkIHRvIHRo?= =?utf-8?Q?e_OpenStack_community?= In-Reply-To: <17bcb74964e.101c130b3259268.4181971006334121517@ghanshyammann.com> References: <6c371e048e99587470d9150399806c60@sslemail.net> <17bcb74964e.101c130b3259268.4181971006334121517@ghanshyammann.com> Message-ID: <12295f57f8814f0294a87087eda4d5ce@inspur.com> Thanks TC team and all contributors. We hope that more contributors can join Venus. If you have any questions, you can contact us via ML ([venus]) or #openstack-venus channel. brinzhang -----邮件原件----- 发件人: Ghanshyam Mann [mailto:gmann at ghanshyammann.com] 发送时间: 2021年9月10日 0:47 收件人: openstack-discuss 主题: [lists.openstack.org代发]Re: [all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community ---- On Mon, 16 Aug 2021 10:06:18 -0500 Ghanshyam Mann wrote ---- > ---- On Wed, 13 Jan 2021 04:59:47 -0600 Thierry Carrez wrote ---- > > Laurent Dumont wrote: > > > This seems really interesting. Tracing events with request-ids is > > > something that is quite useful. > > > > > > What is the current state? Can it be deployed by a third party? > > > > I see code up at https://opendev.org/inspur/ but I haven't tried > > deploying it. > > > > If it gathers momentum, I suspect it will be proposed as a new official > > OpenStack project, and if the Technical Committee approves it, it will > > be moved under the openstack/ namespace on opendev.org. It already > > follows our usual repository structure (venus, python-venusclient, > > venus-tempest-plugin...) > > +1, I agree that this is ready to apply as an official project and then we can start > the discussion in TC about checking the requirement. > > Please propose the patch to governance and I am adding it to the next TC meeting agenda too. > > - https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestions Thanks to the 'Venus' project team for all the discussion and welcome to the OpenStack governance projects. TC has merged the project application, you can start the process of moving it under openstack/ namespace: https://review.opendev.org/c/openstack/governance/+/804824 -gmann > > -gmann > > > > > -- > > Thierry > > > > > > From zhangbailin at inspur.com Fri Sep 10 05:46:42 2021 From: zhangbailin at inspur.com (=?utf-8?B?QnJpbiBaaGFuZyjlvKDnmb7mnpcp?=) Date: Fri, 10 Sep 2021 05:46:42 +0000 Subject: [cyborg] RFE requested for Cyborg Message-ID: Dear Release Team, We experienced strange freezes and timeouts for a long time on our python-cyborgclient need to update the lost feature of device profile 'description' [1] And we have merged this at 7, Sep, that we can update [2]'s hash then to publish it. [1] https://review.opendev.org/c/openstack/python-cyborgclient/+/803886 [2] https://review.opendev.org/c/openstack/releases/+/807624 brinzhang From hberaud at redhat.com Fri Sep 10 07:12:24 2021 From: hberaud at redhat.com (Herve Beraud) Date: Fri, 10 Sep 2021 09:12:24 +0200 Subject: [cyborg] RFE requested for Cyborg In-Reply-To: References: Message-ID: +1 for me, we need validation from the requirement team. Le ven. 10 sept. 2021 à 07:47, Brin Zhang(张百林) a écrit : > Dear Release Team, > > We experienced strange freezes and timeouts for a long time on our > python-cyborgclient need to update the lost feature of device profile > 'description' [1] > > And we have merged this at 7, Sep, that we can update [2]'s hash then to > publish it. > > [1] https://review.opendev.org/c/openstack/python-cyborgclient/+/803886 > [2] https://review.opendev.org/c/openstack/releases/+/807624 > > brinzhang > > > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Fri Sep 10 07:22:11 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 10 Sep 2021 09:22:11 +0200 Subject: [tc][operators][all] Querying operators about their use of TC tags Message-ID: Dear OpenStack Operators, As part of Technical Committee's (TC's) efforts for this cycle (Xena) [1] we are working on an analysis of the usefulness of the Tags framework [2]. To this end, a summarising spreadsheet with tag distribution and applicable notes was started by me. [3] As the Tags framework is aimed at operators, we (TC) have a question to you (operators) whether you are aware of the existence of the Tags framework and whether it is something that you actually use to make deployment decisions. Please reply to this mail if you use the Tags framework and let us know how. [1] https://etherpad.opendev.org/p/tc-xena-tracker [2] https://governance.openstack.org/tc/reference/tags/index.html [3] https://docs.google.com/spreadsheets/d/18GXibtdQnSkIwA7DsBvX9dPwbw9JH76AhhRUknzBb3Q Kind regards, Radosław Piliszek (yoctozepto) OpenStack TC vice-chair From radoslaw.piliszek at gmail.com Fri Sep 10 07:32:51 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 10 Sep 2021 09:32:51 +0200 Subject: Quay.io availability in China Message-ID: Dears, I have a question (mostly to our readers from China) whether https://quay.io/ is freely accessible from there. Asking because Kolla is planning to switch from Docker Hub (which has some local mirror in China as far as we know) to Quay.io for public images. -yoctozepto From eblock at nde.ag Fri Sep 10 07:46:56 2021 From: eblock at nde.ag (Eugen Block) Date: Fri, 10 Sep 2021 07:46:56 +0000 Subject: Horizon connection errors from object store In-Reply-To: References: <20210909121124.Horde.coxF4B2TZ6FfifJpjt6CbRs@webmail.nde.ag> Message-ID: <20210910074656.Horde.X90CJeWqF5dLKyczAtLrz7X@webmail.nde.ag> Check the log files, the dashboard errors suggest that it could be a policy issue, the openstack-dashboard-error-logs should be more verbose. Then check keystone and rgw logs. What errors do you get when you try to create a container from CLI (add '--debug' flag to see more)? Did you source the correct openrc file? Zitat von Michel Niyoyita : > Hello Eugen > > Thank you for your continuous support. now The dashboard is stable is not > dsconnected as before , unfotunately I am not able to create containers and > see the list of created one using openstack CLI or ceph side. > > below is my ceph.conf : > > [client.rgw.ceph-osd3] > rgw frontends = "beast port=8080" > rgw dns name = ceph-osd3 > rgw enable usage log = true > > rgw thread pool size = 512 > rgw keystone api version = 3 > rgw keystone url = http://kolla-open1:5000 > > rgw keystone admin user = rgw > rgw keystone admin password = c8igBKQqEon8jXaG68TkcWgNI4E77m2K3bJD7fCU > rgw keystone admin domain = default > rgw keystone admin project = service > rgw keystone accepted roles = admin,Member,_member_,member,swiftoperator > rgw keystone verify ssl = false > rgw s3 auth use keystone = true > rgw keystone revocation interval = 0 > > > [client.rgw.ceph-osd3.rgw0] > host = ceph-osd3 > keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring > log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log > rgw frontends = beast endpoint=ceph-osd3:8080 > rgw thread pool size = 512 > > openstack role assignment lis --names output: > > > (kolla-open1) stack at kolla-open1:~$ openstack role assignment list --names > > > +------------------+------------------------------------+-------+-----------------+-------- > ----------+--------+-----------+ > | Role | User | Group | Project > | Domain | System | Inherited | > +------------------+------------------------------------+-------+-----------------+-------- > ----------+--------+-----------+ > | swiftoperator | operator:swift at Default | | > service at Default | | | False | > | admin | rgw at Default | | > service at Default | | | False | > | member | rgw at Default | | > service at Default | | | False | > | admin | cinder at Default | | > service at Default | | | False | > | admin | neutron at Default | | > service at Default | | | False | > | admin | placement at Default | | > service at Default | | | False | > | admin | nova at Default | | > service at Default | | | False | > | admin | admin at Default | | > admin at Default | | | False | > | heat_stack_owner | admin at Default | | > admin at Default | | | False | > | admin | admin at Default | | > service at Default | | | False | > | member | admin at Default | | > service at Default | | | False | > | admin | glance at Default | | > service at Default | | | False | > | member | operator at Default | | > service at Default | | | False | > | _member_ | operator at Default | | > service at Default | | | False | > | admin | heat at Default | | > service at Default | | | False | > | admin | heat_domain_admin at heat_user_domain | | > | heat_us er_domain | | False | > | admin | admin at Default | | > | | all | False | > +------------------+------------------------------------+-------+-----------------+-------- > > Michel > > > On Fri, Sep 10, 2021 at 9:33 AM Michel Niyoyita wrote: > >> Hello Eugen >> >> Thank you for your continuous support. now The dashboard is stable is not >> dsconnected as before , unfotunately I am not able to create containers and >> see the list of created one using openstack CLI or ceph side. you will find >> the image at the end. >> >> below is my ceph.conf : >> >> [client.rgw.ceph-osd3] >> rgw frontends = "beast port=8080" >> rgw dns name = ceph-osd3 >> rgw enable usage log = true >> >> rgw thread pool size = 512 >> rgw keystone api version = 3 >> rgw keystone url = http://kolla-open1:5000 >> >> rgw keystone admin user = rgw >> rgw keystone admin password = c8igBKQqEon8jXaG68TkcWgNI4E77m2K3bJD7fCU >> rgw keystone admin domain = default >> rgw keystone admin project = service >> rgw keystone accepted roles = admin,Member,_member_,member,swiftoperator >> rgw keystone verify ssl = false >> rgw s3 auth use keystone = true >> rgw keystone revocation interval = 0 >> >> >> [client.rgw.ceph-osd3.rgw0] >> host = ceph-osd3 >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log >> rgw frontends = beast endpoint=ceph-osd3:8080 >> rgw thread pool size = 512 >> >> openstack role assignment lis --names output: >> >> >> (kolla-open1) stack at kolla-open1:~$ openstack role assignment list --names >> >> >> +------------------+------------------------------------+-------+-----------------+-------- >> ----------+--------+-----------+ >> | Role | User | Group | Project >> | Domain | System | Inherited | >> +------------------+------------------------------------+-------+-----------------+-------- >> ----------+--------+-----------+ >> | swiftoperator | operator:swift at Default | | >> service at Default | | | False | >> | admin | rgw at Default | | >> service at Default | | | False | >> | member | rgw at Default | | >> service at Default | | | False | >> | admin | cinder at Default | | >> service at Default | | | False | >> | admin | neutron at Default | | >> service at Default | | | False | >> | admin | placement at Default | | >> service at Default | | | False | >> | admin | nova at Default | | >> service at Default | | | False | >> | admin | admin at Default | | >> admin at Default | | | False | >> | heat_stack_owner | admin at Default | | >> admin at Default | | | False | >> | admin | admin at Default | | >> service at Default | | | False | >> | member | admin at Default | | >> service at Default | | | False | >> | admin | glance at Default | | >> service at Default | | | False | >> | member | operator at Default | | >> service at Default | | | False | >> | _member_ | operator at Default | | >> service at Default | | | False | >> | admin | heat at Default | | >> service at Default | | | False | >> | admin | heat_domain_admin at heat_user_domain | | >> | heat_us er_domain | | False | >> | admin | admin at Default | | >> | | all | False | >> >> +------------------+------------------------------------+-------+-----------------+-------- >> >> [image: image.png] >> >> Michel >> >> >> On Thu, Sep 9, 2021 at 2:15 PM Eugen Block wrote: >> >>> Hi, >>> >>> I could reproduce this in my lab environment. The issue must be either >>> in your ceph.conf on the RGW host(s) or your openstack role >>> assigments. I have a dedicated user for my setup as you can see in my >>> previous response. The user "rgw" gets then assigned the "member" role >>> to the "service" project. If I login to Horizon dashboard with this >>> user I can see the object-storage panel and see existing containers >>> for that user. If I login as admin and try to see the container panel >>> I get logged out, too. If I replace "rgw" with "admin" in the >>> ceph.conf and restart the RGW it works. But note that in this case the >>> admin user has to have the proper role assignment, too. >>> >>> So to achieve this you need to add a matching role (from "rgw keystone >>> accepted roles") for your admin user in the respective project, like >>> this: >>> >>> # replace rgw with admin in your case, PROJECT_ID is "service" in my case >>> openstack role add --user rgw --project member >>> >>> # check with >>> openstack role assignment list --names >>> >>> To make it easier to follow, please share your current ceph.conf and >>> the openstack role assignment output. >>> >>> Regards, >>> Eugen >>> >>> >>> >>> Zitat von Michel Niyoyita : >>> >>> > Hello team , >>> > >>> > I am facing an issue when I am trying to connect to the object store >>> > containers on the horizon dashboad . Once click on containers it >>> > automatically disconnect. please find below logs I am getting and help >>> for >>> > further analysis. >>> > >>> > [Thu Sep 09 06:35:22.185771 2021] [wsgi:error] [pid 167:tid >>> > 139887608641280] [remote 10.10.29.150:55130] Attempted scope to domain >>> > Default failed, will attempt to scope to another domain. >>> > [Thu Sep 09 06:35:22.572522 2021] [wsgi:error] [pid 167:tid >>> > 139887608641280] [remote 10.10.29.150:55130] Login successful for user >>> > "admin" using domain "Default", remote address 10.10.29.150. >>> > [Thu Sep 09 06:35:51.494815 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i >>> > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H >>> > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" >>> > [Thu Sep 09 06:35:51.495140 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 >>> Unauthorized >>> > [Thu Sep 09 06:35:51.495541 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: >>> > {'Content-Length': '119', 'X-Trans-Id': >>> > 'tx00000000000000000000f-006139ab44-9fc1a-default', >>> > 'X-Openstack-Request-Id': >>> > 'tx00000000000000000000f-006139ab44-9fc1a-default', 'Accept-Ranges': >>> > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': >>> 'Thu, >>> > 09 Sep 2021 06:35:51 GMT', 'Connection': 'Keep-Alive'} >>> > [Thu Sep 09 06:35:51.495792 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: >>> > >>> b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000f-006139ab44-9fc1a-default","HostId":"9fc1a-default-default"}' >>> > [Thu Sep 09 06:35:51.498743 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: >>> > /api/swift/containers/ >>> > [Thu Sep 09 06:35:52.924169 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i >>> > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H >>> > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" >>> > [Thu Sep 09 06:35:52.924520 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 >>> Unauthorized >>> > [Thu Sep 09 06:35:52.924789 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: >>> > {'Content-Length': '119', 'X-Trans-Id': >>> > 'tx000000000000000000010-006139ab48-9fc1a-default', >>> > 'X-Openstack-Request-Id': >>> > 'tx000000000000000000010-006139ab48-9fc1a-default', 'Accept-Ranges': >>> > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': >>> 'Thu, >>> > 09 Sep 2021 06:35:52 GMT', 'Connection': 'Keep-Alive'} >>> > [Thu Sep 09 06:35:52.925034 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: >>> > >>> b'{"Code":"AccessDenied","RequestId":"tx000000000000000000010-006139ab48-9fc1a-default","HostId":"9fc1a-default-default"}' >>> > [Thu Sep 09 06:35:52.929398 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: >>> > /api/swift/containers/ >>> > [Thu Sep 09 06:35:52.935799 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:56016] Logging out user "admin". >>> > [Thu Sep 09 06:35:53.061489 2021] [wsgi:error] [pid 166:tid >>> > 139887608641280] [remote 10.10.29.150:55806] Logging out user "". >>> > [Thu Sep 09 06:35:54.541593 2021] [wsgi:error] [pid 165:tid >>> > 139887608641280] [remote 10.10.29.150:55852] The request's session was >>> > deleted before the request completed. The user may have logged out in a >>> > concurrent request, for example. >>> > [Thu Sep 09 06:35:54.542896 2021] [wsgi:error] [pid 165:tid >>> > 139887608641280] [remote 10.10.29.150:55852] Bad Request: >>> > /api/swift/policies/ >>> > [Thu Sep 09 06:35:54.566055 2021] [wsgi:error] [pid 167:tid >>> > 139887608641280] [remote 10.10.29.150:55860] The request's session was >>> > deleted before the request completed. The user may have logged out in a >>> > concurrent request, for example. >>> > [Thu Sep 09 06:35:54.567130 2021] [wsgi:error] [pid 167:tid >>> > 139887608641280] [remote 10.10.29.150:55860] Bad Request: >>> /api/swift/info/ >>> > (kolla-open1) stack at kolla-open1 >>> > :/var/lib/docker/volumes/kolla_logs/_data/horizon$ >>> > >>> > Michel >>> >>> >>> >>> >>> From eblock at nde.ag Fri Sep 10 07:54:37 2021 From: eblock at nde.ag (Eugen Block) Date: Fri, 10 Sep 2021 07:54:37 +0000 Subject: Cinder API timeout on single-control node In-Reply-To: References: <20210901120225.Horde.Fyn_birethoRxxPJpwzbF3O@webmail.nde.ag> Message-ID: <20210910075437.Horde.Q4WjpcHjOkcXpW99hLbqtdK@webmail.nde.ag> Thanks, Tony. I enabled debug logs for cinder and restarted cinder-api but then the issue was not reproducable. Suddenly all api calls (even 'cinder list --all') took less than a minute so we haven't seen timeouts since then. I turned off debug logs and am still waiting for this to reoccur. In the meantime they also deleted 600 unused volumes, that probably helped, too. Anyway, I doubled the rpc_response_timeout in cinder.conf now and will wait until this happens again. I don't find anything like a cinder wsgi timeout for a vhost, also there's no haproxy involved because it's only one control node deployed with chef and crowbar. I'll report back if anything happens. Thanks! Eugen Zitat von Tony Liu : > The first issue is 504 timeout, update timeout in haproxy helps on that. > The next is the timeout from cinder-api, [1] helps on that. > Then the next is rbd client timeout. I started "debug RBD timeout > issue" thread for that. > It seems that the root cause of this series timeout is from Ceph. > I followed comments from Konstantin to use msgr2 only. > Hopefully that will fix the whole timeout issue. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1930806 > > > Thanks! > Tony > ________________________________________ > From: Eugen Block > Sent: September 1, 2021 05:02 AM > To: openstack-discuss at lists.openstack.org > Subject: Cinder API timeout on single-control node > > Hi *, > > since your last responses were quite helpful regarding rabbitmq I > would like to ask a different question for the same environment. It's > an older openstack version (Pike) running with only one control node. > There already were lots of volumes (way more than 1000) in that cloud, > but after adding a bunch more (not sure how many exactly) in one > project the whole cinder api became extremely slow. Both horizon and > CLI run into timeouts: > > [Wed Sep 01 13:18:52.109178 2021] [wsgi:error] [pid 60440] [client > :58474] Timeout when reading response headers from daemon process > 'horizon': > /srv/www/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi, > referer: http:///project/volumes/ > [Wed Sep 01 13:18:53.664714 2021] [wsgi:error] [pid 13007] Not Found: > /favicon.ico > > Sometimes the volume creation succeeds if you just retry, but it often > fails. The dashboard shows a "504 gateway timeout" after two minutes > (also after four minutes after I increased the timeout for the apache > dashboard config). > > The timeout also shows even if I try to get into the volumes tab of an > empty project. > > A couple of weeks ago I already noticed some performance issues with > cinder api if there are lots of attached volumes, if there are many > "available" volumes it doesn't seem to slow things down. But since > then the total number of volumes has doubled. At the moment there are > more than 960 attached volumes across all projects and more than 750 > detached volumes. I searched the cinder.conf for any helpful setting > but I'm not sure which would actually help. And since it's a > production cloud I would like to avoid restarting services all the > time just to try something. Maybe some of you can point me in the > right direction? I would appreciate any help! > > If there's more information I can provide just let me know. > > Thanks! > Eugen From eblock at nde.ag Fri Sep 10 08:20:58 2021 From: eblock at nde.ag (Eugen Block) Date: Fri, 10 Sep 2021 08:20:58 +0000 Subject: Horizon connection errors from object store In-Reply-To: References: <20210909121124.Horde.coxF4B2TZ6FfifJpjt6CbRs@webmail.nde.ag> <20210910074656.Horde.X90CJeWqF5dLKyczAtLrz7X@webmail.nde.ag> Message-ID: <20210910082058.Horde.ShI7_PgkarDV5P8w0VYBZLg@webmail.nde.ag> Sounds a lot like this: https://bugs.launchpad.net/ubuntu/+source/python-swiftclient/+bug/1902944 and the swift storage policies: https://docs.openstack.org/swift/latest/overview_policies.html I haven't dealt with swift policies yet so I can't really help here. Zitat von Michel Niyoyita : > Actually from the CLI on both side I did not get any errors. containers are > successfully created and added in ceph . The problem is to do it through > the dashboard . > > Michel > > On Fri, Sep 10, 2021 at 9:47 AM Eugen Block wrote: > >> Check the log files, the dashboard errors suggest that it could be a >> policy issue, the openstack-dashboard-error-logs should be more >> verbose. Then check keystone and rgw logs. What errors do you get when >> you try to create a container from CLI (add '--debug' flag to see >> more)? Did you source the correct openrc file? >> >> >> Zitat von Michel Niyoyita : >> >> > Hello Eugen >> > >> > Thank you for your continuous support. now The dashboard is stable is not >> > dsconnected as before , unfotunately I am not able to create containers >> and >> > see the list of created one using openstack CLI or ceph side. >> > >> > below is my ceph.conf : >> > >> > [client.rgw.ceph-osd3] >> > rgw frontends = "beast port=8080" >> > rgw dns name = ceph-osd3 >> > rgw enable usage log = true >> > >> > rgw thread pool size = 512 >> > rgw keystone api version = 3 >> > rgw keystone url = http://kolla-open1:5000 >> > >> > rgw keystone admin user = rgw >> > rgw keystone admin password = c8igBKQqEon8jXaG68TkcWgNI4E77m2K3bJD7fCU >> > rgw keystone admin domain = default >> > rgw keystone admin project = service >> > rgw keystone accepted roles = admin,Member,_member_,member,swiftoperator >> > rgw keystone verify ssl = false >> > rgw s3 auth use keystone = true >> > rgw keystone revocation interval = 0 >> > >> > >> > [client.rgw.ceph-osd3.rgw0] >> > host = ceph-osd3 >> > keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring >> > log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log >> > rgw frontends = beast endpoint=ceph-osd3:8080 >> > rgw thread pool size = 512 >> > >> > openstack role assignment lis --names output: >> > >> > >> > (kolla-open1) stack at kolla-open1:~$ openstack role assignment list >> --names >> > >> > >> > >> +------------------+------------------------------------+-------+-----------------+-------- >> > ----------+--------+-----------+ >> > | Role | User | Group | Project >> > | Domain | System | Inherited | >> > >> +------------------+------------------------------------+-------+-----------------+-------- >> > ----------+--------+-----------+ >> > | swiftoperator | operator:swift at Default | | >> > service at Default | | | False | >> > | admin | rgw at Default | | >> > service at Default | | | False | >> > | member | rgw at Default | | >> > service at Default | | | False | >> > | admin | cinder at Default | | >> > service at Default | | | False | >> > | admin | neutron at Default | | >> > service at Default | | | False | >> > | admin | placement at Default | | >> > service at Default | | | False | >> > | admin | nova at Default | | >> > service at Default | | | False | >> > | admin | admin at Default | | >> > admin at Default | | | False | >> > | heat_stack_owner | admin at Default | | >> > admin at Default | | | False | >> > | admin | admin at Default | | >> > service at Default | | | False | >> > | member | admin at Default | | >> > service at Default | | | False | >> > | admin | glance at Default | | >> > service at Default | | | False | >> > | member | operator at Default | | >> > service at Default | | | False | >> > | _member_ | operator at Default | | >> > service at Default | | | False | >> > | admin | heat at Default | | >> > service at Default | | | False | >> > | admin | heat_domain_admin at heat_user_domain | | >> > | heat_us er_domain | | False | >> > | admin | admin at Default | | >> > | | all | False | >> > >> +------------------+------------------------------------+-------+-----------------+-------- >> > >> > Michel >> > >> > >> > On Fri, Sep 10, 2021 at 9:33 AM Michel Niyoyita >> wrote: >> > >> >> Hello Eugen >> >> >> >> Thank you for your continuous support. now The dashboard is stable is >> not >> >> dsconnected as before , unfotunately I am not able to create containers >> and >> >> see the list of created one using openstack CLI or ceph side. you will >> find >> >> the image at the end. >> >> >> >> below is my ceph.conf : >> >> >> >> [client.rgw.ceph-osd3] >> >> rgw frontends = "beast port=8080" >> >> rgw dns name = ceph-osd3 >> >> rgw enable usage log = true >> >> >> >> rgw thread pool size = 512 >> >> rgw keystone api version = 3 >> >> rgw keystone url = http://kolla-open1:5000 >> >> >> >> rgw keystone admin user = rgw >> >> rgw keystone admin password = c8igBKQqEon8jXaG68TkcWgNI4E77m2K3bJD7fCU >> >> rgw keystone admin domain = default >> >> rgw keystone admin project = service >> >> rgw keystone accepted roles = admin,Member,_member_,member,swiftoperator >> >> rgw keystone verify ssl = false >> >> rgw s3 auth use keystone = true >> >> rgw keystone revocation interval = 0 >> >> >> >> >> >> [client.rgw.ceph-osd3.rgw0] >> >> host = ceph-osd3 >> >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring >> >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log >> >> rgw frontends = beast endpoint=ceph-osd3:8080 >> >> rgw thread pool size = 512 >> >> >> >> openstack role assignment lis --names output: >> >> >> >> >> >> (kolla-open1) stack at kolla-open1:~$ openstack role assignment list >> --names >> >> >> >> >> >> >> +------------------+------------------------------------+-------+-----------------+-------- >> >> ----------+--------+-----------+ >> >> | Role | User | Group | >> Project >> >> | Domain | System | Inherited | >> >> >> +------------------+------------------------------------+-------+-----------------+-------- >> >> ----------+--------+-----------+ >> >> | swiftoperator | operator:swift at Default | | >> >> service at Default | | | False | >> >> | admin | rgw at Default | | >> >> service at Default | | | False | >> >> | member | rgw at Default | | >> >> service at Default | | | False | >> >> | admin | cinder at Default | | >> >> service at Default | | | False | >> >> | admin | neutron at Default | | >> >> service at Default | | | False | >> >> | admin | placement at Default | | >> >> service at Default | | | False | >> >> | admin | nova at Default | | >> >> service at Default | | | False | >> >> | admin | admin at Default | | >> >> admin at Default | | | False | >> >> | heat_stack_owner | admin at Default | | >> >> admin at Default | | | False | >> >> | admin | admin at Default | | >> >> service at Default | | | False | >> >> | member | admin at Default | | >> >> service at Default | | | False | >> >> | admin | glance at Default | | >> >> service at Default | | | False | >> >> | member | operator at Default | | >> >> service at Default | | | False | >> >> | _member_ | operator at Default | | >> >> service at Default | | | False | >> >> | admin | heat at Default | | >> >> service at Default | | | False | >> >> | admin | heat_domain_admin at heat_user_domain | | >> >> | heat_us er_domain | | False | >> >> | admin | admin at Default | | >> >> | | all | False | >> >> >> >> >> +------------------+------------------------------------+-------+-----------------+-------- >> >> >> >> [image: image.png] >> >> >> >> Michel >> >> >> >> >> >> On Thu, Sep 9, 2021 at 2:15 PM Eugen Block wrote: >> >> >> >>> Hi, >> >>> >> >>> I could reproduce this in my lab environment. The issue must be either >> >>> in your ceph.conf on the RGW host(s) or your openstack role >> >>> assigments. I have a dedicated user for my setup as you can see in my >> >>> previous response. The user "rgw" gets then assigned the "member" role >> >>> to the "service" project. If I login to Horizon dashboard with this >> >>> user I can see the object-storage panel and see existing containers >> >>> for that user. If I login as admin and try to see the container panel >> >>> I get logged out, too. If I replace "rgw" with "admin" in the >> >>> ceph.conf and restart the RGW it works. But note that in this case the >> >>> admin user has to have the proper role assignment, too. >> >>> >> >>> So to achieve this you need to add a matching role (from "rgw keystone >> >>> accepted roles") for your admin user in the respective project, like >> >>> this: >> >>> >> >>> # replace rgw with admin in your case, PROJECT_ID is "service" in my >> case >> >>> openstack role add --user rgw --project member >> >>> >> >>> # check with >> >>> openstack role assignment list --names >> >>> >> >>> To make it easier to follow, please share your current ceph.conf and >> >>> the openstack role assignment output. >> >>> >> >>> Regards, >> >>> Eugen >> >>> >> >>> >> >>> >> >>> Zitat von Michel Niyoyita : >> >>> >> >>> > Hello team , >> >>> > >> >>> > I am facing an issue when I am trying to connect to the object store >> >>> > containers on the horizon dashboad . Once click on containers it >> >>> > automatically disconnect. please find below logs I am getting and >> help >> >>> for >> >>> > further analysis. >> >>> > >> >>> > [Thu Sep 09 06:35:22.185771 2021] [wsgi:error] [pid 167:tid >> >>> > 139887608641280] [remote 10.10.29.150:55130] Attempted scope to >> domain >> >>> > Default failed, will attempt to scope to another domain. >> >>> > [Thu Sep 09 06:35:22.572522 2021] [wsgi:error] [pid 167:tid >> >>> > 139887608641280] [remote 10.10.29.150:55130] Login successful for >> user >> >>> > "admin" using domain "Default", remote address 10.10.29.150. >> >>> > [Thu Sep 09 06:35:51.494815 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i >> >>> > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H >> >>> > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" >> >>> > [Thu Sep 09 06:35:51.495140 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 >> >>> Unauthorized >> >>> > [Thu Sep 09 06:35:51.495541 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: >> >>> > {'Content-Length': '119', 'X-Trans-Id': >> >>> > 'tx00000000000000000000f-006139ab44-9fc1a-default', >> >>> > 'X-Openstack-Request-Id': >> >>> > 'tx00000000000000000000f-006139ab44-9fc1a-default', 'Accept-Ranges': >> >>> > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': >> >>> 'Thu, >> >>> > 09 Sep 2021 06:35:51 GMT', 'Connection': 'Keep-Alive'} >> >>> > [Thu Sep 09 06:35:51.495792 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: >> >>> > >> >>> >> b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000f-006139ab44-9fc1a-default","HostId":"9fc1a-default-default"}' >> >>> > [Thu Sep 09 06:35:51.498743 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: >> >>> > /api/swift/containers/ >> >>> > [Thu Sep 09 06:35:52.924169 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i >> >>> > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H >> >>> > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" >> >>> > [Thu Sep 09 06:35:52.924520 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 >> >>> Unauthorized >> >>> > [Thu Sep 09 06:35:52.924789 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: >> >>> > {'Content-Length': '119', 'X-Trans-Id': >> >>> > 'tx000000000000000000010-006139ab48-9fc1a-default', >> >>> > 'X-Openstack-Request-Id': >> >>> > 'tx000000000000000000010-006139ab48-9fc1a-default', 'Accept-Ranges': >> >>> > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': >> >>> 'Thu, >> >>> > 09 Sep 2021 06:35:52 GMT', 'Connection': 'Keep-Alive'} >> >>> > [Thu Sep 09 06:35:52.925034 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: >> >>> > >> >>> >> b'{"Code":"AccessDenied","RequestId":"tx000000000000000000010-006139ab48-9fc1a-default","HostId":"9fc1a-default-default"}' >> >>> > [Thu Sep 09 06:35:52.929398 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: >> >>> > /api/swift/containers/ >> >>> > [Thu Sep 09 06:35:52.935799 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:56016] Logging out user >> "admin". >> >>> > [Thu Sep 09 06:35:53.061489 2021] [wsgi:error] [pid 166:tid >> >>> > 139887608641280] [remote 10.10.29.150:55806] Logging out user "". >> >>> > [Thu Sep 09 06:35:54.541593 2021] [wsgi:error] [pid 165:tid >> >>> > 139887608641280] [remote 10.10.29.150:55852] The request's session >> was >> >>> > deleted before the request completed. The user may have logged out >> in a >> >>> > concurrent request, for example. >> >>> > [Thu Sep 09 06:35:54.542896 2021] [wsgi:error] [pid 165:tid >> >>> > 139887608641280] [remote 10.10.29.150:55852] Bad Request: >> >>> > /api/swift/policies/ >> >>> > [Thu Sep 09 06:35:54.566055 2021] [wsgi:error] [pid 167:tid >> >>> > 139887608641280] [remote 10.10.29.150:55860] The request's session >> was >> >>> > deleted before the request completed. The user may have logged out >> in a >> >>> > concurrent request, for example. >> >>> > [Thu Sep 09 06:35:54.567130 2021] [wsgi:error] [pid 167:tid >> >>> > 139887608641280] [remote 10.10.29.150:55860] Bad Request: >> >>> /api/swift/info/ >> >>> > (kolla-open1) stack at kolla-open1 >> >>> > :/var/lib/docker/volumes/kolla_logs/_data/horizon$ >> >>> > >> >>> > Michel >> >>> >> >>> >> >>> >> >>> >> >>> >> >> >> >> From suzhengwei at inspur.com Fri Sep 10 10:16:16 2021 From: suzhengwei at inspur.com (=?utf-8?B?U2FtIFN1ICjoi4/mraPkvJ8p?=) Date: Fri, 10 Sep 2021 10:16:16 +0000 Subject: =?utf-8?B?562U5aSNOiBRdWF5LmlvIGF2YWlsYWJpbGl0eSBpbiBDaGluYQ==?= In-Reply-To: References: Message-ID: <243d6a9a5fe44e81a4a1cc7537e4b17f@inspur.com> Yoctozepto: I can aceess to Quay.io from China. -----邮件原件----- 发件人: Radosław Piliszek [mailto:radoslaw.piliszek at gmail.com] 发送时间: 2021年9月10日 15:33 收件人: openstack-discuss 主题: Quay.io availability in China Dears, I have a question (mostly to our readers from China) whether https://quay.io/ is freely accessible from there. Asking because Kolla is planning to switch from Docker Hub (which has some local mirror in China as far as we know) to Quay.io for public images. -yoctozepto -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3606 bytes Desc: not available URL: From gao.hanxiang at 99cloud.net Fri Sep 10 10:30:49 2021 From: gao.hanxiang at 99cloud.net (=?utf-8?B?6auY54Ca57+U?=) Date: Fri, 10 Sep 2021 18:30:49 +0800 Subject: Quay.io availability in China In-Reply-To: References: Message-ID: <547D3953-8BC5-4E95-880D-A92E09EC9008@99cloud.net> Hi, It’s a good idea. Access to "quay.io" in China is unimpeded. > 2021年09月10日 15:32，Radosław Piliszek 写道： > > Dears, > > I have a question (mostly to our readers from China) whether > https://quay.io/ is freely accessible from there. > > Asking because Kolla is planning to switch from Docker Hub (which has > some local mirror in China as far as we know) to Quay.io for public > images. > > -yoctozepto > From radoslaw.piliszek at gmail.com Fri Sep 10 11:19:46 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 10 Sep 2021 13:19:46 +0200 Subject: Quay.io availability in China In-Reply-To: <243d6a9a5fe44e81a4a1cc7537e4b17f@inspur.com> References: <243d6a9a5fe44e81a4a1cc7537e4b17f@inspur.com> Message-ID: Thank you both for answering! Much appreciated! :-) -yoctozepto On Fri, Sep 10, 2021 at 12:17 PM Sam Su (苏正伟) wrote: > > Yoctozepto: > > I can aceess to Quay.io from China. > > -----邮件原件----- > 发件人: Radosław Piliszek [mailto:radoslaw.piliszek at gmail.com] > 发送时间: 2021年9月10日 15:33 > 收件人: openstack-discuss > 主题: Quay.io availability in China > > Dears, > > I have a question (mostly to our readers from China) whether https://quay.io/ is freely accessible from there. > > Asking because Kolla is planning to switch from Docker Hub (which has some local mirror in China as far as we know) to Quay.io for public images. > > -yoctozepto > On Fri, Sep 10, 2021 at 12:31 PM 高瀚翔 wrote: > > Hi, > > It’s a good idea. Access to "quay.io" in China is unimpeded. > > > > 2021年09月10日 15:32，Radosław Piliszek 写道： > > > > Dears, > > > > I have a question (mostly to our readers from China) whether > > https://quay.io/ is freely accessible from there. > > > > Asking because Kolla is planning to switch from Docker Hub (which has > > some local mirror in China as far as we know) to Quay.io for public > > images. > > > > -yoctozepto > > > > From Ricardo.SoaresSarto at windriver.com Fri Sep 10 14:19:09 2021 From: Ricardo.SoaresSarto at windriver.com (Soares Sarto, Ricardo) Date: Fri, 10 Sep 2021 14:19:09 +0000 Subject: [horizon][dev] Multiple Login Sessions Enhancement Proposal Message-ID: Hi, Based on a new feature request about Horizon's login session, we would like to know if you're open to considering the improvement described below. If that is the case, we can submit a spec and upstream that implementation when the feature is done. Currently, a specific user is able to create multiple login sessions in Horizon, allowing him to log in to Horizon from multiple browsers/devices at the same time. The proposal is to have a parameterized solution and have a configuration to control how Horizon's login sessions are handled, the configuration options would be as below: 1 - Default option, allow the user to create multiple login sessions - this is the current behavior; 2 - Block subsequent login attempts if the user already has an active session; 3 - Invalidate the active session (if there is one) when the user creates a new one. Option number 2 obligates the user to log out of the system before opening a new one, on the other hand, in option number 3, the Horizon application is in charge of invalidating any previous active session and keeping only the latest one. This solution targets only Horizon login sessions, not affecting other kinds of sessions (e.g.: CLI). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mthode at mthode.org Fri Sep 10 14:22:00 2021 From: mthode at mthode.org (Matthew Thode) Date: Fri, 10 Sep 2021 09:22:00 -0500 Subject: [cyborg] RFE requested for Cyborg In-Reply-To: References: Message-ID: <20210910142200.5dm7ri7yswwytzfs@mthode.org> On 21-09-10 09:12:24, Herve Beraud wrote: > +1 for me, we need validation from the requirement team. > > Le ven. 10 sept. 2021 à 07:47, Brin Zhang(张百林) a > écrit : > > > Dear Release Team, > > > > We experienced strange freezes and timeouts for a long time on our > > python-cyborgclient need to update the lost feature of device profile > > 'description' [1] > > > > And we have merged this at 7, Sep, that we can update [2]'s hash then to > > publish it. > > > > [1] https://review.opendev.org/c/openstack/python-cyborgclient/+/803886 > > [2] https://review.opendev.org/c/openstack/releases/+/807624 > > > > brinzhang > > > > > > > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ > https://twitter.com/4383hberaud +1 from me as well (requirements) -- Matthew Thode -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From gmann at ghanshyammann.com Fri Sep 10 14:21:53 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 10 Sep 2021 09:21:53 -0500 Subject: =?UTF-8?Q?Re:_=E7=AD=94=E5=A4=8D:_[lists.open?= =?UTF-8?Q?stack.org=E4=BB=A3=E5=8F=91]Re:_[al?= =?UTF-8?Q?l]Introduction_to_venus_which_is_the_p?= =?UTF-8?Q?roject_of_log_management_and_has_been_?= =?UTF-8?Q?contributed_to_the_OpenStack_community?= In-Reply-To: <12295f57f8814f0294a87087eda4d5ce@inspur.com> References: <6c371e048e99587470d9150399806c60@sslemail.net> <17bcb74964e.101c130b3259268.4181971006334121517@ghanshyammann.com> <12295f57f8814f0294a87087eda4d5ce@inspur.com> Message-ID: <17bd0161464.edf73eba304712.402300747087401343@ghanshyammann.com> ---- On Fri, 10 Sep 2021 00:35:34 -0500 Brin Zhang(张百林) wrote ---- > Thanks TC team and all contributors. > We hope that more contributors can join Venus. If you have any questions, you can contact us via ML ([venus]) or #openstack-venus channel. Hi Brin, frickler noticed that #openstack-venus channel is not registered in OFTC network, is it on freenode or another network? -gmann > > brinzhang > > > -----邮件原件----- > 发件人: Ghanshyam Mann [mailto:gmann at ghanshyammann.com] > 发送时间: 2021年9月10日 0:47 > 收件人: openstack-discuss > 主题: [lists.openstack.org代发]Re: [all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community > > ---- On Mon, 16 Aug 2021 10:06:18 -0500 Ghanshyam Mann wrote ---- > ---- On Wed, 13 Jan 2021 04:59:47 -0600 Thierry Carrez wrote ---- > > Laurent Dumont wrote: > > > > This seems really interesting. Tracing events with request-ids is > > > something that is quite useful. > > > > > > > > What is the current state? Can it be deployed by a third party? > > > > > > I see code up at https://opendev.org/inspur/ but I haven't tried > > deploying it. > > > > > > If it gathers momentum, I suspect it will be proposed as a new official > > OpenStack project, and if the Technical Committee approves it, it will > > be moved under the openstack/ namespace on opendev.org. It already > > follows our usual repository structure (venus, python-venusclient, > > venus-tempest-plugin...) > > +1, I agree that this is ready to apply as an official project and then we can start > the discussion in TC about checking the requirement. > > > > Please propose the patch to governance and I am adding it to the next TC meeting agenda too. > > > > - https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestions > > Thanks to the 'Venus' project team for all the discussion and welcome to the OpenStack governance projects. > > TC has merged the project application, you can start the process of moving it under openstack/ namespace: > > https://review.opendev.org/c/openstack/governance/+/804824 > > -gmann > > > > > -gmann > > > > > > > > -- > > > Thierry > > > > > > > > > > > > From hberaud at redhat.com Fri Sep 10 14:31:44 2021 From: hberaud at redhat.com (Herve Beraud) Date: Fri, 10 Sep 2021 16:31:44 +0200 Subject: [release] Release countdown for week R-3, Sep 13 - Sep 17 Message-ID: Development Focus ----------------- The Release Candidate (RC) deadline is next Thursday, September 16. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/xena branch will be cut. This branch will track the Xena release. Once stable/xena has been created, the master branch will be ready to switch to Yoga development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing Xena plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/xena branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from the master branch. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/xena branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from the master branch (which will technically then be Yoga) to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any Xena release so far. Finally, now is a good time to finalize release highlights. Release highlights help shape the messaging around the release and make sure that your work is properly represented. Upcoming Deadlines & Dates -------------------------- RC1 deadline: September 16 (R-3 week) Final RC deadline: September 30 (R-1 week) Final Xena release: October 6 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From akanevsk at redhat.com Fri Sep 10 14:36:49 2021 From: akanevsk at redhat.com (Arkady Kanevsky) Date: Fri, 10 Sep 2021 09:36:49 -0500 Subject: [Interop] Proposal for new weekly meeting Message-ID: Team, Friday meetings are a bit late for EMEA folks and too close to weekend plans. Propose that we move the meeting to Mondays (3pm UTC). Feedback, please. Thanks, -- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456 -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Fri Sep 10 14:41:40 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 10 Sep 2021 16:41:40 +0200 Subject: [horizon][dev] Multiple Login Sessions Enhancement Proposal In-Reply-To: References: Message-ID: Hi Ricardo, you may want to consider checking out and contributing to the Skyline project [1]. A more modern approach at Web UI for OpenStack. It has, most likely, brighter future than Horizon. [1] https://opendev.org/skyline -yoctozepto On Fri, Sep 10, 2021 at 4:19 PM Soares Sarto, Ricardo wrote: > > Hi, > > Based on a new feature request about Horizon's login session, > we would like to know if you're open to considering the > improvement described below. If that is the case, we can submit a > spec and upstream that implementation when the feature is done. > > Currently, a specific user is able to create multiple login sessions > in Horizon, allowing him to log in to Horizon from multiple > browsers/devices at the same time. > > The proposal is to have a parameterized solution and have a > configuration to control how Horizon's login sessions are handled, > the configuration options would be as below: > > 1 - Default option, allow the user to create multiple login sessions > - this is the current behavior; > > 2 - Block subsequent login attempts if the user already has an > active session; > > 3 - Invalidate the active session (if there is one) when the user > creates a new one. > > Option number 2 obligates the user to log out of the system before > opening a new one, on the other hand, in option number 3, the > Horizon application is in charge of invalidating any previous active > session and keeping only the latest one. > > This solution targets only Horizon login sessions, not affecting > other kinds of sessions (e.g.: CLI). From os-user157 at protonmail.com Fri Sep 10 15:09:41 2021 From: os-user157 at protonmail.com (os-user157) Date: Fri, 10 Sep 2021 15:09:41 +0000 Subject: [devstack][openstack-qa] DevStack on VirtualBox with Host-only Adapter In-Reply-To: <05c67c42-4527-4031-b291-14bcaa0aa7e6@www.fastmail.com> References: <1CDGGTuntimwQAUEHVBdZvwxrY5s1aYy2ccWSYtqPUl9JbKg4Y6cMBPv1TKCCDCPmNLQ6SWfZNe76BnDmpQUGWtfzZAEdtXJdz4fsCrn3Yo=@protonmail.com> <05c67c42-4527-4031-b291-14bcaa0aa7e6@www.fastmail.com> Message-ID: Hi Clark, thanks for your response! Would you be so kind to share or point me to where I can find the CI local.conf? So that I can test with it. I noticed a very strange behavior. With the minimal local.conf described in the DevStack documentation (setting the passwords and nothing more), if the hostname has a dot, as in "devstack.localdomain", when you try to create instanes, Neutron complains with "Failed to bind port...", saying in the logs that there was no OVN chassis for the host. I fixed this by using a hostname without any dots, and it worked! Is this a known bug? I tried this with only 1 NAT adapter to isolate the different issues I was having. Today I'll try with the Host-only adapter again with my CIDR 192.168.200.0/24 and follow your tips, and I'll give some feedback. Thanks for the help! From mkopec at redhat.com Fri Sep 10 15:27:50 2021 From: mkopec at redhat.com (Martin Kopec) Date: Fri, 10 Sep 2021 17:27:50 +0200 Subject: [Interop] Proposal for new weekly meeting In-Reply-To: References: Message-ID: Hi, let's use the following poll, it might be easier: https://doodle.com/poll/2477h9mm9er4ecnn Focus on the days (not exact dates) and time (UTC) please. Thanks, On Fri, 10 Sept 2021 at 16:37, Arkady Kanevsky wrote: > Team, > Friday meetings are a bit late for EMEA folks and too close to weekend > plans. > Propose that we move the meeting to Mondays (3pm UTC). > Feedback, please. > > Thanks, > -- > Arkady Kanevsky, Ph.D. > Phone: 972 707-6456 > Corporate Phone: 919 729-5744 ext. 8176456 > -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA -------------- next part -------------- An HTML attachment was scrubbed... URL: From akanevsk at redhat.com Fri Sep 10 15:34:26 2021 From: akanevsk at redhat.com (Arkady Kanevsky) Date: Fri, 10 Sep 2021 10:34:26 -0500 Subject: [Interop] Proposal for new weekly meeting In-Reply-To: References: Message-ID: good idea. Thanks Martin On Fri, Sep 10, 2021 at 10:28 AM Martin Kopec wrote: > Hi, > > let's use the following poll, it might be easier: > https://doodle.com/poll/2477h9mm9er4ecnn > Focus on the days (not exact dates) and time (UTC) please. > > Thanks, > > On Fri, 10 Sept 2021 at 16:37, Arkady Kanevsky > wrote: > >> Team, >> Friday meetings are a bit late for EMEA folks and too close to weekend >> plans. >> Propose that we move the meeting to Mondays (3pm UTC). >> Feedback, please. >> >> Thanks, >> -- >> Arkady Kanevsky, Ph.D. >> Phone: 972 707-6456 >> Corporate Phone: 919 729-5744 ext. 8176456 >> > > > -- > Martin Kopec > Senior Software Quality Engineer > Red Hat EMEA > > > > -- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Fri Sep 10 15:43:58 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 10 Sep 2021 15:43:58 +0000 Subject: [devstack][openstack-qa] DevStack on VirtualBox with Host-only Adapter In-Reply-To: References: <1CDGGTuntimwQAUEHVBdZvwxrY5s1aYy2ccWSYtqPUl9JbKg4Y6cMBPv1TKCCDCPmNLQ6SWfZNe76BnDmpQUGWtfzZAEdtXJdz4fsCrn3Yo=@protonmail.com> <05c67c42-4527-4031-b291-14bcaa0aa7e6@www.fastmail.com> Message-ID: <20210910154357.6qkazcnsonf3kns4@yuggoth.org> On 2021-09-10 15:09:41 +0000 (+0000), os-user157 wrote: [...] > Would you be so kind to share or point me to where I can find the > CI local.conf? So that I can test with it. [...] DevStack is used in CI with countless variations of local.conf files depending on what the job is intended to test, there's not just one. It's also generated from merged variable lists embedded in hierarchically inherited job definitions, so there's not a great persistent place to point to where the file itself exists in a raw state as input. However, when we run DevStack-based CI jobs we record a copy of the local.conf it was fed, so referring to one of those might be "good enough" I guess? Here's a link to one from a recent build of the generic "devstack" job (note that these files expire in roughly a month, but you can look through the job build history for new samples): https://zuul.opendev.org/t/openstack/build/0bf56f711ef24dc39d0675abda940720/log/controller/logs/local_conf.txt -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From johnsomor at gmail.com Fri Sep 10 15:46:29 2021 From: johnsomor at gmail.com (Michael Johnson) Date: Fri, 10 Sep 2021 08:46:29 -0700 Subject: [designate][interop] test_create_all_recordset_types ddt issue In-Reply-To: References: <3229108.usfYGdeWWP@whitebase.usersys.redhat.com> Message-ID: Personally I think it is cleaner to just unroll these from DDT into individual stub tests. There are not a lot of them and they are important for the interop testing, so should be cleanly tracked. Frankly we have spent more time talking about the problems than the change would take to write. grin If the interop team agrees, I am happy to do the work on the tests. Michael On Thu, Sep 9, 2021 at 10:07 AM Goutham Pacha Ravi wrote: > > > > On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec wrote: >> >> From interop perspective it's also better not to have multiple tests with the same id. >> We encountered one more problem with ddt - the test names seem not to be generated consistently, see this: >> https://paste.opendev.org/show/809187/ >> The test can have either _00009_TXT suffix or _9_TXT one. >> >> Until we figure this out, I think we will need to flag the test in interop - so that a skip of the test (because of the name mismatch in this case) won't make the whole guideline fail. >> >> Luigi's idea is great. Every test should be identified by a unique id and it shouldn't matter that the test is generated (ddt). Different input data -> different test -> different name -> different id. >> Let's try to explore whether having a unique id per ddt entry is possible. > > > > I think we could annotate the test inputs and include a unique UUID included in the test name/doc per test input. Something along these lines: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114 > However, adding the UUID as a test attr (a functionality of "testtools" [1]) will require a hook within ddt likely. > > However, as we've discovered, there's no way to avoid adding an ordinal into the generated test name when using DDT, and so we'll have to either set a PYTHONHASHSEED and disable hash randomization, or ensure that values appear in the same order all the time (example [2]) > > [1] https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc9d68320330aa/testtools/testcase.py#L892 > [2] https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859 > > > >> >> >> >> On Wed, 8 Sept 2021 at 18:15, Luigi Toscano wrote: >>> >>> On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote: >>> > If the use of "ddt" is a big problem for the compliance testing, we >>> > can consider breaking these out into individual tests (likely to be >>> > duplicates to the existing "ddt" tests to maintain the legacy test >>> > UUIDs). >>> >>> I was wondering: wouldn't it be possible to expand or use ddt somehow to >>> inject different UUIDs for each generated test? You know in advance how many >>> tests are going to be generated. >>> >>> -- >>> Luigi >>> >>> >> >> >> -- >> Martin From akanevsk at redhat.com Fri Sep 10 15:53:34 2021 From: akanevsk at redhat.com (Arkady Kanevsky) Date: Fri, 10 Sep 2021 10:53:34 -0500 Subject: [designate][interop] test_create_all_recordset_types ddt issue In-Reply-To: References: <3229108.usfYGdeWWP@whitebase.usersys.redhat.com> Message-ID: I am good with that. But Martin should respond since it will impact Refstack. On Fri, Sep 10, 2021 at 10:46 AM Michael Johnson wrote: > Personally I think it is cleaner to just unroll these from DDT into > individual stub tests. There are not a lot of them and they are > important for the interop testing, so should be cleanly tracked. > Frankly we have spent more time talking about the problems than the > change would take to write. grin > > If the interop team agrees, I am happy to do the work on the tests. > > Michael > > On Thu, Sep 9, 2021 at 10:07 AM Goutham Pacha Ravi > wrote: > > > > > > > > On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec wrote: > >> > >> From interop perspective it's also better not to have multiple tests > with the same id. > >> We encountered one more problem with ddt - the test names seem not to > be generated consistently, see this: > >> https://paste.opendev.org/show/809187/ > >> The test can have either _00009_TXT suffix or _9_TXT one. > >> > >> Until we figure this out, I think we will need to flag the test in > interop - so that a skip of the test (because of the name mismatch in this > case) won't make the whole guideline fail. > >> > >> Luigi's idea is great. Every test should be identified by a unique id > and it shouldn't matter that the test is generated (ddt). Different input > data -> different test -> different name -> different id. > >> Let's try to explore whether having a unique id per ddt entry is > possible. > > > > > > > > I think we could annotate the test inputs and include a unique UUID > included in the test name/doc per test input. Something along these lines: > https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114 > > However, adding the UUID as a test attr (a functionality of "testtools" > [1]) will require a hook within ddt likely. > > > > However, as we've discovered, there's no way to avoid adding an ordinal > into the generated test name when using DDT, and so we'll have to either > set a PYTHONHASHSEED and disable hash randomization, or ensure that values > appear in the same order all the time (example [2]) > > > > [1] > https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc9d68320330aa/testtools/testcase.py#L892 > > [2] > https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859 > > > > > > > >> > >> > >> > >> On Wed, 8 Sept 2021 at 18:15, Luigi Toscano > wrote: > >>> > >>> On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote: > >>> > If the use of "ddt" is a big problem for the compliance testing, we > >>> > can consider breaking these out into individual tests (likely to be > >>> > duplicates to the existing "ddt" tests to maintain the legacy test > >>> > UUIDs). > >>> > >>> I was wondering: wouldn't it be possible to expand or use ddt somehow > to > >>> inject different UUIDs for each generated test? You know in advance > how many > >>> tests are going to be generated. > >>> > >>> -- > >>> Luigi > >>> > >>> > >> > >> > >> -- > >> Martin > > -- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456 -------------- next part -------------- An HTML attachment was scrubbed... URL: From haiwu.us at gmail.com Fri Sep 10 15:53:45 2021 From: haiwu.us at gmail.com (hai wu) Date: Fri, 10 Sep 2021 10:53:45 -0500 Subject: Openstack nova live migration pain Message-ID: This is Openstack train release. Nova live migration is extremely painful to deal with. Everytime we complete a live migration, nova-compute service on the source host would still be up and runnning, but it would always fail to further increment the report_count database field in nova database, host heartbeat purpose, thus controller would think the hypervisor host failed via its is_up() function checking on report_count and updated_at fields iirc. So we end up having to manually migrate one vm at a time, then restart service, then manually migrate the next vm .. Any ideas? I already tried setting debug=True for nova.conf even for database tracing, but thus far I could not find any obvious error message. Each time the live migration (no shared storage) would succeed, but each time we will have to restart nova-compute service. This is so bad .. Any suggestions on this would be highly appreciated. Thanks, Hai From balazs.gibizer at est.tech Fri Sep 10 16:00:27 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Fri, 10 Sep 2021 18:00:27 +0200 Subject: [neutron] upgrading port binding:profile.allocation value for port-resource-request-groups API extension Message-ID: Hi Neutrinos! We found a technical challenge during implementing the port-resource-request-groups API extension[1]. That extension changes the format of the port.resoruce_request as well as the format of the port.binding:profile.allocation. The former is a calculated field on the port so that is easy. However the bindig:profile is persisted in the database so data migration is needed. What is the canonical way to do such DB data translation in Neutron? Can we translate the data in place during alembic migration? Or should we do some kind of online data migration when the data is translated by neutron when it is read from the db? cheers, gibi [1] https://review.opendev.org/c/openstack/neutron/+/805637/5 From manchandavishal143 at gmail.com Fri Sep 10 16:02:24 2021 From: manchandavishal143 at gmail.com (vishal manchanda) Date: Fri, 10 Sep 2021 21:32:24 +0530 Subject: [horizon][dev] Multiple Login Sessions Enhancement Proposal In-Reply-To: References: Message-ID: Hi Ricardo, Any contributions in horizon are most welcome.Feel free to reach me or other horizon team members on irc (#openstack-horizon) at OFTC n/w. for any related queries. Thanks & Regards, Vishal Manchanda On Fri, Sep 10, 2021 at 7:50 PM Soares Sarto, Ricardo < Ricardo.SoaresSarto at windriver.com> wrote: > Hi, > > Based on a new feature request about Horizon's login session, > we would like to know if you're open to considering the > improvement described below. If that is the case, we can submit a > spec and upstream that implementation when the feature is done. > > Currently, a specific user is able to create multiple login sessions > in Horizon, allowing him to log in to Horizon from multiple > browsers/devices at the same time. > > The proposal is to have a parameterized solution and have a > configuration to control how Horizon's login sessions are handled, > the configuration options would be as below: > > 1 - Default option, allow the user to create multiple login sessions > - this is the current behavior; > > 2 - Block subsequent login attempts if the user already has an > active session; > > 3 - Invalidate the active session (if there is one) when the user > creates a new one. > > Option number 2 obligates the user to log out of the system before > opening a new one, on the other hand, in option number 3, the > Horizon application is in charge of invalidating any previous active > session and keeping only the latest one. > > This solution targets only Horizon login sessions, not affecting > other kinds of sessions (e.g.: CLI). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkopec at redhat.com Fri Sep 10 16:05:07 2021 From: mkopec at redhat.com (Martin Kopec) Date: Fri, 10 Sep 2021 18:05:07 +0200 Subject: [neutron][interop] Yoga PTG session request Message-ID: Hi neutron team, we (interop) would like to schedule a 15-30 minute session on your PTG agenda to go through the latest changes in interop. We also detected a few tests from neutron-tempest-plugin which aren't tracked in interop yet and we need your help to find out whether they are good candidates or not. I put a topic proposal to your ptg agenda [1]. The best time seems to be on Wednesday or Thursday atm. [1] https://etherpad.opendev.org/p/neutron-yoga-ptg Thank you, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkopec at redhat.com Fri Sep 10 16:05:09 2021 From: mkopec at redhat.com (Martin Kopec) Date: Fri, 10 Sep 2021 18:05:09 +0200 Subject: [designate][interop][qa] Yoga PTG - ddt topic Message-ID: Hi designate team, we (interop) would like to schedule a session on your PTG agenda to go through the latest changes in interop and more importantly through the ddt issue [1][2]. I put a topic proposal to your ptg agenda [3]. [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024758.html [2] https://bugs.launchpad.net/designate/+bug/1943115 [3] https://etherpad.opendev.org/p/yoga-ptg-designate Thank you, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA -------------- next part -------------- An HTML attachment was scrubbed... URL: From haiwu.us at gmail.com Fri Sep 10 16:06:27 2021 From: haiwu.us at gmail.com (hai wu) Date: Fri, 10 Sep 2021 11:06:27 -0500 Subject: Openstack nova live migration pain In-Reply-To: References: Message-ID: I also tried to switch from using "Database ServiceGroup driver" to using "Memcache ServiceGroup driver" per doc at https://docs.openstack.org/nova/rocky/admin/service-groups.html, by modifying the following entry in nova.conf for all hosts, including controller host: servicegroup_driver = "mc" memcached_servers = service_down_time = 60 But it failed, the controller host could not see any host's nova-compute service is actually up.. So I am stuck with having to use database servicegroup driver .. On Fri, Sep 10, 2021 at 10:53 AM hai wu wrote: > > This is Openstack train release. Nova live migration is extremely > painful to deal with. Everytime we complete a live migration, > nova-compute service on the source host would still be up and > runnning, but it would always fail to further increment the > report_count database field in nova database, host heartbeat purpose, > thus controller would think the hypervisor host failed via its is_up() > function checking on report_count and updated_at fields iirc. So we > end up having to manually migrate one vm at a time, then restart > service, then manually migrate the next vm .. > > Any ideas? I already tried setting debug=True for nova.conf even for > database tracing, but thus far I could not find any obvious error > message. Each time the live migration (no shared storage) would > succeed, but each time we will have to restart nova-compute service. > This is so bad .. > > Any suggestions on this would be highly appreciated. > > Thanks, > Hai From micou12 at gmail.com Fri Sep 10 07:33:25 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Fri, 10 Sep 2021 09:33:25 +0200 Subject: Horizon connection errors from object store In-Reply-To: <20210909121124.Horde.coxF4B2TZ6FfifJpjt6CbRs@webmail.nde.ag> References: <20210909121124.Horde.coxF4B2TZ6FfifJpjt6CbRs@webmail.nde.ag> Message-ID: Hello Eugen Thank you for your continuous support. now The dashboard is stable is not dsconnected as before , unfotunately I am not able to create containers and see the list of created one using openstack CLI or ceph side. you will find the image at the end. below is my ceph.conf : [client.rgw.ceph-osd3] rgw frontends = "beast port=8080" rgw dns name = ceph-osd3 rgw enable usage log = true rgw thread pool size = 512 rgw keystone api version = 3 rgw keystone url = http://kolla-open1:5000 rgw keystone admin user = rgw rgw keystone admin password = c8igBKQqEon8jXaG68TkcWgNI4E77m2K3bJD7fCU rgw keystone admin domain = default rgw keystone admin project = service rgw keystone accepted roles = admin,Member,_member_,member,swiftoperator rgw keystone verify ssl = false rgw s3 auth use keystone = true rgw keystone revocation interval = 0 [client.rgw.ceph-osd3.rgw0] host = ceph-osd3 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log rgw frontends = beast endpoint=ceph-osd3:8080 rgw thread pool size = 512 openstack role assignment lis --names output: (kolla-open1) stack at kolla-open1:~$ openstack role assignment list --names +------------------+------------------------------------+-------+-----------------+-------- ----------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +------------------+------------------------------------+-------+-----------------+-------- ----------+--------+-----------+ | swiftoperator | operator:swift at Default | | service at Default | | | False | | admin | rgw at Default | | service at Default | | | False | | member | rgw at Default | | service at Default | | | False | | admin | cinder at Default | | service at Default | | | False | | admin | neutron at Default | | service at Default | | | False | | admin | placement at Default | | service at Default | | | False | | admin | nova at Default | | service at Default | | | False | | admin | admin at Default | | admin at Default | | | False | | heat_stack_owner | admin at Default | | admin at Default | | | False | | admin | admin at Default | | service at Default | | | False | | member | admin at Default | | service at Default | | | False | | admin | glance at Default | | service at Default | | | False | | member | operator at Default | | service at Default | | | False | | _member_ | operator at Default | | service at Default | | | False | | admin | heat at Default | | service at Default | | | False | | admin | heat_domain_admin at heat_user_domain | | | heat_us er_domain | | False | | admin | admin at Default | | | | all | False | +------------------+------------------------------------+-------+-----------------+-------- [image: image.png] Michel On Thu, Sep 9, 2021 at 2:15 PM Eugen Block wrote: > Hi, > > I could reproduce this in my lab environment. The issue must be either > in your ceph.conf on the RGW host(s) or your openstack role > assigments. I have a dedicated user for my setup as you can see in my > previous response. The user "rgw" gets then assigned the "member" role > to the "service" project. If I login to Horizon dashboard with this > user I can see the object-storage panel and see existing containers > for that user. If I login as admin and try to see the container panel > I get logged out, too. If I replace "rgw" with "admin" in the > ceph.conf and restart the RGW it works. But note that in this case the > admin user has to have the proper role assignment, too. > > So to achieve this you need to add a matching role (from "rgw keystone > accepted roles") for your admin user in the respective project, like > this: > > # replace rgw with admin in your case, PROJECT_ID is "service" in my case > openstack role add --user rgw --project member > > # check with > openstack role assignment list --names > > To make it easier to follow, please share your current ceph.conf and > the openstack role assignment output. > > Regards, > Eugen > > > > Zitat von Michel Niyoyita : > > > Hello team , > > > > I am facing an issue when I am trying to connect to the object store > > containers on the horizon dashboad . Once click on containers it > > automatically disconnect. please find below logs I am getting and help > for > > further analysis. > > > > [Thu Sep 09 06:35:22.185771 2021] [wsgi:error] [pid 167:tid > > 139887608641280] [remote 10.10.29.150:55130] Attempted scope to domain > > Default failed, will attempt to scope to another domain. > > [Thu Sep 09 06:35:22.572522 2021] [wsgi:error] [pid 167:tid > > 139887608641280] [remote 10.10.29.150:55130] Login successful for user > > "admin" using domain "Default", remote address 10.10.29.150. > > [Thu Sep 09 06:35:51.494815 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i > > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H > > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" > > [Thu Sep 09 06:35:51.495140 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 > Unauthorized > > [Thu Sep 09 06:35:51.495541 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: > > {'Content-Length': '119', 'X-Trans-Id': > > 'tx00000000000000000000f-006139ab44-9fc1a-default', > > 'X-Openstack-Request-Id': > > 'tx00000000000000000000f-006139ab44-9fc1a-default', 'Accept-Ranges': > > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, > > 09 Sep 2021 06:35:51 GMT', 'Connection': 'Keep-Alive'} > > [Thu Sep 09 06:35:51.495792 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: > > > b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000f-006139ab44-9fc1a-default","HostId":"9fc1a-default-default"}' > > [Thu Sep 09 06:35:51.498743 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: > > /api/swift/containers/ > > [Thu Sep 09 06:35:52.924169 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i > > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H > > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" > > [Thu Sep 09 06:35:52.924520 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 > Unauthorized > > [Thu Sep 09 06:35:52.924789 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: > > {'Content-Length': '119', 'X-Trans-Id': > > 'tx000000000000000000010-006139ab48-9fc1a-default', > > 'X-Openstack-Request-Id': > > 'tx000000000000000000010-006139ab48-9fc1a-default', 'Accept-Ranges': > > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Thu, > > 09 Sep 2021 06:35:52 GMT', 'Connection': 'Keep-Alive'} > > [Thu Sep 09 06:35:52.925034 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: > > > b'{"Code":"AccessDenied","RequestId":"tx000000000000000000010-006139ab48-9fc1a-default","HostId":"9fc1a-default-default"}' > > [Thu Sep 09 06:35:52.929398 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: > > /api/swift/containers/ > > [Thu Sep 09 06:35:52.935799 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:56016] Logging out user "admin". > > [Thu Sep 09 06:35:53.061489 2021] [wsgi:error] [pid 166:tid > > 139887608641280] [remote 10.10.29.150:55806] Logging out user "". > > [Thu Sep 09 06:35:54.541593 2021] [wsgi:error] [pid 165:tid > > 139887608641280] [remote 10.10.29.150:55852] The request's session was > > deleted before the request completed. The user may have logged out in a > > concurrent request, for example. > > [Thu Sep 09 06:35:54.542896 2021] [wsgi:error] [pid 165:tid > > 139887608641280] [remote 10.10.29.150:55852] Bad Request: > > /api/swift/policies/ > > [Thu Sep 09 06:35:54.566055 2021] [wsgi:error] [pid 167:tid > > 139887608641280] [remote 10.10.29.150:55860] The request's session was > > deleted before the request completed. The user may have logged out in a > > concurrent request, for example. > > [Thu Sep 09 06:35:54.567130 2021] [wsgi:error] [pid 167:tid > > 139887608641280] [remote 10.10.29.150:55860] Bad Request: > /api/swift/info/ > > (kolla-open1) stack at kolla-open1 > > :/var/lib/docker/volumes/kolla_logs/_data/horizon$ > > > > Michel > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 68661 bytes Desc: not available URL: From micou12 at gmail.com Fri Sep 10 07:36:09 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Fri, 10 Sep 2021 09:36:09 +0200 Subject: Horizon connection errors from object store In-Reply-To: References: <20210909121124.Horde.coxF4B2TZ6FfifJpjt6CbRs@webmail.nde.ag> Message-ID: Hello Eugen Thank you for your continuous support. now The dashboard is stable is not dsconnected as before , unfotunately I am not able to create containers and see the list of created one using openstack CLI or ceph side. below is my ceph.conf : [client.rgw.ceph-osd3] rgw frontends = "beast port=8080" rgw dns name = ceph-osd3 rgw enable usage log = true rgw thread pool size = 512 rgw keystone api version = 3 rgw keystone url = http://kolla-open1:5000 rgw keystone admin user = rgw rgw keystone admin password = c8igBKQqEon8jXaG68TkcWgNI4E77m2K3bJD7fCU rgw keystone admin domain = default rgw keystone admin project = service rgw keystone accepted roles = admin,Member,_member_,member,swiftoperator rgw keystone verify ssl = false rgw s3 auth use keystone = true rgw keystone revocation interval = 0 [client.rgw.ceph-osd3.rgw0] host = ceph-osd3 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log rgw frontends = beast endpoint=ceph-osd3:8080 rgw thread pool size = 512 openstack role assignment lis --names output: (kolla-open1) stack at kolla-open1:~$ openstack role assignment list --names +------------------+------------------------------------+-------+-----------------+-------- ----------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +------------------+------------------------------------+-------+-----------------+-------- ----------+--------+-----------+ | swiftoperator | operator:swift at Default | | service at Default | | | False | | admin | rgw at Default | | service at Default | | | False | | member | rgw at Default | | service at Default | | | False | | admin | cinder at Default | | service at Default | | | False | | admin | neutron at Default | | service at Default | | | False | | admin | placement at Default | | service at Default | | | False | | admin | nova at Default | | service at Default | | | False | | admin | admin at Default | | admin at Default | | | False | | heat_stack_owner | admin at Default | | admin at Default | | | False | | admin | admin at Default | | service at Default | | | False | | member | admin at Default | | service at Default | | | False | | admin | glance at Default | | service at Default | | | False | | member | operator at Default | | service at Default | | | False | | _member_ | operator at Default | | service at Default | | | False | | admin | heat at Default | | service at Default | | | False | | admin | heat_domain_admin at heat_user_domain | | | heat_us er_domain | | False | | admin | admin at Default | | | | all | False | +------------------+------------------------------------+-------+-----------------+-------- Michel On Fri, Sep 10, 2021 at 9:33 AM Michel Niyoyita wrote: > Hello Eugen > > Thank you for your continuous support. now The dashboard is stable is not > dsconnected as before , unfotunately I am not able to create containers and > see the list of created one using openstack CLI or ceph side. you will find > the image at the end. > > below is my ceph.conf : > > [client.rgw.ceph-osd3] > rgw frontends = "beast port=8080" > rgw dns name = ceph-osd3 > rgw enable usage log = true > > rgw thread pool size = 512 > rgw keystone api version = 3 > rgw keystone url = http://kolla-open1:5000 > > rgw keystone admin user = rgw > rgw keystone admin password = c8igBKQqEon8jXaG68TkcWgNI4E77m2K3bJD7fCU > rgw keystone admin domain = default > rgw keystone admin project = service > rgw keystone accepted roles = admin,Member,_member_,member,swiftoperator > rgw keystone verify ssl = false > rgw s3 auth use keystone = true > rgw keystone revocation interval = 0 > > > [client.rgw.ceph-osd3.rgw0] > host = ceph-osd3 > keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring > log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log > rgw frontends = beast endpoint=ceph-osd3:8080 > rgw thread pool size = 512 > > openstack role assignment lis --names output: > > > (kolla-open1) stack at kolla-open1:~$ openstack role assignment list --names > > +------------------+------------------------------------+-------+-----------------+-------- > ----------+--------+-----------+ > | Role | User | Group | Project > | Domain | System | Inherited | > +------------------+------------------------------------+-------+-----------------+-------- > ----------+--------+-----------+ > | swiftoperator | operator:swift at Default | | > service at Default | | | False | > | admin | rgw at Default | | > service at Default | | | False | > | member | rgw at Default | | > service at Default | | | False | > | admin | cinder at Default | | > service at Default | | | False | > | admin | neutron at Default | | > service at Default | | | False | > | admin | placement at Default | | > service at Default | | | False | > | admin | nova at Default | | > service at Default | | | False | > | admin | admin at Default | | > admin at Default | | | False | > | heat_stack_owner | admin at Default | | > admin at Default | | | False | > | admin | admin at Default | | > service at Default | | | False | > | member | admin at Default | | > service at Default | | | False | > | admin | glance at Default | | > service at Default | | | False | > | member | operator at Default | | > service at Default | | | False | > | _member_ | operator at Default | | > service at Default | | | False | > | admin | heat at Default | | > service at Default | | | False | > | admin | heat_domain_admin at heat_user_domain | | > | heat_us er_domain | | False | > | admin | admin at Default | | > | | all | False | > > +------------------+------------------------------------+-------+-----------------+-------- > > [image: image.png] > > Michel > > > On Thu, Sep 9, 2021 at 2:15 PM Eugen Block wrote: > >> Hi, >> >> I could reproduce this in my lab environment. The issue must be either >> in your ceph.conf on the RGW host(s) or your openstack role >> assigments. I have a dedicated user for my setup as you can see in my >> previous response. The user "rgw" gets then assigned the "member" role >> to the "service" project. If I login to Horizon dashboard with this >> user I can see the object-storage panel and see existing containers >> for that user. If I login as admin and try to see the container panel >> I get logged out, too. If I replace "rgw" with "admin" in the >> ceph.conf and restart the RGW it works. But note that in this case the >> admin user has to have the proper role assignment, too. >> >> So to achieve this you need to add a matching role (from "rgw keystone >> accepted roles") for your admin user in the respective project, like >> this: >> >> # replace rgw with admin in your case, PROJECT_ID is "service" in my case >> openstack role add --user rgw --project member >> >> # check with >> openstack role assignment list --names >> >> To make it easier to follow, please share your current ceph.conf and >> the openstack role assignment output. >> >> Regards, >> Eugen >> >> >> >> Zitat von Michel Niyoyita : >> >> > Hello team , >> > >> > I am facing an issue when I am trying to connect to the object store >> > containers on the horizon dashboad . Once click on containers it >> > automatically disconnect. please find below logs I am getting and help >> for >> > further analysis. >> > >> > [Thu Sep 09 06:35:22.185771 2021] [wsgi:error] [pid 167:tid >> > 139887608641280] [remote 10.10.29.150:55130] Attempted scope to domain >> > Default failed, will attempt to scope to another domain. >> > [Thu Sep 09 06:35:22.572522 2021] [wsgi:error] [pid 167:tid >> > 139887608641280] [remote 10.10.29.150:55130] Login successful for user >> > "admin" using domain "Default", remote address 10.10.29.150. >> > [Thu Sep 09 06:35:51.494815 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i >> > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H >> > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" >> > [Thu Sep 09 06:35:51.495140 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 >> Unauthorized >> > [Thu Sep 09 06:35:51.495541 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: >> > {'Content-Length': '119', 'X-Trans-Id': >> > 'tx00000000000000000000f-006139ab44-9fc1a-default', >> > 'X-Openstack-Request-Id': >> > 'tx00000000000000000000f-006139ab44-9fc1a-default', 'Accept-Ranges': >> > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': >> 'Thu, >> > 09 Sep 2021 06:35:51 GMT', 'Connection': 'Keep-Alive'} >> > [Thu Sep 09 06:35:51.495792 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: >> > >> b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000f-006139ab44-9fc1a-default","HostId":"9fc1a-default-default"}' >> > [Thu Sep 09 06:35:51.498743 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: >> > /api/swift/containers/ >> > [Thu Sep 09 06:35:52.924169 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i >> > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H >> > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" >> > [Thu Sep 09 06:35:52.924520 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 >> Unauthorized >> > [Thu Sep 09 06:35:52.924789 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: >> > {'Content-Length': '119', 'X-Trans-Id': >> > 'tx000000000000000000010-006139ab48-9fc1a-default', >> > 'X-Openstack-Request-Id': >> > 'tx000000000000000000010-006139ab48-9fc1a-default', 'Accept-Ranges': >> > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': >> 'Thu, >> > 09 Sep 2021 06:35:52 GMT', 'Connection': 'Keep-Alive'} >> > [Thu Sep 09 06:35:52.925034 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: >> > >> b'{"Code":"AccessDenied","RequestId":"tx000000000000000000010-006139ab48-9fc1a-default","HostId":"9fc1a-default-default"}' >> > [Thu Sep 09 06:35:52.929398 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: >> > /api/swift/containers/ >> > [Thu Sep 09 06:35:52.935799 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:56016] Logging out user "admin". >> > [Thu Sep 09 06:35:53.061489 2021] [wsgi:error] [pid 166:tid >> > 139887608641280] [remote 10.10.29.150:55806] Logging out user "". >> > [Thu Sep 09 06:35:54.541593 2021] [wsgi:error] [pid 165:tid >> > 139887608641280] [remote 10.10.29.150:55852] The request's session was >> > deleted before the request completed. The user may have logged out in a >> > concurrent request, for example. >> > [Thu Sep 09 06:35:54.542896 2021] [wsgi:error] [pid 165:tid >> > 139887608641280] [remote 10.10.29.150:55852] Bad Request: >> > /api/swift/policies/ >> > [Thu Sep 09 06:35:54.566055 2021] [wsgi:error] [pid 167:tid >> > 139887608641280] [remote 10.10.29.150:55860] The request's session was >> > deleted before the request completed. The user may have logged out in a >> > concurrent request, for example. >> > [Thu Sep 09 06:35:54.567130 2021] [wsgi:error] [pid 167:tid >> > 139887608641280] [remote 10.10.29.150:55860] Bad Request: >> /api/swift/info/ >> > (kolla-open1) stack at kolla-open1 >> > :/var/lib/docker/volumes/kolla_logs/_data/horizon$ >> > >> > Michel >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 68661 bytes Desc: not available URL: From micou12 at gmail.com Fri Sep 10 07:55:35 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Fri, 10 Sep 2021 09:55:35 +0200 Subject: Horizon connection errors from object store In-Reply-To: <20210910074656.Horde.X90CJeWqF5dLKyczAtLrz7X@webmail.nde.ag> References: <20210909121124.Horde.coxF4B2TZ6FfifJpjt6CbRs@webmail.nde.ag> <20210910074656.Horde.X90CJeWqF5dLKyczAtLrz7X@webmail.nde.ag> Message-ID: Actually from the CLI on both side I did not get any errors. containers are successfully created and added in ceph . The problem is to do it through the dashboard . Michel On Fri, Sep 10, 2021 at 9:47 AM Eugen Block wrote: > Check the log files, the dashboard errors suggest that it could be a > policy issue, the openstack-dashboard-error-logs should be more > verbose. Then check keystone and rgw logs. What errors do you get when > you try to create a container from CLI (add '--debug' flag to see > more)? Did you source the correct openrc file? > > > Zitat von Michel Niyoyita : > > > Hello Eugen > > > > Thank you for your continuous support. now The dashboard is stable is not > > dsconnected as before , unfotunately I am not able to create containers > and > > see the list of created one using openstack CLI or ceph side. > > > > below is my ceph.conf : > > > > [client.rgw.ceph-osd3] > > rgw frontends = "beast port=8080" > > rgw dns name = ceph-osd3 > > rgw enable usage log = true > > > > rgw thread pool size = 512 > > rgw keystone api version = 3 > > rgw keystone url = http://kolla-open1:5000 > > > > rgw keystone admin user = rgw > > rgw keystone admin password = c8igBKQqEon8jXaG68TkcWgNI4E77m2K3bJD7fCU > > rgw keystone admin domain = default > > rgw keystone admin project = service > > rgw keystone accepted roles = admin,Member,_member_,member,swiftoperator > > rgw keystone verify ssl = false > > rgw s3 auth use keystone = true > > rgw keystone revocation interval = 0 > > > > > > [client.rgw.ceph-osd3.rgw0] > > host = ceph-osd3 > > keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring > > log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log > > rgw frontends = beast endpoint=ceph-osd3:8080 > > rgw thread pool size = 512 > > > > openstack role assignment lis --names output: > > > > > > (kolla-open1) stack at kolla-open1:~$ openstack role assignment list > --names > > > > > > > +------------------+------------------------------------+-------+-----------------+-------- > > ----------+--------+-----------+ > > | Role | User | Group | Project > > | Domain | System | Inherited | > > > +------------------+------------------------------------+-------+-----------------+-------- > > ----------+--------+-----------+ > > | swiftoperator | operator:swift at Default | | > > service at Default | | | False | > > | admin | rgw at Default | | > > service at Default | | | False | > > | member | rgw at Default | | > > service at Default | | | False | > > | admin | cinder at Default | | > > service at Default | | | False | > > | admin | neutron at Default | | > > service at Default | | | False | > > | admin | placement at Default | | > > service at Default | | | False | > > | admin | nova at Default | | > > service at Default | | | False | > > | admin | admin at Default | | > > admin at Default | | | False | > > | heat_stack_owner | admin at Default | | > > admin at Default | | | False | > > | admin | admin at Default | | > > service at Default | | | False | > > | member | admin at Default | | > > service at Default | | | False | > > | admin | glance at Default | | > > service at Default | | | False | > > | member | operator at Default | | > > service at Default | | | False | > > | _member_ | operator at Default | | > > service at Default | | | False | > > | admin | heat at Default | | > > service at Default | | | False | > > | admin | heat_domain_admin at heat_user_domain | | > > | heat_us er_domain | | False | > > | admin | admin at Default | | > > | | all | False | > > > +------------------+------------------------------------+-------+-----------------+-------- > > > > Michel > > > > > > On Fri, Sep 10, 2021 at 9:33 AM Michel Niyoyita > wrote: > > > >> Hello Eugen > >> > >> Thank you for your continuous support. now The dashboard is stable is > not > >> dsconnected as before , unfotunately I am not able to create containers > and > >> see the list of created one using openstack CLI or ceph side. you will > find > >> the image at the end. > >> > >> below is my ceph.conf : > >> > >> [client.rgw.ceph-osd3] > >> rgw frontends = "beast port=8080" > >> rgw dns name = ceph-osd3 > >> rgw enable usage log = true > >> > >> rgw thread pool size = 512 > >> rgw keystone api version = 3 > >> rgw keystone url = http://kolla-open1:5000 > >> > >> rgw keystone admin user = rgw > >> rgw keystone admin password = c8igBKQqEon8jXaG68TkcWgNI4E77m2K3bJD7fCU > >> rgw keystone admin domain = default > >> rgw keystone admin project = service > >> rgw keystone accepted roles = admin,Member,_member_,member,swiftoperator > >> rgw keystone verify ssl = false > >> rgw s3 auth use keystone = true > >> rgw keystone revocation interval = 0 > >> > >> > >> [client.rgw.ceph-osd3.rgw0] > >> host = ceph-osd3 > >> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring > >> log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log > >> rgw frontends = beast endpoint=ceph-osd3:8080 > >> rgw thread pool size = 512 > >> > >> openstack role assignment lis --names output: > >> > >> > >> (kolla-open1) stack at kolla-open1:~$ openstack role assignment list > --names > >> > >> > >> > +------------------+------------------------------------+-------+-----------------+-------- > >> ----------+--------+-----------+ > >> | Role | User | Group | > Project > >> | Domain | System | Inherited | > >> > +------------------+------------------------------------+-------+-----------------+-------- > >> ----------+--------+-----------+ > >> | swiftoperator | operator:swift at Default | | > >> service at Default | | | False | > >> | admin | rgw at Default | | > >> service at Default | | | False | > >> | member | rgw at Default | | > >> service at Default | | | False | > >> | admin | cinder at Default | | > >> service at Default | | | False | > >> | admin | neutron at Default | | > >> service at Default | | | False | > >> | admin | placement at Default | | > >> service at Default | | | False | > >> | admin | nova at Default | | > >> service at Default | | | False | > >> | admin | admin at Default | | > >> admin at Default | | | False | > >> | heat_stack_owner | admin at Default | | > >> admin at Default | | | False | > >> | admin | admin at Default | | > >> service at Default | | | False | > >> | member | admin at Default | | > >> service at Default | | | False | > >> | admin | glance at Default | | > >> service at Default | | | False | > >> | member | operator at Default | | > >> service at Default | | | False | > >> | _member_ | operator at Default | | > >> service at Default | | | False | > >> | admin | heat at Default | | > >> service at Default | | | False | > >> | admin | heat_domain_admin at heat_user_domain | | > >> | heat_us er_domain | | False | > >> | admin | admin at Default | | > >> | | all | False | > >> > >> > +------------------+------------------------------------+-------+-----------------+-------- > >> > >> [image: image.png] > >> > >> Michel > >> > >> > >> On Thu, Sep 9, 2021 at 2:15 PM Eugen Block wrote: > >> > >>> Hi, > >>> > >>> I could reproduce this in my lab environment. The issue must be either > >>> in your ceph.conf on the RGW host(s) or your openstack role > >>> assigments. I have a dedicated user for my setup as you can see in my > >>> previous response. The user "rgw" gets then assigned the "member" role > >>> to the "service" project. If I login to Horizon dashboard with this > >>> user I can see the object-storage panel and see existing containers > >>> for that user. If I login as admin and try to see the container panel > >>> I get logged out, too. If I replace "rgw" with "admin" in the > >>> ceph.conf and restart the RGW it works. But note that in this case the > >>> admin user has to have the proper role assignment, too. > >>> > >>> So to achieve this you need to add a matching role (from "rgw keystone > >>> accepted roles") for your admin user in the respective project, like > >>> this: > >>> > >>> # replace rgw with admin in your case, PROJECT_ID is "service" in my > case > >>> openstack role add --user rgw --project member > >>> > >>> # check with > >>> openstack role assignment list --names > >>> > >>> To make it easier to follow, please share your current ceph.conf and > >>> the openstack role assignment output. > >>> > >>> Regards, > >>> Eugen > >>> > >>> > >>> > >>> Zitat von Michel Niyoyita : > >>> > >>> > Hello team , > >>> > > >>> > I am facing an issue when I am trying to connect to the object store > >>> > containers on the horizon dashboad . Once click on containers it > >>> > automatically disconnect. please find below logs I am getting and > help > >>> for > >>> > further analysis. > >>> > > >>> > [Thu Sep 09 06:35:22.185771 2021] [wsgi:error] [pid 167:tid > >>> > 139887608641280] [remote 10.10.29.150:55130] Attempted scope to > domain > >>> > Default failed, will attempt to scope to another domain. > >>> > [Thu Sep 09 06:35:22.572522 2021] [wsgi:error] [pid 167:tid > >>> > 139887608641280] [remote 10.10.29.150:55130] Login successful for > user > >>> > "admin" using domain "Default", remote address 10.10.29.150. > >>> > [Thu Sep 09 06:35:51.494815 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i > >>> > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H > >>> > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" > >>> > [Thu Sep 09 06:35:51.495140 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 > >>> Unauthorized > >>> > [Thu Sep 09 06:35:51.495541 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: > >>> > {'Content-Length': '119', 'X-Trans-Id': > >>> > 'tx00000000000000000000f-006139ab44-9fc1a-default', > >>> > 'X-Openstack-Request-Id': > >>> > 'tx00000000000000000000f-006139ab44-9fc1a-default', 'Accept-Ranges': > >>> > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': > >>> 'Thu, > >>> > 09 Sep 2021 06:35:51 GMT', 'Connection': 'Keep-Alive'} > >>> > [Thu Sep 09 06:35:51.495792 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: > >>> > > >>> > b'{"Code":"AccessDenied","RequestId":"tx00000000000000000000f-006139ab44-9fc1a-default","HostId":"9fc1a-default-default"}' > >>> > [Thu Sep 09 06:35:51.498743 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: > >>> > /api/swift/containers/ > >>> > [Thu Sep 09 06:35:52.924169 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:55806] REQ: curl -i > >>> > http://ceph-mon2:8080/swift/v1?format=json&limit=1001 -X GET -H > >>> > "X-Auth-Token: gAAAAABhOasqHFyB..." -H "Accept-Encoding: gzip" > >>> > [Thu Sep 09 06:35:52.924520 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:55806] RESP STATUS: 401 > >>> Unauthorized > >>> > [Thu Sep 09 06:35:52.924789 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:55806] RESP HEADERS: > >>> > {'Content-Length': '119', 'X-Trans-Id': > >>> > 'tx000000000000000000010-006139ab48-9fc1a-default', > >>> > 'X-Openstack-Request-Id': > >>> > 'tx000000000000000000010-006139ab48-9fc1a-default', 'Accept-Ranges': > >>> > 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': > >>> 'Thu, > >>> > 09 Sep 2021 06:35:52 GMT', 'Connection': 'Keep-Alive'} > >>> > [Thu Sep 09 06:35:52.925034 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:55806] RESP BODY: > >>> > > >>> > b'{"Code":"AccessDenied","RequestId":"tx000000000000000000010-006139ab48-9fc1a-default","HostId":"9fc1a-default-default"}' > >>> > [Thu Sep 09 06:35:52.929398 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:55806] Unauthorized: > >>> > /api/swift/containers/ > >>> > [Thu Sep 09 06:35:52.935799 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:56016] Logging out user > "admin". > >>> > [Thu Sep 09 06:35:53.061489 2021] [wsgi:error] [pid 166:tid > >>> > 139887608641280] [remote 10.10.29.150:55806] Logging out user "". > >>> > [Thu Sep 09 06:35:54.541593 2021] [wsgi:error] [pid 165:tid > >>> > 139887608641280] [remote 10.10.29.150:55852] The request's session > was > >>> > deleted before the request completed. The user may have logged out > in a > >>> > concurrent request, for example. > >>> > [Thu Sep 09 06:35:54.542896 2021] [wsgi:error] [pid 165:tid > >>> > 139887608641280] [remote 10.10.29.150:55852] Bad Request: > >>> > /api/swift/policies/ > >>> > [Thu Sep 09 06:35:54.566055 2021] [wsgi:error] [pid 167:tid > >>> > 139887608641280] [remote 10.10.29.150:55860] The request's session > was > >>> > deleted before the request completed. The user may have logged out > in a > >>> > concurrent request, for example. > >>> > [Thu Sep 09 06:35:54.567130 2021] [wsgi:error] [pid 167:tid > >>> > 139887608641280] [remote 10.10.29.150:55860] Bad Request: > >>> /api/swift/info/ > >>> > (kolla-open1) stack at kolla-open1 > >>> > :/var/lib/docker/volumes/kolla_logs/_data/horizon$ > >>> > > >>> > Michel > >>> > >>> > >>> > >>> > >>> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From corey.bryant at canonical.com Fri Sep 10 18:11:43 2021 From: corey.bryant at canonical.com (Corey Bryant) Date: Fri, 10 Sep 2021 14:11:43 -0400 Subject: Help with eventlet 0.26.1 and dnspython >= 2 In-Reply-To: References: <56ab7d1e-c8d6-8505-dd81-fb4a20534fdf@debian.org> <5489b743e2ee0052500a961aa99c3aa613e81caa.camel@redhat.com> <4a27c961-354e-270c-5dc7-789a5770fe5c@debian.org> Message-ID: On Wed, Sep 30, 2020 at 11:53 AM Sean Mooney wrote: > we do not know if there are other failrue > neutron has a spereate issue which was tracked by > https://github.com/eventlet/eventlet/issues/619 > and nova hit the ssl issue with websockify and eventlets tracked by > https://github.com/eventlet/eventlet/issues/632 > > so the issue is really eventlets is not compatiabley with dnspython 2.0 > so before openstack can uncap dnspython eventlets need to gain support for > dnspython 2.0 > that should hopefully resolve the issues that nova, neutron and other > projects are now hitting. > > it is unlikely that this is something we can resolve in openstack alone, > not unless we are willing to monkeyptych > eventlets and other dependcies so really we need to work with eventlets > and or dnspython to resolve the incompatiablity > caused by the dnspython changes in 2.0 > It looks like there's been some progress on eventlet supporting dnspython 2.0: https://github.com/eventlet/eventlet/commit/aeb0390094a1c3f29bb4f25a8dab96587a86b3e8 Thanks, Corey -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Fri Sep 10 18:49:46 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 10 Sep 2021 20:49:46 +0200 Subject: [kolla][operators] Kolla Ansible will switch to use Quay.io by default (instead of Docker Hub) Message-ID: Dear Kolla Operators, This is a heads-up message that Kolla Ansible will switch to using Quay.io instead of Docker Hub by default. [1] This was agreed at the last meeting unless we find obstacles not to do it. But it seems it's even a better option for our users from China. [2] Therefore the patch [1] has been proposed and will likely be merged and probably backported to stable branches too. The main reasoning behind this change is that users suffer from Docker Hub's limits. [3] You may have noticed we started (a long time ago) publishing weekly, instead of daily, to Docker Hub to mitigate the issue to some extent. However, we had to switch our CI to Quay.io anyway to really be able to work on the project rather than pray for luck with Docker Hub. Quay.io has been great for us in general so we decided to make the same switch for users. Oh, and, by the way, we publish to Quay.io daily. ;-) If you use your own, local registry, then you are most likely unaffected but make sure you set both `kolla_registry` AND `kolla_namespace` as we have to override both. Finally, kind reminder that operators of multinode and production clusters are strongly advised to set up their own, local registries. The docs show a simple minimal example of that. [4] (use option 1 from the current page as possibility of option 2 will be gone with the switch to quay.io). Thank you for reading up to this point! [1] https://review.opendev.org/c/openstack/kolla-ansible/+/808486 [2] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024811.html [3] https://bugs.launchpad.net/kolla-ansible/+bug/1942134 [4] https://docs.openstack.org/kolla-ansible/wallaby/user/multinode.html#deploy-a-registry -yoctozepto From zigo at debian.org Fri Sep 10 19:34:19 2021 From: zigo at debian.org (Thomas Goirand) Date: Fri, 10 Sep 2021 21:34:19 +0200 Subject: Openstack nova live migration pain In-Reply-To: References: Message-ID: On 9/10/21 5:53 PM, hai wu wrote: > This is Openstack train release. Nova live migration is extremely > painful to deal with. Everytime we complete a live migration, > nova-compute service on the source host would still be up and > runnning, but it would always fail to further increment the > report_count database field in nova database, host heartbeat purpose, > thus controller would think the hypervisor host failed via its is_up() > function checking on report_count and updated_at fields iirc. So we > end up having to manually migrate one vm at a time, then restart > service, then manually migrate the next vm .. > > Any ideas? I already tried setting debug=True for nova.conf even for > database tracing, but thus far I could not find any obvious error > message. Each time the live migration (no shared storage) would > succeed, but each time we will have to restart nova-compute service. > This is so bad .. > > Any suggestions on this would be highly appreciated. > > Thanks, > Hai Hi, I experienced the same kind of trouble with Nova as well, but this seems to be fixed in the more recent versions. What is the exact point release that you're running? Cheers, Thomas Goirand (zigo) From haiwu.us at gmail.com Fri Sep 10 19:50:08 2021 From: haiwu.us at gmail.com (hai wu) Date: Fri, 10 Sep 2021 14:50:08 -0500 Subject: Openstack nova live migration pain In-Reply-To: References: Message-ID: >From controller, this command is saying 4.0.0: $ openstack --version openstack 4.0.0 Is this one what you are looking for? Currently using Debian native openstack deb packages for this train release. On Fri, Sep 10, 2021 at 2:39 PM Thomas Goirand wrote: > > On 9/10/21 5:53 PM, hai wu wrote: > > This is Openstack train release. Nova live migration is extremely > > painful to deal with. Everytime we complete a live migration, > > nova-compute service on the source host would still be up and > > runnning, but it would always fail to further increment the > > report_count database field in nova database, host heartbeat purpose, > > thus controller would think the hypervisor host failed via its is_up() > > function checking on report_count and updated_at fields iirc. So we > > end up having to manually migrate one vm at a time, then restart > > service, then manually migrate the next vm .. > > > > Any ideas? I already tried setting debug=True for nova.conf even for > > database tracing, but thus far I could not find any obvious error > > message. Each time the live migration (no shared storage) would > > succeed, but each time we will have to restart nova-compute service. > > This is so bad .. > > > > Any suggestions on this would be highly appreciated. > > > > Thanks, > > Hai > > Hi, > > I experienced the same kind of trouble with Nova as well, but this seems > to be fixed in the more recent versions. What is the exact point release > that you're running? > > Cheers, > > Thomas Goirand (zigo) > From fungi at yuggoth.org Fri Sep 10 20:01:37 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 10 Sep 2021 20:01:37 +0000 Subject: Openstack nova live migration pain In-Reply-To: References: Message-ID: <20210910200136.ancno6f6adbodsc6@yuggoth.org> On 2021-09-10 14:50:08 -0500 (-0500), hai wu wrote: > From controller, this command is saying 4.0.0: > > $ openstack --version > openstack 4.0.0 > > Is this one what you are looking for? Currently using Debian native > openstack deb packages for this train release. [...] That's reporting the version of OpenStackClient you have installed. More likely you should be looking at something like the output of `dpkg -l python3-nova` to see what version of Nova you have installed on the controller. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From haiwu.us at gmail.com Fri Sep 10 20:16:46 2021 From: haiwu.us at gmail.com (hai wu) Date: Fri, 10 Sep 2021 15:16:46 -0500 Subject: Openstack nova live migration pain In-Reply-To: <20210910200136.ancno6f6adbodsc6@yuggoth.org> References: <20210910200136.ancno6f6adbodsc6@yuggoth.org> Message-ID: Here is what we use for python3-nova: $ dpkg -l python3-nova Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-============-============-================================= ii python3-nova 2:18.1.0-6 all OpenStack Compute - libraries Here is what we have for all things with 'nova' string in the package name: $ dpkg -l | grep nova ii nova-common 2:18.1.0-6 all OpenStack Compute - common files ii nova-compute 2:18.1.0-6 all OpenStack Compute - compute node ii nova-compute-kvm 2:18.1.0-6 all OpenStack Compute - compute node (KVM) ii python3-nova 2:18.1.0-6 all OpenStack Compute - libraries ii python3-novaclient 2:15.1.0-1~bpo10+1 all client library for OpenStack Compute API - 3.x On Fri, Sep 10, 2021 at 3:08 PM Jeremy Stanley wrote: > > On 2021-09-10 14:50:08 -0500 (-0500), hai wu wrote: > > From controller, this command is saying 4.0.0: > > > > $ openstack --version > > openstack 4.0.0 > > > > Is this one what you are looking for? Currently using Debian native > > openstack deb packages for this train release. > [...] > > That's reporting the version of OpenStackClient you have installed. > More likely you should be looking at something like the output of > `dpkg -l python3-nova` to see what version of Nova you have > installed on the controller. > -- > Jeremy Stanley From fungi at yuggoth.org Fri Sep 10 20:29:13 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 10 Sep 2021 20:29:13 +0000 Subject: Openstack nova live migration pain In-Reply-To: References: <20210910200136.ancno6f6adbodsc6@yuggoth.org> Message-ID: <20210910202913.gs2p5hhdqxhp44wz@yuggoth.org> On 2021-09-10 15:16:46 -0500 (-0500), hai wu wrote: > Here is what we use for python3-nova: > > $ dpkg -l python3-nova [...] > ii python3-nova 2:18.1.0-6 all OpenStack Compute - libraries [...] In your first message you stated "this is Openstack train release," but that's a version of Nova from Rocky (originally released a year before Train). It looks like you're using packages which come as part of normal Debian 10 (buster), rather than using something like buster-train-backports from osbpo.debian.net. What made you think it was OpenStack Train, and could this be part of the problem? Do you maybe have a mixed deployment with some components from Train and some from Rocky? -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From haiwu.us at gmail.com Fri Sep 10 21:19:41 2021 From: haiwu.us at gmail.com (hai wu) Date: Fri, 10 Sep 2021 16:19:41 -0500 Subject: Openstack nova live migration pain In-Reply-To: <20210910202913.gs2p5hhdqxhp44wz@yuggoth.org> References: <20210910200136.ancno6f6adbodsc6@yuggoth.org> <20210910202913.gs2p5hhdqxhp44wz@yuggoth.org> Message-ID: I am new to the current environment. I just asked around, and you are correct: we had to use a mix of both buster-train-backports and buster, because some of the packages were breaking systems scripts, also there was one python-nova that caused the rabbitmq to get into an endless loop... Is it possible to just upgrade this package to some later package from Train, which does not have this bug, in order to work around this particular buggy state? we could do some tests here, we do have a test system, which is built the same way, and it is suffering from the exact same live migration issue .. On Fri, Sep 10, 2021 at 3:33 PM Jeremy Stanley wrote: > > On 2021-09-10 15:16:46 -0500 (-0500), hai wu wrote: > > Here is what we use for python3-nova: > > > > $ dpkg -l python3-nova > [...] > > ii python3-nova 2:18.1.0-6 all OpenStack Compute - libraries > [...] > > In your first message you stated "this is Openstack train release," > but that's a version of Nova from Rocky (originally released a year > before Train). It looks like you're using packages which come as > part of normal Debian 10 (buster), rather than using something like > buster-train-backports from osbpo.debian.net. > > What made you think it was OpenStack Train, and could this be part > of the problem? Do you maybe have a mixed deployment with some > components from Train and some from Rocky? > -- > Jeremy Stanley From gmann at ghanshyammann.com Fri Sep 10 21:58:00 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 10 Sep 2021 16:58:00 -0500 Subject: [all][tc] What's happening in Technical Committee: summary 10th Sept, 21: Reading: 5 min Message-ID: <17bd1b7abad.e2d557b9317957.8791261246820666764@ghanshyammann.com> Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC held this week's IRC meeting on Sept 9th Thursday. * Most of the meeting discussions are summarized in the below sections (Completed or in-progress activities section). To know more details, you can check the complete logs @ https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-09-15.00.log.html * We will have next week's meeting on Sept 16th, Thursday 15:00 UTC on IRC, feel free the topic in agenda [1] by Sept 15th. 2. What we completed this week: ========================= * Closeout Yoga PTL Election[2] * New project 'Venus' is an official project now[3] * Added the cinder-lvm charm to Openstack charms[4] * Fixed documentation for "assert:supports-standalone" tag[5] 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ TC is using the etherpad[6] for Xena cycle working item. We will be checking and updating the status biweekly in the same etherpad. Open Reviews ----------------- * Twelve open reviews for ongoing activities[7]. TC Chair/Vice-chair for Yoga ---------------------------------- * TC chair and vice-chair nomination are also up for review/vote[8][9]. TC tags analysis ------------------- * As discussed in last PTG, TC is working on an analysis of the usefulness of the Tags framework[10] or what all tags can be cleaned up. * You might have read the email from yoctozepto[11] about it, if you are operator, please respond to the email and based on the feedback we will continue the discussion on this. Leaderless projects ----------------------- * All the six leaderless projects[12] now have leaders identified and they are being proposed as PTL in governance[13]. Collecting Pain points from projects/SIGs/pop-up ----------------------------------------------------------- * We will continue discussing it in PTG[14], join us if you are interested to help/discuss this topic. Project updates ------------------- * Add openstack-loadbalancer charm and interfaces[15] * Retiring js-openstack-lib [16] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [17]. * Current status: "Secure RBAC" is selected for Yoga cycle[18]. PTG planning ---------------- * We are collecting the PTG topics in etherpad[19], feel free to add any topic you would like to discuss. * We discussed the live stream of one of the TC PTG sessions like we did last time. Once we will have more topics in etherpad then we can select the appropriate one. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[20], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[21]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [22] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [23] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024747.html [3] https://review.opendev.org/c/openstack/governance/+/804824 [4] https://review.opendev.org/c/openstack/governance/+/807203 [5] https://review.opendev.org/c/openstack/governance/+/807728 [6] https://etherpad.opendev.org/p/tc-xena-tracker [7] https://review.opendev.org/q/project:openstack/governance+status:open [8] https://review.opendev.org/c/openstack/governance/+/807163 [9] https://review.opendev.org/c/openstack/governance/+/807178 [10] https://governance.openstack.org/tc/reference/tags/index.html [11] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024804.html [12] https://etherpad.opendev.org/p/yoga-leaderless [13] https://review.opendev.org/q/topic:%2522formal-vote%2522+status:open+message:+PTL [14] https://etherpad.opendev.org/p/pain-point-elimination [15] https://review.opendev.org/c/openstack/governance/+/807837 [16] https://review.opendev.org/c/openstack/governance/+/798540 [17] https://etherpad.opendev.org/p/y-series-goals [18] https://review.opendev.org/c/openstack/governance/+/803783 [19] https://etherpad.opendev.org/p/tc-yoga-ptg [20] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [21] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [22] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [23] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann From zigo at debian.org Sat Sep 11 08:27:50 2021 From: zigo at debian.org (Thomas Goirand) Date: Sat, 11 Sep 2021 10:27:50 +0200 Subject: Openstack nova live migration pain In-Reply-To: References: Message-ID: <2101dc1f-24ea-0062-fcab-bf4af73d31ae@debian.org> On 9/10/21 9:50 PM, hai wu wrote: >>From controller, this command is saying 4.0.0: > > $ openstack --version > openstack 4.0.0 > > Is this one what you are looking for? Currently using Debian native > openstack deb packages for this train release. What you've displayed above is the version of python3-openstackclient. Please login into the controller machine and do: dpkg -l python3-nova Cheers, Thomas Goirand (zigo) From zigo at debian.org Sat Sep 11 13:49:21 2021 From: zigo at debian.org (Thomas Goirand) Date: Sat, 11 Sep 2021 15:49:21 +0200 Subject: Openstack nova live migration pain In-Reply-To: References: <20210910200136.ancno6f6adbodsc6@yuggoth.org> <20210910202913.gs2p5hhdqxhp44wz@yuggoth.org> Message-ID: <5030ccdb-846b-1041-812c-80b786bf69c5@debian.org> On 9/10/21 11:19 PM, hai wu wrote: > I am new to the current environment. I just asked around, and you are > correct: we had to use a mix of both buster-train-backports and > buster, because some of the packages were breaking systems scripts, > also there was one python-nova that caused the rabbitmq to get into an > endless loop... Hi, Could you be more precise please? What package is breaking what system scripts? I never heard of something like this... I never heard about what you wrote on "python-nova that caused the rabbitmq to get into an endless loop" either. We've been running Nova from the Train release for quite some time now, and we never experienced this. Also, we did have the issue you described with Nova from Rocky, but it went away after upgrading to Train. > Is it possible to just upgrade this package to some later package from > Train, which does not have this bug, in order to work around this > particular buggy state? we could do some tests here, we do have a test > system, which is built the same way, and it is suffering from the > exact same live migration issue .. Yes you can upgrade, but not directly from Rocky to Train, you must update your control plane to Stein first, do the db-sync (ie: upgrade your db schema), then upgrade to Train and re-do the db-sync again. In fact, we did this in production. While it took 6 hours, it worked well, without a single issue. BTW, it's nice that we get in touch, though I regret you didn't do it earlier about the other issues you described. We (the Debian OpenStack package maintainers) don't bite and are happy to help! :) Cheers, Thomas Goirand (zigo) From haiwu.us at gmail.com Sat Sep 11 18:17:26 2021 From: haiwu.us at gmail.com (hai wu) Date: Sat, 11 Sep 2021 13:17:26 -0500 Subject: Openstack nova live migration pain In-Reply-To: <5030ccdb-846b-1041-812c-80b786bf69c5@debian.org> References: <20210910200136.ancno6f6adbodsc6@yuggoth.org> <20210910202913.gs2p5hhdqxhp44wz@yuggoth.org> <5030ccdb-846b-1041-812c-80b786bf69c5@debian.org> Message-ID: Thanks a lot! I am not sure about the details on the issues hit before, it might have something to do with certain internally available Debian native python3 packages conflicting with the ones from buster-train-backports. I am not sure about the current state for our environment, maybe its db schema is already the one for Train release. Is there any way to verify (not to change anything) that, in order to see the current state (like issuing some sql query to see what's in place for its db schema, and issuing some dpkg command to see what kind of openstack related packages are currently in place .. )? I am hoping that we might already have db schema for Train release, and we just need to upgrade some python3 packages from buster-train-backports .. I understand that some db schema might come from certain packages, and if those packages are not matching the ones from buster-train-backports, then we might have to go the db-sync path later. Thanks, Hai On Sat, Sep 11, 2021 at 8:55 AM Thomas Goirand wrote: > > On 9/10/21 11:19 PM, hai wu wrote: > > I am new to the current environment. I just asked around, and you are > > correct: we had to use a mix of both buster-train-backports and > > buster, because some of the packages were breaking systems scripts, > > also there was one python-nova that caused the rabbitmq to get into an > > endless loop... > > Hi, > > Could you be more precise please? What package is breaking what system > scripts? I never heard of something like this... I never heard about > what you wrote on "python-nova that caused the rabbitmq to get into an > endless loop" either. > > We've been running Nova from the Train release for quite some time now, > and we never experienced this. > > Also, we did have the issue you described with Nova from Rocky, but it > went away after upgrading to Train. > > > Is it possible to just upgrade this package to some later package from > > Train, which does not have this bug, in order to work around this > > particular buggy state? we could do some tests here, we do have a test > > system, which is built the same way, and it is suffering from the > > exact same live migration issue .. > > Yes you can upgrade, but not directly from Rocky to Train, you must > update your control plane to Stein first, do the db-sync (ie: upgrade > your db schema), then upgrade to Train and re-do the db-sync again. > > In fact, we did this in production. While it took 6 hours, it worked > well, without a single issue. > > BTW, it's nice that we get in touch, though I regret you didn't do it > earlier about the other issues you described. We (the Debian OpenStack > package maintainers) don't bite and are happy to help! :) > > Cheers, > > Thomas Goirand (zigo) > From fungi at yuggoth.org Mon Sep 13 00:51:05 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 13 Sep 2021 00:51:05 +0000 Subject: [all][infra] Mailing lists are back in service again In-Reply-To: <20210902232210.jx2hrxpqjmjrn74z@yuggoth.org> References: <20210902232210.jx2hrxpqjmjrn74z@yuggoth.org> Message-ID: <20210913005105.hdcyul2472s5v7z4@yuggoth.org> Over the course of the past day, the OpenDev sysadmins performed a series of operating system upgrades for the server hosting the lists.airshipit.org, lists.opendev.org, lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org sites. Some unexpected issues arose which we did not encounter in our earlier test upgrades, which caused the outage to run four hours longer than anticipated. Unfortunately, the server began accepting messages while we were still wrestling with a bug which may have caused some queued inbound messages to be silently dropped. If you sent a message to one of the mailing lists on these sites on September 12 between 15:00 and 00:00 UTC and don't see it appear in the list archive, it was likely lost and you should send another copy. All mailing list sites should be back in service as of now, but if you have any questions or notice a problem we've overlooked, feel free to reply to this message, or find us in the #opendev channel on the OFTC IRC network. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From zigo at debian.org Sun Sep 12 19:12:16 2021 From: zigo at debian.org (Thomas Goirand) Date: Sun, 12 Sep 2021 21:12:16 +0200 Subject: Openstack nova live migration pain In-Reply-To: References: <20210910200136.ancno6f6adbodsc6@yuggoth.org> <20210910202913.gs2p5hhdqxhp44wz@yuggoth.org> <5030ccdb-846b-1041-812c-80b786bf69c5@debian.org> Message-ID: On 9/11/21 8:17 PM, hai wu wrote: > Is there any way to verify (not to change anything) that, in > order to see the current state (like issuing some sql query to see > what's in place for its db schema, and issuing some dpkg command to > see what kind of openstack related packages are currently in place .. > )? Some more familiar than me with the nova db schema should be able to answer this question. However... > I am hoping that we might already have db schema for Train release, > and we just need to upgrade some python3 packages from > buster-train-backports .. I understand that some db schema might come > from certain packages, and if those packages are not matching the ones > from buster-train-backports, then we might have to go the db-sync path > later. ... it doesn't really mater, the db-sync thingy is supposed to be idempotent, so you can: - upgrade to stein - run the db-sync in stein - upgrade to train - run the db-sync in train If you've already done the Train db-sync, then the above db-sync will do nothing and that's it... you'll ave performed a working upgrade anyways. Cheers, Thomas Goirand (zigo) From ykarel at redhat.com Mon Sep 13 05:54:03 2021 From: ykarel at redhat.com (Yatin Karel) Date: Mon, 13 Sep 2021 11:24:03 +0530 Subject: goodbye openstack, welcome ansible In-Reply-To: References: Message-ID: On Mon, Sep 6, 2021 at 10:08 PM Sorin Sbarnea wrote: > > Today was my last day as a part of the OpenStack team at Red Hat. After almost 5 years, I've decided to use the opportunity to join the newly created Ansible DevTools team, a team that will improve and maintain development tools such as ansible-lint, molecule and the more recent Ansible extension for VS Code. > > If you are interested in finding out more, make sure to attend the next Ansible Contributor Summit which is happening at the end of this month. > > To those who will be missing me, don't worry I will not be far away. > All the best for new role Sorin !!! > Cheers! > Sorin Sbarnea Thanks and Regards Yatin Karel From micou12 at gmail.com Mon Sep 13 07:31:42 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Mon, 13 Sep 2021 09:31:42 +0200 Subject: Cannot create a swift container, mandatory "Storage Policy" dropdown field is empty Message-ID: Hello team , I am replacing swift by Ceph Radosgateway , and I am successful by creating containers through openstack and ceph CLI side . but once trying to create through the horizon dashboard I get errors: *Error: *Unable to fetch the policy details. , Unable to get the Swift service info and Unable to get the Swift container listing. Kindly help to solve the issue I deployed Openstack wallaby using kolla-ansible running on ubuntu 20.04 and ceph pacific using ansible running on ubuntu 20.04 Michel -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at stackhpc.com Mon Sep 13 08:51:06 2021 From: mark at stackhpc.com (Mark Goddard) Date: Mon, 13 Sep 2021 09:51:06 +0100 Subject: [kolla][operators] Kolla Ansible will switch to use Quay.io by default (instead of Docker Hub) In-Reply-To: References: Message-ID: On Fri, 10 Sept 2021 at 19:53, Rados?aw Piliszek wrote: > > Dear Kolla Operators, > > This is a heads-up message that Kolla Ansible will switch to using > Quay.io instead of Docker Hub by default. [1] > This was agreed at the last meeting unless we find obstacles not to do it. > But it seems it's even a better option for our users from China. [2] > Therefore the patch [1] has been proposed and will likely be merged > and probably backported to stable branches too. > > The main reasoning behind this change is that users suffer from Docker > Hub's limits. [3] > You may have noticed we started (a long time ago) publishing weekly, > instead of daily, to Docker Hub to mitigate the issue to some extent. > However, we had to switch our CI to Quay.io anyway to really be able > to work on the project rather than pray for luck with Docker Hub. > Quay.io has been great for us in general so we decided to make the > same switch for users. > Oh, and, by the way, we publish to Quay.io daily. ;-) > > If you use your own, local registry, then you are most likely > unaffected but make sure you set both `kolla_registry` AND > `kolla_namespace` as we have to override both. These should be `docker_registry` and `docker_namespace`. Otherwise, a nicely written email :) > > Finally, kind reminder that operators of multinode and production > clusters are strongly advised to set up their own, local registries. > The docs show a simple minimal example of that. [4] (use option 1 from > the current page as possibility of option 2 will be gone with the > switch to quay.io). > > Thank you for reading up to this point! > > [1] https://review.opendev.org/c/openstack/kolla-ansible/+/808486 > [2] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024811.html > [3] https://bugs.launchpad.net/kolla-ansible/+bug/1942134 > [4] https://docs.openstack.org/kolla-ansible/wallaby/user/multinode.html#deploy-a-registry > > -yoctozepto > From radoslaw.piliszek at gmail.com Mon Sep 13 09:01:29 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Mon, 13 Sep 2021 11:01:29 +0200 Subject: [kolla][operators] Kolla Ansible will switch to use Quay.io by default (instead of Docker Hub) In-Reply-To: References: Message-ID: On Mon, Sep 13, 2021 at 10:51 AM Mark Goddard wrote: > > On Fri, 10 Sept 2021 at 19:53, Rados?aw Piliszek > wrote: > > > > Dear Kolla Operators, > > > > This is a heads-up message that Kolla Ansible will switch to using > > Quay.io instead of Docker Hub by default. [1] > > This was agreed at the last meeting unless we find obstacles not to do it. > > But it seems it's even a better option for our users from China. [2] > > Therefore the patch [1] has been proposed and will likely be merged > > and probably backported to stable branches too. > > > > The main reasoning behind this change is that users suffer from Docker > > Hub's limits. [3] > > You may have noticed we started (a long time ago) publishing weekly, > > instead of daily, to Docker Hub to mitigate the issue to some extent. > > However, we had to switch our CI to Quay.io anyway to really be able > > to work on the project rather than pray for luck with Docker Hub. > > Quay.io has been great for us in general so we decided to make the > > same switch for users. > > Oh, and, by the way, we publish to Quay.io daily. ;-) > > > > If you use your own, local registry, then you are most likely > > unaffected but make sure you set both `kolla_registry` AND > > `kolla_namespace` as we have to override both. > > These should be `docker_registry` and `docker_namespace`. > > Otherwise, a nicely written email :) Haha, thanks! No idea how I made up their new names. -yoctozepto > > > > Finally, kind reminder that operators of multinode and production > > clusters are strongly advised to set up their own, local registries. > > The docs show a simple minimal example of that. [4] (use option 1 from > > the current page as possibility of option 2 will be gone with the > > switch to quay.io). > > > > Thank you for reading up to this point! > > > > [1] https://review.opendev.org/c/openstack/kolla-ansible/+/808486 > > [2] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024811.html > > [3] https://bugs.launchpad.net/kolla-ansible/+bug/1942134 > > [4] https://docs.openstack.org/kolla-ansible/wallaby/user/multinode.html#deploy-a-registry > > > > -yoctozepto > > From thierry at openstack.org Mon Sep 13 09:29:48 2021 From: thierry at openstack.org (Thierry Carrez) Date: Mon, 13 Sep 2021 11:29:48 +0200 Subject: [largescale-sig] Next meeting: Sept 15th, 15utc Message-ID: <8baf8695-8f7c-4e79-2bd3-a55f26f67c9a@openstack.org> Hi everyone, The Large Scale SIG meeting is back this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210915T15 A number of topics have already been added to the agenda, including discussing the topic of our next OpenInfra.Live show! Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez From smooney at redhat.com Mon Sep 13 10:30:05 2021 From: smooney at redhat.com (Sean Mooney) Date: Mon, 13 Sep 2021 11:30:05 +0100 Subject: Openstack nova live migration pain In-Reply-To: References: <20210910200136.ancno6f6adbodsc6@yuggoth.org> <20210910202913.gs2p5hhdqxhp44wz@yuggoth.org> <5030ccdb-846b-1041-812c-80b786bf69c5@debian.org> Message-ID: <2f8affd4a2c4218f9d3cccc936393362f0e4a9b1.camel@redhat.com> On Sun, 2021-09-12 at 21:12 +0200, Thomas Goirand wrote: > On 9/11/21 8:17 PM, hai wu wrote: > > Is there any way to verify (not to change anything) that, in > > order to see the current state (like issuing some sql query to see > > what's in place for its db schema, and issuing some dpkg command to > > see what kind of openstack related packages are currently in place .. > > )? > > Some more familiar than me with the nova db schema should be able to > answer this question. However... > > > I am hoping that we might already have db schema for Train release, > > and we just need to upgrade some python3 packages from > > buster-train-backports .. I understand that some db schema might come > > from certain packages, and if those packages are not matching the ones > > from buster-train-backports, then we might have to go the db-sync path > > later. > > ... it doesn't really mater, the db-sync thingy is supposed to be > idempotent, so you can: > - upgrade to stein > - run the db-sync in stein > - upgrade to train > - run the db-sync in train > > If you've already done the Train db-sync, then the above db-sync will do > nothing and that's it... you'll ave performed a working upgrade anyways. yep running db sync should be safe if you have already run it. also nova technially does not support mix version with greate then 1 upstream version. e.g. running some nova rocky compents with other using train. you can if you really know what your doing make this work in some cases but its not advised and entirely untested upstream. if the rpc version are pinned correctly on all nodes and the contolers are a newer version the the compute and all contolers are the same version it technially can fucntion correctly. but each contoller must run exactly the same version and some feature like the numa live migration code will automatically disabel its self untill all compute services are on train. upgrading compute nodes before contolers is entirely unsupported and not intended to work. so i would strongly suggest you try and align all host to train and see if that resolve your issue. the compute agent locking up when live migration happend had 2 cause that i know if in the past one was a long running io operation that blocked the main thread and the other was due to not proprly proxying all libvirt object which again caused the main thread to block on a call to libvirt. both were fixed so its proably that your rocky nodes are just missing those fixes. > > Cheers, > > Thomas Goirand (zigo) > From tkajinam at redhat.com Mon Sep 13 12:02:33 2021 From: tkajinam at redhat.com (Takashi Kajinami) Date: Mon, 13 Sep 2021 21:02:33 +0900 Subject: [puppet] Propose retiring puppet-freezer In-Reply-To: References: Message-ID: Because I've heard no objections for one week, I'll move forward patches to retire the module. In case you have any concerns please put a comment in the following reviews. https://review.opendev.org/q/topic:retire-puppet-freezer On Mon, Sep 6, 2021 at 11:53 PM Takashi Kajinami wrote: > Hello, > > > Today I looked into the current code of puppet-freezer and noticed > this module is still very incomplete. > > Most of its implementations are inherited from the cookiecutter template, > and the module doesn't support fundamental resources like packages, > services and so on. > > Because we haven't seen any interest in fixing that feature gap for a long > time, > I'll propose retiring the module during this cycle. > > Please let me know if you have any concerns. If I don't hear any objections > I'll submit a series of patches to retire the module. > > Thank you, > Takashi Kajinami > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wodel.youchi at gmail.com Mon Sep 13 12:07:53 2021 From: wodel.youchi at gmail.com (wodel youchi) Date: Mon, 13 Sep 2021 13:07:53 +0100 Subject: Instances cannot ping each other and cannot ping virtual-router In-Reply-To: References: Message-ID: Hi, After some reading, I found out that I needed to specify the physical port that will be used to connect my infrastructure to the external world. In my configuration I created two ovs bridges over two bonds : br0 : is used for storage and storage management networks. br1 : is used for api, tenant and external networks. So I added this to my network-environment.yaml file : *Neu*tronBridgeMappings: 'datacentre:br1' And it did fix the majority of my connectivity problems, now the instances can ping each other, the instances can ping the internet. I can ping the external vrouter interface, but when I associate a floating IP with an instance, I cannot ping or ssh that instance from the external network. I have any to any security group rules for icmp and ssh (for test). How can I debug that? I have some other questions : What does mean technically these variables ? 1) What is the difference between these two lines? NeutronNetworkVLANRanges: 'datacentre:1:4000' NeutronNetworkVLANRanges: 'datacentre:1:1000,tenant:500:1000' 2) What is the difference between NeutronNetworkType and NeutronTunnelType Regards. Le mer. 8 sept. 2021 ? 17:13, wodel youchi a ?crit : > Hi, > > I deployed OpenStack Train using TripleO using this tutorial : > https://kdjlab.com/deploying-rdo-in-a-cohesive-manner/ and the > documentation of TripleO. > I deployed it with DVR. > > In my deployment I am using virtual machines with nested-kvm. > > The deployment went well, I am using network isolation like this : > - nic1 : provisioning > - nic2 and nic3 (bond0) storage and storage mgmt networks, each one in > it's VLAN > - nic3 and nic5 (bond1) tenant, api and *external* (10.0.2.0/24 VLAN2100) > networks, each one in it's VLAN > > In my physical host (the bare metal KVM) I created a bridge which handles > the provisioning, tenant, api and external networks. > > I created a private tenant network (172.16.100.0/24). > > openstack network create private > neutron subnet-create private 172.16.100.0/24 --name private-sub --dns-nameserver 172.16.0.252 > > > I created a public network and I attached it to the external network using > the same VLAN tag (10.0.2.0/24 VLAN 2100, pool: 10.0.2.100-10.0.2.120) : > > *openstack network create --provider-network-type vlan --provider-physical-network datacentre --provider-segment 2100 --external public* > neutron subnet-create public 10.0.2.0/24 --name public-sub --disable-dhcp --allocation-pool=start=10.0.2.100,end=10.0.2.120 --gateway=10.0.2.1 --dns-nameserver 172.16.0.252 > > > > I created a vrouter, one port in the public network and the other in the > private network. > I created two cirrus instances, each one got it's ip address from the > private network. > > I found : > cirrus-1 : 172.16.100.81 > cirrus-2 : 172.16.100.103 > vrouter : 172.16.100.1 private > : 10.0.2.101 external > neutron:dhcp : 172.16.100.2 > > The problems : > - The instances cannot ping each other. > - The instances cannot ping the vrouter. > - I cannot ping the public vrouter interface. > > But both instances can ping neutron:dhcp > > Could someone help me dig into this. > > Thanks in advance, Regards. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkopec at redhat.com Mon Sep 13 12:12:00 2021 From: mkopec at redhat.com (Martin Kopec) Date: Mon, 13 Sep 2021 14:12:00 +0200 Subject: [designate][interop] test_create_all_recordset_types ddt issue In-Reply-To: References: <3229108.usfYGdeWWP@whitebase.usersys.redhat.com> Message-ID: sounds good, unrolling the related tests from DDT is definitely one of the solutions. This DDT thing raises a few questions because uniquely identifying tests is not just an interop requirement, it should be a general goal. To highlight that I proposed a topic for the PTG in the designate's agenda [1]. [1] https://etherpad.opendev.org/p/yoga-ptg-designate On Fri, 10 Sept 2021 at 17:53, Arkady Kanevsky wrote: > I am good with that. > But Martin should respond since it will impact Refstack. > > On Fri, Sep 10, 2021 at 10:46 AM Michael Johnson > wrote: > >> Personally I think it is cleaner to just unroll these from DDT into >> individual stub tests. There are not a lot of them and they are >> important for the interop testing, so should be cleanly tracked. >> Frankly we have spent more time talking about the problems than the >> change would take to write. grin >> >> If the interop team agrees, I am happy to do the work on the tests. >> >> Michael >> >> On Thu, Sep 9, 2021 at 10:07 AM Goutham Pacha Ravi >> wrote: >> > >> > >> > >> > On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec wrote: >> >> >> >> From interop perspective it's also better not to have multiple tests >> with the same id. >> >> We encountered one more problem with ddt - the test names seem not to >> be generated consistently, see this: >> >> https://paste.opendev.org/show/809187/ >> >> The test can have either _00009_TXT suffix or _9_TXT one. >> >> >> >> Until we figure this out, I think we will need to flag the test in >> interop - so that a skip of the test (because of the name mismatch in this >> case) won't make the whole guideline fail. >> >> >> >> Luigi's idea is great. Every test should be identified by a unique id >> and it shouldn't matter that the test is generated (ddt). Different input >> data -> different test -> different name -> different id. >> >> Let's try to explore whether having a unique id per ddt entry is >> possible. >> > >> > >> > >> > I think we could annotate the test inputs and include a unique UUID >> included in the test name/doc per test input. Something along these lines: >> https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114 >> > However, adding the UUID as a test attr (a functionality of "testtools" >> [1]) will require a hook within ddt likely. >> > >> > However, as we've discovered, there's no way to avoid adding an ordinal >> into the generated test name when using DDT, and so we'll have to either >> set a PYTHONHASHSEED and disable hash randomization, or ensure that values >> appear in the same order all the time (example [2]) >> > >> > [1] >> https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc9d68320330aa/testtools/testcase.py#L892 >> > [2] >> https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859 >> > >> > >> > >> >> >> >> >> >> >> >> On Wed, 8 Sept 2021 at 18:15, Luigi Toscano >> wrote: >> >>> >> >>> On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote: >> >>> > If the use of "ddt" is a big problem for the compliance testing, we >> >>> > can consider breaking these out into individual tests (likely to be >> >>> > duplicates to the existing "ddt" tests to maintain the legacy test >> >>> > UUIDs). >> >>> >> >>> I was wondering: wouldn't it be possible to expand or use ddt somehow >> to >> >>> inject different UUIDs for each generated test? You know in advance >> how many >> >>> tests are going to be generated. >> >>> >> >>> -- >> >>> Luigi >> >>> >> >>> >> >> >> >> >> >> -- >> >> Martin >> >> > > -- > Arkady Kanevsky, Ph.D. > Phone: 972 707-6456 > Corporate Phone: 919 729-5744 ext. 8176456 > -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephine.seifert at secustack.com Mon Sep 13 13:00:10 2021 From: josephine.seifert at secustack.com (Josephine Seifert) Date: Mon, 13 Sep 2021 15:00:10 +0200 Subject: [Image Encryption] Meeting today cancelled Message-ID: Hello, unfortunately I have to cancel the meeting today. Next meeting will be Monday 20th. greetings Josephine (Luzi) From shubjero at gmail.com Mon Sep 13 14:18:28 2021 From: shubjero at gmail.com (shubjero) Date: Mon, 13 Sep 2021 10:18:28 -0400 Subject: MTU configuration for instances Message-ID: Hey Openstackers, I am trying to determine how to configure/manipulate an OpenStack instances MTU configuration upon deployment/creation. I know that in neutron.conf you can set the 'global_physnet_mtu = value' and 'path_mtu = value' in ml2_conf.ini. In our case we are setting these to 9000 as the physical interfaces of our control plane and compute nodes are all using 9000 mtu however the bonded interfaces for our various networks are 8900. With this mtu configured, we've noticed that our OpenStack neutron networks within a tenancy are auto-selecting a mtu of 8958 as from what I've read it will take the 9000 mtu and minus the approriate overhead to pass the underlying physical network properly but I think because our bonded interfaces are 8900 this has started to introduce some problems in instances with operating systems that hard code the MTU from Neutron networks. When an instance gets deployed it may (Ubuntu 20.04) or may not (Ubuntu 18.04) retrieve the network mtu from neutron metadata service and builds it in to the netplan configuration as provided below. My question is, where in Neutron could I configure a lower MTU for the instances? I am not sure if I should change my global_physnet_mtu and path_mtu to 8900 OR some other configuration option? Thanks all! ---- # curl --silent http://169.254.169.254/openstack/2018-08-27/network_data.json | jq . { "links": [ { "id": "tap4318b50a-76", "vif_id": "4318b50a-7685-4777-917f-eea947aa1cb6", "type": "ovs", "mtu": 8958, <--- Matches the MTU of the underlying OpenStack network attached to this instance "ethernet_mac_address": "fa:16:3e:a4:19:44" } ], "networks": [ { "id": "network0", "type": "ipv4_dhcp", "link": "tap4318b50a-76", "network_id": "4acb077c-0656-4731-ba42-031a22af7931" } ], "services": [] } ---- # cat /etc/netplan/50-cloud-init.yaml # This file is generated from information provided by the datasource. Changes # to it will not persist across an instance reboot. To disable cloud-init's # network configuration capabilities, write a file # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following: # network: {config: disabled} network: version: 2 ethernets: ens3: dhcp4: true match: macaddress: fa:16:3e:a4:19:44 mtu: 8958 <--- MTU gets hard-coded in the config for Ubuntu 20.04 but not Ubuntu 18.04. set-name: ens3 From zigo at debian.org Mon Sep 13 15:05:15 2021 From: zigo at debian.org (Thomas Goirand) Date: Mon, 13 Sep 2021 17:05:15 +0200 Subject: [puppet] Propose retiring puppet-freezer In-Reply-To: References: Message-ID: <35dd6e03-7110-41ae-8210-31be5e1691e0@debian.org> On 9/6/21 4:53 PM, Takashi Kajinami wrote: > Hello, > > > Today I looked into the current code of puppet-freezer and noticed > this module is still very incomplete. > > Most of its implementations are inherited from the cookiecutter template, > and the module doesn't support fundamental resources like packages, > services and so on. > > Because we haven't seen any interest in fixing that feature gap for a > long time, > I'll propose retiring the module during this cycle. > > Please let me know if you have any concerns. If I don't hear any objections > I'll submit a series of patches to retire the module. > > Thank you, > Takashi Kajinami Hi, I may have, in a distant future, some interest for it. However, retiring it is probably the best thing to do: I can revert the retirement later on if I decide to go ahead and maintain it. Cheers, Thomas Goirand (zigo) From openstack at nemebean.com Mon Sep 13 15:16:00 2021 From: openstack at nemebean.com (Ben Nemec) Date: Mon, 13 Sep 2021 10:16:00 -0500 Subject: [oslo] RFE requested for oslo-messaging In-Reply-To: References: Message-ID: <66c7dd54-a9c2-aa0e-a2bb-8ecf0b2909f4@nemebean.com> Since we cut the Xena branches for Oslo, I went ahead and merged these patches to master. We'll need to backport them anyway at this point (and I think we want to backport them as far as we can). On 9/9/21 8:38 AM, Herve Beraud wrote: > Yes simply need a RFE (requirement freeze exception) as we past the > requirement freeze date. > > Le?jeu. 9 sept. 2021 ??13:01, Rados?aw Piliszek > > a ?crit?: > > On Thu, Sep 9, 2021 at 12:44 PM Nikita Kalyanov > > wrote: > > > > Dear Release Team, > > > > We experienced strange freezes and timeouts for a long time on our > > OpenStack deployment [1] and some duplicate messages [2]. After > > applying the fixes [3], [4] both bugs are gone. > > Hmm, I have read the bug reports out of curiosity and I think we might > be hitting these issues randomly in Kolla CI (due to load or bad > network). > (I am obviously unsure whether the proposed fixes are correct or not.) > > As these are bugfixes, they should not need a freeze exception per se. > I hope Oslo messaging cores respond soon. > > -yoctozepto > > > > > Is a freeze exception possible for these? > > > > Thanks! > > nkalyanov > > > > [1] https://bugs.launchpad.net/oslo.messaging/+bug/1935864 > > > [2] https://bugs.launchpad.net/oslo.messaging/+bug/1935883 > > > [3] > https://review.opendev.org/c/openstack/oslo.messaging/+/800564 > > > [4] > https://review.opendev.org/c/openstack/oslo.messaging/+/800575 > > > > > > > -- > Herv? Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ > https://twitter.com/4383hberaud > From katonalala at gmail.com Mon Sep 13 15:37:16 2021 From: katonalala at gmail.com (Lajos Katona) Date: Mon, 13 Sep 2021 17:37:16 +0200 Subject: [neutron] Bug deputy report for week of September 6 Message-ID: Hi Neutron Team I was bug deputy in neutron last week. Summary ======= One bug needs more attention but that is for Rocky, and seems like ovs-agent ovs communication issue ( https://bugs.launchpad.net/neutron/+bug/1943112 ) And one question regarding Neutron standalone use with OVN perhaps ( https://bugs.launchpad.net/neutron/+bug/1943355 ) The employee of the week is ralonsoh :-) Needs attention ================ * openstack rocky version, neutron is throwing the following error : ovs|00002|ovsdb_idl|WARN|transaction error: {"details":"Transaction causes multiple rows in \"Manager\" table to have identi https://bugs.launchpad.net/neutron/+bug/1943112 * Can We Use Openstack Neutron as standalone component? https://bugs.launchpad.net/neutron/+bug/1943355 This one is more like a question, not an actual bug, or a bug for documentation. Bugs with assignees =================== * [OVN] "DHCP_Options" is not updated when the metadata port IPs are https://bugs.launchpad.net/neutron/+bug/1942794 Assigned to ralonsoh, patch: https://review.opendev.org/c/openstack/neutron/+/807692 * [OVN] Metadata ports owner is "network:distributed" only since Wallaby release https://bugs.launchpad.net/neutron/+bug/1942866 Assigned to ralonsoh, patch: https://review.opendev.org/c/openstack/neutron/+/807707 * [OVN] "neutron-ovn-tempest-ovs-master-fedora" CI job failing since 2021-08-24 https://bugs.launchpad.net/neutron/+bug/1942913 Assigned to ralonsoh, patches: https://review.opendev.org/c/openstack/neutron/+/808109 & https://review.opendev.org/c/openstack/tempest/+/808081 * [OVN][FT] During the OvnNorthd stop, the unix file could not exist https://bugs.launchpad.net/neutron/+bug/1943029 Assigned to ralonsoh, patch: https://review.opendev.org/c/openstack/neutron/+/807862 * Replace "Inspector.from_engine()" with "sqlalchemy.inspect()" https://bugs.launchpad.net/neutron/+bug/1943155 Assigned to ralonsoh, patch: https://review.opendev.org/c/openstack/neutron/+/808103 RFEs ==== No new RFE for this week. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Mon Sep 13 16:09:31 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Mon, 13 Sep 2021 11:09:31 -0500 Subject: [all][tc] Technical Committee next weekly meeting on Sept 16th at 1500 UTC Message-ID: <17bdfebb4fd.eb8b0d59570336.8347738122602559606@ghanshyammann.com> Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 16th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 15th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann From mrunge at matthias-runge.de Mon Sep 13 16:51:27 2021 From: mrunge at matthias-runge.de (Matthias Runge) Date: Mon, 13 Sep 2021 18:51:27 +0200 Subject: [oslo] RFE requested for oslo-messaging In-Reply-To: References: Message-ID: <80C95442-143A-45DF-8C08-9C0B171D6C74@matthias-runge.de> > Am 09.09.2021 um 13:00 schrieb Rados?aw Piliszek : > > On Thu, Sep 9, 2021 at 12:44 PM Nikita Kalyanov > wrote: >> >> Dear Release Team, >> >> We experienced strange freezes and timeouts for a long time on our >> OpenStack deployment [1] and some duplicate messages [2]. After >> applying the fixes [3], [4] both bugs are gone. > > Hmm, I have read the bug reports out of curiosity and I think we might > be hitting these issues randomly in Kolla CI (due to load or bad > network). > (I am obviously unsure whether the proposed fixes are correct or not.) > > As these are bugfixes, they should not need a freeze exception per se. > I hope Oslo messaging cores respond soon. > > -yoctozepto > I believe we have seen that issue a couple of times already, causing a lot of headaches with missed metrics. IHMO it would be a bugfix and not a new feature and thus a feature freeze would not really take place here. Matthias >> >> Is a freeze exception possible for these? >> >> Thanks! >> nkalyanov >> >> [1] https://bugs.launchpad.net/oslo.messaging/+bug/1935864 >> [2] https://bugs.launchpad.net/oslo.messaging/+bug/1935883 >> [3] https://review.opendev.org/c/openstack/oslo.messaging/+/800564 >> [4] https://review.opendev.org/c/openstack/oslo.messaging/+/800575 >> > From ashlee at openstack.org Mon Sep 13 14:48:52 2021 From: ashlee at openstack.org (Ashlee Ferguson) Date: Mon, 13 Sep 2021 09:48:52 -0500 Subject: OpenInfra Live - September 16 at 9am CT Message-ID: <25D4A028-C71A-449C-9403-6A5D0B577654@openstack.org> Hi everyone, This week?s OpenInfra Live episode is brought to you by the OpenInfra Community. Since the Paris Summit in 2014, the OpenInfra Foundation has hosted our annual Superuser Awards to recognize organizations that have used open infrastructure to meaningfully improve their business while contributing back to the community. Past winners will join us for this episode to discuss where they are now, what is next, and tips for organizations applying for this year's awards. Episode: Superusers: Where are they now? Date and time: September 16 at 9am CT (1400 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=nUwgBdx0lSw LinkedIn: https://www.linkedin.com/feed/update/urn:li:ugcPost:6841804188478009344/ Facebook: https://www.facebook.com/104139126308032/posts/4319519031436666/ WeChat: recording will be posted on OpenStack WeChat after the live stream Speakers: Jared Baker (The Ontario Institute for Cancer Research (OICR)) Belmiro Moreira (CERN) Mohammed Naser (VEXXHOST) Xiaoguang, Zhang and Zhiqiang Yu (China Mobile) Think your organization or someone you know could be a Superuser? Nominations for the 2021 Superuser Awards are open! Have an idea for a future episode? Share it now at ideas.openinfra.live . Thanks, Ashlee -------------- next part -------------- An HTML attachment was scrubbed... URL: From james.slagle at gmail.com Mon Sep 13 18:07:15 2021 From: james.slagle at gmail.com (James Slagle) Date: Mon, 13 Sep 2021 14:07:15 -0400 Subject: [TripleO] Reviving the TripleO review priorities etherpad Message-ID: I'd like to see about helping to focus our review priorities by reviving the tripleo-review-priorities etherpad: https://etherpad.opendev.org/p/tripleo-review-priorities I deleted all the old cruft, and created some new high level topics. There are likely others as well. I'll start by filling in some links to what I think the highest priority items are, but I could use help to create an accurate list of where the core team should focus. I'd like to get some feedback on the approach in the #tripleo meeting tomorrow as well. Let me know any thoughts. -- -- James Slagle -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From ildiko.vancsa at gmail.com Mon Sep 13 21:35:28 2021 From: ildiko.vancsa at gmail.com (Ildiko Vancsa) Date: Mon, 13 Sep 2021 14:35:28 -0700 Subject: [zun][cyborg][kuryr][neutron] Edge for scientific research session on September 13 Edge WG meeting In-Reply-To: <360c63346cfd44fca42eaa744c84ef8d@inspur.com> References: <5DB3E1B1-1D6A-4C0B-AB3D-FE33F0B88961@gmail.com> <360c63346cfd44fca42eaa744c84ef8d@inspur.com> Message-ID: <7B5E5A10-C30A-4F2B-AD64-FF2F8B988FAC@gmail.com> Hi, We had a great session today, thanks to all of you who tuned in. For those of you who couldn?t make it, we have a recording of the presentation and the discussion that followed: https://wiki.openstack.org/wiki/Edge_Computing_Group#CHI.40Edge_-_recording_of_the_session_with_the_Chameleon_project.2C_September_13.2C_2021 Next steps we touched on: * Chameleon integrated Cyborg and Zun ad would like to look into how to upstream that work * Uploading a spec about the integration steps * Discussing the details at the PTG * Chameleon also added a new Neutron ML2 plugin to support Wireguard; it could be a good item to discuss with the Neutron team as well to check upstream potential and make improvements to the agent Please let me know if you have questions to either of the above. Thanks and Best Regards, Ildik? > On Sep 7, 2021, at 17:15, Brin Zhang(???) wrote: > >> I will share the recording of the meeting as soon as it is available in case someone would not be able to make it and would like to check out the presentation and the discussion that follows. > Thanks for you will share of the record, at 1300UTC I have another meeting (start 1100UTC), if I can end up that meeting earlier, I will join this meeting ^^ > > > > > brinzhang > > > -----????----- > ???: Ildiko Vancsa [mailto:ildiko.vancsa at gmail.com] > ????: 2021?9?8? 6:08 > ???: Brin Zhang(???) > ??: openstack-discuss at lists.openstack.org; xin-ran.wang at intel.com > ??: Re: [lists.openstack.org??][zun][cyborg][kuryr] Edge for scientific research session on September 13 Edge WG meeting > > Hi Bailin, > > Thank you for sharing the links. > > Zun is the project in OpenStack that is supporting to run and manage containers directly from OpenStack without the need of Kubernetes. As the Chameleon project is using it to run their containerized workload and they also need hardware accelerators they integrated it with Cyborg. > > Many of their activities fall under the edge computing area and they are starting their collaboration with the OpenInfra Edge Computing group for this reason. As part of that work we are helping them to reach relevant communities and project teams where their requirements identified some gaps and I believe the Cyborg-Zun integration is one of these. They would like to upstream their work if the community finds value in it as it may or may not fall into the current scope of the respective projects. > > I hope that we can start to discuss some of this on the meeting next Monday and take next steps from there. > > I hope this helps a bit with the context and we can have a fruitful discussion next week. > > I will share the recording of the meeting as soon as it is available in case someone would not be able to make it and would like to check out the presentation and the discussion that follows. > > Thanks, > Ildik? > > >> On Sep 3, 2021, at 04:20, Brin Zhang(???) wrote: >> >> Hi Ildiko, >> >> Currently Cyborg and nova support interaction, we are not very familiar with zun yet, if possible, please provide some relevant information. You can refer to [1] for Cyborg related materials, and [2] for the interactive content of nova and cyborg (not yet merged). >> >> [1]https://docs.openstack.org/cyborg/latest/user/architecture.html >> [2]https://review.opendev.org/c/openstack/cyborg/+/794549 >> https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/nova-cyborg-interaction.html >> >> >> Bailin Zhang (brinzhang) >> >> >> -----????----- >> ???: Ildiko Vancsa [mailto:ildiko.vancsa at gmail.com] >> ????: 2021?8?31? 22:39 >> ???: openstack-discuss >> ??: [lists.openstack.org??][zun][cyborg][kuryr] Re: Edge for scientific research session on September 13 Edge WG meeting >> >> Hi, >> >> It is a friendly reminder that the OpenInfra Edge Computing Group is having a joint session with the Chameleon project. >> >> During this session you can learn about the Chameleon project and how they are using OpenStack to build an edge platform for scientific research and academic purposes. They have also found some gaps during their journey. To mention an example the integration between Cyborg and Zun is missing to be able to use acceleration devices with containers orchestrated by OpenStack and they have been working to fill in this gap. >> >> The session will start with a presentation about the project and then we will use the remaining time to talk about solutions and next steps. I would like to invite everyone who is interested in learning about how OpenStack can be turned into an edge platform and what are the missing pieces of the puzzle that still need to be enhanced or implemented. >> >> The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. >> The meeting will be held on Zoom, for the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings >> >> Please let me know if you need a calendar invite and I can send you one to get reminded about the meeting. >> >> Thanks and Best Regards, >> Ildik? >> >> >>> On Aug 25, 2021, at 17:00, Ildiko Vancsa wrote: >>> >>> Hi, >>> >>> I?m very excited to invite you to a session with the Chameleon project to discuss how edge computing is powering education, scientific research and more! As always in the area of edge there are challenges and gaps that we need to solve together. In that sense the session will start with a presentation that we will then turn into a discussion to turn towards discussing solutions. >>> >>> Please also note that the Chameleon project is using OpenStack to build edge solutions. As they identified some gaps they would like to work with the community to address the challenges that they?ve been hitting. Please join us to get their feedback and find the best way to improve OpenStack to deliver a platform that can be used on the edge! >>> >>> The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. >>> For the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings >>> >>> Please find the details of the session below. >>> >>> Thanks and Best Regards, >>> Ildik? >>> >>> >>> Title: Chameleon: a cloud to edge infrastructure for science >>> >>> Abstract: The NSF-funded Chameleon project (www.chameleoncloud.org) provides an OpenStack-based bare metal reconfigurable cloud that supports systems research, education, and emergent application projects in science. The facility has been in existence for 6+ years, and has served ~6,000 users working on a variety of projects ranging from networking, operating systems design, and security, to innovative applications in astronomy, physics, and life sciences. Recent research trends in IoT and edge computing, as evidenced by increased numbers of project applications in those areas, made it clear that Chameleon needed to expand into the edge space. Remaining within the OpenStack framework allowed us to leverage end-user features that we developed for Chameleon, such as access via federated identity and Jupyter integration. Accordingly, we developed infrastructure called CHI at Edge that adapts Zun to provision containers on edge devices and significantly expands its capabilities to work in the edge space. The talk will describe our use case and the emergent product definition, the OpenStack adaptations we made to the current preview implementation, the directions we plan to take in our further work, as well as the ecosystem of projects and NSF infrastructure deployments leveraging this work and collaborating with us. >>> >>> >> >> > From katonalala at gmail.com Tue Sep 14 05:25:18 2021 From: katonalala at gmail.com (Lajos Katona) Date: Tue, 14 Sep 2021 07:25:18 +0200 Subject: MTU configuration for instances In-Reply-To: References: Message-ID: Hi, You can set MTU on the network level, see: https://docs.openstack.org/api-ref/network/v2/index.html?expanded=create-network-detail#create-network Lajos Katona (lajoskatona) shubjero ezt ?rta (id?pont: 2021. szept. 13., H, 16:28): > Hey Openstackers, > > I am trying to determine how to configure/manipulate an OpenStack > instances MTU configuration upon deployment/creation. I know that in > neutron.conf you can set the 'global_physnet_mtu = value' and > 'path_mtu = value' in ml2_conf.ini. In our case we are setting these > to 9000 as the physical interfaces of our control plane and compute > nodes are all using 9000 mtu however the bonded interfaces for our > various networks are 8900. > > With this mtu configured, we've noticed that our OpenStack neutron > networks within a tenancy are auto-selecting a mtu of 8958 as from > what I've read it will take the 9000 mtu and minus the approriate > overhead to pass the underlying physical network properly but I think > because our bonded interfaces are 8900 this has started to introduce > some problems in instances with operating systems that hard code the > MTU from Neutron networks. > > When an instance gets deployed it may (Ubuntu 20.04) or may not > (Ubuntu 18.04) retrieve the network mtu from neutron metadata service > and builds it in to the netplan configuration as provided below. > > My question is, where in Neutron could I configure a lower MTU for the > instances? I am not sure if I should change my global_physnet_mtu and > path_mtu to 8900 OR some other configuration option? Thanks all! > > ---- > > # curl --silent > http://169.254.169.254/openstack/2018-08-27/network_data.json | jq . > { > "links": [ > { > "id": "tap4318b50a-76", > "vif_id": "4318b50a-7685-4777-917f-eea947aa1cb6", > "type": "ovs", > "mtu": 8958, <--- Matches the MTU of the underlying OpenStack > network attached to this instance > "ethernet_mac_address": "fa:16:3e:a4:19:44" > } > ], > "networks": [ > { > "id": "network0", > "type": "ipv4_dhcp", > "link": "tap4318b50a-76", > "network_id": "4acb077c-0656-4731-ba42-031a22af7931" > } > ], > "services": [] > } > > ---- > > # cat /etc/netplan/50-cloud-init.yaml > # This file is generated from information provided by the datasource. > Changes > # to it will not persist across an instance reboot. To disable > cloud-init's > # network configuration capabilities, write a file > # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following: > # network: {config: disabled} > network: > version: 2 > ethernets: > ens3: > dhcp4: true > match: > macaddress: fa:16:3e:a4:19:44 > mtu: 8958 <--- MTU gets hard-coded in the config for > Ubuntu 20.04 but not Ubuntu 18.04. > set-name: ens3 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Tue Sep 14 07:54:23 2021 From: hberaud at redhat.com (Herve Beraud) Date: Tue, 14 Sep 2021 09:54:23 +0200 Subject: [oslo] RFE requested for oslo-messaging In-Reply-To: <80C95442-143A-45DF-8C08-9C0B171D6C74@matthias-runge.de> References: <80C95442-143A-45DF-8C08-9C0B171D6C74@matthias-runge.de> Message-ID: I proposed backports for these both bug fixes [1][2]. Thanks to validate them to allow us to land them with the final Xena. When those patches will be merged I'll propose a new bugfix release for Xena. [1] https://review.opendev.org/c/openstack/oslo.messaging/+/808853 [2] https://review.opendev.org/c/openstack/oslo.messaging/+/808878 Le lun. 13 sept. 2021 ? 18:53, Matthias Runge a ?crit : > > > > Am 09.09.2021 um 13:00 schrieb Rados?aw Piliszek < > radoslaw.piliszek at gmail.com>: > > > > On Thu, Sep 9, 2021 at 12:44 PM Nikita Kalyanov > > wrote: > >> > >> Dear Release Team, > >> > >> We experienced strange freezes and timeouts for a long time on our > >> OpenStack deployment [1] and some duplicate messages [2]. After > >> applying the fixes [3], [4] both bugs are gone. > > > > Hmm, I have read the bug reports out of curiosity and I think we might > > be hitting these issues randomly in Kolla CI (due to load or bad > > network). > > (I am obviously unsure whether the proposed fixes are correct or not.) > > > > As these are bugfixes, they should not need a freeze exception per se. > > I hope Oslo messaging cores respond soon. > > > > -yoctozepto > > > > > I believe we have seen that issue a couple of times already, causing a lot > of headaches with missed metrics. > IHMO it would be a bugfix and not a new feature and thus a feature freeze > would not really take place here. > > Matthias > > > >> > >> Is a freeze exception possible for these? > >> > >> Thanks! > >> nkalyanov > >> > >> [1] https://bugs.launchpad.net/oslo.messaging/+bug/1935864 > >> [2] https://bugs.launchpad.net/oslo.messaging/+bug/1935883 > >> [3] https://review.opendev.org/c/openstack/oslo.messaging/+/800564 > >> [4] https://review.opendev.org/c/openstack/oslo.messaging/+/800575 > >> > > > > > -- Herv? Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhang.lei.fly+os-discuss at gmail.com Tue Sep 14 08:59:55 2021 From: zhang.lei.fly+os-discuss at gmail.com (Jeffrey Zhang) Date: Tue, 14 Sep 2021 16:59:55 +0800 Subject: Question about vDPA support for live migration In-Reply-To: <8e698e1a0adbaabe00db9732fc90f7eba167d22f.camel@redhat.com> References: <8e698e1a0adbaabe00db9732fc90f7eba167d22f.camel@redhat.com> Message-ID: Thanks Sean for the update. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tkajinam at redhat.com Tue Sep 14 09:00:57 2021 From: tkajinam at redhat.com (Takashi Kajinami) Date: Tue, 14 Sep 2021 18:00:57 +0900 Subject: [puppet] Propose retiring puppet-freezer In-Reply-To: <35dd6e03-7110-41ae-8210-31be5e1691e0@debian.org> References: <35dd6e03-7110-41ae-8210-31be5e1691e0@debian.org> Message-ID: On Tue, Sep 14, 2021 at 12:10 AM Thomas Goirand wrote: > On 9/6/21 4:53 PM, Takashi Kajinami wrote: > > Hello, > > > > > > Today I looked into the current code of puppet-freezer and noticed > > this module is still very incomplete. > > > > Most of its implementations are inherited from the cookiecutter template, > > and the module doesn't support fundamental resources like packages, > > services and so on. > > > > Because we haven't seen any interest in fixing that feature gap for a > > long time, > > I'll propose retiring the module during this cycle. > > > > Please let me know if you have any concerns. If I don't hear any > objections > > I'll submit a series of patches to retire the module. > > > > Thank you, > > Takashi Kajinami > > Hi, > > I may have, in a distant future, some interest for it. However, retiring > it is probably the best thing to do: I can revert the retirement later > on if I decide to go ahead and maintain it. > > Cheers, > > Thomas Goirand (zigo) > > Thanks for the feedback. We can revert the retirement when you (or anybody else) start working on it. The sad fact is that this module has never been completed for the past 4 years, and it'd be reasonable to retire the module now instead of keeping the current incomplete implementation any longer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhang.lei.fly+os-discuss at gmail.com Tue Sep 14 09:02:02 2021 From: zhang.lei.fly+os-discuss at gmail.com (Jeffrey Zhang) Date: Tue, 14 Sep 2021 17:02:02 +0800 Subject: Question about vDPA support for live migration In-Reply-To: References: <8e698e1a0adbaabe00db9732fc90f7eba167d22f.camel@redhat.com> Message-ID: Do you have any bug/bp/explanation on the qemu side, which i can follow or tracking. On Tue, Sep 14, 2021 at 4:59 PM Jeffrey Zhang < zhang.lei.fly+os-discuss at gmail.com> wrote: > Thanks Sean for the update. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yasufum.o at gmail.com Tue Sep 14 11:38:54 2021 From: yasufum.o at gmail.com (Yasufumi Ogawa) Date: Tue, 14 Sep 2021 20:38:54 +0900 Subject: [tacker] Ayumu Ueha for Tacker Core In-Reply-To: References: Message-ID: <2194a9ed-57a1-be99-ab6a-123e582c9df7@gmail.com> Hi, Since many of +1s and no objections, I'd add Ayumu to tacker-core. Welcome Ayumu! On 2021/09/06 14:03, Yasufumi Ogawa wrote: > Hi everyone, > > I'd like to propose Ayumu Ueha as a core reviewer. He has made huge > contributions not only for reviewing actively but also developing key > features of NFV Container support and helping us for fixing critical > problems [1][2]. > > Assuming that there are no objections, we will add Ayumu to tacker core > team next week. > > [1] https://review.opendev.org/q/owner:ayumu > [2] > https://www.stackalytics.io/?module=opendev.org%2fopenstack%2ftacker&user_id=ueha > From qiujunting at inspur.com Tue Sep 14 11:55:31 2021 From: qiujunting at inspur.com (=?gb2312?B?SnVudGluZ3FpdSBRaXVqdW50aW5nICjH8b785sMp?=) Date: Tue, 14 Sep 2021 11:55:31 +0000 Subject: The code about BP Allow migrating PMEM's data Message-ID: <3d091e8b74654a2c977ac9c09e460fa9@inspur.com> Hi all: Is there anything in the code logic that needs to be modified? If there is no problem with the implementation structure of the code, I start adding unit tests. https://review.opendev.org/c/openstack/nova/+/802225 Thank you Fossen. -----????----- ???: Balazs Gibizer [mailto:balazs.gibizer at est.tech] ????: 2021?9?14? 16:45 ???: Juntingqiu Qiujunting (???) ??: smooney at redhat.com; Brin Zhang(???) ??: Re: The code about BP Allow migrating PMEM's data Hi, Sorry I cannot read the content of your mail due to smime. Also please keep the discussion on openstack-discuss if possible. Cheers, gibi On Tue, Sep 14 2021 at 01:45:49 AM +0000, Juntingqiu Qiujunting (???) wrote: From zigo at debian.org Tue Sep 14 12:04:30 2021 From: zigo at debian.org (Thomas Goirand) Date: Tue, 14 Sep 2021 14:04:30 +0200 Subject: [horizon] Support for Django 3.x In-Reply-To: <1977ce4f-bf9b-9a03-672d-b85531ea1187@debian.org> References: <1977ce4f-bf9b-9a03-672d-b85531ea1187@debian.org> Message-ID: On 9/6/21 9:29 PM, Thomas Goirand wrote: > On 9/6/21 1:00 PM, vishal manchanda wrote: >> Hi Thomas, >> >> Horizon team working on adding django3.x support in horizon. >> A series of patches to add django3.x support is already up [1] and We >> recently merged a non-voting job to test >> django3.x which is failing right now[2]. The main issue we are facing in >> migration to django3.x is horizon >> using django-pyscss which isn't compatible with django3.x although we >> have some workaround to fix it[3] >> but it is not the right way. So I tried to contact the django-pyscss >> maintainer, but they didn't respond. >> Then we plan to use an alternative of django-pyscss i.e. >> django-sass-processor. >> >> Update on django-sass-processor: Me and radomir from horizon team worked >> on adding django-sass-processor >> support in horizon[4]. As far our testing it works fine with horizon but >> horizon default theme doesn't work with it? >> right now and fails with error[5]. I am working on this issue, any help >> is really appreciated. > > Hi Vishal, > > Thanks a lot for giving a status update, I'm very happy to see there's > work toward Django 3.x support. Thanks to you and all the team. > > FYI, I unfortunately don't have the skills to help. However, I can work > for having new dependencies available in Debian, and so also in Ubuntu. > > Do you think I should package django-sass-processor right now? Or is it > too early to tell it's going to be the adopted solution? > > Cheers, > > Thomas Goirand (zigo) FYI, what I feared happened: Django 3.2.7 was uploaded to Debian Unstable yesterday, rendering Horizon Wallaby and Xena completely broken. Is there a chance that Django 3.2.x support lands in Xena? Or at least, some chance that I can gather a few patches to fix the situation? Cheers, Thomas Goirand (zigo) From ts-takahashi at nec.com Tue Sep 14 12:16:51 2021 From: ts-takahashi at nec.com (=?utf-8?B?VEFLQUhBU0hJIFRPU0hJQUtJKOmrmOapi+OAgOaVj+aYjik=?=) Date: Tue, 14 Sep 2021 12:16:51 +0000 Subject: [tacker] Ayumu Ueha for Tacker Core In-Reply-To: <2194a9ed-57a1-be99-ab6a-123e582c9df7@gmail.com> References: <2194a9ed-57a1-be99-ab6a-123e582c9df7@gmail.com> Message-ID: Welcome! > -----Original Message----- > From: Yasufumi Ogawa > Sent: Tuesday, September 14, 2021 8:39 PM > To: openstack-discuss > Subject: Re: [tacker] Ayumu Ueha for Tacker Core > > Hi, > > Since many of +1s and no objections, I'd add Ayumu to tacker-core. > Welcome Ayumu! > > On 2021/09/06 14:03, Yasufumi Ogawa wrote: > > Hi everyone, > > > > I'd like to propose Ayumu Ueha as a core reviewer. He has made huge > > contributions not only for reviewing actively but also developing key > > features of NFV Container support and helping us for fixing critical > > problems [1][2]. > > > > Assuming that there are no objections, we will add Ayumu to tacker > > core team next week. > > > > [1] https://review.opendev.org/q/owner:ayumu > > [2] > > https://www.stackalytics.io/?module=opendev.org%2fopenstack%2ftacker&u > > ser_id=ueha > > From ueha.ayumu at fujitsu.com Tue Sep 14 12:49:03 2021 From: ueha.ayumu at fujitsu.com (ueha.ayumu at fujitsu.com) Date: Tue, 14 Sep 2021 12:49:03 +0000 Subject: [tacker] Ayumu Ueha for Tacker Core In-Reply-To: References: <2194a9ed-57a1-be99-ab6a-123e582c9df7@gmail.com> Message-ID: Hi Yasufumi and Toshiaki and Team, Thank you. I'm pleased to join Tacker core team! I'll continue to contribute for Tacker community activation! Regards, Ayumu Ueha -----Original Message----- From: TAKAHASHI TOSHIAKI(?????) Sent: Tuesday, September 14, 2021 9:17 PM To: Yasufumi Ogawa ; openstack-discuss Subject: RE: [tacker] Ayumu Ueha for Tacker Core Welcome! > -----Original Message----- > From: Yasufumi Ogawa > Sent: Tuesday, September 14, 2021 8:39 PM > To: openstack-discuss > Subject: Re: [tacker] Ayumu Ueha for Tacker Core > > Hi, > > Since many of +1s and no objections, I'd add Ayumu to tacker-core. > Welcome Ayumu! > > On 2021/09/06 14:03, Yasufumi Ogawa wrote: > > Hi everyone, > > > > I'd like to propose Ayumu Ueha as a core reviewer. He has made huge > > contributions not only for reviewing actively but also developing > > key features of NFV Container support and helping us for fixing > > critical problems [1][2]. > > > > Assuming that there are no objections, we will add Ayumu to tacker > > core team next week. > > > > [1] https://review.opendev.org/q/owner:ayumu > > [2] > > https://www.stackalytics.io/?module=opendev.org%2fopenstack%2ftacker > > &u > > ser_id=ueha > > From katonalala at gmail.com Tue Sep 14 13:41:30 2021 From: katonalala at gmail.com (Lajos Katona) Date: Tue, 14 Sep 2021 15:41:30 +0200 Subject: [tc][neutron] Use Kea insted of Dibbler for IPv6 PD Message-ID: Hi, In Neutron we have a bug (see [1]) that states Dibbler is concluded (see [2]), and the developer of Dibbler suggests using Kea (see [3]) instead. Before we start checking technical things first thing is that Kea is MPL2.0 licensed (see [4]) and not sure if we can pull that as dependency to Openstack. Kea is an ISC project so looks safe, but I am not familiar with these legal things. How can we check and be sure? [1] https://bugs.launchpad.net/neutron/+bug/1916428 [2] https://github.com/tomaszmrugalski/dibbler#project-status [3] https://gitlab.isc.org/isc-projects/kea [4] https://www.isc.org/kea/ Thanks in advance Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Tue Sep 14 14:09:30 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 14 Sep 2021 14:09:30 +0000 Subject: [tc][neutron] Use Kea insted of Dibbler for IPv6 PD In-Reply-To: References: Message-ID: <20210914140930.ikjt4usicvbjhsch@yuggoth.org> On 2021-09-14 15:41:30 +0200 (+0200), Lajos Katona wrote: > In Neutron we have a bug (see [1]) that states Dibbler is > concluded (see [2]), and the developer of Dibbler suggests using > Kea (see [3]) instead. Before we start checking technical things > first thing is that Kea is MPL2.0 licensed (see [4]) and not sure > if we can pull that as dependency to Openstack. > > Kea is an ISC project so looks safe, but I am not familiar with > these legal things. How can we check and be sure? [...] I'm not a lawyer, but a cursory search indicates that MoFo considers the MPL 2.0 to be compatible with the Apache 2.0 license (see the "Licenses Compatible with the MPL" section): https://www.mozilla.org/en-US/MPL/license-policy/ Also the TC has previously declared that MPL dependencies are acceptable: https://governance.openstack.org/tc/reference/licensing.html -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From anyrude10 at gmail.com Tue Sep 14 13:48:05 2021 From: anyrude10 at gmail.com (Anirudh Gupta) Date: Tue, 14 Sep 2021 19:18:05 +0530 Subject: [TripleO] Installation of Undercloud on CentOS 8.4 Message-ID: Hi Team, I was trying to install Undercloud on CentOS 8.4 While installing the repos, I got the error "The host is not stream" So wanted to confirm if only CentOS 8 Stream is supported or is there still a way we can install TripleO on CentOS 8.4 or any specific version? Regards Anirudh Gupta -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Tue Sep 14 16:41:01 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 14 Sep 2021 11:41:01 -0500 Subject: [tc][neutron] Use Kea insted of Dibbler for IPv6 PD In-Reply-To: <20210914140930.ikjt4usicvbjhsch@yuggoth.org> References: <20210914140930.ikjt4usicvbjhsch@yuggoth.org> Message-ID: <17be52ee672.c8943e1d648813.8156633414337881645@ghanshyammann.com> ---- On Tue, 14 Sep 2021 09:09:30 -0500 Jeremy Stanley wrote ---- > On 2021-09-14 15:41:30 +0200 (+0200), Lajos Katona wrote: > > In Neutron we have a bug (see [1]) that states Dibbler is > > concluded (see [2]), and the developer of Dibbler suggests using > > Kea (see [3]) instead. Before we start checking technical things > > first thing is that Kea is MPL2.0 licensed (see [4]) and not sure > > if we can pull that as dependency to Openstack. > > > > Kea is an ISC project so looks safe, but I am not familiar with > > these legal things. How can we check and be sure? > [...] > > I'm not a lawyer, but a cursory search indicates that MoFo considers > the MPL 2.0 to be compatible with the Apache 2.0 license (see the > "Licenses Compatible with the MPL" section): > > https://www.mozilla.org/en-US/MPL/license-policy/ > > Also the TC has previously declared that MPL dependencies are > acceptable: > > https://governance.openstack.org/tc/reference/licensing.html Yes, I think there is no issue on MPL2.0 license as the above doc states that and we already have a few dependencies software with that license like 'pytest-html' - https://opendev.org/openstack/requirements/src/branch/master/global-requirements.txt#L216 -gmann > > -- > Jeremy Stanley > From aschultz at redhat.com Tue Sep 14 16:44:34 2021 From: aschultz at redhat.com (Alex Schultz) Date: Tue, 14 Sep 2021 10:44:34 -0600 Subject: [TripleO] Installation of Undercloud on CentOS 8.4 In-Reply-To: References: Message-ID: On Tue, Sep 14, 2021 at 10:29 AM Anirudh Gupta wrote: > Hi Team, > > I was trying to install Undercloud on CentOS 8.4 > While installing the repos, I got the error "The host is not stream" > > So wanted to confirm if only CentOS 8 Stream is supported or is there > still a way we can install TripleO on CentOS 8.4 or any specific version? > > The containers and rpms that we're building and distributing via RDO are based on 8-stream. If the error message you're getting is from tripleo-repos, you can use --no-stream to get past this warning as we assume stream by default. It likely should work, however YMMV and the current target is really 8-stream going forward. > Regards > Anirudh Gupta > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alifshit at redhat.com Tue Sep 14 17:55:14 2021 From: alifshit at redhat.com (Artom Lifshitz) Date: Tue, 14 Sep 2021 13:55:14 -0400 Subject: [nova] Backporting test refactors to stable/train Message-ID: Hey Elod and Vlad (and the ML, cc'ed) This is continuing the conversation Elod and I had on IRC. The immediate context is [1], which is a test refactor backport that in turn enables conflict-free backporting of at least two fixes. >From Red Hat's point of view, we will have to care about stable/train for *very* long time, so we'd like to do everything we can to make our lives easier, including backportings things are aren't strictly necessary (like test refactors) in order to greatly facilitate future backports, which tend to be of the "func reproducer test + actual fix" variety. Elod made the excellent point that while this approach does make backports to train much easier to both write and review, it also "postpones" all of the manual conflict resolution work onto anyone doing stable/stein backports. I went and gathered data. I looked at the last 6 months of backports in Gerrit. At the time of this writing, stable/train 72 [2] had and stable/stein had 24 [3]. Out of the latter, Vlad Gusev was the main non-Red Hat contributor. So my proposal is simple: I'm willing to help resolve conflicts in stable/stein backport for folks like Vlad, if it means making backports to stable/train easier. I'm confident that it would still mean an overall reduction in man-hours for backports, even if I become personally on the hook to deal with the fallout of backporting test refactors to stable/train. Cheers! [1] https://review.opendev.org/c/openstack/nova/+/791481 [2] https://review.opendev.org/q/project:openstack/nova+branch:stable/train+-age:6month [3] https://review.opendev.org/q/project:openstack/nova+branch:stable/stein+-age:6month From mkopec at redhat.com Tue Sep 14 18:18:54 2021 From: mkopec at redhat.com (Martin Kopec) Date: Tue, 14 Sep 2021 20:18:54 +0200 Subject: [patrole][qa] Proposing Soniya Vyas to Patrole core In-Reply-To: References: <17bc1bcebec.d7cc6895149770.7494147722877953142@ghanshyammann.com> Message-ID: Thank you for your feedback. As no objections were raised, we added Soniya to the patrole-core group. Welcome to the team! On Wed, 8 Sept 2021 at 14:11, Arx Cruz wrote: > +1 > > Congratulations!!! > > On Wed, Sep 8, 2021 at 12:02 PM Chandan Kumar wrote: > >> On Wed, Sep 8, 2021 at 1:03 AM Ghanshyam Mann >> wrote: >> > >> > ---- On Tue, 07 Sep 2021 14:15:04 -0500 Martin Kopec < >> mkopec at redhat.com> wrote ---- >> > > Hi all, >> > > >> > > Soniya Vyas (IRC: soniya29) is very enthusiastic about Patrole >> project and since she has been very helpful, I'd like to propose her for >> Patrole core. >> > > >> > > You can vote/feedback on this email. If no objections within a week, >> I'll add her to the core list. >> > >> > +1, she has been very helpful in Patrole project. >> >> +1, Thank you Sonia for all the great work. >> >> Thanks, >> >> Chandan Kumar >> >> >> > > -- > > Arx Cruz > > Software Engineer > > Red Hat EMEA > > arxcruz at redhat.com > @RedHat Red Hat > Red Hat > > > -- Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlandy at redhat.com Tue Sep 14 19:33:31 2021 From: rlandy at redhat.com (Ronelle Landy) Date: Tue, 14 Sep 2021 15:33:31 -0400 Subject: [patrole][qa] Proposing Soniya Vyas to Patrole core In-Reply-To: References: <17bc1bcebec.d7cc6895149770.7494147722877953142@ghanshyammann.com> Message-ID: Congrats Soniya On Tue, Sep 14, 2021 at 2:23 PM Martin Kopec wrote: > Thank you for your feedback. > > As no objections were raised, we added Soniya to the patrole-core group. > > Welcome to the team! > > On Wed, 8 Sept 2021 at 14:11, Arx Cruz wrote: > >> +1 >> >> Congratulations!!! >> >> On Wed, Sep 8, 2021 at 12:02 PM Chandan Kumar wrote: >> >>> On Wed, Sep 8, 2021 at 1:03 AM Ghanshyam Mann >>> wrote: >>> > >>> > ---- On Tue, 07 Sep 2021 14:15:04 -0500 Martin Kopec < >>> mkopec at redhat.com> wrote ---- >>> > > Hi all, >>> > > >>> > > Soniya Vyas (IRC: soniya29) is very enthusiastic about Patrole >>> project and since she has been very helpful, I'd like to propose her for >>> Patrole core. >>> > > >>> > > You can vote/feedback on this email. If no objections within a >>> week, I'll add her to the core list. >>> > >>> > +1, she has been very helpful in Patrole project. >>> >>> +1, Thank you Sonia for all the great work. >>> >>> Thanks, >>> >>> Chandan Kumar >>> >>> >>> >> >> -- >> >> Arx Cruz >> >> Software Engineer >> >> Red Hat EMEA >> >> arxcruz at redhat.com >> @RedHat Red Hat >> Red Hat >> >> >> > > > -- > Martin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lyarwood at redhat.com Tue Sep 14 19:57:25 2021 From: lyarwood at redhat.com (Lee Yarwood) Date: Tue, 14 Sep 2021 20:57:25 +0100 Subject: [nova] Backporting test refactors to stable/train In-Reply-To: References: Message-ID: <20210914195725.chfty2csxpzkxswk@lyarwood-laptop.usersys.redhat.com> On 14-09-21 13:55:14, Artom Lifshitz wrote: > Hey Elod and Vlad (and the ML, cc'ed) > > This is continuing the conversation Elod and I had on IRC. The > immediate context is [1], which is a test refactor backport that in > turn enables conflict-free backporting of at least two fixes. > > From Red Hat's point of view, we will have to care about stable/train > for *very* long time, so we'd like to do everything we can to make our > lives easier, including backportings things are aren't strictly > necessary (like test refactors) in order to greatly facilitate future > backports, which tend to be of the "func reproducer test + actual fix" > variety. > > Elod made the excellent point that while this approach does make > backports to train much easier to both write and review, it also > "postpones" all of the manual conflict resolution work onto anyone > doing stable/stein backports. I can't say I agree with the objection here but regardless I think the ship has already sailed on us backporting many of these test refactors to stable/train. We have already backported so by manually resolving conflicts and that would now conflict again with any attempt to backport the refactors. At least that has been my experience the last few times I tried to do this and I've now given up and manually resolved conflicts, for example: https://review.opendev.org/c/openstack/nova/+/785847 Going forward however and with my dusty old Red Hat on I want to try to do this as much as possible back to stable/wallaby to allow fixes to flow through our stable branches as seamlessly as possible. I don't see why such contributions should be blocked, the work required to resolve these conflicts would eventually have to be done by someone, pushing that to contributors who care about a given older release only seems fair IMHO. > I went and gathered data. I looked at the last 6 months of backports > in Gerrit. At the time of this writing, stable/train 72 [2] had and > stable/stein had 24 [3]. Out of the latter, Vlad Gusev was the main > non-Red Hat contributor. Slight tangent but only 20 landing for train and 10 landing for stein. If anyone is interested in becoming a stable core on Nova please feel free to reach out as we could really use more reviewers. > So my proposal is simple: I'm willing to help resolve conflicts in > stable/stein backport for folks like Vlad, if it means making > backports to stable/train easier. I'm confident that it would still > mean an overall reduction in man-hours for backports, even if I become > personally on the hook to deal with the fallout of backporting test > refactors to stable/train. Well that's good of you Artom but I don't think that should be required to land such backports on a given stable branch. Cheers, Lee > [1] https://review.opendev.org/c/openstack/nova/+/791481 > [2] https://review.opendev.org/q/project:openstack/nova+branch:stable/train+-age:6month > [3] https://review.opendev.org/q/project:openstack/nova+branch:stable/stein+-age:6month -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From alifshit at redhat.com Tue Sep 14 20:06:15 2021 From: alifshit at redhat.com (Artom Lifshitz) Date: Tue, 14 Sep 2021 16:06:15 -0400 Subject: [nova] Backporting test refactors to stable/train In-Reply-To: <20210914195725.chfty2csxpzkxswk@lyarwood-laptop.usersys.redhat.com> References: <20210914195725.chfty2csxpzkxswk@lyarwood-laptop.usersys.redhat.com> Message-ID: On Tue., Sep. 14, 2021, 15:57 Lee Yarwood, wrote: > On 14-09-21 13:55:14, Artom Lifshitz wrote: > > Hey Elod and Vlad (and the ML, cc'ed) > > > > This is continuing the conversation Elod and I had on IRC. The > > immediate context is [1], which is a test refactor backport that in > > turn enables conflict-free backporting of at least two fixes. > > > > From Red Hat's point of view, we will have to care about stable/train > > for *very* long time, so we'd like to do everything we can to make our > > lives easier, including backportings things are aren't strictly > > necessary (like test refactors) in order to greatly facilitate future > > backports, which tend to be of the "func reproducer test + actual fix" > > variety. > > > > Elod made the excellent point that while this approach does make > > backports to train much easier to both write and review, it also > > "postpones" all of the manual conflict resolution work onto anyone > > doing stable/stein backports. > > I can't say I agree with the objection here but regardless I think the > ship has already sailed on us backporting many of these test refactors to > stable/train. > Maybe, but at least in some cases it's still possible, like my [1] link in my previous message. We have already backported so by manually resolving conflicts and that > would now conflict again with any attempt to backport the refactors. At > least that has been my experience the last few times I tried to do this > and I've now given up and manually resolved conflicts, for example: > > https://review.opendev.org/c/openstack/nova/+/785847 > > Going forward however and with my dusty old Red Hat on I want to try to > do this as much as possible back to stable/wallaby to allow fixes to > flow through our stable branches as seamlessly as possible. > > I don't see why such contributions should be blocked, the work required > to resolve these conflicts would eventually have to be done by someone, > pushing that to contributors who care about a given older release only > seems fair IMHO. > Fair, hard to argue with "if you care about something, put in the work." > I went and gathered data. I looked at the last 6 months of backports > > in Gerrit. At the time of this writing, stable/train 72 [2] had and > > stable/stein had 24 [3]. Out of the latter, Vlad Gusev was the main > > non-Red Hat contributor. > > Slight tangent but only 20 landing for train and 10 landing for stein. > > If anyone is interested in becoming a stable core on Nova please feel > free to reach out as we could really use more reviewers. > > > So my proposal is simple: I'm willing to help resolve conflicts in > > stable/stein backport for folks like Vlad, if it means making > > backports to stable/train easier. I'm confident that it would still > > mean an overall reduction in man-hours for backports, even if I become > > personally on the hook to deal with the fallout of backporting test > > refactors to stable/train. > > Well that's good of you Artom but I don't think that should be required > to land such backports on a given stable branch. > Just trying to be a good community citizen, and show that I'm open to listen to others' concerns :) (And, full disclosure, also making the bet that it'll happen so rarely that it won't affect me.) > Cheers, > > Lee > > > [1] https://review.opendev.org/c/openstack/nova/+/791481 > > [2] > https://review.opendev.org/q/project:openstack/nova+branch:stable/train+-age:6month > > [3] > https://review.opendev.org/q/project:openstack/nova+branch:stable/stein+-age:6month > > -- > Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 > 2D76 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fabian.wiesel at sap.com Wed Sep 15 07:08:40 2021 From: fabian.wiesel at sap.com (Wiesel, Fabian) Date: Wed, 15 Sep 2021 07:08:40 +0000 Subject: [nova] "Migrating" ComputeNode within ComputeHost / vmwareapi ComputeNode per ESXi Message-ID: <9DD35476-3627-436D-B241-442B80E5D432@sap.com> Hi, I am looking at adding the option for the vmwareapi driver to expose the individual ESXi hosts. A compute-host would represent a cluster in vsphere, and each ESXi-host within a cluster would be a ComputeNode. The main question here is, how to handle a hypervisor initiated change of the compute-node and if the community is open to extending the compute-driver interface for such a use-case. The secondary one is, if there is interest in having the upstream vmwareapi driver getting this functionality. * What are we trying to solve by that? Our users are asking for that feature, because they want to know, if a payload of two VMs runs on the same physical host, as the performance characteristics are quite different. We also gain there from an operational and code perspective: - Nova would become aware of the actual resources per ESXi host instead of the aggregated ones. - Resources could also be more faithfully represented in the resource-tree (e.g., shared storage as an aggregate). - Capabilities, (cpu-flags, architecture,? ), Numa topology, and even small things like up-time can be faithfully reflected on a per-host basis. - The internal scheduling decisions in the driver for ephemeral storage and compute (for live-migration) can be handled by standard placement & nova scheduling. * Why do we need to change the driver interface? Main problem here is that VSphere might migrate a VM between two ESXi-hosts for various reasons. So, the driver would need a way to communicate such a change somehow. But even if we disable all kind of migrations on VMware side, we still need a way to migrate the VMs from the single compute-node for the whole cluster to the compute nodes for each individual ESXi-hosts. * Proposed Driver API Change The Compute-Manager is currently periodically polling the driver with ComputeDriver.get_info(self, instance, use_cache) which returns an InstanceInfo, which is essentially only the power-state. Adding two new optional fields to InstanceInfo could allow the driver to inform the manager about - the current compute-node - the compute-node the VM is migrated to This allows the Compute-Manager to take care of updating the node field and the resource-allocations accordingly. Additionally, we can notify the manager over Instance lifecycle events, allowing for a faster reflection of a compute-node change. The lifecycle events could be - EVENT_LIFECYCLE_COMPUTE_NODE_MIGRATION_STARTED - EVENT_LIFECYCLE_COMPUTE_NODE_MIGRATION_COMPLETED Or one could generalize the current life-migration events to include also such compute-node changes. I am looking forward for your feedback. Cheers, Fabian From katonalala at gmail.com Wed Sep 15 07:46:34 2021 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 15 Sep 2021 09:46:34 +0200 Subject: [tc][neutron] Use Kea insted of Dibbler for IPv6 PD In-Reply-To: <17be52ee672.c8943e1d648813.8156633414337881645@ghanshyammann.com> References: <20210914140930.ikjt4usicvbjhsch@yuggoth.org> <17be52ee672.c8943e1d648813.8156633414337881645@ghanshyammann.com> Message-ID: Hi, Thanks, we can continue to check the technical part. Ghanshyam Mann ezt ?rta (id?pont: 2021. szept. 14., K, 18:47): > ---- On Tue, 14 Sep 2021 09:09:30 -0500 Jeremy Stanley > wrote ---- > > On 2021-09-14 15:41:30 +0200 (+0200), Lajos Katona wrote: > > > In Neutron we have a bug (see [1]) that states Dibbler is > > > concluded (see [2]), and the developer of Dibbler suggests using > > > Kea (see [3]) instead. Before we start checking technical things > > > first thing is that Kea is MPL2.0 licensed (see [4]) and not sure > > > if we can pull that as dependency to Openstack. > > > > > > Kea is an ISC project so looks safe, but I am not familiar with > > > these legal things. How can we check and be sure? > > [...] > > > > I'm not a lawyer, but a cursory search indicates that MoFo considers > > the MPL 2.0 to be compatible with the Apache 2.0 license (see the > > "Licenses Compatible with the MPL" section): > > > > https://www.mozilla.org/en-US/MPL/license-policy/ > > > > Also the TC has previously declared that MPL dependencies are > > acceptable: > > > > https://governance.openstack.org/tc/reference/licensing.html > > Yes, I think there is no issue on MPL2.0 license as the above doc states > that > and we already have a few dependencies software with that license like > 'pytest-html' > > - > https://opendev.org/openstack/requirements/src/branch/master/global-requirements.txt#L216 > > -gmann > > > > -- > > Jeremy Stanley > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 15 08:54:12 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 15 Sep 2021 10:54:12 +0200 Subject: kubernetes cluster stuck in CREATE_IN_PROGRESS but stack list shows CREATE_COMPLETE Message-ID: Dear Team, I have configured Magnum and I am able to kubernetes clusters and when I log in to the nodes everything is OK. But Minus the first cluster I created which completed well as per the openstack coe cluster list command and the output on Horizon, the rest of the clusters get stack in CREATE_IN_PROGRESS . Please check the below commands output and advise. (kolla-openstack) stack at deployment:~$ openstack stack list +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | e9bfad7c-c82e-4fc6-9f05-793b31631761 | k8s-cluster2-zsgfkrobjlw6 | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-15T08:01:50Z | None | | 9dde1665-8e50-4482-a986-810eb2a9cbaa | k8s-cluster1-eogcos7ee3ub | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-10T07:27:41Z | None | +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ (kolla-openstack) stack at deployment:~$ openstack coe cluster list +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status | +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | 277e0047-4ae1-4006-869e-a824cc602bcc | k8s-cluster1 | Newkey | 2 | 1 | CREATE_COMPLETE | HEALTHY | | 4336f9d9-9bea-477f-bc13-c16ff0f7e141 | k8s-cluster2 | Newkey | 2 | 1 | CREATE_IN_PROGRESS | None | +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ Regards Tony Karera -------------- next part -------------- An HTML attachment was scrubbed... URL: From balazs.gibizer at est.tech Wed Sep 15 09:02:16 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Wed, 15 Sep 2021 11:02:16 +0200 Subject: [nova] Backporting test refactors to stable/train In-Reply-To: References: Message-ID: On Tue, Sep 14 2021 at 01:55:14 PM -0400, Artom Lifshitz wrote: > Hey Elod and Vlad (and the ML, cc'ed) > > This is continuing the conversation Elod and I had on IRC. The > immediate context is [1], which is a test refactor backport that in > turn enables conflict-free backporting of at least two fixes. > > From Red Hat's point of view, we will have to care about stable/train > for *very* long time, so we'd like to do everything we can to make our > lives easier, including backportings things are aren't strictly > necessary (like test refactors) in order to greatly facilitate future > backports, which tend to be of the "func reproducer test + actual fix" > variety. > > Elod made the excellent point that while this approach does make > backports to train much easier to both write and review, it also > "postpones" all of the manual conflict resolution work onto anyone > doing stable/stein backports. > > I went and gathered data. I looked at the last 6 months of backports > in Gerrit. At the time of this writing, stable/train 72 [2] had and > stable/stein had 24 [3]. Out of the latter, Vlad Gusev was the main > non-Red Hat contributor. > > So my proposal is simple: I'm willing to help resolve conflicts in > stable/stein backport for folks like Vlad, if it means making > backports to stable/train easier. I'm confident that it would still > mean an overall reduction in man-hours for backports, even if I become > personally on the hook to deal with the fallout of backporting test > refactors to stable/train. In my view there is work and risk attached to resolve a conflict during backporting a bugfix, as well as work and risk attached to backporting a test refactor instead. If the former is bigger than the latter then I would do the latter. As the latter potentially helps avoiding multiple instance of the former I can imagine that the latter is a better general approach. Based on this I can even accept non-test but production code refactors to be backported as we did for example with the libvirt device detach series. Cheers, gibi > > Cheers! > > [1] https://review.opendev.org/c/openstack/nova/+/791481 > [2] > https://review.opendev.org/q/project:openstack/nova+branch:stable/train+-age:6month > [3] > https://review.opendev.org/q/project:openstack/nova+branch:stable/stein+-age:6month > > From svyas at redhat.com Wed Sep 15 09:08:09 2021 From: svyas at redhat.com (Soniya Vyas) Date: Wed, 15 Sep 2021 14:38:09 +0530 Subject: [patrole][qa] Proposing Soniya Vyas to Patrole core In-Reply-To: References: <17bc1bcebec.d7cc6895149770.7494147722877953142@ghanshyammann.com> Message-ID: Thanks everyone for your support and motivation. Regards, Soniya Vyas -------------- next part -------------- An HTML attachment was scrubbed... URL: From wodel.youchi at gmail.com Wed Sep 15 10:06:02 2021 From: wodel.youchi at gmail.com (wodel youchi) Date: Wed, 15 Sep 2021 11:06:02 +0100 Subject: Need some clarification about tenant network in TripleO Message-ID: Hi, It is a newbie question, but I didn't find a real answer, and I have trouble with my deployment, so some help will be appreciated. In the deployment of TripleO in network isolation, a Tenant network is created, that network (in the examples I have seen) have a VLAN-ID and a subnet. I did the same in my deployment, I created that network. My questions : - Is the Tenant network, mentioned in the deployment process, is it the first tenant network created for the admin tenant? and for new tenant I have to create a new network tenant, despite I am not seeing how I would do that? - Or is it a pipe (a conduit) used to pass the traffic of all the tenants that will be created after? Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From micou12 at gmail.com Wed Sep 15 10:07:01 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Wed, 15 Sep 2021 12:07:01 +0200 Subject: Cannot create a swift container on Horizon dashboard , mandatory "Storage Policy" dropdown field is empty Message-ID: Hello team , I am replacing swift by Ceph Radosgateway , and I am successful by creating containers through openstack and ceph CLI side . but once trying to create through the horizon dashboard I get errors: *Error: *Unable to fetch the policy details. , Unable to get the Swift service info and Unable to get the Swift container listing. Kindly help to solve the issue I deployed Openstack wallaby using kolla-ansible running on ubuntu 20.04 and ceph pacific using ansible running on ubuntu 20.04 Michel -------------- next part -------------- An HTML attachment was scrubbed... URL: From johfulto at redhat.com Wed Sep 15 11:16:44 2021 From: johfulto at redhat.com (John Fulton) Date: Wed, 15 Sep 2021 07:16:44 -0400 Subject: Need some clarification about tenant network in TripleO In-Reply-To: References: Message-ID: On Wed, Sep 15, 2021 at 6:07 AM wodel youchi wrote: > > Hi, > > It is a newbie question, but I didn't find a real answer, and I have trouble with my deployment, so some help will be appreciated. > > In the deployment of TripleO in network isolation, a Tenant network is created, that network (in the examples I have seen) have a VLAN-ID and a subnet. > I did the same in my deployment, I created that network. > > My questions : > - Is the Tenant network, mentioned in the deployment process, is it the first tenant network created for the admin tenant? and for new tenant I have to create a new network tenant, despite I am not seeing how I would do that? > - Or is it a pipe (a conduit) used to pass the traffic of all the tenants that will be created after? Convention is for it to be a pipe, i.e. the second option. Usually the tenant network will be its own VLAN and then an overlay network (e.g. Geneve, VXLAN, etc) will create tenant networks on top of that VLAN when each tenant requests with a command like `openstack network create my_private_network`. On a default TripleO install you should be able to source the overcloudrc and then run that command to create these overlay networks. John > > > Regards. From kkchn.in at gmail.com Wed Sep 15 11:17:53 2021 From: kkchn.in at gmail.com (KK CHN) Date: Wed, 15 Sep 2021 16:47:53 +0530 Subject: LVM migration from one SAN storage to another New SAN Box Message-ID: List, I want to migrate our old SAN storage backend which contains LVMs used for the cloud storage, to a New SAN Storage box. The requirement was to migrate all storage of the old SAN box, which was mounted to the controller node as /dev/sdb and /dev/sdc around total 6TB storage. 80 % full. Then AMC of this box getting over. So what all are the steps to perform the migration of the old storage to new SAN storage . Any hints most welcome Kris -------------- next part -------------- An HTML attachment was scrubbed... URL: From wodel.youchi at gmail.com Wed Sep 15 12:07:39 2021 From: wodel.youchi at gmail.com (wodel youchi) Date: Wed, 15 Sep 2021 13:07:39 +0100 Subject: Need some clarification about tenant network in TripleO In-Reply-To: References: Message-ID: Thank you for this explanation. To the problem now, I installed Train, then I created a new tenant with its private network, then I created another tenant with its own private network. But I used the same subnet for both networks, then I spawned 4 VMs, 2 per tenant, they got their IP addresses, and the problem is, they all can ping each other, there is no isolation between the tenants. I don't know where to begin searching. Regards. Le mer. 15 sept. 2021 ? 12:16, John Fulton a ?crit : > On Wed, Sep 15, 2021 at 6:07 AM wodel youchi > wrote: > > > > Hi, > > > > It is a newbie question, but I didn't find a real answer, and I have > trouble with my deployment, so some help will be appreciated. > > > > In the deployment of TripleO in network isolation, a Tenant network is > created, that network (in the examples I have seen) have a VLAN-ID and a > subnet. > > I did the same in my deployment, I created that network. > > > > My questions : > > - Is the Tenant network, mentioned in the deployment process, is it the > first tenant network created for the admin tenant? and for new tenant I > have to create a new network tenant, despite I am not seeing how I would do > that? > > - Or is it a pipe (a conduit) used to pass the traffic of all the > tenants that will be created after? > > Convention is for it to be a pipe, i.e. the second option. Usually the > tenant network will be its own VLAN and then an overlay network (e.g. > Geneve, VXLAN, etc) will create tenant networks on top of that VLAN > when each tenant requests with a command like `openstack network > create my_private_network`. On a default TripleO install you should be > able to source the overcloudrc and then run that command to create > these overlay networks. > > John > > > > > > > Regards. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 15 12:36:20 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 15 Sep 2021 14:36:20 +0200 Subject: Changing the default Network Quotas Message-ID: Hello Team, Can anybody advise on how to change the Network Quotas for Openstack Wallaby deployed using Kolla-ansbile. Regards Tony Karera -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Wed Sep 15 13:52:53 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 15 Sep 2021 13:52:53 +0000 Subject: [ops] LVM migration from one SAN storage to another New SAN Box In-Reply-To: References: Message-ID: <20210915135006.k3qncievkoczbniz@yuggoth.org> On 2021-09-15 16:47:53 +0530 (+0530), KK CHN wrote: > I want to migrate our old SAN storage backend which contains LVMs > used for the cloud storage, to a New SAN Storage box. > > The requirement was to migrate all storage of the old SAN box, > which was mounted to the controller node as /dev/sdb and /dev/sdc > around total 6TB storage. 80 % full. Then AMC of this box getting > over. > > So what all are the steps to perform the migration of the old > storage to new SAN storage. Any hints most welcome While I've never tried this on an OpenStack compute node specifically, transparent relocation of logical volumes from one physical volume to another is a very longstanding feature of the LVM2 implementation in the Linux kernel, and I've used it many times on general purpose servers to replace the backend block devices in all sorts of situations. The short answer is that you connect and add the new physical volumes to the same volume group which contains your existing logical volumes, then pvmove the extents for those LVs from the old PV to the new one, then you can remove the old PVs from the VG and disconnect them from the server. There is no need to take the volumes out of production for this, it can be done live while they're actively in use. The main caveat is that pvmove will put additional strain on your I/O channels moving data from one disk to another, possibly slowing your overall disk performance while it runs. I'll leave it to folks who manage OpenStack deployments to say whether this is a terrible idea, but this simple solution has served me well over the decades. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From senrique at redhat.com Wed Sep 15 14:11:26 2021 From: senrique at redhat.com (Sofia Enriquez) Date: Wed, 15 Sep 2021 11:11:26 -0300 Subject: [cinder] Bug deputy report for week of 2021-15-09 Message-ID: This is a bug report from 2021-08-09 to 2021-15-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/cinder/+bug/1943493 "backups.num_dependent_backups doesn't need to be stored in the db as an integer". Unassigned - https://bugs.launchpad.net/cinder/+bug/1934025 "Create a separate [quota] section for quota parameters". Unassigned - https://bugs.launchpad.net/cinder/+bug/1930290 "PowerMax - QoS should not be set on the parent storage group". Unassigned - https://bugs.launchpad.net/os-brick/+bug/1943615 "nvmeof connector gets device failing on ubuntu focal kernel". Unassigned - https://bugs.launchpad.net/os-brick/+bug/1888675 "fail to extend in-use fibre channel volume due to multipath-tools version" . Unassigned Low - https://bugs.launchpad.net/cinder/+bug/1943689 "[Doc] The interface of list snapshots and details supports volume filtering, but this is not documented". Unassigned - https://bugs.launchpad.net/cinder/+bug/1943682 "[Storwize] update rccg name to metadata for clone-group-volumes". Unassigned Invalid - https://bugs.launchpad.net/cinder/+bug/1927330 " Erroneous formula for calculation of virtual free capacity ". Unassigned -- L. Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Wed Sep 15 14:12:02 2021 From: smooney at redhat.com (Sean Mooney) Date: Wed, 15 Sep 2021 15:12:02 +0100 Subject: [ops] LVM migration from one SAN storage to another New SAN Box In-Reply-To: <20210915135006.k3qncievkoczbniz@yuggoth.org> References: <20210915135006.k3qncievkoczbniz@yuggoth.org> Message-ID: <3e4799b5c46aab0934574cb0f186d4ecff6fad30.camel@redhat.com> On Wed, 2021-09-15 at 13:52 +0000, Jeremy Stanley wrote: > On 2021-09-15 16:47:53 +0530 (+0530), KK CHN wrote: > > I want to migrate our old SAN storage backend which contains LVMs > > used for the cloud storage, to a New SAN Storage box. > > > > The requirement was to migrate all storage of the old SAN box, > > which was mounted to the controller node as /dev/sdb and /dev/sdc > > around total 6TB storage. 80 % full. Then AMC of this box getting > > over. > > > > So what all are the steps to perform the migration of the old > > storage to new SAN storage. Any hints most welcome > > While I've never tried this on an OpenStack compute node > specifically, transparent relocation of logical volumes from one > physical volume to another is a very longstanding feature of the > LVM2 implementation in the Linux kernel, and I've used it many times > on general purpose servers to replace the backend block devices in > all sorts of situations. i have done this by hand on my home system moving volume from local disk to a disk shlef. the approch i did was to expand the volume group and add the destiantion as a second PV to the VG then i belive i had lvm move the data betwen the too for each volume then remove the orgin PV form the VG. > > The short answer is that you connect and add the new physical > volumes to the same volume group which contains your existing > logical volumes, then pvmove the extents for those LVs from the old > PV to the new one, then you can remove the old PVs from the VG and > disconnect them from the server. There is no need to take the > volumes out of production for this, it can be done live while > they're actively in use. The main caveat is that pvmove will put > additional strain on your I/O channels moving data from one disk to > another, possibly slowing your overall disk performance while it > runs. yep this is what i did. although since it was just a home cloud i fully stopped the vm while i was doing it. > > I'll leave it to folks who manage OpenStack deployments to say > whether this is a terrible idea, but this simple solution has served > me well over the decades. ya not sure how bad an idea it is the main concern would like be the createion of new vms/volume while you are doing this. if you can block that it woudl certenly help. From adivya1.singh at gmail.com Wed Sep 15 14:42:59 2021 From: adivya1.singh at gmail.com (Adivya Singh) Date: Wed, 15 Sep 2021 20:12:59 +0530 Subject: Steps for Controller Node Removal in Victoria Version Openstack using ansible In-Reply-To: References: Message-ID: Hi Team, Any inputs on this Regards Adivya Singh 9590986094 On Thu, Sep 9, 2021 at 12:01 AM Adivya Singh wrote: > Hello Team, > > Is there any way , We can have a link or guide to decommission a > controller node > We have 3 controller nodes and 4 compute nodes, and we are planning to > Scale down the Infrastructure by 1 controller and 2 compute nodes > > Regards > Adivya Singh > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoralej at redhat.com Wed Sep 15 15:11:15 2021 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Wed, 15 Sep 2021 17:11:15 +0200 Subject: [TripleO][RDO] Plan for TripleO packages in RDO Xena release Message-ID: Hi, With OpenStack Xena GA approaching and TripleO as an independent project, we need to clarify how we will handle it in RDO Xena Release. According to the spec [1]: "TripleO would no longer be able to deploy all versions of OpenStack. One idea that was brough forth in the discussions around this topic thus far, is that we can attempt to address this by designating a range of git tags as compatible with a particular OpenStack stable branch." In practical terms: - When will this list be provided? Can I assume shortly after Xena release (around mid october)? - How will this "list of tags compatible with Xena" be provided by the TripleO project? - How may we describe the maintenance status of TripleO in RDO Xena release? We'd like to include a statement in the release notes. Best regards, Alfredo https://specs.openstack.org/openstack/tripleo-specs/specs/xena/tripleo-independent-release.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From zeeshan.emallates at gmail.com Wed Sep 15 05:19:20 2021 From: zeeshan.emallates at gmail.com (Zeeshan Haider) Date: Wed, 15 Sep 2021 09:19:20 +0400 Subject: Neutron Message-ID: **Scenario**: We are trying to develop our Virtual Network using `Open Virtual Network` or `OVN`, We need our Virtual Network Switches running firstly on the root Virtual Router, and there are some other Virtual Switches as well. I was Working on the `openvswitch` and `OVN` but a lot of things are missing for me, and there are some unlinked dots between the deployment itself and there are some problems. My Question has 3 further Parts. 1. Is it possible at all to use `Neutron` as our Network Manager without OpenStack cloud installation? 2. Is it a good approach? 3. if not then are there any cookbooks available for `OVN`, I am not talking about man-pages. Any help would be highly appreciated. P.S. I am thinking about this because OVN got merged with Neutron, and it could be one of the possibilities. **Note**: Please Don't downvote the question because I don't have that much experience with the network could be a stupid question, will remove it if you provide me the info in the comment section. I have found out that installation is possible I am quoting the answer from `https://bugs.launchpad.net/neutron` by `https://launchpad.net/~amotoki` below. "The way to install neutron is simple: to install neutron with pip (from PyPI package or the git repo), prepare a config file for neutron and start the neutron server (in case of OVN mechanism driver). Distro packages and/or various tools like Ansible playbook are just a wrapper for this. The minimum required config to enable noauth (ie without keystone) is to set auth_strategy to noauth in the config file. I don't know the detail instruction step by step to satisfy your need." Now I want to find out detailed instructions for this purpose. is this possible to get the instructions here -------------- next part -------------- An HTML attachment was scrubbed... URL: From feilong at catalyst.net.nz Wed Sep 15 18:48:06 2021 From: feilong at catalyst.net.nz (feilong) Date: Thu, 16 Sep 2021 06:48:06 +1200 Subject: kubernetes cluster stuck in CREATE_IN_PROGRESS but stack list shows CREATE_COMPLETE In-Reply-To: References: Message-ID: <89aa86e8-2955-53c6-a8b7-2a6ab12a300e@catalyst.net.nz> Hi Tony, Good to see your cluster can be created "successfully" ;) Technically, Magnum doesn't maintain the cluster status, the status is synced from Heat. That said, if the stack is created successfully, but the magnum cluster is in progress, that means something wrong when syncing the status. The issue happened before, I just cannot remember the exact bug ID, you should be able to find it by google. And here is the code, https://github.com/openstack/magnum/blob/master/magnum/service/periodic.py#L159 you can just add some breakpoints to debug it, it shouldn't be hard. On 15/09/21 8:54 pm, Karera Tony wrote: > Dear Team, > > I have configured Magnum and I am able to kubernetes clusters and when > I log in to the nodes everything is OK. > > But Minus the first cluster I created which completed well as per the > openstack coe cluster list command?and the output on Horizon, the rest > of the clusters get stack in CREATE_IN_PROGRESS . > > Please check the below commands output and advise. > > (kolla-openstack) stack at deployment:~$ openstack stack list > +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ > | ID ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? | Stack Name ? ? ? ? ? ? ? ?| > Project ? ? ? ? ? ? ? ? ? ? ? ? ?| Stack Status ? ?| Creation Time ? ? > ? ?| Updated Time | > +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ > | e9bfad7c-c82e-4fc6-9f05-793b31631761 | k8s-cluster2-zsgfkrobjlw6 | > 456586218b26442ebeff03643684faed | CREATE_COMPLETE | > 2021-09-15T08:01:50Z | None ? ? ? ? | > | 9dde1665-8e50-4482-a986-810eb2a9cbaa | k8s-cluster1-eogcos7ee3ub | > 456586218b26442ebeff03643684faed | CREATE_COMPLETE | > 2021-09-10T07:27:41Z | None ? ? ? ? | > +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ > (kolla-openstack) stack at deployment:~$ openstack coe cluster list > +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ > | uuid ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? | name ? ? ? ? | keypair | > node_count | master_count | status ? ? ? ? ? ? | health_status | > +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ > | 277e0047-4ae1-4006-869e-a824cc602bcc | k8s-cluster1 | Newkey ?| ? ? > ? ? ?2 | ? ? ? ? ? ?1 | CREATE_COMPLETE ? ?| HEALTHY ? ? ? | > | 4336f9d9-9bea-477f-bc13-c16ff0f7e141 | k8s-cluster2 | Newkey ?| ? ? > ? ? ?2 | ? ? ? ? ? ?1 | CREATE_IN_PROGRESS | None ? ? ? ? ?| > +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ > Regards > > Tony Karera > > -- Cheers & Best regards, ------------------------------------------------------------------------------ Feilong Wang (???) (he/him) Head of Research & Development Catalyst Cloud Aotearoa's own Mob: +64 21 0832 6348 | www.catalystcloud.nz Level 6, 150 Willis Street, Wellington 6011, New Zealand CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipient, any use, reliance upon, disclosure or copying of this email or its attachments is unauthorised. If you have received this email in error, please reply via email or call +64 21 0832 6348. ------------------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From zigo at debian.org Wed Sep 15 22:31:49 2021 From: zigo at debian.org (Thomas Goirand) Date: Thu, 16 Sep 2021 00:31:49 +0200 Subject: [horizon] Support for Django 3.x In-Reply-To: References: <1977ce4f-bf9b-9a03-672d-b85531ea1187@debian.org> Message-ID: On 9/14/21 2:04 PM, Thomas Goirand wrote: > FYI, what I feared happened: Django 3.2.7 was uploaded to Debian > Unstable yesterday, rendering Horizon Wallaby and Xena completely broken. > > Is there a chance that Django 3.2.x support lands in Xena? Or at least, > some chance that I can gather a few patches to fix the situation? > > Cheers, > > Thomas Goirand (zigo) Guys, I've seen what you're doing, and I'm amazed. THANK YOU ! And a special big-up to Akihiro Motoki! Cheers, Thomas Goirand (zigo) P.S: Feel free to ping me when the last patch merges... From gmann at ghanshyammann.com Wed Sep 15 22:32:53 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 15 Sep 2021 17:32:53 -0500 Subject: [all][tc] Technical Committee next weekly meeting on Sept 16th at 1500 UTC In-Reply-To: <17bdfebb4fd.eb8b0d59570336.8347738122602559606@ghanshyammann.com> References: <17bdfebb4fd.eb8b0d59570336.8347738122602559606@ghanshyammann.com> Message-ID: <17beb97677e.d11ee8ab733059.7735607257359595309@ghanshyammann.com> Hello Everyone, Below is the agenda for Tomorrow's TC meeting schedule at 1500 UTC in #openstack-tc IRC channel. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting == Agenda for tomorrow's TC meeting == * Roll call * Follow up on past action items * Gate health check (dansmith/yoctozepto) ** http://paste.openstack.org/show/jD6kAP9tHk7PZr2nhv8h/ * TC tags analysis ** https://docs.google.com/spreadsheets/d/18GXibtdQnSkIwA7DsBvX9dPwbw9JH76AhhRUknzBb3Q/edit * Open Reviews ** https://review.opendev.org/q/projects:openstack/governance+is:open -gmann ---- On Mon, 13 Sep 2021 11:09:31 -0500 Ghanshyam Mann wrote ---- > Hello Everyone, > > Technical Committee's next weekly meeting is scheduled for Sept 16th at 1500 UTC. > > If you would like to add topics for discussion, please add them to the below wiki page by > Wednesday, Sept 15th, at 2100 UTC. > > https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting > > -gmann > > > From qiujunting at inspur.com Thu Sep 16 01:12:06 2021 From: qiujunting at inspur.com (=?gb2312?B?SnVudGluZ3FpdSBRaXVqdW50aW5nICjH8b785sMp?=) Date: Thu, 16 Sep 2021 01:12:06 +0000 Subject: [nova]The code about BP Allow migrating PMEM's dat Message-ID: <81acb2be96d242279e1a5f0122ed1986@inspur.com> Hi all: Is there anything in the code logic that needs to be modified? If there is no problem with the implementation structure of the code, I start adding unit tests. https://review.opendev.org/c/openstack/nova/+/802225 Thank you Fossen. --------------------------------- Fossen Qiu | ??? CBRD | ?????????? T: 18249256272 E: qiujunting at inspur.com ???? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 3522 bytes Desc: image001.jpg URL: From tonykarera at gmail.com Thu Sep 16 04:57:34 2021 From: tonykarera at gmail.com (Karera Tony) Date: Thu, 16 Sep 2021 06:57:34 +0200 Subject: kubernetes cluster stuck in CREATE_IN_PROGRESS but stack list shows CREATE_COMPLETE In-Reply-To: <89aa86e8-2955-53c6-a8b7-2a6ab12a300e@catalyst.net.nz> References: <89aa86e8-2955-53c6-a8b7-2a6ab12a300e@catalyst.net.nz> Message-ID: Hello Feilong, The Clusters are now Ok and they are all in CEATE_COMPLETE after restarting the heat and magnum containers. Thanks Regards Tony Karera On Wed, Sep 15, 2021 at 8:48 PM feilong wrote: > Hi Tony, > > Good to see your cluster can be created "successfully" ;) Technically, > Magnum doesn't maintain the cluster status, the status is synced from Heat. > That said, if the stack is created successfully, but the magnum cluster is > in progress, that means something wrong when syncing the status. The issue > happened before, I just cannot remember the exact bug ID, you should be > able to find it by google. And here is the code, > https://github.com/openstack/magnum/blob/master/magnum/service/periodic.py#L159 > you can just add some breakpoints to debug it, it shouldn't be hard. > > > On 15/09/21 8:54 pm, Karera Tony wrote: > > Dear Team, > > I have configured Magnum and I am able to kubernetes clusters and when I > log in to the nodes everything is OK. > > But Minus the first cluster I created which completed well as per the > openstack coe cluster list command and the output on Horizon, the rest of > the clusters get stack in CREATE_IN_PROGRESS . > > Please check the below commands output and advise. > > (kolla-openstack) stack at deployment:~$ openstack stack list > > +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ > | ID | Stack Name | > Project | Stack Status | Creation Time | > Updated Time | > > +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ > | e9bfad7c-c82e-4fc6-9f05-793b31631761 | k8s-cluster2-zsgfkrobjlw6 | > 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-15T08:01:50Z | > None | > | 9dde1665-8e50-4482-a986-810eb2a9cbaa | k8s-cluster1-eogcos7ee3ub | > 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-10T07:27:41Z | > None | > > +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ > (kolla-openstack) stack at deployment:~$ openstack coe cluster list > > +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ > | uuid | name | keypair | > node_count | master_count | status | health_status | > > +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ > | 277e0047-4ae1-4006-869e-a824cc602bcc | k8s-cluster1 | Newkey | > 2 | 1 | CREATE_COMPLETE | HEALTHY | > | 4336f9d9-9bea-477f-bc13-c16ff0f7e141 | k8s-cluster2 | Newkey | > 2 | 1 | CREATE_IN_PROGRESS | None | > > +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ > Regards > > Tony Karera > > > -- > Cheers & Best regards, > ------------------------------------------------------------------------------ > Feilong Wang (???) (he/him) > Head of Research & Development > > Catalyst Cloud > Aotearoa's own > > Mob: +64 21 0832 6348 | www.catalystcloud.nz > Level 6, 150 Willis Street, Wellington 6011, New Zealand > > CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. > It may contain privileged, confidential or copyright information. If you are > not the named recipient, any use, reliance upon, disclosure or copying of this > email or its attachments is unauthorised. If you have received this email in > error, please reply via email or call +64 21 0832 6348. > ------------------------------------------------------------------------------ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Thu Sep 16 05:00:27 2021 From: tonykarera at gmail.com (Karera Tony) Date: Thu, 16 Sep 2021 07:00:27 +0200 Subject: Changing Default Network Quotas in Openstack Message-ID: Hello Team, I have a working Wallaby Openstack deployed using kolla-ansible. However, I have not been able to find a way of changing the Network quotas as I need to increase the defaults. Can someone kindly advise on how to go about it ? Regards Tony Karera -------------- next part -------------- An HTML attachment was scrubbed... URL: From syedammad83 at gmail.com Thu Sep 16 05:42:06 2021 From: syedammad83 at gmail.com (Ammad Syed) Date: Thu, 16 Sep 2021 10:42:06 +0500 Subject: kubernetes cluster stuck in CREATE_IN_PROGRESS but stack list shows CREATE_COMPLETE In-Reply-To: References: <89aa86e8-2955-53c6-a8b7-2a6ab12a300e@catalyst.net.nz> Message-ID: Hello Tony, I am suspecting that you are hitting the same bug as mine. You can see full details of bug here. https://storyboard.openstack.org/#!/story/2009141 Usually what happens it magnum-conductor service got crash randomly. You can confirm this by executing the command: openstack coe service list You will see that conductor service is down, once you restart conductor service, the cluster state will be updated to CREATE_COMPLETE. Ammad On Thu, Sep 16, 2021 at 10:02 AM Karera Tony wrote: > Hello Feilong, > > The Clusters are now Ok and they are all in CEATE_COMPLETE after > restarting the heat and magnum containers. > > Thanks > > Regards > > Tony Karera > > > > > On Wed, Sep 15, 2021 at 8:48 PM feilong wrote: > >> Hi Tony, >> >> Good to see your cluster can be created "successfully" ;) Technically, >> Magnum doesn't maintain the cluster status, the status is synced from Heat. >> That said, if the stack is created successfully, but the magnum cluster is >> in progress, that means something wrong when syncing the status. The issue >> happened before, I just cannot remember the exact bug ID, you should be >> able to find it by google. And here is the code, >> https://github.com/openstack/magnum/blob/master/magnum/service/periodic.py#L159 >> you can just add some breakpoints to debug it, it shouldn't be hard. >> >> >> On 15/09/21 8:54 pm, Karera Tony wrote: >> >> Dear Team, >> >> I have configured Magnum and I am able to kubernetes clusters and when I >> log in to the nodes everything is OK. >> >> But Minus the first cluster I created which completed well as per the >> openstack coe cluster list command and the output on Horizon, the rest of >> the clusters get stack in CREATE_IN_PROGRESS . >> >> Please check the below commands output and advise. >> >> (kolla-openstack) stack at deployment:~$ openstack stack list >> >> +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ >> | ID | Stack Name | >> Project | Stack Status | Creation Time | >> Updated Time | >> >> +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ >> | e9bfad7c-c82e-4fc6-9f05-793b31631761 | k8s-cluster2-zsgfkrobjlw6 | >> 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-15T08:01:50Z | >> None | >> | 9dde1665-8e50-4482-a986-810eb2a9cbaa | k8s-cluster1-eogcos7ee3ub | >> 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-10T07:27:41Z | >> None | >> >> +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ >> (kolla-openstack) stack at deployment:~$ openstack coe cluster list >> >> +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ >> | uuid | name | keypair | >> node_count | master_count | status | health_status | >> >> +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ >> | 277e0047-4ae1-4006-869e-a824cc602bcc | k8s-cluster1 | Newkey | >> 2 | 1 | CREATE_COMPLETE | HEALTHY | >> | 4336f9d9-9bea-477f-bc13-c16ff0f7e141 | k8s-cluster2 | Newkey | >> 2 | 1 | CREATE_IN_PROGRESS | None | >> >> +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ >> Regards >> >> Tony Karera >> >> >> -- >> Cheers & Best regards, >> ------------------------------------------------------------------------------ >> Feilong Wang (???) (he/him) >> Head of Research & Development >> >> Catalyst Cloud >> Aotearoa's own >> >> Mob: +64 21 0832 6348 | www.catalystcloud.nz >> Level 6, 150 Willis Street, Wellington 6011, New Zealand >> >> CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. >> It may contain privileged, confidential or copyright information. If you are >> not the named recipient, any use, reliance upon, disclosure or copying of this >> email or its attachments is unauthorised. If you have received this email in >> error, please reply via email or call +64 21 0832 6348. >> ------------------------------------------------------------------------------ >> >> -- Regards, Syed Ammad Ali -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Thu Sep 16 07:13:53 2021 From: katonalala at gmail.com (Lajos Katona) Date: Thu, 16 Sep 2021 09:13:53 +0200 Subject: Changing the default Network Quotas In-Reply-To: References: Message-ID: Hi Tony, Have You checked the Quota API: https://docs.openstack.org/api-ref/network/v2/index.html?expanded=update-quota-for-a-project-detail ? There's a CLI for it: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/quota.html Regards Lajos (lajoskatona) Karera Tony ezt ?rta (id?pont: 2021. szept. 15., Sze, 14:43): > Hello Team, > > Can anybody advise on how to change the Network Quotas for Openstack > Wallaby deployed using Kolla-ansbile. > > Regards > > Tony Karera > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Thu Sep 16 07:46:49 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 16 Sep 2021 09:46:49 +0200 Subject: Neutron In-Reply-To: References: Message-ID: <5473028.3fjBA3ox3t@p1> Hi, On ?roda, 15 wrze?nia 2021 07:19:20 CEST Zeeshan Haider wrote: > **Scenario**: > We are trying to develop our Virtual Network using `Open Virtual Network` > or `OVN`, We need our Virtual Network Switches running firstly on the root > Virtual Router, and there are some other Virtual Switches as well. I was > Working on the `openvswitch` and `OVN` but a lot of things are missing for > me, and there are some unlinked dots between the deployment itself and > there are some problems. My Question has 3 further Parts. > > 1. Is it possible at all to use `Neutron` as our Network Manager without > OpenStack cloud installation? Yes, it's possible as Akihiro already explained. The one thing You need to remember is that if You want ports to be provisioned by neutron they should be plugged into br-int bridge on the node. In the way as e.g. Nova (os-vif) is doing that actually. > 2. Is it a good approach? TBH I don't know. In theory it should works but in u/s Neutron we are not testing such use case at all so I can't tell You what bugs You may hit with that approach. > 3. if not then are there any cookbooks available for `OVN`, I am not > talking about man-pages. > > Any help would be highly appreciated. > > P.S. I am thinking about this because OVN got merged with Neutron, and it > could be one of the possibilities. To be strict OVN is separate project which comes from the Openvswitch originally and it wasn't never merged with Neutron. Neutron have only mechanism driver which allows using OVN as a backend. That driver was recently merged to be one of the "in-tree" mechanism drivers in Neutron. > > **Note**: Please Don't downvote the question because I don't have that much > experience with the network could be a stupid question, will remove it if > you provide me the info in the comment section. > > > I have found out that installation is possible I am quoting the answer from > `https://bugs.launchpad.net/neutron` by `https://launchpad.net/~amotoki` > below. > > "The way to install neutron is simple: to install neutron with pip (from > PyPI package or the git repo), prepare a config file for neutron and start > the neutron server (in case of OVN mechanism driver). Distro packages > and/or various tools like Ansible playbook are just a wrapper for this. The > minimum required config to enable noauth (ie without keystone) is to set > auth_strategy to noauth in the config file. > > I don't know the detail instruction step by step to satisfy your need." > > > Now I want to find out detailed instructions for this purpose. is this > possible to get the instructions here -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From tonykarera at gmail.com Thu Sep 16 07:53:59 2021 From: tonykarera at gmail.com (Karera Tony) Date: Thu, 16 Sep 2021 09:53:59 +0200 Subject: Changing the default Network Quotas In-Reply-To: References: Message-ID: Hi Lajos, This was very helpful. Regards Tony Karera On Thu, Sep 16, 2021 at 9:14 AM Lajos Katona wrote: > Hi Tony, > > Have You checked the Quota API: > > https://docs.openstack.org/api-ref/network/v2/index.html?expanded=update-quota-for-a-project-detail > ? > > There's a CLI for it: > > https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/quota.html > > Regards > Lajos (lajoskatona) > > Karera Tony ezt ?rta (id?pont: 2021. szept. 15., > Sze, 14:43): > >> Hello Team, >> >> Can anybody advise on how to change the Network Quotas for Openstack >> Wallaby deployed using Kolla-ansbile. >> >> Regards >> >> Tony Karera >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Thu Sep 16 09:58:55 2021 From: smooney at redhat.com (Sean Mooney) Date: Thu, 16 Sep 2021 10:58:55 +0100 Subject: [nova]The code about BP Allow migrating PMEM's dat In-Reply-To: <81acb2be96d242279e1a5f0122ed1986@inspur.com> References: <81acb2be96d242279e1a5f0122ed1986@inspur.com> Message-ID: On Thu, 2021-09-16 at 01:12 +0000, Juntingqiu Qiujunting (???) wrote: > Hi all: > > > > Is there anything in the code logic that needs to be modified? If there is no problem with the implementation structure of the code, I start adding unit tests. > > https://review.opendev.org/c/openstack/nova/+/802225 the code is still not implementing the desing in the spec. https://specs.openstack.org/openstack/nova-specs/specs/xena/approved/allow-migrate-pmem-data.html#workflow specificaly it states prep_resize validates that there is a pmem device free and claims it as part of the move_claim process.The claimed device is stored in the instance.migration_context which can be retrieved later. you are not doing that and instead chanign the rpc protocoal to pass the device paths as addtional parmaters to resize_instance https://review.opendev.org/c/openstack/nova/+/802225/5/nova/compute/manager.py#5203 which requires you to change the compute api https://review.opendev.org/c/openstack/nova/+/802225/5/nova/compute/api.py addtionally the spec need to be reporposed for yoga. i have done a quick code review again but in addtion to unit test ideally you would add docs updates, a release note and functional tests. unit test are our bare minium but functional tests are prefered for move opertions. > > > > Thank you Fossen. > > > > --------------------------------- > > Fossen Qiu | ??? > > > > > > CBRD | ?????????? > > > > > > T: 18249256272 > > > > > > E: qiujunting at inspur.com > > > > > > > > > > > > > ???? > > > > > From fungi at yuggoth.org Thu Sep 16 12:44:41 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 16 Sep 2021 12:44:41 +0000 Subject: Changing Default Network Quotas in Openstack In-Reply-To: References: Message-ID: <20210916124441.dlgw43pyfu4mku7d@yuggoth.org> On 2021-09-16 07:00:27 +0200 (+0200), Karera Tony wrote: > I have a working Wallaby Openstack deployed using kolla-ansible. > > However, I have not been able to find a way of changing the > Network quotas as I need to increase the defaults. > > Can someone kindly advise on how to go about it ? Are you looking for something other than what's already explained in the Neutron documentation? https://docs.openstack.org/neutron/latest/admin/ops-quotas.html -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From katonalala at gmail.com Thu Sep 16 13:11:36 2021 From: katonalala at gmail.com (Lajos Katona) Date: Thu, 16 Sep 2021 15:11:36 +0200 Subject: [neutron][networking-midonet] Propose to EOL stable/pike, stable/queens and stable/rocky branches of networking-midonet Message-ID: Hi, The older than rocky branches of networking-midonet are broken, as it seems that the last merge on stable/rocky and stable/queens happened a year ago, and even the master branch of networking-midonet is not maintained. Considering the above I propose to make stable/rocky, stable/queens, stable/pike branches End Of Life. I will wait until the end of next week for anyone who would like to maybe step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in networking-midonet. Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosmaita.fossdev at gmail.com Thu Sep 16 13:20:30 2021 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Thu, 16 Sep 2021 09:20:30 -0400 Subject: [cinder] festival of XS reviews 17 September 2021 Message-ID: <45aa7165-8dae-deea-2c73-86322c62ae27@gmail.com> Hello Cinder community members, This is a reminder that the most recent edition of the Cinder Festival of XS Reviews will be held at the end of this week (tomorrow!) on Friday 17 September. Since we'd like to keep the master branch clean in case any release-critical bugs are discovered that will need to be backported to stable/xena, I propose that we focus on drivers reviews tomorrow, which will not impact the main cinder code. who: Everyone! what: The Cinder Festival of XS Reviews when: Friday 17 September 2021 from 1400-1600 UTC where: https://meetpad.opendev.org/cinder-festival-of-reviews This recurring meeting can be placed on your calendar by using this handy ICS file: http://eavesdrop.openstack.org/calendars/cinder-festival-of-reviews.ics See you there! brian From maryx.camp at intel.com Thu Sep 16 13:41:58 2021 From: maryx.camp at intel.com (Camp, MaryX) Date: Thu, 16 Sep 2021 13:41:58 +0000 Subject: [oslo][docs] Request to release openstackdocstheme Message-ID: Hi to the Oslo team, Can you please release the openstackdocstheme so it can be used by the StarlingX community? This will correct the double headings issue we see now on every doc page. We've made similar historical release requests for 2.2.4 [1], 2.2.5 [2], and 2.2.7 [3] The StarlingX docs site utilizes a theme variant within the openstackdocstheme repo, but consumes packaged releases. Several recent changes merged to address Sphinx 4.x changes, and the StarlingX community would appreciate a new openstackdocstheme release to make use of them. [1]? 2.2.4? https://review.opendev.org/c/openstack/releases/+/737105 [2] ?2.2.5? https://review.opendev.org/c/openstack/releases/+/741550 [3]? 2.2.7? https://review.opendev.org/c/openstack/releases/+/765192 Thanks in advance! Mary Camp StarlingX Documentation Kelly Services Technical Writer | mailto:maryx.camp at intel.com From hberaud at redhat.com Thu Sep 16 14:00:19 2021 From: hberaud at redhat.com (Herve Beraud) Date: Thu, 16 Sep 2021 16:00:19 +0200 Subject: [oslo][docs] Request to release openstackdocstheme In-Reply-To: References: Message-ID: Hello Mary, Done => https://review.opendev.org/c/openstack/releases/+/809419 Cheers Le jeu. 16 sept. 2021 ? 15:43, Camp, MaryX a ?crit : > Hi to the Oslo team, > Can you please release the openstackdocstheme so it can be used by the > StarlingX community? This will correct the double headings issue we see now > on every doc page. > We've made similar historical release requests for 2.2.4 [1], 2.2.5 [2], > and 2.2.7 [3] > > The StarlingX docs site utilizes a theme variant within the > openstackdocstheme repo, but consumes packaged releases. > Several recent changes merged to address Sphinx 4.x changes, and the > StarlingX community would appreciate a new openstackdocstheme release to > make use of them. > > [1] 2.2.4 https://review.opendev.org/c/openstack/releases/+/737105 > [2] 2.2.5 https://review.opendev.org/c/openstack/releases/+/741550 > [3] 2.2.7 https://review.opendev.org/c/openstack/releases/+/765192 > > Thanks in advance! > Mary Camp > StarlingX Documentation > Kelly Services Technical Writer | mailto:maryx.camp at intel.com > > > -- Herv? Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Thu Sep 16 14:11:17 2021 From: hberaud at redhat.com (Herve Beraud) Date: Thu, 16 Sep 2021 16:11:17 +0200 Subject: [oslo] next meeting canceled Message-ID: Hey folks, Our next meeting (September 20) will be canceled. I'll be away from the keyboard all the entire day, hence, I couldn't drive our meeting. According to our schedule our next meeting will be on October the 4th. Thanks for your understanding. See you soon, -- Herv? Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From akekane at redhat.com Thu Sep 16 14:45:32 2021 From: akekane at redhat.com (Abhishek Kekane) Date: Thu, 16 Sep 2021 20:15:32 +0530 Subject: [glance] No meeting on 23rd September Message-ID: Hi All, We decided to cancel our next (September 23rd) weekly meeting. According to schedule we will meet directly on September 30th. In case of any queries, reach us on #openstack-glance IRC channel. Thank you, Abhishek -------------- next part -------------- An HTML attachment was scrubbed... URL: From derekokeeffe85 at yahoo.ie Thu Sep 16 14:56:20 2021 From: derekokeeffe85 at yahoo.ie (Derek O keeffe) Date: Thu, 16 Sep 2021 14:56:20 +0000 (UTC) Subject: Delete Volume on Instance Delete References: <897920775.952758.1631804180386.ref@mail.yahoo.com> Message-ID: <897920775.952758.1631804180386@mail.yahoo.com> Hi all, I was able to set "Create new volume" to no by default on the launch instance page by adding the following to /etc/openstack-dashboard/local_settings.py LAUNCH_INSTANCE_DEFAULTS ={? ? ? ?'create_volume': False,} Is there anyway I can set "Delete Volume on Instance Delete" to yes by default should someone select to create a new volume when launching an instance. I tried the following but no luck LAUNCH_INSTANCE_DEFAULTS ={? ? ? ?'create_volume': False,? ? ? ?'delete_on_termination': True} Thanks for any help/info in advance. Regards,Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashlee at openstack.org Thu Sep 16 15:36:56 2021 From: ashlee at openstack.org (Ashlee Ferguson) Date: Thu, 16 Sep 2021 10:36:56 -0500 Subject: [PTG] Last call to sign up your team! Message-ID: <9C44A048-9B0E-49A4-A7B6-D7EF63A71DE9@openstack.org> Hi everyone, The next virtual PTG (Project Teams Gathering)[1] is taking place October 18-22, 2021! This is your final chance to signup your team. To do so you must complete BOTH the survey[2] AND reserve time in the ethercalc[3] by end of day Friday, September 17. Then *register* so you don?t miss important updates: https://openinfra-ptg.eventbrite.com/ Teams that have already signed up for PTG include: - Airship - Barbican - Blazar - Cinder - CloudKitty - Cyborg - Designate - Diversity and Inclusion WG - First Contact SIG - Glance - Heat - Horizon - Interop/Refstack - Ironic - Kata-Containers - Keystone - Kolla - Kuryr - Magma - Magnum - Manila - Masakari - Multi-arch SIG - Neutron - Nova - Octavia - OpenDev - OpenInfra Edge Computing Group - Openstack-Ansible - OpenStack Ansible Modules - OpenStack Charms - OpenStack-Helm - OpenStack Telemetry - QA - Scientific SIG - SDK/CLI - Security SIG - Skyline - StarlingX - Swift - Tacker - Technical Committee - TripleO - Venus We look forward to seeing you at PTG! If you have any questions, please email ptg at openstack.org . Thanks! Ashlee [1] PTG Registration: https://www.openstack.org/ptg/ [2] Team Survey: https://openinfrafoundation.formstack.com/forms/oct2021_vptg_survey [3] Ethercalc Signup: https://ethercalc.openstack.org/8tum5yl1bx43 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennelson11 at gmail.com Thu Sep 16 15:42:03 2021 From: kennelson11 at gmail.com (Kendall Nelson) Date: Thu, 16 Sep 2021 08:42:03 -0700 Subject: [cinder] festival of XS reviews 17 September 2021 In-Reply-To: <45aa7165-8dae-deea-2c73-86322c62ae27@gmail.com> References: <45aa7165-8dae-deea-2c73-86322c62ae27@gmail.com> Message-ID: I just have to say I love this. -Kendall (diablo_rojo) On Thu, Sep 16, 2021 at 6:21 AM Brian Rosmaita wrote: > Hello Cinder community members, > > This is a reminder that the most recent edition of the Cinder Festival > of XS Reviews will be held at the end of this week (tomorrow!) on Friday > 17 September. > > Since we'd like to keep the master branch clean in case any > release-critical bugs are discovered that will need to be backported to > stable/xena, I propose that we focus on drivers reviews tomorrow, which > will not impact the main cinder code. > > who: Everyone! > what: The Cinder Festival of XS Reviews > when: Friday 17 September 2021 from 1400-1600 UTC > where: https://meetpad.opendev.org/cinder-festival-of-reviews > > This recurring meeting can be placed on your calendar by using this > handy ICS file: > http://eavesdrop.openstack.org/calendars/cinder-festival-of-reviews.ics > > > See you there! > brian > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rguyett at datto.com Thu Sep 16 13:30:52 2021 From: rguyett at datto.com (Reid Guyett) Date: Thu, 16 Sep 2021 09:30:52 -0400 Subject: [Swift] Rebalancing EC question Message-ID: Hello, We were working on expanding one of our clusters (Ussuri on Ubuntu 18.04) and are wondering about the rebalance behavior of swift-ring-builder. When we run it in debug mode on a 15/4 EC ring, we see this message about "Unable to finish rebalance plan after 2 attempts" and are seeing 100% partitions reassigned. DEBUG: Placed 10899/2 onto dev r1z3-10.40.48.72/d10 DEBUG: Placed 2183/3 onto dev r1z5-10.40.48.76/d11 DEBUG: Placed 1607/1 onto dev r1z3-10.40.48.70/d28 DEBUG: Assigned 32768 parts DEBUG: Gather start is 10278 (Last start was 25464) DEBUG: Unable to finish rebalance plan after 2 attempts Reassigned 32768 (100.00%) partitions. Balance is now 63.21. Dispersion is now 0.00 ------------------------------------------------------------------------------- NOTE: Balance of 63.21 indicates you should push this ring, wait at least 1 hours, and rebalance/repush. ------------------------------------------------------------------------------- Moving 100% seems scary, what does that mean in this situation? Is this message because 1 fragment from every partition is moved and that is the most that it can do per rebalance because they are technically the same partition? When we compare the swift-ring-builder output (partitions per device) between rebalances we can see some partitions move each time until we no longer see the push/wait/rebalance message again. So it's not really moving 100% partitions. Reid From clay.gerrard at gmail.com Thu Sep 16 18:02:23 2021 From: clay.gerrard at gmail.com (Clay Gerrard) Date: Thu, 16 Sep 2021 13:02:23 -0500 Subject: [Swift] Rebalancing EC question In-Reply-To: References: Message-ID: Not scary! Because you have a 15/4 EC policy, we say each partition has 19 "replicas". And since rebalance will only move one "replica" of any partition max at each rebalance: up to 100% of your partitions may have at least one replica assignment move. That means, after you push out this ring, 100% of your object GET requests will experience at most one "replica" is out of place. But that's ok! In a 15/4 you only need 15 EC fragments to respond successfully and you have 18 total fragments that did NOT get reassigned. It's unfortunate the language is a little ambiguous, but it is talking about % of *partitions* that had a replica moved. Since each object resides in single a partition - the % of partitions affected most directly communicates the % of client objects affected by the rebalance. We do NOT display % of *partition-replicas* moved because while the number would be smaller - it could never be 100% because of the restriction that only one "replica" may move. When doing a large topology change - particularly with EC - it may be the case that more than one replica of each part will need to move (imagine doubling your capacity into a second zone on a 8+4 ring) - so it'll take a few cranks. Eventually you'll want to have moved 6 replicas of each part (6 in z1 and 6 in z2), but if we allowed you to move six replicas of 100% of your parts you'd only have 6/8 required parts to service reads! Protip: when you push out the new ring you can turn on handoffs_only mode for the reconstructor for a little while to get things rebalanced MUCH more quickly - just don't forget to turn it off! (sending second time because I forgot to reply all to the list) On Thu, Sep 16, 2021 at 11:35 AM Reid Guyett wrote: > Hello, > > We were working on expanding one of our clusters (Ussuri on Ubuntu > 18.04) and are wondering about the rebalance behavior of > swift-ring-builder. When we run it in debug mode on a 15/4 EC ring, we > see this message about "Unable to finish rebalance plan after 2 > attempts" and are seeing 100% partitions reassigned. > > DEBUG: Placed 10899/2 onto dev r1z3-10.40.48.72/d10 > DEBUG: Placed 2183/3 onto dev r1z5-10.40.48.76/d11 > DEBUG: Placed 1607/1 onto dev r1z3-10.40.48.70/d28 > DEBUG: Assigned 32768 parts > DEBUG: Gather start is 10278 (Last start was 25464) > DEBUG: Unable to finish rebalance plan after 2 attempts > Reassigned 32768 (100.00%) partitions. Balance is now 63.21. > Dispersion is now 0.00 > > ------------------------------------------------------------------------------- > NOTE: Balance of 63.21 indicates you should push this > ring, wait at least 1 hours, and rebalance/repush. > > ------------------------------------------------------------------------------- > > Moving 100% seems scary, what does that mean in this situation? Is > this message because 1 fragment from every partition is moved and that > is the most that it can do per rebalance because they are technically > the same partition? > When we compare the swift-ring-builder output (partitions per device) > between rebalances we can see some partitions move each time until we > no longer see the push/wait/rebalance message again. So it's not > really moving 100% partitions. > > Reid > > > -- Clay Gerrard -------------- next part -------------- An HTML attachment was scrubbed... URL: From jasonanderson at uchicago.edu Thu Sep 16 22:58:21 2021 From: jasonanderson at uchicago.edu (Jason Anderson) Date: Thu, 16 Sep 2021 22:58:21 +0000 Subject: [blazar] Third-party resource type plugins Message-ID: <9BEF73B0-2E8F-4EBB-B6D1-3C9DEB77BC2D@uchicago.edu> Hi all, I?ve just submitted a spec for making resources in Blazar follow a more explicit plugin architecture: https://review.opendev.org/c/openstack/blazar-specs/+/809485 The intent is to allow out-of-tree resource types, to allow deployers/operators to define their own abstractions. This could support some pretty wide-ranging implementations, like reserving bandwidth allocations on switches or perhaps access to other controlled/limited resources. I hope that this change will also enable Blazar to be adopted by a wider set of users and use cases. Please, have a look?it is currently proposed under Xena because there is not yet a Yoga spec section in blazar-specs. Cheers, /Jason From openinfradn at gmail.com Fri Sep 17 05:25:27 2021 From: openinfradn at gmail.com (open infra) Date: Fri, 17 Sep 2021 10:55:27 +0530 Subject: Autoscaling based on requests Message-ID: Hi, I have a requirement to allocate a server(VM) for each user request and make this automatic. Is it possible with Openstack autoscale? Scaling should not be done based on either cpu or memory usage as only one user is supposed to access a single VM. Regards, Danishka -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Fri Sep 17 06:13:37 2021 From: katonalala at gmail.com (Lajos Katona) Date: Fri, 17 Sep 2021 08:13:37 +0200 Subject: [neutron] Drivers meeting - Friday 17.09.2021 - cancelled Message-ID: Hi, Due to lack of topics for Today's Drivers meeting, let's cancel the meeting. Lajos (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingjisun at vmware.com Fri Sep 17 07:42:24 2021 From: yingjisun at vmware.com (Yingji Sun) Date: Fri, 17 Sep 2021 07:42:24 +0000 Subject: How to resume/abort a migration in error state Message-ID: Buddies, I am in openstack train and I have some failure live-migration cases that leave some migration states to error. These failure cases may block the following migration request in ?queued? state. Is it possible that I force to complete the failure cases ? live_migrate_force_complete only works for a ?running?migration and live_migrate_abort works only for ?running/queued/preparing?migration. Any suggestion would be appreciated. Yingji. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Fri Sep 17 08:32:10 2021 From: eblock at nde.ag (Eugen Block) Date: Fri, 17 Sep 2021 08:32:10 +0000 Subject: Delete Volume on Instance Delete In-Reply-To: <897920775.952758.1631804180386@mail.yahoo.com> References: <897920775.952758.1631804180386.ref@mail.yahoo.com> <897920775.952758.1631804180386@mail.yahoo.com> Message-ID: <20210917083210.Horde.Wppta4KVIuImVwuTSoVSV8F@webmail.nde.ag> Hi, I have a lab environment here with an older version (Pike), but I believe the workflow has not changed significantly since then (please correct me if I'm wrong). The delete_on_termination flag can be changed in a different file than the create_volume option: ---snip--- root at control:/srv/www/openstack-dashboard/openstack_dashboard/dashboards/project/static/dashboard/project/workflow/launch-instance # diff -u launch-instance-model.service.spec.js.dist launch-instance-model.service.spec.js --- launch-instance-model.service.spec.js.dist 2021-09-17 10:13:18.343019968 +0200 +++ launch-instance-model.service.spec.js 2021-09-17 10:13:33.190524361 +0200 @@ -957,7 +957,7 @@ var expectedBlockDevice = [{ source_type: 'image', destination_type: 'volume', - delete_on_termination: true, + delete_on_termination: false, uuid: 'cirros', boot_index: '0', volume_size: 10 ---snip--- The path to this file is probably different in your case as you write about /etc/openstack-dashboard, so I would search in that path. Changing it in the .js file worked for me here. Regards, Eugen Zitat von Derek O keeffe : > Hi all, > I was able to set "Create new volume" to no by default on the launch > instance page by adding the following to > /etc/openstack-dashboard/local_settings.py > LAUNCH_INSTANCE_DEFAULTS ={? ? ? ?'create_volume': False,} > > Is there anyway I can set "Delete Volume on Instance Delete" to yes > by default should someone select to create a new volume when > launching an instance. I tried the following but no luck > LAUNCH_INSTANCE_DEFAULTS ={? ? ? ?'create_volume': False,? ? ? > ?'delete_on_termination': True} > > Thanks for any help/info in advance. > Regards,Derek From derekokeeffe85 at yahoo.ie Fri Sep 17 08:45:08 2021 From: derekokeeffe85 at yahoo.ie (Derek O keeffe) Date: Fri, 17 Sep 2021 08:45:08 +0000 (UTC) Subject: Delete Volume on Instance Delete In-Reply-To: <20210917083210.Horde.Wppta4KVIuImVwuTSoVSV8F@webmail.nde.ag> References: <897920775.952758.1631804180386.ref@mail.yahoo.com> <897920775.952758.1631804180386@mail.yahoo.com> <20210917083210.Horde.Wppta4KVIuImVwuTSoVSV8F@webmail.nde.ag> Message-ID: <1347239937.1242689.1631868308560@mail.yahoo.com> Hi Eugen, Thanks for that info, I will take a look shortly and let you know if I find it. What I did for the time being was: 'create_volume': False,'hide_create_volume': True Which actually disables the option for the user to create boot volumes altogether. Regards,Derek On Friday 17 September 2021, 09:34:44 IST, Eugen Block wrote: Hi, I have a lab environment here with an older version (Pike), but I? believe the workflow has not changed significantly since then (please? correct me if I'm wrong). The delete_on_termination flag can be? changed in a different file than the create_volume option: ---snip--- root at control:/srv/www/openstack-dashboard/openstack_dashboard/dashboards/project/static/dashboard/project/workflow/launch-instance # diff -u launch-instance-model.service.spec.js.dist? launch-instance-model.service.spec.js --- launch-instance-model.service.spec.js.dist? 2021-09-17? 10:13:18.343019968 +0200 +++ launch-instance-model.service.spec.js? ? ? 2021-09-17? 10:13:33.190524361 +0200 @@ -957,7 +957,7 @@ ? ? ? ? ? ? var expectedBlockDevice = [{ ? ? ? ? ? ? ? source_type: 'image', ? ? ? ? ? ? ? destination_type: 'volume', -? ? ? ? ? ? delete_on_termination: true, +? ? ? ? ? ? delete_on_termination: false, ? ? ? ? ? ? ? uuid: 'cirros', ? ? ? ? ? ? ? boot_index: '0', ? ? ? ? ? ? ? volume_size: 10 ---snip--- The path to this file is probably different in your case as you write? about /etc/openstack-dashboard, so I would search in that path. Changing it in the .js file worked for me here. Regards, Eugen Zitat von Derek O keeffe : > Hi all, > I was able to set "Create new volume" to no by default on the launch? > instance page by adding the following to? > /etc/openstack-dashboard/local_settings.py > LAUNCH_INSTANCE_DEFAULTS ={? ? ? ?'create_volume': False,} > > Is there anyway I can set "Delete Volume on Instance Delete" to yes? > by default should someone select to create a new volume when? > launching an instance. I tried the following but no luck > LAUNCH_INSTANCE_DEFAULTS ={? ? ? ?'create_volume': False,? ? ?? > ?'delete_on_termination': True} > > Thanks for any help/info in advance. > Regards,Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Fri Sep 17 10:41:17 2021 From: hberaud at redhat.com (Herve Beraud) Date: Fri, 17 Sep 2021 12:41:17 +0200 Subject: [release][PTL] broken gates and stuck releases Message-ID: Hello PTLs, For your information our gates are currently broken so we are not able to merge patches and so to release deliverables. indeed openstack/release rely on openstack/governance to validate releases. The governance repo relies on pybot2 which is no longer maintained and became not installable a few hours ago. That broke the release gate and led us to a catch-22 situation where we are not able to release the governance repo to fix our repo. Our current solution is to pull openstack/governance from git source instead of from pypi. By doing that we will pull the fixed version from opendev git and we will unblock our gates. In this way, we will be able to restart to merge patches and release deliverables. We currently wait for some feedback from the infra team to help us to force merging the needed fixes by using their gerrit super admin rights, hence, bypassing the catch-22 situation. This situation highlighted the cross dependency between these two repos (governance and releases) and I don't think that this is something healthy. I propose to keep pulling the governance repo from git source even after we will fix the catch-22 situation. Any opinions? Thanks for your understanding, we will inform you when we will have updates. https://opendev.org/openstack/governance/commit/9128e502253db9aac346ce25726a23378c07d478 https://review.opendev.org/c/openstack/releases/+/809626 -- Herv? Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Fri Sep 17 13:51:54 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 17 Sep 2021 13:51:54 +0000 Subject: [release][PTL] broken gates and stuck releases In-Reply-To: References: Message-ID: <20210917135154.ddbmrojefh5kxjfl@yuggoth.org> On 2021-09-17 12:41:17 +0200 (+0200), Herve Beraud wrote: [...] > Our current solution is to pull openstack/governance from git > source instead of from pypi. By doing that we will pull the fixed > version from opendev git and we will unblock our gates. In this > way, we will be able to restart to merge patches and release > deliverables. > > We currently wait for some feedback from the infra team to help us > to force merging the needed fixes by using their gerrit super > admin rights, hence, bypassing the catch-22 situation. By the time I was around, a solution was found which involved merely adjusting the release project requirements list, so no action was needed outside normal code review workflow. > This situation highlighted the cross dependency between these two > repos (governance and releases) and I don't think that this is > something healthy. I propose to keep pulling the governance repo > from git source even after we will fix the catch-22 situation. Any > opinions? [...] The "proper" way to do this is to declare openstack/governance in the required-projects lists for the jobs which are using it, and then the tox role from zuul-jobs will know to consume it from locally supplied source instead of fetching it from PyPI. This also allows Zuul to handle cross-repository dependencies properly, so a requirements change would be able to successfully Depends-On some proposed change in governance even before it's merged. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From gmann at ghanshyammann.com Fri Sep 17 14:31:37 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 17 Sep 2021 09:31:37 -0500 Subject: [release][PTL] broken gates and stuck releases In-Reply-To: <20210917135154.ddbmrojefh5kxjfl@yuggoth.org> References: <20210917135154.ddbmrojefh5kxjfl@yuggoth.org> Message-ID: <17bf42b82d0.fd6e4953853458.5358036126858462962@ghanshyammann.com> ---- On Fri, 17 Sep 2021 08:51:54 -0500 Jeremy Stanley wrote ---- > On 2021-09-17 12:41:17 +0200 (+0200), Herve Beraud wrote: > [...] > > Our current solution is to pull openstack/governance from git > > source instead of from pypi. By doing that we will pull the fixed > > version from opendev git and we will unblock our gates. In this > > way, we will be able to restart to merge patches and release > > deliverables. > > > > We currently wait for some feedback from the infra team to help us > > to force merging the needed fixes by using their gerrit super > > admin rights, hence, bypassing the catch-22 situation. > > By the time I was around, a solution was found which involved merely > adjusting the release project requirements list, so no action was > needed outside normal code review workflow. > > > This situation highlighted the cross dependency between these two > > repos (governance and releases) and I don't think that this is > > something healthy. I propose to keep pulling the governance repo > > from git source even after we will fix the catch-22 situation. Any > > opinions? > [...] > > The "proper" way to do this is to declare openstack/governance in > the required-projects lists for the jobs which are using it, and > then the tox role from zuul-jobs will know to consume it from > locally supplied source instead of fetching it from PyPI. This also > allows Zuul to handle cross-repository dependencies properly, so a > requirements change would be able to successfully Depends-On some > proposed change in governance even before it's merged. +1, and using data directly from governance repo make more sense than pip. I thought we release governance only for election scripts. can we remove governance release dependencies also, I have not checked those election scripts yet? and after that, we stop releasing governance itself and everyone can use it from the source. -gmann > -- > Jeremy Stanley > From hberaud at redhat.com Fri Sep 17 15:31:53 2021 From: hberaud at redhat.com (Herve Beraud) Date: Fri, 17 Sep 2021 17:31:53 +0200 Subject: [release][PTL] broken gates and stuck releases In-Reply-To: <17bf42b82d0.fd6e4953853458.5358036126858462962@ghanshyammann.com> References: <20210917135154.ddbmrojefh5kxjfl@yuggoth.org> <17bf42b82d0.fd6e4953853458.5358036126858462962@ghanshyammann.com> Message-ID: Just a heads up to inform everyone that the situation is now fixed and our gates are now ok. Deliverables can be now released. Le ven. 17 sept. 2021 ? 16:33, Ghanshyam Mann a ?crit : > ---- On Fri, 17 Sep 2021 08:51:54 -0500 Jeremy Stanley > wrote ---- > > On 2021-09-17 12:41:17 +0200 (+0200), Herve Beraud wrote: > > [...] > > > Our current solution is to pull openstack/governance from git > > > source instead of from pypi. By doing that we will pull the fixed > > > version from opendev git and we will unblock our gates. In this > > > way, we will be able to restart to merge patches and release > > > deliverables. > > > > > > We currently wait for some feedback from the infra team to help us > > > to force merging the needed fixes by using their gerrit super > > > admin rights, hence, bypassing the catch-22 situation. > > > > By the time I was around, a solution was found which involved merely > > adjusting the release project requirements list, so no action was > > needed outside normal code review workflow. > > > > > This situation highlighted the cross dependency between these two > > > repos (governance and releases) and I don't think that this is > > > something healthy. I propose to keep pulling the governance repo > > > from git source even after we will fix the catch-22 situation. Any > > > opinions? > > [...] > > > > The "proper" way to do this is to declare openstack/governance in > > the required-projects lists for the jobs which are using it, and > > then the tox role from zuul-jobs will know to consume it from > > locally supplied source instead of fetching it from PyPI. This also > > allows Zuul to handle cross-repository dependencies properly, so a > > requirements change would be able to successfully Depends-On some > > proposed change in governance even before it's merged. > > +1, and using data directly from governance repo make more sense than pip. > I thought we release governance only for election scripts. can we remove > governance > release dependencies also, I have not checked those election scripts yet? > and after > that, we stop releasing governance itself and everyone can use it from the > source. > > -gmann > > > -- > > Jeremy Stanley > > > > -- Herv? Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From bkslash at poczta.onet.pl Fri Sep 17 12:00:58 2021 From: bkslash at poczta.onet.pl (Adam Tomas) Date: Fri, 17 Sep 2021 14:00:58 +0200 Subject: RabbitMQ annoying disconnections In-Reply-To: References: Message-ID: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> Hi, after some struggling I have almost ?clear? logs (clear=error free :) ) Almost?. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description: https://bugzilla.redhat.com/show_bug.cgi?id=1711794 in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.? but? It is reported :( And produces tons of logs. I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :( [oslo_messaging_rabbit] heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4 How to avoid this disconnections?? Or what to do to avoid writing this ?errors?to logs? Best regards Adam Tomas /var/log/kolla/rabbitmq/rabbitXXX.log 2021-09-17 06:43:08.771 [error] <0.16056.678> closing AMQP connection <0.16056.678> (x.x.x.x00:44882 -> x.x.x.x00:5672 - mod_wsgi:959:aaa12af5-bb0b-456e-a129-a466eccc24f0): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:08.773 [info] <0.7412.679> Closing all channels from connection 'x.x.x.x00:44882 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:09.282 [info] <0.4574.679> accepting AMQP connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) 2021-09-17 06:43:09.283 [info] <0.28410.678> accepting AMQP connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) 2021-09-17 06:43:09.285 [info] <0.28410.678> Connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394 2021-09-17 06:43:09.285 [info] <0.4574.679> Connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93 2021-09-17 06:43:09.286 [info] <0.4574.679> connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672 - magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.286 [info] <0.28410.678> connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672 - magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.504 [error] <0.14465.678> closing AMQP connection <0.14465.678> (x.x.x.x00:44914 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:09.507 [info] <0.14985.679> Closing all channels from connection 'x.x.x.x00:44914 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:09.521 [info] <0.11574.678> accepting AMQP connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) 2021-09-17 06:43:09.523 [info] <0.11574.678> Connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) has a client-provided name: mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3 2021-09-17 06:43:09.525 [info] <0.11574.678> connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.654 [info] <0.31575.679> accepting AMQP connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) 2021-09-17 06:43:09.656 [info] <0.31575.679> Connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7 2021-09-17 06:43:09.656 [info] <0.31180.678> accepting AMQP connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) 2021-09-17 06:43:09.658 [info] <0.31575.679> connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672 - magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.658 [info] <0.31180.678> Connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b 2021-09-17 06:43:09.660 [info] <0.2757.679> accepting AMQP connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) 2021-09-17 06:43:09.660 [info] <0.31180.678> connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672 - magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.662 [info] <0.2757.679> Connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315 2021-09-17 06:43:09.663 [info] <0.2757.679> connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672 - magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.814 [error] <0.22797.678> closing AMQP connection <0.22797.678> (x.x.x.x05:57676 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:09.817 [info] <0.7903.679> Closing all channels from connection 'x.x.x.x05:57676 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:10.319 [error] <0.18268.678> closing AMQP connection <0.18268.678> (x.x.x.x00:44924 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:10.322 [info] <0.28032.678> Closing all channels from connection 'x.x.x.x00:44924 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:10.330 [info] <0.19596.678> accepting AMQP connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) 2021-09-17 06:43:10.333 [info] <0.19596.678> Connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) has a client-provided name: nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09 2021-09-17 06:43:10.334 [info] <0.19596.678> connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:10.831 [info] <0.9838.678> accepting AMQP connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) 2021-09-17 06:43:10.834 [info] <0.9838.678> Connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) has a client-provided name: neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e 2021-09-17 06:43:10.835 [info] <0.9838.678> connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:11.738 [error] <0.15275.678> closing AMQP connection <0.15275.678> (x.x.x.x01:53414 -> x.x.x.x00:5672 - mod_wsgi:968:b60f723c-882e-4780-a162-55c03ebe6179): missed heartbeats from client, timeout: 60s /var/log/kolla/glance/glance-api.log 2021-09-17 13:17:23.212 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:17:29.208 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:17:42.190 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:17:42] "GET / HTTP/1.1" 300 1446 0.001949 2021-09-17 13:17:46.351 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:12.336 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:12] "GET / HTTP/1.1" 300 1446 0.003754 2021-09-17 13:18:13.703 956 INFO eventlet.wsgi.server [req-1a00db34-349b-4ed3-8625-edfadc939be4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:13] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.139229 2021-09-17 13:18:33.065 955 INFO eventlet.wsgi.server [req-a57046eb-549e-4837-95c7-d0994c05449e d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.096302 2021-09-17 13:18:33.077 955 INFO eventlet.wsgi.server [req-3a8af281-6732-4afd-b3ec-e502367bce51 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.003982 2021-09-17 13:18:36.747 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:36.759 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:42.460 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:42] "GET / HTTP/1.1" 300 1446 0.001383 2021-09-17 13:18:52.659 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:52.670 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:12.598 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:19:12] "GET / HTTP/1.1" 300 1446 0.001590 2021-09-17 13:19:16.367 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:27.127 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:27.184 955 INFO eventlet.wsgi.server [req-8f68efc9-f46a-450d-8138-928e4f2585b4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:19:27] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.032917 /var/log/kolla/keystone/keystone.log 2021-09-17 13:13:54.670 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.148 960 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.150 962 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.345 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:56.044 961 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:18.402 966 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:31.372 963 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:36.580 967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:38.771 965 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:16:12.226 964 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer /var/log/kolla/nova/nova-compute.log 2021-09-17 13:10:07.710 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56226. 2021-09-17 13:11:11.473 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:11:34.046 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:11:34.721 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:07.699 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:13:08.734 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56434. 2021-09-17 13:14:11.501 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:34.070 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:34.745 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:16:08.724 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:16:09.756 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56638. 2021-09-17 13:17:34.098 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:09.747 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:19:10.781 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56834. 2021-09-17 13:20:34.124 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:20:34.777 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:20:51.789 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:22:10.772 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:22:11.806 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57038. 2021-09-17 13:23:34.149 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:23:34.803 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:23:51.818 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:25:11.795 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:25:12.842 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57240. 2021-09-17 13:26:34.176 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:26:34.830 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:28:12.812 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:28:13.848 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57446. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Fri Sep 17 16:35:13 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 17 Sep 2021 11:35:13 -0500 Subject: [all][tc] What's happening in Technical Committee: summary 17th Sept, 21: Reading: 5 min Message-ID: <17bf49cad63.10b0435a2860827.3145744043532592549@ghanshyammann.com> Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC this week IRC meeting held on Sept 16th Thursday. * Most of the meeting discussions are summarized in the below sections (Completed or in-progress activities section). To know more details, you can check the complete logs @ https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-16-15.00.log.html * We will have next week's meeting on Sept 23rd, Thursday 15:00 UTC on IRC, feel free the topic in agenda [1] by Sept 22nd. 2. What we completed this week: ========================= * PTLs for all the six leaderless projects are now assigned and merged[2] * Added openstack-loadbalancer charm and interfaces[3] * TC chair and vice-chair for Yoga cycle are selected. I will continue as chair[3] and yoctozepto as vice-chair[4]. 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ TC is using the etherpad[6] for Xena cycle working item. We will be checking and updating the status biweekly in the same etherpad. Open Reviews ----------------- * Three open reviews for ongoing activities[7]. TC tags analysis ------------------- * As discussed in the last PTG, TC is working on an analysis of the usefulness of the Tags framework[8] or what all tags can be cleaned up. * It seems no response yet on email from yoctozepto[9], if you are an operator, please respond to the email and based on the feedback we will continue the discussion in PTG. Collecting Pain points from projects/SIGs/pop-up ----------------------------------------------------------- * No updates on this, we will continue discussing it in PTG[10], join us if you are interested to help/discuss this topic. Project updates ------------------- * Add the cinder-netapp charm to Openstack charms[11] * Retiring js-openstack-lib [12] * Retire puppet-freezer[13] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [14]. * Current status: "Secure RBAC" is selected for Yoga cycle[15]. PTG planning ---------------- * We are collecting the PTG topics in etherpad[16], feel free to add any topic you would like to discuss. * We discussed the live stream of one of the TC PTG sessions like we did last time. Once we will have more topics in etherpad then we can select the appropriate one. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[17], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[18]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [19] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [20] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://etherpad.opendev.org/p/yoga-leaderless [3] https://review.opendev.org/c/openstack/governance/+/807837 [4] https://review.opendev.org/c/openstack/governance/+/807163 [5] https://review.opendev.org/c/openstack/governance/+/807178 [6] https://etherpad.opendev.org/p/tc-xena-tracker [7] https://review.opendev.org/q/project:openstack/governance+status:open [8] https://governance.openstack.org/tc/reference/tags/index.html [9] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024804.html [10] https://etherpad.opendev.org/p/pain-point-elimination [11] https://review.opendev.org/c/openstack/governance/+/809011 [12] https://review.opendev.org/c/openstack/governance/+/798540 [13] https://review.opendev.org/c/openstack/governance/+/807163 [14] https://etherpad.opendev.org/p/y-series-goals [15] https://review.opendev.org/c/openstack/governance/+/803783 [16] https://etherpad.opendev.org/p/tc-yoga-ptg [17] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [18] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [19] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [20] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann From DHilsbos at performair.com Fri Sep 17 16:38:06 2021 From: DHilsbos at performair.com (DHilsbos at performair.com) Date: Fri, 17 Sep 2021 16:38:06 +0000 Subject: RabbitMQ annoying disconnections In-Reply-To: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> References: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> Message-ID: <0670B960225633449A24709C291A525251CA0CCB@COM03.performair.local> Adam; If I'm reading this correctly; Rabbit is timing out, but you're increasing the heartbeat period of OpenStack. This would make the issue worse, wouldn't it? It seems to me that you would want to lower the heartbeat interval of OpenStack, and raise the timeout of Rabbit. That said; it looks like you're using Kola, and I know nothing about Kola. Thank you, Dominic L. Hilsbos, MBA Vice President ? Information Technology Perform Air International Inc. DHilsbos at PerformAir.com www.PerformAir.com From: Adam Tomas [mailto:bkslash at poczta.onet.pl] Sent: Friday, September 17, 2021 5:01 AM To: openstack-discuss Subject: RabbitMQ annoying disconnections Hi, after some struggling I have almost ?clear? logs (clear=error free :) ) Almost?. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description: https://bugzilla.redhat.com/show_bug.cgi?id=1711794 in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.? but? It is reported :( And produces tons of logs. I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :( [oslo_messaging_rabbit] heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4 How to avoid this disconnections?? Or what to do to avoid writing this ?errors?to logs? Best regards Adam Tomas /var/log/kolla/rabbitmq/rabbitXXX.log 2021-09-17 06:43:08.771 [error] <0.16056.678> closing AMQP connection <0.16056.678> (x.x.x.x00:44882 -> x.x.x.x00:5672 - mod_wsgi:959:aaa12af5-bb0b-456e-a129-a466eccc24f0): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:08.773 [info] <0.7412.679> Closing all channels from connection 'x.x.x.x00:44882 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:09.282 [info] <0.4574.679> accepting AMQP connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) 2021-09-17 06:43:09.283 [info] <0.28410.678> accepting AMQP connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) 2021-09-17 06:43:09.285 [info] <0.28410.678> Connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394 2021-09-17 06:43:09.285 [info] <0.4574.679> Connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93 2021-09-17 06:43:09.286 [info] <0.4574.679> connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672 - magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.286 [info] <0.28410.678> connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672 - magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.504 [error] <0.14465.678> closing AMQP connection <0.14465.678> (x.x.x.x00:44914 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:09.507 [info] <0.14985.679> Closing all channels from connection 'x.x.x.x00:44914 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:09.521 [info] <0.11574.678> accepting AMQP connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) 2021-09-17 06:43:09.523 [info] <0.11574.678> Connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) has a client-provided name: mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3 2021-09-17 06:43:09.525 [info] <0.11574.678> connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.654 [info] <0.31575.679> accepting AMQP connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) 2021-09-17 06:43:09.656 [info] <0.31575.679> Connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7 2021-09-17 06:43:09.656 [info] <0.31180.678> accepting AMQP connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) 2021-09-17 06:43:09.658 [info] <0.31575.679> connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672 - magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.658 [info] <0.31180.678> Connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b 2021-09-17 06:43:09.660 [info] <0.2757.679> accepting AMQP connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) 2021-09-17 06:43:09.660 [info] <0.31180.678> connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672 - magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.662 [info] <0.2757.679> Connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315 2021-09-17 06:43:09.663 [info] <0.2757.679> connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672 - magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.814 [error] <0.22797.678> closing AMQP connection <0.22797.678> (x.x.x.x05:57676 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:09.817 [info] <0.7903.679> Closing all channels from connection 'x.x.x.x05:57676 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:10.319 [error] <0.18268.678> closing AMQP connection <0.18268.678> (x.x.x.x00:44924 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:10.322 [info] <0.28032.678> Closing all channels from connection 'x.x.x.x00:44924 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:10.330 [info] <0.19596.678> accepting AMQP connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) 2021-09-17 06:43:10.333 [info] <0.19596.678> Connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) has a client-provided name: nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09 2021-09-17 06:43:10.334 [info] <0.19596.678> connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:10.831 [info] <0.9838.678> accepting AMQP connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) 2021-09-17 06:43:10.834 [info] <0.9838.678> Connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) has a client-provided name: neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e 2021-09-17 06:43:10.835 [info] <0.9838.678> connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:11.738 [error] <0.15275.678> closing AMQP connection <0.15275.678> (x.x.x.x01:53414 -> x.x.x.x00:5672 - mod_wsgi:968:b60f723c-882e-4780-a162-55c03ebe6179): missed heartbeats from client, timeout: 60s /var/log/kolla/glance/glance-api.log 2021-09-17 13:17:23.212 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:17:29.208 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:17:42.190 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:17:42] "GET / HTTP/1.1" 300 1446 0.001949 2021-09-17 13:17:46.351 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:12.336 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:12] "GET / HTTP/1.1" 300 1446 0.003754 2021-09-17 13:18:13.703 956 INFO eventlet.wsgi.server [req-1a00db34-349b-4ed3-8625-edfadc939be4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:13] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.139229 2021-09-17 13:18:33.065 955 INFO eventlet.wsgi.server [req-a57046eb-549e-4837-95c7-d0994c05449e d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.096302 2021-09-17 13:18:33.077 955 INFO eventlet.wsgi.server [req-3a8af281-6732-4afd-b3ec-e502367bce51 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.003982 2021-09-17 13:18:36.747 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:36.759 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:42.460 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:42] "GET / HTTP/1.1" 300 1446 0.001383 2021-09-17 13:18:52.659 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:52.670 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:12.598 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:19:12] "GET / HTTP/1.1" 300 1446 0.001590 2021-09-17 13:19:16.367 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:27.127 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:27.184 955 INFO eventlet.wsgi.server [req-8f68efc9-f46a-450d-8138-928e4f2585b4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:19:27] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.032917 /var/log/kolla/keystone/keystone.log 2021-09-17 13:13:54.670 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.148 960 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.150 962 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.345 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:56.044 961 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:18.402 966 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:31.372 963 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:36.580 967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:38.771 965 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:16:12.226 964 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer /var/log/kolla/nova/nova-compute.log 2021-09-17 13:10:07.710 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56226. 2021-09-17 13:11:11.473 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:11:34.046 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:11:34.721 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:07.699 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:13:08.734 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56434. 2021-09-17 13:14:11.501 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:34.070 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:34.745 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:16:08.724 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:16:09.756 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56638. 2021-09-17 13:17:34.098 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:09.747 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:19:10.781 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56834. 2021-09-17 13:20:34.124 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:20:34.777 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:20:51.789 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:22:10.772 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:22:11.806 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57038. 2021-09-17 13:23:34.149 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:23:34.803 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:23:51.818 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:25:11.795 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:25:12.842 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57240. 2021-09-17 13:26:34.176 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:26:34.830 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:28:12.812 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:28:13.848 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57446. From openstack at nemebean.com Fri Sep 17 16:57:28 2021 From: openstack at nemebean.com (Ben Nemec) Date: Fri, 17 Sep 2021 11:57:28 -0500 Subject: RabbitMQ annoying disconnections In-Reply-To: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> References: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> Message-ID: <6eb72199-d368-3e8a-a6e7-1c0e8523bc0e@nemebean.com> This sounds like you're hitting the heartbeat/eventlet bug. Basically eventlet won't schedule the heartbeat thread unless something is regularly waking up the api threads. Depending on what release you're running, you may be able to set heartbeat_in_pthread[0] to fix this. 0: https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/_drivers/impl_rabbit.py#L90 On 9/17/21 7:00 AM, Adam Tomas wrote: > Hi, > after some struggling I have almost ?clear? logs (clear=error free :) ) > Almost?. RabbitMQ keeps disconnecting sessions and there is a huge > amount of disconnect errors in all logs (see below). I found this bug > description: > > https://bugzilla.redhat.com/show_bug.cgi?id=1711794 > > > in which we can read as follows: "this is a recoverable issue that is > already handled by how oslo.messaging is designed. disconnection is not > an error and should not be reported as such in the logs.? > > but? It is reported :( And produces tons of logs. > > I tried to modify heartbeat values - helped a little bit, but I had to > increase [database] max_pool_size = 5 and that of course multiplied > number of disconnection errors by 5 :( > > [oslo_messaging_rabbit] > heartbeat_timeout_threshold = 720 heartbeat_interval = 360 > heartbeat_rate = 4 > > How to avoid this disconnections?? Or what to do to avoid writing this > ?errors?to logs? > > Best regards > Adam Tomas > > /var/log/kolla/rabbitmq/rabbitXXX.log > > 2021-09-17 06:43:08.771 [error] <0.16056.678> closing AMQP connection > <0.16056.678> (x.x.x.x00:44882 -> x.x.x.x00:5672 - > mod_wsgi:959:aaa12af5-bb0b-456e-a129-a466eccc24f0): > missed heartbeats from client, timeout: 60s > 2021-09-17 06:43:08.773 [info] <0.7412.679> Closing all channels from > connection 'x.x.x.x00:44882 -> x.x.x.x00:5672' because it has been closed > 2021-09-17 06:43:09.282 [info] <0.4574.679> accepting AMQP connection > <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) > 2021-09-17 06:43:09.283 [info] <0.28410.678> accepting AMQP connection > <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) > 2021-09-17 06:43:09.285 [info] <0.28410.678> Connection <0.28410.678> > (x.x.x.x11:58172 -> x.x.x.x00:5672) has a client-provided name: > magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394 > 2021-09-17 06:43:09.285 [info] <0.4574.679> Connection <0.4574.679> > (x.x.x.x11:58170 -> x.x.x.x00:5672) has a client-provided name: > magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93 > 2021-09-17 06:43:09.286 [info] <0.4574.679> connection <0.4574.679> > (x.x.x.x11:58170 -> x.x.x.x00:5672 - > magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93): user > 'openstack' authenticated and granted access to vhost '/' > 2021-09-17 06:43:09.286 [info] <0.28410.678> connection <0.28410.678> > (x.x.x.x11:58172 -> x.x.x.x00:5672 - > magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394): user > 'openstack' authenticated and granted access to vhost '/' > 2021-09-17 06:43:09.504 [error] <0.14465.678> closing AMQP connection > <0.14465.678> (x.x.x.x00:44914 -> x.x.x.x00:5672 - > mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): > missed heartbeats from client, timeout: 60s > 2021-09-17 06:43:09.507 [info] <0.14985.679> Closing all channels from > connection 'x.x.x.x00:44914 -> x.x.x.x00:5672' because it has been closed > 2021-09-17 06:43:09.521 [info] <0.11574.678> accepting AMQP connection > <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) > 2021-09-17 06:43:09.523 [info] <0.11574.678> Connection <0.11574.678> > (x.x.x.x00:46834 -> x.x.x.x00:5672) has a client-provided name: > mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3 > 2021-09-17 06:43:09.525 [info] <0.11574.678> connection <0.11574.678> > (x.x.x.x00:46834 -> x.x.x.x00:5672 - > mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): user 'openstack' > authenticated and granted access to vhost '/' > 2021-09-17 06:43:09.654 [info] <0.31575.679> accepting AMQP connection > <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) > 2021-09-17 06:43:09.656 [info] <0.31575.679> Connection <0.31575.679> > (x.x.x.x01:55304 -> x.x.x.x00:5672) has a client-provided name: > magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7 > 2021-09-17 06:43:09.656 [info] <0.31180.678> accepting AMQP connection > <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) > 2021-09-17 06:43:09.658 [info] <0.31575.679> connection <0.31575.679> > (x.x.x.x01:55304 -> x.x.x.x00:5672 - > magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7): user > 'openstack' authenticated and granted access to vhost '/' > 2021-09-17 06:43:09.658 [info] <0.31180.678> Connection <0.31180.678> > (x.x.x.x01:55306 -> x.x.x.x00:5672) has a client-provided name: > magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b > 2021-09-17 06:43:09.660 [info] <0.2757.679> accepting AMQP connection > <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) > 2021-09-17 06:43:09.660 [info] <0.31180.678> connection <0.31180.678> > (x.x.x.x01:55306 -> x.x.x.x00:5672 - > magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b): user > 'openstack' authenticated and granted access to vhost '/' > 2021-09-17 06:43:09.662 [info] <0.2757.679> Connection <0.2757.679> > (x.x.x.x11:58174 -> x.x.x.x00:5672) has a client-provided name: > magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315 > 2021-09-17 06:43:09.663 [info] <0.2757.679> connection <0.2757.679> > (x.x.x.x11:58174 -> x.x.x.x00:5672 - > magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315): user > 'openstack' authenticated and granted access to vhost '/' > 2021-09-17 06:43:09.814 [error] <0.22797.678> closing AMQP connection > <0.22797.678> (x.x.x.x05:57676 -> x.x.x.x00:5672 - > neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): > missed heartbeats from client, timeout: 60s > 2021-09-17 06:43:09.817 [info] <0.7903.679> Closing all channels from > connection 'x.x.x.x05:57676 -> x.x.x.x00:5672' because it has been closed > 2021-09-17 06:43:10.319 [error] <0.18268.678> closing AMQP connection > <0.18268.678> (x.x.x.x00:44924 -> x.x.x.x00:5672 - > nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): > missed heartbeats from client, timeout: 60s > 2021-09-17 06:43:10.322 [info] <0.28032.678> Closing all channels from > connection 'x.x.x.x00:44924 -> x.x.x.x00:5672' because it has been closed > 2021-09-17 06:43:10.330 [info] <0.19596.678> accepting AMQP connection > <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) > 2021-09-17 06:43:10.333 [info] <0.19596.678> Connection <0.19596.678> > (x.x.x.x00:46850 -> x.x.x.x00:5672) has a client-provided name: > nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09 > 2021-09-17 06:43:10.334 [info] <0.19596.678> connection <0.19596.678> > (x.x.x.x00:46850 -> x.x.x.x00:5672 - > nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): user > 'openstack' authenticated and granted access to vhost '/' > 2021-09-17 06:43:10.831 [info] <0.9838.678> accepting AMQP connection > <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) > 2021-09-17 06:43:10.834 [info] <0.9838.678> Connection <0.9838.678> > (x.x.x.x05:57878 -> x.x.x.x00:5672) has a client-provided name: > neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e > 2021-09-17 06:43:10.835 [info] <0.9838.678> connection <0.9838.678> > (x.x.x.x05:57878 -> x.x.x.x00:5672 - > neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): user > 'openstack' authenticated and granted access to vhost '/' > 2021-09-17 06:43:11.738 [error] <0.15275.678> closing AMQP connection > <0.15275.678> (x.x.x.x01:53414 -> x.x.x.x00:5672 - > mod_wsgi:968:b60f723c-882e-4780-a162-55c03ebe6179): > missed heartbeats from client, timeout: 60s > > > /var/log/kolla/glance/glance-api.log > 2021-09-17 13:17:23.212 958 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:17:29.208 955 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:17:42.190 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - > [17/Sep/2021 13:17:42] "GET / HTTP/1.1" 300 1446 0.001949 > 2021-09-17 13:17:46.351 957 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:18:12.336 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - > [17/Sep/2021 13:18:12] "GET / HTTP/1.1" 300 1446 0.003754 > 2021-09-17 13:18:13.703 956 INFO eventlet.wsgi.server > [req-1a00db34-349b-4ed3-8625-edfadc939be4 > d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - > default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:13] "GET > /v2/images?limit=20 HTTP/1.1" 200 2990 0.139229 > 2021-09-17 13:18:33.065 955 INFO eventlet.wsgi.server > [req-a57046eb-549e-4837-95c7-d0994c05449e > d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - > default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET > /v2/images?limit=20 HTTP/1.1" 200 2990 0.096302 > 2021-09-17 13:18:33.077 955 INFO eventlet.wsgi.server > [req-3a8af281-6732-4afd-b3ec-e502367bce51 > d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - > default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET > /v2/schemas/image HTTP/1.1" 200 6259 0.003982 > 2021-09-17 13:18:36.747 959 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:18:36.759 959 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:18:42.460 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - > [17/Sep/2021 13:18:42] "GET / HTTP/1.1" 300 1446 0.001383 > 2021-09-17 13:18:52.659 956 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:18:52.670 956 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:19:12.598 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - > [17/Sep/2021 13:19:12] "GET / HTTP/1.1" 300 1446 0.001590 > 2021-09-17 13:19:16.367 957 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:19:27.127 955 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:19:27.184 955 INFO eventlet.wsgi.server > [req-8f68efc9-f46a-450d-8138-928e4f2585b4 > d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - > default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:19:27] "GET > /v2/images?limit=20 HTTP/1.1" 200 2990 0.032917 > > > > /var/log/kolla/keystone/keystone.log > 2021-09-17 13:13:54.670 958 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:13:55.148 960 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:13:55.150 962 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:13:55.345 959 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:13:56.044 961 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:14:18.402 966 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:14:31.372 963 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:14:36.580 967 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:14:38.771 965 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:16:12.226 964 INFO oslo.messaging._drivers.impl_rabbit [-] > A recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > > > /var/log/kolla/nova/nova-compute.log > 2021-09-17 13:10:07.710 7 INFO oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on > x.x.x.x00:5672 via [amqp] client with port 56226. > 2021-09-17 13:11:11.473 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:11:34.046 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:11:34.721 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:13:07.699 7 ERROR oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is > unreachable: Server unexpectedly closed connection. Trying again in 1 > seconds.: OSError: Server unexpectedly closed connection > 2021-09-17 13:13:08.734 7 INFO oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on > x.x.x.x00:5672 via [amqp] client with port 56434. > 2021-09-17 13:14:11.501 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:14:34.070 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:14:34.745 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:16:08.724 7 ERROR oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is > unreachable: Server unexpectedly closed connection. Trying again in 1 > seconds.: OSError: Server unexpectedly closed connection > 2021-09-17 13:16:09.756 7 INFO oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on > x.x.x.x00:5672 via [amqp] client with port 56638. > 2021-09-17 13:17:34.098 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:19:09.747 7 ERROR oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is > unreachable: Server unexpectedly closed connection. Trying again in 1 > seconds.: OSError: Server unexpectedly closed connection > 2021-09-17 13:19:10.781 7 INFO oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on > x.x.x.x00:5672 via [amqp] client with port 56834. > 2021-09-17 13:20:34.124 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:20:34.777 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:20:51.789 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:22:10.772 7 ERROR oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is > unreachable: Server unexpectedly closed connection. Trying again in 1 > seconds.: OSError: Server unexpectedly closed connection > 2021-09-17 13:22:11.806 7 INFO oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on > x.x.x.x00:5672 via [amqp] client with port 57038. > 2021-09-17 13:23:34.149 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:23:34.803 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:23:51.818 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:25:11.795 7 ERROR oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is > unreachable: Server unexpectedly closed connection. Trying again in 1 > seconds.: OSError: Server unexpectedly closed connection > 2021-09-17 13:25:12.842 7 INFO oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on > x.x.x.x00:5672 via [amqp] client with port 57240. > 2021-09-17 13:26:34.176 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:26:34.830 7 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: > [Errno 104] Connection reset by peer > 2021-09-17 13:28:12.812 7 ERROR oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is > unreachable: Server unexpectedly closed connection. Trying again in 1 > seconds.: OSError: Server unexpectedly closed connection > 2021-09-17 13:28:13.848 7 INFO oslo.messaging._drivers.impl_rabbit [-] > [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on > x.x.x.x00:5672 via [amqp] client with port 57446. > From gmann at ghanshyammann.com Fri Sep 17 17:06:50 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 17 Sep 2021 12:06:50 -0500 Subject: [all][tc][goals] Project Specific PTL and Contributor Documentation: Completion update In-Reply-To: <1797cf1bde6.b09dbcff679085.3250944423641646471@ghanshyammann.com> References: <1797cf1bde6.b09dbcff679085.3250944423641646471@ghanshyammann.com> Message-ID: <17bf4b99c6a.b799bfc5862082.7869873480841941495@ghanshyammann.com> ---- On Mon, 17 May 2021 19:48:08 -0500 Ghanshyam Mann wrote ---- > Hello Everyone, > > As you know, we did not select any community-wide goal for Xena cycle and in PTG, TC > decided to spend more time on the previous cycle pending community-wide goal work[1] > > One of the previous cycle (Ussuri) goals is 'Project Specific PTL and Contributor Documentation' which still not > completed by many projects. > - https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-contrib-docs.html > > I am starting the work to finish this goal which is pretty straight forwards and sending the updates in this ML thread. > Please review the patches and merge them or push the patches for your project's repo if not yet done. > > Gerrit Topic: https://review.opendev.org/q/topic:%22project-ptl-and-contrib-docs%22+(status:open%20OR%20status:merged) > > Tracking: https://storyboard.openstack.org/#!/story/2007236 Hello Everyone, We have merged all the patches and all projects completed this community-wide goal now. - https://review.opendev.org/q/topic:%22project-ptl-and-contrib-docs%22+(status:open%20OR%20status:merged I have marked it complete on the storyboard too - https://storyboard.openstack.org/#!/story/2007236 Thanks, everyone for helping to complete this community-wide goal. -gmann > > Progress Summary: > =============== > > * Projects completed: 32 > * Projects for which patches are up and required to merge those: 10 > * Projects for which patches are not yet up: 12 > > [1] https://etherpad.opendev.org/p/tc-xena-tracker (Item#2) > > -gmann > > > > From kennelson11 at gmail.com Fri Sep 17 17:21:21 2021 From: kennelson11 at gmail.com (Kendall Nelson) Date: Fri, 17 Sep 2021 10:21:21 -0700 Subject: [all][tc][goals] Project Specific PTL and Contributor Documentation: Completion update In-Reply-To: <17bf4b99c6a.b799bfc5862082.7869873480841941495@ghanshyammann.com> References: <1797cf1bde6.b09dbcff679085.3250944423641646471@ghanshyammann.com> <17bf4b99c6a.b799bfc5862082.7869873480841941495@ghanshyammann.com> Message-ID: Thank you so much for all your hard work everyone! Gmann in particular :) -Kendall (diablo_rojo) On Fri, Sep 17, 2021 at 10:07 AM Ghanshyam Mann wrote: > ---- On Mon, 17 May 2021 19:48:08 -0500 Ghanshyam Mann < > gmann at ghanshyammann.com> wrote ---- > > Hello Everyone, > > > > As you know, we did not select any community-wide goal for Xena cycle > and in PTG, TC > > decided to spend more time on the previous cycle pending community-wide > goal work[1] > > > > One of the previous cycle (Ussuri) goals is 'Project Specific PTL and > Contributor Documentation' which still not > > completed by many projects. > > - > https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-contrib-docs.html > > > > I am starting the work to finish this goal which is pretty straight > forwards and sending the updates in this ML thread. > > Please review the patches and merge them or push the patches for your > project's repo if not yet done. > > > > Gerrit Topic: > https://review.opendev.org/q/topic:%22project-ptl-and-contrib-docs%22+(status:open%20OR%20status:merged) > > > > Tracking: https://storyboard.openstack.org/#!/story/2007236 > > Hello Everyone, > > We have merged all the patches and all projects completed this > community-wide goal now. > > - > https://review.opendev.org/q/topic:%22project-ptl-and-contrib-docs%22+(status:open%20OR%20status:merged > > I have marked it complete on the storyboard too > - https://storyboard.openstack.org/#!/story/2007236 > > Thanks, everyone for helping to complete this community-wide goal. > > -gmann > > > > > Progress Summary: > > =============== > > > > * Projects completed: 32 > > * Projects for which patches are up and required to merge those: 10 > > * Projects for which patches are not yet up: 12 > > > > [1] https://etherpad.opendev.org/p/tc-xena-tracker (Item#2) > > > > -gmann > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akanevsk at redhat.com Fri Sep 17 23:40:27 2021 From: akanevsk at redhat.com (Arkady Kanevsky) Date: Fri, 17 Sep 2021 18:40:27 -0500 Subject: [Interop] New meeting time Message-ID: The interop WG changed its weekly meeting time to Thursday 16:00pm UTC. The Google meet and etherpad stay the same. See https://wiki.openstack.org/wiki/Governance/InteropWG#Meetings Thanks, Arkady -- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456 -------------- next part -------------- An HTML attachment was scrubbed... URL: From satish.txt at gmail.com Sat Sep 18 11:14:34 2021 From: satish.txt at gmail.com (Satish Patel) Date: Sat, 18 Sep 2021 07:14:34 -0400 Subject: OVS_AFXDP for production Message-ID: <3226DC41-C05F-4BE0-B92A-ED5FF8B4D46D@gmail.com> Folks, I?m running some high performance workload on SRIOV and planning to move from SRIOV to DPDK but recently Hearing a lots about AFDXP data path and just wanted to check with expert, is this ready for production? How good it is to compare with DPDK? Sent from my iPhone From bkslash at poczta.onet.pl Sat Sep 18 13:51:12 2021 From: bkslash at poczta.onet.pl (bkslash) Date: Sat, 18 Sep 2021 15:51:12 +0200 Subject: RabbitMQ annoying disconnections In-Reply-To: <6eb72199-d368-3e8a-a6e7-1c0e8523bc0e@nemebean.com> References: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> <6eb72199-d368-3e8a-a6e7-1c0e8523bc0e@nemebean.com> Message-ID: <02C6F39B-C041-469F-9217-C1EE9FC3CBF7@poczta.onet.pl> Hi Ben, Thank you for the answer. I?m using open stack Victoria deployed with Kolla-ansible. So I assume I should add (in /etc/kolla/config/global.conf) [oslo_messaging_rabbit] heartbeat_in_pthread = True But for Victoria (nova sample configuration file) it seems to be deprecated? # DEPRECATED: Run the health check heartbeat thread through a native python # thread by default. If this option is equal to False then the health check # heartbeat will inherit the execution model from the parent process. For # example if the parent process has monkey patched the stdlib by using # eventlet/greenlet then the heartbeat will be run through a green thread # (boolean value) # This option is deprecated for removal. # Its value may be silently ignored in the future. #heartbeat_in_pthread = true So will it work? Best regards Adam Tomas > Wiadomo?? napisana przez Ben Nemec w dniu 17.09.2021, o godz. 18:57: > > This sounds like you're hitting the heartbeat/eventlet bug. Basically eventlet won't schedule the heartbeat thread unless something is regularly waking up the api threads. Depending on what release you're running, you may be able to set heartbeat_in_pthread[0] to fix this. > > 0: https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/_drivers/impl_rabbit.py#L90 > > On 9/17/21 7:00 AM, Adam Tomas wrote: >> Hi, >> after some struggling I have almost ?clear? logs (clear=error free :) ) Almost?. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description: >> https://bugzilla.redhat.com/show_bug.cgi?id=1711794 >> in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.? >> but? It is reported :( And produces tons of logs. >> I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :( >> [oslo_messaging_rabbit] >> heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4 >> How to avoid this disconnections?? Or what to do to avoid writing this ?errors?to logs? >> Best regards >> Adam Tomas >> /var/log/kolla/rabbitmq/rabbitXXX.log >> 2021-09-17 06:43:08.771 [error] <0.16056.678> closing AMQP connection <0.16056.678> (x.x.x.x00:44882 -> x.x.x.x00:5672 - mod_wsgi:959:aaa12af5-bb0b-456e-a129-a466eccc24f0): >> missed heartbeats from client, timeout: 60s >> 2021-09-17 06:43:08.773 [info] <0.7412.679> Closing all channels from connection 'x.x.x.x00:44882 -> x.x.x.x00:5672' because it has been closed >> 2021-09-17 06:43:09.282 [info] <0.4574.679> accepting AMQP connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) >> 2021-09-17 06:43:09.283 [info] <0.28410.678> accepting AMQP connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) >> 2021-09-17 06:43:09.285 [info] <0.28410.678> Connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394 >> 2021-09-17 06:43:09.285 [info] <0.4574.679> Connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93 >> 2021-09-17 06:43:09.286 [info] <0.4574.679> connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672 - magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93): user 'openstack' authenticated and granted access to vhost '/' >> 2021-09-17 06:43:09.286 [info] <0.28410.678> connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672 - magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394): user 'openstack' authenticated and granted access to vhost '/' >> 2021-09-17 06:43:09.504 [error] <0.14465.678> closing AMQP connection <0.14465.678> (x.x.x.x00:44914 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): >> missed heartbeats from client, timeout: 60s >> 2021-09-17 06:43:09.507 [info] <0.14985.679> Closing all channels from connection 'x.x.x.x00:44914 -> x.x.x.x00:5672' because it has been closed >> 2021-09-17 06:43:09.521 [info] <0.11574.678> accepting AMQP connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) >> 2021-09-17 06:43:09.523 [info] <0.11574.678> Connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) has a client-provided name: mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3 >> 2021-09-17 06:43:09.525 [info] <0.11574.678> connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): user 'openstack' authenticated and granted access to vhost '/' >> 2021-09-17 06:43:09.654 [info] <0.31575.679> accepting AMQP connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) >> 2021-09-17 06:43:09.656 [info] <0.31575.679> Connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7 >> 2021-09-17 06:43:09.656 [info] <0.31180.678> accepting AMQP connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) >> 2021-09-17 06:43:09.658 [info] <0.31575.679> connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672 - magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7): user 'openstack' authenticated and granted access to vhost '/' >> 2021-09-17 06:43:09.658 [info] <0.31180.678> Connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b >> 2021-09-17 06:43:09.660 [info] <0.2757.679> accepting AMQP connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) >> 2021-09-17 06:43:09.660 [info] <0.31180.678> connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672 - magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b): user 'openstack' authenticated and granted access to vhost '/' >> 2021-09-17 06:43:09.662 [info] <0.2757.679> Connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315 >> 2021-09-17 06:43:09.663 [info] <0.2757.679> connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672 - magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315): user 'openstack' authenticated and granted access to vhost '/' >> 2021-09-17 06:43:09.814 [error] <0.22797.678> closing AMQP connection <0.22797.678> (x.x.x.x05:57676 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): >> missed heartbeats from client, timeout: 60s >> 2021-09-17 06:43:09.817 [info] <0.7903.679> Closing all channels from connection 'x.x.x.x05:57676 -> x.x.x.x00:5672' because it has been closed >> 2021-09-17 06:43:10.319 [error] <0.18268.678> closing AMQP connection <0.18268.678> (x.x.x.x00:44924 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): >> missed heartbeats from client, timeout: 60s >> 2021-09-17 06:43:10.322 [info] <0.28032.678> Closing all channels from connection 'x.x.x.x00:44924 -> x.x.x.x00:5672' because it has been closed >> 2021-09-17 06:43:10.330 [info] <0.19596.678> accepting AMQP connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) >> 2021-09-17 06:43:10.333 [info] <0.19596.678> Connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) has a client-provided name: nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09 >> 2021-09-17 06:43:10.334 [info] <0.19596.678> connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): user 'openstack' authenticated and granted access to vhost '/' >> 2021-09-17 06:43:10.831 [info] <0.9838.678> accepting AMQP connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) >> 2021-09-17 06:43:10.834 [info] <0.9838.678> Connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) has a client-provided name: neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e >> 2021-09-17 06:43:10.835 [info] <0.9838.678> connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): user 'openstack' authenticated and granted access to vhost '/' >> 2021-09-17 06:43:11.738 [error] <0.15275.678> closing AMQP connection <0.15275.678> (x.x.x.x01:53414 -> x.x.x.x00:5672 - mod_wsgi:968:b60f723c-882e-4780-a162-55c03ebe6179): >> missed heartbeats from client, timeout: 60s >> /var/log/kolla/glance/glance-api.log >> 2021-09-17 13:17:23.212 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:17:29.208 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:17:42.190 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:17:42] "GET / HTTP/1.1" 300 1446 0.001949 >> 2021-09-17 13:17:46.351 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:18:12.336 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:12] "GET / HTTP/1.1" 300 1446 0.003754 >> 2021-09-17 13:18:13.703 956 INFO eventlet.wsgi.server [req-1a00db34-349b-4ed3-8625-edfadc939be4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:13] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.139229 >> 2021-09-17 13:18:33.065 955 INFO eventlet.wsgi.server [req-a57046eb-549e-4837-95c7-d0994c05449e d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.096302 >> 2021-09-17 13:18:33.077 955 INFO eventlet.wsgi.server [req-3a8af281-6732-4afd-b3ec-e502367bce51 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.003982 >> 2021-09-17 13:18:36.747 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:18:36.759 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:18:42.460 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:42] "GET / HTTP/1.1" 300 1446 0.001383 >> 2021-09-17 13:18:52.659 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:18:52.670 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:19:12.598 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:19:12] "GET / HTTP/1.1" 300 1446 0.001590 >> 2021-09-17 13:19:16.367 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:19:27.127 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:19:27.184 955 INFO eventlet.wsgi.server [req-8f68efc9-f46a-450d-8138-928e4f2585b4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:19:27] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.032917 >> /var/log/kolla/keystone/keystone.log >> 2021-09-17 13:13:54.670 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:13:55.148 960 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:13:55.150 962 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:13:55.345 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:13:56.044 961 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:14:18.402 966 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:14:31.372 963 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:14:36.580 967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:14:38.771 965 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:16:12.226 964 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> /var/log/kolla/nova/nova-compute.log >> 2021-09-17 13:10:07.710 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56226. >> 2021-09-17 13:11:11.473 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:11:34.046 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:11:34.721 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:13:07.699 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >> 2021-09-17 13:13:08.734 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56434. >> 2021-09-17 13:14:11.501 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:14:34.070 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:14:34.745 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:16:08.724 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >> 2021-09-17 13:16:09.756 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56638. >> 2021-09-17 13:17:34.098 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:19:09.747 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >> 2021-09-17 13:19:10.781 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56834. >> 2021-09-17 13:20:34.124 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:20:34.777 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:20:51.789 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:22:10.772 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >> 2021-09-17 13:22:11.806 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57038. >> 2021-09-17 13:23:34.149 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:23:34.803 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:23:51.818 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:25:11.795 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >> 2021-09-17 13:25:12.842 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57240. >> 2021-09-17 13:26:34.176 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:26:34.830 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >> 2021-09-17 13:28:12.812 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >> 2021-09-17 13:28:13.848 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57446. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhangbailin at inspur.com Sun Sep 19 03:11:17 2021 From: zhangbailin at inspur.com (=?utf-8?B?QnJpbiBaaGFuZyjlvKDnmb7mnpcp?=) Date: Sun, 19 Sep 2021 03:11:17 +0000 Subject: =?utf-8?B?562U5aSNOiBbenVuXVtjeWJvcmddW2t1cnlyXVtuZXV0cm9uXSBFZGdlIGZv?= =?utf-8?B?ciBzY2llbnRpZmljIHJlc2VhcmNoIHNlc3Npb24gb24gU2VwdGVtYmVyIDEz?= =?utf-8?Q?_Edge_WG_meeting?= In-Reply-To: <7B5E5A10-C30A-4F2B-AD64-FF2F8B988FAC@gmail.com> References: <5DB3E1B1-1D6A-4C0B-AB3D-FE33F0B88961@gmail.com> <360c63346cfd44fca42eaa744c84ef8d@inspur.com> <7B5E5A10-C30A-4F2B-AD64-FF2F8B988FAC@gmail.com> Message-ID: <7960ddb241ca4575a0005546249b753e@inspur.com> Hi Ildiko, We watched your meeting minutes and also discussed within the Cyborg project team (xinranwang, chenke, wenpingsong). I known that zun and Cyborg have been integrated for a while. Is there any problem now? I added this topic to Cyborg PTG's etherpad [1], you can leave a comment on it. [1] https://etherpad.opendev.org/p/cyborg-yoga-ptg brinzhang -----????----- ???: Ildiko Vancsa [mailto:ildiko.vancsa at gmail.com] ????: 2021?9?14? 5:35 ???: openstack-discuss at lists.openstack.org ??: xin-ran.wang at intel.com; Brin Zhang(???) ??: Re: [zun][cyborg][kuryr][neutron] Edge for scientific research session on September 13 Edge WG meeting Hi, We had a great session today, thanks to all of you who tuned in. For those of you who couldn?t make it, we have a recording of the presentation and the discussion that followed: https://wiki.openstack.org/wiki/Edge_Computing_Group#CHI.40Edge_-_recording_of_the_session_with_the_Chameleon_project.2C_September_13.2C_2021 Next steps we touched on: * Chameleon integrated Cyborg and Zun ad would like to look into how to upstream that work * Uploading a spec about the integration steps * Discussing the details at the PTG * Chameleon also added a new Neutron ML2 plugin to support Wireguard; it could be a good item to discuss with the Neutron team as well to check upstream potential and make improvements to the agent Please let me know if you have questions to either of the above. Thanks and Best Regards, Ildik? > On Sep 7, 2021, at 17:15, Brin Zhang(???) wrote: > >> I will share the recording of the meeting as soon as it is available in case someone would not be able to make it and would like to check out the presentation and the discussion that follows. > Thanks for you will share of the record, at 1300UTC I have another meeting (start 1100UTC), if I can end up that meeting earlier, I will join this meeting ^^ > > > > > brinzhang > > > -----????----- > ???: Ildiko Vancsa [mailto:ildiko.vancsa at gmail.com] > ????: 2021?9?8? 6:08 > ???: Brin Zhang(???) > ??: openstack-discuss at lists.openstack.org; xin-ran.wang at intel.com > ??: Re: [lists.openstack.org??][zun][cyborg][kuryr] Edge for scientific research session on September 13 Edge WG meeting > > Hi Bailin, > > Thank you for sharing the links. > > Zun is the project in OpenStack that is supporting to run and manage containers directly from OpenStack without the need of Kubernetes. As the Chameleon project is using it to run their containerized workload and they also need hardware accelerators they integrated it with Cyborg. > > Many of their activities fall under the edge computing area and they are starting their collaboration with the OpenInfra Edge Computing group for this reason. As part of that work we are helping them to reach relevant communities and project teams where their requirements identified some gaps and I believe the Cyborg-Zun integration is one of these. They would like to upstream their work if the community finds value in it as it may or may not fall into the current scope of the respective projects. > > I hope that we can start to discuss some of this on the meeting next Monday and take next steps from there. > > I hope this helps a bit with the context and we can have a fruitful discussion next week. > > I will share the recording of the meeting as soon as it is available in case someone would not be able to make it and would like to check out the presentation and the discussion that follows. > > Thanks, > Ildik? > > >> On Sep 3, 2021, at 04:20, Brin Zhang(???) wrote: >> >> Hi Ildiko, >> >> Currently Cyborg and nova support interaction, we are not very familiar with zun yet, if possible, please provide some relevant information. You can refer to [1] for Cyborg related materials, and [2] for the interactive content of nova and cyborg (not yet merged). >> >> [1]https://docs.openstack.org/cyborg/latest/user/architecture.html >> [2]https://review.opendev.org/c/openstack/cyborg/+/794549 >> https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/nova-cyborg-interaction.html >> >> >> Bailin Zhang (brinzhang) >> >> >> -----????----- >> ???: Ildiko Vancsa [mailto:ildiko.vancsa at gmail.com] >> ????: 2021?8?31? 22:39 >> ???: openstack-discuss >> ??: [lists.openstack.org??][zun][cyborg][kuryr] Re: Edge for scientific research session on September 13 Edge WG meeting >> >> Hi, >> >> It is a friendly reminder that the OpenInfra Edge Computing Group is having a joint session with the Chameleon project. >> >> During this session you can learn about the Chameleon project and how they are using OpenStack to build an edge platform for scientific research and academic purposes. They have also found some gaps during their journey. To mention an example the integration between Cyborg and Zun is missing to be able to use acceleration devices with containers orchestrated by OpenStack and they have been working to fill in this gap. >> >> The session will start with a presentation about the project and then we will use the remaining time to talk about solutions and next steps. I would like to invite everyone who is interested in learning about how OpenStack can be turned into an edge platform and what are the missing pieces of the puzzle that still need to be enhanced or implemented. >> >> The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. >> The meeting will be held on Zoom, for the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings >> >> Please let me know if you need a calendar invite and I can send you one to get reminded about the meeting. >> >> Thanks and Best Regards, >> Ildik? >> >> >>> On Aug 25, 2021, at 17:00, Ildiko Vancsa wrote: >>> >>> Hi, >>> >>> I?m very excited to invite you to a session with the Chameleon project to discuss how edge computing is powering education, scientific research and more! As always in the area of edge there are challenges and gaps that we need to solve together. In that sense the session will start with a presentation that we will then turn into a discussion to turn towards discussing solutions. >>> >>> Please also note that the Chameleon project is using OpenStack to build edge solutions. As they identified some gaps they would like to work with the community to address the challenges that they?ve been hitting. Please join us to get their feedback and find the best way to improve OpenStack to deliver a platform that can be used on the edge! >>> >>> The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. >>> For the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings >>> >>> Please find the details of the session below. >>> >>> Thanks and Best Regards, >>> Ildik? >>> >>> >>> Title: Chameleon: a cloud to edge infrastructure for science >>> >>> Abstract: The NSF-funded Chameleon project (www.chameleoncloud.org) provides an OpenStack-based bare metal reconfigurable cloud that supports systems research, education, and emergent application projects in science. The facility has been in existence for 6+ years, and has served ~6,000 users working on a variety of projects ranging from networking, operating systems design, and security, to innovative applications in astronomy, physics, and life sciences. Recent research trends in IoT and edge computing, as evidenced by increased numbers of project applications in those areas, made it clear that Chameleon needed to expand into the edge space. Remaining within the OpenStack framework allowed us to leverage end-user features that we developed for Chameleon, such as access via federated identity and Jupyter integration. Accordingly, we developed infrastructure called CHI at Edge that adapts Zun to provision containers on edge devices and significantly expands its capabilities to work in the edge space. The talk will describe our use case and the emergent product definition, the OpenStack adaptations we made to the current preview implementation, the directions we plan to take in our further work, as well as the ecosystem of projects and NSF infrastructure deployments leveraging this work and collaborating with us. >>> >>> >> >> > From eduardo.experimental at gmail.com Mon Sep 20 02:31:32 2021 From: eduardo.experimental at gmail.com (Eduardo Santos) Date: Mon, 20 Sep 2021 02:31:32 +0000 Subject: [keystone] Ed25519 support Message-ID: Hi folks, I'm trying to create an instance with an assigned keypair that uses the Ed25519 algorithm, but I'm unable to login into the instance without providing a password (I generated the public key without a passphrase). RSA keypairs work fine. Is Ed25519 not supported? I didn't find a support matrix in the documentation. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephenfin at redhat.com Mon Sep 20 07:09:13 2021 From: stephenfin at redhat.com (stephenfin at redhat.com) Date: Mon, 20 Sep 2021 08:09:13 +0100 Subject: [ops][cinder] is anyone using RateLimitingMiddleware ? In-Reply-To: References: Message-ID: <62625b0e41da3e90e227bb9990938ce3066c7f96.camel@redhat.com> On Wed, 2021-09-08 at 12:00 -0400, Brian Rosmaita wrote: > Hello operators, > > Takashi Kajinami noticed that cinder's RateLimitingMiddleware class > lives among the legacy v2 classes and should be moved to live with the > v3 classes, but then it occurred to him that maybe it should simply be > deprecated and removed [0]. > > There are a few reasons to believe that nobody is actually using this > functionality, for example, it's not included in the default > api-paste.ini file, this middleware no longer exists in Nova, it gives > you inexact behavior with multiple API nodes, etc. > > We discussed this topic at today's cinder meeting and decided to ask > whether anyone is using the RateLimitingMiddleware with cinder before > deprecating it for removal. > > If it's going to be deprecated, we'd like to do it in Xena, so please > reply to this email or leave a comment on the deprecation patch [1] (or > preferably both) before 23:59 UTC on Tuesday 14 September 2021. > > Thank you! > > > [0] https://bugs.launchpad.net/cinder/+bug/1942696 > [1] https://review.opendev.org/c/openstack/cinder/+/807469 Not an ansewr, but I have a relatively old patch to document this [1]. If we're not deprecating this, we should probably merge the docs patch. If we are deprecating it, the doc patch can obviously be abandoned. Cheers, Stephen [1] https://review.opendev.org/c/openstack/cinder/+/782518 From bkslash at poczta.onet.pl Mon Sep 20 08:17:39 2021 From: bkslash at poczta.onet.pl (Adam Tomas) Date: Mon, 20 Sep 2021 10:17:39 +0200 Subject: RabbitMQ annoying disconnections In-Reply-To: <0670B960225633449A24709C291A525251CA0CCB@COM03.performair.local> References: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> <0670B960225633449A24709C291A525251CA0CCB@COM03.performair.local> Message-ID: <69239B77-1552-49BC-ACE9-A410900C63FD@poczta.onet.pl> Dominic, according to documentation: heartbeat_rate = 2 integer value How often times during the heartbeat_timeout_threshold we check the heartbeat. heartbeat_timeout_threshold = 60 integer value Number of seconds after which the Rabbit broker is considered down if heartbeat?s keep-alive fails (0 disables heartbeat). So to avoid disconnection from service side (nova, keystone, etc.) I?ve increased Heartbeat_timeout_treshold from 60 to 720 and set heartbeat_rate to 4, so Rabbit will be considered dead after 360s not after 60s as in default. In addition I?ve increased heartbeat to 600 from 60 in rabbitmq.conf. And again - according to documentation: The heartbeat timeout value defines after what period of time the peer TCP connection should be considered unreachable (down) by RabbitMQ and client libraries. So disconnection from Rabbit?s side should be after 600s of no heartbeat And I still got disconnections. Best regards Adam Tomas > Wiadomo?? napisana przez DHilsbos at performair.com w dniu 17.09.2021, o godz. 18:38: > > Adam; > > If I'm reading this correctly; Rabbit is timing out, but you're increasing the heartbeat period of OpenStack. This would make the issue worse, wouldn't it? > > It seems to me that you would want to lower the heartbeat interval of OpenStack, and raise the timeout of Rabbit. > > That said; it looks like you're using Kola, and I know nothing about Kola. > > Thank you, > > Dominic L. Hilsbos, MBA > Vice President ? Information Technology > Perform Air International Inc. > DHilsbos at PerformAir.com > www.PerformAir.com > > > From: Adam Tomas [mailto:bkslash at poczta.onet.pl] > Sent: Friday, September 17, 2021 5:01 AM > To: openstack-discuss > Subject: RabbitMQ annoying disconnections > > Hi, > after some struggling I have almost ?clear? logs (clear=error free :) ) Almost?. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description: > > https://bugzilla.redhat.com/show_bug.cgi?id=1711794 > > in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.? > > > but? It is reported :( And produces tons of logs. > > > I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :( > > > [oslo_messaging_rabbit] > heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bkslash at poczta.onet.pl Mon Sep 20 08:21:42 2021 From: bkslash at poczta.onet.pl (Adam Tomas) Date: Mon, 20 Sep 2021 10:21:42 +0200 Subject: RabbitMQ annoying disconnections In-Reply-To: <6eb72199-d368-3e8a-a6e7-1c0e8523bc0e@nemebean.com> References: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> <6eb72199-d368-3e8a-a6e7-1c0e8523bc0e@nemebean.com> Message-ID: Hi Ben, I?ve set [oslo_messaging_rabbit] heartbeat_in_pthread = True and of course restarted the control plane, but it seems nothing has changed - still have disconnections in logs > Wiadomo?? napisana przez Ben Nemec w dniu 17.09.2021, o godz. 18:57: > > This sounds like you're hitting the heartbeat/eventlet bug. Basically eventlet won't schedule the heartbeat thread unless something is regularly waking up the api threads. Depending on what release you're running, you may be able to set heartbeat_in_pthread[0] to fix this. > > 0: https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/_drivers/impl_rabbit.py#L90 > > On 9/17/21 7:00 AM, Adam Tomas wrote: From smooney at redhat.com Mon Sep 20 09:47:36 2021 From: smooney at redhat.com (Sean Mooney) Date: Mon, 20 Sep 2021 10:47:36 +0100 Subject: [keystone] [nova] Ed25519 support In-Reply-To: References: Message-ID: On Mon, 2021-09-20 at 02:31 +0000, Eduardo Santos wrote: > Hi folks, > > I'm trying to create an instance with an assigned keypair that uses the > Ed25519 algorithm, but I'm unable to login into the instance without > providing a password (I generated the public key without a passphrase). RSA > keypairs work fine. > > Is Ed25519 not supported? I didn't find a support matrix in the > documentation. this need a newer version of python-cryptography to work. specifically in the 2.6+. also keypairs are part of nova not keystone so updated the subject. i have left keystone incase there is some use of keypairs in keystone im unaware of but if you are using OSP or any other distibution of openstack they proably have a bug like https://bugzilla.redhat.com/show_bug.cgi?id=1873581 tor track updating the version. in redhat osp 16.2 this is now suppported on rhel 8.4 which is based on upstream train. so this is suported but you just need the corret lib version to allow it to work. if you have the correct version of python-cryptography then it might depend on your guest OS as it will also need to support that key algorthim too. > > Thanks. From openinfradn at gmail.com Mon Sep 20 11:42:48 2021 From: openinfradn at gmail.com (open infra) Date: Mon, 20 Sep 2021 17:12:48 +0530 Subject: FreeNAS intergration witn Openstack Message-ID: Hi, Is it possible to use a freenas storage instead of ceph cluster? Regards, Danishka -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Mon Sep 20 12:15:40 2021 From: smooney at redhat.com (Sean Mooney) Date: Mon, 20 Sep 2021 13:15:40 +0100 Subject: FreeNAS intergration witn Openstack In-Reply-To: References: Message-ID: On Mon, 2021-09-20 at 17:12 +0530, open infra wrote: > Hi, > > Is it possible to use a freenas storage instead of ceph cluster? you could use it as a provider of nfs shares using the generic cinder nfs driver. it also can provide isci volumes but there is no driver that can talk to free nas to actully create those. https://github.com/iXsystems/cinder there is a FreeNAS cinder driver from iXsystems but i dont think they are involded with upstream development much. at least they are not listed on https://docs.openstack.org/cinder/rocky/reference/support-matrix.html so your milage will vary but it should technially be possible to do. this is wehre there user docs are https://www.ixsystems.com/documentation/freenas/11.2/ which just point you back to the git repo so it does not look like there is much support unless you are willing to support it yourself. > > Regards, > Danishka From tpb at dyncloud.net Mon Sep 20 12:47:07 2021 From: tpb at dyncloud.net (Tom Barron) Date: Mon, 20 Sep 2021 08:47:07 -0400 Subject: FreeNAS intergration witn Openstack In-Reply-To: References: Message-ID: <20210920124707.xkeywhib55vvnxa6@barron.net> On 20/09/21 13:15 +0100, Sean Mooney wrote: >On Mon, 2021-09-20 at 17:12 +0530, open infra wrote: >> Hi, >> >> Is it possible to use a freenas storage instead of ceph cluster? >you could use it as a provider of nfs shares using the generic cinder nfs driver. > >it also can provide isci volumes but there is no driver that can talk to free nas >to actully create those. > >https://github.com/iXsystems/cinder >there is a FreeNAS cinder driver from iXsystems but i dont think they are involded with upstream development much. >at least they are not listed on https://docs.openstack.org/cinder/rocky/reference/support-matrix.html > >so your milage will vary but it should technially be possible to do. >this is wehre there user docs are https://www.ixsystems.com/documentation/freenas/11.2/ >which just point you back to the git repo so it does not look like there is much support unless you are willing to support it yourself. > > >> >> Regards, >> Danishka > > > Note that these solutions for Cinder will give you a raw block device, even if an NFS file is used to back it. An OpenStack compute instance will attach the block device. If you want to use applications that write to a filesystem, you need to format the block device and create a file system, and mount it yourself from within the compute instance. And this is a fine thing to do. Perhaps the most natural way to use a FreeNAS with OpenStack would be with Manila, which presents shared file systems like NFS directly for guest VMs to mount. Unfortunately, I don't know of a Manila driver for FreeNAS -- don't see one in the directory Sean referenced and we don't have one in-tree. I think it would be pretty easy to write, and I considered doing this at one point, but I no longer have a FreeNAS at home so I put it off. The quickest way to use it in OpenStack without writing any driver code, though, as Sean said, would be with the generic Cinder nfs driver, assuming that a block device rather than a shared filesystem will work for your use case. From wodel.youchi at gmail.com Mon Sep 20 15:45:11 2021 From: wodel.youchi at gmail.com (wodel youchi) Date: Mon, 20 Sep 2021 16:45:11 +0100 Subject: Install controller computehci on 3 nodes with tripleO Message-ID: Hi, Is it technically possible to install openstack tripleO on 3 nodes, such that all 3 nodes play the roles of controller compute and ceph storage (hci) at the same time, with replication and HA and all? Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From johfulto at redhat.com Mon Sep 20 15:59:31 2021 From: johfulto at redhat.com (John Fulton) Date: Mon, 20 Sep 2021 11:59:31 -0400 Subject: Install controller computehci on 3 nodes with tripleO In-Reply-To: References: Message-ID: On Mon, Sep 20, 2021 at 11:46 AM wodel youchi wrote: > > Hi, > > Is it technically possible to install openstack tripleO on 3 nodes, such that all 3 nodes play the roles of controller compute and ceph storage (hci) at the same time, with replication and HA and all? It's technically possible for a POC (just compose a role [1] accordingly) but it's not a recommended configuration because the controllers use pacemaker with power fencing which you don't want running on your OSD hosts. E.g. pacemaker could take out 1/3rd of your OSDs and make the cluster backfill. If you're short on nodes then one option is to use four. One as a basic KVM hypervisor to host VMs including a virtual undercloud and three virtual controllers. Then use three physical HCI nodes which host the VMs and OSDs. John [1] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/custom_roles.html From mahati.chamarthy at gmail.com Mon Sep 20 07:38:27 2021 From: mahati.chamarthy at gmail.com (Mahati Chamarthy) Date: Mon, 20 Sep 2021 13:08:27 +0530 Subject: [Outreachy] Call for mentors, project submission by September 26 Message-ID: Hello! *TL;DR OpenStack is participating in Outreachy for the upcoming December round!* *Please submit projects as soon as possible, the deadline is September 26th, 4 pm UTC:* *https://www.outreachy.org/communities/cfp/openstack/ * Outreachy's goal is to support people from groups underrepresented in the technology industry. Interns will work remotely with mentors from our community. We are seeking mentors to propose projects that Outreachy interns can work on during their internship. If you or your colleagues are interested in mentoring in the next round, please submit your project proposal by *September 26th,* *2021 *over here https://www.outreachy.org/communities/cfp/openstack/ Mentors should read the mentor FAQ https://www.outreachy.org/mentor/mentor-faq and find more details about the Outreachy program and timeline in https://www.outreachy.org/communities/cfp/ . If you want help crafting your project proposal, please contact me Mahati Chamarthy or Samuel Queiroz < samueldmq at gmail.com> and we will be glad to assist you. Thank you, Samuel Queiroz -------------- next part -------------- An HTML attachment was scrubbed... URL: From anyrude10 at gmail.com Mon Sep 20 12:45:45 2021 From: anyrude10 at gmail.com (Anirudh Gupta) Date: Mon, 20 Sep 2021 18:15:45 +0530 Subject: [TripleO] Installation of Undercloud machine on Centos 8 fails in prefetching the Images Message-ID: Hi Team, I was trying to install Undercloud on Centos 8 using the link below, but facing error in executing "openstack undercloud install" command. The error is in prefetch image Steps followed: 1. Created a non root stack user - sudo useradd stack - sudo passwd stack # specify a password - echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack - sudo chmod 0440 /etc/sudoers.d/stack - su - stack 2. Install the python-tripleo-repos - sudo dnf install -y https://trunk.rdoproject.org/centos8/component/tripleo/current/python3-tripleo-repos-0.1.1-0.20210915165829.6d878bb.el8.noarch.rpm - In order to install victoria release, executed the command - *sudo -E tripleo-repos -b victoria current ceph* 3. As per the document, "*The remaining repositories configuration steps below should not be done for stable releases!"* - *So, I didn't executed the command* - "sudo -E tripleo-repos current-tripleo-dev ceph" - *Is my understanding correct? This step has to be skipped in case we are going with a stable victoria release?* 4. Modified undercloud.conf file and executed the command - openstack undercloud install *ISSUE:* 2021-09-20 11:16:18,724 p=59556 u=root n=ansible | 2021-09-20 11:16:18.723422 | 525400a8-fc39-4771-208c-000000000e71 | FATAL | Pre-fetch all the containers | undercloud | item= quay.io/tripleovictoria/openstack-swift-account:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg":" *Failed to pull image quay.io/tripleovictoria/openstack-swift-account:current-tripleo ", "prefetch_image": "quay.io/tripleovictoria/openstack-swift-account:current-tripleo "}* 5. In order to eliminate the problem, I manually installed the image - - sudo podman pull *quay.io/tripleovictoria/openstack-swift-account:current-tripleo * After re-executing, then it again failed for the keystone image, which was already installed on the system. I verified in *sudo podman images* command *I tried running this 5-6 times and issue generates randomly for any openstack image ( sometimes for rabbitmq sometimes for swift, keystone) which is already installed in machine* Can someone please help or suggest what step I am missing Note: This I have tried using Centos 8 Stream and Centos 8.4(using --no-stream), but same issue is observed in both Regards Anirudh Gupta -------------- next part -------------- An HTML attachment was scrubbed... URL: From rguyett at datto.com Mon Sep 20 13:41:32 2021 From: rguyett at datto.com (Reid Guyett) Date: Mon, 20 Sep 2021 09:41:32 -0400 Subject: [Swift] Rebalancing EC question In-Reply-To: References: Message-ID: Thanks for that explanation. It is clear now how the rebalancing is working for the EC policies. We have adjusted our reconstruction workers to speed up the rebalance and it seemed to have helped. It went from weeks to days. Reid Guyett On Thu, Sep 16, 2021 at 2:02 PM Clay Gerrard wrote: > > Not scary! > > Because you have a 15/4 EC policy, we say each partition has 19 "replicas". And since rebalance will only move one "replica" of any partition max at each rebalance: up to 100% of your partitions may have at least one replica assignment move. > > That means, after you push out this ring, 100% of your object GET requests will experience at most one "replica" is out of place. But that's ok! In a 15/4 you only need 15 EC fragments to respond successfully and you have 18 total fragments that did NOT get reassigned. > > It's unfortunate the language is a little ambiguous, but it is talking about % of *partitions* that had a replica moved. Since each object resides in single a partition - the % of partitions affected most directly communicates the % of client objects affected by the rebalance. We do NOT display % of *partition-replicas* moved because while the number would be smaller - it could never be 100% because of the restriction that only one "replica" may move. > > When doing a large topology change - particularly with EC - it may be the case that more than one replica of each part will need to move (imagine doubling your capacity into a second zone on a 8+4 ring) - so it'll take a few cranks. Eventually you'll want to have moved 6 replicas of each part (6 in z1 and 6 in z2), but if we allowed you to move six replicas of 100% of your parts you'd only have 6/8 required parts to service reads! > > Protip: when you push out the new ring you can turn on handoffs_only mode for the reconstructor for a little while to get things rebalanced MUCH more quickly - just don't forget to turn it off! > > (sending second time because I forgot to reply all to the list) > > On Thu, Sep 16, 2021 at 11:35 AM Reid Guyett wrote: >> >> Hello, >> >> We were working on expanding one of our clusters (Ussuri on Ubuntu >> 18.04) and are wondering about the rebalance behavior of >> swift-ring-builder. When we run it in debug mode on a 15/4 EC ring, we >> see this message about "Unable to finish rebalance plan after 2 >> attempts" and are seeing 100% partitions reassigned. >> >> DEBUG: Placed 10899/2 onto dev r1z3-10.40.48.72/d10 >> DEBUG: Placed 2183/3 onto dev r1z5-10.40.48.76/d11 >> DEBUG: Placed 1607/1 onto dev r1z3-10.40.48.70/d28 >> DEBUG: Assigned 32768 parts >> DEBUG: Gather start is 10278 (Last start was 25464) >> DEBUG: Unable to finish rebalance plan after 2 attempts >> Reassigned 32768 (100.00%) partitions. Balance is now 63.21. >> Dispersion is now 0.00 >> ------------------------------------------------------------------------------- >> NOTE: Balance of 63.21 indicates you should push this >> ring, wait at least 1 hours, and rebalance/repush. >> ------------------------------------------------------------------------------- >> >> Moving 100% seems scary, what does that mean in this situation? Is >> this message because 1 fragment from every partition is moved and that >> is the most that it can do per rebalance because they are technically >> the same partition? >> When we compare the swift-ring-builder output (partitions per device) >> between rebalances we can see some partitions move each time until we >> no longer see the push/wait/rebalance message again. So it's not >> really moving 100% partitions. >> >> Reid >> >> > > > -- > Clay Gerrard From wodel.youchi at gmail.com Mon Sep 20 16:18:42 2021 From: wodel.youchi at gmail.com (wodel youchi) Date: Mon, 20 Sep 2021 17:18:42 +0100 Subject: Install controller computehci on 3 nodes with tripleO In-Reply-To: References: Message-ID: Hi, Thanks, your answer about pacemaker raised another question, VM (instances) high availability is managed by peacemaker in TripleO, could this be a problem when using HCI deployment? Regards. Le lun. 20 sept. 2021 ? 16:59, John Fulton a ?crit : > On Mon, Sep 20, 2021 at 11:46 AM wodel youchi > wrote: > > > > Hi, > > > > Is it technically possible to install openstack tripleO on 3 nodes, such > that all 3 nodes play the roles of controller compute and ceph storage > (hci) at the same time, with replication and HA and all? > > It's technically possible for a POC (just compose a role [1] > accordingly) but it's not a recommended configuration because the > controllers use pacemaker with power fencing which you don't want > running on your OSD hosts. E.g. pacemaker could take out 1/3rd of your > OSDs and make the cluster backfill. If you're short on nodes then one > option is to use four. One as a basic KVM hypervisor to host VMs > including a virtual undercloud and three virtual controllers. Then use > three physical HCI nodes which host the VMs and OSDs. > > John > > [1] > https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/custom_roles.html > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johfulto at redhat.com Mon Sep 20 16:22:42 2021 From: johfulto at redhat.com (John Fulton) Date: Mon, 20 Sep 2021 12:22:42 -0400 Subject: [tripleo] Re: Install controller computehci on 3 nodes with tripleO In-Reply-To: References: Message-ID: On Mon, Sep 20, 2021 at 12:18 PM wodel youchi wrote: > > Hi, > > Thanks, your answer about pacemaker raised another question, VM (instances) high availability is managed by peacemaker in TripleO, could this be a problem when using HCI deployment? Yes, it's not a recommended combination for the same reasons. John > > > Regards. > > Le lun. 20 sept. 2021 ? 16:59, John Fulton a ?crit : >> >> On Mon, Sep 20, 2021 at 11:46 AM wodel youchi wrote: >> > >> > Hi, >> > >> > Is it technically possible to install openstack tripleO on 3 nodes, such that all 3 nodes play the roles of controller compute and ceph storage (hci) at the same time, with replication and HA and all? >> >> It's technically possible for a POC (just compose a role [1] >> accordingly) but it's not a recommended configuration because the >> controllers use pacemaker with power fencing which you don't want >> running on your OSD hosts. E.g. pacemaker could take out 1/3rd of your >> OSDs and make the cluster backfill. If you're short on nodes then one >> option is to use four. One as a basic KVM hypervisor to host VMs >> including a virtual undercloud and three virtual controllers. Then use >> three physical HCI nodes which host the VMs and OSDs. >> >> John >> >> [1] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/custom_roles.html >> From gfidente at redhat.com Mon Sep 20 16:25:13 2021 From: gfidente at redhat.com (Giulio Fidente) Date: Mon, 20 Sep 2021 18:25:13 +0200 Subject: Install controller computehci on 3 nodes with tripleO In-Reply-To: References: Message-ID: <5dd74161-cf13-5fc2-e117-aefc15042ada@redhat.com> On 9/20/21 17:45, wodel youchi wrote: > Hi, > > Is it technically possible to install openstack tripleO on 3 nodes, such > that all 3 nodes play the roles of controller compute and ceph storage > (hci) at the same time, with replication and HA and all? I think it could be done by creating a custom role which merges the Controller and ComputeHCI services together but it has important implications; for example, should pacemaker fence a node, that would kill the workload *and* the osds hosted on that node this can be acceptable or not, depending on expectations and other possible mitigations I guess -- Giulio Fidente GPG KEY: 08D733BA From laurentfdumont at gmail.com Mon Sep 20 16:26:29 2021 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Mon, 20 Sep 2021 12:26:29 -0400 Subject: [openstack-cli] Missing volume backend commands Message-ID: Hey everyone, Trying to use the volume backend command to get some information regarding storage pools. https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/volume-backend.html Not sure if it's something on my side, but I cannot see to access a subsection of the volume commands from the CLI client. openstack volume backend pool list openstack: 'volume backend pool list' is not an openstack command. See 'openstack --help'. Did you mean one of these? volume attachment complete volume attachment create volume attachment delete volume attachment list volume attachment set volume attachment show volume backup create volume backup delete volume backup list /usr/local/lib/python3.9/site-packages/openstack/ drwxr-xr-x 41 laurentdumont admin 1.3K 2 Jun 17:03 . drwxr-xr-x 187 laurentdumont admin 5.8K 20 Sep 09:47 .. -rw-r--r-- 1 laurentdumont admin 2.4K 2 Jun 17:03 __init__.py -rw-r--r-- 1 laurentdumont admin 1.2K 2 Jun 17:03 __main__.py drwxr-xr-x 15 laurentdumont admin 480B 2 Jun 17:03 __pycache__ -rw-r--r-- 1 laurentdumont admin 1.4K 2 Jun 17:03 _hacking.py -rw-r--r-- 1 laurentdumont admin 4.6K 2 Jun 17:03 _log.py -rw-r--r-- 1 laurentdumont admin 5.6K 2 Jun 17:03 _services_mixin.py drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 accelerator drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 baremetal drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 baremetal_introspection drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 block_storage drwxr-xr-x 28 laurentdumont admin 896B 2 Jun 17:03 cloud drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 clustering drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 compute drwxr-xr-x 14 laurentdumont admin 448B 2 Jun 17:03 config -rw-r--r-- 1 laurentdumont admin 21K 2 Jun 17:03 connection.py drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 database drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 dns -rw-r--r-- 1 laurentdumont admin 8.6K 2 Jun 17:03 exceptions.py drwxr-xr-x 5 laurentdumont admin 160B 2 Jun 17:03 fixture -rw-r--r-- 1 laurentdumont admin 1.6K 2 Jun 17:03 format.py drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 identity drwxr-xr-x 11 laurentdumont admin 352B 2 Jun 17:03 image drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 instance_ha drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 key_manager drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 load_balancer drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 message drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 network drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 object_store drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 orchestration drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 placement -rw-r--r-- 1 laurentdumont admin 29K 2 Jun 17:03 proxy.py -rw-r--r-- 1 laurentdumont admin 0B 2 Jun 17:03 py.typed -rw-r--r-- 1 laurentdumont admin 88K 2 Jun 17:03 resource.py -rw-r--r-- 1 laurentdumont admin 13K 2 Jun 17:03 service_description.py drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 shared_file_system drwxr-xr-x 9 laurentdumont admin 288B 2 Jun 17:03 tests -rw-r--r-- 1 laurentdumont admin 12K 2 Jun 17:03 utils.py -rw-r--r-- 1 laurentdumont admin 641B 2 Jun 17:03 version.py drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 workflow usr/local/?/site-packages/openstack % python3 Python 3.9.2 (default, Mar 26 2021, 23:27:12) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> I cannot see the volume folder (but it's on the opendev) The command was added here https://opendev.org/openstack/python-openstackclient/commit/9647d43bd5e77c815ac2f08b7de2226a657a66bc Removing + reinstall the client doesn't seem to work either. Any clues? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lyarwood at redhat.com Mon Sep 20 17:16:11 2021 From: lyarwood at redhat.com (Lee Yarwood) Date: Mon, 20 Sep 2021 18:16:11 +0100 Subject: [openstack-cli] Missing volume backend commands In-Reply-To: References: Message-ID: <20210920171611.62nc2mo5ux3m6773@lyarwood-laptop.usersys.redhat.com> On 20-09-21 12:26:29, Laurent Dumont wrote: > Hey everyone, > > Trying to use the volume backend command to get some information regarding > storage pools. > > https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/volume-backend.html > > Not sure if it's something on my side, but I cannot see to access a > subsection of the volume commands from the CLI client. As noted at the top of that page it's only available with v2 API of Cinder that has recently been dropped. Assuming you still have it in your env use --os-volume-api-version 2 with osc. $ openstack --os-volume-api-version 2 volume backend capability show ${host} Cheers, Lee > openstack volume backend pool list > openstack: 'volume backend pool list' is not an openstack command. See > 'openstack --help'. > Did you mean one of these? > volume attachment complete > volume attachment create > volume attachment delete > volume attachment list > volume attachment set > volume attachment show > volume backup create > volume backup delete > volume backup list > > /usr/local/lib/python3.9/site-packages/openstack/ > > drwxr-xr-x 41 laurentdumont admin 1.3K 2 Jun 17:03 . > drwxr-xr-x 187 laurentdumont admin 5.8K 20 Sep 09:47 .. > -rw-r--r-- 1 laurentdumont admin 2.4K 2 Jun 17:03 __init__.py > -rw-r--r-- 1 laurentdumont admin 1.2K 2 Jun 17:03 __main__.py > drwxr-xr-x 15 laurentdumont admin 480B 2 Jun 17:03 __pycache__ > -rw-r--r-- 1 laurentdumont admin 1.4K 2 Jun 17:03 _hacking.py > -rw-r--r-- 1 laurentdumont admin 4.6K 2 Jun 17:03 _log.py > -rw-r--r-- 1 laurentdumont admin 5.6K 2 Jun 17:03 _services_mixin.py > drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 accelerator > drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 baremetal > drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 > baremetal_introspection > drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 block_storage > drwxr-xr-x 28 laurentdumont admin 896B 2 Jun 17:03 cloud > drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 clustering > drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 compute > drwxr-xr-x 14 laurentdumont admin 448B 2 Jun 17:03 config > -rw-r--r-- 1 laurentdumont admin 21K 2 Jun 17:03 connection.py > drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 database > drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 dns > -rw-r--r-- 1 laurentdumont admin 8.6K 2 Jun 17:03 exceptions.py > drwxr-xr-x 5 laurentdumont admin 160B 2 Jun 17:03 fixture > -rw-r--r-- 1 laurentdumont admin 1.6K 2 Jun 17:03 format.py > drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 identity > drwxr-xr-x 11 laurentdumont admin 352B 2 Jun 17:03 image > drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 instance_ha > drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 key_manager > drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 load_balancer > drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 message > drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 network > drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 object_store > drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 orchestration > drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 placement > -rw-r--r-- 1 laurentdumont admin 29K 2 Jun 17:03 proxy.py > -rw-r--r-- 1 laurentdumont admin 0B 2 Jun 17:03 py.typed > -rw-r--r-- 1 laurentdumont admin 88K 2 Jun 17:03 resource.py > -rw-r--r-- 1 laurentdumont admin 13K 2 Jun 17:03 > service_description.py > drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 shared_file_system > drwxr-xr-x 9 laurentdumont admin 288B 2 Jun 17:03 tests > -rw-r--r-- 1 laurentdumont admin 12K 2 Jun 17:03 utils.py > -rw-r--r-- 1 laurentdumont admin 641B 2 Jun 17:03 version.py > drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 workflow > usr/local/?/site-packages/openstack > % python3 > Python 3.9.2 (default, Mar 26 2021, 23:27:12) > [Clang 12.0.0 (clang-1200.0.32.29)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> > > I cannot see the volume folder (but it's on the opendev) > > The command was added here > https://opendev.org/openstack/python-openstackclient/commit/9647d43bd5e77c815ac2f08b7de2226a657a66bc > > Removing + reinstall the client doesn't seem to work either. > > Any clues? -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From lyarwood at redhat.com Mon Sep 20 17:23:30 2021 From: lyarwood at redhat.com (Lee Yarwood) Date: Mon, 20 Sep 2021 18:23:30 +0100 Subject: [openstack-cli] Missing volume backend commands In-Reply-To: <20210920171611.62nc2mo5ux3m6773@lyarwood-laptop.usersys.redhat.com> References: <20210920171611.62nc2mo5ux3m6773@lyarwood-laptop.usersys.redhat.com> Message-ID: <20210920172330.cc3ee3z5yigchh5b@lyarwood-laptop.usersys.redhat.com> On 20-09-21 18:16:14, Lee Yarwood wrote: > On 20-09-21 12:26:29, Laurent Dumont wrote: > > Hey everyone, > > > > Trying to use the volume backend command to get some information regarding > > storage pools. > > > > https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/volume-backend.html > > > > Not sure if it's something on my side, but I cannot see to access a > > subsection of the volume commands from the CLI client. > > As noted at the top of that page it's only available with v2 API of > Cinder that has recently been dropped. Assuming you still have it in > your env use --os-volume-api-version 2 with osc. > > $ openstack --os-volume-api-version 2 volume backend capability show ${host} I forgot to say that it looks like the API for both these commands made it to v3 so someone just needs to do the leg work in osc to enable them: https://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=list-all-back-end-storage-pools-detail#back-end-storage-pools -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From ashlee at openstack.org Mon Sep 20 18:13:30 2021 From: ashlee at openstack.org (Ashlee Ferguson) Date: Mon, 20 Sep 2021 13:13:30 -0500 Subject: OpenInfra Live - September 23 at 9am CT Message-ID: Hi everyone, This week?s OpenInfra Live episode is brought to you by the OpenInfra Community. Join the volunteer organizers of OpenInfra Days to hear about the happenings at their latest local events, what they're looking forward to in the future, and learn what's happening with OpenInfra all over the world. Episode: OpenInfra Days 2021 Recap Date and time: September 23, 2021 at 9am CT (1400 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=-s1eJa6u_2A LinkedIn: https://www.linkedin.com/feed/update/urn:li:ugcPost:6844374141978734592/ Facebook: https://www.facebook.com/104139126308032/posts/4341771172544785/ WeChat: recording will be posted on OpenStack WeChat after the live stream Speakers: Akihiro Hasegawa, Cloud Operator Days Tokyo Organizer Rico Lin, OpenInfra Days Asia Organizer Sartika Lestari, OpenInfra Days Indonesia Organizer Have an idea for a future episode? Share it now at ideas.openinfra.live . Register now for OpenInfra Live: Keynotes, a special edition of OpenInfra Live on November 17-18th starting at 1500 UTC: https://openinfralivekeynotes.eventbrite.com/ Thanks, Ashlee -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal.arbet at ultimum.io Mon Sep 20 20:38:45 2021 From: michal.arbet at ultimum.io (Michal Arbet) Date: Mon, 20 Sep 2021 22:38:45 +0200 Subject: Changing Default Network Quotas in Openstack In-Reply-To: <20210916124441.dlgw43pyfu4mku7d@yuggoth.org> References: <20210916124441.dlgw43pyfu4mku7d@yuggoth.org> Message-ID: Hi Tony, Kolla-ansible is not rendering [quotas] configuration block in neutron.conf file by default. But you can add this configuration block to /etc/kolla/config/neutron.conf and this will be merged into kolla's final rendered neutron.conf file. Added file : ubuntu at deploy:/opt/kolla-ansible$ cat /etc/kolla/config/neutron.conf [quotas] quota_network = 10 quota_subnet = 10 quota_port = 50 quota_driver = neutron.quota.DbQuotaNoLockDriver Then run kolla-ansible -i inventory reconfigure -t neutron Let me know if you have any questions Thanks, Michal Arbet (kevko) Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Po???? 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet at ultimum.io *https://ultimum.io * LinkedIn | Twitter | Facebook ?t 16. 9. 2021 v 14:51 odes?latel Jeremy Stanley napsal: > On 2021-09-16 07:00:27 +0200 (+0200), Karera Tony wrote: > > I have a working Wallaby Openstack deployed using kolla-ansible. > > > > However, I have not been able to find a way of changing the > > Network quotas as I need to increase the defaults. > > > > Can someone kindly advise on how to go about it ? > > Are you looking for something other than what's already explained in > the Neutron documentation? > > https://docs.openstack.org/neutron/latest/admin/ops-quotas.html > > -- > Jeremy Stanley > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Mon Sep 20 22:04:35 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Mon, 20 Sep 2021 17:04:35 -0500 Subject: [all][tc] Technical Committee next weekly meeting on Sept 23rd at 1500 UTC Message-ID: <17c053d4c0a.be874cfa77515.5986366594963704309@ghanshyammann.com> Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 23rd at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 22th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann From hiwkby at yahoo.com Tue Sep 21 03:38:45 2021 From: hiwkby at yahoo.com (Hirotaka Wakabayashi) Date: Tue, 21 Sep 2021 03:38:45 +0000 (UTC) Subject: [oslo] oslo.metrics package for Fedora References: <1578604251.1736431.1632195525621.ref@mail.yahoo.com> Message-ID: <1578604251.1736431.1632195525621@mail.yahoo.com> Hello Oslo Team, I am a Fedora packager. I want to package oslo.metrics for Fedora because my package uses oslo.messaging that requires oslo.metrics as you know. oslo.messaging package repository already exists in Fedora. I will take over it from the former package maintainer. the oslo.metrics repository doesn't existso I need to make it. If any concerns with it, please reply. I can update the version as soon as the new version releases by using Fedora's release monitoring system. Thanks in advance, Hirotaka Wakabayashi -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Tue Sep 21 05:57:11 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Tue, 21 Sep 2021 07:57:11 +0200 Subject: [oslo] oslo.metrics package for Fedora In-Reply-To: <1578604251.1736431.1632195525621@mail.yahoo.com> References: <1578604251.1736431.1632195525621.ref@mail.yahoo.com> <1578604251.1736431.1632195525621@mail.yahoo.com> Message-ID: <20210921055711.coiwgigvp22imrhd@p1.localdomain> Hi, On Tue, Sep 21, 2021 at 03:38:45AM +0000, Hirotaka Wakabayashi wrote: > Hello Oslo Team, > > I am a Fedora packager. I want to package oslo.metrics for Fedora because > my package uses oslo.messaging that requires oslo.metrics as you know. > oslo.messaging package repository already exists in Fedora. I will take over > it from the former package maintainer. the oslo.metrics repository doesn't existso I need to make it. > > If any concerns with it, please reply. I can update the version as soon as the > new version releases by using Fedora's release monitoring system. > > Thanks in advance, > Hirotaka Wakabayashi Did You check RDO project [1]? I think that it is already there. [1] https://www.rdoproject.org/documentation/rdo-packaging/ -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From hberaud at redhat.com Tue Sep 21 06:07:14 2021 From: hberaud at redhat.com (Herve Beraud) Date: Tue, 21 Sep 2021 08:07:14 +0200 Subject: [release] Release countdown for week R-2, Sep 20 - Sep 24 Message-ID: Development Focus ----------------- At this point we should have release candidates (RC1 or recent intermediary release) for all the Xena deliverables. Teams should be working on any release-critical bugs that would require another RC or intermediary release before the final release. Actions ------- Early in the week, the release team will be proposing stable/xena branch creation for all deliverables that have not branched yet, using the latest available Xena release as the branch point. If your team is ready to go for creating that branch, please let us know by leaving a +1 on these patches. If you would like to wait for another release before branching, you can -1 the patch and update it later in the week with the new release you would like to use. By the end of the week the release team will merge those patches though, unless an exception is granted. Once stable/xena branches are created, if a release-critical bug is detected, you will need to fix the issue in the master branch first, then backport the fix to the stable/xena branch before releasing out of the stable/xena branch. After all of the cycle-with-rc projects have branched we will branch devstack, grenade, and the requirements repos. This will effectively open them up for Yoga development, though the focus should still be on finishing up Xena until the final release. For projects with translations, watch for any translation patches coming through and merge them quickly. A new release should be produced so that translations are included in the final Xena release. Finally, now is a good time to finalize release notes. In particular, consider adding any relevant "prelude" content. Release notes are targeted for the downstream consumers of your project, so it would be great to include any useful information for those that are going to pick up and use or deploy the Xena version of your project. Upcoming Deadlines & Dates -------------------------- Final RC deadline: September 30 (R-1 week) Final Xena release: October 6 -- Herv? Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From artem.goncharov at gmail.com Tue Sep 21 06:09:14 2021 From: artem.goncharov at gmail.com (Artem Goncharov) Date: Tue, 21 Sep 2021 08:09:14 +0200 Subject: [oslo] oslo.metrics package for Fedora In-Reply-To: <20210921055711.coiwgigvp22imrhd@p1.localdomain> References: <1578604251.1736431.1632195525621.ref@mail.yahoo.com> <1578604251.1736431.1632195525621@mail.yahoo.com> <20210921055711.coiwgigvp22imrhd@p1.localdomain> Message-ID: On Tue, Sep 21, 2021, 08:00 Slawek Kaplonski wrote: > Hi, > > On Tue, Sep 21, 2021 at 03:38:45AM +0000, Hirotaka Wakabayashi wrote: > > Hello Oslo Team, > > > > I am a Fedora packager. I want to package oslo.metrics for Fedora because > > my package uses oslo.messaging that requires oslo.metrics as you know. > > oslo.messaging package repository already exists in Fedora. I will take > over > > it from the former package maintainer. the oslo.metrics repository > doesn't existso I need to make it. > > > > If any concerns with it, please reply. I can update the version as soon > as the > > new version releases by using Fedora's release monitoring system. > > > > Thanks in advance, > > Hirotaka Wakabayashi > > Did You check RDO project [1]? I think that it is already there. > RDO is not automatically in Fedora and Fedora already has own packages and sadly in not best shape. Some years ago I tried to use RDO packages in Fedora and failed in something (probably it was back in py2/py3 split days). My personal opinion is that we should either fix Fedora OpenStack packages or completely get rid of them in favor of RDO. In the later issue is in over complexity for those willing only OpenStack SDK/CLI (to which I personally often belong). > [1] https://www.rdoproject.org/documentation/rdo-packaging/ > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat Artem -------------- next part -------------- An HTML attachment was scrubbed... URL: From amotoki at gmail.com Tue Sep 21 12:51:06 2021 From: amotoki at gmail.com (Akihiro Motoki) Date: Tue, 21 Sep 2021 21:51:06 +0900 Subject: [neutron] Bug deputy report (Sep 13 - 19) Message-ID: Hi, Here is my bug deputy report last week. Sorry for late. [Summary] * There are several RC2 candidate bugs. I added a DVR one to Xena RC2 milestone. * dynamic-routing and vpnaas bugs need to be investigated. I would be nice if someone can look into the dynamic-routing one. Regarding vpnaas ones, unfortunately I had no time to check them last week but I will visit them this week. [RC candidates] * https://bugs.launchpad.net/neutron/+bug/1943714 DB session commit error in resource_registry.set_resources_dirty In Progress, High, Fix already approved * https://bugs.launchpad.net/neutron/+bug/1943930 [DHCP] new line symbol in opt_name of extra_dhcp_opt causes dnsmasq to fail In Progress, High, Fix already approved * https://bugs.launchpad.net/neutron/+bug/1943846 Create floating agent gw port for DVR fails In Progress, High, Fix already approved NOTE: It is caused by the change in the notification payload during Xena, so I targeted to Xena-RC2. [QoS] * https://bugs.launchpad.net/neutron/+bug/1943724 Change a QoS policy on a bound port does not handle minimum bandwidth direction change In Progress, Medium NOTE: A fix is proposed but it is in the feature patch series of the min pps rule. Hopefully someone familiar with QoS can check the dependency of the fix. [dynamic-routing] * https://bugs.launchpad.net/neutron/+bug/1943725 Automatic cleanup of BGP speakers is too aggressive New, Unassigned [VPNaaS] * https://bugs.launchpad.net/neutron/+bug/1943449 VPNaaS reconfiguration creates duplicate IPtables rules causes the VPN connection to remain DOWN New, Unassigned * https://bugs.launchpad.net/neutron/+bug/1943716 vpn services / vpn connections are stuck in PENDING CREATE New, Unassigned NOTE: RabbitMQ topic mismatch is reported here. It is interesting. I will check later this week. Thanks, Akihiro Motoki (irc: amotoki) From bkslash at poczta.onet.pl Tue Sep 21 07:44:26 2021 From: bkslash at poczta.onet.pl (bkslash) Date: Tue, 21 Sep 2021 09:44:26 +0200 Subject: RabbitMQ annoying disconnections [kolla-ansible, rabbitmq] In-Reply-To: References: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> <6eb72199-d368-3e8a-a6e7-1c0e8523bc0e@nemebean.com> <02C6F39B-C041-469F-9217-C1EE9FC3CBF7@poczta.onet.pl> Message-ID: <9418AB32-0847-4A84-B844-59EBB045FA63@poczta.onet.pl> Hi, I?ve tried to set heartbeat_in_pthread = true, but nothing has changed after kolla-ansible reconfigure - still got disconnections and heartbeat_in_pthread option was present inside configs on all nodes. But in the meantime I tried to alternate rabbit configuration and I realized that Kolla ignored /etc/kolla/config/rabbitmq.conf file (of course at first I checked permissions, etc.) So I tried to put heartbeat = 600 inside /etc/kolla/config/rabbitmq/rabbitmq.conf - this overwrote rabbitmq.conf totally (I mean Kolla didn?t just add heartbeat = 600 to rabbitmq.conf on each controller node, but put ONLY this line in all rabbitmq.conf files ! - rest of the file was cleared). So finally I tried to create separate config for each controller node. So I copied default content of /etc/kolla/rabbitmq/rabbitmq.conf on each controller (because of course these files differs from each other - they have different IPs inside) to /etc/kolla/config/rabbitmq/controller1(2,3)/rabbitmq.conf And finally heartbeat = 600 started to work - no more disconnections. I left Oslo heartbeats and timeouts intact (with the values I?ve set previously) and I don?t have disconnections from services side too. So there are two questions: 1. Why heartbeat_in_pthread didn?t change situation? And if it?s deprecated it was kind of risky to use it long term (after upgrade to wallaby it would stop working anyway?) 2. Why Kolla-ansible ignored rabbitmq.conf file if it is located in /etc/kolla/config, and overwrites ALL content of rabbitmq.conf file when it is located in /etc/kolla/config/rabbitmq/ ? Best regards Adam Tomas > Wiadomo?? napisana przez Herve Beraud > w dniu 21.09.2021, o godz. 08:12: > > > > Le sam. 18 sept. 2021 ? 16:12, bkslash > a ?crit : > Hi Ben, > Thank you for the answer. I?m using open stack Victoria deployed with Kolla-ansible. > So I assume I should add (in /etc/kolla/config/global.conf) > > [oslo_messaging_rabbit] > heartbeat_in_pthread = True > > > But for Victoria (nova sample configuration file) it seems to be deprecated? > # DEPRECATED: Run the health check heartbeat thread through a native python > # thread by default. If this option is equal to False then the health check > # heartbeat will inherit the execution model from the parent process. For > # example if the parent process has monkey patched the stdlib by using > # eventlet/greenlet then the heartbeat will be run through a green thread > # (boolean value) > # This option is deprecated for removal. > # Its value may be silently ignored in the future. > #heartbeat_in_pthread = true > So will it work? > > > Yes it will work. > > Few months ago I proposed to remove this deprecation and to keep the option available [1], unfortunately my removal isn't yet merged. > Only the default value is changed and the option will be turned on by default. > > [1] https://review.opendev.org/c/openstack/oslo.messaging/+/800621 > > Best regards > Adam Tomas > > >> Wiadomo?? napisana przez Ben Nemec > w dniu 17.09.2021, o godz. 18:57: >> >> This sounds like you're hitting the heartbeat/eventlet bug. Basically eventlet won't schedule the heartbeat thread unless something is regularly waking up the api threads. Depending on what release you're running, you may be able to set heartbeat_in_pthread[0] to fix this. >> >> 0: https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/_drivers/impl_rabbit.py#L90 >> >> On 9/17/21 7:00 AM, Adam Tomas wrote: >>> Hi, >>> after some struggling I have almost ?clear? logs (clear=error free :) ) Almost?. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description: >>> https://bugzilla.redhat.com/show_bug.cgi?id=1711794 > >>> in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.? >>> but? It is reported :( And produces tons of logs. >>> I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :( >>> [oslo_messaging_rabbit] >>> heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4 >>> How to avoid this disconnections?? Or what to do to avoid writing this ?errors?to logs? >>> Best regards >>> Adam Tomas >>> /var/log/kolla/rabbitmq/rabbitXXX.log >>> 2021-09-17 06:43:08.771 [error] <0.16056.678> closing AMQP connection <0.16056.678> (x.x.x.x00:44882 -> x.x.x.x00:5672 - mod_wsgi:959:aaa12af5-bb0b-456e-a129-a466eccc24f0): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:08.773 [info] <0.7412.679> Closing all channels from connection 'x.x.x.x00:44882 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:09.282 [info] <0.4574.679> accepting AMQP connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.283 [info] <0.28410.678> accepting AMQP connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.285 [info] <0.28410.678> Connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394 >>> 2021-09-17 06:43:09.285 [info] <0.4574.679> Connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93 >>> 2021-09-17 06:43:09.286 [info] <0.4574.679> connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672 - magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.286 [info] <0.28410.678> connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672 - magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.504 [error] <0.14465.678> closing AMQP connection <0.14465.678> (x.x.x.x00:44914 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:09.507 [info] <0.14985.679> Closing all channels from connection 'x.x.x.x00:44914 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:09.521 [info] <0.11574.678> accepting AMQP connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.523 [info] <0.11574.678> Connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) has a client-provided name: mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3 >>> 2021-09-17 06:43:09.525 [info] <0.11574.678> connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.654 [info] <0.31575.679> accepting AMQP connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.656 [info] <0.31575.679> Connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7 >>> 2021-09-17 06:43:09.656 [info] <0.31180.678> accepting AMQP connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.658 [info] <0.31575.679> connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672 - magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.658 [info] <0.31180.678> Connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b >>> 2021-09-17 06:43:09.660 [info] <0.2757.679> accepting AMQP connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.660 [info] <0.31180.678> connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672 - magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.662 [info] <0.2757.679> Connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315 >>> 2021-09-17 06:43:09.663 [info] <0.2757.679> connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672 - magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.814 [error] <0.22797.678> closing AMQP connection <0.22797.678> (x.x.x.x05:57676 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:09.817 [info] <0.7903.679> Closing all channels from connection 'x.x.x.x05:57676 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:10.319 [error] <0.18268.678> closing AMQP connection <0.18268.678> (x.x.x.x00:44924 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:10.322 [info] <0.28032.678> Closing all channels from connection 'x.x.x.x00:44924 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:10.330 [info] <0.19596.678> accepting AMQP connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:10.333 [info] <0.19596.678> Connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) has a client-provided name: nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09 >>> 2021-09-17 06:43:10.334 [info] <0.19596.678> connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:10.831 [info] <0.9838.678> accepting AMQP connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:10.834 [info] <0.9838.678> Connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) has a client-provided name: neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e >>> 2021-09-17 06:43:10.835 [info] <0.9838.678> connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:11.738 [error] <0.15275.678> closing AMQP connection <0.15275.678> (x.x.x.x01:53414 -> x.x.x.x00:5672 - mod_wsgi:968:b60f723c-882e-4780-a162-55c03ebe6179): >>> missed heartbeats from client, timeout: 60s >>> /var/log/kolla/glance/glance-api.log >>> 2021-09-17 13:17:23.212 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:17:29.208 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:17:42.190 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:17:42] "GET / HTTP/1.1" 300 1446 0.001949 >>> 2021-09-17 13:17:46.351 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:12.336 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:12] "GET / HTTP/1.1" 300 1446 0.003754 >>> 2021-09-17 13:18:13.703 956 INFO eventlet.wsgi.server [req-1a00db34-349b-4ed3-8625-edfadc939be4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:13] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.139229 >>> 2021-09-17 13:18:33.065 955 INFO eventlet.wsgi.server [req-a57046eb-549e-4837-95c7-d0994c05449e d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.096302 >>> 2021-09-17 13:18:33.077 955 INFO eventlet.wsgi.server [req-3a8af281-6732-4afd-b3ec-e502367bce51 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.003982 >>> 2021-09-17 13:18:36.747 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:36.759 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:42.460 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:42] "GET / HTTP/1.1" 300 1446 0.001383 >>> 2021-09-17 13:18:52.659 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:52.670 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:12.598 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:19:12] "GET / HTTP/1.1" 300 1446 0.001590 >>> 2021-09-17 13:19:16.367 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:27.127 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:27.184 955 INFO eventlet.wsgi.server [req-8f68efc9-f46a-450d-8138-928e4f2585b4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:19:27] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.032917 >>> /var/log/kolla/keystone/keystone.log >>> 2021-09-17 13:13:54.670 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.148 960 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.150 962 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.345 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:56.044 961 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:18.402 966 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:31.372 963 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:36.580 967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:38.771 965 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:16:12.226 964 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> /var/log/kolla/nova/nova-compute.log >>> 2021-09-17 13:10:07.710 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56226. >>> 2021-09-17 13:11:11.473 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:11:34.046 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:11:34.721 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:07.699 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:13:08.734 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56434. >>> 2021-09-17 13:14:11.501 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:34.070 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:34.745 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:16:08.724 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:16:09.756 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56638. >>> 2021-09-17 13:17:34.098 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:09.747 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:19:10.781 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56834. >>> 2021-09-17 13:20:34.124 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:20:34.777 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:20:51.789 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:22:10.772 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:22:11.806 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57038. >>> 2021-09-17 13:23:34.149 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:23:34.803 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:23:51.818 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:25:11.795 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:25:12.842 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57240. >>> 2021-09-17 13:26:34.176 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:26:34.830 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:28:12.812 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:28:13.848 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57446. > > > > -- > Herv? Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ > https://twitter.com/4383hberaud > -------------- next part -------------- An HTML attachment was scrubbed... URL: From iurygregory at gmail.com Tue Sep 21 15:21:44 2021 From: iurygregory at gmail.com (Iury Gregory) Date: Tue, 21 Sep 2021 17:21:44 +0200 Subject: [requirements][release][ironic] Exception for ironic deliverables - Xena Message-ID: Hello everyone, The ironic CI last week was in a very bad shape and we couldn't merge some patches that we wanted to have in the Xena release, we knew last week was the RC1 and we should have proposed the release for our derivables. Today we were able to merge a workaround to get our CI green again, we will probably have all the required patches merged by EOD or tomorrow morning. I would like to ask for an exception so we can release our deliverables tomorrow. Thanks! -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory at gmail.com * -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Tue Sep 21 15:26:52 2021 From: tonykarera at gmail.com (Karera Tony) Date: Tue, 21 Sep 2021 17:26:52 +0200 Subject: Not registering Hypervisors Message-ID: Hello Team, I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance. It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors. Any idea on how to resolve this ? Regards Tony Karera -------------- next part -------------- An HTML attachment was scrubbed... URL: From justin.moravec at wildcardcorp.com Tue Sep 21 15:29:16 2021 From: justin.moravec at wildcardcorp.com (Justin Moravec) Date: Tue, 21 Sep 2021 10:29:16 -0500 Subject: Constant HA Virtual Router Flapping Message-ID: Any ideas welcome, Kolla-ansible Ussuri deployment. Have some projects that are using virtual routers that are in HA setup that are constantly flapping (flipping between one neutron node to another) causing disconnects. This only happens in SOME projects, other projects with HA routers are working as intended. This only affects HA router deployments, non HA routers work as intended no matter the neutron node they are on. Have verified that each virtual router can reach the external network gateway while it is on each of the neutron nodes. Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mthode at mthode.org Tue Sep 21 15:36:20 2021 From: mthode at mthode.org (Matthew Thode) Date: Tue, 21 Sep 2021 10:36:20 -0500 Subject: [requirements][release][ironic] Exception for ironic deliverables - Xena In-Reply-To: References: Message-ID: <20210921153620.tvlgnj5hds37huh4@mthode.org> On 21-09-21 17:21:44, Iury Gregory wrote: > Hello everyone, > > The ironic CI last week was in a very bad shape and we couldn't merge some > patches that we wanted to have in the Xena release, we knew last week was > the RC1 and we should have proposed the release for our derivables. > Today we were able to merge a workaround to get our CI green again, we will > probably have all the required patches merged by EOD or tomorrow morning. > I would like to ask for an exception so we can release our deliverables > tomorrow. > > Thanks! > -- > > > *Att[]'sIury Gregory Melo Ferreira * > *MSc in Computer Science at UFCG* > *Part of the ironic-core and puppet-manager-core team in OpenStack* > *Software Engineer at Red Hat Czech* > *Social*: https://www.linkedin.com/in/iurygregory > *E-mail: iurygregory at gmail.com * No issues on the requirements side. -- Matthew Thode -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From juliaashleykreger at gmail.com Tue Sep 21 15:39:48 2021 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Tue, 21 Sep 2021 08:39:48 -0700 Subject: [requirements][release][ironic] Exception for ironic deliverables - Xena In-Reply-To: References: Message-ID: I don't believe an exception is required. The assets to be released are cycle-with-intermediary and as such Ironic does not cut Release Candidate releases. Our drop dead absolute window for intermediary releases is R-1 per the release schedule[1]. If requirements.txt requires a change, then I believe that will require an explicit individual approval. For what it is worth, everyone's CI is in horrible shape this week. [0]: https://releases.openstack.org/reference/release_models.html#cycle-with-intermediary [1]:https://releases.openstack.org/xena/schedule.html On Tue, Sep 21, 2021 at 8:25 AM Iury Gregory wrote: > > Hello everyone, > > The ironic CI last week was in a very bad shape and we couldn't merge some patches that we wanted to have in the Xena release, we knew last week was the RC1 and we should have proposed the release for our derivables. > Today we were able to merge a workaround to get our CI green again, we will probably have all the required patches merged by EOD or tomorrow morning. > I would like to ask for an exception so we can release our deliverables tomorrow. > > Thanks! > -- > Att[]'s > Iury Gregory Melo Ferreira > MSc in Computer Science at UFCG > Part of the ironic-core and puppet-manager-core team in OpenStack > Software Engineer at Red Hat Czech > Social: https://www.linkedin.com/in/iurygregory > E-mail: iurygregory at gmail.com From ralonsoh at redhat.com Tue Sep 21 16:30:46 2021 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Tue, 21 Sep 2021 18:30:46 +0200 Subject: [neutron] upgrading port binding:profile.allocation value for port-resource-request-groups API extension In-Reply-To: References: Message-ID: Hello Balazs: Sorry for the late reply, I was on PTO. If I'm not wrong, now port['binding:profile']['allocation'] is a UUID and you need it to be a list of UUIDs. Am I correct? To make this change in the DB you should use the Alembic migrations, as you said. That should ensure all registers are translated. We should also include a sanity check to ensure the DB migration was done correctly. Is that what you needed? Don't hesitate to ping me in IRC if needed. Regards. On Fri, Sep 10, 2021 at 6:06 PM Balazs Gibizer wrote: > Hi Neutrinos! > > We found a technical challenge during implementing the > port-resource-request-groups API extension[1]. That extension changes > the format of the port.resoruce_request as well as the format of the > port.binding:profile.allocation. The former is a calculated field on > the port so that is easy. However the bindig:profile is persisted in > the database so data migration is needed. What is the canonical way to > do such DB data translation in Neutron? Can we translate the data in > place during alembic migration? Or should we do some kind of online > data migration when the data is translated by neutron when it is read > from the db? > > cheers, > gibi > > [1] https://review.opendev.org/c/openstack/neutron/+/805637/5 > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 22 01:47:56 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 22 Sep 2021 03:47:56 +0200 Subject: Not registering Hypervisors In-Reply-To: References: Message-ID: Hello Team, I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance. It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors. Any idea on how to resolve this ? Regards Tony Karera -------------- next part -------------- An HTML attachment was scrubbed... URL: From hiwkby at yahoo.com Wed Sep 22 03:01:57 2021 From: hiwkby at yahoo.com (Hirotaka Wakabayashi) Date: Wed, 22 Sep 2021 03:01:57 +0000 (UTC) Subject: [oslo] oslo.metrics package for Fedora In-Reply-To: <20210921055711.coiwgigvp22imrhd@p1.localdomain> References: <1578604251.1736431.1632195525621.ref@mail.yahoo.com> <1578604251.1736431.1632195525621@mail.yahoo.com> <20210921055711.coiwgigvp22imrhd@p1.localdomain> Message-ID: <663593806.87658.1632279717177@mail.yahoo.com> Hello Slawek, Thank you for your kind reply. I will use the RDO's spec file to make the Fedora package. :) My application packed for Fedora is a simple notification listener using oslo.messaging that requires oslo.metrics. As Artem says, any packages in Fedora must resolve the dependencies without using RDO packages. Best Regards, Hirotaka Wakabayashi On Tuesday, September 21, 2021, 02:57:21 PM GMT+9, Slawek Kaplonski wrote: Hi, On Tue, Sep 21, 2021 at 03:38:45AM +0000, Hirotaka Wakabayashi wrote: > Hello Oslo Team, > > I am a Fedora packager. I want to package oslo.metrics for Fedora because > my package uses oslo.messaging that requires oslo.metrics as you know. > oslo.messaging package repository already exists in Fedora. I will take over > it from the former package maintainer. the oslo.metrics repository doesn't existso I need to make it. > > If any concerns with it, please reply. I can update the version as soon as the > new version releases by using Fedora's release monitoring system. > > Thanks in advance, > Hirotaka Wakabayashi Did You check RDO project [1]? I think that it is already there. [1] https://www.rdoproject.org/documentation/rdo-packaging/ -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From ricolin at ricolky.com Wed Sep 22 05:04:05 2021 From: ricolin at ricolky.com (Rico Lin) Date: Wed, 22 Sep 2021 13:04:05 +0800 Subject: [heat] Proposing Brendan Shephard for heat core Message-ID: Hi all I would like to suggest to add Brendan Shephard(bshephar at redhat.com) to the heat core reviewer group. He has been working on heat actively. provide good quality reviews and commits. I think it will be great to have him as the heat core reviewer. *Rico Lin* OIF Individual Board of directors, OpenStack TC, Multi-arch SIG chair, Heat PTL, Senior Software Engineer at EasyStack -------------- next part -------------- An HTML attachment was scrubbed... URL: From ramishra at redhat.com Wed Sep 22 05:30:40 2021 From: ramishra at redhat.com (Rabi Mishra) Date: Wed, 22 Sep 2021 11:00:40 +0530 Subject: [heat] Proposing Brendan Shephard for heat core In-Reply-To: References: Message-ID: On Wed, Sep 22, 2021 at 10:36 AM Rico Lin wrote: > Hi all > I would like to suggest to add > > Brendan Shephard(bshephar at redhat.com) > > to the heat core reviewer group. > > He has been working on heat actively. provide good quality reviews and > commits. > > I think it will be great to have him as the heat core reviewer. > +1, Brendan has a solid background of working with heat(downstream) and would be a good addition to help review/merge upstream patches. > > *Rico Lin* > OIF Individual Board of directors, OpenStack TC, Multi-arch SIG chair, > Heat PTL, > Senior Software Engineer at EasyStack > -- Regards, Rabi Mishra -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Wed Sep 22 09:59:18 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Wed, 22 Sep 2021 11:59:18 +0200 Subject: [neutron] CI update Message-ID: <2741059.qUHmEk80i5@p1> Hi, Due to very urgent things which I had yesterday, I wasn't able to run Neutron CI meeting as usually. Fortunatelly we don't have many new issues in our CI. There is only one new issue in our scenario jobs which I wanted to discuss [1]. It's impacting Ironic gates but I noticed it also in the Neutron CI as well. See [2] or [3] for example. I'm not sure about Ironic jobs but in Neutron I saw it mostly (or only, I'm not sure) in the multinode jobs. [1] https://bugs.launchpad.net/neutron/+bug/1944201 [2] https:// e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ logs/screen-q-agt.txt [3] https://storage.bhs.cloud.ovh.net/v1/ AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/check/ neutron-ovs-tempest-slow/88f8bb7/job-output.txt -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From tomasz.rutkowski at netart.pl Wed Sep 22 10:04:12 2021 From: tomasz.rutkowski at netart.pl (Tomasz Rutkowski) Date: Wed, 22 Sep 2021 12:04:12 +0200 Subject: [monasca][cloudkitty] [kolla-ansible][victoria] metrics metadata Message-ID: <4cc139a6d4983eea33a5ef6d2d76be441c5bb6cd.camel@netart.pl> Hi, perhaps I'm missing something - monasca returns metrics metadata when asked for measurements (/metrics/measurements) however cloudkitty asks only for aggregated data (/metrics/statistics). Is there some way to push cloudkitty to work with monasca also by the field-id mappings (metadata values)? with best regards -- Tomasz Rutkowski -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3728 bytes Desc: not available URL: From micou12 at gmail.com Wed Sep 22 10:35:12 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Wed, 22 Sep 2021 12:35:12 +0200 Subject: Issue on accessing object storage containers from outside Message-ID: Dear all I have created containers and upload objects into containers . I can access these containers and objects trough horizon dashboard but once make them public I can not reach them through the provided link . If any one have an idea on how to access containers and their object using the link from outside he can help and guide. I deployed openstack wallaby using kolla-ansible and ceph pacific using ansible , both are using ubuntu 20.04 as OS Michel -------------- next part -------------- An HTML attachment was scrubbed... URL: From balazs.gibizer at est.tech Wed Sep 22 12:11:30 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Wed, 22 Sep 2021 14:11:30 +0200 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable Message-ID: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Hi, The latest setuptools (58.0) removed support for "use_2to3" [1] (deprecated since 46.2). Many OpenStack modules defines decorator==3.4.0 as lower constraints[2]. However, decorator 3.4.0 cannot be installed anymore with the latest setuptool as it depends on "use_2to3". On master, this can be solved easily by bumping the dependency to decorator 4.0.0. On stable/xena we can still solve it the same way with a new RC. But on older stable branches such solution might be against the stable policy as it would require a major bump on our dependencies. This issue is not limited to lower-constraints testing it just hit us there first. A similar break could happen in our upper constraints as well on old stable branches. The root of the problem is that we always use the latest setuptools in our CI testing even on old stable branches. Zuul's ensure-tox task[4] installs tox which installs the virtualenv package which bundles the latest setuptools package. This happens without applying any constraints. Then tox is used to install the project's dependencies under lower or upper constraints with the unconstrained setuptools available. During and after yesterday's nova meeting [3] we discussed possible solutions. Option 1: Bump the major version of the decorator dependency on stable. Pros: * easy to implement Cons: * might against the policy / breaks downstream packagers * needs to be done in each affected project[3] separately * not future proof. After a future setuptools release we can see a similar break with another of our dependencies. @Stable Cores: how do you feel about such bump? Option 2: Pin the setuptools version during tox installation Pros: * single solution for all affected projects * future proof against breaks introduced by future setuptools releases Cons: * complex change as it probably requires to extend the base task in zuul itself Option 3: turn off lower-constraints testing Pros: * easy to implement * some of us want to get rid of lower constraints anyhow Cons: * needs per project changes * not future proof against similar break affecting our upper constraints on old stable branches. Option 4: utilize pyproject.toml[6] to specify build-time requirements Unfortunately, I'm not sure if it is possible to restrict the maximum setuptools version with it As a side note I tried applying [tox]requires in tox.ini to restrict setuptools version[7] but It does not seem to take effect in our case[8]. @Stable Cores: what do you think what could be the way forward? Cheers, gibi [1] https://setuptools.pypa.io/en/latest/history.html#v58-0-2 [2] https://codesearch.openstack.org/?q=decorator%3D%3D3.4.0&i=nope&literal=nope&files=&excludeFiles=&repos= [3] https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2021-09-21.log.html#t2021-09-21T16:31:49 [4] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks/main.yaml#L28 [5] https://tox.readthedocs.io/en/latest/config.html#conf-requires [6] https://www.python.org/dev/peps/pep-0518/ [7] https://review.opendev.org/c/openstack/placement/+/810293/1/tox.ini [8] https://zuul.opendev.org/t/openstack/build/f78a0300c8734e7c8475309af1d2e1a4/log/tox/lower-constraints-0.log#7 From elod.illes at est.tech Wed Sep 22 12:56:43 2021 From: elod.illes at est.tech (=?UTF-8?B?RWzFkWQgSWxsw6lz?=) Date: Wed, 22 Sep 2021 14:56:43 +0200 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: <6J4UZQ.VOBD0LVDTPUX1@est.tech> References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: Hi, Thanks for the detailed summary gibi! Let me share my view: Option 1: Since the time I am reviewing stable patches I remember that we usually avoided dependency's version bumps (except the case when the new pip dependency resolver started to work properly and revealed that lot of the lower constraints settings were broken) on stable branches. The written rule is "Following the Review guidelines. Specifically, not allowing backports of new features, *new* *dependencies*, or backward incompatible changes" [1]. Option 2: In OpenStack projects (who follows stable policy) we use global requirements, and upper constraints for stable branches. Why not use the same for the tools as well? It caused problem in the past as well (changes in virtualenv, setuptools, etc). In my opinion this would be better or future proof solution. Though it's true, we need to find the proper place and way to add the new constraints when installing tox in ensure-tox zuul role. Option 3: it is really just a temporary solution (maybe worse than Option 1), though it is probably acceptable to unblock gate on stable branches until the appropriate solution will be chosen. Also worth to emphasize that this problem doesn't only affect lower-constraints tests, but also could affect branches where we have old packages (versions) in upper-constraints.txt! Option 4: I'm not familiar with pyproject.toml, but I'm curious about the opinions of those who already used/uses it. Long story short: I'd recommend option 2, as I think that is the best solution and fits the best to OpenStack stable way of working... If that is possible to add in our zuul job roles. Nevertheless, I am curious about further opinions from the community. Thanks, El?d [1] https://docs.openstack.org/project-team-guide/stable-branches.html#active-maintenance On 2021. 09. 22. 14:11, Balazs Gibizer wrote: > Hi, > > > The latest setuptools (58.0) removed support for "use_2to3" [1] > (deprecated since 46.2). Many OpenStack modules defines > decorator==3.4.0 as lower constraints[2]. However, decorator 3.4.0 > cannot be installed anymore with the latest setuptool as it depends on > "use_2to3". > > On master, this can be solved easily by bumping the dependency to > decorator 4.0.0. On stable/xena we can still solve it the same way > with a new RC. But on older stable branches such solution might be > against the stable policy as it would require a major bump on our > dependencies. > > This issue is not limited to lower-constraints testing it just hit us > there first. A similar break could happen in our upper constraints as > well on old stable branches. > > The root of the problem is that we always use the latest setuptools in > our CI testing even on old stable branches. Zuul's ensure-tox task[4] > installs tox which installs the virtualenv package which bundles the > latest setuptools package. This happens without applying any > constraints. Then tox is used to install the project's dependencies > under lower or upper constraints with the unconstrained setuptools > available. > > During and after yesterday's nova meeting [3] we discussed possible > solutions. > > Option 1: Bump the major version of the decorator dependency on stable. > Pros: > * easy to implement > Cons: > * might against the policy / breaks downstream packagers > * needs to be done in each affected project[3] separately > * not future proof. After a future setuptools release we can see a > similar break with another of our dependencies. > > @Stable Cores: how do you feel about such bump? > > > Option 2: Pin the setuptools version during tox installation > Pros: > * single solution for all affected projects > * future proof against breaks introduced by future setuptools releases > Cons: > * complex change as it probably requires to extend the base task in > zuul itself > > > Option 3: turn off lower-constraints testing > Pros: > * easy to implement > * some of us want to get rid of lower constraints anyhow > Cons: > * needs per project changes > * not future proof against similar break affecting our upper > constraints on old stable branches. > > > Option 4: utilize pyproject.toml[6] to specify build-time requirements > Unfortunately, I'm not sure if it is possible to restrict the maximum > setuptools version with it > > > As a side note I tried applying [tox]requires in tox.ini to restrict > setuptools version[7] but It does not seem to take effect in our case[8]. > > > @Stable Cores: what do you think what could be the way forward? > > Cheers, > gibi > > > [1] https://setuptools.pypa.io/en/latest/history.html#v58-0-2 > [2] > https://codesearch.openstack.org/?q=decorator%3D%3D3.4.0&i=nope&literal=nope&files=&excludeFiles=&repos= > [3] > https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2021-09-21.log.html#t2021-09-21T16:31:49 > [4] > https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks/main.yaml#L28 > [5] https://tox.readthedocs.io/en/latest/config.html#conf-requires > [6] https://www.python.org/dev/peps/pep-0518/ > [7] https://review.opendev.org/c/openstack/placement/+/810293/1/tox.ini > [8] > https://zuul.opendev.org/t/openstack/build/f78a0300c8734e7c8475309af1d2e1a4/log/tox/lower-constraints-0.log#7 > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clay.gerrard at gmail.com Wed Sep 22 13:03:12 2021 From: clay.gerrard at gmail.com (Clay Gerrard) Date: Wed, 22 Sep 2021 08:03:12 -0500 Subject: Issue on accessing object storage containers from outside In-Reply-To: References: Message-ID: I'm not familiar with horizon, but it's probably using container ACLs to make the objects in the buckets publicly available: https://docs.openstack.org/swift/latest/overview_acl.html#container-acls Try to using a swift client (like python swiftclient CLI) to check the x-container-read and verify it includes .r:* On Wed, Sep 22, 2021 at 5:39 AM Michel Niyoyita wrote: > Dear all > > I have created containers and upload objects into containers . I can > access these containers and objects trough horizon dashboard but once make > them public I can not reach them through the provided link . > > If any one have an idea on how to access containers and their object using > the link from outside he can help and guide. > > I deployed openstack wallaby using kolla-ansible and ceph pacific using > ansible , both are using ubuntu 20.04 as OS > > Michel > -- Clay Gerrard 210 788 9431 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Wed Sep 22 13:15:13 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 22 Sep 2021 13:15:13 +0000 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: <20210922131513.7h622qyit54lgteq@yuggoth.org> On 2021-09-22 14:56:43 +0200 (+0200), El?d Ill?s wrote: [...] > Option 3: it is really just a temporary solution (maybe worse than > Option 1), though it is probably acceptable to unblock gate on > stable branches until the appropriate solution will be chosen. > Also worth to emphasize that this problem doesn't only affect > lower-constraints tests, but also could affect branches where we > have old packages (versions) in upper-constraints.txt! [...] Do also note that all solutions in this space are effectively temporary. When we drafted the stable branch and extended maintenance policies, we accepted that there comes a time for any job where circumstances may prevent us from being able to effectively continue running it. If the complexity of the solution outweighs the *temporary* benefits of being able to run the job for longer, it may not be worth the effort. While we almost certainly could come up with a non-default replacement for Zuul's standard tox job which allows us to use specific versions of tox, virtualenv, pip, setuptools, et cetera, that added complexity does not come for free. It has a maintenance cost which extends far into the future. Also think back on the times when we've reached for the "easy" solution of pinning such tools in other frameworks like DevStack: everyone's in a rush to stop the bleeding and get back to testing, but nobody cares about fixing the underlying issue and so such things end up pinned for years until that pinning breaks something else because we're blind to all the bitrot around us. And then there's the fact that while we can "fix" things this way in CI jobs, these are ultimately tools for our developers which we just happen to also be able to run in the CI system, not the other way around. Pinning setuptools in the tox jobs does not solve this for developers running tox locally on their own systems. Are we going to start providing them with a list of specific versions of these tools which we're only able to test with, substantially increasing the complexity of setting up local dev environments and making it even harder for newcomers to contribute to the project? The stable branch policy is a set of guidelines and suggestions, not an unbreakable covenant. It's there to help us make sound choices under normal circumstances, but we should not be afraid to come to a collective decision about making policy exceptions in unusual situations such as this one. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From juliaashleykreger at gmail.com Wed Sep 22 13:17:42 2021 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Wed, 22 Sep 2021 06:17:42 -0700 Subject: [neutron] CI update In-Reply-To: <2741059.qUHmEk80i5@p1> References: <2741059.qUHmEk80i5@p1> Message-ID: Greetings Slawek, Ironic, for now, has disabled the firewall code path in our jobs since it is not a critical item for us to run against integration tests, but I've seen mention to the #opendev folks observing the same failure causing the entire openstack gate to reset and re-run jobs. Given the hit counts from the logs where the error is being observed, it seems more likely this issue is a pan-openstack gate issue causing instability at this time. -Julia On Wed, Sep 22, 2021 at 3:04 AM Slawek Kaplonski wrote: > > Hi, > > Due to very urgent things which I had yesterday, I wasn't able to run Neutron > CI meeting as usually. Fortunatelly we don't have many new issues in our CI. > There is only one new issue in our scenario jobs which I wanted to discuss > [1]. It's impacting Ironic gates but I noticed it also in the Neutron CI as > well. See [2] or [3] for example. I'm not sure about Ironic jobs but in > Neutron I saw it mostly (or only, I'm not sure) in the multinode jobs. > > [1] https://bugs.launchpad.net/neutron/+bug/1944201 > [2] https:// > e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ > 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ > logs/screen-q-agt.txt > [3] https://storage.bhs.cloud.ovh.net/v1/ > AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/check/ > neutron-ovs-tempest-slow/88f8bb7/job-output.txt > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat From mahati.chamarthy at gmail.com Wed Sep 22 03:59:54 2021 From: mahati.chamarthy at gmail.com (Mahati Chamarthy) Date: Wed, 22 Sep 2021 09:29:54 +0530 Subject: [Outreachy] Call for mentors, project submission by September 26 In-Reply-To: References: Message-ID: Hi All - here to let you know that the date for Outreachy OpenStack project submission has been extended to September 29th. Please let me know if you have any questions. Best, Mahati On Mon, Sep 20, 2021, 1:08 PM Mahati Chamarthy wrote: > Hello! > > *TL;DR OpenStack is participating in Outreachy for the upcoming December > round!* > *Please submit projects as soon as possible, the deadline is September > 26th, 4 pm UTC:* > *https://www.outreachy.org/communities/cfp/openstack/ > * > > Outreachy's goal is to support people from groups underrepresented in the > technology industry. Interns will work remotely with mentors from our > community. > > We are seeking mentors to propose projects that Outreachy interns can work > on during their internship. If you or your colleagues are interested in > mentoring in the next round, please submit your project proposal by *September > 26th,* *2021 *over here > https://www.outreachy.org/communities/cfp/openstack/ > > Mentors should read the mentor FAQ > https://www.outreachy.org/mentor/mentor-faq and find more details about > the Outreachy program and timeline in > https://www.outreachy.org/communities/cfp/. > > If you want help crafting your project proposal, please contact me Mahati > Chamarthy or Samuel Queiroz < > samueldmq at gmail.com> and we will be glad to assist you. > > Thank you, > Samuel Queiroz > -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Wed Sep 22 13:30:08 2021 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 22 Sep 2021 15:30:08 +0200 Subject: [neutron] CI update In-Reply-To: <2741059.qUHmEk80i5@p1> References: <2741059.qUHmEk80i5@p1> Message-ID: Hi Slawek, Thanks for the summary. Regarding https://bugs.launchpad.net/neutron/+bug/1944201 not sure about if it is related to the number of hosts, there's some failure in singlenode jobs as well: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20Datapath%20Invalid%5C%22 example: https://9f672e0630f459ee81cb-e4093b1756a9a5a7c7d28e6575b4af7f.ssl.cf5.rackcdn.com/805849/10/check/neutron-ovs-rally-task/5981df8/controller/logs/screen-q-agt.txt Lajos (lajoskatona) Slawek Kaplonski ezt ?rta (id?pont: 2021. szept. 22., Sze, 12:10): > Hi, > > Due to very urgent things which I had yesterday, I wasn't able to run > Neutron > CI meeting as usually. Fortunatelly we don't have many new issues in our > CI. > There is only one new issue in our scenario jobs which I wanted to discuss > [1]. It's impacting Ironic gates but I noticed it also in the Neutron CI > as > well. See [2] or [3] for example. I'm not sure about Ironic jobs but in > Neutron I saw it mostly (or only, I'm not sure) in the multinode jobs. > > [1] https://bugs.launchpad.net/neutron/+bug/1944201 > [2] https:// > e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ > 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ > logs/screen-q-agt.txt > > [3] https://storage.bhs.cloud.ovh.net/v1/ > > AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/check/ > neutron-ovs-tempest-slow/88f8bb7/job-output.txt > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From senrique at redhat.com Wed Sep 22 13:38:48 2021 From: senrique at redhat.com (Sofia Enriquez) Date: Wed, 22 Sep 2021 10:38:48 -0300 Subject: [cinder] Bug deputy report for week of 2021-22-09 Message-ID: This is a bug report from 2021-08-09 to 2021-15-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- High - https://bugs.launchpad.net/cinder/+bug/1944118 " cinder-manage db version fails with: AttributeError: 'Engine' object has no attribute 'get_main_option'". Assigned to Stephen Finucane. Medium - https://bugs.launchpad.net/os-brick/+bug/1944474 "iSCSI connections are not reinitiated after host reboot". Assigned to Sophie Huang. - https://bugs.launchpad.net/cinder/+bug/1926412 "When the backup cannot find an available service, update the volume status to none". Assigned to ZhaoYixin - https://bugs.launchpad.net/cinder/+bug/1920729 "PowerStore driver reports no iSCSI targets found". Assigned to Ivan Pchelintsev. Low - https://bugs.launchpad.net/cinder/+bug/1924798 "NetApp ONTAP cannot initialize the driver with SVM scoped mode". Assigned to Felipe Rodrigues. - https://bugs.launchpad.net/cinder/+bug/1924643 "NetApp ONTAP cannot resize a volume LUN with a sub-clone". Assigned to Felipe Rodrigues. Invalid - https://bugs.launchpad.net/cinder/+bug/1922507 "cinder-volume-driver test fails "ModuleNotFoundError: No module named 'fixtures'" Thanks -- Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 22 13:39:00 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 22 Sep 2021 15:39:00 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: References: Message-ID: Hello Team, I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance. It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors. kolla-openstack) stack at deployment:~$ openstack hypervisor list (kolla-openstack) stack at deployment:~$ Any idea on how to resolve this ? Regards Tony Karera -------------- next part -------------- An HTML attachment was scrubbed... URL: From balazs.gibizer at est.tech Wed Sep 22 13:44:24 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Wed, 22 Sep 2021 15:44:24 +0200 Subject: [neutron] upgrading port binding:profile.allocation value for port-resource-request-groups API extension In-Reply-To: References: Message-ID: <0U8UZQ.B1WQBDR9U22T1@est.tech> On Tue, Sep 21 2021 at 06:30:46 PM +0200, Rodolfo Alonso Hernandez wrote: > Hello Balazs: > > Sorry for the late reply, I was on PTO. > > If I'm not wrong, now port['binding:profile']['allocation'] is a UUID > and you need it to be a list of UUIDs. Am I correct? It is a bit more complicated than that. The old value is a single RP UUID. the new value is a dict where the key is the group_id and the value is the RP UUID fulfilling that group. So the transformation needs to access to the group_id. The group_id is a stable UUID generated by neutron server as part of the port.resource_request value, but it is not persisted. > > To make this change in the DB you should use the Alembic migrations, > as you said. That should ensure all registers are translated. We > should also include a sanity check to ensure the DB migration was > done correctly. I'm not 100% sure but I think such data migration can only be done in the contract part as it needs to be done while the neutron server is down as the old code can only use the old data format while the new code can only use the new format. Is it OK to introduce a new contract migration in Yoga in neutron? Cheers, gibi > > Is that what you needed? Don't hesitate to ping me in IRC if needed. > > Regards. > > On Fri, Sep 10, 2021 at 6:06 PM Balazs Gibizer > wrote: >> Hi Neutrinos! >> >> We found a technical challenge during implementing the >> port-resource-request-groups API extension[1]. That extension >> changes >> the format of the port.resoruce_request as well as the format of the >> port.binding:profile.allocation. The former is a calculated field on >> the port so that is easy. However the bindig:profile is persisted in >> the database so data migration is needed. What is the canonical way >> to >> do such DB data translation in Neutron? Can we translate the data in >> place during alembic migration? Or should we do some kind of online >> data migration when the data is translated by neutron when it is >> read >> from the db? >> >> cheers, >> gibi >> >> [1] https://review.opendev.org/c/openstack/neutron/+/805637/5 >> >> >> From skaplons at redhat.com Wed Sep 22 14:28:06 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Wed, 22 Sep 2021 16:28:06 +0200 Subject: [neutron] CI update In-Reply-To: References: <2741059.qUHmEk80i5@p1> Message-ID: <20210922142806.4ovhy3k5zwahkarn@p1.localdomain> Hi On Wed, Sep 22, 2021 at 03:30:08PM +0200, Lajos Katona wrote: > Hi Slawek, > Thanks for the summary. > Regarding https://bugs.launchpad.net/neutron/+bug/1944201 not sure about if > it is related to the number of hosts, there's some failure in > singlenode jobs as well: > http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20Datapath%20Invalid%5C%22 > > example: > https://9f672e0630f459ee81cb-e4093b1756a9a5a7c7d28e6575b4af7f.ssl.cf5.rackcdn.com/805849/10/check/neutron-ovs-rally-task/5981df8/controller/logs/screen-q-agt.txt Ok. So it happens on all types of jobs where neutron-ovs-agent is used :/ > > Lajos (lajoskatona) > > Slawek Kaplonski ezt ?rta (id?pont: 2021. szept. 22., > Sze, 12:10): > > > Hi, > > > > Due to very urgent things which I had yesterday, I wasn't able to run > > Neutron > > CI meeting as usually. Fortunatelly we don't have many new issues in our > > CI. > > There is only one new issue in our scenario jobs which I wanted to discuss > > [1]. It's impacting Ironic gates but I noticed it also in the Neutron CI > > as > > well. See [2] or [3] for example. I'm not sure about Ironic jobs but in > > Neutron I saw it mostly (or only, I'm not sure) in the multinode jobs. > > > > [1] https://bugs.launchpad.net/neutron/+bug/1944201 > > [2] https:// > > e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ > > 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ > > logs/screen-q-agt.txt > > > > [3] https://storage.bhs.cloud.ovh.net/v1/ > > > > AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/check/ > > neutron-ovs-tempest-slow/88f8bb7/job-output.txt > > > > -- > > Slawek Kaplonski > > Principal Software Engineer > > Red Hat -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From smooney at redhat.com Wed Sep 22 14:31:18 2021 From: smooney at redhat.com (Sean Mooney) Date: Wed, 22 Sep 2021 15:31:18 +0100 Subject: Openstack hypervisor list is empty In-Reply-To: References: Message-ID: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > Hello Team, > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and > ceph as the backend storage for Nova, Cinder and Glance. > > It finished with no error but it has failed to register any on the Compute > Nodes under Hypervisors. > > kolla-openstack) stack at deployment:~$ openstack hypervisor list > > (kolla-openstack) stack at deployment:~$ > > > Any idea on how to resolve this ? that usually means that somehthing prevented the comptue agent form strating properly for example incorrect ceph keyrings there are several other case but you mentioned you are using ceph. if this is hte case you should see error in the compute agent log. > > Regards > > Tony Karera From laurentfdumont at gmail.com Wed Sep 22 14:46:29 2021 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Wed, 22 Sep 2021 10:46:29 -0400 Subject: Openstack hypervisor list is empty In-Reply-To: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> Message-ID: It could also be a compute cell discovery issue maybe? Do you see anything under "openstack compute service list"? On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney wrote: > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > Hello Team, > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 > and > > ceph as the backend storage for Nova, Cinder and Glance. > > > > It finished with no error but it has failed to register any on the > Compute > > Nodes under Hypervisors. > > > > kolla-openstack) stack at deployment:~$ openstack hypervisor list > > > > (kolla-openstack) stack at deployment:~$ > > > > > > Any idea on how to resolve this ? > that usually means that somehthing prevented the comptue agent form > strating properly > > for example incorrect ceph keyrings there are several other case but you > mentioned you are > using ceph. > > if this is hte case you should see error in the compute agent log. > > > > > Regards > > > > Tony Karera > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Wed Sep 22 14:50:19 2021 From: smooney at redhat.com (Sean Mooney) Date: Wed, 22 Sep 2021 15:50:19 +0100 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> Message-ID: On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > It could also be a compute cell discovery issue maybe? no they shoudl still show up in the hypervior list api > > Do you see anything under "openstack compute service list"? if they show up in the service list but not they hyperiors api it means that the comptue service started and registered its service entry but something broke it before it could create a compute node recored in the db. with ceph the case i have hit this most often is when the keyright used by nova to get the avaiable capastiy of the ceph cluster is wrong whihc prevent the resoucetack and compute manager form actully creating the compute node record. it can happen for other reason too but best place to start is check if there is an error in the nova compute agent log and go from there. > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney wrote: > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > Hello Team, > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 > > and > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > It finished with no error but it has failed to register any on the > > Compute > > > Nodes under Hypervisors. > > > > > > kolla-openstack) stack at deployment:~$ openstack hypervisor list > > > > > > (kolla-openstack) stack at deployment:~$ > > > > > > > > > Any idea on how to resolve this ? > > that usually means that somehthing prevented the comptue agent form > > strating properly > > > > for example incorrect ceph keyrings there are several other case but you > > mentioned you are > > using ceph. > > > > if this is hte case you should see error in the compute agent log. > > > > > > > > Regards > > > > > > Tony Karera > > > > > > > > From ralonsoh at redhat.com Wed Sep 22 14:55:34 2021 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Wed, 22 Sep 2021 16:55:34 +0200 Subject: [neutron] CI update In-Reply-To: <20210922142806.4ovhy3k5zwahkarn@p1.localdomain> References: <2741059.qUHmEk80i5@p1> <20210922142806.4ovhy3k5zwahkarn@p1.localdomain> Message-ID: Hello folks: I replied in [1]. I think we could have a potential problem when executing a DB change the impies a OF controller re-initialization. Most of the time we don't have problems but as we see in the CI, we could sometimes. I'll push a patch to add a retry decorator on the methods that trigger this OF controller restart. Regards. [1]https://bugs.launchpad.net/neutron/+bug/1944201/comments/4 On Wed, Sep 22, 2021 at 4:36 PM Slawek Kaplonski wrote: > Hi > > On Wed, Sep 22, 2021 at 03:30:08PM +0200, Lajos Katona wrote: > > Hi Slawek, > > Thanks for the summary. > > Regarding https://bugs.launchpad.net/neutron/+bug/1944201 not sure > about if > > it is related to the number of hosts, there's some failure in > > singlenode jobs as well: > > > http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20Datapath%20Invalid%5C%22 > > > > example: > > > https://9f672e0630f459ee81cb-e4093b1756a9a5a7c7d28e6575b4af7f.ssl.cf5.rackcdn.com/805849/10/check/neutron-ovs-rally-task/5981df8/controller/logs/screen-q-agt.txt > > Ok. So it happens on all types of jobs where neutron-ovs-agent is used :/ > > > > > Lajos (lajoskatona) > > > > Slawek Kaplonski ezt ?rta (id?pont: 2021. szept. > 22., > > Sze, 12:10): > > > > > Hi, > > > > > > Due to very urgent things which I had yesterday, I wasn't able to run > > > Neutron > > > CI meeting as usually. Fortunatelly we don't have many new issues in > our > > > CI. > > > There is only one new issue in our scenario jobs which I wanted to > discuss > > > [1]. It's impacting Ironic gates but I noticed it also in the Neutron > CI > > > as > > > well. See [2] or [3] for example. I'm not sure about Ironic jobs but in > > > Neutron I saw it mostly (or only, I'm not sure) in the multinode jobs. > > > > > > [1] https://bugs.launchpad.net/neutron/+bug/1944201 > > > [2] https:// > > > > e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ > > > > 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ > > > logs/screen-q-agt.txt > > > < > http://e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/logs/screen-q-agt.txt > > > > > [3] https://storage.bhs.cloud.ovh.net/v1/ > > > > > > > AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/check/ > > > neutron-ovs-tempest-slow/88f8bb7/job-output.txt > > > > > > -- > > > Slawek Kaplonski > > > Principal Software Engineer > > > Red Hat > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 22 15:08:03 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 22 Sep 2021 17:08:03 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> Message-ID: Hello Guys, Thanks a lot. I had actually checked the nova -compute.log on the compute node and they were showing the error I will post at the end about the cinder keyring but I know its correct because its the same I was using on wallaby, I even tried to use another ceph cluster with ofcouse different keyrings but its the same issue. Below is the error r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc05a8b8) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc060698) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbce2f4e020) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n[errno 2] RADOS object not found (error connecting to the cluster)\n' 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of the above exception, another exception occurred: Regards Tony Karera On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney wrote: > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > > It could also be a compute cell discovery issue maybe? > no they shoudl still show up in the hypervior list api > > > > Do you see anything under "openstack compute service list"? > if they show up in the service list but not they hyperiors api it > means that the comptue service started and registered its service entry but > something broke it before it could create a compute node recored in the db. > > with ceph the case i have hit this most often is when the keyright used by > nova to > get the avaiable capastiy of the ceph cluster is wrong whihc prevent the > resoucetack and compute manager > form actully creating the compute node record. > > > it can happen for other reason too but best place to start is check if > there is an error in the nova compute agent log and go from there. > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney wrote: > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > > Hello Team, > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu > 20.04 > > > and > > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > > > It finished with no error but it has failed to register any on the > > > Compute > > > > Nodes under Hypervisors. > > > > > > > > kolla-openstack) stack at deployment:~$ openstack hypervisor list > > > > > > > > (kolla-openstack) stack at deployment:~$ > > > > > > > > > > > > Any idea on how to resolve this ? > > > that usually means that somehthing prevented the comptue agent form > > > strating properly > > > > > > for example incorrect ceph keyrings there are several other case but > you > > > mentioned you are > > > using ceph. > > > > > > if this is hte case you should see error in the compute agent log. > > > > > > > > > > > Regards > > > > > > > > Tony Karera > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralonsoh at redhat.com Wed Sep 22 15:09:17 2021 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Wed, 22 Sep 2021 17:09:17 +0200 Subject: [neutron] upgrading port binding:profile.allocation value for port-resource-request-groups API extension In-Reply-To: <0U8UZQ.B1WQBDR9U22T1@est.tech> References: <0U8UZQ.B1WQBDR9U22T1@est.tech> Message-ID: Hello Balazs: About the group_id, I see this is built using the port_id and the qos_rules. We have all of this in the DB and we can build it statically (I think so, maybe I'm very optimistic). About the code, that was something I was thinking about after sending the last mail. For at least two releases, we need to support both RP formats in the DB. If we read only the UUID (old format), then we should convert it and store it in the new format. About the migration, we don't support contract migrations anymore. But this is not true as we have done some migrations that have added new restrictions in the DB schema. In any case, this could be done as an expansion migration. If the code is in place, I don't see any problem of doing this migration with the server running. Each "ml2_port_bindings" register will be updated atomically, while the Neutron server will be able to handle both versions. Regards. On Wed, Sep 22, 2021 at 3:44 PM Balazs Gibizer wrote: > > > On Tue, Sep 21 2021 at 06:30:46 PM +0200, Rodolfo Alonso Hernandez > wrote: > > Hello Balazs: > > > > Sorry for the late reply, I was on PTO. > > > > If I'm not wrong, now port['binding:profile']['allocation'] is a UUID > > and you need it to be a list of UUIDs. Am I correct? > > It is a bit more complicated than that. The old value is a single RP > UUID. the new value is a dict where the key is the group_id and the > value is the RP UUID fulfilling that group. So the transformation needs > to access to the group_id. > The group_id is a stable UUID generated by neutron server as part of > the port.resource_request value, but it is not persisted. > > > > > To make this change in the DB you should use the Alembic migrations, > > as you said. That should ensure all registers are translated. We > > should also include a sanity check to ensure the DB migration was > > done correctly. > > I'm not 100% sure but I think such data migration can only be done in > the contract part as it needs to be done while the neutron server is > down as the old code can only use the old data format while the new > code can only use the new format. Is it OK to introduce a new contract > migration in Yoga in neutron? > > Cheers, > gibi > > > > > > Is that what you needed? Don't hesitate to ping me in IRC if needed. > > > > Regards. > > > > On Fri, Sep 10, 2021 at 6:06 PM Balazs Gibizer > > wrote: > >> Hi Neutrinos! > >> > >> We found a technical challenge during implementing the > >> port-resource-request-groups API extension[1]. That extension > >> changes > >> the format of the port.resoruce_request as well as the format of the > >> port.binding:profile.allocation. The former is a calculated field on > >> the port so that is easy. However the bindig:profile is persisted in > >> the database so data migration is needed. What is the canonical way > >> to > >> do such DB data translation in Neutron? Can we translate the data in > >> place during alembic migration? Or should we do some kind of online > >> data migration when the data is translated by neutron when it is > >> read > >> from the db? > >> > >> cheers, > >> gibi > >> > >> [1] https://review.opendev.org/c/openstack/neutron/+/805637/5 > >> > >> > >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Wed Sep 22 15:20:52 2021 From: tonykarera at gmail.com (Karera Tony) Date: Wed, 22 Sep 2021 17:20:52 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> Message-ID: Just to add on that, compute service is listed, I can create Volumes, I have the same cinder keyring in the /etc/kolla/config/nova directory as I have in the /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring Regards Tony Karera On Wed, Sep 22, 2021 at 5:08 PM Karera Tony wrote: > Hello Guys, > > Thanks a lot. > > I had actually checked the nova -compute.log on the compute node and they > were showing the error I will post at the end about the cinder keyring but > I know its correct because its the same I was using on wallaby, I even > tried to use another ceph cluster with ofcouse different keyrings but its > the same issue. > > Below is the error > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to > find a keyring on > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 > AuthRegistry(0x7fbcdc05a8b8) no keyring found at > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > to find a keyring on > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > AuthRegistry(0x7fbcdc060698) no keyring found at > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > to find a keyring on > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > AuthRegistry(0x7fbce2f4e020) no keyring found at > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > disabling cephx\n[errno 2] RADOS object not found (error connecting to the > cluster)\n' > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of > the above exception, another exception occurred: > Regards > > Tony Karera > > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney wrote: > >> On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >> > It could also be a compute cell discovery issue maybe? >> no they shoudl still show up in the hypervior list api >> > >> > Do you see anything under "openstack compute service list"? >> if they show up in the service list but not they hyperiors api it >> means that the comptue service started and registered its service entry >> but >> something broke it before it could create a compute node recored in the >> db. >> >> with ceph the case i have hit this most often is when the keyright used >> by nova to >> get the avaiable capastiy of the ceph cluster is wrong whihc prevent the >> resoucetack and compute manager >> form actully creating the compute node record. >> >> >> it can happen for other reason too but best place to start is check if >> there is an error in the nova compute agent log and go from there. >> > >> > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney >> wrote: >> > >> > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >> > > > Hello Team, >> > > > >> > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu >> 20.04 >> > > and >> > > > ceph as the backend storage for Nova, Cinder and Glance. >> > > > >> > > > It finished with no error but it has failed to register any on the >> > > Compute >> > > > Nodes under Hypervisors. >> > > > >> > > > kolla-openstack) stack at deployment:~$ openstack hypervisor list >> > > > >> > > > (kolla-openstack) stack at deployment:~$ >> > > > >> > > > >> > > > Any idea on how to resolve this ? >> > > that usually means that somehthing prevented the comptue agent form >> > > strating properly >> > > >> > > for example incorrect ceph keyrings there are several other case but >> you >> > > mentioned you are >> > > using ceph. >> > > >> > > if this is hte case you should see error in the compute agent log. >> > > >> > > > >> > > > Regards >> > > > >> > > > Tony Karera >> > > >> > > >> > > >> > > >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Wed Sep 22 15:39:03 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Wed, 22 Sep 2021 08:39:03 -0700 Subject: =?UTF-8?Q?Re:_[stable][requirements][zuul]_unpinned_setuptools_dependenc?= =?UTF-8?Q?y_on_stable?= In-Reply-To: <6J4UZQ.VOBD0LVDTPUX1@est.tech> References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: On Wed, Sep 22, 2021, at 5:11 AM, Balazs Gibizer wrote: > Hi, > Snip > > The root of the problem is that we always use the latest setuptools in > our CI testing even on old stable branches. Zuul's ensure-tox task[4] > installs tox which installs the virtualenv package which bundles the > latest setuptools package. This happens without applying any > constraints. Then tox is used to install the project's dependencies > under lower or upper constraints with the unconstrained setuptools > available. > > During and after yesterday's nova meeting [3] we discussed possible > solutions. > > Option 1: Bump the major version of the decorator dependency on stable. > Pros: > * easy to implement > Cons: > * might against the policy / breaks downstream packagers > * needs to be done in each affected project[3] separately > * not future proof. After a future setuptools release we can see a > similar break with another of our dependencies. > > @Stable Cores: how do you feel about such bump? > > > Option 2: Pin the setuptools version during tox installation > Pros: > * single solution for all affected projects > * future proof against breaks introduced by future setuptools releases > Cons: > * complex change as it probably requires to extend the base task in > zuul itself Note as mentioned during the meeting yesterday I believe you actually need to pin virtualenv during the tox installation. If you want to pin setuptools it needs to happen when tox creates its virtualenvs. One major con to this approach is now you've stopped testing that your software can deploy with newer versions of setuptools. It doesn't work right this moment, but as circumstances change you'll be without up to date information. > > > Option 3: turn off lower-constraints testing > Pros: > * easy to implement > * some of us want to get rid of lower constraints anyhow > Cons: > * needs per project changes > * not future proof against similar break affecting our upper > constraints on old stable branches. > > > Option 4: utilize pyproject.toml[6] to specify build-time requirements > Unfortunately, I'm not sure if it is possible to restrict the maximum > setuptools version with it > > > As a side note I tried applying [tox]requires in tox.ini to restrict > setuptools version[7] but It does not seem to take effect in our > case[8]. This didn't work because the requires specification seems to only affect the meta venv that tox uses to bootstrap itself if necessary [9]. Basically this pinned setuptools in the wrong virtualenv. This might work if you pinned virtualenv here as suggested above. > > > @Stable Cores: what do you think what could be the way forward? > > Cheers, > gibi > > > [1] https://setuptools.pypa.io/en/latest/history.html#v58-0-2 > [2] > https://codesearch.openstack.org/?q=decorator%3D%3D3.4.0&i=nope&literal=nope&files=&excludeFiles=&repos= > [3] > https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2021-09-21.log.html#t2021-09-21T16:31:49 > [4] > https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks/main.yaml#L28 > [5] https://tox.readthedocs.io/en/latest/config.html#conf-requires > [6] https://www.python.org/dev/peps/pep-0518/ > [7] https://review.opendev.org/c/openstack/placement/+/810293/1/tox.ini > [8] > https://zuul.opendev.org/t/openstack/build/f78a0300c8734e7c8475309af1d2e1a4/log/tox/lower-constraints-0.log#7 [9] https://tox.readthedocs.io/en/latest/example/basic.html#tox-auto-provisioning From cboylan at sapwetik.org Wed Sep 22 15:44:32 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Wed, 22 Sep 2021 08:44:32 -0700 Subject: =?UTF-8?Q?Re:_[stable][requirements][zuul]_unpinned_setuptools_dependenc?= =?UTF-8?Q?y_on_stable?= In-Reply-To: <20210922131513.7h622qyit54lgteq@yuggoth.org> References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> <20210922131513.7h622qyit54lgteq@yuggoth.org> Message-ID: <42e13e66-3468-4a1c-b03c-b5b361a13e81@www.fastmail.com> On Wed, Sep 22, 2021, at 6:15 AM, Jeremy Stanley wrote: > On 2021-09-22 14:56:43 +0200 (+0200), El?d Ill?s wrote: > [...] >> Option 3: it is really just a temporary solution (maybe worse than >> Option 1), though it is probably acceptable to unblock gate on >> stable branches until the appropriate solution will be chosen. >> Also worth to emphasize that this problem doesn't only affect >> lower-constraints tests, but also could affect branches where we >> have old packages (versions) in upper-constraints.txt! > [...] > > Do also note that all solutions in this space are effectively > temporary. When we drafted the stable branch and extended > maintenance policies, we accepted that there comes a time for any > job where circumstances may prevent us from being able to > effectively continue running it. If the complexity of the solution > outweighs the *temporary* benefits of being able to run the job for > longer, it may not be worth the effort. > > While we almost certainly could come up with a non-default > replacement for Zuul's standard tox job which allows us to use > specific versions of tox, virtualenv, pip, setuptools, et cetera, > that added complexity does not come for free. It has a maintenance > cost which extends far into the future. Also think back on the times > when we've reached for the "easy" solution of pinning such tools in > other frameworks like DevStack: everyone's in a rush to stop the > bleeding and get back to testing, but nobody cares about fixing the > underlying issue and so such things end up pinned for years until > that pinning breaks something else because we're blind to all the > bitrot around us. > > And then there's the fact that while we can "fix" things this way in > CI jobs, these are ultimately tools for our developers which we just > happen to also be able to run in the CI system, not the other way > around. Pinning setuptools in the tox jobs does not solve this for > developers running tox locally on their own systems. Are we going to > start providing them with a list of specific versions of these tools > which we're only able to test with, substantially increasing the > complexity of setting up local dev environments and making it even > harder for newcomers to contribute to the project? Related to this is the trap that we can often see passing CI as the goal. In reality the goal should be building quality reliable software that functions properly. The testing and CI gives us a measurement against that goal. We do pre merge testing so that we can avoid merging code that does not function. Occasionally external factors break functionality and in those cases the CI and testing have done their job. We are alerted to non functioning software that needs to be fixed. In this case we need to be careful to avoid fixing this only in a CI context because we see passing CI as the goal. Instead we should try to fix the software so that it is functional for the CI system, developers on their local machines, and the individuals/orgs that ultimately deploy the software into production. > > The stable branch policy is a set of guidelines and suggestions, not > an unbreakable covenant. It's there to help us make sound choices > under normal circumstances, but we should not be afraid to come to a > collective decision about making policy exceptions in unusual > situations such as this one. > -- > Jeremy Stanley From balazs.gibizer at est.tech Wed Sep 22 16:02:40 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Wed, 22 Sep 2021 18:02:40 +0200 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: <20210922131513.7h622qyit54lgteq@yuggoth.org> References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> <20210922131513.7h622qyit54lgteq@yuggoth.org> Message-ID: On Wed, Sep 22 2021 at 01:15:13 PM +0000, Jeremy Stanley wrote: > On 2021-09-22 14:56:43 +0200 (+0200), El?d Ill?s wrote: > [...] >> Option 3: it is really just a temporary solution (maybe worse than >> Option 1), though it is probably acceptable to unblock gate on >> stable branches until the appropriate solution will be chosen. >> Also worth to emphasize that this problem doesn't only affect >> lower-constraints tests, but also could affect branches where we >> have old packages (versions) in upper-constraints.txt! > [...] > > Do also note that all solutions in this space are effectively > temporary. When we drafted the stable branch and extended > maintenance policies, we accepted that there comes a time for any > job where circumstances may prevent us from being able to > effectively continue running it. If the complexity of the solution > outweighs the *temporary* benefits of being able to run the job for > longer, it may not be worth the effort. As the issue affects even the recent stable branches this would effectively mean we lose all the lower-constraints testing. And if later the same thing happen with one of our upper requirements then we will to lose all the testing. I think we cannot simply say that we ignore the problem by turning off the job. > > While we almost certainly could come up with a non-default > replacement for Zuul's standard tox job which allows us to use > specific versions of tox, virtualenv, pip, setuptools, et cetera, > that added complexity does not come for free. It has a maintenance > cost which extends far into the future. Also think back on the times > when we've reached for the "easy" solution of pinning such tools in > other frameworks like DevStack: everyone's in a rush to stop the > bleeding and get back to testing, but nobody cares about fixing the > underlying issue and so such things end up pinned for years until > that pinning breaks something else because we're blind to all the > bitrot around us. > > And then there's the fact that while we can "fix" things this way in > CI jobs, these are ultimately tools for our developers which we just > happen to also be able to run in the CI system, not the other way > around. Pinning setuptools in the tox jobs does not solve this for > developers running tox locally on their own systems. Are we going to > start providing them with a list of specific versions of these tools > which we're only able to test with, substantially increasing the > complexity of setting up local dev environments and making it even > harder for newcomers to contribute to the project? > > The stable branch policy is a set of guidelines and suggestions, not > an unbreakable covenant. It's there to help us make sound choices > under normal circumstances, but we should not be afraid to come to a > collective decision about making policy exceptions in unusual > situations such as this one. Does it mean you suggest bumping the decorator dependency to 4.0.0? > -- > Jeremy Stanley From balazs.gibizer at est.tech Wed Sep 22 16:06:49 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Wed, 22 Sep 2021 18:06:49 +0200 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: On Wed, Sep 22 2021 at 08:39:03 AM -0700, Clark Boylan wrote: > On Wed, Sep 22, 2021, at 5:11 AM, Balazs Gibizer wrote: >> Hi, >> > > Snip > >> >> The root of the problem is that we always use the latest setuptools >> in >> our CI testing even on old stable branches. Zuul's ensure-tox >> task[4] >> installs tox which installs the virtualenv package which bundles the >> latest setuptools package. This happens without applying any >> constraints. Then tox is used to install the project's dependencies >> under lower or upper constraints with the unconstrained setuptools >> available. >> >> During and after yesterday's nova meeting [3] we discussed possible >> solutions. >> >> Option 1: Bump the major version of the decorator dependency on >> stable. >> Pros: >> * easy to implement >> Cons: >> * might against the policy / breaks downstream packagers >> * needs to be done in each affected project[3] separately >> * not future proof. After a future setuptools release we can see a >> similar break with another of our dependencies. >> >> @Stable Cores: how do you feel about such bump? >> >> >> Option 2: Pin the setuptools version during tox installation >> Pros: >> * single solution for all affected projects >> * future proof against breaks introduced by future setuptools >> releases >> Cons: >> * complex change as it probably requires to extend the base task in >> zuul itself > > Note as mentioned during the meeting yesterday I believe you actually > need to pin virtualenv during the tox installation. If you want to > pin setuptools it needs to happen when tox creates its virtualenvs. good point > > One major con to this approach is now you've stopped testing that > your software can deploy with newer versions of setuptools. It > doesn't work right this moment, but as circumstances change you'll be > without up to date information. I agree not to pin this on master. We need to support the latest and greatest on master. But on stable I don't see the problem saying we only support the tooling version that was the latest when the major release was created. > >> >> >> Option 3: turn off lower-constraints testing >> Pros: >> * easy to implement >> * some of us want to get rid of lower constraints anyhow >> Cons: >> * needs per project changes >> * not future proof against similar break affecting our upper >> constraints on old stable branches. >> >> >> Option 4: utilize pyproject.toml[6] to specify build-time >> requirements >> Unfortunately, I'm not sure if it is possible to restrict the >> maximum >> setuptools version with it >> >> >> As a side note I tried applying [tox]requires in tox.ini to restrict >> setuptools version[7] but It does not seem to take effect in our >> case[8]. > > This didn't work because the requires specification seems to only > affect the meta venv that tox uses to bootstrap itself if necessary > [9]. Basically this pinned setuptools in the wrong virtualenv. This > might work if you pinned virtualenv here as suggested above. Thanks for the pointers. I think I managed to make this work locally. We will see if it works in CI too when [1] goes through the check pipeline. Cheers, gibi [1] https://review.opendev.org/c/openstack/nova/+/810461 From balazs.gibizer at est.tech Wed Sep 22 16:11:46 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Wed, 22 Sep 2021 18:11:46 +0200 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: <42e13e66-3468-4a1c-b03c-b5b361a13e81@www.fastmail.com> References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> <20210922131513.7h622qyit54lgteq@yuggoth.org> <42e13e66-3468-4a1c-b03c-b5b361a13e81@www.fastmail.com> Message-ID: On Wed, Sep 22 2021 at 08:44:32 AM -0700, Clark Boylan wrote: > On Wed, Sep 22, 2021, at 6:15 AM, Jeremy Stanley wrote: >> On 2021-09-22 14:56:43 +0200 (+0200), El?d Ill?s wrote: >> [...] >>> Option 3: it is really just a temporary solution (maybe worse than >>> Option 1), though it is probably acceptable to unblock gate on >>> stable branches until the appropriate solution will be chosen. >>> Also worth to emphasize that this problem doesn't only affect >>> lower-constraints tests, but also could affect branches where we >>> have old packages (versions) in upper-constraints.txt! >> [...] >> >> Do also note that all solutions in this space are effectively >> temporary. When we drafted the stable branch and extended >> maintenance policies, we accepted that there comes a time for any >> job where circumstances may prevent us from being able to >> effectively continue running it. If the complexity of the solution >> outweighs the *temporary* benefits of being able to run the job for >> longer, it may not be worth the effort. >> >> While we almost certainly could come up with a non-default >> replacement for Zuul's standard tox job which allows us to use >> specific versions of tox, virtualenv, pip, setuptools, et cetera, >> that added complexity does not come for free. It has a maintenance >> cost which extends far into the future. Also think back on the times >> when we've reached for the "easy" solution of pinning such tools in >> other frameworks like DevStack: everyone's in a rush to stop the >> bleeding and get back to testing, but nobody cares about fixing the >> underlying issue and so such things end up pinned for years until >> that pinning breaks something else because we're blind to all the >> bitrot around us. >> >> And then there's the fact that while we can "fix" things this way in >> CI jobs, these are ultimately tools for our developers which we just >> happen to also be able to run in the CI system, not the other way >> around. Pinning setuptools in the tox jobs does not solve this for >> developers running tox locally on their own systems. Are we going to >> start providing them with a list of specific versions of these tools >> which we're only able to test with, substantially increasing the >> complexity of setting up local dev environments and making it even >> harder for newcomers to contribute to the project? > > Related to this is the trap that we can often see passing CI as the > goal. In reality the goal should be building quality reliable > software that functions properly. The testing and CI gives us a > measurement against that goal. We do pre merge testing so that we can > avoid merging code that does not function. Occasionally external > factors break functionality and in those cases the CI and testing > have done their job. We are alerted to non functioning software that > needs to be fixed. > > In this case we need to be careful to avoid fixing this only in a CI > context because we see passing CI as the goal. Instead we should try > to fix the software so that it is functional for the CI system, > developers on their local machines, and the individuals/orgs that > ultimately deploy the software into production. The [tox]requires based fix helps the CI and the developer locally as both are using tox. But it does not help individulas/orgs that are trying to deploy OpenStack with the lowest possible dependencies we say we support. To help them we have to signal either that our decorator lower constraint is now moved from 3.4.0 to 4.0.0 in all our stable branches OR that stable branches does not support setuptools 58.0.0 or newer. Which message we want to send? Cheers, gibi > >> >> The stable branch policy is a set of guidelines and suggestions, not >> an unbreakable covenant. It's there to help us make sound choices >> under normal circumstances, but we should not be afraid to come to a >> collective decision about making policy exceptions in unusual >> situations such as this one. >> -- >> Jeremy Stanley > From fungi at yuggoth.org Wed Sep 22 16:15:15 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 22 Sep 2021 16:15:15 +0000 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> <20210922131513.7h622qyit54lgteq@yuggoth.org> Message-ID: <20210922161515.42353vdjpegr52vk@yuggoth.org> On 2021-09-22 18:02:40 +0200 (+0200), Balazs Gibizer wrote: [...] > As the issue affects even the recent stable branches this would > effectively mean we lose all the lower-constraints testing. I've been strongly recommending dropping those jobs from all stable branches as soon as the branch is created. They're inherently broken by design, I did my best to explain that when the idea first arose, they don't really test what they set out to and the cost of keeping them functioning is disproportionately high. Many projects have already gotten rid of them, they're not mandated by the PTI anyway. > And if later the same thing happen with one of our upper > requirements then we will to lose all the testing. I think we > cannot simply say that we ignore the problem by turning off the > job. But that's not the current situation, right? I can come up with all sorts of scenarios where we might break upper bounds testing, that's merely one of them, we should evaluate our options for those on a case by case basis as they arise since the nature of the conflict will determine which solutions are viable and at what cost/benefit ratios. > Does it mean you suggest bumping the decorator dependency to 4.0.0? In master, yes. On stable branches I recommend tossing the lower-constraints.txt based jobs out the nearest window. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From sylvain.bauza at gmail.com Wed Sep 22 16:15:05 2021 From: sylvain.bauza at gmail.com (Sylvain Bauza) Date: Wed, 22 Sep 2021 18:15:05 +0200 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> <20210922131513.7h622qyit54lgteq@yuggoth.org> Message-ID: Le mer. 22 sept. 2021 ? 18:05, Balazs Gibizer a ?crit : > > > On Wed, Sep 22 2021 at 01:15:13 PM +0000, Jeremy Stanley > wrote: > > On 2021-09-22 14:56:43 +0200 (+0200), El?d Ill?s wrote: > > [...] > >> Option 3: it is really just a temporary solution (maybe worse than > >> Option 1), though it is probably acceptable to unblock gate on > >> stable branches until the appropriate solution will be chosen. > >> Also worth to emphasize that this problem doesn't only affect > >> lower-constraints tests, but also could affect branches where we > >> have old packages (versions) in upper-constraints.txt! > > [...] > > > > Do also note that all solutions in this space are effectively > > temporary. When we drafted the stable branch and extended > > maintenance policies, we accepted that there comes a time for any > > job where circumstances may prevent us from being able to > > effectively continue running it. If the complexity of the solution > > outweighs the *temporary* benefits of being able to run the job for > > longer, it may not be worth the effort. > > As the issue affects even the recent stable branches this would > effectively mean we lose all the lower-constraints testing. And if > later the same thing happen with one of our upper requirements then we > will to lose all the testing. I think we cannot simply say that we > ignore the problem by turning off the job. > > > > > While we almost certainly could come up with a non-default > > replacement for Zuul's standard tox job which allows us to use > > specific versions of tox, virtualenv, pip, setuptools, et cetera, > > that added complexity does not come for free. It has a maintenance > > cost which extends far into the future. Also think back on the times > > when we've reached for the "easy" solution of pinning such tools in > > other frameworks like DevStack: everyone's in a rush to stop the > > bleeding and get back to testing, but nobody cares about fixing the > > underlying issue and so such things end up pinned for years until > > that pinning breaks something else because we're blind to all the > > bitrot around us. > > > > And then there's the fact that while we can "fix" things this way in > > CI jobs, these are ultimately tools for our developers which we just > > happen to also be able to run in the CI system, not the other way > > around. Pinning setuptools in the tox jobs does not solve this for > > developers running tox locally on their own systems. Are we going to > > start providing them with a list of specific versions of these tools > > which we're only able to test with, substantially increasing the > > complexity of setting up local dev environments and making it even > > harder for newcomers to contribute to the project? > > > > The stable branch policy is a set of guidelines and suggestions, not > > an unbreakable covenant. It's there to help us make sound choices > > under normal circumstances, but we should not be afraid to come to a > > collective decision about making policy exceptions in unusual > > situations such as this one. > > Does it mean you suggest bumping the decorator dependency to 4.0.0? > > > -- > > Jeremy Stanley > > > > I have to say fungi made great points. While option 2 has some ideal, option 1 seems the most pragmatic and won't divert us with developing efforts for getting there (AFAIK, we don't yet have a working solution for option 2). Also, I don't know for all distros but the one I know doesn't use our lower constraints but rather redefine the dependencies by what they currently support. So, I'm even not sure bumping our lower dep here would create an issue for them if we tell them it's a minor release. Operators, thoughts about it ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From openstack at nemebean.com Wed Sep 22 16:22:51 2021 From: openstack at nemebean.com (Ben Nemec) Date: Wed, 22 Sep 2021 11:22:51 -0500 Subject: RabbitMQ annoying disconnections [kolla-ansible, rabbitmq] In-Reply-To: <9418AB32-0847-4A84-B844-59EBB045FA63@poczta.onet.pl> References: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> <6eb72199-d368-3e8a-a6e7-1c0e8523bc0e@nemebean.com> <02C6F39B-C041-469F-9217-C1EE9FC3CBF7@poczta.onet.pl> <9418AB32-0847-4A84-B844-59EBB045FA63@poczta.onet.pl> Message-ID: <405ed974-3066-15bf-a9ce-06debffc1614@nemebean.com> On 9/21/21 2:44 AM, bkslash wrote: > So there are two questions: > > 1. Why heartbeat_in_pthread didn?t change situation? And if it?s > deprecated it was kind of risky to use it long term (after upgrade to > wallaby it would stop working anyway?) It's deprecated because it became the default behavior in subsequent releases. The functionality is not going away, just the ability to turn it off. The only reason it was ever configurable was that we had concerns about mixing pthreads and greenthreads and wanted the people affected by the original bug to test the combination for a couple of releases before we turned it on for everyone. From balazs.gibizer at est.tech Wed Sep 22 16:54:08 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Wed, 22 Sep 2021 18:54:08 +0200 Subject: [neutron] upgrading port binding:profile.allocation value for port-resource-request-groups API extension In-Reply-To: References: <0U8UZQ.B1WQBDR9U22T1@est.tech> Message-ID: <8MHUZQ.NIR8KHQMZ56L2@est.tech> On Wed, Sep 22 2021 at 05:09:17 PM +0200, Rodolfo Alonso Hernandez wrote: > Hello Balazs: > > About the group_id, I see this is built using the port_id and the > qos_rules. We have all of this in the DB and we can build it > statically (I think so, maybe I'm very optimistic). Yes, that is probably doable. But I let Przemek work out the details in the patch. > > About the code, that was something I was thinking about after sending > the last mail. For at least two releases, we need to support both RP > formats in the DB. If we read only the UUID (old format), then we > should convert it and store it in the new format. > > About the migration, we don't support contract migrations anymore. > But this is not true as we have done some migrations that have added > new restrictions in the DB schema. In any case, this could be done as > an expansion migration. If the code is in place, I don't see any > problem of doing this migration with the server running. Each > "ml2_port_bindings" register will be updated atomically, while the > Neutron server will be able to handle both versions. > If I understand correctly what you described is an online data migration where 1) neutron does not even need an expand migration as no new field is needed in the database as we use the old field both for the old and the new data 2) neutron server converts between data formats when the data is read 3) neutron can drop the conversion code only after every register is upgraded this way. As there could be ports that are not touched between upgrades we cannot simply say that we are done with the migration after waiting a release cycle. I think we have to add an upgrade check in Yoga that warns if there are still ports with the old format. And also neutron needs to provide a tool for the deployer to trigger the conversion of those remaining port before the upgrade to X. Do I understand your suggestion correct? Do you agree with the above solution proposal? Cheers, gibi > Regards. > > > On Wed, Sep 22, 2021 at 3:44 PM Balazs Gibizer > wrote: >> >> >> On Tue, Sep 21 2021 at 06:30:46 PM +0200, Rodolfo Alonso Hernandez >> wrote: >> > Hello Balazs: >> > >> > Sorry for the late reply, I was on PTO. >> > >> > If I'm not wrong, now port['binding:profile']['allocation'] is a >> UUID >> > and you need it to be a list of UUIDs. Am I correct? >> >> It is a bit more complicated than that. The old value is a single RP >> UUID. the new value is a dict where the key is the group_id and the >> value is the RP UUID fulfilling that group. So the transformation >> needs >> to access to the group_id. >> The group_id is a stable UUID generated by neutron server as part of >> the port.resource_request value, but it is not persisted. >> >> > >> > To make this change in the DB you should use the Alembic >> migrations, >> > as you said. That should ensure all registers are translated. We >> > should also include a sanity check to ensure the DB migration was >> > done correctly. >> >> I'm not 100% sure but I think such data migration can only be done >> in >> the contract part as it needs to be done while the neutron server is >> down as the old code can only use the old data format while the new >> code can only use the new format. Is it OK to introduce a new >> contract >> migration in Yoga in neutron? >> >> Cheers, >> gibi >> >> >> > >> > Is that what you needed? Don't hesitate to ping me in IRC if >> needed. >> > >> > Regards. >> > >> > On Fri, Sep 10, 2021 at 6:06 PM Balazs Gibizer >> > wrote: >> >> Hi Neutrinos! >> >> >> >> We found a technical challenge during implementing the >> >> port-resource-request-groups API extension[1]. That extension >> >> changes >> >> the format of the port.resoruce_request as well as the format >> of the >> >> port.binding:profile.allocation. The former is a calculated >> field on >> >> the port so that is easy. However the bindig:profile is >> persisted in >> >> the database so data migration is needed. What is the canonical >> way >> >> to >> >> do such DB data translation in Neutron? Can we translate the >> data in >> >> place during alembic migration? Or should we do some kind of >> online >> >> data migration when the data is translated by neutron when it is >> >> read >> >> from the db? >> >> >> >> cheers, >> >> gibi >> >> >> >> [1] https://review.opendev.org/c/openstack/neutron/+/805637/5 >> >> >> >> >> >> >> >> From gmann at ghanshyammann.com Wed Sep 22 17:23:04 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 22 Sep 2021 12:23:04 -0500 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> <20210922131513.7h622qyit54lgteq@yuggoth.org> Message-ID: <17c0e884634.1147e11ec220224.4626140220974493274@ghanshyammann.com> ---- On Wed, 22 Sep 2021 11:15:05 -0500 Sylvain Bauza wrote ---- > > > Le mer. 22 sept. 2021 ? 18:05, Balazs Gibizer a ?crit : > > > On Wed, Sep 22 2021 at 01:15:13 PM +0000, Jeremy Stanley > wrote: > > On 2021-09-22 14:56:43 +0200 (+0200), El?d Ill?s wrote: > > [...] > >> Option 3: it is really just a temporary solution (maybe worse than > >> Option 1), though it is probably acceptable to unblock gate on > >> stable branches until the appropriate solution will be chosen. > >> Also worth to emphasize that this problem doesn't only affect > >> lower-constraints tests, but also could affect branches where we > >> have old packages (versions) in upper-constraints.txt! > > [...] > > > > Do also note that all solutions in this space are effectively > > temporary. When we drafted the stable branch and extended > > maintenance policies, we accepted that there comes a time for any > > job where circumstances may prevent us from being able to > > effectively continue running it. If the complexity of the solution > > outweighs the *temporary* benefits of being able to run the job for > > longer, it may not be worth the effort. > > As the issue affects even the recent stable branches this would > effectively mean we lose all the lower-constraints testing. And if > later the same thing happen with one of our upper requirements then we > will to lose all the testing. I think we cannot simply say that we > ignore the problem by turning off the job. > > > > > While we almost certainly could come up with a non-default > > replacement for Zuul's standard tox job which allows us to use > > specific versions of tox, virtualenv, pip, setuptools, et cetera, > > that added complexity does not come for free. It has a maintenance > > cost which extends far into the future. Also think back on the times > > when we've reached for the "easy" solution of pinning such tools in > > other frameworks like DevStack: everyone's in a rush to stop the > > bleeding and get back to testing, but nobody cares about fixing the > > underlying issue and so such things end up pinned for years until > > that pinning breaks something else because we're blind to all the > > bitrot around us. > > > > And then there's the fact that while we can "fix" things this way in > > CI jobs, these are ultimately tools for our developers which we just > > happen to also be able to run in the CI system, not the other way > > around. Pinning setuptools in the tox jobs does not solve this for > > developers running tox locally on their own systems. Are we going to > > start providing them with a list of specific versions of these tools > > which we're only able to test with, substantially increasing the > > complexity of setting up local dev environments and making it even > > harder for newcomers to contribute to the project? > > > > The stable branch policy is a set of guidelines and suggestions, not > > an unbreakable covenant. It's there to help us make sound choices > > under normal circumstances, but we should not be afraid to come to a > > collective decision about making policy exceptions in unusual > > situations such as this one. > > Does it mean you suggest bumping the decorator dependency to 4.0.0? > > > -- > > Jeremy Stanley > > > > > I have to say fungi made great points. While option 2 has some ideal, option 1 seems the most pragmatic and won't divert us with developing efforts for getting there (AFAIK, we don't yet have a working solution for option 2). > Also, I don't know for all distros but the one I know doesn't use our lower constraints but rather redefine the dependencies by what they currently support. So, I'm even not sure bumping our lower dep here would create an issue for them if we tell them it's a minor release. > Operators, thoughts about it ? As per TC checks with distro/package maintainers on l-c usage, it is found that only Debian uses it and others do not care about it even on master branch[1]. I am also on the side of dropping the l-c from stable branches as soon as they are created instead of waiting for the broken gate and spend time on that where we know that they are hardly being used. As Fungi mentioned, TC has updated the PTI also mentioning it clearly that testing the l-c is all up to project decisions and it is totally fine to drop them (many projects already dropped it from stable) [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019877.html -gmann > From zbitter at redhat.com Wed Sep 22 17:26:45 2021 From: zbitter at redhat.com (Zane Bitter) Date: Wed, 22 Sep 2021 13:26:45 -0400 Subject: [heat] Proposing Brendan Shephard for heat core In-Reply-To: References: Message-ID: <41f63fa8-7bcb-25fc-bbfa-96e1993d40cb@redhat.com> On 22/09/21 01:04, Rico Lin wrote: > Hi all > I would like to suggest to add > > Brendan Shephard(bshephar at redhat.com ) > > to the heat core reviewer group. > > He has been working on heat actively. provide good quality reviews and > commits. > > I think it will be great to have him as the heat core reviewer. I'm all for adding more people, but I don't feel like I can make an informed judgement on the basis of 3 (non-backport) reviews (or 5 in total), all of which were +1s. From elod.illes at est.tech Wed Sep 22 19:01:02 2021 From: elod.illes at est.tech (=?UTF-8?B?RWzFkWQgSWxsw6lz?=) Date: Wed, 22 Sep 2021 21:01:02 +0200 Subject: [stable] remove formal 'unmaintained' phase from EM to EOL transition Message-ID: Hi stable maintainers! On the Xena Virtual PTG there was a discussion about 'Stable Extended Maintenance to End of Life transition' [1] and at some point we touched that the listed 'Unmaintained' phase [2] does not add value in the process (rather some confusion) so that was agreed that it should be removed (line 397). I've seen that the removal was not yet proposed so I created a patch [3] for it now. So this is just a warning for the change. Thanks, El?d [1] Line 342-397 https://etherpad.opendev.org/p/tc-xena-ptg [2] https://docs.openstack.org/project-team-guide/stable-branches.html#maintenance-phases [3] https://review.opendev.org/c/openstack/project-team-guide/+/810499 -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Wed Sep 22 19:05:27 2021 From: smooney at redhat.com (Sean Mooney) Date: Wed, 22 Sep 2021 20:05:27 +0100 Subject: RabbitMQ annoying disconnections [kolla-ansible, rabbitmq] In-Reply-To: <405ed974-3066-15bf-a9ce-06debffc1614@nemebean.com> References: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> <6eb72199-d368-3e8a-a6e7-1c0e8523bc0e@nemebean.com> <02C6F39B-C041-469F-9217-C1EE9FC3CBF7@poczta.onet.pl> <9418AB32-0847-4A84-B844-59EBB045FA63@poczta.onet.pl> <405ed974-3066-15bf-a9ce-06debffc1614@nemebean.com> Message-ID: <54af904fad040dc5618ce2bfc6426d3b76761348.camel@redhat.com> On Wed, 2021-09-22 at 11:22 -0500, Ben Nemec wrote: > > On 9/21/21 2:44 AM, bkslash wrote: > > So there are two questions: > > > > 1. Why heartbeat_in_pthread didn?t change situation? And if it?s > > deprecated it was kind of risky to use it long term (after upgrade to > > wallaby it would stop working anyway?) > > It's deprecated because it became the default behavior in subsequent > releases. The functionality is not going away, just the ability to turn > it off. > > The only reason it was ever configurable was that we had concerns about > mixing pthreads and greenthreads and wanted the people affected by the > original bug to test the combination for a couple of releases before we > turned it on for everyone. actully the ablity to trun it off is not going away either turing it on can break the nova comptue agent by breaking eventlet. https://bugs.launchpad.net/oslo.messaging/+bug/1934937 here is the patch to undeprecate it https://review.opendev.org/c/openstack/oslo.messaging/+/800621 based on the bug above this looke like it should be turned off unless you runing under mod_wsgi or uwsgi where the pthread is needed to escape there treadign lifecycle. > From openstack at nemebean.com Wed Sep 22 19:33:00 2021 From: openstack at nemebean.com (Ben Nemec) Date: Wed, 22 Sep 2021 14:33:00 -0500 Subject: RabbitMQ annoying disconnections [kolla-ansible, rabbitmq] In-Reply-To: <54af904fad040dc5618ce2bfc6426d3b76761348.camel@redhat.com> References: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> <6eb72199-d368-3e8a-a6e7-1c0e8523bc0e@nemebean.com> <02C6F39B-C041-469F-9217-C1EE9FC3CBF7@poczta.onet.pl> <9418AB32-0847-4A84-B844-59EBB045FA63@poczta.onet.pl> <405ed974-3066-15bf-a9ce-06debffc1614@nemebean.com> <54af904fad040dc5618ce2bfc6426d3b76761348.camel@redhat.com> Message-ID: On 9/22/21 2:05 PM, Sean Mooney wrote: > On Wed, 2021-09-22 at 11:22 -0500, Ben Nemec wrote: >> >> On 9/21/21 2:44 AM, bkslash wrote: >>> So there are two questions: >>> >>> 1. Why heartbeat_in_pthread didn?t change situation? And if it?s >>> deprecated it was kind of risky to use it long term (after upgrade to >>> wallaby it would stop working anyway?) >> >> It's deprecated because it became the default behavior in subsequent >> releases. The functionality is not going away, just the ability to turn >> it off. >> >> The only reason it was ever configurable was that we had concerns about >> mixing pthreads and greenthreads and wanted the people affected by the >> original bug to test the combination for a couple of releases before we >> turned it on for everyone. > actully the ablity to trun it off is not going away either > turing it on can break the nova comptue agent by breaking eventlet. > https://bugs.launchpad.net/oslo.messaging/+bug/1934937 > here is the patch to undeprecate it https://review.opendev.org/c/openstack/oslo.messaging/+/800621 > based on the bug above this looke like it should be turned off unless you runing under mod_wsgi or uwsgi > where the pthread is needed to escape there treadign lifecycle. Ah, okay. I thought I remembered seeing some discussion about that but I missed the patch. +2. From gmann at ghanshyammann.com Wed Sep 22 22:13:13 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 22 Sep 2021 17:13:13 -0500 Subject: [all][tc] Technical Committee next weekly meeting on Sept 23rd at 1500 UTC In-Reply-To: <17c053d4c0a.be874cfa77515.5986366594963704309@ghanshyammann.com> References: <17c053d4c0a.be874cfa77515.5986366594963704309@ghanshyammann.com> Message-ID: <17c0f91eb4a.c2739e7b228246.2453466278532798261@ghanshyammann.com> Hello Everyone, Below is the agenda for Tomorrow's TC meeting schedule at 1500 UTC in #openstack-tc IRC channel. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting == Agenda for tomorrow's TC meeting == * Roll call * Follow up on past action items * Gate health check (dansmith/yoctozepto) ** http://paste.openstack.org/show/jD6kAP9tHk7PZr2nhv8h/ * TC tags analysis ** https://docs.google.com/spreadsheets/d/18GXibtdQnSkIwA7DsBvX9dPwbw9JH76AhhRUknzBb3Q/edit * Project Health checks framework ** https://etherpad.opendev.org/p/health_check ** https://review.opendev.org/c/openstack/governance/+/810037 * Xena Tracker ** https://etherpad.opendev.org/p/tc-xena-tracker * Open Reviews ** https://review.opendev.org/q/projects:openstack/governance+is:open -gmann ---- On Mon, 20 Sep 2021 17:04:35 -0500 Ghanshyam Mann wrote ---- > > Hello Everyone, > > Technical Committee's next weekly meeting is scheduled for Sept 23rd at 1500 UTC. > > If you would like to add topics for discussion, please add them to the below wiki page by > Wednesday, Sept 22th, at 2100 UTC. > > https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting > > -gmann > > > From berndbausch at gmail.com Thu Sep 23 00:41:20 2021 From: berndbausch at gmail.com (Bernd Bausch) Date: Thu, 23 Sep 2021 09:41:20 +0900 Subject: Issue on accessing object storage containers from outside In-Reply-To: References: Message-ID: Please reveal the precise symptoms. In particular, how do you attempt to reach the public containers, and what exactly happens when you do so. Without having relevant information about your set up, my guess it that this is a networking problem. You may have configured an external network that is not reachable from your browser. The first thing, therefore, is to check whether there is any connectivity to the network to which the public link belongs. Bernd Bausch On 2021/09/22 7:35 PM, Michel Niyoyita wrote: > Dear all > > I have created containers and upload objects into containers . I can > access these containers and objects trough horizon dashboard but once > make them public I can not reach them through?the provided link . > > If any one have an idea on how to access containers and their object > using the link from outside he can help and guide. > > I deployed openstack wallaby using kolla-ansible and ceph pacific > using ansible , both are using ubuntu 20.04 as OS > > Michel From bkslash at poczta.onet.pl Thu Sep 23 07:53:25 2021 From: bkslash at poczta.onet.pl (Adam Tomas) Date: Thu, 23 Sep 2021 09:53:25 +0200 Subject: RabbitMQ annoying disconnections [kolla-ansible, rabbitmq] In-Reply-To: <54af904fad040dc5618ce2bfc6426d3b76761348.camel@redhat.com> References: <7063772E-2DBF-444B-8720-FEDD28A9336C@poczta.onet.pl> <6eb72199-d368-3e8a-a6e7-1c0e8523bc0e@nemebean.com> <02C6F39B-C041-469F-9217-C1EE9FC3CBF7@poczta.onet.pl> <9418AB32-0847-4A84-B844-59EBB045FA63@poczta.onet.pl> <405ed974-3066-15bf-a9ce-06debffc1614@nemebean.com> <54af904fad040dc5618ce2bfc6426d3b76761348.camel@redhat.com> Message-ID: Well, then see what I have in impl_rabbit.py (kolla-ansible 11, newest container images, Victoria): cfg.BoolOpt('heartbeat_in_pthread', default=False, help="EXPERIMENTAL: Run the health check heartbeat thread " "through a native python thread. By default if this " "option isn't provided the health check heartbeat will " "inherit the execution model from the parent process. By " "example if the parent process have monkey patched the " "stdlib by using eventlet/greenlet then the heartbeat " "will be run through a green thread."), 1. It is disabled by default, so heartbeats run through green thread? 2. Why enabling it didn?t change anything? 3. heartbeat = 600 for rabbit resolved problem for now (but I?ve seen 1-2 timeouts after 2 days anyway), but 10-minute timeout is kind of big, so I would like to use another solution Best regards Adam Tomas > Wiadomo?? napisana przez Sean Mooney w dniu 22.09.2021, o godz. 21:05: > > On Wed, 2021-09-22 at 11:22 -0500, Ben Nemec wrote: >> >> On 9/21/21 2:44 AM, bkslash wrote: >>> So there are two questions: >>> >>> 1. Why heartbeat_in_pthread didn?t change situation? And if it?s >>> deprecated it was kind of risky to use it long term (after upgrade to >>> wallaby it would stop working anyway?) >> >> It's deprecated because it became the default behavior in subsequent >> releases. The functionality is not going away, just the ability to turn >> it off. >> >> The only reason it was ever configurable was that we had concerns about >> mixing pthreads and greenthreads and wanted the people affected by the >> original bug to test the combination for a couple of releases before we >> turned it on for everyone. > actully the ablity to trun it off is not going away either > turing it on can break the nova comptue agent by breaking eventlet. > https://bugs.launchpad.net/oslo.messaging/+bug/1934937 > here is the patch to undeprecate it https://review.opendev.org/c/openstack/oslo.messaging/+/800621 > based on the bug above this looke like it should be turned off unless you runing under mod_wsgi or uwsgi > where the pthread is needed to escape there treadign lifecycle. > >> > > From tonykarera at gmail.com Thu Sep 23 09:01:59 2021 From: tonykarera at gmail.com (Karera Tony) Date: Thu, 23 Sep 2021 11:01:59 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> Message-ID: Hey Guys, Any other idea ? Regards Tony Karera On Wed, Sep 22, 2021 at 5:20 PM Karera Tony wrote: > Just to add on that, > > compute service is listed, I can create Volumes, I have the same cinder > keyring in the /etc/kolla/config/nova directory as I have in the > /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring > Regards > > Tony Karera > > > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony wrote: > >> Hello Guys, >> >> Thanks a lot. >> >> I had actually checked the nova -compute.log on the compute node and >> they were showing the error I will post at the end about the cinder keyring >> but I know its correct because its the same I was using on wallaby, I even >> tried to use another ceph cluster with ofcouse different keyrings but its >> the same issue. >> >> Below is the error >> >> r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to >> find a keyring on >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >> AuthRegistry(0x7fbcdc05a8b8) no keyring found at >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >> disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable >> to find a keyring on >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >> AuthRegistry(0x7fbcdc060698) no keyring found at >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >> disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable >> to find a keyring on >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >> AuthRegistry(0x7fbce2f4e020) no keyring found at >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >> disabling cephx\n[errno 2] RADOS object not found (error connecting to the >> cluster)\n' >> 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >> 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of >> the above exception, another exception occurred: >> Regards >> >> Tony Karera >> >> >> >> >> On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney wrote: >> >>> On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>> > It could also be a compute cell discovery issue maybe? >>> no they shoudl still show up in the hypervior list api >>> > >>> > Do you see anything under "openstack compute service list"? >>> if they show up in the service list but not they hyperiors api it >>> means that the comptue service started and registered its service entry >>> but >>> something broke it before it could create a compute node recored in the >>> db. >>> >>> with ceph the case i have hit this most often is when the keyright used >>> by nova to >>> get the avaiable capastiy of the ceph cluster is wrong whihc prevent the >>> resoucetack and compute manager >>> form actully creating the compute node record. >>> >>> >>> it can happen for other reason too but best place to start is check if >>> there is an error in the nova compute agent log and go from there. >>> > >>> > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney >>> wrote: >>> > >>> > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >>> > > > Hello Team, >>> > > > >>> > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu >>> 20.04 >>> > > and >>> > > > ceph as the backend storage for Nova, Cinder and Glance. >>> > > > >>> > > > It finished with no error but it has failed to register any on the >>> > > Compute >>> > > > Nodes under Hypervisors. >>> > > > >>> > > > kolla-openstack) stack at deployment:~$ openstack hypervisor list >>> > > > >>> > > > (kolla-openstack) stack at deployment:~$ >>> > > > >>> > > > >>> > > > Any idea on how to resolve this ? >>> > > that usually means that somehthing prevented the comptue agent form >>> > > strating properly >>> > > >>> > > for example incorrect ceph keyrings there are several other case but >>> you >>> > > mentioned you are >>> > > using ceph. >>> > > >>> > > if this is hte case you should see error in the compute agent log. >>> > > >>> > > > >>> > > > Regards >>> > > > >>> > > > Tony Karera >>> > > >>> > > >>> > > >>> > > >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.sgaravatto at gmail.com Thu Sep 23 10:20:53 2021 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Thu, 23 Sep 2021 12:20:53 +0200 Subject: [ops][neutron] Is it possible to "lock" a floating IP to an instance ? Message-ID: Hello I have the following use case: A user creates a VM and associates a floating IP to such instance Is in some way possible to prevent that the floating IP is disassociated from that instance by another user of the same project ? If it helps, the user owning the instance could be admin (but allowing only the admin user to manage floating IPs is not an option) Thanks, Massimo -------------- next part -------------- An HTML attachment was scrubbed... URL: From syedammad83 at gmail.com Thu Sep 23 10:33:13 2021 From: syedammad83 at gmail.com (Ammad Syed) Date: Thu, 23 Sep 2021 15:33:13 +0500 Subject: Neutron FIP Qos with OVN Backend Message-ID: Hello, I am using Neutron wallaby with OVN backend. I am testing QoS FIP and it works fine. I have created a QoS policy and attached it with FIP, It works fine. Is it possible to set one of the QoS policy as default that each time I acquire a floating IP, the QoS should be attached to it by default ? Ammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Thu Sep 23 10:38:56 2021 From: smooney at redhat.com (Sean Mooney) Date: Thu, 23 Sep 2021 11:38:56 +0100 Subject: [ops][neutron] Is it possible to "lock" a floating IP to an instance ? In-Reply-To: References: Message-ID: <1d449263168f0e90524ca0c5d7b9d761959c82be.camel@redhat.com> On Thu, 2021-09-23 at 12:20 +0200, Massimo Sgaravatto wrote: > Hello > > I have the following use case: > > A user creates a VM and associates a floating IP to such instance > > Is in some way possible to prevent that the floating IP is > disassociated from that instance by another user of the same project ? > > If it helps, the user owning the instance could be admin (but allowing only > the admin user to manage floating IPs is not an option) if you are using novas api to manage floating ips then you might be able to lock the instnace which should prevent changing the ip assocations and most other instnace actions however if you were to manage teh floating ips form neutron that ouls entirly bypass that. we had talk about adding the ablity to lock ports for a different usecasue and haing nova lock the port whenever an instance is locked that might be the way to adress this in the future but for now i dont think you can do this without custom midelware. > > > Thanks, Massimo From skaplons at redhat.com Thu Sep 23 11:07:25 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 23 Sep 2021 13:07:25 +0200 Subject: Neutron FIP Qos with OVN Backend In-Reply-To: References: Message-ID: <5698647.Ru2sQNNtPn@p1> Hi, On czwartek, 23 wrze?nia 2021 12:33:13 CEST Ammad Syed wrote: > Hello, > > I am using Neutron wallaby with OVN backend. I am testing QoS FIP and it > works fine. > > I have created a QoS policy and attached it with FIP, It works fine. Is it > possible to set one of the QoS policy as default that each time I acquire a > floating IP, the QoS should be attached to it by default ? > > Ammad You can mark QoS policy as default for the project: https:// docs.openstack.org/api-ref/network/v2/index.html?expanded=create-qos-policy- detail#qos-default-extension But TBH I'm not sure if that will work with FIPs - You should test it and if that will not work, probably You can open bug for Neutron about that :) -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From ralonsoh at redhat.com Thu Sep 23 11:35:32 2021 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Thu, 23 Sep 2021 13:35:32 +0200 Subject: Neutron FIP Qos with OVN Backend In-Reply-To: <5698647.Ru2sQNNtPn@p1> References: <5698647.Ru2sQNNtPn@p1> Message-ID: Hello Ammad: This is not possible. You can create a default QoS policy that will apply to a network and thus to the internal ports created on this network. When you create a FIP, a port on the external network is created and this port receives a QoS policy (the default policy). However, the FIP QoS driver reads the *FIP QoS* policy, not the associated *FIP port QoS* policy. That means, for now, what you need is not possible. Regards. On Thu, Sep 23, 2021 at 1:14 PM Slawek Kaplonski wrote: > Hi, > > On czwartek, 23 wrze?nia 2021 12:33:13 CEST Ammad Syed wrote: > > Hello, > > > > I am using Neutron wallaby with OVN backend. I am testing QoS FIP and it > > works fine. > > > > I have created a QoS policy and attached it with FIP, It works fine. Is > it > > possible to set one of the QoS policy as default that each time I > acquire a > > floating IP, the QoS should be attached to it by default ? > > > > Ammad > > You can mark QoS policy as default for the project: https:// > > docs.openstack.org/api-ref/network/v2/index.html?expanded=create-qos-policy- > detail#qos-default-extension > > But TBH I'm not sure if that will work with FIPs - You should test it and > if > that will not work, probably You can open bug for Neutron about that :) > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurentfdumont at gmail.com Thu Sep 23 12:59:21 2021 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Thu, 23 Sep 2021 08:59:21 -0400 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> Message-ID: I would investigate that compute error first. Creating volumes means the controllers are doing the action. Starting a VM on a compute means you also need Ceph to works on the compute to mount the rdb target. Working in Wallaby with the error doesn't mean it would 100% work in Victoria. On Thu, Sep 23, 2021 at 5:02 AM Karera Tony wrote: > Hey Guys, Any other idea ? > > Regards > > Tony Karera > > > > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony wrote: > >> Just to add on that, >> >> compute service is listed, I can create Volumes, I have the same cinder >> keyring in the /etc/kolla/config/nova directory as I have in the >> /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring >> Regards >> >> Tony Karera >> >> >> >> >> On Wed, Sep 22, 2021 at 5:08 PM Karera Tony wrote: >> >>> Hello Guys, >>> >>> Thanks a lot. >>> >>> I had actually checked the nova -compute.log on the compute node and >>> they were showing the error I will post at the end about the cinder keyring >>> but I know its correct because its the same I was using on wallaby, I even >>> tried to use another ceph cluster with ofcouse different keyrings but its >>> the same issue. >>> >>> Below is the error >>> >>> r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to >>> find a keyring on >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>> AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable >>> to find a keyring on >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>> AuthRegistry(0x7fbcdc060698) no keyring found at >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable >>> to find a keyring on >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>> AuthRegistry(0x7fbce2f4e020) no keyring found at >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> disabling cephx\n[errno 2] RADOS object not found (error connecting to the >>> cluster)\n' >>> 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>> 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of >>> the above exception, another exception occurred: >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney wrote: >>> >>>> On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>>> > It could also be a compute cell discovery issue maybe? >>>> no they shoudl still show up in the hypervior list api >>>> > >>>> > Do you see anything under "openstack compute service list"? >>>> if they show up in the service list but not they hyperiors api it >>>> means that the comptue service started and registered its service entry >>>> but >>>> something broke it before it could create a compute node recored in the >>>> db. >>>> >>>> with ceph the case i have hit this most often is when the keyright used >>>> by nova to >>>> get the avaiable capastiy of the ceph cluster is wrong whihc prevent >>>> the resoucetack and compute manager >>>> form actully creating the compute node record. >>>> >>>> >>>> it can happen for other reason too but best place to start is check if >>>> there is an error in the nova compute agent log and go from there. >>>> > >>>> > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney >>>> wrote: >>>> > >>>> > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >>>> > > > Hello Team, >>>> > > > >>>> > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu >>>> 20.04 >>>> > > and >>>> > > > ceph as the backend storage for Nova, Cinder and Glance. >>>> > > > >>>> > > > It finished with no error but it has failed to register any on the >>>> > > Compute >>>> > > > Nodes under Hypervisors. >>>> > > > >>>> > > > kolla-openstack) stack at deployment:~$ openstack hypervisor list >>>> > > > >>>> > > > (kolla-openstack) stack at deployment:~$ >>>> > > > >>>> > > > >>>> > > > Any idea on how to resolve this ? >>>> > > that usually means that somehthing prevented the comptue agent form >>>> > > strating properly >>>> > > >>>> > > for example incorrect ceph keyrings there are several other case >>>> but you >>>> > > mentioned you are >>>> > > using ceph. >>>> > > >>>> > > if this is hte case you should see error in the compute agent log. >>>> > > >>>> > > > >>>> > > > Regards >>>> > > > >>>> > > > Tony Karera >>>> > > >>>> > > >>>> > > >>>> > > >>>> >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Thu Sep 23 13:20:26 2021 From: smooney at redhat.com (Sean Mooney) Date: Thu, 23 Sep 2021 14:20:26 +0100 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> Message-ID: <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: > I would investigate that compute error first. Creating volumes means the > controllers are doing the action. Starting a VM on a compute means you also > need Ceph to works on the compute to mount the rdb target. nova as part of its startup process in aintiallying the resouce tracker will try to connect to ceph if you are using the rbd image backend to report how much stroage is avaiable. if the keyring does not work on the vms pool as the user nova is connecting as then that will block the agent from starting up fully and will cause it to be missing the hypervior list. the error seams to indicate that the cinder keyring is not in the nova container that likely means you have not put it in /etc/kolla/config/nova i woudl check /etc/kolla/config/nova on the deployment host and sudo ls /etc/kolla/nova-compute/ on the compute node to ensure the cinder keyring is actully copied and has the correct content i have stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf [client.cinder] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" stack at cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.client.nova.keyring [client.nova] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" blanked out the key wiht *************** after the fact but you should have something similar in my case i decied to use a seperate key for nova rbd backend because i was also using EC poosl with a seperate data and metadata pool so i neede to modify my ceph.conf to make that work with kolla stack at cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.conf # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f [global] fsid = ********************* mon_host = [*********************] [client.glance] rbd default data pool = images-data [client.cinder] rbd default data pool = volumes-data [client.nova] rbd default data pool = vms-data using 2 keyrings/user allows me to set different default data pools for cinder and nova. > > Working in Wallaby with the error doesn't mean it would 100% work in > Victoria. > > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony wrote: > > > Hey Guys, Any other idea ? > > > > Regards > > > > Tony Karera > > > > > > > > > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony wrote: > > > > > Just to add on that, > > > > > > compute service is listed, I can create Volumes, I have the same cinder > > > keyring in the /etc/kolla/config/nova directory as I have in the > > > /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring > > > Regards > > > > > > Tony Karera > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony wrote: > > > > > > > Hello Guys, > > > > > > > > Thanks a lot. > > > > > > > > I had actually checked the nova -compute.log on the compute node and > > > > they were showing the error I will post at the end about the cinder keyring > > > > but I know its correct because its the same I was using on wallaby, I even > > > > tried to use another ceph cluster with ofcouse different keyrings but its > > > > the same issue. > > > > > > > > Below is the error > > > > > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to > > > > find a keyring on > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > > > to find a keyring on > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > > > AuthRegistry(0x7fbcdc060698) no keyring found at > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > > > to find a keyring on > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > disabling cephx\n[errno 2] RADOS object not found (error connecting to the > > > > cluster)\n' > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of > > > > the above exception, another exception occurred: > > > > Regards > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney wrote: > > > > > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > > > > > > It could also be a compute cell discovery issue maybe? > > > > > no they shoudl still show up in the hypervior list api > > > > > > > > > > > > Do you see anything under "openstack compute service list"? > > > > > if they show up in the service list but not they hyperiors api it > > > > > means that the comptue service started and registered its service entry > > > > > but > > > > > something broke it before it could create a compute node recored in the > > > > > db. > > > > > > > > > > with ceph the case i have hit this most often is when the keyright used > > > > > by nova to > > > > > get the avaiable capastiy of the ceph cluster is wrong whihc prevent > > > > > the resoucetack and compute manager > > > > > form actully creating the compute node record. > > > > > > > > > > > > > > > it can happen for other reason too but best place to start is check if > > > > > there is an error in the nova compute agent log and go from there. > > > > > > > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney > > > > > wrote: > > > > > > > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > > > > > > Hello Team, > > > > > > > > > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu > > > > > 20.04 > > > > > > > and > > > > > > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > > > > > > > > > > > It finished with no error but it has failed to register any on the > > > > > > > Compute > > > > > > > > Nodes under Hypervisors. > > > > > > > > > > > > > > > > kolla-openstack) stack at deployment:~$ openstack hypervisor list > > > > > > > > > > > > > > > > (kolla-openstack) stack at deployment:~$ > > > > > > > > > > > > > > > > > > > > > > > > Any idea on how to resolve this ? > > > > > > > that usually means that somehthing prevented the comptue agent form > > > > > > > strating properly > > > > > > > > > > > > > > for example incorrect ceph keyrings there are several other case > > > > > but you > > > > > > > mentioned you are > > > > > > > using ceph. > > > > > > > > > > > > > > if this is hte case you should see error in the compute agent log. > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From tonykarera at gmail.com Thu Sep 23 15:24:24 2021 From: tonykarera at gmail.com (Karera Tony) Date: Thu, 23 Sep 2021 17:24:24 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> Message-ID: Hello Sean, Below are the output on the compute node and deployment root at compute1:/etc/kolla/nova-compute# ls ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf And I can confirm that the content is the same. Regards Tony Karera On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney wrote: > On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: > > I would investigate that compute error first. Creating volumes means the > > controllers are doing the action. Starting a VM on a compute means you > also > > need Ceph to works on the compute to mount the rdb target. > > nova as part of its startup process in aintiallying the resouce tracker > will > try to connect to ceph if you are using the rbd image backend to report > how much stroage > is avaiable. if the keyring does not work on the vms pool as the user > nova is connecting as > then that will block the agent from starting up fully and will cause it to > be missing the hypervior list. > > the error seams to indicate that the cinder keyring is not in the nova > container > that likely means you have not put it in /etc/kolla/config/nova > i woudl check /etc/kolla/config/nova on the deployment host and sudo ls > /etc/kolla/nova-compute/ > on the compute node to ensure the cinder keyring is actully copied and has > the correct content > > i have > stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ > ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf > config.json nova.conf > > > [client.cinder] > key = ********************************* > caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" > caps mon = "profile rbd" > caps osd = "profile rbd pool=volumes, profile rbd pool=vms, > profile rbd pool=images" > stack at cloud:/opt/repos/devstack$ sudo cat > /etc/kolla/nova-compute/ceph.client.nova.keyring > [client.nova] > key = ********************************* > caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" > caps mon = "profile rbd" > caps osd = "profile rbd pool=volumes, profile rbd pool=vms, > profile rbd pool=images" > > blanked out the key wiht *************** after the fact but you should > have something similar > > > in my case i decied to use a seperate key for nova rbd backend because i > was also using EC poosl with a seperate data and metadata pool > so i neede to modify my ceph.conf to make that work with kolla > > stack at cloud:/opt/repos/devstack$ sudo cat > /etc/kolla/nova-compute/ceph.conf > # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f > [global] > fsid = ********************* > mon_host = [*********************] > > [client.glance] > rbd default data pool = images-data > > [client.cinder] > rbd default data pool = volumes-data > > [client.nova] > rbd default data pool = vms-data > > using 2 keyrings/user allows me to set different default data pools for > cinder and nova. > > > > > Working in Wallaby with the error doesn't mean it would 100% work in > > Victoria. > > > > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony > wrote: > > > > > Hey Guys, Any other idea ? > > > > > > Regards > > > > > > Tony Karera > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony > wrote: > > > > > > > Just to add on that, > > > > > > > > compute service is listed, I can create Volumes, I have the same > cinder > > > > keyring in the /etc/kolla/config/nova directory as I have in the > > > > /etc/kolla/config/cinder/cinder-volume directory along with the nova > keyring > > > > Regards > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony > wrote: > > > > > > > > > Hello Guys, > > > > > > > > > > Thanks a lot. > > > > > > > > > > I had actually checked the nova -compute.log on the compute node > and > > > > > they were showing the error I will post at the end about the > cinder keyring > > > > > but I know its correct because its the same I was using on > wallaby, I even > > > > > tried to use another ceph cluster with ofcouse different keyrings > but its > > > > > the same issue. > > > > > > > > > > Below is the error > > > > > > > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: > unable to > > > > > find a keyring on > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 > 7fbce2f4f700 -1 > > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > auth: unable > > > > > to find a keyring on > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 > 7fbce2f4f700 -1 > > > > > AuthRegistry(0x7fbcdc060698) no keyring found at > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > auth: unable > > > > > to find a keyring on > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 > 7fbce2f4f700 -1 > > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > > disabling cephx\n[errno 2] RADOS object not found (error > connecting to the > > > > > cluster)\n' > > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager > > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During > handling of > > > > > the above exception, another exception occurred: > > > > > Regards > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney > wrote: > > > > > > > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > > > > > > > It could also be a compute cell discovery issue maybe? > > > > > > no they shoudl still show up in the hypervior list api > > > > > > > > > > > > > > Do you see anything under "openstack compute service list"? > > > > > > if they show up in the service list but not they hyperiors api it > > > > > > means that the comptue service started and registered its > service entry > > > > > > but > > > > > > something broke it before it could create a compute node recored > in the > > > > > > db. > > > > > > > > > > > > with ceph the case i have hit this most often is when the > keyright used > > > > > > by nova to > > > > > > get the avaiable capastiy of the ceph cluster is wrong whihc > prevent > > > > > > the resoucetack and compute manager > > > > > > form actully creating the compute node record. > > > > > > > > > > > > > > > > > > it can happen for other reason too but best place to start is > check if > > > > > > there is an error in the nova compute agent log and go from > there. > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < > smooney at redhat.com> > > > > > > wrote: > > > > > > > > > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > > > > > > > Hello Team, > > > > > > > > > > > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on > Ubuntu > > > > > > 20.04 > > > > > > > > and > > > > > > > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > > > > > > > > > > > > > It finished with no error but it has failed to register > any on the > > > > > > > > Compute > > > > > > > > > Nodes under Hypervisors. > > > > > > > > > > > > > > > > > > kolla-openstack) stack at deployment:~$ openstack hypervisor > list > > > > > > > > > > > > > > > > > > (kolla-openstack) stack at deployment:~$ > > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea on how to resolve this ? > > > > > > > > that usually means that somehthing prevented the comptue > agent form > > > > > > > > strating properly > > > > > > > > > > > > > > > > for example incorrect ceph keyrings there are several other > case > > > > > > but you > > > > > > > > mentioned you are > > > > > > > > using ceph. > > > > > > > > > > > > > > > > if this is hte case you should see error in the compute > agent log. > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralonsoh at redhat.com Thu Sep 23 15:33:11 2021 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Thu, 23 Sep 2021 17:33:11 +0200 Subject: [neutron] upgrading port binding:profile.allocation value for port-resource-request-groups API extension In-Reply-To: <8MHUZQ.NIR8KHQMZ56L2@est.tech> References: <0U8UZQ.B1WQBDR9U22T1@est.tech> <8MHUZQ.NIR8KHQMZ56L2@est.tech> Message-ID: Hello Balazs: You are right: a DB migration is not necessary but a DB sanitization. I did something similar in https://review.opendev.org/c/openstack/neutron/+/789831. This patch includes the script to, in one shot, modify the whole DB and an upgrade check to test it. About the code, if something like the script provided and the upgrade check is included, that will ensure the DB is correctly migrated in Y release. That means the code supporting both formats could be removed in Z. Regards. On Wed, Sep 22, 2021 at 6:54 PM Balazs Gibizer wrote: > > > On Wed, Sep 22 2021 at 05:09:17 PM +0200, Rodolfo Alonso Hernandez > wrote: > > Hello Balazs: > > > > About the group_id, I see this is built using the port_id and the > > qos_rules. We have all of this in the DB and we can build it > > statically (I think so, maybe I'm very optimistic). > > Yes, that is probably doable. But I let Przemek work out the details in > the patch. > > > > About the code, that was something I was thinking about after sending > > the last mail. For at least two releases, we need to support both RP > > formats in the DB. If we read only the UUID (old format), then we > > should convert it and store it in the new format. > > > > About the migration, we don't support contract migrations anymore. > > But this is not true as we have done some migrations that have added > > new restrictions in the DB schema. In any case, this could be done as > > an expansion migration. If the code is in place, I don't see any > > problem of doing this migration with the server running. Each > > "ml2_port_bindings" register will be updated atomically, while the > > Neutron server will be able to handle both versions. > > > > If I understand correctly what you described is an online data > migration where > 1) neutron does not even need an expand migration as no new field is > needed in the database as we use the old field both for the old and the > new data > 2) neutron server converts between data formats when the data is read > 3) neutron can drop the conversion code only after every register is > upgraded this way. As there could be ports that are not touched between > upgrades we cannot simply say that we are done with the migration after > waiting a release cycle. I think we have to add an upgrade check in > Yoga that warns if there are still ports with the old format. And also > neutron needs to provide a tool for the deployer to trigger the > conversion of those remaining port before the upgrade to X. > > Do I understand your suggestion correct? Do you agree with the above > solution proposal? > > Cheers, > gibi > > > Regards. > > > > > > On Wed, Sep 22, 2021 at 3:44 PM Balazs Gibizer > > wrote: > >> > >> > >> On Tue, Sep 21 2021 at 06:30:46 PM +0200, Rodolfo Alonso Hernandez > >> wrote: > >> > Hello Balazs: > >> > > >> > Sorry for the late reply, I was on PTO. > >> > > >> > If I'm not wrong, now port['binding:profile']['allocation'] is a > >> UUID > >> > and you need it to be a list of UUIDs. Am I correct? > >> > >> It is a bit more complicated than that. The old value is a single RP > >> UUID. the new value is a dict where the key is the group_id and the > >> value is the RP UUID fulfilling that group. So the transformation > >> needs > >> to access to the group_id. > >> The group_id is a stable UUID generated by neutron server as part of > >> the port.resource_request value, but it is not persisted. > >> > >> > > >> > To make this change in the DB you should use the Alembic > >> migrations, > >> > as you said. That should ensure all registers are translated. We > >> > should also include a sanity check to ensure the DB migration was > >> > done correctly. > >> > >> I'm not 100% sure but I think such data migration can only be done > >> in > >> the contract part as it needs to be done while the neutron server is > >> down as the old code can only use the old data format while the new > >> code can only use the new format. Is it OK to introduce a new > >> contract > >> migration in Yoga in neutron? > >> > >> Cheers, > >> gibi > >> > >> > >> > > >> > Is that what you needed? Don't hesitate to ping me in IRC if > >> needed. > >> > > >> > Regards. > >> > > >> > On Fri, Sep 10, 2021 at 6:06 PM Balazs Gibizer > >> > wrote: > >> >> Hi Neutrinos! > >> >> > >> >> We found a technical challenge during implementing the > >> >> port-resource-request-groups API extension[1]. That extension > >> >> changes > >> >> the format of the port.resoruce_request as well as the format > >> of the > >> >> port.binding:profile.allocation. The former is a calculated > >> field on > >> >> the port so that is easy. However the bindig:profile is > >> persisted in > >> >> the database so data migration is needed. What is the canonical > >> way > >> >> to > >> >> do such DB data translation in Neutron? Can we translate the > >> data in > >> >> place during alembic migration? Or should we do some kind of > >> online > >> >> data migration when the data is translated by neutron when it is > >> >> read > >> >> from the db? > >> >> > >> >> cheers, > >> >> gibi > >> >> > >> >> [1] https://review.opendev.org/c/openstack/neutron/+/805637/5 > >> >> > >> >> > >> >> > >> > >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Thu Sep 23 15:50:39 2021 From: katonalala at gmail.com (Lajos Katona) Date: Thu, 23 Sep 2021 17:50:39 +0200 Subject: [neutron] Drivers meeting - Friday 24.09.2021 - cancelled Message-ID: Hi Neutron Team, Due to lack of agenda, let's cancel tomorrow's drivers meeting. Let's concentrate on the ongoing release and see You on the meeting next week :-) Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From iurygregory at gmail.com Thu Sep 23 15:55:21 2021 From: iurygregory at gmail.com (Iury Gregory) Date: Thu, 23 Sep 2021 17:55:21 +0200 Subject: [ironic] Yoga PTG In-Reply-To: References: Message-ID: Hello Ironicers, Fellow reminder that the PTG is in less than 4 weeks. Don't forget to register [1] and add your topics to our etherpad [2]. [1] https://openinfra-ptg.eventbrite.com [2] https://etherpad.opendev.org/p/ironic-yoga-ptg Em ter., 20 de jul. de 2021 ?s 16:21, Iury Gregory escreveu: > Hello Ironicers! > > We are approaching the time of the next PTG (it's in October but we need > to make some decisions). > Our yoga etherpad is available[1], feel free to add any topics we shall > discuss. In the ethercalc[2] you can find the slots we already booked, we > may add more slots for APAC if someone is willing to run the meeting and > discussions. > Don't forget to register! [3]. > > [1] https://etherpad.opendev.org/p/ironic-yoga-ptg > [2] https://ethercalc.openstack.org/8tum5yl1bx43 > [3] https://openinfra-ptg.eventbrite.com > -- > > > *Att[]'sIury Gregory Melo Ferreira * > *MSc in Computer Science at UFCG* > *Part of the ironic-core and puppet-manager-core team in OpenStack* > *Software Engineer at Red Hat Czech* > *Social*: https://www.linkedin.com/in/iurygregory > *E-mail: iurygregory at gmail.com * > -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory at gmail.com * -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurentfdumont at gmail.com Fri Sep 24 00:47:11 2021 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Thu, 23 Sep 2021 20:47:11 -0400 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> Message-ID: I do believe Kolla runs a container version of each service on computes. Are you looking inside the nova-compute container ( etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) On Thu, Sep 23, 2021 at 11:24 AM Karera Tony wrote: > Hello Sean, > > Below are the output on the compute node and deployment > > root at compute1:/etc/kolla/nova-compute# ls > ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf > config.json nova.conf > > (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ > ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf > > And I can confirm that the content is the same. > > > > Regards > > Tony Karera > > > > > On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney wrote: > >> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >> > I would investigate that compute error first. Creating volumes means the >> > controllers are doing the action. Starting a VM on a compute means you >> also >> > need Ceph to works on the compute to mount the rdb target. >> >> nova as part of its startup process in aintiallying the resouce tracker >> will >> try to connect to ceph if you are using the rbd image backend to report >> how much stroage >> is avaiable. if the keyring does not work on the vms pool as the user >> nova is connecting as >> then that will block the agent from starting up fully and will cause it >> to be missing the hypervior list. >> >> the error seams to indicate that the cinder keyring is not in the nova >> container >> that likely means you have not put it in /etc/kolla/config/nova >> i woudl check /etc/kolla/config/nova on the deployment host and sudo ls >> /etc/kolla/nova-compute/ >> on the compute node to ensure the cinder keyring is actully copied and >> has the correct content >> >> i have >> stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >> config.json nova.conf >> >> >> [client.cinder] >> key = ********************************* >> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >> caps mon = "profile rbd" >> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >> profile rbd pool=images" >> stack at cloud:/opt/repos/devstack$ sudo cat >> /etc/kolla/nova-compute/ceph.client.nova.keyring >> [client.nova] >> key = ********************************* >> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >> caps mon = "profile rbd" >> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >> profile rbd pool=images" >> >> blanked out the key wiht *************** after the fact but you should >> have something similar >> >> >> in my case i decied to use a seperate key for nova rbd backend because i >> was also using EC poosl with a seperate data and metadata pool >> so i neede to modify my ceph.conf to make that work with kolla >> >> stack at cloud:/opt/repos/devstack$ sudo cat >> /etc/kolla/nova-compute/ceph.conf >> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >> [global] >> fsid = ********************* >> mon_host = [*********************] >> >> [client.glance] >> rbd default data pool = images-data >> >> [client.cinder] >> rbd default data pool = volumes-data >> >> [client.nova] >> rbd default data pool = vms-data >> >> using 2 keyrings/user allows me to set different default data pools for >> cinder and nova. >> >> > >> > Working in Wallaby with the error doesn't mean it would 100% work in >> > Victoria. >> > >> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony >> wrote: >> > >> > > Hey Guys, Any other idea ? >> > > >> > > Regards >> > > >> > > Tony Karera >> > > >> > > >> > > >> > > >> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony >> wrote: >> > > >> > > > Just to add on that, >> > > > >> > > > compute service is listed, I can create Volumes, I have the same >> cinder >> > > > keyring in the /etc/kolla/config/nova directory as I have in the >> > > > /etc/kolla/config/cinder/cinder-volume directory along with the >> nova keyring >> > > > Regards >> > > > >> > > > Tony Karera >> > > > >> > > > >> > > > >> > > > >> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony >> wrote: >> > > > >> > > > > Hello Guys, >> > > > > >> > > > > Thanks a lot. >> > > > > >> > > > > I had actually checked the nova -compute.log on the compute node >> and >> > > > > they were showing the error I will post at the end about the >> cinder keyring >> > > > > but I know its correct because its the same I was using on >> wallaby, I even >> > > > > tried to use another ceph cluster with ofcouse different keyrings >> but its >> > > > > the same issue. >> > > > > >> > > > > Below is the error >> > > > > >> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: >> unable to >> > > > > find a keyring on >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 >> 7fbce2f4f700 -1 >> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >> auth: unable >> > > > > to find a keyring on >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >> 7fbce2f4f700 -1 >> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >> auth: unable >> > > > > to find a keyring on >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >> 7fbce2f4f700 -1 >> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >> > > > > disabling cephx\n[errno 2] RADOS object not found (error >> connecting to the >> > > > > cluster)\n' >> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During >> handling of >> > > > > the above exception, another exception occurred: >> > > > > Regards >> > > > > >> > > > > Tony Karera >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney >> wrote: >> > > > > >> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >> > > > > > > It could also be a compute cell discovery issue maybe? >> > > > > > no they shoudl still show up in the hypervior list api >> > > > > > > >> > > > > > > Do you see anything under "openstack compute service list"? >> > > > > > if they show up in the service list but not they hyperiors api >> it >> > > > > > means that the comptue service started and registered its >> service entry >> > > > > > but >> > > > > > something broke it before it could create a compute node >> recored in the >> > > > > > db. >> > > > > > >> > > > > > with ceph the case i have hit this most often is when the >> keyright used >> > > > > > by nova to >> > > > > > get the avaiable capastiy of the ceph cluster is wrong whihc >> prevent >> > > > > > the resoucetack and compute manager >> > > > > > form actully creating the compute node record. >> > > > > > >> > > > > > >> > > > > > it can happen for other reason too but best place to start is >> check if >> > > > > > there is an error in the nova compute agent log and go from >> there. >> > > > > > > >> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >> smooney at redhat.com> >> > > > > > wrote: >> > > > > > > >> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >> > > > > > > > > Hello Team, >> > > > > > > > > >> > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on >> Ubuntu >> > > > > > 20.04 >> > > > > > > > and >> > > > > > > > > ceph as the backend storage for Nova, Cinder and Glance. >> > > > > > > > > >> > > > > > > > > It finished with no error but it has failed to register >> any on the >> > > > > > > > Compute >> > > > > > > > > Nodes under Hypervisors. >> > > > > > > > > >> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack >> hypervisor list >> > > > > > > > > >> > > > > > > > > (kolla-openstack) stack at deployment:~$ >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > Any idea on how to resolve this ? >> > > > > > > > that usually means that somehthing prevented the comptue >> agent form >> > > > > > > > strating properly >> > > > > > > > >> > > > > > > > for example incorrect ceph keyrings there are several other >> case >> > > > > > but you >> > > > > > > > mentioned you are >> > > > > > > > using ceph. >> > > > > > > > >> > > > > > > > if this is hte case you should see error in the compute >> agent log. >> > > > > > > > >> > > > > > > > > >> > > > > > > > > Regards >> > > > > > > > > >> > > > > > > > > Tony Karera >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > >> > > > > > >> > > > > > >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Fri Sep 24 05:30:28 2021 From: tonykarera at gmail.com (Karera Tony) Date: Fri, 24 Sep 2021 07:30:28 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> Message-ID: Hello Laurent, It turns out I only have one keyring in the container. root at compute1:/home/stack# docker exec -it nova_compute bash (nova-compute)[nova at compute1 /]$ cd /etc/ceph/ (nova-compute)[nova at compute1 ceph]$ ls ceph.client.nova.keyring ceph.conf rbdmap Regards Tony Karera On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont wrote: > I do believe Kolla runs a container version of each service on computes. > Are you looking inside the nova-compute container ( > etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. > keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) > > On Thu, Sep 23, 2021 at 11:24 AM Karera Tony wrote: > >> Hello Sean, >> >> Below are the output on the compute node and deployment >> >> root at compute1:/etc/kolla/nova-compute# ls >> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >> config.json nova.conf >> >> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ >> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >> >> And I can confirm that the content is the same. >> >> >> >> Regards >> >> Tony Karera >> >> >> >> >> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney wrote: >> >>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>> > I would investigate that compute error first. Creating volumes means >>> the >>> > controllers are doing the action. Starting a VM on a compute means you >>> also >>> > need Ceph to works on the compute to mount the rdb target. >>> >>> nova as part of its startup process in aintiallying the resouce tracker >>> will >>> try to connect to ceph if you are using the rbd image backend to report >>> how much stroage >>> is avaiable. if the keyring does not work on the vms pool as the user >>> nova is connecting as >>> then that will block the agent from starting up fully and will cause it >>> to be missing the hypervior list. >>> >>> the error seams to indicate that the cinder keyring is not in the nova >>> container >>> that likely means you have not put it in /etc/kolla/config/nova >>> i woudl check /etc/kolla/config/nova on the deployment host and sudo ls >>> /etc/kolla/nova-compute/ >>> on the compute node to ensure the cinder keyring is actully copied and >>> has the correct content >>> >>> i have >>> stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>> config.json nova.conf >>> >>> >>> [client.cinder] >>> key = ********************************* >>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>> caps mon = "profile rbd" >>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>> profile rbd pool=images" >>> stack at cloud:/opt/repos/devstack$ sudo cat >>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>> [client.nova] >>> key = ********************************* >>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>> caps mon = "profile rbd" >>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>> profile rbd pool=images" >>> >>> blanked out the key wiht *************** after the fact but you should >>> have something similar >>> >>> >>> in my case i decied to use a seperate key for nova rbd backend because i >>> was also using EC poosl with a seperate data and metadata pool >>> so i neede to modify my ceph.conf to make that work with kolla >>> >>> stack at cloud:/opt/repos/devstack$ sudo cat >>> /etc/kolla/nova-compute/ceph.conf >>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>> [global] >>> fsid = ********************* >>> mon_host = [*********************] >>> >>> [client.glance] >>> rbd default data pool = images-data >>> >>> [client.cinder] >>> rbd default data pool = volumes-data >>> >>> [client.nova] >>> rbd default data pool = vms-data >>> >>> using 2 keyrings/user allows me to set different default data pools for >>> cinder and nova. >>> >>> > >>> > Working in Wallaby with the error doesn't mean it would 100% work in >>> > Victoria. >>> > >>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony >>> wrote: >>> > >>> > > Hey Guys, Any other idea ? >>> > > >>> > > Regards >>> > > >>> > > Tony Karera >>> > > >>> > > >>> > > >>> > > >>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony >>> wrote: >>> > > >>> > > > Just to add on that, >>> > > > >>> > > > compute service is listed, I can create Volumes, I have the same >>> cinder >>> > > > keyring in the /etc/kolla/config/nova directory as I have in the >>> > > > /etc/kolla/config/cinder/cinder-volume directory along with the >>> nova keyring >>> > > > Regards >>> > > > >>> > > > Tony Karera >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony >>> wrote: >>> > > > >>> > > > > Hello Guys, >>> > > > > >>> > > > > Thanks a lot. >>> > > > > >>> > > > > I had actually checked the nova -compute.log on the compute >>> node and >>> > > > > they were showing the error I will post at the end about the >>> cinder keyring >>> > > > > but I know its correct because its the same I was using on >>> wallaby, I even >>> > > > > tried to use another ceph cluster with ofcouse different >>> keyrings but its >>> > > > > the same issue. >>> > > > > >>> > > > > Below is the error >>> > > > > >>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: >>> unable to >>> > > > > find a keyring on >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 >>> 7fbce2f4f700 -1 >>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>> auth: unable >>> > > > > to find a keyring on >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>> 7fbce2f4f700 -1 >>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>> auth: unable >>> > > > > to find a keyring on >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>> 7fbce2f4f700 -1 >>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>> connecting to the >>> > > > > cluster)\n' >>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During >>> handling of >>> > > > > the above exception, another exception occurred: >>> > > > > Regards >>> > > > > >>> > > > > Tony Karera >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney >>> wrote: >>> > > > > >>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>> > > > > > > It could also be a compute cell discovery issue maybe? >>> > > > > > no they shoudl still show up in the hypervior list api >>> > > > > > > >>> > > > > > > Do you see anything under "openstack compute service list"? >>> > > > > > if they show up in the service list but not they hyperiors api >>> it >>> > > > > > means that the comptue service started and registered its >>> service entry >>> > > > > > but >>> > > > > > something broke it before it could create a compute node >>> recored in the >>> > > > > > db. >>> > > > > > >>> > > > > > with ceph the case i have hit this most often is when the >>> keyright used >>> > > > > > by nova to >>> > > > > > get the avaiable capastiy of the ceph cluster is wrong whihc >>> prevent >>> > > > > > the resoucetack and compute manager >>> > > > > > form actully creating the compute node record. >>> > > > > > >>> > > > > > >>> > > > > > it can happen for other reason too but best place to start is >>> check if >>> > > > > > there is an error in the nova compute agent log and go from >>> there. >>> > > > > > > >>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>> smooney at redhat.com> >>> > > > > > wrote: >>> > > > > > > >>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >>> > > > > > > > > Hello Team, >>> > > > > > > > > >>> > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible >>> on Ubuntu >>> > > > > > 20.04 >>> > > > > > > > and >>> > > > > > > > > ceph as the backend storage for Nova, Cinder and Glance. >>> > > > > > > > > >>> > > > > > > > > It finished with no error but it has failed to register >>> any on the >>> > > > > > > > Compute >>> > > > > > > > > Nodes under Hypervisors. >>> > > > > > > > > >>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack >>> hypervisor list >>> > > > > > > > > >>> > > > > > > > > (kolla-openstack) stack at deployment:~$ >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > Any idea on how to resolve this ? >>> > > > > > > > that usually means that somehthing prevented the comptue >>> agent form >>> > > > > > > > strating properly >>> > > > > > > > >>> > > > > > > > for example incorrect ceph keyrings there are several >>> other case >>> > > > > > but you >>> > > > > > > > mentioned you are >>> > > > > > > > using ceph. >>> > > > > > > > >>> > > > > > > > if this is hte case you should see error in the compute >>> agent log. >>> > > > > > > > >>> > > > > > > > > >>> > > > > > > > > Regards >>> > > > > > > > > >>> > > > > > > > > Tony Karera >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Fri Sep 24 09:13:17 2021 From: tonykarera at gmail.com (Karera Tony) Date: Fri, 24 Sep 2021 11:13:17 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> Message-ID: Hello Team, I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying. I don't understand why there is a certain issue now. Regards Tony Karera On Fri, Sep 24, 2021 at 7:30 AM Karera Tony wrote: > Hello Laurent, > > It turns out I only have one keyring in the container. > > root at compute1:/home/stack# docker exec -it nova_compute bash > (nova-compute)[nova at compute1 /]$ cd /etc/ceph/ > (nova-compute)[nova at compute1 ceph]$ ls > ceph.client.nova.keyring ceph.conf rbdmap > > Regards > > Tony Karera > > > > > On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont > wrote: > >> I do believe Kolla runs a container version of each service on computes. >> Are you looking inside the nova-compute container ( >> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >> >> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony >> wrote: >> >>> Hello Sean, >>> >>> Below are the output on the compute node and deployment >>> >>> root at compute1:/etc/kolla/nova-compute# ls >>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>> config.json nova.conf >>> >>> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ >>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>> >>> And I can confirm that the content is the same. >>> >>> >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney wrote: >>> >>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>> > I would investigate that compute error first. Creating volumes means >>>> the >>>> > controllers are doing the action. Starting a VM on a compute means >>>> you also >>>> > need Ceph to works on the compute to mount the rdb target. >>>> >>>> nova as part of its startup process in aintiallying the resouce tracker >>>> will >>>> try to connect to ceph if you are using the rbd image backend to report >>>> how much stroage >>>> is avaiable. if the keyring does not work on the vms pool as the user >>>> nova is connecting as >>>> then that will block the agent from starting up fully and will cause it >>>> to be missing the hypervior list. >>>> >>>> the error seams to indicate that the cinder keyring is not in the nova >>>> container >>>> that likely means you have not put it in /etc/kolla/config/nova >>>> i woudl check /etc/kolla/config/nova on the deployment host and sudo ls >>>> /etc/kolla/nova-compute/ >>>> on the compute node to ensure the cinder keyring is actully copied and >>>> has the correct content >>>> >>>> i have >>>> stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>> config.json nova.conf >>>> >>>> >>>> [client.cinder] >>>> key = ********************************* >>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>> caps mon = "profile rbd" >>>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>>> profile rbd pool=images" >>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>> [client.nova] >>>> key = ********************************* >>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>> caps mon = "profile rbd" >>>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>>> profile rbd pool=images" >>>> >>>> blanked out the key wiht *************** after the fact but you should >>>> have something similar >>>> >>>> >>>> in my case i decied to use a seperate key for nova rbd backend because >>>> i was also using EC poosl with a seperate data and metadata pool >>>> so i neede to modify my ceph.conf to make that work with kolla >>>> >>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>> /etc/kolla/nova-compute/ceph.conf >>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>> [global] >>>> fsid = ********************* >>>> mon_host = [*********************] >>>> >>>> [client.glance] >>>> rbd default data pool = images-data >>>> >>>> [client.cinder] >>>> rbd default data pool = volumes-data >>>> >>>> [client.nova] >>>> rbd default data pool = vms-data >>>> >>>> using 2 keyrings/user allows me to set different default data pools for >>>> cinder and nova. >>>> >>>> > >>>> > Working in Wallaby with the error doesn't mean it would 100% work in >>>> > Victoria. >>>> > >>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony >>>> wrote: >>>> > >>>> > > Hey Guys, Any other idea ? >>>> > > >>>> > > Regards >>>> > > >>>> > > Tony Karera >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony >>>> wrote: >>>> > > >>>> > > > Just to add on that, >>>> > > > >>>> > > > compute service is listed, I can create Volumes, I have the same >>>> cinder >>>> > > > keyring in the /etc/kolla/config/nova directory as I have in the >>>> > > > /etc/kolla/config/cinder/cinder-volume directory along with the >>>> nova keyring >>>> > > > Regards >>>> > > > >>>> > > > Tony Karera >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony >>>> wrote: >>>> > > > >>>> > > > > Hello Guys, >>>> > > > > >>>> > > > > Thanks a lot. >>>> > > > > >>>> > > > > I had actually checked the nova -compute.log on the compute >>>> node and >>>> > > > > they were showing the error I will post at the end about the >>>> cinder keyring >>>> > > > > but I know its correct because its the same I was using on >>>> wallaby, I even >>>> > > > > tried to use another ceph cluster with ofcouse different >>>> keyrings but its >>>> > > > > the same issue. >>>> > > > > >>>> > > > > Below is the error >>>> > > > > >>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: >>>> unable to >>>> > > > > find a keyring on >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 >>>> 7fbce2f4f700 -1 >>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>> auth: unable >>>> > > > > to find a keyring on >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>>> 7fbce2f4f700 -1 >>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>> auth: unable >>>> > > > > to find a keyring on >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>>> 7fbce2f4f700 -1 >>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>> connecting to the >>>> > > > > cluster)\n' >>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During >>>> handling of >>>> > > > > the above exception, another exception occurred: >>>> > > > > Regards >>>> > > > > >>>> > > > > Tony Karera >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney >>>> wrote: >>>> > > > > >>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>>> > > > > > > It could also be a compute cell discovery issue maybe? >>>> > > > > > no they shoudl still show up in the hypervior list api >>>> > > > > > > >>>> > > > > > > Do you see anything under "openstack compute service list"? >>>> > > > > > if they show up in the service list but not they hyperiors >>>> api it >>>> > > > > > means that the comptue service started and registered its >>>> service entry >>>> > > > > > but >>>> > > > > > something broke it before it could create a compute node >>>> recored in the >>>> > > > > > db. >>>> > > > > > >>>> > > > > > with ceph the case i have hit this most often is when the >>>> keyright used >>>> > > > > > by nova to >>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong whihc >>>> prevent >>>> > > > > > the resoucetack and compute manager >>>> > > > > > form actully creating the compute node record. >>>> > > > > > >>>> > > > > > >>>> > > > > > it can happen for other reason too but best place to start is >>>> check if >>>> > > > > > there is an error in the nova compute agent log and go from >>>> there. >>>> > > > > > > >>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>> smooney at redhat.com> >>>> > > > > > wrote: >>>> > > > > > > >>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >>>> > > > > > > > > Hello Team, >>>> > > > > > > > > >>>> > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible >>>> on Ubuntu >>>> > > > > > 20.04 >>>> > > > > > > > and >>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and Glance. >>>> > > > > > > > > >>>> > > > > > > > > It finished with no error but it has failed to register >>>> any on the >>>> > > > > > > > Compute >>>> > > > > > > > > Nodes under Hypervisors. >>>> > > > > > > > > >>>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack >>>> hypervisor list >>>> > > > > > > > > >>>> > > > > > > > > (kolla-openstack) stack at deployment:~$ >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > Any idea on how to resolve this ? >>>> > > > > > > > that usually means that somehthing prevented the comptue >>>> agent form >>>> > > > > > > > strating properly >>>> > > > > > > > >>>> > > > > > > > for example incorrect ceph keyrings there are several >>>> other case >>>> > > > > > but you >>>> > > > > > > > mentioned you are >>>> > > > > > > > using ceph. >>>> > > > > > > > >>>> > > > > > > > if this is hte case you should see error in the compute >>>> agent log. >>>> > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > Regards >>>> > > > > > > > > >>>> > > > > > > > > Tony Karera >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From elod.illes at est.tech Fri Sep 24 09:40:07 2021 From: elod.illes at est.tech (=?UTF-8?B?RWzFkWQgSWxsw6lz?=) Date: Fri, 24 Sep 2021 11:40:07 +0200 Subject: [release][qa][requirements][i18n] time to finalize stable branch setup Message-ID: Hi, All the projects that are enabled in devstack by default have been branched, so it's time to - [requirements]: create branch for 'requirements' to be able to lift the requirements freeze - [qa]: do the necessary QA preparation patches ( https://wiki.openstack.org/wiki/QA/releases ) - [i18n]: update the translation tools for the new stable series So I ask the PTLs of these teams to do the above necessary preparation. Thanks, El?d -------------- next part -------------- An HTML attachment was scrubbed... URL: From qiujunting at inspur.com Fri Sep 24 10:04:33 2021 From: qiujunting at inspur.com (=?gb2312?B?SnVudGluZ3FpdSBRaXVqdW50aW5nICjH8b785sMp?=) Date: Fri, 24 Sep 2021 10:04:33 +0000 Subject: [Sahara]Currently about the development of the Sahara community there are some points Message-ID: Hi all: Currently about the development of the Sahara community there are some points as following: 1. About the schedule of the regular meeting of the Sahara project? What is your suggestion? How about the regular meeting time every Wednesday afternoon 15:00 to 16:30? 2. Regarding the Sahara project maintenance switch from StoryBoard to launchpad. https://storyboard.openstack.org/ https://blueprints.launchpad.net/openstack/ The reasons are as follows: 1. OpenSatck core projects are maintained on launchpad, such as nova, cinder, neutron, etc. 2. Most OpenStack contributors are used to working on launchpad. 3. Do you have any suggestions? If you think this is feasible, I will post this content in the Sahara community later. Thank you for your help. Thank you Fossen. --------------------------------- Fossen Qiu | ??? CBRD | ?????????? T: 18249256272 E: qiujunting at inspur.com ???? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 3519 bytes Desc: image001.jpg URL: From ltoscano at redhat.com Fri Sep 24 10:24:17 2021 From: ltoscano at redhat.com (Luigi Toscano) Date: Fri, 24 Sep 2021 12:24:17 +0200 Subject: [Sahara]Currently about the development of the Sahara community there are some points In-Reply-To: References: Message-ID: <3225773.irdbgypaU6@whitebase.usersys.redhat.com> On Friday, 24 September 2021 12:04:33 CEST Juntingqiu Qiujunting (???) wrote: > Hi all: > > Hello, > > Currently about the development of the Sahara community there are some > points as following: > > > 1. About the schedule of the regular meeting of the Sahara project? What is > your suggestion? > How about the regular meeting time every Wednesday afternoon 15:00 to > 16:30? I guess you are proposing to change the current time, which is currently set for Thursday at 14.00 UTC, but which we haven't used much lately? If Wednesday 15.00 UTC (is that UTC) works better for you, why not? > > 2. Regarding the Sahara project maintenance switch from StoryBoard to > launchpad. > > https://storyboard.openstack.org/ > > https://blueprints.launchpad.net/openstack/ > > The reasons are as follows: > > 1. OpenSatck core projects are maintained on launchpad, such as > nova, cinder, neutron, etc. > 2. Most OpenStack contributors are used to working on launchpad. Not all OpenStack projects are maintained on launchpad, some switched to storyboard. Unfortunately the mass migration didn't happen and the storyboard development has slowed down. That said, Sahara switched to storyboard long ago and moving to launchpad will lose a lot of tickets and history. All the tickets are in storyboard. > > > 3. Do you have any suggestions? > > > > If you think this is feasible, I will post this content in the Sahara > community later. Thank you for your help. As this email is going to openstack-discuss, which other community? The IRC channel? -- Luigi From neil at tigera.io Fri Sep 24 10:48:58 2021 From: neil at tigera.io (Neil Jerram) Date: Fri, 24 Sep 2021 11:48:58 +0100 Subject: [devstack][requirements][stable/ussuri] New setuptools doesn't like suds-jurko? Message-ID: I use the stable/ussuri branches of devstack and tempest and have recently started seeing this stack.sh failure: Collecting suds-jurko===0.6 Downloading suds-jurko-0.6.tar.bz2 (143 kB) ERROR: Command errored out with exit status 1: command: /usr/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wsiug8gg/suds-jurko/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wsiug8gg/suds-jurko/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-h8s2x8w4 cwd: /tmp/pip-install-wsiug8gg/suds-jurko/ Complete output (1 lines): error in suds-jurko setup command: use_2to3 is invalid. IIUC that last error is from setuptools, specifically version 58.0.4. In previous runs with setuptools 57.4.0, this error did not occur. Does it mean that stable/ussuri global-requirements.txt for setuptools should have "<58" ? I haven't investigated why suds-jurko is pulled in, but I guess it's from the reference Neutron code - as opposed to something that my particular testing has added - and so this could affect anyone using stable/ussuri devstack. Best wishes, Neil -- Neil Jerram Principal Software Engineer Tigera neil at tigera.io | @neiljerram Follow Tigera: Blog | Twitter | LinkedIn Leader in Kubernetes Security and Observability -------------- next part -------------- An HTML attachment was scrubbed... URL: From balazs.gibizer at est.tech Fri Sep 24 10:58:46 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Fri, 24 Sep 2021 12:58:46 +0200 Subject: [devstack][requirements][stable/ussuri] New setuptools doesn't like suds-jurko? In-Reply-To: References: Message-ID: On Fri, Sep 24 2021 at 11:48:58 AM +0100, Neil Jerram wrote: > I use the stable/ussuri branches of devstack and tempest and have > recently started seeing this stack.sh failure: > > Collecting suds-jurko===0.6 > Downloading suds-jurko-0.6.tar.bz2 (143 kB) > ERROR: Command errored out with exit status 1: > command: /usr/bin/python3.6 -c 'import sys, setuptools, > tokenize; sys.argv[0] = > '"'"'/tmp/pip-install-wsiug8gg/suds-jurko/setup.py'"'"'; > __file__='"'"'/tmp/pip-install-wsiug8gg/suds-jurko/setup.py'"'"';f=getattr(tokenize, > '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', > '"'"'\n'"'"');f.close();exec(compile(code, __file__, > '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-h8s2x8w4 > cwd: /tmp/pip-install-wsiug8gg/suds-jurko/ > Complete output (1 lines): > error in suds-jurko setup command: use_2to3 is invalid. > > IIUC that last error is from setuptools, specifically version 58.0.4. > In previous runs with setuptools 57.4.0, this error did not occur. It is correct. setuptools >= 58.0 removed support for use_2to3 > > Does it mean that stable/ussuri global-requirements.txt for > setuptools should have "<58" ? > > I haven't investigated why suds-jurko is pulled in, but I guess it's > from the reference Neutron code - as opposed to something that my > particular testing has added - and so this could affect anyone using > stable/ussuri devstack. Does suds-jurko 0.6 is you upper constraint? If it is not your upper constraint the you can look at if suds-jurko has a newer minor version that works with latest setuptools. If this is your upper then I think we have a situation where setuptools needs to be pinned on stable not due to lower constraints (where it seems we say we don't care and drop lower testing) but due to upper constraints. See the other mailthread [1] about our options regarding this setuptools issues. Cheers, gibi [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024987.html > > Best wishes, > Neil > > > -- > Neil Jerram > > Principal Software Engineer > > Tigera > > neil at tigera.io | @neiljerram > > Follow Tigera: Blog | Twitter | LinkedIn > > > Leader in Kubernetes Security and Observability > > From pierre at stackhpc.com Fri Sep 24 11:11:19 2021 From: pierre at stackhpc.com (Pierre Riteau) Date: Fri, 24 Sep 2021 13:11:19 +0200 Subject: [requirements] Please consider bumping docker to 5.0.2 in Xena u-c Message-ID: Hello, Xena upper constraints currently contain docker 5.0.1, released on August 31. This version includes a critical issue [1] which was fixed by version 5.0.2 released the next day. Except for a docstring update this is the only change, according to git diff. Would it be possible to update Xena u-c to use docker 5.0.2? We currently have to work around it in Kayobe to avoid installing the buggy version [2]. Thanks, Pierre Riteau (priteau) [1] https://github.com/docker/docker-py/issues/2885 [2] https://review.opendev.org/c/openstack/kayobe/+/807345/9/ansible/kolla-target-venv.yml From neil at tigera.io Fri Sep 24 11:30:36 2021 From: neil at tigera.io (Neil Jerram) Date: Fri, 24 Sep 2021 12:30:36 +0100 Subject: [devstack][requirements][stable/ussuri] New setuptools doesn't like suds-jurko? In-Reply-To: References: Message-ID: Many thanks gibi! On Fri, Sep 24, 2021 at 11:58 AM Balazs Gibizer wrote: > > Does suds-jurko 0.6 is you upper constraint? If it is not your upper > constraint the you can look at if suds-jurko has a newer minor version > that works with latest setuptools. If this is your upper then I think > we have a situation where setuptools needs to be pinned on stable not > due to lower constraints (where it seems we say we don't care and drop > lower testing) but due to upper constraints. See the other mailthread > [1] about our options regarding this setuptools issues. > AFAIK, my build and coding don't impose any additional constraints. (Is that what you meant?) 0.6 is the latest on PyPI. https://github.com/Affirm/suds-jurko/releases has a 0.6.4 release, but I don't know if that is the official GitHub project for suds-jurko (or if it addresses the 2to3 problem). > > Cheers, > gibi > > [1] > > http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024987.html I will follow up in that thread to support Option 2. -------------- next part -------------- An HTML attachment was scrubbed... URL: From neil at tigera.io Fri Sep 24 11:33:14 2021 From: neil at tigera.io (Neil Jerram) Date: Fri, 24 Sep 2021 12:33:14 +0100 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: On Wed, Sep 22, 2021 at 5:07 PM Balazs Gibizer wrote: > > I agree not to pin this on master. We need to support the latest and > greatest on master. But on stable I don't see the problem saying we > only support the tooling version that was the latest when the major > release was created. > +1, this seems like a straightforward fix to me. https://review.opendev.org/c/openstack/requirements/+/810859 proposes what I think is the right change. Is there any downside? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sombrafam at gmail.com Fri Sep 24 11:50:00 2021 From: sombrafam at gmail.com (Erlon Cruz) Date: Fri, 24 Sep 2021 08:50:00 -0300 Subject: Neutron FIP Qos with OVN Backend In-Reply-To: References: <5698647.Ru2sQNNtPn@p1> Message-ID: Hi folks, I'm confused now. I have recently read that [1]: "Currently the Neutron L3-agent supports floating IP and gateway IP bandwidth limiting based on Linux TC. OVN L3 plugin supports floating IP bandwidth limiting based on the OVN?s QoS rules. Neutron OVN backend does not yet support bandwidth limiting for gateway IP" So, QoS, especially FIP QoS is not present in OVN yet. I'm I missing something? If it supports, what versions that was added? Erlon ________________ [1] https://docs.openstack.org/neutron/latest/ovn/gaps.html Em qui., 23 de set. de 2021 ?s 08:37, Rodolfo Alonso Hernandez < ralonsoh at redhat.com> escreveu: > Hello Ammad: > > This is not possible. You can create a default QoS policy that will apply > to a network and thus to the internal ports created on this network. > > When you create a FIP, a port on the external network is created and this > port receives a QoS policy (the default policy). However, the FIP QoS > driver reads the *FIP QoS* policy, not the associated *FIP port QoS* > policy. That means, for now, what you need is not possible. > > Regards. > > > On Thu, Sep 23, 2021 at 1:14 PM Slawek Kaplonski > wrote: > >> Hi, >> >> On czwartek, 23 wrze?nia 2021 12:33:13 CEST Ammad Syed wrote: >> > Hello, >> > >> > I am using Neutron wallaby with OVN backend. I am testing QoS FIP and it >> > works fine. >> > >> > I have created a QoS policy and attached it with FIP, It works fine. Is >> it >> > possible to set one of the QoS policy as default that each time I >> acquire a >> > floating IP, the QoS should be attached to it by default ? >> > >> > Ammad >> >> You can mark QoS policy as default for the project: https:// >> >> docs.openstack.org/api-ref/network/v2/index.html?expanded=create-qos-policy- >> detail#qos-default-extension >> >> But TBH I'm not sure if that will work with FIPs - You should test it and >> if >> that will not work, probably You can open bug for Neutron about that :) >> >> -- >> Slawek Kaplonski >> Principal Software Engineer >> Red Hat > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralonsoh at redhat.com Fri Sep 24 13:12:07 2021 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Fri, 24 Sep 2021 15:12:07 +0200 Subject: Neutron FIP Qos with OVN Backend In-Reply-To: References: <5698647.Ru2sQNNtPn@p1> Message-ID: Hello Erlon: "OVN L3 plugin supports floating IP bandwidth limiting based on the OVN?s QoS rules". That means OVN L3 plugin supports FIP QoS. This feature is supported since Wallaby. OVN L3 plugin still does not support QoS on router GW port. Regards. On Fri, Sep 24, 2021 at 1:50 PM Erlon Cruz wrote: > Hi folks, > > I'm confused now. I have recently read that [1]: > > "Currently the Neutron L3-agent supports floating IP and gateway IP > bandwidth limiting based on Linux TC. OVN L3 plugin supports floating IP > bandwidth limiting based on the OVN?s QoS rules. Neutron OVN backend does > not yet support bandwidth limiting for gateway IP" > > So, QoS, especially FIP QoS is not present in OVN yet. I'm I missing > something? If it supports, what versions that was added? > > Erlon > ________________ > [1] https://docs.openstack.org/neutron/latest/ovn/gaps.html > > > > Em qui., 23 de set. de 2021 ?s 08:37, Rodolfo Alonso Hernandez < > ralonsoh at redhat.com> escreveu: > >> Hello Ammad: >> >> This is not possible. You can create a default QoS policy that will apply >> to a network and thus to the internal ports created on this network. >> >> When you create a FIP, a port on the external network is created and this >> port receives a QoS policy (the default policy). However, the FIP QoS >> driver reads the *FIP QoS* policy, not the associated *FIP port QoS* >> policy. That means, for now, what you need is not possible. >> >> Regards. >> >> >> On Thu, Sep 23, 2021 at 1:14 PM Slawek Kaplonski >> wrote: >> >>> Hi, >>> >>> On czwartek, 23 wrze?nia 2021 12:33:13 CEST Ammad Syed wrote: >>> > Hello, >>> > >>> > I am using Neutron wallaby with OVN backend. I am testing QoS FIP and >>> it >>> > works fine. >>> > >>> > I have created a QoS policy and attached it with FIP, It works fine. >>> Is it >>> > possible to set one of the QoS policy as default that each time I >>> acquire a >>> > floating IP, the QoS should be attached to it by default ? >>> > >>> > Ammad >>> >>> You can mark QoS policy as default for the project: https:// >>> >>> docs.openstack.org/api-ref/network/v2/index.html?expanded=create-qos-policy- >>> detail#qos-default-extension >>> >>> But TBH I'm not sure if that will work with FIPs - You should test it >>> and if >>> that will not work, probably You can open bug for Neutron about that :) >>> >>> -- >>> Slawek Kaplonski >>> Principal Software Engineer >>> Red Hat >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From aschultz at redhat.com Fri Sep 24 13:19:03 2021 From: aschultz at redhat.com (Alex Schultz) Date: Fri, 24 Sep 2021 07:19:03 -0600 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: <6J4UZQ.VOBD0LVDTPUX1@est.tech> References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: On Wed, Sep 22, 2021 at 6:13 AM Balazs Gibizer wrote: > Hi, > > > The latest setuptools (58.0) removed support for "use_2to3" [1] > (deprecated since 46.2). Many OpenStack modules defines > decorator==3.4.0 as lower constraints[2]. However, decorator 3.4.0 > cannot be installed anymore with the latest setuptool as it depends on > "use_2to3". > > JFYI as I was looking into some other requirements issues yesterday, I hit this error with anyjson[0] 0.3.3 as well. It's used in a handful of projects[1] and there has not been a release since 2012[2] so this might be a problem in xena. I haven't checked the projects respective gates, but just want to highlight we'll probably have additional fallout from the setuptools change. [0] https://github.com/pypa/setuptools/issues/1120#issuecomment-919239967 [1] https://codesearch.opendev.org/?q=anyjson&i=nope&literal=nope&files=&excludeFiles=&repos= [2] https://pypi.org/project/anyjson/#history > On master, this can be solved easily by bumping the dependency to > decorator 4.0.0. On stable/xena we can still solve it the same way with > a new RC. But on older stable branches such solution might be against > the stable policy as it would require a major bump on our dependencies. > > This issue is not limited to lower-constraints testing it just hit us > there first. A similar break could happen in our upper constraints as > well on old stable branches. > > The root of the problem is that we always use the latest setuptools in > our CI testing even on old stable branches. Zuul's ensure-tox task[4] > installs tox which installs the virtualenv package which bundles the > latest setuptools package. This happens without applying any > constraints. Then tox is used to install the project's dependencies > under lower or upper constraints with the unconstrained setuptools > available. > > During and after yesterday's nova meeting [3] we discussed possible > solutions. > > Option 1: Bump the major version of the decorator dependency on stable. > Pros: > * easy to implement > Cons: > * might against the policy / breaks downstream packagers > * needs to be done in each affected project[3] separately > * not future proof. After a future setuptools release we can see a > similar break with another of our dependencies. > > @Stable Cores: how do you feel about such bump? > > > Option 2: Pin the setuptools version during tox installation > Pros: > * single solution for all affected projects > * future proof against breaks introduced by future setuptools releases > Cons: > * complex change as it probably requires to extend the base task in > zuul itself > > > Option 3: turn off lower-constraints testing > Pros: > * easy to implement > * some of us want to get rid of lower constraints anyhow > Cons: > * needs per project changes > * not future proof against similar break affecting our upper > constraints on old stable branches. > > > Option 4: utilize pyproject.toml[6] to specify build-time requirements > Unfortunately, I'm not sure if it is possible to restrict the maximum > setuptools version with it > > > As a side note I tried applying [tox]requires in tox.ini to restrict > setuptools version[7] but It does not seem to take effect in our > case[8]. > > > @Stable Cores: what do you think what could be the way forward? > > Cheers, > gibi > > > [1] https://setuptools.pypa.io/en/latest/history.html#v58-0-2 > [2] > > https://codesearch.openstack.org/?q=decorator%3D%3D3.4.0&i=nope&literal=nope&files=&excludeFiles=&repos= > [3] > > https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2021-09-21.log.html#t2021-09-21T16:31:49 > [4] > > https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks/main.yaml#L28 > [5] https://tox.readthedocs.io/en/latest/config.html#conf-requires > [6] https://www.python.org/dev/peps/pep-0518/ > [7] https://review.opendev.org/c/openstack/placement/+/810293/1/tox.ini > [8] > > https://zuul.opendev.org/t/openstack/build/f78a0300c8734e7c8475309af1d2e1a4/log/tox/lower-constraints-0.log#7 > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremyfreudberg at gmail.com Fri Sep 24 13:50:50 2021 From: jeremyfreudberg at gmail.com (Jeremy Freudberg) Date: Fri, 24 Sep 2021 09:50:50 -0400 Subject: [Sahara]Currently about the development of the Sahara community there are some points In-Reply-To: References: Message-ID: Hi Fossen, thanks for your questions. I don't have a comment on the specific meeting time. It sounds like the proposed time is more convenient for new contributors. Switching to Launchpad is not the best idea. Sahara switched to Storyboard years ago and I think it is not possible to migrate backwards. On Fri, Sep 24, 2021 at 6:05 AM Juntingqiu Qiujunting (???) < qiujunting at inspur.com> wrote: > Hi all: > > > > Currently about the development of the Sahara community there are some > points as following: > > > > 1. About the schedule of the regular meeting of the Sahara project? What > is your suggestion? > > > > How about the regular meeting time every Wednesday afternoon 15:00 to > 16:30? > > > > 2. Regarding the Sahara project maintenance switch from StoryBoard to > launchpad. > > > > https://storyboard.openstack.org/ > > https://blueprints.launchpad.net/openstack/ > > The reasons are as follows: > > 1. OpenSatck core projects are maintained on launchpad, such as > nova, cinder, neutron, etc. > > 2. Most OpenStack contributors are used to working on launchpad. > > > > 3. Do you have any suggestions? > > > > If you think this is feasible, I will post this content in the Sahara > community later. Thank you for your help. > > > > Thank you Fossen. > > > > > > --------------------------------- > > Fossen Qiu *|* ??? > > > > > > CBRD * |* ?????????? > > > > > > *T:* 18249256272 > > > > > > *E:* qiujunting at inspur.com > > > > > > > > [image: signature_1653277958] > > ???? > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 3519 bytes Desc: not available URL: From mthode at mthode.org Fri Sep 24 14:30:32 2021 From: mthode at mthode.org (Matthew Thode) Date: Fri, 24 Sep 2021 09:30:32 -0500 Subject: [requirements] Please consider bumping docker to 5.0.2 in Xena u-c In-Reply-To: References: Message-ID: <20210924143032.gx4d4pzicug6s6ad@mthode.org> After we branch, ya, probably On 21-09-24 13:11:19, Pierre Riteau wrote: > Hello, > > Xena upper constraints currently contain docker 5.0.1, released on > August 31. This version includes a critical issue [1] which was fixed > by version 5.0.2 released the next day. Except for a docstring update > this is the only change, according to git diff. > > Would it be possible to update Xena u-c to use docker 5.0.2? We > currently have to work around it in Kayobe to avoid installing the > buggy version [2]. > > Thanks, > Pierre Riteau (priteau) > > [1] https://github.com/docker/docker-py/issues/2885 > [2] https://review.opendev.org/c/openstack/kayobe/+/807345/9/ansible/kolla-target-venv.yml > -- Matthew Thode From fungi at yuggoth.org Fri Sep 24 14:36:01 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 24 Sep 2021 14:36:01 +0000 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: <20210924143600.yfjuxerlid52vlji@yuggoth.org> On 2021-09-24 07:19:03 -0600 (-0600), Alex Schultz wrote: [...] > JFYI as I was looking into some other requirements issues > yesterday, I hit this error with anyjson[0] 0.3.3 as well. It's > used in a handful of projects[1] and there has not been a release > since 2012[2] so this might be a problem in xena. I haven't > checked the projects respective gates, but just want to highlight > we'll probably have additional fallout from the setuptools change. [...] Yes, we've also run into similar problems with pydot2 and funcparserlib, and I'm sure there's plenty more of what is effectively abandonware lingering in various projects' requirements lists. The long and short of it is that people with newer versions of SetupTools are going to be unable to install those, full stop. The maintainers of some of them may be spurred to action and release a new version, but in so doing may also drop support for older interpreters we still test with on some stable branches (this was the case with funcparserlib). On the other hand, controlling what version of SetupTools others have and use isn't always possible, unlike runtime dependencies, so that really should be a solution of last resort. Making exceptions to stable branch policy in unusual circumstances such as this seems like a reasonable and more effective compromise. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From fungi at yuggoth.org Fri Sep 24 14:38:21 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 24 Sep 2021 14:38:21 +0000 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: <20210924143821.ht2qd23gkvii4itn@yuggoth.org> On 2021-09-24 12:33:14 +0100 (+0100), Neil Jerram wrote: > On Wed, Sep 22, 2021 at 5:07 PM Balazs Gibizer > wrote: > > > > > I agree not to pin this on master. We need to support the latest and > > greatest on master. But on stable I don't see the problem saying we > > only support the tooling version that was the latest when the major > > release was created. > > > > +1, this seems like a straightforward fix to me. > https://review.opendev.org/c/openstack/requirements/+/810859 proposes what > I think is the right change. Is there any downside? Did that change actually fix the observed problem with tox jobs? Last I recall, the requirements/constraints on SetupTools were only effectively applied by DevStack because that includes a separate step for installing or downgrading it. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From neil at tigera.io Fri Sep 24 15:07:42 2021 From: neil at tigera.io (Neil Jerram) Date: Fri, 24 Sep 2021 16:07:42 +0100 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: <20210924143821.ht2qd23gkvii4itn@yuggoth.org> References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> <20210924143821.ht2qd23gkvii4itn@yuggoth.org> Message-ID: On Fri, Sep 24, 2021 at 3:38 PM Jeremy Stanley wrote: > On 2021-09-24 12:33:14 +0100 (+0100), Neil Jerram wrote: > > On Wed, Sep 22, 2021 at 5:07 PM Balazs Gibizer > > wrote: > > > > > > > > I agree not to pin this on master. We need to support the latest and > > > greatest on master. But on stable I don't see the problem saying we > > > only support the tooling version that was the latest when the major > > > release was created. > > > > > > > +1, this seems like a straightforward fix to me. > > https://review.opendev.org/c/openstack/requirements/+/810859 proposes > what > > I think is the right change. Is there any downside? > > Did that change actually fix the observed problem with tox jobs? > Last I recall, the requirements/constraints on SetupTools were only > effectively applied by DevStack because that includes a separate > step for installing or downgrading it. > I'm afraid I haven't yet tested it, or worked out how I would test it prior to merge - any hints for that? My reasoning is based on reading the stack.sh stdout and underlying devstack code (for stable/ussuri): + tools/install_pip.sh:main:152 : pip_install_gr setuptools + inc/python:pip_install_gr:76 : local name=setuptools + inc/python:pip_install_gr:77 : local clean_name ++ inc/python:pip_install_gr:78 : get_from_global_requirements setuptools ++ inc/python:get_from_global_requirements:238 : local package=setuptools ++ inc/python:get_from_global_requirements:239 : local required_pkg +++ inc/python:get_from_global_requirements:240 : grep -i -h '^setuptools' /opt/stack/requirements/global-requirements.txt +++ inc/python:get_from_global_requirements:240 : cut -d# -f1 ++ inc/python:get_from_global_requirements:240 : required_pkg='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\'' setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\'' ' ++ inc/python:get_from_global_requirements:241 : [[ setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='3.5' setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7' == '' ]] ++ inc/python:get_from_global_requirements:244 : echo 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\''' 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\''' + inc/python:pip_install_gr:78 : clean_name='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\'' setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\''' + inc/python:pip_install_gr:79 : pip_install 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\''' 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\''' Using python 3.6 to install setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7' because python3_enabled=True It clearly seems to be installing a version of setuptools as constrained by global-requirements.txt, and it appears that that installed version is the relevant factor, because runs with 58.* fail and older runs with 57.* succeeded. -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.sgaravatto at gmail.com Fri Sep 24 15:17:46 2021 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Fri, 24 Sep 2021 17:17:46 +0200 Subject: [ops][neutron] Is it possible to "lock" a floating IP to an instance ? In-Reply-To: <1d449263168f0e90524ca0c5d7b9d761959c82be.camel@redhat.com> References: <1d449263168f0e90524ca0c5d7b9d761959c82be.camel@redhat.com> Message-ID: Thanks a lot I tried associating the floating IP using: curl -i "${NOVA_ENDPOINT_URL}/${TENANT_ID}/servers/${SERVER}/action" -X POST -H "X-Auth-Project-Id: ${TENANT_ID}" -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: $TOKEN" -d '{"addFloatingIp": {"address": "90.147.77.102"}}' I hope this is what you mean with "using novas api to manage floating ips" Then I locked the instance However another user is then still able to disassociate that floating IP Cheers, Massimo On Thu, Sep 23, 2021 at 12:39 PM Sean Mooney wrote: > On Thu, 2021-09-23 at 12:20 +0200, Massimo Sgaravatto wrote: > > Hello > > > > I have the following use case: > > > > A user creates a VM and associates a floating IP to such instance > > > > Is in some way possible to prevent that the floating IP is > > disassociated from that instance by another user of the same project ? > > > > If it helps, the user owning the instance could be admin (but allowing > only > > the admin user to manage floating IPs is not an option) > > if you are using novas api to manage floating ips then you might be able > to lock the instnace which should prevent changing > the ip assocations and most other instnace actions however if you were to > manage teh floating ips form neutron that ouls entirly bypass that. > we had talk about adding the ablity to lock ports for a different usecasue > and haing nova lock the port whenever an instance is locked > that might be the way to adress this in the future but for now i dont > think you can do this without custom midelware. > > > > > > Thanks, Massimo > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From openinfradn at gmail.com Fri Sep 24 15:18:16 2021 From: openinfradn at gmail.com (open infra) Date: Fri, 24 Sep 2021 20:48:16 +0530 Subject: Creating multiple servers with a shared volume Message-ID: Hi, I am planning to spin hundreds of servers with a shared storage. I have created one instance and attached a 1 TB volume (multiattach) and created a snapshot of the instance. But when I spin another server using the snapshot, I noticed that instead of expected shared volume, individual 1 TB size additional (dedicated) volume has been attached to each VM. Is there a mechanism where I can create multiple instances along with a shared volume attached to the instance instead of attaching later? Regards, Danishka -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Fri Sep 24 15:19:27 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Fri, 24 Sep 2021 08:19:27 -0700 Subject: =?UTF-8?Q?Re:_[stable][requirements][zuul]_unpinned_setuptools_dependenc?= =?UTF-8?Q?y_on_stable?= In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> <20210924143821.ht2qd23gkvii4itn@yuggoth.org> Message-ID: <9c410982-ad47-45c1-87da-bc3edbe54270@www.fastmail.com> On Fri, Sep 24, 2021, at 8:07 AM, Neil Jerram wrote: > On Fri, Sep 24, 2021 at 3:38 PM Jeremy Stanley wrote: >> On 2021-09-24 12:33:14 +0100 (+0100), Neil Jerram wrote: >> > On Wed, Sep 22, 2021 at 5:07 PM Balazs Gibizer >> > wrote: >> > >> > > >> > > I agree not to pin this on master. We need to support the latest and >> > > greatest on master. But on stable I don't see the problem saying we >> > > only support the tooling version that was the latest when the major >> > > release was created. >> > > >> > >> > +1, this seems like a straightforward fix to me. >> > https://review.opendev.org/c/openstack/requirements/+/810859 proposes what >> > I think is the right change. Is there any downside? >> >> Did that change actually fix the observed problem with tox jobs? >> Last I recall, the requirements/constraints on SetupTools were only >> effectively applied by DevStack because that includes a separate >> step for installing or downgrading it. > > I'm afraid I haven't yet tested it, or worked out how I would test it > prior to merge - any hints for that? > > My reasoning is based on reading the stack.sh stdout and underlying > devstack code (for stable/ussuri): > > + tools/install_pip.sh:main:152 : pip_install_gr setuptools > + inc/python:pip_install_gr:76 : local name=setuptools > + inc/python:pip_install_gr:77 : local clean_name > ++ inc/python:pip_install_gr:78 : > get_from_global_requirements setuptools > ++ inc/python:get_from_global_requirements:238 : local > package=setuptools > ++ inc/python:get_from_global_requirements:239 : local required_pkg > +++ inc/python:get_from_global_requirements:240 : grep -i -h > '^setuptools' /opt/stack/requirements/global-requirements.txt > +++ inc/python:get_from_global_requirements:240 : cut -d# -f1 > ++ inc/python:get_from_global_requirements:240 : > required_pkg='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\'' > > setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\'' > ' > ++ inc/python:get_from_global_requirements:241 : [[ > setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='3.5' > > setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7' > == '' ]] > ++ inc/python:get_from_global_requirements:244 : echo > 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\''' > 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\''' > + inc/python:pip_install_gr:78 : > clean_name='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\'' > setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\''' > + inc/python:pip_install_gr:79 : pip_install > 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\''' > 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\''' > Using python 3.6 to install > setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7' > because python3_enabled=True > > It clearly seems to be installing a version of setuptools as > constrained by global-requirements.txt, and it appears that that > installed version is the relevant factor, because runs with 58.* fail > and older runs with 57.* succeeded. Yup, as Fungi mentions this will work for devstack because devstack manages the setuptools installation. It won't work for tox because requirements and constraints are applied after tox+virtualenv have installed setuptools. That said there may not be a one size fits all fix here and we'll have to address it several different ways. I think Fungi's concern is that we think a single fix has corrected the issue when that isn't the case. From neil at tigera.io Fri Sep 24 15:33:05 2021 From: neil at tigera.io (Neil Jerram) Date: Fri, 24 Sep 2021 16:33:05 +0100 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: <9c410982-ad47-45c1-87da-bc3edbe54270@www.fastmail.com> References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> <20210924143821.ht2qd23gkvii4itn@yuggoth.org> <9c410982-ad47-45c1-87da-bc3edbe54270@www.fastmail.com> Message-ID: On Fri, Sep 24, 2021 at 4:19 PM Clark Boylan wrote: > On Fri, Sep 24, 2021, at 8:07 AM, Neil Jerram wrote: > > On Fri, Sep 24, 2021 at 3:38 PM Jeremy Stanley > wrote: > >> On 2021-09-24 12:33:14 +0100 (+0100), Neil Jerram wrote: > >> > On Wed, Sep 22, 2021 at 5:07 PM Balazs Gibizer > > >> > wrote: > >> > > >> > > > >> > > I agree not to pin this on master. We need to support the latest and > >> > > greatest on master. But on stable I don't see the problem saying we > >> > > only support the tooling version that was the latest when the major > >> > > release was created. > >> > > > >> > > >> > +1, this seems like a straightforward fix to me. > >> > https://review.opendev.org/c/openstack/requirements/+/810859 > proposes what > >> > I think is the right change. Is there any downside? > >> > >> Did that change actually fix the observed problem with tox jobs? > >> Last I recall, the requirements/constraints on SetupTools were only > >> effectively applied by DevStack because that includes a separate > >> step for installing or downgrading it. > > > > I'm afraid I haven't yet tested it, or worked out how I would test it > > prior to merge - any hints for that? > > > > My reasoning is based on reading the stack.sh stdout and underlying > > devstack code (for stable/ussuri): > > > > + tools/install_pip.sh:main:152 : pip_install_gr setuptools > > + inc/python:pip_install_gr:76 : local name=setuptools > > + inc/python:pip_install_gr:77 : local clean_name > > ++ inc/python:pip_install_gr:78 : > > get_from_global_requirements setuptools > > ++ inc/python:get_from_global_requirements:238 : local > > package=setuptools > > ++ inc/python:get_from_global_requirements:239 : local required_pkg > > +++ inc/python:get_from_global_requirements:240 : grep -i -h > > '^setuptools' /opt/stack/requirements/global-requirements.txt > > +++ inc/python:get_from_global_requirements:240 : cut -d# -f1 > > ++ inc/python:get_from_global_requirements:240 : > > > required_pkg='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\'' > > > > > > setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\'' > > > ' > > ++ inc/python:get_from_global_requirements:241 : [[ > > > setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='3.5' > > > > > > setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7' > > > == '' ]] > > ++ inc/python:get_from_global_requirements:244 : echo > > > 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\''' > > > > 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\''' > > + inc/python:pip_install_gr:78 : > > > clean_name='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\'' > > > > setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\''' > > + inc/python:pip_install_gr:79 : pip_install > > > 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\''' > > > > 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\''' > > Using python 3.6 to install > > > setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7' > > > because python3_enabled=True > > > > It clearly seems to be installing a version of setuptools as > > constrained by global-requirements.txt, and it appears that that > > installed version is the relevant factor, because runs with 58.* fail > > and older runs with 57.* succeeded. > > Yup, as Fungi mentions this will work for devstack because devstack > manages the setuptools installation. It won't work for tox because > requirements and constraints are applied after tox+virtualenv have > installed setuptools. > > That said there may not be a one size fits all fix here and we'll have to > address it several different ways. I think Fungi's concern is that we think > a single fix has corrected the issue when that isn't the case. > Ah right, I see more of the complexity now. Still, if a better overall solution can't be found quickly, I think it would be better to have some things working than nothing working. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Fri Sep 24 17:05:17 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Fri, 24 Sep 2021 10:05:17 -0700 Subject: All Devstack Based Jobs Failing to Create Swap on master Message-ID: Hello everyone, At approximately 12:53 UTC September 24 (today) [0] Devstack's Xena branch was created. Due to an unexpected side effect in Devstack's Zuul configs this has caused all Devstack based jobs against master and possible stable/xena to RETRY_LIMIT and fail. The underlying issue you'll see in the jobs is that mkswap cannot create swap because it already exists [1]. This happens because Devstack's pre-run playbook is running twice [2] and the configure-swap role is not idempotent. The role is running twice because the Devstack base job is loading master and stable/xena's configs and composing them together which results in two pre-run playbooks being selected one for each branch. This happens because Devstack's stable/xena branch explicitly states that its configs apply to master and feature/r1 branches [3]. I've pushed up a change [4] to remove this Zuul configuration from the Xena branch which I expect will fix things for us. [0] https://zuul.opendev.org/t/openstack/build/ebe77aee0c104c9e9b917ff28a4731ee/log/job-output.txt#715 [1] https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9/log/job-output.txt#1138 [2] https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9/log/job-output.txt#923 [3] https://opendev.org/openstack/devstack/src/branch/stable/xena/.zuul.yaml#L1-L7 [4] https://review.opendev.org/c/openstack/devstack/+/810947 From gouthampravi at gmail.com Fri Sep 24 17:48:25 2021 From: gouthampravi at gmail.com (Goutham Pacha Ravi) Date: Fri, 24 Sep 2021 10:48:25 -0700 Subject: All Devstack Based Jobs Failing to Create Swap on master In-Reply-To: References: Message-ID: On Fri, Sep 24, 2021 at 10:14 AM Clark Boylan wrote: > Hello everyone, > > At approximately 12:53 UTC September 24 (today) [0] Devstack's Xena branch > was created. Due to an unexpected side effect in Devstack's Zuul configs > this has caused all Devstack based jobs against master and possible > stable/xena to RETRY_LIMIT and fail. > > The underlying issue you'll see in the jobs is that mkswap cannot create > swap because it already exists [1]. This happens because Devstack's pre-run > playbook is running twice [2] and the configure-swap role is not > idempotent. The role is running twice because the Devstack base job is > loading master and stable/xena's configs and composing them together which > results in two pre-run playbooks being selected one for each branch. This > happens because Devstack's stable/xena branch explicitly states that its > configs apply to master and feature/r1 branches [3]. > > I've pushed up a change [4] to remove this Zuul configuration from the > Xena branch which I expect will fix things for us. > Thanks Clark - this was puzzling. Made a depends-on change in a devstack job elsewhere, and it went past the failure. > > [0] > https://zuul.opendev.org/t/openstack/build/ebe77aee0c104c9e9b917ff28a4731ee/log/job-output.txt#715 > [1] > https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9/log/job-output.txt#1138 > [2] > https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9/log/job-output.txt#923 > [3] > https://opendev.org/openstack/devstack/src/branch/stable/xena/.zuul.yaml#L1-L7 > [4] https://review.opendev.org/c/openstack/devstack/+/810947 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elod.illes at est.tech Fri Sep 24 19:41:21 2021 From: elod.illes at est.tech (=?UTF-8?B?RWzFkWQgSWxsw6lz?=) Date: Fri, 24 Sep 2021 21:41:21 +0200 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: <972f884e-e509-fc0d-ebfc-4414fa18cc64@est.tech> I completely agree with gibi: - on master (and thus, stable/xena) we need to support the latest setuptools - on older stable branches almost everything are already constrained (upper-constraints.txt), so I think that pinning setuptools on older branches is not that different than what we have now, plus it could eliminate a lot of possible similar issues in the future (I know, everything is 'temporary' as nothing will work forever, but if we want to make the stable maintenance easier then this is the better approach) For me gibi's patch [1] looks quite promising for old stable branches. I know that it means that we have to create a similar patch for (at least) all broken old stable branches, but it unblocks the gate and seems quite trivial change. Thanks, El?d [1] https://review.opendev.org/c/openstack/nova/+/810461 On 2021. 09. 22. 18:06, Balazs Gibizer wrote: > > > On Wed, Sep 22 2021 at 08:39:03 AM -0700, Clark Boylan > wrote: >> On Wed, Sep 22, 2021, at 5:11 AM, Balazs Gibizer wrote: >>> ?Hi, >>> >> >> Snip >> >>> >>> ?The root of the problem is that we always use the latest setuptools in >>> ?our CI testing even on old stable branches. Zuul's ensure-tox task[4] >>> ?installs tox which installs the virtualenv package which bundles the >>> ?latest setuptools package. This happens without applying any >>> ?constraints. Then tox is used to install the project's dependencies >>> ?under lower or upper constraints with the unconstrained setuptools >>> ?available. >>> >>> ?During and after yesterday's nova meeting [3] we discussed possible >>> ?solutions. >>> >>> ?Option 1: Bump the major version of the decorator dependency on >>> stable. >>> ?Pros: >>> ?* easy to implement >>> ?Cons: >>> ?* might against the policy / breaks downstream packagers >>> ?* needs to be done in each affected project[3] separately >>> ?* not future proof. After a future setuptools release we can see a >>> ?similar break with another of our dependencies. >>> >>> ?@Stable Cores: how do you feel about such bump? >>> >>> >>> ?Option 2: Pin the setuptools version during tox installation >>> ?Pros: >>> ?* single solution for all affected projects >>> ?* future proof against breaks introduced by future setuptools releases >>> ?Cons: >>> ?* complex change as it probably requires to extend the base task in >>> ?zuul itself >> >> Note as mentioned during the meeting yesterday I believe you actually >> need to pin virtualenv during the tox installation. If you want to >> pin setuptools it needs to happen when tox creates its virtualenvs. > > good point > >> >> One major con to this approach is now you've stopped testing that >> your software can deploy with newer versions of setuptools. It >> doesn't work right this moment, but as circumstances change you'll be >> without up to date information. > > I agree not to pin this on master. We need to support the latest and > greatest on master. But on stable I don't see the problem saying we > only support the tooling version that was the latest when the major > release was created. > >> >>> >>> >>> ?Option 3: turn off lower-constraints testing >>> ?Pros: >>> ?* easy to implement >>> ?* some of us want to get rid of lower constraints anyhow >>> ?Cons: >>> ?* needs per project changes >>> ?* not future proof against similar break affecting our upper >>> ?constraints on old stable branches. >>> >>> >>> ?Option 4: utilize pyproject.toml[6] to specify build-time requirements >>> ?Unfortunately, I'm not sure if it is possible to restrict the maximum >>> ?setuptools version with it >>> >>> >>> ?As a side note I tried applying [tox]requires in tox.ini to restrict >>> ?setuptools version[7] but It does not seem to take effect in our >>> ?case[8]. >> >> This didn't work because the requires specification seems to only >> affect the meta venv that tox uses to bootstrap itself if necessary >> [9]. Basically this pinned setuptools in the wrong virtualenv. This >> might work if you pinned virtualenv here as suggested above. > > Thanks for the pointers. I think I managed to make this work locally. > We will see if it works in CI too when [1] goes through the check > pipeline. > > Cheers, > gibi > > [1] https://review.opendev.org/c/openstack/nova/+/810461 > > > From gmann at ghanshyammann.com Fri Sep 24 19:55:29 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 24 Sep 2021 14:55:29 -0500 Subject: [all][tc] What's happening in Technical Committee: summary 24th Sept, 21: Reading: 5 min Message-ID: <17c19608a3f.fe9ca8ee27235.5538124727480865722@ghanshyammann.com> Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC this week IRC meeting held on Sept 23rd Thursday. * Most of the meeting discussions are summarized in the below sections (Completed or in-progress activities section). To know more details, you can check the complete logs @ https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-23-15.00.log.html * We will have next week's meeting on Sept 30th, Thursday 15:00 UTC on IRC, feel free the topic in agenda [1] by Sept 29th. 2. What we completed this week: ========================= * Moved Cloud Research SIG to active state[2] 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ * TC is using the etherpad[3] for Xena cycle working item. We will be checking and updating the status biweekly on the same etherpad. * Current status is: 7 completed, 6 in-progress Open Reviews ----------------- * Five open reviews for ongoing activities[4]. Add project health check tool ----------------------------------- * We are automating the project health checks. Currently, every TC member is assigned a number of projects to keep checking the project health/ listening to their issues but this is not so successful so far. * Rico has proposed the code[5] to collect the contribution stats and we are in the process of documenting the usage and interpretation of those stats. Stable Core team process change --------------------------------------- * As discussed in Shanghai and Xena PTG, there are some concerns over the current stable core team process (especially now when there are very few global stable team members) and to improve the process, mnaser has proposed the draft resolution[6] . Feel free to provide early feedback if you have any. Removing stable release 'Unmaintained' phase ------------------------------------------------------- * 'Unmaintained' phase in the stable release process has been confusing than adding any value. * Elod has proposed the removal of the 'unmaintained' phase[7], feel free to reply if any objection otherwise we will merge the patch on Monday(27th Sept). Places for projects to spreading the word ------------------------------------------------- * Amy has started to list down all the places that projects can use to spread the word out about your project which will help to get additional contributors and users[8]. She also sent emails to PTLs. * Please make note of it and use such places/blogs to highlights/educate operators/users/community etc. TC tags analysis ------------------- * As discussed in the last PTG, TC is working on an analysis of the usefulness of the Tags framework[9] or what all tags can be cleaned up. * We are still waiting for the operator's response to the email on openstack-disscuss ML[10]. If you are an operator, please respond to the email and based on the feedback we will continue the discussion in PTG. Project updates ------------------- * Add the cinder-netapp charm to Openstack charms[11] * Retiring js-openstack-lib [12] * Retire puppet-freezer[13] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [14]. * Current status: "Secure RBAC" is selected for Yoga cycle[15]. PTG planning ---------------- * We are collecting the PTG topics in etherpad[16], feel free to add any topic you would like to discuss. * We discussed the live stream of one of the TC PTG sessions like we did last time. Once we will have more topics in etherpad then we can select the appropriate one. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[17], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[18]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [19] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [20] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance-sigs/+/809887 [3] https://etherpad.opendev.org/p/tc-xena-tracker [4] https://review.opendev.org/q/projects:openstack/governance+status:open [5] https://review.opendev.org/c/openstack/governance/+/810037 [6] https://review.opendev.org/c/openstack/governance/+/810721 [7] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025016.html [8] https://review.opendev.org/c/openstack/project-team-guide/+/810766 [9] https://governance.openstack.org/tc/reference/tags/index.html [10] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024804.html [11] https://review.opendev.org/c/openstack/governance/+/809011 [12] https://review.opendev.org/c/openstack/governance/+/798540 [13] https://review.opendev.org/c/openstack/governance/+/807163 [14] https://etherpad.opendev.org/p/y-series-goals [15] https://review.opendev.org/c/openstack/governance/+/803783 [16] https://etherpad.opendev.org/p/tc-yoga-ptg [17] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [18] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [19] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [20] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann From zigo at debian.org Fri Sep 24 20:21:33 2021 From: zigo at debian.org (Thomas Goirand) Date: Fri, 24 Sep 2021 22:21:33 +0200 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: <6J4UZQ.VOBD0LVDTPUX1@est.tech> References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: Hi Gibi! Thanks for bringing this up. As a distro package maintainer, here's my view. On 9/22/21 2:11 PM, Balazs Gibizer wrote: > Option 1: Bump the major version of the decorator dependency on stable. Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable), for which I don't even maintain OpenStack anymore (that's OpenStack Newton...). So I don't see how switching to decorator 4.0.0 is a problem, and I don't understand how OpenStack could be using 3.4.0 which is in Jessie (ie: 6 years old Debian release). PyPi says Decorator 3.4.0 is from 2012: https://pypi.org/project/decorator/#history Do you have your release numbers correct? If so, then switching to Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 feels a bit crazy (and I wasn't aware of it). > Option 2: Pin the setuptools version during tox installation Please don't do this for the master branch, we need OpenStack to stay current with setuptools (yeah, even if this means breaking changes...). For already released OpenStack: I don't mind much if this is done (I could backport fixes if something breaks). > Option 3: turn off lower-constraints testing I already expressed myself about this: this is dangerous as distros rely on it for setting lower bounds as low as possible (which is always preferred from a distro point of view). > Option 4: utilize pyproject.toml[6] to specify build-time requirements I don't know about pyproject.toml. Just my 2 cents, hoping it's useful, Cheers, Thomas Goirand (zigo) From tonykarera at gmail.com Fri Sep 24 21:10:16 2021 From: tonykarera at gmail.com (Karera Tony) Date: Fri, 24 Sep 2021 23:10:16 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> Message-ID: I would really appreciate any support on this On Fri, 24 Sep 2021, 11:13 Karera Tony, wrote: > Hello Team, > > I don't know if there has been any change in the packages but the way I am > deploying is the same way I have been deploying. > > I don't understand why there is a certain issue now. > Regards > > Tony Karera > > > > > On Fri, Sep 24, 2021 at 7:30 AM Karera Tony wrote: > >> Hello Laurent, >> >> It turns out I only have one keyring in the container. >> >> root at compute1:/home/stack# docker exec -it nova_compute bash >> (nova-compute)[nova at compute1 /]$ cd /etc/ceph/ >> (nova-compute)[nova at compute1 ceph]$ ls >> ceph.client.nova.keyring ceph.conf rbdmap >> >> Regards >> >> Tony Karera >> >> >> >> >> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont >> wrote: >> >>> I do believe Kolla runs a container version of each service on computes. >>> Are you looking inside the nova-compute container ( >>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >>> >>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony >>> wrote: >>> >>>> Hello Sean, >>>> >>>> Below are the output on the compute node and deployment >>>> >>>> root at compute1:/etc/kolla/nova-compute# ls >>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>> config.json nova.conf >>>> >>>> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ >>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>> >>>> And I can confirm that the content is the same. >>>> >>>> >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney wrote: >>>> >>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>>> > I would investigate that compute error first. Creating volumes means >>>>> the >>>>> > controllers are doing the action. Starting a VM on a compute means >>>>> you also >>>>> > need Ceph to works on the compute to mount the rdb target. >>>>> >>>>> nova as part of its startup process in aintiallying the resouce >>>>> tracker will >>>>> try to connect to ceph if you are using the rbd image backend to >>>>> report how much stroage >>>>> is avaiable. if the keyring does not work on the vms pool as the user >>>>> nova is connecting as >>>>> then that will block the agent from starting up fully and will cause >>>>> it to be missing the hypervior list. >>>>> >>>>> the error seams to indicate that the cinder keyring is not in the nova >>>>> container >>>>> that likely means you have not put it in /etc/kolla/config/nova >>>>> i woudl check /etc/kolla/config/nova on the deployment host and sudo >>>>> ls /etc/kolla/nova-compute/ >>>>> on the compute node to ensure the cinder keyring is actully copied and >>>>> has the correct content >>>>> >>>>> i have >>>>> stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>> config.json nova.conf >>>>> >>>>> >>>>> [client.cinder] >>>>> key = ********************************* >>>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>>> caps mon = "profile rbd" >>>>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>>>> profile rbd pool=images" >>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>>> [client.nova] >>>>> key = ********************************* >>>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>>> caps mon = "profile rbd" >>>>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>>>> profile rbd pool=images" >>>>> >>>>> blanked out the key wiht *************** after the fact but you should >>>>> have something similar >>>>> >>>>> >>>>> in my case i decied to use a seperate key for nova rbd backend because >>>>> i was also using EC poosl with a seperate data and metadata pool >>>>> so i neede to modify my ceph.conf to make that work with kolla >>>>> >>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>> /etc/kolla/nova-compute/ceph.conf >>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>>> [global] >>>>> fsid = ********************* >>>>> mon_host = [*********************] >>>>> >>>>> [client.glance] >>>>> rbd default data pool = images-data >>>>> >>>>> [client.cinder] >>>>> rbd default data pool = volumes-data >>>>> >>>>> [client.nova] >>>>> rbd default data pool = vms-data >>>>> >>>>> using 2 keyrings/user allows me to set different default data pools >>>>> for cinder and nova. >>>>> >>>>> > >>>>> > Working in Wallaby with the error doesn't mean it would 100% work in >>>>> > Victoria. >>>>> > >>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony >>>>> wrote: >>>>> > >>>>> > > Hey Guys, Any other idea ? >>>>> > > >>>>> > > Regards >>>>> > > >>>>> > > Tony Karera >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony >>>>> wrote: >>>>> > > >>>>> > > > Just to add on that, >>>>> > > > >>>>> > > > compute service is listed, I can create Volumes, I have the same >>>>> cinder >>>>> > > > keyring in the /etc/kolla/config/nova directory as I have in the >>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along with the >>>>> nova keyring >>>>> > > > Regards >>>>> > > > >>>>> > > > Tony Karera >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>>>> tonykarera at gmail.com> wrote: >>>>> > > > >>>>> > > > > Hello Guys, >>>>> > > > > >>>>> > > > > Thanks a lot. >>>>> > > > > >>>>> > > > > I had actually checked the nova -compute.log on the compute >>>>> node and >>>>> > > > > they were showing the error I will post at the end about the >>>>> cinder keyring >>>>> > > > > but I know its correct because its the same I was using on >>>>> wallaby, I even >>>>> > > > > tried to use another ceph cluster with ofcouse different >>>>> keyrings but its >>>>> > > > > the same issue. >>>>> > > > > >>>>> > > > > Below is the error >>>>> > > > > >>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: >>>>> unable to >>>>> > > > > find a keyring on >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 >>>>> 7fbce2f4f700 -1 >>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>> auth: unable >>>>> > > > > to find a keyring on >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>>>> 7fbce2f4f700 -1 >>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>> auth: unable >>>>> > > > > to find a keyring on >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>>>> 7fbce2f4f700 -1 >>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>>> connecting to the >>>>> > > > > cluster)\n' >>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During >>>>> handling of >>>>> > > > > the above exception, another exception occurred: >>>>> > > > > Regards >>>>> > > > > >>>>> > > > > Tony Karera >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>>>> smooney at redhat.com> wrote: >>>>> > > > > >>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>>>> > > > > > > It could also be a compute cell discovery issue maybe? >>>>> > > > > > no they shoudl still show up in the hypervior list api >>>>> > > > > > > >>>>> > > > > > > Do you see anything under "openstack compute service list"? >>>>> > > > > > if they show up in the service list but not they hyperiors >>>>> api it >>>>> > > > > > means that the comptue service started and registered its >>>>> service entry >>>>> > > > > > but >>>>> > > > > > something broke it before it could create a compute node >>>>> recored in the >>>>> > > > > > db. >>>>> > > > > > >>>>> > > > > > with ceph the case i have hit this most often is when the >>>>> keyright used >>>>> > > > > > by nova to >>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong whihc >>>>> prevent >>>>> > > > > > the resoucetack and compute manager >>>>> > > > > > form actully creating the compute node record. >>>>> > > > > > >>>>> > > > > > >>>>> > > > > > it can happen for other reason too but best place to start >>>>> is check if >>>>> > > > > > there is an error in the nova compute agent log and go from >>>>> there. >>>>> > > > > > > >>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>>> smooney at redhat.com> >>>>> > > > > > wrote: >>>>> > > > > > > >>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >>>>> > > > > > > > > Hello Team, >>>>> > > > > > > > > >>>>> > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible >>>>> on Ubuntu >>>>> > > > > > 20.04 >>>>> > > > > > > > and >>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>>>> Glance. >>>>> > > > > > > > > >>>>> > > > > > > > > It finished with no error but it has failed to >>>>> register any on the >>>>> > > > > > > > Compute >>>>> > > > > > > > > Nodes under Hypervisors. >>>>> > > > > > > > > >>>>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack >>>>> hypervisor list >>>>> > > > > > > > > >>>>> > > > > > > > > (kolla-openstack) stack at deployment:~$ >>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > Any idea on how to resolve this ? >>>>> > > > > > > > that usually means that somehthing prevented the comptue >>>>> agent form >>>>> > > > > > > > strating properly >>>>> > > > > > > > >>>>> > > > > > > > for example incorrect ceph keyrings there are several >>>>> other case >>>>> > > > > > but you >>>>> > > > > > > > mentioned you are >>>>> > > > > > > > using ceph. >>>>> > > > > > > > >>>>> > > > > > > > if this is hte case you should see error in the compute >>>>> agent log. >>>>> > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > Regards >>>>> > > > > > > > > >>>>> > > > > > > > > Tony Karera >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> >>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ihrachys at redhat.com Fri Sep 24 21:26:09 2021 From: ihrachys at redhat.com (Ihar Hrachyshka) Date: Fri, 24 Sep 2021 17:26:09 -0400 Subject: [neutron][ovn] support for stateless NAT for floating ip in ml2 ovn In-Reply-To: References: <20210817002002.252368-1-ihrachys@redhat.com> <9096201007244d0732f019d1452575e61080e09c.camel@redhat.com> <3b84603ca14830e712154091692f05538fdcacef.camel@redhat.com> Message-ID: On 8/18/21 11:35 AM, Ihar Hrachyshka wrote: > Not too hard to fallback; but on this note, do we maintain minimal OVN > version anywhere in neutron? I remember when I was adding support for > allow-stateless ACLs, I was told we don't track it (hence runtime > schema inspection in > https://review.opendev.org/c/openstack/neutron/+/789974) Considering > potential backports in downstream products, perhaps a runtime schema > check is a better approach anyway. To close the loop, the patch that uses stateless dnat_and_snat has merged in master. Including migration path for existing NAT objects in nbdb and runtime check for core OVN support. Probably not a backport material for upstream. Ihar From jimmy at openstack.org Fri Sep 24 21:27:05 2021 From: jimmy at openstack.org (Jimmy McArthur) Date: Fri, 24 Sep 2021 16:27:05 -0500 Subject: Openstack hypervisor list is empty In-Reply-To: References: Message-ID: <5BD1D336-6C7B-4CD3-80F6-8FF60A7B29CC@getmailspring.com> Keep in mind that this is asynchronous communication with a community of volunteers. Lots of folks are already off today or will be soon. Sometimes it could be a couple of days before you get a response. I hope you're able to solve your issue. Jimmy On Sep 24 2021, at 4:10 pm, Karera Tony wrote: > I would really appreciate any support on this > > On Fri, 24 Sep 2021, 11:13 Karera Tony, wrote: > > Hello Team, > > > > I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying. > > > > I don't understand why there is a certain issue now. > > Regards > > > > Tony Karera > > > > > > > > > > On Fri, Sep 24, 2021 at 7:30 AM Karera Tony wrote: > > > Hello Laurent, > > > > > > It turns out I only have one keyring in the container. > > > > > > root at compute1:/home/stack# docker exec -it nova_compute bash > > > (nova-compute)[nova at compute1 /]$ cd /etc/ceph/ > > > (nova-compute)[nova at compute1 ceph]$ ls > > > ceph.client.nova.keyring ceph.conf rbdmap > > > > > > > > > Regards > > > > > > Tony Karera > > > > > > > > > > > > > > > On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont wrote: > > > > I do believe Kolla runs a container version of each service on computes. Are you looking inside the nova-compute container (etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) > > > > > > > > On Thu, Sep 23, 2021 at 11:24 AM Karera Tony wrote: > > > > > Hello Sean, > > > > > > > > > > Below are the output on the compute node and deployment > > > > > > > > > > root at compute1:/etc/kolla/nova-compute# ls > > > > > ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf > > > > > > > > > > > > > > > (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ > > > > > ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf > > > > > > > > > > > > > > > And I can confirm that the content is the same. > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney wrote: > > > > > > On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: > > > > > > > I would investigate that compute error first. Creating volumes means the > > > > > > > controllers are doing the action. Starting a VM on a compute means you also > > > > > > > need Ceph to works on the compute to mount the rdb target. > > > > > > > > > > > > nova as part of its startup process in aintiallying the resouce tracker will > > > > > > try to connect to ceph if you are using the rbd image backend to report how much stroage > > > > > > is avaiable. if the keyring does not work on the vms pool as the user nova is connecting as > > > > > > then that will block the agent from starting up fully and will cause it to be missing the hypervior list. > > > > > > > > > > > > the error seams to indicate that the cinder keyring is not in the nova container > > > > > > that likely means you have not put it in /etc/kolla/config/nova > > > > > > i woudl check /etc/kolla/config/nova on the deployment host and sudo ls /etc/kolla/nova-compute/ > > > > > > on the compute node to ensure the cinder keyring is actully copied and has the correct content > > > > > > > > > > > > i have > > > > > > stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ > > > > > > ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf > > > > > > > > > > > > > > > > > > [client.cinder] > > > > > > key = ********************************* > > > > > > caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" > > > > > > caps mon = "profile rbd" > > > > > > caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" > > > > > > stack at cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.client.nova.keyring > > > > > > [client.nova] > > > > > > key = ********************************* > > > > > > caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" > > > > > > caps mon = "profile rbd" > > > > > > caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" > > > > > > > > > > > > blanked out the key wiht *************** after the fact but you should have something similar > > > > > > > > > > > > in my case i decied to use a seperate key for nova rbd backend because i was also using EC poosl with a seperate data and metadata pool > > > > > > so i neede to modify my ceph.conf to make that work with kolla > > > > > > > > > > > > stack at cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.conf > > > > > > # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f > > > > > > [global] > > > > > > fsid = ********************* > > > > > > mon_host = [*********************] > > > > > > > > > > > > [client.glance] > > > > > > rbd default data pool = images-data > > > > > > > > > > > > [client.cinder] > > > > > > rbd default data pool = volumes-data > > > > > > > > > > > > [client.nova] > > > > > > rbd default data pool = vms-data > > > > > > > > > > > > using 2 keyrings/user allows me to set different default data pools for cinder and nova. > > > > > > > > > > > > > > Working in Wallaby with the error doesn't mean it would 100% work in > > > > > > > Victoria. > > > > > > > > > > > > > > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony wrote: > > > > > > > > > > > > > > > Hey Guys, Any other idea ? > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony wrote: > > > > > > > > > > > > > > > > > Just to add on that, > > > > > > > > > > > > > > > > > > compute service is listed, I can create Volumes, I have the same cinder > > > > > > > > > keyring in the /etc/kolla/config/nova directory as I have in the > > > > > > > > > /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony wrote: > > > > > > > > > > > > > > > > > > > Hello Guys, > > > > > > > > > > > > > > > > > > > > Thanks a lot. > > > > > > > > > > > > > > > > > > > > I had actually checked the nova -compute.log on the compute node and > > > > > > > > > > they were showing the error I will post at the end about the cinder keyring > > > > > > > > > > but I know its correct because its the same I was using on wallaby, I even > > > > > > > > > > tried to use another ceph cluster with ofcouse different keyrings but its > > > > > > > > > > the same issue. > > > > > > > > > > > > > > > > > > > > Below is the error > > > > > > > > > > > > > > > > > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to > > > > > > > > > > find a keyring on > > > > > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > > > > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 > > > > > > > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at > > > > > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > > > > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > > > > > > > > > to find a keyring on > > > > > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > > > > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > > > > > > > > > AuthRegistry(0x7fbcdc060698) no keyring found at > > > > > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > > > > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > > > > > > > > > to find a keyring on > > > > > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > > > > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > > > > > > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at > > > > > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > > > > > > > disabling cephx\n[errno 2] RADOS object not found (error connecting to the > > > > > > > > > > cluster)\n' > > > > > > > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager > > > > > > > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of > > > > > > > > > > the above exception, another exception occurred: > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney wrote: > > > > > > > > > > > > > > > > > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > > > > > > > > > > > > It could also be a compute cell discovery issue maybe? > > > > > > > > > > > no they shoudl still show up in the hypervior list api > > > > > > > > > > > > > > > > > > > > > > > > Do you see anything under "openstack compute service list"? > > > > > > > > > > > if they show up in the service list but not they hyperiors api it > > > > > > > > > > > means that the comptue service started and registered its service entry > > > > > > > > > > > but > > > > > > > > > > > something broke it before it could create a compute node recored in the > > > > > > > > > > > db. > > > > > > > > > > > > > > > > > > > > > > with ceph the case i have hit this most often is when the keyright used > > > > > > > > > > > by nova to > > > > > > > > > > > get the avaiable capastiy of the ceph cluster is wrong whihc prevent > > > > > > > > > > > the resoucetack and compute manager > > > > > > > > > > > form actully creating the compute node record. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > it can happen for other reason too but best place to start is check if > > > > > > > > > > > there is an error in the nova compute agent log and go from there. > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > > > > > > > > > > > > Hello Team, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu > > > > > > > > > > > 20.04 > > > > > > > > > > > > > and > > > > > > > > > > > > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > > > > > > > > > > > > > > > > > > > > > > > It finished with no error but it has failed to register any on the > > > > > > > > > > > > > Compute > > > > > > > > > > > > > > Nodes under Hypervisors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > kolla-openstack) stack at deployment:~$ openstack hypervisor list > > > > > > > > > > > > > > > > > > > > > > > > > > > > (kolla-openstack) stack at deployment:~$ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea on how to resolve this ? > > > > > > > > > > > > > that usually means that somehthing prevented the comptue agent form > > > > > > > > > > > > > strating properly > > > > > > > > > > > > > > > > > > > > > > > > > > for example incorrect ceph keyrings there are several other case > > > > > > > > > > > but you > > > > > > > > > > > > > mentioned you are > > > > > > > > > > > > > using ceph. > > > > > > > > > > > > > > > > > > > > > > > > > > if this is hte case you should see error in the compute agent log. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Fri Sep 24 23:37:00 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Fri, 24 Sep 2021 16:37:00 -0700 Subject: All Devstack Based Jobs Failing to Create Swap on master In-Reply-To: References: Message-ID: On Fri, Sep 24, 2021, at 10:05 AM, Clark Boylan wrote: > snip > > I've pushed up a change [4] to remove this Zuul configuration from the > Xena branch which I expect will fix things for us. > snip > [4] https://review.opendev.org/c/openstack/devstack/+/810947 This change has merged and things appear happier. You should be clear to recheck at this point. From laurentfdumont at gmail.com Sat Sep 25 02:07:08 2021 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Fri, 24 Sep 2021 22:07:08 -0400 Subject: Creating multiple servers with a shared volume In-Reply-To: References: Message-ID: Just to be sure, the flow is. - Create VM with root disk on a volume/image. - Create another 1TB volume. - Attach the volume to the VM as a second drive using a multi attach approach ( https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiattach.html )? - Create snapshot of the original VM (with it's root drive + 1 TB volume)? - Create new VM from snapshot. - Instead of a new VM + 1 new root volume + the same 1TB multiattach volume attached, you get a new multiattach volume (new volume ID and everything)? I've never had the pleasure to play around with this, but creating a new VM from a snapshot could mean a new volume too. I'm not sure how snapshot will work with multi volume VMs. On Fri, Sep 24, 2021 at 11:20 AM open infra wrote: > Hi, > > I am planning to spin hundreds of servers with a shared storage. > I have created one instance and attached a 1 TB volume (multiattach) and > created a snapshot of the instance. > But when I spin another server using the snapshot, I noticed that instead > of expected shared volume, individual 1 TB size additional (dedicated) > volume has been attached to each VM. > > Is there a mechanism where I can create multiple instances along with a > shared volume attached to the instance instead of attaching later? > > Regards, > Danishka > -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurentfdumont at gmail.com Sat Sep 25 02:07:57 2021 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Fri, 24 Sep 2021 22:07:57 -0400 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> Message-ID: I know that there is some Kolla folks around but keep in mind that this is a volunteer based list :) I think you might get a bit more one to one help on IRC in their kolla channel. On Fri, Sep 24, 2021 at 5:10 PM Karera Tony wrote: > I would really appreciate any support on this > > On Fri, 24 Sep 2021, 11:13 Karera Tony, wrote: > >> Hello Team, >> >> I don't know if there has been any change in the packages but the way I >> am deploying is the same way I have been deploying. >> >> I don't understand why there is a certain issue now. >> Regards >> >> Tony Karera >> >> >> >> >> On Fri, Sep 24, 2021 at 7:30 AM Karera Tony wrote: >> >>> Hello Laurent, >>> >>> It turns out I only have one keyring in the container. >>> >>> root at compute1:/home/stack# docker exec -it nova_compute bash >>> (nova-compute)[nova at compute1 /]$ cd /etc/ceph/ >>> (nova-compute)[nova at compute1 ceph]$ ls >>> ceph.client.nova.keyring ceph.conf rbdmap >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont >>> wrote: >>> >>>> I do believe Kolla runs a container version of each service on >>>> computes. Are you looking inside the nova-compute container ( >>>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >>>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >>>> >>>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony >>>> wrote: >>>> >>>>> Hello Sean, >>>>> >>>>> Below are the output on the compute node and deployment >>>>> >>>>> root at compute1:/etc/kolla/nova-compute# ls >>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>> config.json nova.conf >>>>> >>>>> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ >>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>> >>>>> And I can confirm that the content is the same. >>>>> >>>>> >>>>> >>>>> Regards >>>>> >>>>> Tony Karera >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney >>>>> wrote: >>>>> >>>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>>>> > I would investigate that compute error first. Creating volumes >>>>>> means the >>>>>> > controllers are doing the action. Starting a VM on a compute means >>>>>> you also >>>>>> > need Ceph to works on the compute to mount the rdb target. >>>>>> >>>>>> nova as part of its startup process in aintiallying the resouce >>>>>> tracker will >>>>>> try to connect to ceph if you are using the rbd image backend to >>>>>> report how much stroage >>>>>> is avaiable. if the keyring does not work on the vms pool as the >>>>>> user nova is connecting as >>>>>> then that will block the agent from starting up fully and will cause >>>>>> it to be missing the hypervior list. >>>>>> >>>>>> the error seams to indicate that the cinder keyring is not in the >>>>>> nova container >>>>>> that likely means you have not put it in /etc/kolla/config/nova >>>>>> i woudl check /etc/kolla/config/nova on the deployment host and sudo >>>>>> ls /etc/kolla/nova-compute/ >>>>>> on the compute node to ensure the cinder keyring is actully copied >>>>>> and has the correct content >>>>>> >>>>>> i have >>>>>> stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>> config.json nova.conf >>>>>> >>>>>> >>>>>> [client.cinder] >>>>>> key = ********************************* >>>>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>>>> caps mon = "profile rbd" >>>>>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>>>>> profile rbd pool=images" >>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>>>> [client.nova] >>>>>> key = ********************************* >>>>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>>>> caps mon = "profile rbd" >>>>>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>>>>> profile rbd pool=images" >>>>>> >>>>>> blanked out the key wiht *************** after the fact but you >>>>>> should have something similar >>>>>> >>>>>> >>>>>> in my case i decied to use a seperate key for nova rbd backend >>>>>> because i was also using EC poosl with a seperate data and metadata pool >>>>>> so i neede to modify my ceph.conf to make that work with kolla >>>>>> >>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>> /etc/kolla/nova-compute/ceph.conf >>>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>>>> [global] >>>>>> fsid = ********************* >>>>>> mon_host = [*********************] >>>>>> >>>>>> [client.glance] >>>>>> rbd default data pool = images-data >>>>>> >>>>>> [client.cinder] >>>>>> rbd default data pool = volumes-data >>>>>> >>>>>> [client.nova] >>>>>> rbd default data pool = vms-data >>>>>> >>>>>> using 2 keyrings/user allows me to set different default data pools >>>>>> for cinder and nova. >>>>>> >>>>>> > >>>>>> > Working in Wallaby with the error doesn't mean it would 100% work in >>>>>> > Victoria. >>>>>> > >>>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony >>>>>> wrote: >>>>>> > >>>>>> > > Hey Guys, Any other idea ? >>>>>> > > >>>>>> > > Regards >>>>>> > > >>>>>> > > Tony Karera >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony >>>>>> wrote: >>>>>> > > >>>>>> > > > Just to add on that, >>>>>> > > > >>>>>> > > > compute service is listed, I can create Volumes, I have the >>>>>> same cinder >>>>>> > > > keyring in the /etc/kolla/config/nova directory as I have in >>>>>> the >>>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along with the >>>>>> nova keyring >>>>>> > > > Regards >>>>>> > > > >>>>>> > > > Tony Karera >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>>>>> tonykarera at gmail.com> wrote: >>>>>> > > > >>>>>> > > > > Hello Guys, >>>>>> > > > > >>>>>> > > > > Thanks a lot. >>>>>> > > > > >>>>>> > > > > I had actually checked the nova -compute.log on the compute >>>>>> node and >>>>>> > > > > they were showing the error I will post at the end about the >>>>>> cinder keyring >>>>>> > > > > but I know its correct because its the same I was using on >>>>>> wallaby, I even >>>>>> > > > > tried to use another ceph cluster with ofcouse different >>>>>> keyrings but its >>>>>> > > > > the same issue. >>>>>> > > > > >>>>>> > > > > Below is the error >>>>>> > > > > >>>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: >>>>>> unable to >>>>>> > > > > find a keyring on >>>>>> > > > > >>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 >>>>>> 7fbce2f4f700 -1 >>>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>>>> > > > > >>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>>> auth: unable >>>>>> > > > > to find a keyring on >>>>>> > > > > >>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>>>>> 7fbce2f4f700 -1 >>>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>>>> > > > > >>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>>> auth: unable >>>>>> > > > > to find a keyring on >>>>>> > > > > >>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>>>>> 7fbce2f4f700 -1 >>>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>>>> > > > > >>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>>>> connecting to the >>>>>> > > > > cluster)\n' >>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During >>>>>> handling of >>>>>> > > > > the above exception, another exception occurred: >>>>>> > > > > Regards >>>>>> > > > > >>>>>> > > > > Tony Karera >>>>>> > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>>>>> smooney at redhat.com> wrote: >>>>>> > > > > >>>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>>>>> > > > > > > It could also be a compute cell discovery issue maybe? >>>>>> > > > > > no they shoudl still show up in the hypervior list api >>>>>> > > > > > > >>>>>> > > > > > > Do you see anything under "openstack compute service >>>>>> list"? >>>>>> > > > > > if they show up in the service list but not they hyperiors >>>>>> api it >>>>>> > > > > > means that the comptue service started and registered its >>>>>> service entry >>>>>> > > > > > but >>>>>> > > > > > something broke it before it could create a compute node >>>>>> recored in the >>>>>> > > > > > db. >>>>>> > > > > > >>>>>> > > > > > with ceph the case i have hit this most often is when the >>>>>> keyright used >>>>>> > > > > > by nova to >>>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>>>>> whihc prevent >>>>>> > > > > > the resoucetack and compute manager >>>>>> > > > > > form actully creating the compute node record. >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > > it can happen for other reason too but best place to start >>>>>> is check if >>>>>> > > > > > there is an error in the nova compute agent log and go from >>>>>> there. >>>>>> > > > > > > >>>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>>>> smooney at redhat.com> >>>>>> > > > > > wrote: >>>>>> > > > > > > >>>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >>>>>> > > > > > > > > Hello Team, >>>>>> > > > > > > > > >>>>>> > > > > > > > > I have deployed Openstack Victoria using >>>>>> Kolla-ansible on Ubuntu >>>>>> > > > > > 20.04 >>>>>> > > > > > > > and >>>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>>>>> Glance. >>>>>> > > > > > > > > >>>>>> > > > > > > > > It finished with no error but it has failed to >>>>>> register any on the >>>>>> > > > > > > > Compute >>>>>> > > > > > > > > Nodes under Hypervisors. >>>>>> > > > > > > > > >>>>>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack >>>>>> hypervisor list >>>>>> > > > > > > > > >>>>>> > > > > > > > > (kolla-openstack) stack at deployment:~$ >>>>>> > > > > > > > > >>>>>> > > > > > > > > >>>>>> > > > > > > > > Any idea on how to resolve this ? >>>>>> > > > > > > > that usually means that somehthing prevented the >>>>>> comptue agent form >>>>>> > > > > > > > strating properly >>>>>> > > > > > > > >>>>>> > > > > > > > for example incorrect ceph keyrings there are several >>>>>> other case >>>>>> > > > > > but you >>>>>> > > > > > > > mentioned you are >>>>>> > > > > > > > using ceph. >>>>>> > > > > > > > >>>>>> > > > > > > > if this is hte case you should see error in the compute >>>>>> agent log. >>>>>> > > > > > > > >>>>>> > > > > > > > > >>>>>> > > > > > > > > Regards >>>>>> > > > > > > > > >>>>>> > > > > > > > > Tony Karera >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> >>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Sat Sep 25 07:02:44 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Sat, 25 Sep 2021 09:02:44 +0200 Subject: [neutron] CI update In-Reply-To: References: <2741059.qUHmEk80i5@p1> <20210922142806.4ovhy3k5zwahkarn@p1.localdomain> Message-ID: <20210925070244.wz3cj4kpg2b6rgui@p1.localdomain> Hi, On Wed, Sep 22, 2021 at 04:55:34PM +0200, Rodolfo Alonso Hernandez wrote: > Hello folks: > > I replied in [1]. I think we could have a potential problem when executing > a DB change the impies a OF controller re-initialization. Most of the time > we don't have problems but as we see in the CI, we could sometimes. > > I'll push a patch to add a retry decorator on the methods that trigger this > OF controller restart. I see that patch https://review.opendev.org/c/openstack/neutron/+/810592 is already merged. Thx Rodolfo for quick fix. @Julia: can You check if Ironic jobs are ok now? > > Regards. > > [1]https://bugs.launchpad.net/neutron/+bug/1944201/comments/4 > > > On Wed, Sep 22, 2021 at 4:36 PM Slawek Kaplonski > wrote: > > > Hi > > > > On Wed, Sep 22, 2021 at 03:30:08PM +0200, Lajos Katona wrote: > > > Hi Slawek, > > > Thanks for the summary. > > > Regarding https://bugs.launchpad.net/neutron/+bug/1944201 not sure > > about if > > > it is related to the number of hosts, there's some failure in > > > singlenode jobs as well: > > > > > http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20Datapath%20Invalid%5C%22 > > > > > > example: > > > > > https://9f672e0630f459ee81cb-e4093b1756a9a5a7c7d28e6575b4af7f.ssl.cf5.rackcdn.com/805849/10/check/neutron-ovs-rally-task/5981df8/controller/logs/screen-q-agt.txt > > > > Ok. So it happens on all types of jobs where neutron-ovs-agent is used :/ > > > > > > > > Lajos (lajoskatona) > > > > > > Slawek Kaplonski ezt ?rta (id?pont: 2021. szept. > > 22., > > > Sze, 12:10): > > > > > > > Hi, > > > > > > > > Due to very urgent things which I had yesterday, I wasn't able to run > > > > Neutron > > > > CI meeting as usually. Fortunatelly we don't have many new issues in > > our > > > > CI. > > > > There is only one new issue in our scenario jobs which I wanted to > > discuss > > > > [1]. It's impacting Ironic gates but I noticed it also in the Neutron > > CI > > > > as > > > > well. See [2] or [3] for example. I'm not sure about Ironic jobs but in > > > > Neutron I saw it mostly (or only, I'm not sure) in the multinode jobs. > > > > > > > > [1] https://bugs.launchpad.net/neutron/+bug/1944201 > > > > [2] https:// > > > > > > e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ > > > > > > 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ > > > > logs/screen-q-agt.txt > > > > < > > http://e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/logs/screen-q-agt.txt > > > > > > > [3] https://storage.bhs.cloud.ovh.net/v1/ > > > > > > > > > > AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/check/ > > > > neutron-ovs-tempest-slow/88f8bb7/job-output.txt > > > > > > > > -- > > > > Slawek Kaplonski > > > > Principal Software Engineer > > > > Red Hat > > > > -- > > Slawek Kaplonski > > Principal Software Engineer > > Red Hat > > -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From iurygregory at gmail.com Sat Sep 25 07:10:04 2021 From: iurygregory at gmail.com (Iury Gregory) Date: Sat, 25 Sep 2021 09:10:04 +0200 Subject: [neutron] CI update In-Reply-To: <20210925070244.wz3cj4kpg2b6rgui@p1.localdomain> References: <2741059.qUHmEk80i5@p1> <20210922142806.4ovhy3k5zwahkarn@p1.localdomain> <20210925070244.wz3cj4kpg2b6rgui@p1.localdomain> Message-ID: Hi Slawek, I just pushed a patch to revert the change we did to workaround the issue in Ironic [1]. Thanks! [1] https://review.opendev.org/c/openstack/ironic/+/810973 Em s?b., 25 de set. de 2021 ?s 09:04, Slawek Kaplonski escreveu: > Hi, > > On Wed, Sep 22, 2021 at 04:55:34PM +0200, Rodolfo Alonso Hernandez wrote: > > Hello folks: > > > > I replied in [1]. I think we could have a potential problem when > executing > > a DB change the impies a OF controller re-initialization. Most of the > time > > we don't have problems but as we see in the CI, we could sometimes. > > > > I'll push a patch to add a retry decorator on the methods that trigger > this > > OF controller restart. > > I see that patch https://review.opendev.org/c/openstack/neutron/+/810592 > is > already merged. Thx Rodolfo for quick fix. > @Julia: can You check if Ironic jobs are ok now? > > > > > Regards. > > > > [1]https://bugs.launchpad.net/neutron/+bug/1944201/comments/4 > > > > > > On Wed, Sep 22, 2021 at 4:36 PM Slawek Kaplonski > > wrote: > > > > > Hi > > > > > > On Wed, Sep 22, 2021 at 03:30:08PM +0200, Lajos Katona wrote: > > > > Hi Slawek, > > > > Thanks for the summary. > > > > Regarding https://bugs.launchpad.net/neutron/+bug/1944201 not sure > > > about if > > > > it is related to the number of hosts, there's some failure in > > > > singlenode jobs as well: > > > > > > > > http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20Datapath%20Invalid%5C%22 > > > > > > > > example: > > > > > > > > https://9f672e0630f459ee81cb-e4093b1756a9a5a7c7d28e6575b4af7f.ssl.cf5.rackcdn.com/805849/10/check/neutron-ovs-rally-task/5981df8/controller/logs/screen-q-agt.txt > > > > > > Ok. So it happens on all types of jobs where neutron-ovs-agent is used > :/ > > > > > > > > > > > Lajos (lajoskatona) > > > > > > > > Slawek Kaplonski ezt ?rta (id?pont: 2021. > szept. > > > 22., > > > > Sze, 12:10): > > > > > > > > > Hi, > > > > > > > > > > Due to very urgent things which I had yesterday, I wasn't able to > run > > > > > Neutron > > > > > CI meeting as usually. Fortunatelly we don't have many new issues > in > > > our > > > > > CI. > > > > > There is only one new issue in our scenario jobs which I wanted to > > > discuss > > > > > [1]. It's impacting Ironic gates but I noticed it also in the > Neutron > > > CI > > > > > as > > > > > well. See [2] or [3] for example. I'm not sure about Ironic jobs > but in > > > > > Neutron I saw it mostly (or only, I'm not sure) in the multinode > jobs. > > > > > > > > > > [1] https://bugs.launchpad.net/neutron/+bug/1944201 > > > > > [2] https:// > > > > > > > > > e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ > > > > > > > > > 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ > > > > > logs/screen-q-agt.txt > > > > > < > > > > http://e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/logs/screen-q-agt.txt > > > > > > > > > [3] https://storage.bhs.cloud.ovh.net/v1/ > > > > > > > > > > > > > > AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/check/ > > > > > neutron-ovs-tempest-slow/88f8bb7/job-output.txt > > > > > > > > > > -- > > > > > Slawek Kaplonski > > > > > Principal Software Engineer > > > > > Red Hat > > > > > > -- > > > Slawek Kaplonski > > > Principal Software Engineer > > > Red Hat > > > > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat > -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory at gmail.com * -------------- next part -------------- An HTML attachment was scrubbed... URL: From eduardo.experimental at gmail.com Sat Sep 25 10:54:34 2021 From: eduardo.experimental at gmail.com (Eduardo Santos) Date: Sat, 25 Sep 2021 10:54:34 +0000 Subject: [openstack-qa] opendev.org mirrors Message-ID: Hi folks, On US-based servers, a DevStack deploy takes 15 minutes. When deploying it from Brazil (where I live) however, it takes more than an hour. Examining the stack.sh logs, git_timed appears to be the culprit. I'm using a 250 Mbps connection. Do opendev.org mirrors exist for other regions other than the US? If not, has this been discussed yet? I believe this is more of a financial discussion othet than technical, but I'm curious if other users have raised this question before. I'm not sure if openstack-qa is the most appropriate flag. If not, please feel free to correct. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Sat Sep 25 12:23:29 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Sat, 25 Sep 2021 12:23:29 +0000 Subject: [openstack-qa][infra] opendev.org mirrors In-Reply-To: References: Message-ID: <20210925122328.5ipl3q2ruux6ejha@yuggoth.org> On 2021-09-25 10:54:34 +0000 (+0000), Eduardo Santos wrote: > On US-based servers, a DevStack deploy takes 15 minutes. When > deploying it from Brazil (where I live) however, it takes more > than an hour. Examining the stack.sh logs, git_timed appears to be > the culprit. I'm using a 250 Mbps connection. Is this only the first time you install DevStack, or is that how long it takes to update an existing DevStack install? It shouldn't be cloning every time. > Do opendev.org mirrors exist for other regions other than the US? > If not, has this been discussed yet? [...] Technically yes, though not for everything hosted in the OpenDev collaboratory, but many projects (including OpenStack) do continuously replicate all their branches and tags to GitHub. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From openinfradn at gmail.com Sat Sep 25 07:41:19 2021 From: openinfradn at gmail.com (open infra) Date: Sat, 25 Sep 2021 13:11:19 +0530 Subject: Creating multiple servers with a shared volume In-Reply-To: References: Message-ID: On Sat, Sep 25, 2021 at 7:37 AM Laurent Dumont wrote: > Just to be sure, the flow is. > > - Create VM with root disk on a volume/image. > - Create another 1TB volume. > - Attach the volume to the VM as a second drive using a multi attach > approach ( > https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiattach.html > )? > - Create snapshot of the original VM (with it's root drive + 1 TB > volume)? > - Create new VM from snapshot. > - Instead of a new VM + 1 new root volume + the same 1TB multiattach > volume attached, you get a new multiattach volume (new volume ID and > everything)? > > Yes, here is the block device mapping of the snapshot (from a VM with 100GB of root disk and 1TB of multiattach volume) block_device_mapping[{"image_id": null, "delete_on_termination": true, "device_name": "/dev/vda", "disk_bus": "virtio", "volume_id": null, "volume_type": null, "destination_type": "volume", "source_type": "snapshot", "guest_format": null, "volume_size": 100, "device_type": "disk", "snapshot_id": "7b2c7f3a-8420-4a33-820e-2d2231d930c7", "boot_index": 0, "no_device": null, "tag": null}, {"image_id": null, "delete_on_termination": false, "device_name": "/dev/vdb", "disk_bus": null, "volume_id": null, "volume_type": null, "destination_type": "volume", "source_type": "snapshot", "guest_format": null, "volume_size": 1000, "device_type": null, "snapshot_id": "c2a0695f-8c9e-46f9-80a8-560c47521eeb", "boot_index": null, "no_device": null, "tag": null}] openstack image list +--------------------------------------+----------------+--------+ | ID | Name | Status | +--------------------------------------+----------------+--------+ | e9e0e6dc-6389-49a3-bd52-467520cd2c9e | vm-04-snap-new | active | +--------------------------------------+----------------+--------+ openstack volume list +--------------------------------------+---------------+--------+------+--------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+---------------+--------+------+--------------------------------------------------------------+ | b239c3da-3276-4497-ad4c-1e900e907426 | | in-use | 100 | Attached to VM-04 on /dev/vda | | 28088321-f87a-4392-bd6c-bcd0174184ea | | in-use | 100 | Attached to VM-03 on /dev/vda | | c693b9ad-6d47-4a2c-93e4-36baaf25f062 | shared-volume | in-use | 1000 | Attached to VM-03 on /dev/vdb Attached to VM-04 on /dev/vdb | Under volume >. snapshots I can see two new volumes created openstack volume snapshot list +--------------------------------------+-----------------------------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+-----------------------------+-------------+-----------+------+ | c2a0695f-8c9e-46f9-80a8-560c47521eeb | snapshot for vm-04-snap-new | None | available | 1000 | | 7b2c7f3a-8420-4a33-820e-2d2231d930c7 | snapshot for vm-04-snap-new | | available | 100 | +--------------------------------------+-----------------------------+-------------+-----------+------+ controller-1:~$ openstack server create --flavor WinGPU16 --image vm-04-snap-new --max 10 vm +-------------------------------------+-------------------------------------------------------+ | Field | Value | +-------------------------------------+-------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | adminPass | se8pWoZ8AQ5Y | | config_drive | | | created | 2021-09-25T07:25:51Z | | flavor | WinGPU16 (e3cd48c5-9675-42ad-a766-29b92c674224) | | hostId | | | id | facfc95d-bcb5-452c-a7fb-fcfa22f4257b | | image | vm-04-snap-new (e9e0e6dc-6389-49a3-bd52-467520cd2c9e) | | key_name | None | | name | vm-1 | | progress | 0 | | project_id | 21639a96c609474390c837accd2de337 | | properties | | | security_groups | name='default' | | status | BUILD | | updated | 2021-09-25T07:25:51Z | | user_id | 19faf07bb5da40b897e5dc73073cdec6 | | volumes_attached | | +-------------------------------------+-------------------------------------------------------+ controller-1:~$ controller-1:~$ openstack server list +--------------------------------------+----------+---------+----------------------+----------------+----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+----------+---------+----------------------+----------------+----------+ | 24352c37-0c2d-4760-bb6e-aeb3f0432833 | vm-2 | ACTIVE | g3-ext=192.168.6.168 | vm-04-snap-new | WinGPU16 | | 39f2a8d5-a895-47ef-8cdc-bd9f9fe6acd2 | vm-7 | ACTIVE | g3-ext=192.168.6.214 | vm-04-snap-new | WinGPU16 | | 59330c7c-4c57-4aaa-9c83-69d05d670b53 | vm-6 | ACTIVE | g3-ext=192.168.6.251 | vm-04-snap-new | WinGPU16 | | 60fc3dc6-c873-4faa-99e6-657a98fa2275 | vm-3 | ACTIVE | g3-ext=192.168.6.210 | vm-04-snap-new | WinGPU16 | | 88af8a71-93f9-438b-93f3-4169b35842ca | vm-4 | ACTIVE | g3-ext=192.168.6.241 | vm-04-snap-new | WinGPU16 | | 90fd144a-2126-45f0-85b7-59d68bff4ba1 | vm-9 | ACTIVE | g3-ext=192.168.6.201 | vm-04-snap-new | WinGPU16 | | 91a069c3-7b31-40d2-b030-5cd23a53619f | vm-10 | ACTIVE | g3-ext=192.168.6.183 | vm-04-snap-new | WinGPU16 | | 9f01af21-101c-412f-a6e3-89632170b932 | vm-8 | ACTIVE | g3-ext=192.168.6.249 | vm-04-snap-new | WinGPU16 | | d345c99b-dac4-4ba2-8d2a-4d2aab5ab73c | vm-5 | ACTIVE | g3-ext=192.168.6.229 | vm-04-snap-new | WinGPU16 | | facfc95d-bcb5-452c-a7fb-fcfa22f4257b | vm-1 | ACTIVE | g3-ext=192.168.6.222 | vm-04-snap-new | WinGPU16 | | cec33607-a668-4ce8-ac16-c9f5b143d676 | VM-04 | SHUTOFF | g3-ext=192.168.6.239 | | WinGPU16 | | 87efb650-d7b0-4459-a401-80ae95cd35c4 | VM-03 | SHUTOFF | g3-ext=192.168.6.179 | | WinGPU16 | +--------------------------------------+----------+---------+----------------------+----------------+----------+ controller-1:~$ openstack volume list +--------------------------------------+---------------+--------+------+--------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+---------------+--------+------+--------------------------------------------------------------+ | d2d51653-23d5-44a6-a713-b69c010c440d | | in-use | 1000 | Attached to vm-10 on /dev/vdb | | 86f5b026-cdd6-46e1-9726-d159d38d2d1a | | in-use | 1000 | Attached to vm-8 on /dev/vdb | | 775a21c3-9229-4487-a30d-267a844a9ed4 | | in-use | 1000 | Attached to vm-9 on /dev/vdb | | 021cbae9-6610-4bbd-ac5d-6663c419532f | | in-use | 1000 | Attached to vm-7 on /dev/vdb | | f46cefa0-3cfe-4a52-b6d2-8bccb6d1b2d9 | | in-use | 1000 | Attached to vm-6 on /dev/vdb | | 887131fe-c2b8-41ca-b0c8-72dd75b4cd5b | | in-use | 1000 | Attached to vm-5 on /dev/vdb | | f5aafed0-4b4b-4bce-ba5e-e96f717ecfa3 | | in-use | 1000 | Attached to vm-4 on /dev/vdb | | ce07fe51-3808-4947-acea-a6344d39ef2b | | in-use | 1000 | Attached to vm-2 on /dev/vdb | | c7088c07-e4cd-4baa-8c1a-1deeab5b486c | | in-use | 1000 | Attached to vm-1 on /dev/vdb | | b1816c08-53c3-4391-8396-e710acf24edf | | in-use | 1000 | Attached to vm-3 on /dev/vdb | | e3b8ccc9-0bdd-4c4f-8256-7e4905637d56 | | in-use | 100 | Attached to vm-8 on /dev/vda | | a4a933e2-b139-4ba0-8cbe-523ac0eb5036 | | in-use | 100 | Attached to vm-10 on /dev/vda | | 81997801-840a-4f23-98d6-af961abad590 | | in-use | 100 | Attached to vm-9 on /dev/vda | | 3d22538f-6f88-4529-a689-df0c37f2a00f | | in-use | 100 | Attached to vm-7 on /dev/vda | | de0551a6-e27e-4dd9-9a5b-ac2569c0dc76 | | in-use | 100 | Attached to vm-4 on /dev/vda | | b080eb16-c17e-40bd-b345-40933d6d1977 | | in-use | 100 | Attached to vm-3 on /dev/vda | | 8b514c8f-f190-4617-be12-be21f15cbd2e | | in-use | 100 | Attached to vm-5 on /dev/vda | | 0e0b0a37-060f-4bb0-b78b-288830d627e5 | | in-use | 100 | Attached to vm-6 on /dev/vda | | 86fba86b-9fdd-4d4c-8d41-f56f03152206 | | in-use | 100 | Attached to vm-2 on /dev/vda | | 69c1de5d-f4ca-4fe8-a617-bd22863610cb | | in-use | 100 | Attached to vm-1 on /dev/vda | | b239c3da-3276-4497-ad4c-1e900e907426 | | in-use | 100 | Attached to VM-04 on /dev/vda | | 28088321-f87a-4392-bd6c-bcd0174184ea | | in-use | 100 | Attached to VM-03 on /dev/vda | | c693b9ad-6d47-4a2c-93e4-36baaf25f062 | shared-volume | in-use | 1000 | Attached to VM-03 on /dev/vdb Attached to VM-04 on /dev/vdb | +--------------------------------------+---------------+--------+------+--------------------------------------------------------------+ As you can see the original multi attach volume "shared-volume" is attached to VM-03 but all the other 1TB volumes are not shared but directly attach only per one sevrer/vm. Seems we have to attach the shared volume separately into each VM. I've never had the pleasure to play around with this, but creating a new VM > from a snapshot could mean a new volume too. I'm not sure how snapshot will > work with multi volume VMs. > > On Fri, Sep 24, 2021 at 11:20 AM open infra wrote: > >> Hi, >> >> I am planning to spin hundreds of servers with a shared storage. >> I have created one instance and attached a 1 TB volume (multiattach) and >> created a snapshot of the instance. >> But when I spin another server using the snapshot, I noticed that instead >> of expected shared volume, individual 1 TB size additional (dedicated) >> volume has been attached to each VM. >> >> Is there a mechanism where I can create multiple instances along with a >> shared volume attached to the instance instead of attaching later? >> >> Regards, >> Danishka >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From satish.txt at gmail.com Sun Sep 26 03:31:19 2021 From: satish.txt at gmail.com (Satish Patel) Date: Sat, 25 Sep 2021 23:31:19 -0400 Subject: anyone running ovn with ovs+dpdk Message-ID: Folks, Anyone running ovn with dpdk on ubuntu 20.04 (Focal) with ovs version 2.13.3 ? I am trying to do that but my ovs switch keep crashing as soon as i spin up vm and it creates socket file in /var/run/openvswitch and I am not able to understand what is wrong with it. (no generating crash dump also) Please advise or share your thoughts. From skaplons at redhat.com Sun Sep 26 07:14:07 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Sun, 26 Sep 2021 09:14:07 +0200 Subject: [neutron] CI update In-Reply-To: References: <2741059.qUHmEk80i5@p1> <20210925070244.wz3cj4kpg2b6rgui@p1.localdomain> Message-ID: <3451555.SPnqAGylcH@p1> Hi, Thx. Unfortunatelly it looks that the problem is still there: https:// f17f24740cbe042b8de3-2bd0a90c56ef0ba69bd0efa56ba93067.ssl.cf1.rackcdn.com/ 810973/1/check/ironic-tempest-ipa-partition-pxe_ipmitool/a36b93f/controller/ logs/screen-q-agt.txt We will need to investigate it on Monday. On sobota, 25 wrze?nia 2021 09:10:04 CEST Iury Gregory wrote: > Hi Slawek, > > I just pushed a patch to revert the change we did to workaround the issue > in Ironic [1]. Thanks! > > [1] https://review.opendev.org/c/openstack/ironic/+/810973 > > Em s?b., 25 de set. de 2021 ?s 09:04, Slawek Kaplonski > > escreveu: > > Hi, > > > > On Wed, Sep 22, 2021 at 04:55:34PM +0200, Rodolfo Alonso Hernandez wrote: > > > Hello folks: > > > > > > I replied in [1]. I think we could have a potential problem when > > > > executing > > > > > a DB change the impies a OF controller re-initialization. Most of the > > > > time > > > > > we don't have problems but as we see in the CI, we could sometimes. > > > > > > I'll push a patch to add a retry decorator on the methods that trigger > > > > this > > > > > OF controller restart. > > > > I see that patch https://review.opendev.org/c/openstack/neutron/+/810592 > > is > > already merged. Thx Rodolfo for quick fix. > > @Julia: can You check if Ironic jobs are ok now? > > > > > Regards. > > > > > > [1]https://bugs.launchpad.net/neutron/+bug/1944201/comments/4 > > > > > > > > > On Wed, Sep 22, 2021 at 4:36 PM Slawek Kaplonski > > > > > > wrote: > > > > Hi > > > > > > > > On Wed, Sep 22, 2021 at 03:30:08PM +0200, Lajos Katona wrote: > > > > > Hi Slawek, > > > > > Thanks for the summary. > > > > > Regarding https://bugs.launchpad.net/neutron/+bug/1944201 not sure > > > > > > > > about if > > > > > > > > > it is related to the number of hosts, there's some failure in > > > > > > > singlenode jobs as well: > > http://logstash.openstack.org/#dashboard/file/logstash.json? query=message%3A > > %5C%22error%20Datapath%20Invalid%5C%22> > > > > > example: > > https://9f672e0630f459ee81cb-e4093b1756a9a5a7c7d28e6575b4af7f.ssl.cf5.rackcd > > n.com/805849/10/check/neutron-ovs-rally-task/5981df8/controller/logs/ screen- > > q-agt.txt> > > > > Ok. So it happens on all types of jobs where neutron-ovs-agent is used > > : > > :/ > > : > > > > > Lajos (lajoskatona) > > > > > > > > > > Slawek Kaplonski ezt ?rta (id?pont: 2021. > > > > szept. > > > > > > 22., > > > > > > > > > Sze, 12:10): > > > > > > Hi, > > > > > > > > > > > > Due to very urgent things which I had yesterday, I wasn't able to > > > > run > > > > > > > > Neutron > > > > > > CI meeting as usually. Fortunatelly we don't have many new issues > > > > in > > > > > > our > > > > > > > > > > CI. > > > > > > There is only one new issue in our scenario jobs which I wanted to > > > > > > > > discuss > > > > > > > > > > [1]. It's impacting Ironic gates but I noticed it also in the > > > > Neutron > > > > > > CI > > > > > > > > > > as > > > > > > well. See [2] or [3] for example. I'm not sure about Ironic jobs > > > > but in > > > > > > > > Neutron I saw it mostly (or only, I'm not sure) in the multinode > > > > jobs. > > > > > > > > [1] https://bugs.launchpad.net/neutron/+bug/1944201 > > > > > > [2] https:// > > > > e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ > > > > > > 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ > > > > > > > > logs/screen-q-agt.txt > > > > > > < > > > > http:// e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn > > .com/805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/ comput > > e1/logs/screen-q-agt.txt> > > > > > > [3] https://storage.bhs.cloud.ovh.net/v1/ > > > > AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/ check/ > > > > > > > > neutron-ovs-tempest-slow/88f8bb7/job-output.txt > > > > > > > > > > > > -- > > > > > > Slawek Kaplonski > > > > > > Principal Software Engineer > > > > > > Red Hat > > > > > > > > -- > > > > Slawek Kaplonski > > > > Principal Software Engineer > > > > Red Hat > > > > -- > > Slawek Kaplonski > > Principal Software Engineer > > Red Hat -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From kira034 at 163.com Sun Sep 26 08:14:19 2021 From: kira034 at 163.com (Hongbin Lu) Date: Sun, 26 Sep 2021 16:14:19 +0800 (CST) Subject: [zun][requirements] request to blacklist docker 5.0.1 for X release Message-ID: <6d3b5197.46a9.17c212b51b9.Coremail.kira034@163.com> Hi, It appears that docker 5.0.1 is broken [1]. All the OpenStack projects that depends on docker-py (i.e. Zun [2]) are impacted. Would the requirements team consider blacklist docker 5.0.1 for X release? Please note that Zun must have such blacklist. Otherwise, the X release will be totally broken. [1] https://github.com/docker/docker-py/issues/2885 [2] https://bugs.launchpad.net/zun/+bug/1945088 Best regards, Hongbin -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Sun Sep 26 08:32:13 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Sun, 26 Sep 2021 10:32:13 +0200 Subject: [zun][requirements] request to blacklist docker 5.0.1 for X release In-Reply-To: <6d3b5197.46a9.17c212b51b9.Coremail.kira034@163.com> References: <6d3b5197.46a9.17c212b51b9.Coremail.kira034@163.com> Message-ID: Already being discussed (from a slightly different angle) in http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025042.html -yoctozepto On Sun, 26 Sept 2021 at 10:15, Hongbin Lu wrote: > > Hi, > > It appears that docker 5.0.1 is broken [1]. All the OpenStack projects that depends on docker-py (i.e. Zun [2]) are impacted. Would the requirements team consider blacklist docker 5.0.1 for X release? > > Please note that Zun must have such blacklist. Otherwise, the X release will be totally broken. > > [1] https://github.com/docker/docker-py/issues/2885 > [2] https://bugs.launchpad.net/zun/+bug/1945088 > > Best regards, > Hongbin > > > From kira034 at 163.com Sun Sep 26 09:39:08 2021 From: kira034 at 163.com (Hongbin Lu) Date: Sun, 26 Sep 2021 17:39:08 +0800 (GMT+08:00) Subject: [zun][requirements] request to blacklist docker 5.0.1 for X release In-Reply-To: References: <6d3b5197.46a9.17c212b51b9.Coremail.kira034@163.com> Message-ID: <410b46f3.567b.17c2178f9a7.Coremail.kira034@163.com> the discussion there didn't address my concern. ?? ?????? ---- ?????? ---- | ??? | Rados?aw Piliszek | | ?? | 2021?09?26? 16:32 | | ??? | Hongbin Lu | | ??? | openstack at lists.openstack.org | | ?? | Re: [zun][requirements] request to blacklist docker 5.0.1 for X release | Already being discussed (from a slightly different angle) in http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025042.html -yoctozepto On Sun, 26 Sept 2021 at 10:15, Hongbin Lu wrote: > > Hi, > > It appears that docker 5.0.1 is broken [1]. All the OpenStack projects that depends on docker-py (i.e. Zun [2]) are impacted. Would the requirements team consider blacklist docker 5.0.1 for X release? > > Please note that Zun must have such blacklist. Otherwise, the X release will be totally broken. > > [1] https://github.com/docker/docker-py/issues/2885 > [2] https://bugs.launchpad.net/zun/+bug/1945088 > > Best regards, > Hongbin > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kira034 at 163.com Mon Sep 27 02:19:41 2021 From: kira034 at 163.com (Hongbin Lu) Date: Mon, 27 Sep 2021 10:19:41 +0800 (GMT+08:00) Subject: [requirements] Please consider bumping docker to 5.0.2 in Xena u-c In-Reply-To: <20210924143032.gx4d4pzicug6s6ad@mthode.org> References: <20210924143032.gx4d4pzicug6s6ad@mthode.org> Message-ID: <711e6d08.74df.17c250d0084.Coremail.kira034@163.com> I don't understand. Why after you branch. This will lead to a broken X release for Zun and other projects that depends on docker. What are the rationals here? ?? ?????? ---- ?????? ---- | ??? | Matthew Thode | | ?? | 2021?09?24? 22:30 | | ??? | Pierre Riteau | | ??? | OpenStack Discuss | | ?? | Re: [requirements] Please consider bumping docker to 5.0.2 in Xena u-c | After we branch, ya, probably On 21-09-24 13:11:19, Pierre Riteau wrote: > Hello, > > Xena upper constraints currently contain docker 5.0.1, released on > August 31. This version includes a critical issue [1] which was fixed > by version 5.0.2 released the next day. Except for a docstring update > this is the only change, according to git diff. > > Would it be possible to update Xena u-c to use docker 5.0.2? We > currently have to work around it in Kayobe to avoid installing the > buggy version [2]. > > Thanks, > Pierre Riteau (priteau) > > [1] https://github.com/docker/docker-py/issues/2885 > [2] https://review.opendev.org/c/openstack/kayobe/+/807345/9/ansible/kolla-target-venv.yml > -- Matthew Thode -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Mon Sep 27 05:48:59 2021 From: tonykarera at gmail.com (Karera Tony) Date: Mon, 27 Sep 2021 07:48:59 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> Message-ID: Hello Team, I even tried to manually put the ceph.client.cinder.keyring in the nova_compute container but the issue persisted. I also tried reinstalling Openstack on another Environment but I still have the same issue. Anyone with any idea on how to proceed ? Regards Tony Karera On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont wrote: > I know that there is some Kolla folks around but keep in mind that this is > a volunteer based list :) > > I think you might get a bit more one to one help on IRC in their kolla > channel. > > On Fri, Sep 24, 2021 at 5:10 PM Karera Tony wrote: > >> I would really appreciate any support on this >> >> On Fri, 24 Sep 2021, 11:13 Karera Tony, wrote: >> >>> Hello Team, >>> >>> I don't know if there has been any change in the packages but the way I >>> am deploying is the same way I have been deploying. >>> >>> I don't understand why there is a certain issue now. >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Fri, Sep 24, 2021 at 7:30 AM Karera Tony >>> wrote: >>> >>>> Hello Laurent, >>>> >>>> It turns out I only have one keyring in the container. >>>> >>>> root at compute1:/home/stack# docker exec -it nova_compute bash >>>> (nova-compute)[nova at compute1 /]$ cd /etc/ceph/ >>>> (nova-compute)[nova at compute1 ceph]$ ls >>>> ceph.client.nova.keyring ceph.conf rbdmap >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < >>>> laurentfdumont at gmail.com> wrote: >>>> >>>>> I do believe Kolla runs a container version of each service on >>>>> computes. Are you looking inside the nova-compute container ( >>>>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >>>>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >>>>> >>>>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony >>>>> wrote: >>>>> >>>>>> Hello Sean, >>>>>> >>>>>> Below are the output on the compute node and deployment >>>>>> >>>>>> root at compute1:/etc/kolla/nova-compute# ls >>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>> config.json nova.conf >>>>>> >>>>>> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ >>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>> >>>>>> And I can confirm that the content is the same. >>>>>> >>>>>> >>>>>> >>>>>> Regards >>>>>> >>>>>> Tony Karera >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney >>>>>> wrote: >>>>>> >>>>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>>>>> > I would investigate that compute error first. Creating volumes >>>>>>> means the >>>>>>> > controllers are doing the action. Starting a VM on a compute means >>>>>>> you also >>>>>>> > need Ceph to works on the compute to mount the rdb target. >>>>>>> >>>>>>> nova as part of its startup process in aintiallying the resouce >>>>>>> tracker will >>>>>>> try to connect to ceph if you are using the rbd image backend to >>>>>>> report how much stroage >>>>>>> is avaiable. if the keyring does not work on the vms pool as the >>>>>>> user nova is connecting as >>>>>>> then that will block the agent from starting up fully and will cause >>>>>>> it to be missing the hypervior list. >>>>>>> >>>>>>> the error seams to indicate that the cinder keyring is not in the >>>>>>> nova container >>>>>>> that likely means you have not put it in /etc/kolla/config/nova >>>>>>> i woudl check /etc/kolla/config/nova on the deployment host and sudo >>>>>>> ls /etc/kolla/nova-compute/ >>>>>>> on the compute node to ensure the cinder keyring is actully copied >>>>>>> and has the correct content >>>>>>> >>>>>>> i have >>>>>>> stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>> config.json nova.conf >>>>>>> >>>>>>> >>>>>>> [client.cinder] >>>>>>> key = ********************************* >>>>>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>>>>> caps mon = "profile rbd" >>>>>>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>>>>>> profile rbd pool=images" >>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>>>>> [client.nova] >>>>>>> key = ********************************* >>>>>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>>>>> caps mon = "profile rbd" >>>>>>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>>>>>> profile rbd pool=images" >>>>>>> >>>>>>> blanked out the key wiht *************** after the fact but you >>>>>>> should have something similar >>>>>>> >>>>>>> >>>>>>> in my case i decied to use a seperate key for nova rbd backend >>>>>>> because i was also using EC poosl with a seperate data and metadata pool >>>>>>> so i neede to modify my ceph.conf to make that work with kolla >>>>>>> >>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>>> /etc/kolla/nova-compute/ceph.conf >>>>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>>>>> [global] >>>>>>> fsid = ********************* >>>>>>> mon_host = [*********************] >>>>>>> >>>>>>> [client.glance] >>>>>>> rbd default data pool = images-data >>>>>>> >>>>>>> [client.cinder] >>>>>>> rbd default data pool = volumes-data >>>>>>> >>>>>>> [client.nova] >>>>>>> rbd default data pool = vms-data >>>>>>> >>>>>>> using 2 keyrings/user allows me to set different default data pools >>>>>>> for cinder and nova. >>>>>>> >>>>>>> > >>>>>>> > Working in Wallaby with the error doesn't mean it would 100% work >>>>>>> in >>>>>>> > Victoria. >>>>>>> > >>>>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony >>>>>>> wrote: >>>>>>> > >>>>>>> > > Hey Guys, Any other idea ? >>>>>>> > > >>>>>>> > > Regards >>>>>>> > > >>>>>>> > > Tony Karera >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >>>>>>> tonykarera at gmail.com> wrote: >>>>>>> > > >>>>>>> > > > Just to add on that, >>>>>>> > > > >>>>>>> > > > compute service is listed, I can create Volumes, I have the >>>>>>> same cinder >>>>>>> > > > keyring in the /etc/kolla/config/nova directory as I have in >>>>>>> the >>>>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along with >>>>>>> the nova keyring >>>>>>> > > > Regards >>>>>>> > > > >>>>>>> > > > Tony Karera >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>>>>>> tonykarera at gmail.com> wrote: >>>>>>> > > > >>>>>>> > > > > Hello Guys, >>>>>>> > > > > >>>>>>> > > > > Thanks a lot. >>>>>>> > > > > >>>>>>> > > > > I had actually checked the nova -compute.log on the compute >>>>>>> node and >>>>>>> > > > > they were showing the error I will post at the end about the >>>>>>> cinder keyring >>>>>>> > > > > but I know its correct because its the same I was using on >>>>>>> wallaby, I even >>>>>>> > > > > tried to use another ceph cluster with ofcouse different >>>>>>> keyrings but its >>>>>>> > > > > the same issue. >>>>>>> > > > > >>>>>>> > > > > Below is the error >>>>>>> > > > > >>>>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>>>> auth: unable to >>>>>>> > > > > find a keyring on >>>>>>> > > > > >>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 >>>>>>> 7fbce2f4f700 -1 >>>>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>>>>> > > > > >>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >>>>>>> -1 auth: unable >>>>>>> > > > > to find a keyring on >>>>>>> > > > > >>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>>>>>> 7fbce2f4f700 -1 >>>>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>>>>> > > > > >>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >>>>>>> -1 auth: unable >>>>>>> > > > > to find a keyring on >>>>>>> > > > > >>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>>>>>> 7fbce2f4f700 -1 >>>>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>>>>> > > > > >>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>>>>> connecting to the >>>>>>> > > > > cluster)\n' >>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During >>>>>>> handling of >>>>>>> > > > > the above exception, another exception occurred: >>>>>>> > > > > Regards >>>>>>> > > > > >>>>>>> > > > > Tony Karera >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>>>>>> smooney at redhat.com> wrote: >>>>>>> > > > > >>>>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>>>>>> > > > > > > It could also be a compute cell discovery issue maybe? >>>>>>> > > > > > no they shoudl still show up in the hypervior list api >>>>>>> > > > > > > >>>>>>> > > > > > > Do you see anything under "openstack compute service >>>>>>> list"? >>>>>>> > > > > > if they show up in the service list but not they hyperiors >>>>>>> api it >>>>>>> > > > > > means that the comptue service started and registered its >>>>>>> service entry >>>>>>> > > > > > but >>>>>>> > > > > > something broke it before it could create a compute node >>>>>>> recored in the >>>>>>> > > > > > db. >>>>>>> > > > > > >>>>>>> > > > > > with ceph the case i have hit this most often is when the >>>>>>> keyright used >>>>>>> > > > > > by nova to >>>>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>>>>>> whihc prevent >>>>>>> > > > > > the resoucetack and compute manager >>>>>>> > > > > > form actully creating the compute node record. >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > it can happen for other reason too but best place to start >>>>>>> is check if >>>>>>> > > > > > there is an error in the nova compute agent log and go >>>>>>> from there. >>>>>>> > > > > > > >>>>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>>>>> smooney at redhat.com> >>>>>>> > > > > > wrote: >>>>>>> > > > > > > >>>>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >>>>>>> > > > > > > > > Hello Team, >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > I have deployed Openstack Victoria using >>>>>>> Kolla-ansible on Ubuntu >>>>>>> > > > > > 20.04 >>>>>>> > > > > > > > and >>>>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>>>>>> Glance. >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > It finished with no error but it has failed to >>>>>>> register any on the >>>>>>> > > > > > > > Compute >>>>>>> > > > > > > > > Nodes under Hypervisors. >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack >>>>>>> hypervisor list >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > (kolla-openstack) stack at deployment:~$ >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > Any idea on how to resolve this ? >>>>>>> > > > > > > > that usually means that somehthing prevented the >>>>>>> comptue agent form >>>>>>> > > > > > > > strating properly >>>>>>> > > > > > > > >>>>>>> > > > > > > > for example incorrect ceph keyrings there are several >>>>>>> other case >>>>>>> > > > > > but you >>>>>>> > > > > > > > mentioned you are >>>>>>> > > > > > > > using ceph. >>>>>>> > > > > > > > >>>>>>> > > > > > > > if this is hte case you should see error in the >>>>>>> compute agent log. >>>>>>> > > > > > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > Regards >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > Tony Karera >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> >>>>>>> >>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnasiadka at gmail.com Mon Sep 27 05:57:31 2021 From: mnasiadka at gmail.com (=?UTF-8?Q?Micha=C5=82_Nasiadka?=) Date: Mon, 27 Sep 2021 07:57:31 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> Message-ID: Hi Tony, As Laurent mentioned - it would be best if you would reach out on #openstack-kolla IRC channel - and we?ll try to do our best to help you. Here?s an IRC guide from Contributors guide - if you?re not familiar: https://docs.openstack.org/contributors/common/irc.html Regards, Michal Nasiadka W dniu pon., 27.09.2021 o 07:52 Karera Tony napisa?(a): > Hello Team, > > I even tried to manually put the ceph.client.cinder.keyring in the > nova_compute container but the issue persisted. > > I also tried reinstalling Openstack on another Environment but I still > have the same issue. > > Anyone with any idea on how to proceed ? > Regards > > Tony Karera > > > > > On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont > wrote: > >> I know that there is some Kolla folks around but keep in mind that this >> is a volunteer based list :) >> >> I think you might get a bit more one to one help on IRC in their kolla >> channel. >> >> On Fri, Sep 24, 2021 at 5:10 PM Karera Tony wrote: >> >>> I would really appreciate any support on this >>> >>> On Fri, 24 Sep 2021, 11:13 Karera Tony, wrote: >>> >>>> Hello Team, >>>> >>>> I don't know if there has been any change in the packages but the way I >>>> am deploying is the same way I have been deploying. >>>> >>>> I don't understand why there is a certain issue now. >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Fri, Sep 24, 2021 at 7:30 AM Karera Tony >>>> wrote: >>>> >>>>> Hello Laurent, >>>>> >>>>> It turns out I only have one keyring in the container. >>>>> >>>>> root at compute1:/home/stack# docker exec -it nova_compute bash >>>>> (nova-compute)[nova at compute1 /]$ cd /etc/ceph/ >>>>> (nova-compute)[nova at compute1 ceph]$ ls >>>>> ceph.client.nova.keyring ceph.conf rbdmap >>>>> >>>>> Regards >>>>> >>>>> Tony Karera >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < >>>>> laurentfdumont at gmail.com> wrote: >>>>> >>>>>> I do believe Kolla runs a container version of each service on >>>>>> computes. Are you looking inside the nova-compute container ( >>>>>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >>>>>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >>>>>> >>>>>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony >>>>>> wrote: >>>>>> >>>>>>> Hello Sean, >>>>>>> >>>>>>> Below are the output on the compute node and deployment >>>>>>> >>>>>>> root at compute1:/etc/kolla/nova-compute# ls >>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>> config.json nova.conf >>>>>>> >>>>>>> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ >>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>> >>>>>>> And I can confirm that the content is the same. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> Tony Karera >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney >>>>>>> wrote: >>>>>>> >>>>>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>>>>>> > I would investigate that compute error first. Creating volumes >>>>>>>> means the >>>>>>>> > controllers are doing the action. Starting a VM on a compute >>>>>>>> means you also >>>>>>>> > need Ceph to works on the compute to mount the rdb target. >>>>>>>> >>>>>>>> nova as part of its startup process in aintiallying the resouce >>>>>>>> tracker will >>>>>>>> try to connect to ceph if you are using the rbd image backend to >>>>>>>> report how much stroage >>>>>>>> is avaiable. if the keyring does not work on the vms pool as the >>>>>>>> user nova is connecting as >>>>>>>> then that will block the agent from starting up fully and will >>>>>>>> cause it to be missing the hypervior list. >>>>>>>> >>>>>>>> the error seams to indicate that the cinder keyring is not in the >>>>>>>> nova container >>>>>>>> that likely means you have not put it in /etc/kolla/config/nova >>>>>>>> i woudl check /etc/kolla/config/nova on the deployment host and >>>>>>>> sudo ls /etc/kolla/nova-compute/ >>>>>>>> on the compute node to ensure the cinder keyring is actully copied >>>>>>>> and has the correct content >>>>>>>> >>>>>>>> i have >>>>>>>> stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>>> config.json nova.conf >>>>>>>> >>>>>>>> >>>>>>>> [client.cinder] >>>>>>>> key = ********************************* >>>>>>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>>>>>> caps mon = "profile rbd" >>>>>>>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>>>>>>> profile rbd pool=images" >>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>>>>>> [client.nova] >>>>>>>> key = ********************************* >>>>>>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>>>>>> caps mon = "profile rbd" >>>>>>>> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >>>>>>>> profile rbd pool=images" >>>>>>>> >>>>>>>> blanked out the key wiht *************** after the fact but you >>>>>>>> should have something similar >>>>>>>> >>>>>>>> >>>>>>>> in my case i decied to use a seperate key for nova rbd backend >>>>>>>> because i was also using EC poosl with a seperate data and metadata pool >>>>>>>> so i neede to modify my ceph.conf to make that work with kolla >>>>>>>> >>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>>>> /etc/kolla/nova-compute/ceph.conf >>>>>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>>>>>> [global] >>>>>>>> fsid = ********************* >>>>>>>> mon_host = [*********************] >>>>>>>> >>>>>>>> [client.glance] >>>>>>>> rbd default data pool = images-data >>>>>>>> >>>>>>>> [client.cinder] >>>>>>>> rbd default data pool = volumes-data >>>>>>>> >>>>>>>> [client.nova] >>>>>>>> rbd default data pool = vms-data >>>>>>>> >>>>>>>> using 2 keyrings/user allows me to set different default data pools >>>>>>>> for cinder and nova. >>>>>>>> >>>>>>>> > >>>>>>>> > Working in Wallaby with the error doesn't mean it would 100% work >>>>>>>> in >>>>>>>> > Victoria. >>>>>>>> > >>>>>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > > Hey Guys, Any other idea ? >>>>>>>> > > >>>>>>>> > > Regards >>>>>>>> > > >>>>>>>> > > Tony Karera >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>> > > >>>>>>>> > > > Just to add on that, >>>>>>>> > > > >>>>>>>> > > > compute service is listed, I can create Volumes, I have the >>>>>>>> same cinder >>>>>>>> > > > keyring in the /etc/kolla/config/nova directory as I have in >>>>>>>> the >>>>>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along with >>>>>>>> the nova keyring >>>>>>>> > > > Regards >>>>>>>> > > > >>>>>>>> > > > Tony Karera >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>> > > > >>>>>>>> > > > > Hello Guys, >>>>>>>> > > > > >>>>>>>> > > > > Thanks a lot. >>>>>>>> > > > > >>>>>>>> > > > > I had actually checked the nova -compute.log on the >>>>>>>> compute node and >>>>>>>> > > > > they were showing the error I will post at the end about >>>>>>>> the cinder keyring >>>>>>>> > > > > but I know its correct because its the same I was using on >>>>>>>> wallaby, I even >>>>>>>> > > > > tried to use another ceph cluster with ofcouse different >>>>>>>> keyrings but its >>>>>>>> > > > > the same issue. >>>>>>>> > > > > >>>>>>>> > > > > Below is the error >>>>>>>> > > > > >>>>>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>>>>> auth: unable to >>>>>>>> > > > > find a keyring on >>>>>>>> > > > > >>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 >>>>>>>> 7fbce2f4f700 -1 >>>>>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>>>>>> > > > > >>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >>>>>>>> -1 auth: unable >>>>>>>> > > > > to find a keyring on >>>>>>>> > > > > >>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>>>>>>> 7fbce2f4f700 -1 >>>>>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>>>>>> > > > > >>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >>>>>>>> -1 auth: unable >>>>>>>> > > > > to find a keyring on >>>>>>>> > > > > >>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >>>>>>>> 7fbce2f4f700 -1 >>>>>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>>>>>> > > > > >>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>>>>>> connecting to the >>>>>>>> > > > > cluster)\n' >>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During >>>>>>>> handling of >>>>>>>> > > > > the above exception, another exception occurred: >>>>>>>> > > > > Regards >>>>>>>> > > > > >>>>>>>> > > > > Tony Karera >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>>>>>>> smooney at redhat.com> wrote: >>>>>>>> > > > > >>>>>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>>>>>>> > > > > > > It could also be a compute cell discovery issue maybe? >>>>>>>> > > > > > no they shoudl still show up in the hypervior list api >>>>>>>> > > > > > > >>>>>>>> > > > > > > Do you see anything under "openstack compute service >>>>>>>> list"? >>>>>>>> > > > > > if they show up in the service list but not they >>>>>>>> hyperiors api it >>>>>>>> > > > > > means that the comptue service started and registered its >>>>>>>> service entry >>>>>>>> > > > > > but >>>>>>>> > > > > > something broke it before it could create a compute node >>>>>>>> recored in the >>>>>>>> > > > > > db. >>>>>>>> > > > > > >>>>>>>> > > > > > with ceph the case i have hit this most often is when the >>>>>>>> keyright used >>>>>>>> > > > > > by nova to >>>>>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>>>>>>> whihc prevent >>>>>>>> > > > > > the resoucetack and compute manager >>>>>>>> > > > > > form actully creating the compute node record. >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > it can happen for other reason too but best place to >>>>>>>> start is check if >>>>>>>> > > > > > there is an error in the nova compute agent log and go >>>>>>>> from there. >>>>>>>> > > > > > > >>>>>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>>>>>> smooney at redhat.com> >>>>>>>> > > > > > wrote: >>>>>>>> > > > > > > >>>>>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >>>>>>>> > > > > > > > > Hello Team, >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > I have deployed Openstack Victoria using >>>>>>>> Kolla-ansible on Ubuntu >>>>>>>> > > > > > 20.04 >>>>>>>> > > > > > > > and >>>>>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>>>>>>> Glance. >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > It finished with no error but it has failed to >>>>>>>> register any on the >>>>>>>> > > > > > > > Compute >>>>>>>> > > > > > > > > Nodes under Hypervisors. >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack >>>>>>>> hypervisor list >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > (kolla-openstack) stack at deployment:~$ >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > Any idea on how to resolve this ? >>>>>>>> > > > > > > > that usually means that somehthing prevented the >>>>>>>> comptue agent form >>>>>>>> > > > > > > > strating properly >>>>>>>> > > > > > > > >>>>>>>> > > > > > > > for example incorrect ceph keyrings there are several >>>>>>>> other case >>>>>>>> > > > > > but you >>>>>>>> > > > > > > > mentioned you are >>>>>>>> > > > > > > > using ceph. >>>>>>>> > > > > > > > >>>>>>>> > > > > > > > if this is hte case you should see error in the >>>>>>>> compute agent log. >>>>>>>> > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > Regards >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > Tony Karera >>>>>>>> > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> >>>>>>>> >>>>>>>> -- Micha? Nasiadka mnasiadka at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Mon Sep 27 07:08:53 2021 From: hberaud at redhat.com (Herve Beraud) Date: Mon, 27 Sep 2021 09:08:53 +0200 Subject: [release] Release countdown for week R-1, Sep 27 - Oct 1 Message-ID: Development Focus ----------------- We are on the final mile of the Xena development cycle! Remember that the Xena final release will include the latest release candidate (for cycle-with-rc deliverables) or the latest intermediary release (for cycle-with-intermediary deliverables) available. September 30 is the deadline for final Xena release candidates as well as any last cycle-with-intermediary deliverables. We will then enter a quiet period until we tag the final release on October 6, 2021. Teams should be prioritizing fixing release-critical bugs, before that deadline. Otherwise it's time to start planning the Yoga development cycle, including discussing sessions content in preparation of the PTG on the week of October 18. Actions ------- Watch for any translation patches coming through on the stable/xena branch and merge them quickly. If you discover a release-critical issue, please make sure to fix it on the master branch first, then backport the bugfix to the stable/xena branch before triggering a new release. Please drop by #openstack-release with any questions or concerns about the upcoming release ! Upcoming Deadlines & Dates -------------------------- Final Xena release: October 6 Yoga PTG: October 18-22 -- Herv? Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcafarel at redhat.com Mon Sep 27 07:29:49 2021 From: bcafarel at redhat.com (Bernard Cafarelli) Date: Mon, 27 Sep 2021 09:29:49 +0200 Subject: [neutron] Bug deputy report (week starting on 2021-09-20) Message-ID: Hello neutrinos, here is the list of bugs reported for last week, overall all bugs have assignees or fixes in progress. Both critical bugs are merged and with pending xena backports (this week is the final RCs week) In details: Critical * neutron-openvswitch-agent crashes on start with firewall config of br-int - https://bugs.launchpad.net/neutron/+bug/1944201 First seen in Ironic jobs but we also see it in our jobs, fix by ralonsoh merged: https://review.opendev.org/c/openstack/neutron/+/810592 (pending xena backport) * neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle fails with Port already has an attached device - https://bugs.launchpad.net/tripleo/+bug/1943708 Previously reported as https://bugs.launchpad.net/neutron/+bug/1943679, fix with revert proposed by slaweq and merged: https://review.opendev.org/c/openstack/neutron/+/809550 (pending xena backport) High * Traffic leaked from dhcp port before vlan tag is applied - https://bugs.launchpad.net/neutron/+bug/1930414 Possible security issue that was under investigation and now public, assigned to rubasov * OVN does not update the network segment tag - https://bugs.launchpad.net/neutron/+bug/1944708 Assigned to lucasagomes, fix in review: https://review.opendev.org/c/openstack/neutron/+/810704 Medium * Fix OVN metadata agent functional tests for Chassis_Private - https://bugs.launchpad.net/neutron/+bug/1944460 Functional OVN tests failing when using newer OVN, fix merged: https://review.opendev.org/c/openstack/neutron/+/810309 * IPv6 slaac subnet creation causes FixedIpsSubnetsNotOnSameSegment error - https://bugs.launchpad.net/neutron/+bug/1944948 Error raised when creating an IPv6 slaac subnet with multisegment, assigned to elvira Incomplete * Instances with SRIOV ports loose access after failed live migrations - https://bugs.launchpad.net/neutron/+bug/1944619 SRIOV live migration issue, liuyulong gave some pointers -- Bernard Cafarelli -------------- next part -------------- An HTML attachment was scrubbed... URL: From neil at tigera.io Mon Sep 27 08:43:46 2021 From: neil at tigera.io (Neil Jerram) Date: Mon, 27 Sep 2021 09:43:46 +0100 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: On Fri, Sep 24, 2021 at 9:22 PM Thomas Goirand wrote: > Hi Gibi! > > Thanks for bringing this up. > > As a distro package maintainer, here's my view. > > On 9/22/21 2:11 PM, Balazs Gibizer wrote: > > Option 1: Bump the major version of the decorator dependency on stable. > > Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable), for > which I don't even maintain OpenStack anymore (that's OpenStack > Newton...). So I don't see how switching to decorator 4.0.0 is a > problem, and I don't understand how OpenStack could be using 3.4.0 which > is in Jessie (ie: 6 years old Debian release). > > PyPi says Decorator 3.4.0 is from 2012: > https://pypi.org/project/decorator/#history > > Do you have your release numbers correct? If so, then switching to > Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) > and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 > feels a bit crazy (and I wasn't aware of it). > Irrelevant now because there are several more packages affected, not just Decorator. > > > Option 2: Pin the setuptools version during tox installation > > Please don't do this for the master branch, we need OpenStack to stay > current with setuptools (yeah, even if this means breaking changes...). > Pretty sure no one has suggested pinning on master. (And the subject of this email includes "stable", twice!) > > For already released OpenStack: I don't mind much if this is done (I > could backport fixes if something breaks). > +1, I've posted this: https://review.opendev.org/c/openstack/requirements/+/810859 > > > Option 3: turn off lower-constraints testing > > I already expressed myself about this: this is dangerous as distros rely > on it for setting lower bounds as low as possible (which is always > preferred from a distro point of view). > How does this option help for the specific problem in hand? Namely: - setuptools >=58.0.0 does not support use_2to3 option - several packages that are pulled into OpenStack stable builds need use_2to3. (Not saying it doesn't; I'm just ignorant here and would appreciate further explanation.) > > > Option 4: utilize pyproject.toml[6] to specify build-time requirements > > I don't know about pyproject.toml. > > Just my 2 cents, hoping it's useful, > Cheers, > > Thomas Goirand (zigo) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lyarwood at redhat.com Mon Sep 27 08:45:33 2021 From: lyarwood at redhat.com (Lee Yarwood) Date: Mon, 27 Sep 2021 09:45:33 +0100 Subject: Creating multiple servers with a shared volume In-Reply-To: References: Message-ID: <20210927084533.jm2iv337u5burfbe@lyarwood-laptop.usersys.redhat.com> On 25-09-21 13:11:19, open infra wrote: > On Sat, Sep 25, 2021 at 7:37 AM Laurent Dumont > wrote: > > > Just to be sure, the flow is. > > > > - Create VM with root disk on a volume/image. > > - Create another 1TB volume. > > - Attach the volume to the VM as a second drive using a multi attach > > approach ( > > https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiattach.html > > )? > > - Create snapshot of the original VM (with it's root drive + 1 TB > > volume)? > > - Create new VM from snapshot. > > - Instead of a new VM + 1 new root volume + the same 1TB multiattach > > volume attached, you get a new multiattach volume (new volume ID and > > everything)? Because you're attaching a snapshot and not the original volume? See below for more. > Yes, here is the block device mapping of the snapshot (from a VM with 100GB > of root disk and 1TB of multiattach volume) > > block_device_mapping[{"image_id": null, "delete_on_termination": true, > "device_name": "/dev/vda", "disk_bus": "virtio", "volume_id": null, > "volume_type": null, "destination_type": "volume", "source_type": > "snapshot", "guest_format": null, "volume_size": 100, "device_type": > "disk", "snapshot_id": "7b2c7f3a-8420-4a33-820e-2d2231d930c7", > "boot_index": 0, "no_device": null, "tag": null}, {"image_id": null, > "delete_on_termination": false, "device_name": "/dev/vdb", "disk_bus": > null, "volume_id": null, "volume_type": null, "destination_type": "volume", > "source_type": "snapshot", "guest_format": null, "volume_size": 1000, > "device_type": null, "snapshot_id": "c2a0695f-8c9e-46f9-80a8-560c47521eeb", > "boot_index": null, "no_device": null, "tag": null}] As you can see above you're using source_type = "snapshot" and destination_type of "volume". As set out in the following docs this creates a new volume from a given snapshot and attaches that to the instance: https://docs.openstack.org/nova/latest/user/block-device-mapping.html#valid-source-destination-combinations snapshot -> volume - this works exactly as passing type=snap does. It would create a volume from a Cinder volume snapshot and attach that volume to the instance. Can be marked bootable. https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server block_device_mapping_v2.source_type (Optional) The source type of the block device. Valid values are: [..] snapshot: This is only valid with destination_type=volume; creates a volume backed by the given volume snapshot referenced via the block_device_mapping_v2.uuid parameter and attaches it to the server When attaching multiattach enabled volumes to multiple instances you need to use source_type = 'volume' *and* destination_type = 'volume'. Hope this helps, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From thierry at openstack.org Mon Sep 27 09:17:56 2021 From: thierry at openstack.org (Thierry Carrez) Date: Mon, 27 Sep 2021 11:17:56 +0200 Subject: [largescale-sig] Next meeting: Sept 29th, 15utc Message-ID: Hi everyone, The Large Scale SIG meeting is back this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210929T15 A number of topics have already been added to the agenda, including discussing the final details of our next OpenInfra.Live show! Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez From skaplons at redhat.com Mon Sep 27 09:58:17 2021 From: skaplons at redhat.com (Slawek Kaplonski) Date: Mon, 27 Sep 2021 11:58:17 +0200 Subject: [neutron] CI update In-Reply-To: <3451555.SPnqAGylcH@p1> References: <2741059.qUHmEk80i5@p1> <20210925070244.wz3cj4kpg2b6rgui@p1.localdomain> <3451555.SPnqAGylcH@p1> Message-ID: <20210927095817.gbkizllrym7rruys@p1.localdomain> Hi, On Sun, Sep 26, 2021 at 09:14:07AM +0200, Slawek Kaplonski wrote: > Hi, > > Thx. Unfortunatelly it looks that the problem is still there: https:// > f17f24740cbe042b8de3-2bd0a90c56ef0ba69bd0efa56ba93067.ssl.cf1.rackcdn.com/ > 810973/1/check/ironic-tempest-ipa-partition-pxe_ipmitool/a36b93f/controller/ > logs/screen-q-agt.txt > > We will need to investigate it on Monday. Ok, it looks I was too fast with saying that the patch didn't help. It seems that in the job https://zuul.opendev.org/t/openstack/build/a36b93faefde49dc8ac0b3426ec0281d/logs it failed for some different reason and neutron-ovs-agent didn't actually crashed there. > > On sobota, 25 wrze?nia 2021 09:10:04 CEST Iury Gregory wrote: > > Hi Slawek, > > > > I just pushed a patch to revert the change we did to workaround the issue > > in Ironic [1]. Thanks! > > > > [1] https://review.opendev.org/c/openstack/ironic/+/810973 > > > > Em s?b., 25 de set. de 2021 ?s 09:04, Slawek Kaplonski > > > > escreveu: > > > Hi, > > > > > > On Wed, Sep 22, 2021 at 04:55:34PM +0200, Rodolfo Alonso Hernandez wrote: > > > > Hello folks: > > > > > > > > I replied in [1]. I think we could have a potential problem when > > > > > > executing > > > > > > > a DB change the impies a OF controller re-initialization. Most of the > > > > > > time > > > > > > > we don't have problems but as we see in the CI, we could sometimes. > > > > > > > > I'll push a patch to add a retry decorator on the methods that trigger > > > > > > this > > > > > > > OF controller restart. > > > > > > I see that patch https://review.opendev.org/c/openstack/neutron/+/810592 > > > is > > > already merged. Thx Rodolfo for quick fix. > > > @Julia: can You check if Ironic jobs are ok now? > > > > > > > Regards. > > > > > > > > [1]https://bugs.launchpad.net/neutron/+bug/1944201/comments/4 > > > > > > > > > > > > On Wed, Sep 22, 2021 at 4:36 PM Slawek Kaplonski > > > > > > > > wrote: > > > > > Hi > > > > > > > > > > On Wed, Sep 22, 2021 at 03:30:08PM +0200, Lajos Katona wrote: > > > > > > Hi Slawek, > > > > > > Thanks for the summary. > > > > > > Regarding https://bugs.launchpad.net/neutron/+bug/1944201 not sure > > > > > > > > > > about if > > > > > > > > > > > it is related to the number of hosts, there's some failure in > > > > > > > > > singlenode jobs as well: > > > http://logstash.openstack.org/#dashboard/file/logstash.json? > query=message%3A > > > %5C%22error%20Datapath%20Invalid%5C%22> > > > > > > example: > > > https://9f672e0630f459ee81cb-e4093b1756a9a5a7c7d28e6575b4af7f.ssl.cf5.rackcd > > > n.com/805849/10/check/neutron-ovs-rally-task/5981df8/controller/logs/ > screen- > > > q-agt.txt> > > > > > Ok. So it happens on all types of jobs where neutron-ovs-agent is used > > > : > > > :/ > > > : > > > > > > Lajos (lajoskatona) > > > > > > > > > > > > Slawek Kaplonski ezt ?rta (id?pont: 2021. > > > > > > szept. > > > > > > > > 22., > > > > > > > > > > > Sze, 12:10): > > > > > > > Hi, > > > > > > > > > > > > > > Due to very urgent things which I had yesterday, I wasn't able to > > > > > > run > > > > > > > > > > Neutron > > > > > > > CI meeting as usually. Fortunatelly we don't have many new issues > > > > > > in > > > > > > > > our > > > > > > > > > > > > CI. > > > > > > > There is only one new issue in our scenario jobs which I wanted to > > > > > > > > > > discuss > > > > > > > > > > > > [1]. It's impacting Ironic gates but I noticed it also in the > > > > > > Neutron > > > > > > > > CI > > > > > > > > > > > > as > > > > > > > well. See [2] or [3] for example. I'm not sure about Ironic jobs > > > > > > but in > > > > > > > > > > Neutron I saw it mostly (or only, I'm not sure) in the multinode > > > > > > jobs. > > > > > > > > > > [1] https://bugs.launchpad.net/neutron/+bug/1944201 > > > > > > > [2] https:// > > > > > > e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ > > > > > > > > > 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ > > > > > > > > > > logs/screen-q-agt.txt > > > > > > > < > > > > > > http:// > e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn > > > .com/805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/ > comput > > > e1/logs/screen-q-agt.txt> > > > > > > > [3] https://storage.bhs.cloud.ovh.net/v1/ > > > > > > AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/ > check/ > > > > > > > > > > neutron-ovs-tempest-slow/88f8bb7/job-output.txt > > > > > > > > > > > > > > -- > > > > > > > Slawek Kaplonski > > > > > > > Principal Software Engineer > > > > > > > Red Hat > > > > > > > > > > -- > > > > > Slawek Kaplonski > > > > > Principal Software Engineer > > > > > Red Hat > > > > > > -- > > > Slawek Kaplonski > > > Principal Software Engineer > > > Red Hat > > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From finarffin at gmail.com Mon Sep 27 10:47:56 2021 From: finarffin at gmail.com (Jan Wasilewski) Date: Mon, 27 Sep 2021 12:47:56 +0200 Subject: [keystone] Correct policies setup for System Administrators Message-ID: Hello, I am preparing policies configuration before an upgrade to the newer OpenStack release(Stein) and I would like to create a group of System Administrators to be able to get i.e. a list of all projects in the OpenStack cloud. I was following a description from this page [1] but it seems my admin user is able to get only a list of projects where it is directly added(i.e. with member role, reader role, or admin role). I am just wondering if we can list all of the OpenStack projects by System Administrator user without role reader added to every single project? To summarize what steps were done so far: - Original policy.json file which was used is here [2] - Only one option was changed so far: from: "identity:list_projects": "rule:cloud_admin or rule:admin_and_matching_domain_id", to: "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)", - Output for command: openstack role assignment list --system all --role member --role reader +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+ | e39e97c23bfe45d1a2f9689b6985f990 | | a0841b83f583477887219f27dd95477b | | | all | False | +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+Shows only role reader, not role member, which is a bit strange if we compare with linked page above. But we have this in implied roles:openstack implied role list +----------------------------------+-----------------+----------------------------------+-------------------+ | Prior Role ID | Prior Role Name | Implied Role ID | Implied Role Name | +----------------------------------+-----------------+----------------------------------+-------------------+ | a3c7bb5d06884b048c1bfb4403b82b42 | admin | 3f20cb7be46346a8b2ba65c4684d50a3 | member | | a3c7bb5d06884b048c1bfb4403b82b42 | admin | 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | | 3f20cb7be46346a8b2ba65c4684d50a3 | member | e39e97c23bfe45d1a2f9689b6985f990 | reader | +----------------------------------+-----------------+----------------------------------+-------------------+- Admin roles are grouped in a group ATM.Admin: openstack role assignment list --names --system all --role admin: +-------+---------------------+-------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +-------+---------------------+-------------------+---------+--------+--------+-----------+ | admin | | ATM.Admin at Default | | | all | False | | admin | admin at Default | | | | all | False | | admin | jwasilewski at Default | | | | all | False | +-------+---------------------+-------------------+---------+--------+--------+-----------+Just to be sure that IDs are linked, we can check it here:openstack role assignment list --system all --role admin +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ | a3c7bb5d06884b048c1bfb4403b82b42 | | a0841b83f583477887219f27dd95477b | | | all | False | | a3c7bb5d06884b048c1bfb4403b82b42 | 19416fe5a2da45c88eb66c3aaf856c73 | | | | all | False | | a3c7bb5d06884b048c1bfb4403b82b42 | f42df418fd404d04b9bdabf2f1b49fd9 | | | | all | False | +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ So by linking roles(implied roles): admin(a3c7bb5d06884b048c1bfb4403b82b42 ) -> member(3f20cb7be46346a8b2ba65c4684d50a3) -> reader(e39e97c23bfe45d1a2f9689b6985f990). Correlation is visible, based on that my user(jwasilewski) should retrieve a full project list, but it seems only three projects are visible where this user is an admin. I do not want to add my user as a reader to every single project to be able to list all of them. Is there a way how to make it or the only way is to add this role(reader) for a user to all projects? Thank you in advance for any suggestions. Best regards, Jan Wasilewski [1] https://docs.openstack.org/keystone/stein/admin/service-api-protection.html#system-administrators -> https://docs.openstack.org/keystone/latest/admin/service-api-protection.html#system-administrators [2] https://paste.openstack.org/show/bq0HgyqouZF1vywKVkGn/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonykarera at gmail.com Mon Sep 27 12:45:26 2021 From: tonykarera at gmail.com (Karera Tony) Date: Mon, 27 Sep 2021 14:45:26 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> Message-ID: I tried it but no one was replying to the chats Regards Tony Karera On Mon, Sep 27, 2021 at 7:57 AM Micha? Nasiadka wrote: > Hi Tony, > > As Laurent mentioned - it would be best if you would reach out on > #openstack-kolla IRC channel - and we?ll try to do our best to help you. > > Here?s an IRC guide from Contributors guide - if you?re not familiar: > https://docs.openstack.org/contributors/common/irc.html > > Regards, > Michal Nasiadka > > W dniu pon., 27.09.2021 o 07:52 Karera Tony > napisa?(a): > >> Hello Team, >> >> I even tried to manually put the ceph.client.cinder.keyring in the >> nova_compute container but the issue persisted. >> >> I also tried reinstalling Openstack on another Environment but I still >> have the same issue. >> >> Anyone with any idea on how to proceed ? >> Regards >> >> Tony Karera >> >> >> >> >> On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont >> wrote: >> >>> I know that there is some Kolla folks around but keep in mind that this >>> is a volunteer based list :) >>> >>> I think you might get a bit more one to one help on IRC in their kolla >>> channel. >>> >>> On Fri, Sep 24, 2021 at 5:10 PM Karera Tony >>> wrote: >>> >>>> I would really appreciate any support on this >>>> >>>> On Fri, 24 Sep 2021, 11:13 Karera Tony, wrote: >>>> >>>>> Hello Team, >>>>> >>>>> I don't know if there has been any change in the packages but the way >>>>> I am deploying is the same way I have been deploying. >>>>> >>>>> I don't understand why there is a certain issue now. >>>>> Regards >>>>> >>>>> Tony Karera >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Sep 24, 2021 at 7:30 AM Karera Tony >>>>> wrote: >>>>> >>>>>> Hello Laurent, >>>>>> >>>>>> It turns out I only have one keyring in the container. >>>>>> >>>>>> root at compute1:/home/stack# docker exec -it nova_compute bash >>>>>> (nova-compute)[nova at compute1 /]$ cd /etc/ceph/ >>>>>> (nova-compute)[nova at compute1 ceph]$ ls >>>>>> ceph.client.nova.keyring ceph.conf rbdmap >>>>>> >>>>>> Regards >>>>>> >>>>>> Tony Karera >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < >>>>>> laurentfdumont at gmail.com> wrote: >>>>>> >>>>>>> I do believe Kolla runs a container version of each service on >>>>>>> computes. Are you looking inside the nova-compute container ( >>>>>>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >>>>>>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >>>>>>> >>>>>>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony >>>>>>> wrote: >>>>>>> >>>>>>>> Hello Sean, >>>>>>>> >>>>>>>> Below are the output on the compute node and deployment >>>>>>>> >>>>>>>> root at compute1:/etc/kolla/nova-compute# ls >>>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>>> config.json nova.conf >>>>>>>> >>>>>>>> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ >>>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>>> >>>>>>>> And I can confirm that the content is the same. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> Tony Karera >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>>>>>>> > I would investigate that compute error first. Creating volumes >>>>>>>>> means the >>>>>>>>> > controllers are doing the action. Starting a VM on a compute >>>>>>>>> means you also >>>>>>>>> > need Ceph to works on the compute to mount the rdb target. >>>>>>>>> >>>>>>>>> nova as part of its startup process in aintiallying the resouce >>>>>>>>> tracker will >>>>>>>>> try to connect to ceph if you are using the rbd image backend to >>>>>>>>> report how much stroage >>>>>>>>> is avaiable. if the keyring does not work on the vms pool as the >>>>>>>>> user nova is connecting as >>>>>>>>> then that will block the agent from starting up fully and will >>>>>>>>> cause it to be missing the hypervior list. >>>>>>>>> >>>>>>>>> the error seams to indicate that the cinder keyring is not in the >>>>>>>>> nova container >>>>>>>>> that likely means you have not put it in /etc/kolla/config/nova >>>>>>>>> i woudl check /etc/kolla/config/nova on the deployment host and >>>>>>>>> sudo ls /etc/kolla/nova-compute/ >>>>>>>>> on the compute node to ensure the cinder keyring is actully copied >>>>>>>>> and has the correct content >>>>>>>>> >>>>>>>>> i have >>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>>>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>>>> config.json nova.conf >>>>>>>>> >>>>>>>>> >>>>>>>>> [client.cinder] >>>>>>>>> key = ********************************* >>>>>>>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>>>>>>> caps mon = "profile rbd" >>>>>>>>> caps osd = "profile rbd pool=volumes, profile rbd >>>>>>>>> pool=vms, profile rbd pool=images" >>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>>>>>>> [client.nova] >>>>>>>>> key = ********************************* >>>>>>>>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>>>>>>>> caps mon = "profile rbd" >>>>>>>>> caps osd = "profile rbd pool=volumes, profile rbd >>>>>>>>> pool=vms, profile rbd pool=images" >>>>>>>>> >>>>>>>>> blanked out the key wiht *************** after the fact but you >>>>>>>>> should have something similar >>>>>>>>> >>>>>>>>> >>>>>>>>> in my case i decied to use a seperate key for nova rbd backend >>>>>>>>> because i was also using EC poosl with a seperate data and metadata pool >>>>>>>>> so i neede to modify my ceph.conf to make that work with kolla >>>>>>>>> >>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>>>>> /etc/kolla/nova-compute/ceph.conf >>>>>>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>>>>>>> [global] >>>>>>>>> fsid = ********************* >>>>>>>>> mon_host = [*********************] >>>>>>>>> >>>>>>>>> [client.glance] >>>>>>>>> rbd default data pool = images-data >>>>>>>>> >>>>>>>>> [client.cinder] >>>>>>>>> rbd default data pool = volumes-data >>>>>>>>> >>>>>>>>> [client.nova] >>>>>>>>> rbd default data pool = vms-data >>>>>>>>> >>>>>>>>> using 2 keyrings/user allows me to set different default data >>>>>>>>> pools for cinder and nova. >>>>>>>>> >>>>>>>>> > >>>>>>>>> > Working in Wallaby with the error doesn't mean it would 100% >>>>>>>>> work in >>>>>>>>> > Victoria. >>>>>>>>> > >>>>>>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony < >>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>> > >>>>>>>>> > > Hey Guys, Any other idea ? >>>>>>>>> > > >>>>>>>>> > > Regards >>>>>>>>> > > >>>>>>>>> > > Tony Karera >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>> > > >>>>>>>>> > > > Just to add on that, >>>>>>>>> > > > >>>>>>>>> > > > compute service is listed, I can create Volumes, I have the >>>>>>>>> same cinder >>>>>>>>> > > > keyring in the /etc/kolla/config/nova directory as I have >>>>>>>>> in the >>>>>>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along with >>>>>>>>> the nova keyring >>>>>>>>> > > > Regards >>>>>>>>> > > > >>>>>>>>> > > > Tony Karera >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>> > > > >>>>>>>>> > > > > Hello Guys, >>>>>>>>> > > > > >>>>>>>>> > > > > Thanks a lot. >>>>>>>>> > > > > >>>>>>>>> > > > > I had actually checked the nova -compute.log on the >>>>>>>>> compute node and >>>>>>>>> > > > > they were showing the error I will post at the end about >>>>>>>>> the cinder keyring >>>>>>>>> > > > > but I know its correct because its the same I was using on >>>>>>>>> wallaby, I even >>>>>>>>> > > > > tried to use another ceph cluster with ofcouse different >>>>>>>>> keyrings but its >>>>>>>>> > > > > the same issue. >>>>>>>>> > > > > >>>>>>>>> > > > > Below is the error >>>>>>>>> > > > > >>>>>>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>>>>>> auth: unable to >>>>>>>>> > > > > find a keyring on >>>>>>>>> > > > > >>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>>> > > > > (2) No such file or >>>>>>>>> directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>>>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>>>>>>> > > > > >>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >>>>>>>>> -1 auth: unable >>>>>>>>> > > > > to find a keyring on >>>>>>>>> > > > > >>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>>> > > > > (2) No such file or >>>>>>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>>>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>>>>>>> > > > > >>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >>>>>>>>> -1 auth: unable >>>>>>>>> > > > > to find a keyring on >>>>>>>>> > > > > >>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>>> > > > > (2) No such file or >>>>>>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>>>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>>>>>>> > > > > >>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>>>>>>> connecting to the >>>>>>>>> > > > > cluster)\n' >>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>>>>>> During handling of >>>>>>>>> > > > > the above exception, another exception occurred: >>>>>>>>> > > > > Regards >>>>>>>>> > > > > >>>>>>>>> > > > > Tony Karera >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>>>>>>>> smooney at redhat.com> wrote: >>>>>>>>> > > > > >>>>>>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>>>>>>>> > > > > > > It could also be a compute cell discovery issue maybe? >>>>>>>>> > > > > > no they shoudl still show up in the hypervior list api >>>>>>>>> > > > > > > >>>>>>>>> > > > > > > Do you see anything under "openstack compute service >>>>>>>>> list"? >>>>>>>>> > > > > > if they show up in the service list but not they >>>>>>>>> hyperiors api it >>>>>>>>> > > > > > means that the comptue service started and registered >>>>>>>>> its service entry >>>>>>>>> > > > > > but >>>>>>>>> > > > > > something broke it before it could create a compute node >>>>>>>>> recored in the >>>>>>>>> > > > > > db. >>>>>>>>> > > > > > >>>>>>>>> > > > > > with ceph the case i have hit this most often is when >>>>>>>>> the keyright used >>>>>>>>> > > > > > by nova to >>>>>>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>>>>>>>> whihc prevent >>>>>>>>> > > > > > the resoucetack and compute manager >>>>>>>>> > > > > > form actully creating the compute node record. >>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>>> > > > > > it can happen for other reason too but best place to >>>>>>>>> start is check if >>>>>>>>> > > > > > there is an error in the nova compute agent log and go >>>>>>>>> from there. >>>>>>>>> > > > > > > >>>>>>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>>>>>>> smooney at redhat.com> >>>>>>>>> > > > > > wrote: >>>>>>>>> > > > > > > >>>>>>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >>>>>>>>> > > > > > > > > Hello Team, >>>>>>>>> > > > > > > > > >>>>>>>>> > > > > > > > > I have deployed Openstack Victoria using >>>>>>>>> Kolla-ansible on Ubuntu >>>>>>>>> > > > > > 20.04 >>>>>>>>> > > > > > > > and >>>>>>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>>>>>>>> Glance. >>>>>>>>> > > > > > > > > >>>>>>>>> > > > > > > > > It finished with no error but it has failed to >>>>>>>>> register any on the >>>>>>>>> > > > > > > > Compute >>>>>>>>> > > > > > > > > Nodes under Hypervisors. >>>>>>>>> > > > > > > > > >>>>>>>>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack >>>>>>>>> hypervisor list >>>>>>>>> > > > > > > > > >>>>>>>>> > > > > > > > > (kolla-openstack) stack at deployment:~$ >>>>>>>>> > > > > > > > > >>>>>>>>> > > > > > > > > >>>>>>>>> > > > > > > > > Any idea on how to resolve this ? >>>>>>>>> > > > > > > > that usually means that somehthing prevented the >>>>>>>>> comptue agent form >>>>>>>>> > > > > > > > strating properly >>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > for example incorrect ceph keyrings there are >>>>>>>>> several other case >>>>>>>>> > > > > > but you >>>>>>>>> > > > > > > > mentioned you are >>>>>>>>> > > > > > > > using ceph. >>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > if this is hte case you should see error in the >>>>>>>>> compute agent log. >>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > > >>>>>>>>> > > > > > > > > Regards >>>>>>>>> > > > > > > > > >>>>>>>>> > > > > > > > > Tony Karera >>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>>> >>>>>>>>> >>>>>>>>> -- > Micha? Nasiadka > mnasiadka at gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balazs.gibizer at est.tech Mon Sep 27 12:47:48 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Mon, 27 Sep 2021 14:47:48 +0200 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: On Fri, Sep 24 2021 at 10:21:33 PM +0200, Thomas Goirand wrote: > Hi Gibi! > > Thanks for bringing this up. > > As a distro package maintainer, here's my view. > > On 9/22/21 2:11 PM, Balazs Gibizer wrote: >> Option 1: Bump the major version of the decorator dependency on >> stable. > > Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable), > for > which I don't even maintain OpenStack anymore (that's OpenStack > Newton...). So I don't see how switching to decorator 4.0.0 is a > problem, and I don't understand how OpenStack could be using 3.4.0 > which > is in Jessie (ie: 6 years old Debian release). > > PyPi says Decorator 3.4.0 is from 2012: > https://pypi.org/project/decorator/#history > > Do you have your release numbers correct? If so, then switching to > Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) > and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 > feels a bit crazy (and I wasn't aware of it). Thanks for the info. So from Debian perspective it is OK to bump the decorator version on stable. As others noted in this thread it seems to be more than just decorator that broke. :/ > >> Option 2: Pin the setuptools version during tox installation > > Please don't do this for the master branch, we need OpenStack to stay > current with setuptools (yeah, even if this means breaking > changes...). I've no intention to pin it on master. Master needs to work with the latest and greatest. Also on master it is easier to fix / replace the dependencies that become broken with new setuptools. > > For already released OpenStack: I don't mind much if this is done (I > could backport fixes if something breaks). ack > >> Option 3: turn off lower-constraints testing > > I already expressed myself about this: this is dangerous as distros > rely > on it for setting lower bounds as low as possible (which is always > preferred from a distro point of view). > >> Option 4: utilize pyproject.toml[6] to specify build-time >> requirements > > I don't know about pyproject.toml. > > Just my 2 cents, hoping it's useful, Thanks! Cheers, gibi > Cheers, > > Thomas Goirand (zigo) > From bkslash at poczta.onet.pl Mon Sep 27 13:14:15 2021 From: bkslash at poczta.onet.pl (Adam Tomas) Date: Mon, 27 Sep 2021 15:14:15 +0200 Subject: Problems with Fluentd buffer [fluentd][kolla-ansible] Message-ID: Hi, after some time of using newly deployed kolla-ansible I have fluentd errors: on all controllers, storage and network nodes: /var/log/kolla/fluentd/fluentd.log:2021-09-27 01:40:46 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 01:40:47.847564807 +0200 chunk="5ccee7b1762157f0ae10dcbeb4e14fd8" error_class=RestClient::GatewayTimeout error="504 Gateway Timeout" /var/log/kolla/fluentd/fluentd.log:2021-09-27 04:49:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout? on compute nodes: /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-statsd.log" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"openstack_python", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"openstack_python\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"kolla.var.log.kolla.monasca.agent-statsd.log", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"kolla.var.log.kolla.monasca.agent-statsd.log\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-forwarder.log" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data? and on monitoring node: 2021-09-27 14:40:50 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=84.05643947119825 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:42:53 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=122.65328024700284 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:44:15 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=82.32426812895574 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:46:05 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=# 2021-09-27 14:46:05 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:46:06.042150289 +0200 chunk="5ccf89c9ec54b09e616f7167d5f93cc1" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" 2021-09-27 14:46:05 +0200 [warn]: #0 suppressed same stacktrace 2021-09-27 14:47:58 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=112.90601075813174 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:47:58 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89c9ec54b09e616f7167d5f93cc1" 2021-09-27 14:50:16 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=136.24769522389397 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:52:27 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=129.86474119895138 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:54:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=100.71324555086903 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:56:31 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=# 2021-09-27 14:56:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:56:32.140741991 +0200 chunk="5ccf89e1feb471b7ce25a26d37977baa" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" 2021-09-27 14:56:31 +0200 [warn]: #0 suppressed same stacktrace 2021-09-27 14:58:03 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=90.91785193886608 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:58:03 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89e1feb471b7ce25a26d37977baa" 2021-09-27 15:00:01 +0200 [info]: #0 detected rotation of /var/log/kolla/kafka/server.log; waiting 5 seconds 2021-09-27 15:00:01 +0200 [info]: #0 following tail of /var/log/kolla/kafka/server.log 2021-09-27 15:00:10 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=127.54797655600123 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:02:29 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=137.74220423400402 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:03:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=38.81670297612436 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:03:13 +0200 [info]: #0 detected rotation of /var/log/kolla/kafka/controller.log; waiting 5 seconds 2021-09-27 15:03:13 +0200 [info]: #0 following tail of /var/log/kolla/kafka/controller.log What seems to be the problem? Is there any way to run more than one fluentd worker in kolla? Best regards Adam Tomas From balazs.gibizer at est.tech Mon Sep 27 13:23:04 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Mon, 27 Sep 2021 15:23:04 +0200 Subject: [neutron] upgrading port binding:profile.allocation value for port-resource-request-groups API extension In-Reply-To: References: <0U8UZQ.B1WQBDR9U22T1@est.tech> <8MHUZQ.NIR8KHQMZ56L2@est.tech> Message-ID: On Thu, Sep 23 2021 at 05:33:11 PM +0200, Rodolfo Alonso Hernandez wrote: > Hello Balazs: > > You are right: a DB migration is not necessary but a DB sanitization. > I did something similar in > https://review.opendev.org/c/openstack/neutron/+/789831. This patch > includes the script to, in one shot, modify the whole DB and an > upgrade check to test it. > > About the code, if something like the script provided and the upgrade > check is included, that will ensure the DB is correctly migrated in Y > release. That means the code supporting both formats could be removed > in Z. Thank you Rodolfo, I think we are on the same page now. cheers, gibi > > Regards. > > > On Wed, Sep 22, 2021 at 6:54 PM Balazs Gibizer > wrote: >> >> >> On Wed, Sep 22 2021 at 05:09:17 PM +0200, Rodolfo Alonso Hernandez >> wrote: >> > Hello Balazs: >> > >> > About the group_id, I see this is built using the port_id and the >> > qos_rules. We have all of this in the DB and we can build it >> > statically (I think so, maybe I'm very optimistic). >> >> Yes, that is probably doable. But I let Przemek work out the >> details in >> the patch. >> > >> > About the code, that was something I was thinking about after >> sending >> > the last mail. For at least two releases, we need to support both >> RP >> > formats in the DB. If we read only the UUID (old format), then we >> > should convert it and store it in the new format. >> > >> > About the migration, we don't support contract migrations anymore. >> > But this is not true as we have done some migrations that have >> added >> > new restrictions in the DB schema. In any case, this could be >> done as >> > an expansion migration. If the code is in place, I don't see any >> > problem of doing this migration with the server running. Each >> > "ml2_port_bindings" register will be updated atomically, while the >> > Neutron server will be able to handle both versions. >> > >> >> If I understand correctly what you described is an online data >> migration where >> 1) neutron does not even need an expand migration as no new field is >> needed in the database as we use the old field both for the old and >> the >> new data >> 2) neutron server converts between data formats when the data is >> read >> 3) neutron can drop the conversion code only after every register is >> upgraded this way. As there could be ports that are not touched >> between >> upgrades we cannot simply say that we are done with the migration >> after >> waiting a release cycle. I think we have to add an upgrade check in >> Yoga that warns if there are still ports with the old format. And >> also >> neutron needs to provide a tool for the deployer to trigger the >> conversion of those remaining port before the upgrade to X. >> >> Do I understand your suggestion correct? Do you agree with the above >> solution proposal? >> >> Cheers, >> gibi >> >> > Regards. >> > >> > >> > On Wed, Sep 22, 2021 at 3:44 PM Balazs Gibizer >> > wrote: >> >> >> >> >> >> On Tue, Sep 21 2021 at 06:30:46 PM +0200, Rodolfo Alonso >> Hernandez >> >> wrote: >> >> > Hello Balazs: >> >> > >> >> > Sorry for the late reply, I was on PTO. >> >> > >> >> > If I'm not wrong, now port['binding:profile']['allocation'] >> is a >> >> UUID >> >> > and you need it to be a list of UUIDs. Am I correct? >> >> >> >> It is a bit more complicated than that. The old value is a >> single RP >> >> UUID. the new value is a dict where the key is the group_id and >> the >> >> value is the RP UUID fulfilling that group. So the >> transformation >> >> needs >> >> to access to the group_id. >> >> The group_id is a stable UUID generated by neutron server as >> part of >> >> the port.resource_request value, but it is not persisted. >> >> >> >> > >> >> > To make this change in the DB you should use the Alembic >> >> migrations, >> >> > as you said. That should ensure all registers are translated. >> We >> >> > should also include a sanity check to ensure the DB migration >> was >> >> > done correctly. >> >> >> >> I'm not 100% sure but I think such data migration can only be >> done >> >> in >> >> the contract part as it needs to be done while the neutron >> server is >> >> down as the old code can only use the old data format while the >> new >> >> code can only use the new format. Is it OK to introduce a new >> >> contract >> >> migration in Yoga in neutron? >> >> >> >> Cheers, >> >> gibi >> >> >> >> >> >> > >> >> > Is that what you needed? Don't hesitate to ping me in IRC if >> >> needed. >> >> > >> >> > Regards. >> >> > >> >> > On Fri, Sep 10, 2021 at 6:06 PM Balazs Gibizer >> >> > wrote: >> >> >> Hi Neutrinos! >> >> >> >> >> >> We found a technical challenge during implementing the >> >> >> port-resource-request-groups API extension[1]. That >> extension >> >> >> changes >> >> >> the format of the port.resoruce_request as well as the >> format >> >> of the >> >> >> port.binding:profile.allocation. The former is a calculated >> >> field on >> >> >> the port so that is easy. However the bindig:profile is >> >> persisted in >> >> >> the database so data migration is needed. What is the >> canonical >> >> way >> >> >> to >> >> >> do such DB data translation in Neutron? Can we translate the >> >> data in >> >> >> place during alembic migration? Or should we do some kind of >> >> online >> >> >> data migration when the data is translated by neutron when >> it is >> >> >> read >> >> >> from the db? >> >> >> >> >> >> cheers, >> >> >> gibi >> >> >> >> >> >> [1] >> https://review.opendev.org/c/openstack/neutron/+/805637/5 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> From tonykarera at gmail.com Mon Sep 27 13:58:29 2021 From: tonykarera at gmail.com (Karera Tony) Date: Mon, 27 Sep 2021 15:58:29 +0200 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> Message-ID: Hello Laurent, Thanks for the advice. I got a solution from the IRC chat group. Apparently, The ceph_nova_user defaults to the ceph_cinder_username. So even if you used nova at the ceph side, You have to uncomment the ceph_nova_user: "nova" in global.yml since it will default to cinder if not uncommented. It was merged recently Regards Tony Karera On Mon, Sep 27, 2021 at 2:45 PM Karera Tony wrote: > I tried it but no one was replying to the chats > > Regards > > Tony Karera > > > > > On Mon, Sep 27, 2021 at 7:57 AM Micha? Nasiadka > wrote: > >> Hi Tony, >> >> As Laurent mentioned - it would be best if you would reach out on >> #openstack-kolla IRC channel - and we?ll try to do our best to help you. >> >> Here?s an IRC guide from Contributors guide - if you?re not familiar: >> https://docs.openstack.org/contributors/common/irc.html >> >> Regards, >> Michal Nasiadka >> >> W dniu pon., 27.09.2021 o 07:52 Karera Tony >> napisa?(a): >> >>> Hello Team, >>> >>> I even tried to manually put the ceph.client.cinder.keyring in the >>> nova_compute container but the issue persisted. >>> >>> I also tried reinstalling Openstack on another Environment but I still >>> have the same issue. >>> >>> Anyone with any idea on how to proceed ? >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont >>> wrote: >>> >>>> I know that there is some Kolla folks around but keep in mind that this >>>> is a volunteer based list :) >>>> >>>> I think you might get a bit more one to one help on IRC in their kolla >>>> channel. >>>> >>>> On Fri, Sep 24, 2021 at 5:10 PM Karera Tony >>>> wrote: >>>> >>>>> I would really appreciate any support on this >>>>> >>>>> On Fri, 24 Sep 2021, 11:13 Karera Tony, wrote: >>>>> >>>>>> Hello Team, >>>>>> >>>>>> I don't know if there has been any change in the packages but the way >>>>>> I am deploying is the same way I have been deploying. >>>>>> >>>>>> I don't understand why there is a certain issue now. >>>>>> Regards >>>>>> >>>>>> Tony Karera >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Sep 24, 2021 at 7:30 AM Karera Tony >>>>>> wrote: >>>>>> >>>>>>> Hello Laurent, >>>>>>> >>>>>>> It turns out I only have one keyring in the container. >>>>>>> >>>>>>> root at compute1:/home/stack# docker exec -it nova_compute bash >>>>>>> (nova-compute)[nova at compute1 /]$ cd /etc/ceph/ >>>>>>> (nova-compute)[nova at compute1 ceph]$ ls >>>>>>> ceph.client.nova.keyring ceph.conf rbdmap >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> Tony Karera >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < >>>>>>> laurentfdumont at gmail.com> wrote: >>>>>>> >>>>>>>> I do believe Kolla runs a container version of each service on >>>>>>>> computes. Are you looking inside the nova-compute container ( >>>>>>>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >>>>>>>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >>>>>>>> >>>>>>>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hello Sean, >>>>>>>>> >>>>>>>>> Below are the output on the compute node and deployment >>>>>>>>> >>>>>>>>> root at compute1:/etc/kolla/nova-compute# ls >>>>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>>>> config.json nova.conf >>>>>>>>> >>>>>>>>> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ >>>>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>>>> >>>>>>>>> And I can confirm that the content is the same. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> >>>>>>>>> Tony Karera >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>>>>>>>> > I would investigate that compute error first. Creating volumes >>>>>>>>>> means the >>>>>>>>>> > controllers are doing the action. Starting a VM on a compute >>>>>>>>>> means you also >>>>>>>>>> > need Ceph to works on the compute to mount the rdb target. >>>>>>>>>> >>>>>>>>>> nova as part of its startup process in aintiallying the resouce >>>>>>>>>> tracker will >>>>>>>>>> try to connect to ceph if you are using the rbd image backend to >>>>>>>>>> report how much stroage >>>>>>>>>> is avaiable. if the keyring does not work on the vms pool as the >>>>>>>>>> user nova is connecting as >>>>>>>>>> then that will block the agent from starting up fully and will >>>>>>>>>> cause it to be missing the hypervior list. >>>>>>>>>> >>>>>>>>>> the error seams to indicate that the cinder keyring is not in the >>>>>>>>>> nova container >>>>>>>>>> that likely means you have not put it in /etc/kolla/config/nova >>>>>>>>>> i woudl check /etc/kolla/config/nova on the deployment host and >>>>>>>>>> sudo ls /etc/kolla/nova-compute/ >>>>>>>>>> on the compute node to ensure the cinder keyring is actully >>>>>>>>>> copied and has the correct content >>>>>>>>>> >>>>>>>>>> i have >>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>>>>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>>>>> config.json nova.conf >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [client.cinder] >>>>>>>>>> key = ********************************* >>>>>>>>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>>>>>>>> pool=vms" >>>>>>>>>> caps mon = "profile rbd" >>>>>>>>>> caps osd = "profile rbd pool=volumes, profile rbd >>>>>>>>>> pool=vms, profile rbd pool=images" >>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>>>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>>>>>>>> [client.nova] >>>>>>>>>> key = ********************************* >>>>>>>>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>>>>>>>> pool=vms" >>>>>>>>>> caps mon = "profile rbd" >>>>>>>>>> caps osd = "profile rbd pool=volumes, profile rbd >>>>>>>>>> pool=vms, profile rbd pool=images" >>>>>>>>>> >>>>>>>>>> blanked out the key wiht *************** after the fact but you >>>>>>>>>> should have something similar >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> in my case i decied to use a seperate key for nova rbd backend >>>>>>>>>> because i was also using EC poosl with a seperate data and metadata pool >>>>>>>>>> so i neede to modify my ceph.conf to make that work with kolla >>>>>>>>>> >>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>>>>>> /etc/kolla/nova-compute/ceph.conf >>>>>>>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>>>>>>>> [global] >>>>>>>>>> fsid = ********************* >>>>>>>>>> mon_host = [*********************] >>>>>>>>>> >>>>>>>>>> [client.glance] >>>>>>>>>> rbd default data pool = images-data >>>>>>>>>> >>>>>>>>>> [client.cinder] >>>>>>>>>> rbd default data pool = volumes-data >>>>>>>>>> >>>>>>>>>> [client.nova] >>>>>>>>>> rbd default data pool = vms-data >>>>>>>>>> >>>>>>>>>> using 2 keyrings/user allows me to set different default data >>>>>>>>>> pools for cinder and nova. >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> > Working in Wallaby with the error doesn't mean it would 100% >>>>>>>>>> work in >>>>>>>>>> > Victoria. >>>>>>>>>> > >>>>>>>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony < >>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>> > >>>>>>>>>> > > Hey Guys, Any other idea ? >>>>>>>>>> > > >>>>>>>>>> > > Regards >>>>>>>>>> > > >>>>>>>>>> > > Tony Karera >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>> > > >>>>>>>>>> > > > Just to add on that, >>>>>>>>>> > > > >>>>>>>>>> > > > compute service is listed, I can create Volumes, I have the >>>>>>>>>> same cinder >>>>>>>>>> > > > keyring in the /etc/kolla/config/nova directory as I have >>>>>>>>>> in the >>>>>>>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along with >>>>>>>>>> the nova keyring >>>>>>>>>> > > > Regards >>>>>>>>>> > > > >>>>>>>>>> > > > Tony Karera >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>> > > > >>>>>>>>>> > > > > Hello Guys, >>>>>>>>>> > > > > >>>>>>>>>> > > > > Thanks a lot. >>>>>>>>>> > > > > >>>>>>>>>> > > > > I had actually checked the nova -compute.log on the >>>>>>>>>> compute node and >>>>>>>>>> > > > > they were showing the error I will post at the end about >>>>>>>>>> the cinder keyring >>>>>>>>>> > > > > but I know its correct because its the same I was using >>>>>>>>>> on wallaby, I even >>>>>>>>>> > > > > tried to use another ceph cluster with ofcouse different >>>>>>>>>> keyrings but its >>>>>>>>>> > > > > the same issue. >>>>>>>>>> > > > > >>>>>>>>>> > > > > Below is the error >>>>>>>>>> > > > > >>>>>>>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>>>>>>> auth: unable to >>>>>>>>>> > > > > find a keyring on >>>>>>>>>> > > > > >>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>>>> > > > > (2) No such file or >>>>>>>>>> directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>>>>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>>>>>>>> > > > > >>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>>>>>>>> 7fbce2f4f700 -1 auth: unable >>>>>>>>>> > > > > to find a keyring on >>>>>>>>>> > > > > >>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>>>> > > > > (2) No such file or >>>>>>>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>>>>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>>>>>>>> > > > > >>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>>>>>>>> 7fbce2f4f700 -1 auth: unable >>>>>>>>>> > > > > to find a keyring on >>>>>>>>>> > > > > >>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>>>> > > > > (2) No such file or >>>>>>>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>>>>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>>>>>>>> > > > > >>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>>>>>>>> connecting to the >>>>>>>>>> > > > > cluster)\n' >>>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>>>>>>> During handling of >>>>>>>>>> > > > > the above exception, another exception occurred: >>>>>>>>>> > > > > Regards >>>>>>>>>> > > > > >>>>>>>>>> > > > > Tony Karera >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>>>>>>>>> smooney at redhat.com> wrote: >>>>>>>>>> > > > > >>>>>>>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>>>>>>>>> > > > > > > It could also be a compute cell discovery issue maybe? >>>>>>>>>> > > > > > no they shoudl still show up in the hypervior list api >>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > Do you see anything under "openstack compute service >>>>>>>>>> list"? >>>>>>>>>> > > > > > if they show up in the service list but not they >>>>>>>>>> hyperiors api it >>>>>>>>>> > > > > > means that the comptue service started and registered >>>>>>>>>> its service entry >>>>>>>>>> > > > > > but >>>>>>>>>> > > > > > something broke it before it could create a compute >>>>>>>>>> node recored in the >>>>>>>>>> > > > > > db. >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > with ceph the case i have hit this most often is when >>>>>>>>>> the keyright used >>>>>>>>>> > > > > > by nova to >>>>>>>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>>>>>>>>> whihc prevent >>>>>>>>>> > > > > > the resoucetack and compute manager >>>>>>>>>> > > > > > form actully creating the compute node record. >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > it can happen for other reason too but best place to >>>>>>>>>> start is check if >>>>>>>>>> > > > > > there is an error in the nova compute agent log and go >>>>>>>>>> from there. >>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>>>>>>>> smooney at redhat.com> >>>>>>>>>> > > > > > wrote: >>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony >>>>>>>>>> wrote: >>>>>>>>>> > > > > > > > > Hello Team, >>>>>>>>>> > > > > > > > > >>>>>>>>>> > > > > > > > > I have deployed Openstack Victoria using >>>>>>>>>> Kolla-ansible on Ubuntu >>>>>>>>>> > > > > > 20.04 >>>>>>>>>> > > > > > > > and >>>>>>>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>>>>>>>>> Glance. >>>>>>>>>> > > > > > > > > >>>>>>>>>> > > > > > > > > It finished with no error but it has failed to >>>>>>>>>> register any on the >>>>>>>>>> > > > > > > > Compute >>>>>>>>>> > > > > > > > > Nodes under Hypervisors. >>>>>>>>>> > > > > > > > > >>>>>>>>>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack >>>>>>>>>> hypervisor list >>>>>>>>>> > > > > > > > > >>>>>>>>>> > > > > > > > > (kolla-openstack) stack at deployment:~$ >>>>>>>>>> > > > > > > > > >>>>>>>>>> > > > > > > > > >>>>>>>>>> > > > > > > > > Any idea on how to resolve this ? >>>>>>>>>> > > > > > > > that usually means that somehthing prevented the >>>>>>>>>> comptue agent form >>>>>>>>>> > > > > > > > strating properly >>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > for example incorrect ceph keyrings there are >>>>>>>>>> several other case >>>>>>>>>> > > > > > but you >>>>>>>>>> > > > > > > > mentioned you are >>>>>>>>>> > > > > > > > using ceph. >>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > if this is hte case you should see error in the >>>>>>>>>> compute agent log. >>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > > >>>>>>>>>> > > > > > > > > Regards >>>>>>>>>> > > > > > > > > >>>>>>>>>> > > > > > > > > Tony Karera >>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >> Micha? Nasiadka >> mnasiadka at gmail.com >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moshele at nvidia.com Mon Sep 27 06:35:44 2021 From: moshele at nvidia.com (Moshe Levi) Date: Mon, 27 Sep 2021 06:35:44 +0000 Subject: [neutron][ovn] support for stateless NAT for floating ip in ml2 ovn In-Reply-To: References: <20210817002002.252368-1-ihrachys@redhat.com> <9096201007244d0732f019d1452575e61080e09c.camel@redhat.com> <3b84603ca14830e712154091692f05538fdcacef.camel@redhat.com> Message-ID: Thank Ihar ? -----Original Message----- From: Ihar Hrachyshka Sent: Saturday, September 25, 2021 12:26 AM To: Sean Mooney Cc: Moshe Levi ; openstack-discuss at lists.openstack.org Subject: Re: [neutron][ovn] support for stateless NAT for floating ip in ml2 ovn External email: Use caution opening links or attachments On 8/18/21 11:35 AM, Ihar Hrachyshka wrote: > Not too hard to fallback; but on this note, do we maintain minimal OVN > version anywhere in neutron? I remember when I was adding support for > allow-stateless ACLs, I was told we don't track it (hence runtime > schema inspection in > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Frevi > ew.opendev.org%2Fc%2Fopenstack%2Fneutron%2F%2B%2F789974&data=04%7C > 01%7Cmoshele%40nvidia.com%7Ccb32ecd84e314d168dd208d97fa1eff7%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637681155759121363%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=OhrwxnPs8xcsB4b3PyautFbg3dfPJp6pblziGpRvoak%3D&reserved=0) Considering potential backports in downstream products, perhaps a runtime schema check is a better approach anyway. To close the loop, the patch that uses stateless dnat_and_snat has merged in master. Including migration path for existing NAT objects in nbdb and runtime check for core OVN support. Probably not a backport material for upstream. Ihar From anyrude10 at gmail.com Mon Sep 27 12:04:04 2021 From: anyrude10 at gmail.com (Anirudh Gupta) Date: Mon, 27 Sep 2021 17:34:04 +0530 Subject: [Tripleo] Support of Alma/Rocky Linux Message-ID: Hi Team As per the document, the supporting OS mentioned in the documentation is only Centos Stream and No Stream version (as of now) Since Centos has officially stopped releasing new versions and looking for EOL. I Just wanted to check if there is a possibility that we can install Tripleo on a specific version of Alma Linux or Rocky Linux Regards Anirudh Gupta -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Mon Sep 27 17:08:17 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 27 Sep 2021 17:08:17 +0000 Subject: [OSSA-2021-002] Nova: Open Redirect in noVNC proxy (CVE-2021-3654) Message-ID: <20210927170817.z5dshda7fctgpjtw@yuggoth.org> =========================================== OSSA-2021-002: Open Redirect in noVNC proxy =========================================== :Date: July 29, 2021 :CVE: CVE-2021-3654 Affects ~~~~~~~ - Nova: <21.2.3, >=22.0.0 <22.2.3, >=23.0.0 <23.0.3 Description ~~~~~~~~~~~ Swe Aung, Shahaan Ayyub, and Salman Khan with the Monash University Cyber Security team reported a vulnerability affecting Nova's noVNC proxying implementation which exposed access to a well-known redirect behavior in the Python standard library's http.server.SimpleHTTPRequestHandler and thus noVNC's WebSockifyRequestHandler which uses it. By convincing a user to follow a specially-crafted novncproxy URL, the user could be redirected to an unrelated site under control of the attacker in an attempt to convince them to divulge credentials or other sensitive data. All Nova deployments with novncproxy enabled are affected. Errata ~~~~~~ The initial fix did not take into account the possibility of bypass using exactly three slashes. This update provides a more thorough revised fix for the issue. The affected versions list has been updated to indicate versions expected to include the newer solution. Patches ~~~~~~~ - https://review.opendev.org/791807 (Train) - https://review.opendev.org/806629 (errata 1) (Train) - https://review.opendev.org/791806 (Ussuri) - https://review.opendev.org/806628 (errata 1) (Ussuri) - https://review.opendev.org/791805 (Victoria) - https://review.opendev.org/806626 (errata 1) (Victoria) - https://review.opendev.org/791577 (Wallaby) - https://review.opendev.org/805818 (errata 1) (Wallaby) - https://review.opendev.org/791297 (Xena) - https://review.opendev.org/805654 (errata 1) (Xena) Credits ~~~~~~~ - Swe Aung from Monash University Cyber Security team (CVE-2021-3654) - Shahaan Ayyub from Monash University Cyber Security team (CVE-2021-3654) - Salman Khan from Monash University Cyber Security team (CVE-2021-3654) References ~~~~~~~~~~ - https://launchpad.net/bugs/1927677 - https://bugs.python.org/issue32084 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3654 Notes ~~~~~ - The stable/train branch is under extended maintenance and will receive no new point releases, but a patch for it is provided as a courtesy. OSSA History ~~~~~~~~~~~~ - 2021-09-27 - Errata 1 - 2021-07-29 - Original Version -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From gmann at ghanshyammann.com Mon Sep 27 17:42:57 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Mon, 27 Sep 2021 12:42:57 -0500 Subject: [all][tc] Technical Committee next weekly meeting on Sept 30th at 1500 UTC Message-ID: <17c285a45d8.1251b48bd112192.5860450617549878983@ghanshyammann.com> Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 30th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 29th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann From elod.illes at est.tech Mon Sep 27 18:48:36 2021 From: elod.illes at est.tech (=?UTF-8?B?RWzFkWQgSWxsw6lz?=) Date: Mon, 27 Sep 2021 20:48:36 +0200 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> Message-ID: <827e99c6-99b2-54c8-a627-5153e3b84e6b@est.tech> Hi again, as I see there is no objection yet about using gibi's solution [1] (as I already summarized the situation in my previous mail [2]) for a fix for similar cases, so with a general stable core hat on, I *suggest* everyone to use that solution to pin the setuptools in tox for every failing cases (so that to avoid similar future errors as well). [1] https://review.opendev.org/810461 [2] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025059.html El?d On 2021. 09. 27. 14:47, Balazs Gibizer wrote: > > > On Fri, Sep 24 2021 at 10:21:33 PM +0200, Thomas Goirand > wrote: >> Hi Gibi! >> >> Thanks for bringing this up. >> >> As a distro package maintainer, here's my view. >> >> On 9/22/21 2:11 PM, Balazs Gibizer wrote: >>> ?Option 1: Bump the major version of the decorator dependency on >>> stable. >> >> Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable), for >> which I don't even maintain OpenStack anymore (that's OpenStack >> Newton...). So I don't see how switching to decorator 4.0.0 is a >> problem, and I don't understand how OpenStack could be using 3.4.0 which >> is in Jessie (ie: 6 years old Debian release). >> >> PyPi says Decorator 3.4.0 is from 2012: >> https://pypi.org/project/decorator/#history >> >> Do you have your release numbers correct? If so, then switching to >> Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) >> and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 >> feels a bit crazy (and I wasn't aware of it). > > Thanks for the info. So from Debian perspective it is OK to bump the > decorator version on stable. As others noted in this thread it seems > to be more than just decorator that broke. :/ > >> >>> ?Option 2: Pin the setuptools version during tox installation >> >> Please don't do this for the master branch, we need OpenStack to stay >> current with setuptools (yeah, even if this means breaking changes...). > > I've no intention to pin it on master. Master needs to work with the > latest and greatest. Also on master it is easier to fix / replace the > dependencies that become broken with new setuptools. > >> >> For already released OpenStack: I don't mind much if this is done (I >> could backport fixes if something breaks). > > ack > >> >>> ?Option 3: turn off lower-constraints testing >> >> I already expressed myself about this: this is dangerous as distros rely >> on it for setting lower bounds as low as possible (which is always >> preferred from a distro point of view). >> >>> ?Option 4: utilize pyproject.toml[6] to specify build-time requirements >> >> I don't know about pyproject.toml. >> >> Just my 2 cents, hoping it's useful, > > Thanks! > > Cheers, > gibi > >> Cheers, >> >> Thomas Goirand (zigo) >> > > > From mkopec at redhat.com Mon Sep 27 19:13:45 2021 From: mkopec at redhat.com (Martin Kopec) Date: Mon, 27 Sep 2021 21:13:45 +0200 Subject: [qa] Office Hour Sep 28th is cancelled Message-ID: Hi everyone, the next QA Office Hour on September 28th at 2pm UTC is cancelled due to a public holiday in my country and I won't be able to moderate it. Thank you for your understanding, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosmaita.fossdev at gmail.com Mon Sep 27 20:38:44 2021 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Mon, 27 Sep 2021 16:38:44 -0400 Subject: [cinder] reminder: this week's meeting in video+IRC Message-ID: <6d2613e2-a1d6-ca60-5a5d-05832ad7f38f@gmail.com> Quick reminder that this week's Cinder team meeting on Wednesday 29 September, being the final meeting of the month, will be held in both videoconference and IRC at the regularly scheduled time of 1400 UTC. These are the video meeting rules we've agreed to: * Everyone will keep IRC open during the meeting. * We'll take notes in IRC to leave a record similar to what we have for our regular IRC meetings. * Some people are more comfortable communicating in written English. So at any point, any attendee may request that the discussion of the current topic be conducted entirely in IRC. * The meeting will be recorded. connection info: https://bluejeans.com/3228528973 meeting agenda: https://etherpad.opendev.org/p/cinder-xena-meetings cheers, brian From aschultz at redhat.com Mon Sep 27 22:06:31 2021 From: aschultz at redhat.com (Alex Schultz) Date: Mon, 27 Sep 2021 16:06:31 -0600 Subject: [Tripleo] Support of Alma/Rocky Linux In-Reply-To: References: Message-ID: On Mon, Sep 27, 2021 at 8:42 AM Anirudh Gupta wrote: > Hi Team > > As per the document, the supporting OS mentioned in the documentation is > only Centos Stream and No Stream version (as of now) > > Since Centos has officially stopped releasing new versions and looking for > EOL. > I Just wanted to check if there is a possibility that we can install > Tripleo on a specific version of Alma Linux or Rocky Linux > > We don't plan on rebuilding for this. It may still work but YMMV. There was a previous thread about this back in May. http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022297.html > Regards > Anirudh Gupta > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Mon Sep 27 22:20:31 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Mon, 27 Sep 2021 17:20:31 -0500 Subject: [all] Gate status: All devstack based jobs are failing for placement url issue in devstack Message-ID: <17c29586680.120e3c448119950.8945886018923104354@ghanshyammann.com> Hello Everyone, All the devstack based jobs are failing with placement URL issue in devstack. The root cause is not known yet. You will see the below error in devstack/summary log- + ./stack.sh:main:1455 : /usr/local/bin/nova-status --config-file /etc/nova/nova.conf upgrade check Placement logs: INFO placement.requestlog [None req-3ad7ab8b-7c85-477a-b156-f5ca652a38cc service placement] 10.208.226.48 "GET /placemen//" status: 404 len: 162 microversion: 1.0 Please do not recheck until the issue is fixed. -gmann From melwittt at gmail.com Tue Sep 28 01:24:18 2021 From: melwittt at gmail.com (melanie witt) Date: Mon, 27 Sep 2021 18:24:18 -0700 Subject: [all] Gate status: All devstack based jobs are failing for placement url issue in devstack In-Reply-To: <17c29586680.120e3c448119950.8945886018923104354@ghanshyammann.com> References: <17c29586680.120e3c448119950.8945886018923104354@ghanshyammann.com> Message-ID: On 9/27/21 15:20, Ghanshyam Mann wrote: > Hello Everyone, > > All the devstack based jobs are failing with placement URL issue in devstack. The root cause is not known yet. > > You will see the below error in devstack/summary log- > > + ./stack.sh:main:1455 : /usr/local/bin/nova-status --config-file /etc/nova/nova.conf upgrade check > > Placement logs: > INFO placement.requestlog [None req-3ad7ab8b-7c85-477a-b156-f5ca652a38cc service placement] 10.208.226.48 "GET /placemen//" status: 404 len: 162 microversion: 1.0 I looked at this for awhile and found that placement is returning a 404 because it fails to match [1] any of the declared routes [2]. This issue reproduces on a local devstack. I added some debug print statements to determine this [3]. (The "ENVIRON" and "NO MATCH FOUND" are the print statements I added in PlacementHandler.__call__() and dispatch() in placement/handler.py). Example: def dispatch(environ, start_response, mapper): """Find a matching route for the current request. If no match is found, raise a 404 response. If there is a matching route, but no matching handler for the given method, raise a 405. """ result = mapper.match(environ=environ) if result is None: print('NO MATCH FOUND') raise webob.exc.HTTPNotFound( json_formatter=util.json_error_formatter) In short, the WSGI environment dict is giving a bad/not matching url in the PATH_INFO variable if the call path is not simply '/'. Example of a good PATH_INFO: ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '/', [...] 'REQUEST_URI': '/placement', 'SCRIPT_NAME': '/placement', [...]} INFO placement.requestlog [None req-bde0b07f-e6a4-4b3d-bb78-51bee95524e1 None None] 127.0.0.1 "GET /placement/" status: 200 len: 136 mi croversion: 1.0 Example of a bad PATH_INFO and SCRIPT_NAME: ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '//', [...] 'REQUEST_URI': '/placement/', 'SCRIPT_NAME': '/placemen', [...]} NO MATCH FOUND INFO placement.requestlog [None req-317708d5-8c6f-48f5-abce-225a242bb31a admin admin] 127.0.0.1 "GET /placemen//" status: 404 len: 198 microversion: 1.29 Another example of bad PATH_INFO and SCRIPT_NAME: ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '//resource_providers', [...] 'REQUEST_URI': '/placement/resource_providers', 'SCRIPT_NAME': '/placemen', [...]} Note that the REQUEST_URI in all cases looks correct. This supports what I observed that the clients are calling the correct urls. The same 404s happen using curl -X GET statements as well. So it's definitely something wrong in the server. I'm not sure how to debug further. I don't know a great deal about WSGI and where the contents of the environment dict come from. If anyone can help, it would be appreciated. Cheers, -melanie [1] https://github.com/openstack/placement/blob/bd5b19c00e1ab293fc157f4699bc4f4719731c25/placement/handler.py#L142 [2] https://github.com/openstack/placement/blob/bd5b19c00e1ab293fc157f4699bc4f4719731c25/placement/handler.py#L51 [3] https://paste.opendev.org/show/809636 From ildiko.vancsa at gmail.com Tue Sep 28 01:31:24 2021 From: ildiko.vancsa at gmail.com (Ildiko Vancsa) Date: Mon, 27 Sep 2021 18:31:24 -0700 Subject: Edge sessions at the upcoming PTG In-Reply-To: References: Message-ID: Hi, It is a friendly reminder to please check out the edge session the OpenInfra Edge Computing Group is planning for the PTG: https://superuser.openstack.org/articles/the-project-teams-gathering-is-coming-lets-talk-edge/ We have topics such as networking, APIs, and automation in edge infrastructures that are relevant for OpenStack as well and it would be great to have the community?s input on these. Our etherpad for the sessions: https://etherpad.opendev.org/p/ecg-ptg-october-2021 Please let me know if you have any questions about the agenda or topics. Thanks and Best Regards, Ildik? > On Sep 7, 2021, at 16:22, Ildiko Vancsa wrote: > > Hi, > > I?m reaching out to you to share the agenda of the OpenInfra Edge Computing Group that we put together for the upcoming PTG. I would like to invite everyone who is interested in discussing edge challenges and finding solutions! > > We summarized our plans for the event in a short blog post to give some context to each of the topic that we picked to discuss in details. We picked key areas like security, networking, automation and tools, containers and more: https://superuser.openstack.org/articles/the-project-teams-gathering-is-coming-lets-talk-edge/ > > Our etherpad for the sessions: https://etherpad.opendev.org/p/ecg-ptg-october-2021 > > Please let me know if you have any questions about the agenda or topics. > > Thanks and Best Regards, > Ildik? > > From gmann at ghanshyammann.com Tue Sep 28 04:30:54 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Mon, 27 Sep 2021 23:30:54 -0500 Subject: [all] Gate status: All devstack based jobs are failing for placement url issue in devstack In-Reply-To: References: <17c29586680.120e3c448119950.8945886018923104354@ghanshyammann.com> Message-ID: <17c2aab7e5c.e789de92121944.8148200687813866688@ghanshyammann.com> ---- On Mon, 27 Sep 2021 20:24:18 -0500 melanie witt wrote ---- > On 9/27/21 15:20, Ghanshyam Mann wrote: > > Hello Everyone, > > > > All the devstack based jobs are failing with placement URL issue in devstack. The root cause is not known yet. > > > > You will see the below error in devstack/summary log- > > > > + ./stack.sh:main:1455 : /usr/local/bin/nova-status --config-file /etc/nova/nova.conf upgrade check > > > > Placement logs: > > INFO placement.requestlog [None req-3ad7ab8b-7c85-477a-b156-f5ca652a38cc service placement] 10.208.226.48 "GET /placemen//" status: 404 len: 162 microversion: 1.0 > > I looked at this for awhile and found that placement is returning a 404 > because it fails to match [1] any of the declared routes [2]. This issue > reproduces on a local devstack. > > I added some debug print statements to determine this [3]. (The > "ENVIRON" and "NO MATCH FOUND" are the print statements I added in > PlacementHandler.__call__() and dispatch() in placement/handler.py). > > Example: > > def dispatch(environ, start_response, mapper): > """Find a matching route for the current request. > > If no match is found, raise a 404 response. > If there is a matching route, but no matching handler > for the given method, raise a 405. > """ > result = mapper.match(environ=environ) > if result is None: > print('NO MATCH FOUND') > raise webob.exc.HTTPNotFound( > json_formatter=util.json_error_formatter) > > In short, the WSGI environment dict is giving a bad/not matching url in > the PATH_INFO variable if the call path is not simply '/'. > > Example of a good PATH_INFO: > > ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '/', [...] > 'REQUEST_URI': '/placement', 'SCRIPT_NAME': '/placement', [...]} > INFO placement.requestlog [None req-bde0b07f-e6a4-4b3d-bb78-51bee95524e1 > None None] 127.0.0.1 "GET /placement/" status: 200 len: 136 mi > croversion: 1.0 > > Example of a bad PATH_INFO and SCRIPT_NAME: > > ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '//', [...] > 'REQUEST_URI': '/placement/', 'SCRIPT_NAME': '/placemen', [...]} > NO MATCH FOUND > INFO placement.requestlog [None req-317708d5-8c6f-48f5-abce-225a242bb31a > admin admin] 127.0.0.1 "GET /placemen//" status: 404 len: 198 > microversion: 1.29 > > Another example of bad PATH_INFO and SCRIPT_NAME: > > ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': > '//resource_providers', [...] 'REQUEST_URI': > '/placement/resource_providers', 'SCRIPT_NAME': '/placemen', [...]} > > Note that the REQUEST_URI in all cases looks correct. This supports what > I observed that the clients are calling the correct urls. The same 404s > happen using curl -X GET statements as well. So it's definitely > something wrong in the server. Yeah. To find out who is corrupting the PATH_INFO and SCRIPT_NAME, I tried to trace the environ (could not finish it) in webob layer and while building the Request in webob, environ variables are already corrupted. 'PATH_INFO': '//resource_providers' 'REQUEST_URI': '/placement/resource_providers?in_tree=baba07a8-3c17-474c-82e6-5d63308a1f9d', 'SCRIPT_NAME': '/placemen', Below log is from webob.des.py->__call__method - https://paste.opendev.org/show/809639/ There is no change in webob recently and as it is only failing in placement (nova, keystone request works fine) so webob or wsgi is less doubtful and it might be only in placement server-side in our Request/Response objects ? -gmann > > I'm not sure how to debug further. I don't know a great deal about WSGI > and where the contents of the environment dict come from. If anyone can > help, it would be appreciated. > > Cheers, > -melanie > > [1] > https://github.com/openstack/placement/blob/bd5b19c00e1ab293fc157f4699bc4f4719731c25/placement/handler.py#L142 > [2] > https://github.com/openstack/placement/blob/bd5b19c00e1ab293fc157f4699bc4f4719731c25/placement/handler.py#L51 > [3] https://paste.opendev.org/show/809636 > > > From iwienand at redhat.com Tue Sep 28 05:27:23 2021 From: iwienand at redhat.com (Ian Wienand) Date: Tue, 28 Sep 2021 15:27:23 +1000 Subject: [all] Gate status: All devstack based jobs are failing for placement url issue in devstack In-Reply-To: References: <17c29586680.120e3c448119950.8945886018923104354@ghanshyammann.com> Message-ID: On Mon, Sep 27, 2021 at 06:24:18PM -0700, melanie witt wrote: > In short, the WSGI environment dict is giving a bad/not matching url in the > PATH_INFO variable if the call path is not simply '/'. This definitely seems to be due to a recent apache ugprade in focal. If you pick out the prior versions $ wget http://launchpadlibrarian.net/547306461/apache2-utils_2.4.41-4ubuntu3.4_amd64.deb $ wget http://launchpadlibrarian.net/547306448/apache2-data_2.4.41-4ubuntu3.4_all.deb $ wget http://launchpadlibrarian.net/547306458/apache2-bin_2.4.41-4ubuntu3.4_amd64.deb $ wget http://launchpadlibrarian.net/547306455/apache2_2.4.41-4ubuntu3.4_amd64.deb $ dpkg -i apache2*.deb $ service apache2 restart it "just works". $ /usr/local/bin/nova-status --debug --config-file /etc/nova/nova.conf upgrade check +-------------------------------------------+ | Upgrade Check Results | +-------------------------------------------+ ... | Check: Placement API | | Result: Success | | Details: None | +-------------------------------------------+ The recent security update @ https://launchpad.net/ubuntu/focal/+source/apache2/+changelog does a lot of stuff. It is an open question if this is actually an Apache error, or the update has just triggered something in our uwsgi setup that is wrong but just happened to work previously... -i From melwittt at gmail.com Tue Sep 28 05:59:45 2021 From: melwittt at gmail.com (melanie witt) Date: Mon, 27 Sep 2021 22:59:45 -0700 Subject: [all] Gate status: All devstack based jobs are failing for placement url issue in devstack In-Reply-To: <17c2aab7e5c.e789de92121944.8148200687813866688@ghanshyammann.com> References: <17c29586680.120e3c448119950.8945886018923104354@ghanshyammann.com> <17c2aab7e5c.e789de92121944.8148200687813866688@ghanshyammann.com> Message-ID: On Mon Sep 27 2021 21:30:54 GMT-0700 (Pacific Daylight Time), Ghanshyam Mann wrote: > Yeah. To find out who is corrupting the PATH_INFO and SCRIPT_NAME, > I tried to trace the environ (could not finish it) in webob layer and while > building the Request in webob, environ variables are already corrupted. > > 'PATH_INFO': '//resource_providers' > 'REQUEST_URI': > '/placement/resource_providers?in_tree=baba07a8-3c17-474c-82e6-5d63308a1f9d', > 'SCRIPT_NAME': '/placemen', > > Below log is from webob.des.py->__call__method > - https://paste.opendev.org/show/809639/ > > There is no change in webob recently and as it is only failing in placement (nova, > keystone request works fine) so webob or wsgi is less doubtful and it might be only > in placement server-side in our Request/Response objects ? I looked at this some more and I think I found where the environ gets populated in wsgiref.simple_server [1][2][3][4] in the case of devstack. (pbr generates the WSGI run script which ends up at /usr/local/bin/placement-api in a devstack env and this script uses wsgiref.simple_server as the WSGI server). I can't tell whether there is anything that could now be wrong in [3] or [4] given the recent apache upgrade in focal that Ian found is connected to the change in behavior (thank you Ian!). Maybe someone else can spot something. -melanie [1] https://github.com/openstack/placement/blob/bd5b19c00e1ab293fc157f4699bc4f4719731c25/setup.cfg#L55 [2] https://github.com/openstack/pbr/blob/2297988a7f41b8cb9d03aaea7d5ad9ba4378e54c/pbr/packaging.py#L403 [3] https://github.com/openstack/pbr/blob/2297988a7f41b8cb9d03aaea7d5ad9ba4378e54c/pbr/packaging.py#L365 [4] https://github.com/python/cpython/blob/3.8/Lib/wsgiref/simple_server.py#L85 From bkslash at poczta.onet.pl Tue Sep 28 06:34:15 2021 From: bkslash at poczta.onet.pl (Adam Tomas) Date: Tue, 28 Sep 2021 08:34:15 +0200 Subject: key_repository does not exist or Keystone does not have sufficient permission to access it [kolla-ansible][keystone] Message-ID: Hi, some next problem with my kolla-ansible deployment. On controller nodes I have (several times per hour) error messages like below: ERROR keystone.common.fernet_utils [req-386e223e-5f08-4959-bd0b-3dbbfd3b534f 73c7a9a7175e46e9971181963267225b efa5d1ce80324476b4ce189686dda2d1 - default default] Either [None] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/ I?ve checked - such path doesn?t exist inside keystone containers. I found bug #1863643 (https://bugs.launchpad.net/kolla/+bug/1863643 ) but I don?t see any progress on it nor information that victoria is not affected (so I assume it?s the same bug and Victoria is affected?). What should be done to get rid of this errors? Best regards Adam Tomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From bkslash at poczta.onet.pl Tue Sep 28 06:49:56 2021 From: bkslash at poczta.onet.pl (Adam Tomas) Date: Tue, 28 Sep 2021 08:49:56 +0200 Subject: Problems with Fluentd buffer [fluentd][kolla-ansible] Message-ID: Hi, I?ve turned off ?central_logging? and now I have only several errors like below on all nodes (3-7 per day) 0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout? On compute node, which had errors like this : /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag=?openstack_python" I?ve restarted fluentd container on that node and now it?s not showing errors like this anymore. So what?s the problem? Performance of Monasca/Kibana/Elasticsearch? But the node (telemetry) on which this stack is running is not overloaded - 20-30% CPU (24 cores on that node) , 40% RAM (there is 64GB), a lot of space on SSD. Best regards Adam Tomas > Hi, > after some time of using newly deployed kolla-ansible I have fluentd errors: > > on all controllers, storage and network nodes: > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 01:40:46 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 01:40:47.847564807 +0200 chunk="5ccee7b1762157f0ae10dcbeb4e14fd8" error_class=RestClient::GatewayTimeout error="504 Gateway Timeout" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 04:49:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout? > > > on compute nodes: > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-statsd.log" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"openstack_python", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"openstack_python\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"kolla.var.log.kolla.monasca.agent-statsd.log", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"kolla.var.log.kolla.monasca.agent-statsd.log\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-forwarder.log" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data? > > and on monitoring node: > > 2021-09-27 14:40:50 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=84.05643947119825 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:42:53 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=122.65328024700284 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:44:15 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=82.32426812895574 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:46:05 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=# > 2021-09-27 14:46:05 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:46:06.042150289 +0200 chunk="5ccf89c9ec54b09e616f7167d5f93cc1" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" > 2021-09-27 14:46:05 +0200 [warn]: #0 suppressed same stacktrace > 2021-09-27 14:47:58 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=112.90601075813174 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:47:58 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89c9ec54b09e616f7167d5f93cc1" > 2021-09-27 14:50:16 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=136.24769522389397 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:52:27 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=129.86474119895138 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:54:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=100.71324555086903 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:56:31 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=# > > From frickler at offenerstapel.de Tue Sep 28 07:09:14 2021 From: frickler at offenerstapel.de (Jens Harbott) Date: Tue, 28 Sep 2021 09:09:14 +0200 Subject: [all] Gate status: All devstack based jobs are failing for placement url issue in devstack In-Reply-To: References: <17c29586680.120e3c448119950.8945886018923104354@ghanshyammann.com> Message-ID: <5da7386f-c0ae-1358-c6cf-c9f0558de400@offenerstapel.de> On 9/28/21 7:27 AM, Ian Wienand wrote: ... > It is an open question if this is actually an Apache error, or the > update has just triggered something in our uwsgi setup that is wrong > but just happened to work previously... Actually mostly the latter I would say. See my proposed fix/workaround [0], the bug report I opened [1] and the apache2 docs [2]. The patch that changed apache2 behaviour is [3]. [0] https://review.opendev.org/c/openstack/devstack/+/811303 [1] https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1945274 [2] https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxypass [3] https://git.launchpad.net/ubuntu/+source/apache2/tree/debian/patches/CVE-2021-36160.patch?h=applied/ubuntu/focal-security From iwienand at redhat.com Tue Sep 28 07:19:29 2021 From: iwienand at redhat.com (Ian Wienand) Date: Tue, 28 Sep 2021 17:19:29 +1000 Subject: [all] Gate status: All devstack based jobs are failing for placement url issue in devstack In-Reply-To: References: <17c29586680.120e3c448119950.8945886018923104354@ghanshyammann.com> Message-ID: On Tue, Sep 28, 2021 at 03:27:23PM +1000, Ian Wienand wrote: > The recent security update @ > > https://launchpad.net/ubuntu/focal/+source/apache2/+changelog > > does a lot of stuff. frickler was actually onto this before me; see [1]. In short, it looks like we have found an issue with our ProxyPass config and the way it uses trailing slashes. See [2] We can either do ProxyPass "/placement/" "unix:/var/run/uwsgi/placement-api.socket|uwsgi://uwsgi-uds-placement-api/" retry=0 (note "placement/" and then "uwsgi://uwsgi-uds-placement-api/") or ProxyPass "/placement" "unix:/var/run/uwsgi/placement-api.socket|uwsgi://uwsgi-uds-placement-api" retry=0 both appear to work. I'm not sure which is better? We could enforce either in the function. As of right now 811303 is still testing, but it's looking good. -i [1] https://meetings.opendev.org/irclogs/%23openstack-qa/%23openstack-qa.2021-09-28.log.html#t2021-09-28T03:54:49 [2] https://review.opendev.org/c/openstack/devstack/+/811303 From balazs.gibizer at est.tech Tue Sep 28 07:28:26 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Tue, 28 Sep 2021 09:28:26 +0200 Subject: [all] Gate status: All devstack based jobs are failing for placement url issue in devstack In-Reply-To: References: <17c29586680.120e3c448119950.8945886018923104354@ghanshyammann.com> Message-ID: On Tue, Sep 28 2021 at 05:19:29 PM +1000, Ian Wienand wrote: > On Tue, Sep 28, 2021 at 03:27:23PM +1000, Ian Wienand wrote: >> The recent security update @ >> >> https://launchpad.net/ubuntu/focal/+source/apache2/+changelog >> >> does a lot of stuff. > > frickler was actually onto this before me; see [1]. > > In short, it looks like we have found an issue with our ProxyPass > config and the way it uses trailing slashes. See [2] > > We can either do > > ProxyPass "/placement/" > "unix:/var/run/uwsgi/placement-api.socket|uwsgi://uwsgi-uds-placement-api/" > retry=0 > > (note "placement/" and then "uwsgi://uwsgi-uds-placement-api/") or > > ProxyPass "/placement" > "unix:/var/run/uwsgi/placement-api.socket|uwsgi://uwsgi-uds-placement-api" > retry=0 > > both appear to work. I'm not sure which is better? We could enforce > either in the function. As of right now 811303 is still testing, but > it's looking good. Thank you all for troubleshooting and finding the solution. I think we need to use the '/placement' variant to allow GET http:///placement to be routed to the placement app. As far as I see if we use '/placement/' variant then /placement is not routed and that would be a breaking API change. Cheers, gibi > > -i > > [1] > https://meetings.opendev.org/irclogs/%23openstack-qa/%23openstack-qa.2021-09-28.log.html#t2021-09-28T03:54:49 > [2] https://review.opendev.org/c/openstack/devstack/+/811303 > > From mark at stackhpc.com Tue Sep 28 08:02:37 2021 From: mark at stackhpc.com (Mark Goddard) Date: Tue, 28 Sep 2021 09:02:37 +0100 Subject: Problems with Fluentd buffer [fluentd][kolla-ansible] In-Reply-To: References: Message-ID: On Tue, 28 Sept 2021 at 07:54, Adam Tomas wrote: > > Hi, > > I?ve turned off ?central_logging? and now I have only several errors like below on all nodes (3-7 per day) > > 0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout? > > > On compute node, which had errors like this : > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag=?openstack_python" > > I?ve restarted fluentd container on that node and now it?s not showing errors like this anymore. > > So what?s the problem? Performance of Monasca/Kibana/Elasticsearch? > But the node (telemetry) on which this stack is running is not overloaded - 20-30% CPU (24 cores on that node) , 40% RAM (there is 64GB), a lot of space on SSD. Hi Adam, Have you checked whether the logs are being pushed to Elasticsearch via the monasca log API? That might cause fluentd to start overflowing its buffers. We do have an issue where our monasca logging CI job is failing currently, but we have not yet had time to find the root cause. Thanks, Mark > > Best regards > Adam Tomas > > > > > > Hi, > > after some time of using newly deployed kolla-ansible I have fluentd errors: > > > > on all controllers, storage and network nodes: > > > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 01:40:46 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 01:40:47.847564807 +0200 chunk="5ccee7b1762157f0ae10dcbeb4e14fd8" error_class=RestClient::GatewayTimeout error="504 Gateway Timeout" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 04:49:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout? > > > > > > on compute nodes: > > > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-statsd.log" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"openstack_python", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"openstack_python\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"kolla.var.log.kolla.monasca.agent-statsd.log", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"kolla.var.log.kolla.monasca.agent-statsd.log\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-forwarder.log" > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data? > > > > and on monitoring node: > > > > 2021-09-27 14:40:50 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=84.05643947119825 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > > 2021-09-27 14:42:53 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=122.65328024700284 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > > 2021-09-27 14:44:15 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=82.32426812895574 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > > 2021-09-27 14:46:05 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=# > > 2021-09-27 14:46:05 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:46:06.042150289 +0200 chunk="5ccf89c9ec54b09e616f7167d5f93cc1" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" > > 2021-09-27 14:46:05 +0200 [warn]: #0 suppressed same stacktrace > > 2021-09-27 14:47:58 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=112.90601075813174 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > > 2021-09-27 14:47:58 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89c9ec54b09e616f7167d5f93cc1" > > 2021-09-27 14:50:16 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=136.24769522389397 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > > 2021-09-27 14:52:27 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=129.86474119895138 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > > 2021-09-27 14:54:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=100.71324555086903 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > > 2021-09-27 14:56:31 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=# > > > > > > > From midhunlaln66 at gmail.com Tue Sep 28 08:18:01 2021 From: midhunlaln66 at gmail.com (Midhunlal Nb) Date: Tue, 28 Sep 2021 13:48:01 +0530 Subject: Open stack Swift Message-ID: Hi team, I am using Openstack rocky setup with 1 controller node 2 compute node and 1 sStorage node,Now i am planning to configure a storage backup service Swift. Where do I need to configure swift,It needs an extra Node?Please explain the requirements. Thanks & Regards Midhunlal N B +918921245637 -------------- next part -------------- An HTML attachment was scrubbed... URL: From neil at tigera.io Tue Sep 28 08:20:38 2021 From: neil at tigera.io (Neil Jerram) Date: Tue, 28 Sep 2021 09:20:38 +0100 Subject: [stable][requirements][zuul] unpinned setuptools dependency on stable In-Reply-To: <827e99c6-99b2-54c8-a627-5153e3b84e6b@est.tech> References: <6J4UZQ.VOBD0LVDTPUX1@est.tech> <827e99c6-99b2-54c8-a627-5153e3b84e6b@est.tech> Message-ID: But I don't think that solution works for devstack, does it? Is there a way to pin setuptools in a stable/ussuri devstack run, except by changing the stable branch of the requirements project? On Mon, Sep 27, 2021 at 7:50 PM El?d Ill?s wrote: > Hi again, > > as I see there is no objection yet about using gibi's solution [1] (as I > already summarized the situation in my previous mail [2]) for a fix for > similar cases, so with a general stable core hat on, I *suggest* > everyone to use that solution to pin the setuptools in tox for every > failing cases (so that to avoid similar future errors as well). > > [1] https://review.opendev.org/810461 > [2] > > http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025059.html > > El?d > > > On 2021. 09. 27. 14:47, Balazs Gibizer wrote: > > > > > > On Fri, Sep 24 2021 at 10:21:33 PM +0200, Thomas Goirand > > wrote: > >> Hi Gibi! > >> > >> Thanks for bringing this up. > >> > >> As a distro package maintainer, here's my view. > >> > >> On 9/22/21 2:11 PM, Balazs Gibizer wrote: > >>> Option 1: Bump the major version of the decorator dependency on > >>> stable. > >> > >> Decorator 4.0.11 is even in Debian Stretch (currently oldoldstable), for > >> which I don't even maintain OpenStack anymore (that's OpenStack > >> Newton...). So I don't see how switching to decorator 4.0.0 is a > >> problem, and I don't understand how OpenStack could be using 3.4.0 which > >> is in Jessie (ie: 6 years old Debian release). > >> > >> PyPi says Decorator 3.4.0 is from 2012: > >> https://pypi.org/project/decorator/#history > >> > >> Do you have your release numbers correct? If so, then switching to > >> Decorator 4.4.2 (available in Debian Bullseye (shipped with Victoria) > >> and Ubuntu >=Focal) looks like reasonable to me... Sticking with 3.4.0 > >> feels a bit crazy (and I wasn't aware of it). > > > > Thanks for the info. So from Debian perspective it is OK to bump the > > decorator version on stable. As others noted in this thread it seems > > to be more than just decorator that broke. :/ > > > >> > >>> Option 2: Pin the setuptools version during tox installation > >> > >> Please don't do this for the master branch, we need OpenStack to stay > >> current with setuptools (yeah, even if this means breaking changes...). > > > > I've no intention to pin it on master. Master needs to work with the > > latest and greatest. Also on master it is easier to fix / replace the > > dependencies that become broken with new setuptools. > > > >> > >> For already released OpenStack: I don't mind much if this is done (I > >> could backport fixes if something breaks). > > > > ack > > > >> > >>> Option 3: turn off lower-constraints testing > >> > >> I already expressed myself about this: this is dangerous as distros rely > >> on it for setting lower bounds as low as possible (which is always > >> preferred from a distro point of view). > >> > >>> Option 4: utilize pyproject.toml[6] to specify build-time requirements > >> > >> I don't know about pyproject.toml. > >> > >> Just my 2 cents, hoping it's useful, > > > > Thanks! > > > > Cheers, > > gibi > > > >> Cheers, > >> > >> Thomas Goirand (zigo) > >> > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From micou12 at gmail.com Tue Sep 28 08:27:05 2021 From: micou12 at gmail.com (Michel Niyoyita) Date: Tue, 28 Sep 2021 10:27:05 +0200 Subject: Issue on accessing object storage containers from outside In-Reply-To: References: Message-ID: Hello Bernd Thank you for your response Both openstack and Ceph are in the same network and are pingable from each other using their IP and names . But once copy the container link is not reachable [image: image.png] kindly advise on how link provided can be accessible . Best Regards Michel On Thu, Sep 23, 2021 at 2:46 AM Bernd Bausch wrote: > Please reveal the precise symptoms. In particular, how do you attempt to > reach the public containers, and what exactly happens when you do so. > > Without having relevant information about your set up, my guess it that > this is a networking problem. You may have configured an external > network that is not reachable from your browser. The first thing, > therefore, is to check whether there is any connectivity to the network > to which the public link belongs. > > Bernd Bausch > > On 2021/09/22 7:35 PM, Michel Niyoyita wrote: > > Dear all > > > > I have created containers and upload objects into containers . I can > > access these containers and objects trough horizon dashboard but once > > make them public I can not reach them through the provided link . > > > > If any one have an idea on how to access containers and their object > > using the link from outside he can help and guide. > > > > I deployed openstack wallaby using kolla-ansible and ceph pacific > > using ansible , both are using ubuntu 20.04 as OS > > > > Michel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 20693 bytes Desc: not available URL: From tomasz.rutkowski at netart.pl Tue Sep 28 08:30:54 2021 From: tomasz.rutkowski at netart.pl (Tomasz Rutkowski) Date: Tue, 28 Sep 2021 10:30:54 +0200 Subject: [kolla][neutron][victoria] metering agent on compute nodes Message-ID: Hi All, I have issue with neutron-metering-agent - on network node it works ok, however on compute nodes the list of routers is constantly empty (and l3-agent works normal)... how to debug why outside network node the metering-agent does not recieve list of routers? with best regards -- Tomasz Rutkowski -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3728 bytes Desc: not available URL: From berndbausch at gmail.com Tue Sep 28 09:07:42 2021 From: berndbausch at gmail.com (Bernd Bausch) Date: Tue, 28 Sep 2021 18:07:42 +0900 Subject: Open stack Swift In-Reply-To: References: Message-ID: <4d5a90a0-eb48-c299-6596-8749334b1a71@gmail.com> The installation guide (https://docs.openstack.org/swift/wallaby/install/environment-networking.html) uses two additional servers, but it should be possible to adapt the instructions to a single-server setup. You could add Swift to the controller, for example. There are also instructions for an all-in-one setup https://docs.openstack.org/swift/latest/development_saio.html. On 2021/09/28 5:18 PM, Midhunlal Nb wrote: > Hi team, > I am using Openstack rocky setup with > 1 controller node > 2 compute node and 1 sStorage node,Now i am planning to configure a > storage backup service Swift. > Where do I need to configure swift,It needs an extra Node?Please > explain the requirements. > > Thanks & Regards > Midhunlal N B > +918921245637 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at stackhpc.com Tue Sep 28 09:09:22 2021 From: pierre at stackhpc.com (Pierre Riteau) Date: Tue, 28 Sep 2021 11:09:22 +0200 Subject: [openstack-qa] opendev.org mirrors In-Reply-To: References: Message-ID: On Sat, 25 Sept 2021 at 12:57, Eduardo Santos wrote: > > Hi folks, > > On US-based servers, a DevStack deploy takes 15 minutes. When deploying it from Brazil (where I live) however, it takes more than an hour. Examining the stack.sh logs, git_timed appears to be the culprit. I'm using a 250 Mbps connection. > > Do opendev.org mirrors exist for other regions other than the US? If not, has this been discussed yet? > > I believe this is more of a financial discussion othet than technical, but I'm curious if other users have raised this question before. > > I'm not sure if openstack-qa is the most appropriate flag. If not, please feel free to correct. > > Thanks! You can use the GIT_BASE variable to override the Git hosting service used for cloning repositories. So you could try setting GIT_BASE=https://github.com in local.conf, it might be faster for you. From openinfradn at gmail.com Tue Sep 28 12:47:14 2021 From: openinfradn at gmail.com (open infra) Date: Tue, 28 Sep 2021 18:17:14 +0530 Subject: Creating multiple servers with a shared volume In-Reply-To: <20210927084533.jm2iv337u5burfbe@lyarwood-laptop.usersys.redhat.com> References: <20210927084533.jm2iv337u5burfbe@lyarwood-laptop.usersys.redhat.com> Message-ID: On Mon, Sep 27, 2021 at 2:15 PM Lee Yarwood wrote: > On 25-09-21 13:11:19, open infra wrote: > > On Sat, Sep 25, 2021 at 7:37 AM Laurent Dumont > > > wrote: > > > > > Just to be sure, the flow is. > > > > > > - Create VM with root disk on a volume/image. > > > - Create another 1TB volume. > > > - Attach the volume to the VM as a second drive using a multi attach > > > approach ( > > > > https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiattach.html > > > )? > > > - Create snapshot of the original VM (with it's root drive + 1 TB > > > volume)? > > > - Create new VM from snapshot. > > > - Instead of a new VM + 1 new root volume + the same 1TB multiattach > > > volume attached, you get a new multiattach volume (new volume ID and > > > everything)? > > Because you're attaching a snapshot and not the original volume? See > below for more. > > > Yes, here is the block device mapping of the snapshot (from a VM with > 100GB > > of root disk and 1TB of multiattach volume) > > > > block_device_mapping[{"image_id": null, "delete_on_termination": true, > > "device_name": "/dev/vda", "disk_bus": "virtio", "volume_id": null, > > "volume_type": null, "destination_type": "volume", "source_type": > > "snapshot", "guest_format": null, "volume_size": 100, "device_type": > > "disk", "snapshot_id": "7b2c7f3a-8420-4a33-820e-2d2231d930c7", > > "boot_index": 0, "no_device": null, "tag": null}, {"image_id": null, > > "delete_on_termination": false, "device_name": "/dev/vdb", "disk_bus": > > null, "volume_id": null, "volume_type": null, "destination_type": > "volume", > > "source_type": "snapshot", "guest_format": null, "volume_size": 1000, > > "device_type": null, "snapshot_id": > "c2a0695f-8c9e-46f9-80a8-560c47521eeb", > > "boot_index": null, "no_device": null, "tag": null}] > > As you can see above you're using source_type = "snapshot" and > destination_type of "volume". As set out in the following docs this > creates a new volume from a given snapshot and attaches that to the > instance: > > > https://docs.openstack.org/nova/latest/user/block-device-mapping.html#valid-source-destination-combinations > > snapshot -> volume - this works exactly as passing type=snap does. It > would create a volume from a Cinder volume snapshot and attach that > volume to the instance. Can be marked bootable. > > > https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server > > block_device_mapping_v2.source_type (Optional) > > The source type of the block device. Valid values are: > [..] > snapshot: This is only valid with destination_type=volume; creates a > volume backed by the given volume snapshot referenced via the > block_device_mapping_v2.uuid parameter and attaches it to the server > > When attaching multiattach enabled volumes to multiple instances you > need to use source_type = 'volume' *and* destination_type = 'volume'. > > Hope this helps, > It very clears what went wrong. The above block device mapping was found when I try to figure out what went wrong. But I am not sure where exactly I should define block device mapping when I use CLI. Appreciate if you can give an example. Thanks a lot Lee Yarwood! > > -- > Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 > 2D76 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aschultz at redhat.com Tue Sep 28 13:19:59 2021 From: aschultz at redhat.com (Alex Schultz) Date: Tue, 28 Sep 2021 07:19:59 -0600 Subject: [Tripleo] Support of Alma/Rocky Linux In-Reply-To: References: Message-ID: On Tue, Sep 28, 2021 at 1:10 AM Anirudh Gupta wrote: > Hi Alex, > > Thanks for your response. > > I tried installing tripleo-repos on rocky linux for wallaby and faced the > below issue > > [stack at localhost ~]$ sudo -E tripleo-repos -b wallaby current ceph > --no-stream > WARNING: Unsupported platform 'rocky8' detected by tripleo-repos, centos7 > will be used unless you use CLI param to change it. > > Similar error was observed for Alma Linux as well. > [stack at localhost ~]$ sudo -E tripleo-repos -b wallaby current ceph > --no-stream > WARNING: Unsupported platform 'almalinux8' detected by tripleo-repos, > centos7 will be used unless you use CLI param to change it. > > Yea tripleo-repos doesn't support anything but CentOS. But you can configure the repositories yourself. https://trunk.rdoproject.org/centos8-wallaby/current/delorean.repo https://trunk.rdoproject.org/centos8-wallaby/delorean-deps.repo I don't know if there's a rockylinux ceph repo. We use the centos one http://mirror.centos.org/centos/8-stream/storage/x86_64/ceph-pacific/ Like I said, this is a YMMV thing where you'll have to put together the repositories yourself. > Regards > Anirudh Gupta > > On Tue, Sep 28, 2021 at 3:37 AM Alex Schultz wrote: > >> >> >> On Mon, Sep 27, 2021 at 8:42 AM Anirudh Gupta >> wrote: >> >>> Hi Team >>> >>> As per the document, the supporting OS mentioned in the documentation is >>> only Centos Stream and No Stream version (as of now) >>> >>> Since Centos has officially stopped releasing new versions and looking >>> for EOL. >>> I Just wanted to check if there is a possibility that we can install >>> Tripleo on a specific version of Alma Linux or Rocky Linux >>> >>> >> We don't plan on rebuilding for this. It may still work but YMMV. There >> was a previous thread about this back in May. >> >> >> http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022297.html >> >> >>> Regards >>> Anirudh Gupta >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From lyarwood at redhat.com Tue Sep 28 13:39:18 2021 From: lyarwood at redhat.com (Lee Yarwood) Date: Tue, 28 Sep 2021 14:39:18 +0100 Subject: Creating multiple servers with a shared volume In-Reply-To: References: <20210927084533.jm2iv337u5burfbe@lyarwood-laptop.usersys.redhat.com> Message-ID: <20210928133918.dpzjkdqnux5hfrtt@lyarwood-laptop.usersys.redhat.com> On 28-09-21 18:17:14, open infra wrote: > On Mon, Sep 27, 2021 at 2:15 PM Lee Yarwood wrote: > > > On 25-09-21 13:11:19, open infra wrote: > > > On Sat, Sep 25, 2021 at 7:37 AM Laurent Dumont > > > > > wrote: > > > > > > > Just to be sure, the flow is. > > > > > > > > - Create VM with root disk on a volume/image. > > > > - Create another 1TB volume. > > > > - Attach the volume to the VM as a second drive using a multi attach > > > > approach ( > > > > > > https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiattach.html > > > > )? > > > > - Create snapshot of the original VM (with it's root drive + 1 TB > > > > volume)? > > > > - Create new VM from snapshot. > > > > - Instead of a new VM + 1 new root volume + the same 1TB multiattach > > > > volume attached, you get a new multiattach volume (new volume ID and > > > > everything)? > > > > Because you're attaching a snapshot and not the original volume? See > > below for more. > > > > > Yes, here is the block device mapping of the snapshot (from a VM with > > 100GB > > > of root disk and 1TB of multiattach volume) > > > > > > block_device_mapping[{"image_id": null, "delete_on_termination": true, > > > "device_name": "/dev/vda", "disk_bus": "virtio", "volume_id": null, > > > "volume_type": null, "destination_type": "volume", "source_type": > > > "snapshot", "guest_format": null, "volume_size": 100, "device_type": > > > "disk", "snapshot_id": "7b2c7f3a-8420-4a33-820e-2d2231d930c7", > > > "boot_index": 0, "no_device": null, "tag": null}, {"image_id": null, > > > "delete_on_termination": false, "device_name": "/dev/vdb", "disk_bus": > > > null, "volume_id": null, "volume_type": null, "destination_type": > > "volume", > > > "source_type": "snapshot", "guest_format": null, "volume_size": 1000, > > > "device_type": null, "snapshot_id": > > "c2a0695f-8c9e-46f9-80a8-560c47521eeb", > > > "boot_index": null, "no_device": null, "tag": null}] > > > > As you can see above you're using source_type = "snapshot" and > > destination_type of "volume". As set out in the following docs this > > creates a new volume from a given snapshot and attaches that to the > > instance: > > > > > > https://docs.openstack.org/nova/latest/user/block-device-mapping.html#valid-source-destination-combinations > > > > snapshot -> volume - this works exactly as passing type=snap does. It > > would create a volume from a Cinder volume snapshot and attach that > > volume to the instance. Can be marked bootable. > > > > > > https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server > > > > block_device_mapping_v2.source_type (Optional) > > > > The source type of the block device. Valid values are: > > [..] > > snapshot: This is only valid with destination_type=volume; creates a > > volume backed by the given volume snapshot referenced via the > > block_device_mapping_v2.uuid parameter and attaches it to the server > > > > When attaching multiattach enabled volumes to multiple instances you > > need to use source_type = 'volume' *and* destination_type = 'volume'. > > > > Hope this helps, > > > > > It very clears what went wrong. > The above block device mapping was found when I try to figure out what went > wrong. > But I am not sure where exactly I should define block device mapping when I > use CLI. > Appreciate if you can give an example. > > Thanks a lot Lee Yarwood! Okay so given an existing multiattach volume of 02a27e61-b242-460e-8cf6-8525c5353698 I can create two instances with it attached as a secondary disks as follows: $ openstack volume show -c multiattach -f json 02a27e61-b242-460e-8cf6-8525c5353698 { "multiattach": true } $ openstack --os-compute-api-version 2.latest server create \ --image cirros-0.5.2-x86_64-disk \ --flavor 1 \ --block-device source_type=volume,destination_type=volume,uuid=02a27e61-b242-460e-8cf6-8525c5353698 \ --network private test-1 $ openstack --os-compute-api-version 2.latest server create \ --image cirros-0.5.2-x86_64-disk \ --flavor 1 \ --block-device source_type=volume,destination_type=volume,uuid=02a27e61-b242-460e-8cf6-8525c5353698 \ --network private test-2 You should be able to see both attachments listed against the volume: $ openstack volume show 02a27e61-b242-460e-8cf6-8525c5353698 -c attachments -f json { "attachments": [ { "id": "02a27e61-b242-460e-8cf6-8525c5353698", "attachment_id": "2f3a515a-3ef4-499e-a48a-04c6cd45a05a", "volume_id": "02a27e61-b242-460e-8cf6-8525c5353698", "server_id": "74adb548-f3ed-4d68-b282-b278dd1ec3f2", "host_name": "localhost.localdomain", "device": "/dev/vdb", "attached_at": "2021-09-28T13:31:31.000000" }, { "id": "02a27e61-b242-460e-8cf6-8525c5353698", "attachment_id": "5f0cdd6e-4586-4e9d-ad70-dfcf2a093c03", "volume_id": "02a27e61-b242-460e-8cf6-8525c5353698", "server_id": "2fdc931e-781a-4430-86af-4debe92c885a", "host_name": "localhost.localdomain", "device": "/dev/vdb", "attached_at": "2021-09-28T13:31:45.000000" } ] } Hope this helps, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From eduardo.experimental at gmail.com Tue Sep 28 14:36:00 2021 From: eduardo.experimental at gmail.com (Eduardo Santos) Date: Tue, 28 Sep 2021 14:36:00 +0000 Subject: [openstack-qa] opendev.org mirrors In-Reply-To: References: Message-ID: Thanks Jeremy and Pierre! Will try following that advice. DevStack is facing some issues related to an Apache update right now but I'll try again with the GitHub mirrors as soon as it's fixed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From clay.gerrard at gmail.com Tue Sep 28 14:38:53 2021 From: clay.gerrard at gmail.com (Clay Gerrard) Date: Tue, 28 Sep 2021 09:38:53 -0500 Subject: Issue on accessing object storage containers from outside In-Reply-To: References: Message-ID: nxdomain is a dns error - the machine you're using to open that web page doesn't know how to connect to the hostname ceph-osd3 it appears your issue is a basic networking issue - not openstack specific On Tue, Sep 28, 2021 at 3:31 AM Michel Niyoyita wrote: > Hello Bernd > > Thank you for your response > > Both openstack and Ceph are in the same network and are pingable from each > other using their IP and names . > But once copy the container link is not reachable > [image: image.png] > kindly advise on how link provided can be accessible . > > Best Regards > > Michel > > On Thu, Sep 23, 2021 at 2:46 AM Bernd Bausch > wrote: > >> Please reveal the precise symptoms. In particular, how do you attempt to >> reach the public containers, and what exactly happens when you do so. >> >> Without having relevant information about your set up, my guess it that >> this is a networking problem. You may have configured an external >> network that is not reachable from your browser. The first thing, >> therefore, is to check whether there is any connectivity to the network >> to which the public link belongs. >> >> Bernd Bausch >> >> On 2021/09/22 7:35 PM, Michel Niyoyita wrote: >> > Dear all >> > >> > I have created containers and upload objects into containers . I can >> > access these containers and objects trough horizon dashboard but once >> > make them public I can not reach them through the provided link . >> > >> > If any one have an idea on how to access containers and their object >> > using the link from outside he can help and guide. >> > >> > I deployed openstack wallaby using kolla-ansible and ceph pacific >> > using ansible , both are using ubuntu 20.04 as OS >> > >> > Michel >> >> -- Clay Gerrard 210 788 9431 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 20693 bytes Desc: not available URL: From laurentfdumont at gmail.com Mon Sep 27 15:44:12 2021 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Mon, 27 Sep 2021 11:44:12 -0400 Subject: Openstack hypervisor list is empty In-Reply-To: References: <8d13d73ecb4a4315ca301119bc184a08ec2bba48.camel@redhat.com> <8c2d5e826b207a003c15049579533aa87556f54c.camel@redhat.com> Message-ID: Perfect! Glad you were able to find the cause. On Mon, Sep 27, 2021 at 9:58 AM Karera Tony wrote: > Hello Laurent, > > Thanks for the advice. > > I got a solution from the IRC chat group. Apparently, The ceph_nova_user > defaults to the ceph_cinder_username. So even if you used nova at the ceph > side, You have to uncomment the ceph_nova_user: "nova" in global.yml since > it will default to cinder if not uncommented. > > It was merged recently > > Regards > > Tony Karera > > > > > On Mon, Sep 27, 2021 at 2:45 PM Karera Tony wrote: > >> I tried it but no one was replying to the chats >> >> Regards >> >> Tony Karera >> >> >> >> >> On Mon, Sep 27, 2021 at 7:57 AM Micha? Nasiadka >> wrote: >> >>> Hi Tony, >>> >>> As Laurent mentioned - it would be best if you would reach out on >>> #openstack-kolla IRC channel - and we?ll try to do our best to help you. >>> >>> Here?s an IRC guide from Contributors guide - if you?re not familiar: >>> https://docs.openstack.org/contributors/common/irc.html >>> >>> Regards, >>> Michal Nasiadka >>> >>> W dniu pon., 27.09.2021 o 07:52 Karera Tony >>> napisa?(a): >>> >>>> Hello Team, >>>> >>>> I even tried to manually put the ceph.client.cinder.keyring in the >>>> nova_compute container but the issue persisted. >>>> >>>> I also tried reinstalling Openstack on another Environment but I still >>>> have the same issue. >>>> >>>> Anyone with any idea on how to proceed ? >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont < >>>> laurentfdumont at gmail.com> wrote: >>>> >>>>> I know that there is some Kolla folks around but keep in mind that >>>>> this is a volunteer based list :) >>>>> >>>>> I think you might get a bit more one to one help on IRC in their kolla >>>>> channel. >>>>> >>>>> On Fri, Sep 24, 2021 at 5:10 PM Karera Tony >>>>> wrote: >>>>> >>>>>> I would really appreciate any support on this >>>>>> >>>>>> On Fri, 24 Sep 2021, 11:13 Karera Tony, wrote: >>>>>> >>>>>>> Hello Team, >>>>>>> >>>>>>> I don't know if there has been any change in the packages but the >>>>>>> way I am deploying is the same way I have been deploying. >>>>>>> >>>>>>> I don't understand why there is a certain issue now. >>>>>>> Regards >>>>>>> >>>>>>> Tony Karera >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 24, 2021 at 7:30 AM Karera Tony >>>>>>> wrote: >>>>>>> >>>>>>>> Hello Laurent, >>>>>>>> >>>>>>>> It turns out I only have one keyring in the container. >>>>>>>> >>>>>>>> root at compute1:/home/stack# docker exec -it nova_compute bash >>>>>>>> (nova-compute)[nova at compute1 /]$ cd /etc/ceph/ >>>>>>>> (nova-compute)[nova at compute1 ceph]$ ls >>>>>>>> ceph.client.nova.keyring ceph.conf rbdmap >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> Tony Karera >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < >>>>>>>> laurentfdumont at gmail.com> wrote: >>>>>>>> >>>>>>>>> I do believe Kolla runs a container version of each service on >>>>>>>>> computes. Are you looking inside the nova-compute container ( >>>>>>>>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >>>>>>>>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >>>>>>>>> >>>>>>>>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hello Sean, >>>>>>>>>> >>>>>>>>>> Below are the output on the compute node and deployment >>>>>>>>>> >>>>>>>>>> root at compute1:/etc/kolla/nova-compute# ls >>>>>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>>>>> config.json nova.conf >>>>>>>>>> >>>>>>>>>> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/ >>>>>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>>>>> >>>>>>>>>> And I can confirm that the content is the same. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> >>>>>>>>>> Tony Karera >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>>>>>>>>> > I would investigate that compute error first. Creating volumes >>>>>>>>>>> means the >>>>>>>>>>> > controllers are doing the action. Starting a VM on a compute >>>>>>>>>>> means you also >>>>>>>>>>> > need Ceph to works on the compute to mount the rdb target. >>>>>>>>>>> >>>>>>>>>>> nova as part of its startup process in aintiallying the resouce >>>>>>>>>>> tracker will >>>>>>>>>>> try to connect to ceph if you are using the rbd image backend to >>>>>>>>>>> report how much stroage >>>>>>>>>>> is avaiable. if the keyring does not work on the vms pool as >>>>>>>>>>> the user nova is connecting as >>>>>>>>>>> then that will block the agent from starting up fully and will >>>>>>>>>>> cause it to be missing the hypervior list. >>>>>>>>>>> >>>>>>>>>>> the error seams to indicate that the cinder keyring is not in >>>>>>>>>>> the nova container >>>>>>>>>>> that likely means you have not put it in /etc/kolla/config/nova >>>>>>>>>>> i woudl check /etc/kolla/config/nova on the deployment host and >>>>>>>>>>> sudo ls /etc/kolla/nova-compute/ >>>>>>>>>>> on the compute node to ensure the cinder keyring is actully >>>>>>>>>>> copied and has the correct content >>>>>>>>>>> >>>>>>>>>>> i have >>>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo ls >>>>>>>>>>> /etc/kolla/nova-compute/ >>>>>>>>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>>>>>>>> config.json nova.conf >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [client.cinder] >>>>>>>>>>> key = ********************************* >>>>>>>>>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>>>>>>>>> pool=vms" >>>>>>>>>>> caps mon = "profile rbd" >>>>>>>>>>> caps osd = "profile rbd pool=volumes, profile rbd >>>>>>>>>>> pool=vms, profile rbd pool=images" >>>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>>>>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>>>>>>>>> [client.nova] >>>>>>>>>>> key = ********************************* >>>>>>>>>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>>>>>>>>> pool=vms" >>>>>>>>>>> caps mon = "profile rbd" >>>>>>>>>>> caps osd = "profile rbd pool=volumes, profile rbd >>>>>>>>>>> pool=vms, profile rbd pool=images" >>>>>>>>>>> >>>>>>>>>>> blanked out the key wiht *************** after the fact but you >>>>>>>>>>> should have something similar >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> in my case i decied to use a seperate key for nova rbd backend >>>>>>>>>>> because i was also using EC poosl with a seperate data and metadata pool >>>>>>>>>>> so i neede to modify my ceph.conf to make that work with kolla >>>>>>>>>>> >>>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat >>>>>>>>>>> /etc/kolla/nova-compute/ceph.conf >>>>>>>>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>>>>>>>>> [global] >>>>>>>>>>> fsid = ********************* >>>>>>>>>>> mon_host = [*********************] >>>>>>>>>>> >>>>>>>>>>> [client.glance] >>>>>>>>>>> rbd default data pool = images-data >>>>>>>>>>> >>>>>>>>>>> [client.cinder] >>>>>>>>>>> rbd default data pool = volumes-data >>>>>>>>>>> >>>>>>>>>>> [client.nova] >>>>>>>>>>> rbd default data pool = vms-data >>>>>>>>>>> >>>>>>>>>>> using 2 keyrings/user allows me to set different default data >>>>>>>>>>> pools for cinder and nova. >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> > Working in Wallaby with the error doesn't mean it would 100% >>>>>>>>>>> work in >>>>>>>>>>> > Victoria. >>>>>>>>>>> > >>>>>>>>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony < >>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>> > >>>>>>>>>>> > > Hey Guys, Any other idea ? >>>>>>>>>>> > > >>>>>>>>>>> > > Regards >>>>>>>>>>> > > >>>>>>>>>>> > > Tony Karera >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>> > > >>>>>>>>>>> > > > Just to add on that, >>>>>>>>>>> > > > >>>>>>>>>>> > > > compute service is listed, I can create Volumes, I have >>>>>>>>>>> the same cinder >>>>>>>>>>> > > > keyring in the /etc/kolla/config/nova directory as I have >>>>>>>>>>> in the >>>>>>>>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along >>>>>>>>>>> with the nova keyring >>>>>>>>>>> > > > Regards >>>>>>>>>>> > > > >>>>>>>>>>> > > > Tony Karera >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>>>>>>>>>> tonykarera at gmail.com> wrote: >>>>>>>>>>> > > > >>>>>>>>>>> > > > > Hello Guys, >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > Thanks a lot. >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > I had actually checked the nova -compute.log on the >>>>>>>>>>> compute node and >>>>>>>>>>> > > > > they were showing the error I will post at the end about >>>>>>>>>>> the cinder keyring >>>>>>>>>>> > > > > but I know its correct because its the same I was using >>>>>>>>>>> on wallaby, I even >>>>>>>>>>> > > > > tried to use another ceph cluster with ofcouse different >>>>>>>>>>> keyrings but its >>>>>>>>>>> > > > > the same issue. >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > Below is the error >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>>>>>>>> auth: unable to >>>>>>>>>>> > > > > find a keyring on >>>>>>>>>>> > > > > >>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>>>>> > > > > (2) No such file or >>>>>>>>>>> directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>>>>>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>>>>>>>>> > > > > >>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>>>>>>>>> 7fbce2f4f700 -1 auth: unable >>>>>>>>>>> > > > > to find a keyring on >>>>>>>>>>> > > > > >>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>>>>> > > > > (2) No such file or >>>>>>>>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>>>>>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>>>>>>>>> > > > > >>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>>>>>>>>> 7fbce2f4f700 -1 auth: unable >>>>>>>>>>> > > > > to find a keyring on >>>>>>>>>>> > > > > >>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>>>>>>>> > > > > (2) No such file or >>>>>>>>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>>>>>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>>>>>>>>> > > > > >>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>>>>>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>>>>>>>>> connecting to the >>>>>>>>>>> > > > > cluster)\n' >>>>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>>>>>>>> During handling of >>>>>>>>>>> > > > > the above exception, another exception occurred: >>>>>>>>>>> > > > > Regards >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > Tony Karera >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>>>>>>>>>> smooney at redhat.com> wrote: >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont >>>>>>>>>>> wrote: >>>>>>>>>>> > > > > > > It could also be a compute cell discovery issue >>>>>>>>>>> maybe? >>>>>>>>>>> > > > > > no they shoudl still show up in the hypervior list api >>>>>>>>>>> > > > > > > >>>>>>>>>>> > > > > > > Do you see anything under "openstack compute service >>>>>>>>>>> list"? >>>>>>>>>>> > > > > > if they show up in the service list but not they >>>>>>>>>>> hyperiors api it >>>>>>>>>>> > > > > > means that the comptue service started and registered >>>>>>>>>>> its service entry >>>>>>>>>>> > > > > > but >>>>>>>>>>> > > > > > something broke it before it could create a compute >>>>>>>>>>> node recored in the >>>>>>>>>>> > > > > > db. >>>>>>>>>>> > > > > > >>>>>>>>>>> > > > > > with ceph the case i have hit this most often is when >>>>>>>>>>> the keyright used >>>>>>>>>>> > > > > > by nova to >>>>>>>>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>>>>>>>>>> whihc prevent >>>>>>>>>>> > > > > > the resoucetack and compute manager >>>>>>>>>>> > > > > > form actully creating the compute node record. >>>>>>>>>>> > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> > > > > > it can happen for other reason too but best place to >>>>>>>>>>> start is check if >>>>>>>>>>> > > > > > there is an error in the nova compute agent log and go >>>>>>>>>>> from there. >>>>>>>>>>> > > > > > > >>>>>>>>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>>>>>>>>> smooney at redhat.com> >>>>>>>>>>> > > > > > wrote: >>>>>>>>>>> > > > > > > >>>>>>>>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony >>>>>>>>>>> wrote: >>>>>>>>>>> > > > > > > > > Hello Team, >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> > > > > > > > > I have deployed Openstack Victoria using >>>>>>>>>>> Kolla-ansible on Ubuntu >>>>>>>>>>> > > > > > 20.04 >>>>>>>>>>> > > > > > > > and >>>>>>>>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>>>>>>>>>> Glance. >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> > > > > > > > > It finished with no error but it has failed to >>>>>>>>>>> register any on the >>>>>>>>>>> > > > > > > > Compute >>>>>>>>>>> > > > > > > > > Nodes under Hypervisors. >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack >>>>>>>>>>> hypervisor list >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> > > > > > > > > (kolla-openstack) stack at deployment:~$ >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> > > > > > > > > Any idea on how to resolve this ? >>>>>>>>>>> > > > > > > > that usually means that somehthing prevented the >>>>>>>>>>> comptue agent form >>>>>>>>>>> > > > > > > > strating properly >>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > for example incorrect ceph keyrings there are >>>>>>>>>>> several other case >>>>>>>>>>> > > > > > but you >>>>>>>>>>> > > > > > > > mentioned you are >>>>>>>>>>> > > > > > > > using ceph. >>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > if this is hte case you should see error in the >>>>>>>>>>> compute agent log. >>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> > > > > > > > > Regards >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> > > > > > > > > Tony Karera >>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>> Micha? Nasiadka >>> mnasiadka at gmail.com >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From anyrude10 at gmail.com Tue Sep 28 07:10:08 2021 From: anyrude10 at gmail.com (Anirudh Gupta) Date: Tue, 28 Sep 2021 12:40:08 +0530 Subject: [Tripleo] Support of Alma/Rocky Linux In-Reply-To: References: Message-ID: Hi Alex, Thanks for your response. I tried installing tripleo-repos on rocky linux for wallaby and faced the below issue [stack at localhost ~]$ sudo -E tripleo-repos -b wallaby current ceph --no-stream WARNING: Unsupported platform 'rocky8' detected by tripleo-repos, centos7 will be used unless you use CLI param to change it. Similar error was observed for Alma Linux as well. [stack at localhost ~]$ sudo -E tripleo-repos -b wallaby current ceph --no-stream WARNING: Unsupported platform 'almalinux8' detected by tripleo-repos, centos7 will be used unless you use CLI param to change it. Regards Anirudh Gupta On Tue, Sep 28, 2021 at 3:37 AM Alex Schultz wrote: > > > On Mon, Sep 27, 2021 at 8:42 AM Anirudh Gupta wrote: > >> Hi Team >> >> As per the document, the supporting OS mentioned in the documentation is >> only Centos Stream and No Stream version (as of now) >> >> Since Centos has officially stopped releasing new versions and looking >> for EOL. >> I Just wanted to check if there is a possibility that we can install >> Tripleo on a specific version of Alma Linux or Rocky Linux >> >> > We don't plan on rebuilding for this. It may still work but YMMV. There > was a previous thread about this back in May. > > http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022297.html > > >> Regards >> Anirudh Gupta >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From erin at openstack.org Tue Sep 28 18:39:50 2021 From: erin at openstack.org (Erin Disney) Date: Tue, 28 Sep 2021 13:39:50 -0500 Subject: OpenInfra Live - September 30th at 9am CT Message-ID: Hi everyone, This week?s OpenInfra Live episode is brought to you by the OpenStack and Ceph communities. Ceph is a highly scalable distributed-storage open source solution offering object, block, and file storage. Join us as various Community members discuss the basics, ongoing development and integration of OpenStack with Ceph. Date and time: September 30th at 9am CT (1400 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=zJVoleSpSOk LinkedIn: https://www.linkedin.com/video/event/urn:li:ugcPost:6846862917225320448/ Facebook: https://www.facebook.com/104139126308032/posts/4364096940312208/ WeChat: Recording will be posted on OpenStack WeChat after the live stream Speakers: Kendall Nelson (OpenInfra Foundation) John Fulton (Red Hat) Francesco Pantano (Red Hat) Tom Barron (Red Hat) Loan Harrouin (Societe Generale) Pawel Sadowski (OVH) Mohammed Naser (VEXXHOST) Have an idea for a future episode? Share it now at ideas.openinfra.live . Register now for OpenInfra Live: Keynotes, a special edition of OpenInfra Live on November 17-18th starting at 1500 UTC: https://openinfralivekeynotes.eventbrite.com/ Thanks, Erin Erin Disney Event Marketing Open Infrastructure Foundation -------------- next part -------------- An HTML attachment was scrubbed... URL: From vhariria at redhat.com Tue Sep 28 20:54:38 2021 From: vhariria at redhat.com (Vida Haririan) Date: Tue, 28 Sep 2021 16:54:38 -0400 Subject: [Cinder] [Nova] [Interop] "compute-volume-list" API guideline Message-ID: Hi Cinder and Nova teams, The Interop team is reviewing all related open stories to determine next steps. We researched the current status of 1638112:?os-volumes should be flagged for removal in the compute API guidelines? [1] and noticed that "compute-volume-list" was removed in 2017.01 guidelines [2][3]. However, two related tests still remain in tempest [4][5] and we are reaching out to determine if the API is still in use or has been moved/renamed. Please let us know if you have any questions. [1] https://storyboard.openstack.org/#!/story/1638112 [2] https://opendev.org/osf/interop/src/commit/dd7ae2660de5daa82ab4fe080e875d6326f88185/guidelines/2017.01.json#L1324-L1348 [3] https://opendev.org/osf/interop/src/branch/master/guidelines/next.json [4] https://opendev.org/openstack/tempest/src/commit/ae41052a51f5dbb748eb6bf4f23e9145853f4639/tempest/api/compute/volumes/test_volumes_list.py#L60 [5] https://opendev.org/openstack/tempest/src/commit/ae41052a51f5dbb748eb6bf4f23e9145853f4639/tempest/api/compute/volumes/test_volumes_list.py#L75 Thank you, -- *Vida Haririan* Senior Software Quality Engineer Red Hat EMEA -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Tue Sep 28 22:29:02 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 28 Sep 2021 17:29:02 -0500 Subject: [all] Gate status: All devstack based jobs are failing for placement url issue in devstack In-Reply-To: References: <17c29586680.120e3c448119950.8945886018923104354@ghanshyammann.com> Message-ID: <17c2e868dc5.f7179b7f186808.5893029040673482428@ghanshyammann.com> Updates: placement API URL (uwsgi setup bug#1945359) issue fixes are merged now, please recheck your failing patches. Big thanks to frickler, ianw, clarkb, and melwitt. Nova two jobs still failing on different issues and fixes are under testing: 1. "nova-ceph-multistore job is broken on stable branches" - https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1945358 - Fixes are under testing: https://review.opendev.org/q/I7061f8d1491ff957452c9c777e40186a4e9c324e 2. nova-grenade-multinode job failing on master, stable/xena due to neutron-truck service/extension - https://bugs.launchpad.net/grenade/+bug/1945346 - Fixes are under testing: https://review.opendev.org/q/topic:%22bug%252F1945346%22+(status:open%20OR%20status:merged) -gmann ---- On Tue, 28 Sep 2021 02:28:26 -0500 Balazs Gibizer wrote ---- > > > On Tue, Sep 28 2021 at 05:19:29 PM +1000, Ian Wienand > wrote: > > On Tue, Sep 28, 2021 at 03:27:23PM +1000, Ian Wienand wrote: > >> The recent security update @ > >> > >> https://launchpad.net/ubuntu/focal/+source/apache2/+changelog > >> > >> does a lot of stuff. > > > > frickler was actually onto this before me; see [1]. > > > > In short, it looks like we have found an issue with our ProxyPass > > config and the way it uses trailing slashes. See [2] > > > > We can either do > > > > ProxyPass "/placement/" > > "unix:/var/run/uwsgi/placement-api.socket|uwsgi://uwsgi-uds-placement-api/" > > retry=0 > > > > (note "placement/" and then "uwsgi://uwsgi-uds-placement-api/") or > > > > ProxyPass "/placement" > > "unix:/var/run/uwsgi/placement-api.socket|uwsgi://uwsgi-uds-placement-api" > > retry=0 > > > > both appear to work. I'm not sure which is better? We could enforce > > either in the function. As of right now 811303 is still testing, but > > it's looking good. > > Thank you all for troubleshooting and finding the solution. > > I think we need to use the '/placement' variant to allow GET > http:///placement to be routed to the placement app. As far as I > see if we use '/placement/' variant then /placement is not routed and > that would be a breaking API change. > > Cheers, > gibi > > > > > -i > > > > [1] > > https://meetings.opendev.org/irclogs/%23openstack-qa/%23openstack-qa.2021-09-28.log.html#t2021-09-28T03:54:49 > > [2] https://review.opendev.org/c/openstack/devstack/+/811303 > > > > > > > > From katonalala at gmail.com Wed Sep 29 07:40:52 2021 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 29 Sep 2021 09:40:52 +0200 Subject: [kolla][neutron][victoria] metering agent on compute nodes In-Reply-To: References: Message-ID: Hi, By documentation (see [1]) and quick code check for me it looks like metering agent resides l3-agent. Do You need metrics from DVR? [1]: https://docs.openstack.org/neutron/latest/admin/archives/config-agents.html#configure-metering-agent Tomasz Rutkowski ezt ?rta (id?pont: 2021. szept. 28., K, 10:36): > Hi All, > > I have issue with neutron-metering-agent - on network node it works ok, > however on compute nodes the list of routers is constantly empty (and > l3-agent works normal)... how to debug why outside network node the > metering-agent does not recieve list of routers? > > > with best regards > -- > Tomasz Rutkowski > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomasz.rutkowski at netart.pl Wed Sep 29 08:04:38 2021 From: tomasz.rutkowski at netart.pl (Tomasz Rutkowski) Date: Wed, 29 Sep 2021 10:04:38 +0200 Subject: [kolla][neutron][victoria] metering agent on compute nodes In-Reply-To: References: Message-ID: W?dniu ?ro, 29.09.2021 o?godzinie 09?40?+0200, u?ytkownik Lajos Katona napisa?: > Hi, > By documentation (see [1]) and quick code check for me it looks like > metering agent resides l3-agent. > Do You need metrics from DVR? > yes, I'm trying to fetch metrics from dvr :) l3-agents seems ok, the routers and fips works fine all around network and compute nodes metering agent with some minor tweaks (include snat_iptables_manager in creating chains and fetching metrics) works good on network node, however on compute nodes the router list that metering agent should work on is empty DEBUG neutron.services.metering.agents.metering_agent [-] Traffic counters [{}] retrieved for routers [{}]. _add_metering_infos /var/lib/kolla/venv/lib/python3.8/site- packages/neutron/services/metering/agents/metering_agent.py:216 and I can't find the reason... with best regards Tomasz Rutkowski -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3728 bytes Desc: not available URL: From zigo at debian.org Wed Sep 29 08:27:02 2021 From: zigo at debian.org (Thomas Goirand) Date: Wed, 29 Sep 2021 10:27:02 +0200 Subject: [announce][debian][xena] general availability of OpenStack Xena in Debian Message-ID: Hi! It's my pleasure to announce the general availability of OpenStack Xena in Debian. Indeed, I've just finished uploading everything to Debian Experimental at the beginning of this week, and was able yesterday, to pop my first VM over on top of Xena. This marks the first readiness of the release, from the Debian point of view (even though there probably will be things to fix, like after addressing https://bugs.launchpad.net/oslo.policy/+bug/1945336 for example, and uploading the final release to replace the RCs when that becomes available). The Bullseye backports are available the usual way, for example using extrepo: apt-get install extrepo extrepo enable openstack_xena apt-get update ... or directly setting-up the http://bullseye-xena.debian.net repository the usual way. I'm currently in the process of uploading all of Xena to Debian Unstable (but please bare with me: there's 225+ packages in this update). Note that Xena can already be installed by OCI [1]. Please report any issue you may find, on OCI or on the Debian packages, through the Debian BTS as usual. Cheers, Thomas Goirand (zigo) [1] https://salsa.debian.org/openstack-team/debian/openstack-cluster-installer From balazs.gibizer at est.tech Wed Sep 29 09:44:35 2021 From: balazs.gibizer at est.tech (Balazs Gibizer) Date: Wed, 29 Sep 2021 11:44:35 +0200 Subject: [Cinder] [Nova] [Interop] "compute-volume-list" API guideline In-Reply-To: References: Message-ID: On Tue, Sep 28 2021 at 04:54:38 PM -0400, Vida Haririan wrote: > Hi Cinder and Nova teams, > > The Interop team is reviewing all related open stories to determine > next steps. > > We researched the current status of 1638112:?os-volumes should be > flagged for removal in the compute API guidelines? [1] and noticed > that "compute-volume-list" was removed in 2017.01 guidelines [2][3]. > However, two related tests still remain in tempest [4][5]and we are > reaching out to determine if the API is still in use or has been > moved/renamed. Nova has been deprecated those proxy APIs and with microversion >= 2.36 Nova returns 404. [6][7]. Still nova did not deleted the os-volumes code, so with microversion < 2.36 the API works this is why the tempest test is still passing. I think we should only remove the tempest test when nova deletes the os-volumes API resource from the code. (We did delete other proxy resources like os-networks in Ussuri) Cheers, gibi > > Please let us know if you have any questions. > > [1] https://storyboard.openstack.org/#!/story/1638112 > [2] > https://opendev.org/osf/interop/src/commit/dd7ae2660de5daa82ab4fe080e875d6326f88185/guidelines/2017.01.json#L1324-L1348 > [3] > https://opendev.org/osf/interop/src/branch/master/guidelines/next.json > [4] > https://opendev.org/openstack/tempest/src/commit/ae41052a51f5dbb748eb6bf4f23e9145853f4639/tempest/api/compute/volumes/test_volumes_list.py#L60 > [5] > https://opendev.org/openstack/tempest/src/commit/ae41052a51f5dbb748eb6bf4f23e9145853f4639/tempest/api/compute/volumes/test_volumes_list.py#L75 [6] https://docs.openstack.org/api-ref/compute/#list-volumes [7] https://docs.openstack.org/nova/latest/reference/api-microversion-history.html Cheers, gibi > > Thank you, > > -- > Vida Haririan > Senior Software Quality Engineer > > Red Hat EMEA > > From senrique at redhat.com Wed Sep 29 13:43:55 2021 From: senrique at redhat.com (Sofia Enriquez) Date: Wed, 29 Sep 2021 10:43:55 -0300 Subject: [cinder] Bug deputy report for week of 2021-29-09 Message-ID: This is a bug report from 2021-15-09 to 2021-29-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/os-brick/+bug/1943977 "os_bricks fails to rerty "iscsiadm -m session" when iser_use_multipath and iscsi_use_multipath are set in nova.conf". Assigned to Sophie Huang. - https://bugs.launchpad.net/os-brick/+bug/1945323 "Can't attach NVMe volume" Low - https://bugs.launchpad.net/cinder/+bug/1945169 " [swift] volume backup's status keep in 'restoring' and Lost connection to MySQL server during query". Assigned to silphon zhang. Wishlist - https://bugs.launchpad.net/cinder/+bug/1945249 '[charm] no upgrade path to cinder-lvm subordinate'. Unassigned. Cheers, -- Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnasiadka at gmail.com Wed Sep 29 14:52:55 2021 From: mnasiadka at gmail.com (=?utf-8?Q?Micha=C5=82_Nasiadka?=) Date: Wed, 29 Sep 2021 16:52:55 +0200 Subject: [kolla] Yoga PTG In-Reply-To: <4a3f0007-6465-4dad-b253-de530026f0aa@Spark> References: <4a3f0007-6465-4dad-b253-de530026f0aa@Spark> Message-ID: <5dba4480-1f54-48b9-a705-878308228e90@Spark> Hola Koalas! We are approaching the time of the next PTG. Kolla Yoga etherpad is available and initially populated[1], feel free to add any topics that shall be discussed and register your planned attendance. Don't forget to register! [2]. [1]?https://etherpad.opendev.org/p/kolla-yoga-ptg [3]?https://openinfra-ptg.eventbrite.com Best regards, Michal Nasiadka -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry at openstack.org Wed Sep 29 15:18:00 2021 From: thierry at openstack.org (Thierry Carrez) Date: Wed, 29 Sep 2021 17:18:00 +0200 Subject: [largescale-sig] Next meeting: Sept 29th, 15utc In-Reply-To: References: Message-ID: We held our meeting today. Small attendance, we discussed potential guest speakers for the upcoming "Large Scale OpenStack" episode on OpenInfra.Live (Oct 14). The topic will be the hot question that everybody asks: what are Neutron scaling best practices? Drivers to use? Features to disable? Tips and tricks? You can read the meeting logs at: https://meetings.opendev.org/meetings/large_scale_sig/2021/large_scale_sig.2021-09-29-15.00.html Our next IRC meeting will be Oct 27, at 1500utc on #openstack-operators on OFTC. Regards, -- Thierry Carrez (ttx) From radoslaw.piliszek at gmail.com Wed Sep 29 16:06:48 2021 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Wed, 29 Sep 2021 18:06:48 +0200 Subject: [masakari] Proposal to cancel weekly meetings Message-ID: Dears, Due to low attendance and the current schedule being uncomfortable for me, I propose to cancel the weekly meetings and suggest we coordinate via this mailing list and do ad-hoc chats on IRC as I'm simply lurking there most of the time and answering the messages. Kind regards, -yoctozepto From satish.txt at gmail.com Wed Sep 29 16:33:14 2021 From: satish.txt at gmail.com (Satish Patel) Date: Wed, 29 Sep 2021 12:33:14 -0400 Subject: DPDK Poor performance Message-ID: Folk, I have deployed DPDK on one of my compute nodes to reply to my sriov deployment which doesn't support bonding. This is what I am doing to do a performance benchmark. Spun up VM with 8 vCPU and 8GB memory with single virtual nic. Then install nuttcp tool and start nuttcp -S (server mode) and then use Physical server outside my lab and use following command to blast some traffic nuttcp -T 30 -i -u -w4m -R 2G 10.69.7.130 I found 120kpps rate working fine but after that I noticed a packet drop on nuttcp. Here is the full details of my deployment and load-test result - https://paste.opendev.org/show/809675/ Small confusion related hugepage - I am seeing following in /dev/hugepages/ I believe rtemap_0 created by DPDK root at ovn-lab-comp-dpdk-1:~# ls -l /dev/hugepages/rtemap_0 -rw------- 1 root root 1073741824 Sep 29 02:28 /dev/hugepages/rtemap_0 Why is the qemu instance file empty? Does that mean it's not using a huge page for dpdk performance? root at ovn-lab-comp-dpdk-1:~# ls -l /dev/hugepages/libvirt/qemu/1-instance-00000045/ total 0 I have hw:mem_page_size='large' setting in flavor so it should use hugepage. am i missing something? From gmann at ghanshyammann.com Wed Sep 29 18:37:43 2021 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 29 Sep 2021 13:37:43 -0500 Subject: [all][tc] Technical Committee next weekly meeting on Sept 30th at 1500 UTC In-Reply-To: <17c285a45d8.1251b48bd112192.5860450617549878983@ghanshyammann.com> References: <17c285a45d8.1251b48bd112192.5860450617549878983@ghanshyammann.com> Message-ID: <17c32d92476.fe081aa6247097.6188601264447602857@ghanshyammann.com> Hello Everyone, Below is the agenda for Tomorrow's TC meeting schedule at 1500 UTC in #openstack-tc IRC channel. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting == Agenda for tomorrow's TC meeting == * Roll call * Follow up on past action items * Gate health check (dansmith/yoctozepto) ** http://paste.openstack.org/show/jD6kAP9tHk7PZr2nhv8h/ * Place to maintain the external hosted ELK, E-R, O-H services ** https://etherpad.opendev.org/p/elk-service-maintenance-plan * Project Health checks framework ** https://etherpad.opendev.org/p/health_check ** https://review.opendev.org/c/openstack/governance/+/810037 * Open Reviews ** https://review.opendev.org/q/projects:openstack/governance+is:open -gmann ---- On Mon, 27 Sep 2021 12:42:57 -0500 Ghanshyam Mann wrote ---- > Hello Everyone, > > Technical Committee's next weekly meeting is scheduled for Sept 30th at 1500 UTC. > > If you would like to add topics for discussion, please add them to the below wiki page by > Wednesday, Sept 29th, at 2100 UTC. > > https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting > > -gmann > > > From anyrude10 at gmail.com Wed Sep 29 11:17:33 2021 From: anyrude10 at gmail.com (Anirudh Gupta) Date: Wed, 29 Sep 2021 16:47:33 +0530 Subject: [TripleO] Issue in running Pre-Introspection Message-ID: Hi Team, I tried installing Undercloud using the below link: https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/install_overcloud.html#deploy-the-overcloud I am getting the following error: (undercloud) [stack at undercloud ~]$ openstack tripleo validator run --group pre-introspection Selected log directory '/home/stack/validations' does not exist. Attempting to create it. +--------------------------------------+---------------------------------+--------+------------+-----------------+-------------------+-------------+ | UUID | Validations | Status | Host_Group | Status_by_Host | Unreachable_Hosts | Duration | +--------------------------------------+---------------------------------+--------+------------+-----------------+-------------------+-------------+ | 7029c1f6-5ab4-465d-82d7-3f29058012ce | check-cpu | PASSED | localhost | localhost | | 0:00:02.531 | | db059017-30f1-4b97-925e-3f55b586d492 | check-disk-space | PASSED | localhost | localhost | | 0:00:04.432 | | e23dd9a1-90d3-4797-ae0a-b43e55ab6179 | check-ram | PASSED | localhost | localhost | | 0:00:01.324 | | 598ca02d-258a-44ad-b78d-3877321cdfe6 | check-selinux-mode | PASSED | localhost | localhost | | 0:00:01.591 | | c4435b4c-b432-4a1e-8a99-00638034a884 | *check-network-gateway | FAILED* | undercloud | *No host matched* | | | | cb1eed23-ef2f-4acd-a43a-86fb09bf0372 | *undercloud-disk-space | FAILED* | undercloud | *No host matched* | | | | abde5329-9289-4b24-bf16-c4d82b03e67a | *undercloud-neutron-sanity-check | FAILED* | undercloud | *No host matched* | | | | d0e5fdca-ece6-4a37-b759-ed1fac31a10f | *ctlplane-ip-range | FAILED* | undercloud | No host matched | | | | 91511807-225c-4852-bb52-6d0003c51d49 | *dhcp-introspection | FAILED* | undercloud | No host matched | | | | e96f7704-d2fb-465d-972b-47e2f057449c |* undercloud-tokenflush | FAILED *| undercloud | No host matched | | | As per the validation link, https://docs.openstack.org/tripleo-validations/wallaby/validations-pre-introspection-details.html check-network-gateway If gateway in undercloud.conf is different from local_ip, verify that the gateway exists and is reachable Observation - In my case IP specified in local_ip and gateway, both are pingable, but still this error is being observed ctlplane-ip-range? Check the number of IP addresses available for the overcloud nodes. Verify that the number of IP addresses defined in dhcp_start and dhcp_end fields in undercloud.conf is not too low. - ctlplane_iprange_min_size: 20 Observation - In my case I have defined more than 20 IPs Similarly for disk related issue, I have dedicated 100 GB space in /var and / Filesystem Size Used Avail Use% Mounted on devtmpfs 12G 0 12G 0% /dev tmpfs 12G 84K 12G 1% /dev/shm tmpfs 12G 8.7M 12G 1% /run tmpfs 12G 0 12G 0% /sys/fs/cgroup /dev/mapper/cl-root 100G 2.5G 98G 3% / /dev/mapper/cl-home 47G 365M 47G 1% /home /dev/mapper/cl-var 103G 1.1G 102G 2% /var /dev/vda1 947M 200M 747M 22% /boot tmpfs 2.4G 0 2.4G 0% /run/user/0 tmpfs 2.4G 0 2.4G 0% /run/user/1000 Despite setting al the parameters, still I am not able to pass pre-introspection checks. *"NO Host Matched" *is found in the table. Regards Anirudh Gupta -------------- next part -------------- An HTML attachment was scrubbed... URL: From corey.bryant at canonical.com Wed Sep 29 20:11:20 2021 From: corey.bryant at canonical.com (Corey Bryant) Date: Wed, 29 Sep 2021 16:11:20 -0400 Subject: Help with eventlet 0.26.1 and dnspython >= 2 In-Reply-To: References: <56ab7d1e-c8d6-8505-dd81-fb4a20534fdf@debian.org> <5489b743e2ee0052500a961aa99c3aa613e81caa.camel@redhat.com> <4a27c961-354e-270c-5dc7-789a5770fe5c@debian.org> Message-ID: On Fri, Sep 10, 2021 at 2:11 PM Corey Bryant wrote: > > > On Wed, Sep 30, 2020 at 11:53 AM Sean Mooney wrote: > > >> we do not know if there are other failrue >> neutron has a spereate issue which was tracked by >> https://github.com/eventlet/eventlet/issues/619 >> and nova hit the ssl issue with websockify and eventlets tracked by >> https://github.com/eventlet/eventlet/issues/632 >> >> so the issue is really eventlets is not compatiabley with dnspython 2.0 >> so before openstack can uncap dnspython eventlets need to gain support >> for dnspython 2.0 >> that should hopefully resolve the issues that nova, neutron and other >> projects are now hitting. >> >> it is unlikely that this is something we can resolve in openstack alone, >> not unless we are willing to monkeyptych >> eventlets and other dependcies so really we need to work with eventlets >> and or dnspython to resolve the incompatiablity >> caused by the dnspython changes in 2.0 >> > > It looks like there's been some progress on eventlet supporting dnspython > 2.0: > https://github.com/eventlet/eventlet/commit/aeb0390094a1c3f29bb4f25a8dab96587a86b3e8 > Does anyone know if there are plans to (attempt to) move to dnspython 2.0 in yoga? Thanks, Corey -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnsomor at gmail.com Wed Sep 29 20:41:19 2021 From: johnsomor at gmail.com (Michael Johnson) Date: Wed, 29 Sep 2021 13:41:19 -0700 Subject: Help with eventlet 0.26.1 and dnspython >= 2 In-Reply-To: References: <56ab7d1e-c8d6-8505-dd81-fb4a20534fdf@debian.org> <5489b743e2ee0052500a961aa99c3aa613e81caa.camel@redhat.com> <4a27c961-354e-270c-5dc7-789a5770fe5c@debian.org> Message-ID: I would like to for Designate. Assuming the eventlet issues get resolved. There is at least one bug in 1.16 that has been resolved on the 2.x chain and some of the new features set us up for new security related features. Michael On Wed, Sep 29, 2021 at 1:19 PM Corey Bryant wrote: > > > > On Fri, Sep 10, 2021 at 2:11 PM Corey Bryant wrote: >> >> >> >> On Wed, Sep 30, 2020 at 11:53 AM Sean Mooney wrote: >> >>> >>> we do not know if there are other failrue >>> neutron has a spereate issue which was tracked by https://github.com/eventlet/eventlet/issues/619 >>> and nova hit the ssl issue with websockify and eventlets tracked by https://github.com/eventlet/eventlet/issues/632 >>> >>> so the issue is really eventlets is not compatiabley with dnspython 2.0 >>> so before openstack can uncap dnspython eventlets need to gain support for dnspython 2.0 >>> that should hopefully resolve the issues that nova, neutron and other projects are now hitting. >>> >>> it is unlikely that this is something we can resolve in openstack alone, not unless we are willing to monkeyptych >>> eventlets and other dependcies so really we need to work with eventlets and or dnspython to resolve the incompatiablity >>> caused by the dnspython changes in 2.0 >> >> >> It looks like there's been some progress on eventlet supporting dnspython 2.0: https://github.com/eventlet/eventlet/commit/aeb0390094a1c3f29bb4f25a8dab96587a86b3e8 > > > Does anyone know if there are plans to (attempt to) move to dnspython 2.0 in yoga? > > Thanks, > Corey From kkchn.in at gmail.com Thu Sep 30 05:36:03 2021 From: kkchn.in at gmail.com (KK CHN) Date: Thu, 30 Sep 2021 11:06:03 +0530 Subject: DC DR high availability - Snapshot taking feature and user experience Message-ID: List, I am in the process of setting up a DC and DR sites with openstack(ussuri, qemu kvm with debian as base OS). I would like to know what all are the options in DC and DR sites for High availability . What the architecture methods to follow at DC and DR so that, If VM crashes at one physical machine at DC ( for example 3 controllers in DC), or Host machine crashes( RAM, Hardware failure etc. ) / machine power cable detached accidentally so that no services down / VMs and applications not down. How can achieve this ? share the best practices here. Also If I take snapshots of the running VMs ( what will the user experience ? will they in freeze / logged out from applications right now they are logged in ? ) . Can we avoid the service unavailable for users while taking snapshots ? To DR we are rsyncing these snapshots after converting each VM image to qcow2 then rsyncing . ( All these we are performing on Controller node . Generally 100 GB VM will output 16 GB qcow2 image for example. ) its taking 2 minutes for snapshot creation and 20 to 30 Minutes for qcow2 conversion . ) How many snapshots and conversion to qcow2( qemu-img convert) can be performed on this controller machine where we performing this operation . ( Can we apply parallel processing for this and how ? ) . Seet he controller specs where we perform this. The controller spec ( 20 core CPUs with 1.5 TB RAM total 160 processors ) Then copying to DR ( which is 200 KM away from our DC takes 4 minutes with rsync). Then we populate each VM with these copied qcow2 images and attaching the Network ips and nat there. Is this the practice or any other good way to perform this for a robust DC and DR setup . Kindly share your thoughts. Regards, Kris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayachander.it at gmail.com Thu Sep 30 07:03:42 2021 From: jayachander.it at gmail.com (Jay See) Date: Thu, 30 Sep 2021 09:03:42 +0200 Subject: [GPU] Request for suggestions for OpenStack deployment Message-ID: Hi, We are planning to buy some servers with GPU cards from DELL(R740/R750)/HPE/Supermicro with NVIDIA A100 40GB (GPU). I have the following questions: 1. What is the best ratio for Memory to GPU? or It doesn?t matter. 2. Anyone using A100? A100 has any known issues with OpenStack deployment? 3. If you have set up with similar server with GPU, could you share the server configuration? It will help us in deciding / identifying the components we have missed. Thanks in advance. Regards, ~ Jayachander. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Daniel.Pereira at windriver.com Thu Sep 30 12:33:51 2021 From: Daniel.Pereira at windriver.com (Daniel de Oliveira Pereira) Date: Thu, 30 Sep 2021 09:33:51 -0300 Subject: [dev][cinder] Consultation about new cinder-backup features In-Reply-To: <20210906132813.xsaxbsyyvf4ey4vm@localhost> References: <20210906132813.xsaxbsyyvf4ey4vm@localhost> Message-ID: On 06/09/2021 10:28, Gorka Eguileor wrote: > [Please note: This e-mail is from an EXTERNAL e-mail address] > > On 27/08, Daniel de Oliveira Pereira wrote: >> Hello everyone, >> >> We have prototyped some new features on Cinder for our clients, and we >> think that they are nice features and good candidates to be part of >> upstream Cinder, so we would like to get feedback from OpenStack >> community about these features and if you would be willing to accept >> them in upstream OpenStack. > > Hi Daniel, > > Thank you very much for your willingness to give back!!! > > >> >> Our team implemented the following features for cinder-backup service: >> >> 1. A multi-backend backup driver, that allow OpenStack users to >> choose, via API/CLI/Horizon, which backup driver (Ceph or NFS, in our >> prototype) will be used during a backup operation to create a new volume >> backup. > > This is a feature that has been discussed before, and e0ne already did > some of the prerequisites for it. > > >> 2. An improved NFS backup driver, that allow OpenStack users to back >> up their volumes to private NFS servers, providing the NFS hostpath at >> runtime via API/CLI/Horizon, while creating the volume backup. >> > > What about the username and password? Hi Gorka, thanks for your feedback. Our prototype doesn't support authentication using username/password, since this is a feature that NFS doesn't provide built-in support. > Can backups be restored from a remote location as well? Yes, if the location is the one where the backup was originally saved (same NFS hostpath), as the backup location is stored on Cinder backups table during the backup creation. It doesn't support restoring the backup from an arbitrary remote NFS server. > > This sounds like a very cool feature, but I'm not too comfortable with > having it in Cinder. > > The idea is that Cinder provides an abstraction and doesn't let users > know about implementation details. > > With that feature as it is a user could request a backup to an off-site > location that could result in congestion in one of the outbound > connections. I think this is a very good point, that we weren't taking into consideration in our prototype. > > I can only think of this being acceptable for admin users, and in that > case I think it would be best to use the multi-backup destination > feature instead. > > After all, how many times do we have to backup to a different location? > Maybe I'm missing a use case. Our clients have privacy and security concerns with the same NFS server being shared by OpenStack tenants to store volume backups, so they required cinder-backup to be able to back up volumes to private NFS servers. > > If the community thinks this as a desired feature I would encourage > adding it with a policy that disables it by default. > > >> Considering that cinder was configured to use the multi-backend backup >> driver, this is how it works: >> >> During a volume backup operation, the user provides a "location" >> parameter to indicate which backend will be used, and the backup >> hostpath, if applicable (for NFS driver), to create the volume backup. >> For instance: >> >> - Creating a backup using Ceph backend: >> $ openstack volume backup create --name --location >> ceph >> >> - Creating a backup using the improved NFS backend: >> $ openstack volume backup create --name --location >> nfs://my.nfs.server:/backups >> >> If the user chooses Ceph backend, the Ceph driver will be used to >> create the backup. If the user chooses the NFS backend, the improved NFS >> driver, previously mentioned, will be used to create the backup. >> >> The backup location, if provided, is stored on Cinder database, and >> can be seen fetching the backup details: >> $ openstack volume backup show >> >> Briefly, this is how the features were implemented: >> >> - Cinder API was updated to add an optional location parameter to >> "create backup" method. Horizon, and OpenStack and Cinder CLIs were >> updated accordingly, to handle the new parameter. >> - Cinder backup controller was updated to handle the backup location >> parameter, and a validator for the parameter was implemented using the >> oslo config library. >> - Cinder backup object model was updated to add a nullable location >> property, so that the backup location could be stored on cinder database. >> - a new backup driver base class, that extends BackupDriver and >> accepts a backup context object, was implemented to handle the backup >> configuration provided at runtime by the user. This new backup base >> class requires that the concrete drivers implement a method to validate >> the backup context (similar to BackupDriver.check_for_setup_error) >> - the 2 new backup drivers, previously mentioned, were implemented >> using these new backup base class. >> - in BackupManager class, the "service" attribute, that on upstream >> OpenStack holds the backup driver class name, was re-implemented as a >> factory function that accepts a backup context object and return an >> instance of a backup driver, according to the backup driver configured >> on cinder.conf file and the backup context provided at runtime by the user. >> - All the backup operations continue working as usual. >> > > When this feature was discussed upstream we liked the idea of > implementing this like we do multi-backends for the volume service, > adding backup-types. I found this approved spec [1] (that, I believe, is product of the work done by eOne that you mentioned before), but I couldn't find any work items in progress related to it. Do you know the current status of this spec? Is it ready to be implemented or is there some more work to be done until there? If we decide to work on its implementation, would be required to review, and possibly update, the spec for the current development cycle? [1] https://specs.openstack.org/openstack/cinder-specs/specs/victoria/backup-backends-configuration.html > > In latest code backup creation operations have been modified to go > through the scheduler, so that's a piece that is already implemented. > > >> Could you please let us know your thoughts about these features and if >> you would be open to adding them to upstream Cinder? If yes, we would be >> willing to submit the specs and work on the upstream implementation, if >> they are approved. >> >> Regards, >> Daniel Pereira >> > > I believe you will have the full community's support on the first idea > (though probably not on the proposed implementation). > > I'm not so sure on the second one, iti will most likely depend on the > use cases. Many times the reasons why features are dismissed upstream > is because there are no clear use cases that justify the addition of the > code. > > Looking forward to continuing this conversation at the PTG, IRC, in a > spec, or through here. > > Cheers, > Gorka. > From syedammad83 at gmail.com Thu Sep 30 13:36:37 2021 From: syedammad83 at gmail.com (Ammad Syed) Date: Thu, 30 Sep 2021 18:36:37 +0500 Subject: [nova] WSGI Apache vs Eventlet HTTP Servers Message-ID: Hi, I was reviewing below document. https://docs.openstack.org/nova/wallaby/user/wsgi.html Is apache2 wsgi provide better performance then eventlet http servers for nova api ? If yes, can you guys please share the sample or reference apache-nova-api file for wsgi ? - Ammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From satish.txt at gmail.com Thu Sep 30 14:29:23 2021 From: satish.txt at gmail.com (Satish Patel) Date: Thu, 30 Sep 2021 10:29:23 -0400 Subject: how dpdk improve guest performance Message-ID: Folks, If I am running dpdk in my compute nodes then sure it will reduce kernel overhead but how does it help to improve guest network performance? Inside guest vm it's going to use kernel to copy packet to userspace. in my example we run high performance haproxy which handles 300k to 500k connections but I have noticed performance is poor and lots of packet drops. If I run the same workload on SRIOV vms it works fine and I notice less kernel context switching inside the guest VM. I think when you run DPDK in compute then it only help compute but not your guest network performance. How do I improve guest performance also with dpdk based compute? From zaitcev at redhat.com Thu Sep 30 15:57:31 2021 From: zaitcev at redhat.com (Pete Zaitcev) Date: Thu, 30 Sep 2021 10:57:31 -0500 Subject: Open stack Swift In-Reply-To: <4d5a90a0-eb48-c299-6596-8749334b1a71@gmail.com> References: <4d5a90a0-eb48-c299-6596-8749334b1a71@gmail.com> Message-ID: <20210930105731.16354922@suzdal.zaitcev.lan> Please don't send people to SAIO when they are in production. The appropriate guide is the former "multinode" guide. It appears to be folded into OpenStack Installation here: https://docs.openstack.org/swift/wallaby/install/storage-install.html Also, the source is readable in RST files: https://opendev.org/openstack/swift/src/branch/master/doc/source/install -- Pete On Tue, 28 Sep 2021 18:07:42 +0900 Bernd Bausch wrote: > The installation guide > (https://docs.openstack.org/swift/wallaby/install/environment-networking.html) > uses two additional servers, but it should be possible to adapt the > instructions to a single-server setup. You could add Swift to the > controller, for example. There are also instructions for an all-in-one > setup https://docs.openstack.org/swift/latest/development_saio.html. > > On 2021/09/28 5:18 PM, Midhunlal Nb wrote: > > Hi team, > > I am using Openstack rocky setup with > > 1 controller node > > 2 compute node and 1 sStorage node,Now i am planning to configure a > > storage backup service Swift. > > Where do I need to configure swift,It needs an extra Node?Please > > explain the requirements. > > > > Thanks & Regards > > Midhunlal N B > > +918921245637 From katonalala at gmail.com Thu Sep 30 16:15:07 2021 From: katonalala at gmail.com (Lajos Katona) Date: Thu, 30 Sep 2021 18:15:07 +0200 Subject: [neutron] Drivers meeting agenda - 01.10.2021 Message-ID: Hi, The agenda for tomorrow's drivers meeting is at [1]. We have 1 RFE to discuss: * https://bugs.launchpad.net/neutron/+bug/1885261 Add stateless firewall support to OVS firewall ** It is Liu who would like to implement this, see logs of last team meeting at [2] ** Liu can't participate as there's some celebration in China, and he is on holiday, but still we can discuss it to have better understanding. [1] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers#Agenda [2] https://meetings.opendev.org/meetings/networking/2021/networking.2021-09-28-14.00.log.html#l-124 See you at the meeting tomorrow. Lajos Katona (lajoskatona -------------- next part -------------- An HTML attachment was scrubbed... URL: From juliaashleykreger at gmail.com Thu Sep 30 16:24:47 2021 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Thu, 30 Sep 2021 09:24:47 -0700 Subject: [ironic][molteniron][qa] Anyone still using MoltenIron? Message-ID: Out of curiosity, is anyone still using MotltenIron? A little historical context: It was originally tooling that came out of IBM to reserve physical nodes in a CI cluster in order to perform testing. It was never intended to be released as it was tooling *purely* for CI job usage. The reason I'm asking is that the ironic team is considering going ahead and retiring the repository. Thanks! -Julia From rlandy at redhat.com Thu Sep 30 17:54:02 2021 From: rlandy at redhat.com (Ronelle Landy) Date: Thu, 30 Sep 2021 13:54:02 -0400 Subject: [TripleO] Gate blocker - please hold rechecks Message-ID: Hello All, We have a gate blocker for tripleo at: https://bugs.launchpad.net/tripleo/+bug/1945682 This tox error is impacting tox jobs on multiple tripleo-related repos. A resolution is being worked on by infra. Please hold rechecks if you are rechecking for this failure. Please note that third-party OVB jobs are also down - following an update to DNS on a network (related bug: https://bugs.launchpad.net/tripleo/+bug/1945678). We will update this list when the error is cleared. Thanks for your patience! -------------- next part -------------- An HTML attachment was scrubbed... URL: From juliaashleykreger at gmail.com Thu Sep 30 18:27:00 2021 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Thu, 30 Sep 2021 11:27:00 -0700 Subject: [ironic][molteniron][qa] Anyone still using MoltenIron? In-Reply-To: References: Message-ID: I could have sworn that zuul had support to be basic selection/checkout of a resource instead of calling out to something else. Oh well! Good to know. Thanks Eric! On Thu, Sep 30, 2021 at 11:23 AM Barrera, Eric wrote: > > Hi Julia, > > Yea, the Zuul based Third Party CI I'm building uses Molten Iron to manage bare metal. I believe other Ironic 3rd Party CI projects are also using it. > > Though, I don't see it as an absolute necessity. > > > Regards, > Eric > > > > Internal Use - Confidential > > -----Original Message----- > From: Julia Kreger > Sent: Thursday, September 30, 2021 11:25 AM > To: openstack-discuss > Subject: [ironic][molteniron][qa] Anyone still using MoltenIron? > > > [EXTERNAL EMAIL] > > Out of curiosity, is anyone still using MotltenIron? > > A little historical context: It was originally tooling that came out of IBM to reserve physical nodes in a CI cluster in order to perform testing. It was never intended to be released as it was tooling > *purely* for CI job usage. > > The reason I'm asking is that the ironic team is considering going ahead and retiring the repository. > > Thanks! > > -Julia From fungi at yuggoth.org Thu Sep 30 19:20:28 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 30 Sep 2021 19:20:28 +0000 Subject: [TripleO] Gate blocker - please hold rechecks In-Reply-To: References: Message-ID: <20210930192027.tytawpypbirzylyk@yuggoth.org> On 2021-09-30 13:54:02 -0400 (-0400), Ronelle Landy wrote: > We have a gate blocker for tripleo at: > https://bugs.launchpad.net/tripleo/+bug/1945682 > > This tox error is impacting tox jobs on multiple tripleo-related repos. > A resolution is being worked on by infra. [...] This was due to a regression in a bug fix change[0] which merged to zuul-jobs, and the emergency revert[1] of that fix merged roughly an hour ago (18:17 UTC) so should no longer be causing new failures. I'm working on a regression test to exercise the tox feature TripleO was using and incorporate a solution for that so we can make sure it's not impacted when we re-merge[2] the original fix. [0] https://review.opendev.org/806612 [1] https://review.opendev.org/812001 [2] https://review.opendev.org/812005 -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From rlandy at redhat.com Thu Sep 30 20:17:07 2021 From: rlandy at redhat.com (Ronelle Landy) Date: Thu, 30 Sep 2021 16:17:07 -0400 Subject: [TripleO] Gate blocker - please hold rechecks In-Reply-To: <20210930192027.tytawpypbirzylyk@yuggoth.org> References: <20210930192027.tytawpypbirzylyk@yuggoth.org> Message-ID: On Thu, Sep 30, 2021 at 3:25 PM Jeremy Stanley wrote: > On 2021-09-30 13:54:02 -0400 (-0400), Ronelle Landy wrote: > > We have a gate blocker for tripleo at: > > https://bugs.launchpad.net/tripleo/+bug/1945682 > > > > This tox error is impacting tox jobs on multiple tripleo-related repos. > > A resolution is being worked on by infra. > [...] > > This was due to a regression in a bug fix change[0] which merged to > zuul-jobs, and the emergency revert[1] of that fix merged roughly an > hour ago (18:17 UTC) so should no longer be causing new failures. > I'm working on a regression test to exercise the tox feature TripleO > was using and incorporate a solution for that so we can make sure > it's not impacted when we re-merge[2] the original fix. > Thanks for the quick resolution here. Failed jobs are clearing the gate and will be rechecked if needed. > > [0] https://review.opendev.org/806612 > [1] https://review.opendev.org/812001 > [2] https://review.opendev.org/812005 > > -- > Jeremy Stanley > Note that the OVB issue is still ongoing. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosmaita.fossdev at gmail.com Thu Sep 30 22:40:44 2021 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Thu, 30 Sep 2021 18:40:44 -0400 Subject: [cinder][nova][requirements] RFE requested for os-brick Message-ID: Hello Requirements Team, The Cinder team recently became aware of a regression in os-brick that prevents Nova from attaching NVMe volumes [0]. A fix has been merged to master (yoga development) [1] and has been backported to stable/xena [2]. We've proposed a release of os-brick 5.0.1 from stable/xena [3], and are petitioning for an Requirements Freeze Exception to include os-brick 5.0.1 in the xena release. Thank you for your attention to this matter. [0] https://bugs.launchpad.net/os-brick/+bug/1945323 [1] https://review.opendev.org/c/openstack/os-brick/+/811944 [2] https://review.opendev.org/c/openstack/os-brick/+/812010 [3] https://review.opendev.org/c/openstack/releases/+/812048 From mthode at mthode.org Thu Sep 30 23:38:25 2021 From: mthode at mthode.org (Matthew Thode) Date: Thu, 30 Sep 2021 18:38:25 -0500 Subject: [cinder][nova][requirements] RFE requested for os-brick In-Reply-To: References: Message-ID: <20210930233825.rrbblojk3sspdon4@mthode.org> On 21-09-30 18:40:44, Brian Rosmaita wrote: > Hello Requirements Team, > > The Cinder team recently became aware of a regression in os-brick that > prevents Nova from attaching NVMe volumes [0]. A fix has been merged to > master (yoga development) [1] and has been backported to stable/xena [2]. > We've proposed a release of os-brick 5.0.1 from stable/xena [3], and are > petitioning for an Requirements Freeze Exception to include os-brick 5.0.1 > in the xena release. > > Thank you for your attention to this matter. > > > [0] https://bugs.launchpad.net/os-brick/+bug/1945323 > [1] https://review.opendev.org/c/openstack/os-brick/+/811944 > [2] https://review.opendev.org/c/openstack/os-brick/+/812010 > [3] https://review.opendev.org/c/openstack/releases/+/812048 > Already approved on irc, approving on ML too now :D -- Matthew Thode -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From Eric.Barrera at dell.com Thu Sep 30 18:23:02 2021 From: Eric.Barrera at dell.com (Barrera, Eric) Date: Thu, 30 Sep 2021 18:23:02 +0000 Subject: [ironic][molteniron][qa] Anyone still using MoltenIron? In-Reply-To: References: Message-ID: Hi Julia, Yea, the Zuul based Third Party CI I'm building uses Molten Iron to manage bare metal. I believe other Ironic 3rd Party CI projects are also using it. Though, I don't see it as an absolute necessity. Regards, Eric Internal Use - Confidential -----Original Message----- From: Julia Kreger Sent: Thursday, September 30, 2021 11:25 AM To: openstack-discuss Subject: [ironic][molteniron][qa] Anyone still using MoltenIron? [EXTERNAL EMAIL] Out of curiosity, is anyone still using MotltenIron? A little historical context: It was originally tooling that came out of IBM to reserve physical nodes in a CI cluster in order to perform testing. It was never intended to be released as it was tooling *purely* for CI job usage. The reason I'm asking is that the ironic team is considering going ahead and retiring the repository. Thanks! -Julia