August 2021 - openstack-discuss - lists.openstack.org

[magnum] [kolla-ansible] [kayobe] [Victoria]
by Tony Pearce 01 Sep '21

01 Sep '21

Testing out Kubernetes with Magnum project, deployed via kayobe on Victoria we have deployed an auto scaling cluster and have run into a problem and I'm not sure how to proceed. I understand that the cluster tried to scale up but the openstack project did not have enough CPU resources to accommodate it (error= Quota exceeded for cores: Requested 4, but already used 20 of 20 cores). So the situation is that the cluster shows "healthy" and "UPDATE_FAILED" but also kubectl commands are failing [1]. What is required to return the cluster back to a working status at this point? I have tried: - cluster resize to reduce number of workers - cluster resize to increase number of workers after increasing project quota - cluster resize and maintaining the same number of workers When trying any of the above, horizon shows an immediate error "Unable to resize given cluster" but magnum logs and heat logs do not show any log update at all at that time. - using "check stack" and resume stack in the stack horizon menu gives this error [2] Investigating the kubectl issue, it was noted that some services had failed on the master node [3]. Manual start as well as reboot the node did not bring up the services. Unfortunately I dont have ssh access to the master and no further information has been forthcoming with regards to logs for those service failures so I am unable to provide anything around that here. I found this link [4] so I decided to delete the master node then run "check" cluster again but the check cluster just fails in the same way except this time it fails saying that it cannot find the master [5] while the previous error was that it could not find a node. Ideally I would prefer to recover the cluster - whether this is still possible I am unsure. I can probably recreate this scenario again. What steps should be performed in this case to restore the cluster? [1] kubectl get no Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes) [2] Resource CHECK failed: ["['NotFound: resources[4].resources.kube-minion: Instance None could not be found. (HTTP 404) (Request-ID: req-6069ff6a-9eb6-4bce-bb25-4ef001ebc428)']. 'CHECK' not fully supported (see resources)"] [3] [systemd] Failed Units: 3 etcd.service heat-container-agent.service logrotate.service [4] https://bugzilla.redhat.com/show_bug.cgi?id=1459854 [5] ["['NotFound: resources.kube_masters.resources[0].resources.kube-master: Instance c6185e8e-1a98-4925-959b-0a56210b8c9e could not be found. (HTTP 404) (Request-ID: req-bdfcc853-7dbb-4022-9208-68b1ab31008a)']. 'CHECK' not fully supported (see resources)"]. Kind regards, Tony Pearce

3 6

[all][tc] Technical Committee next weekly meeting on Sept 2nd at 1500 UTC
by Ghanshyam Mann 01 Sep '21

01 Sep '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 2nd at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 1st, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

2 2

[nova][cyborg][gpu] GPU performance testing
by Ildiko Vancsa 01 Sep '21

01 Sep '21

Hi, I got a question about tools and practices to check GPU performance in an OpenStack environment that I need some help to answer. The question is about recommended GPU performance testing/benchmarking tools if there are a few that people in the community are using and would recommend? The scope of the testing work is to check GPU performance in OpenStack VMs (both virtualized and passthrough). All the help and pointers are very much appreciated! Thanks, Ildikó

6 9

[largescale-sig] Next meeting: Sept 1st, 15utc
by Thierry Carrez 01 Sep '21

01 Sep '21

Hi everyone, Our next Large Scale SIG meeting will be this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210901T15 A number of topics have already been added to the agenda, including discussing the topic of our next OpenInfra.Live show! Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[Kolla][Kolla-Ansible] Ironic Node Cleaning Failed
by Anirudh Gupta 01 Sep '21

01 Sep '21

Hi Team, We have deployed 2 node kolla ansible *12.0.0* in order to deploy openstack *wallaby* release. We have also enabled ironic in order to provision the bare metal nodes. On each server we have 3 nics - *eno1* - OAM for external connectivity and endpoint's publicURL - *eno2* - Mgmt for internal communication between various openstack services. - *ens2f0* - Data Interface Corresponding to this we have defined the following fields in globals.yml - kolla_base_distro: "centos" - kolla_install_type: "source" - openstack_release: "wallaby" - network_interface: "eno2" # MGMT interface - kolla_external_vip_interface: "eno1" # OAM Interface - kolla_internal_vip_address: "192.168.10.3" # MGMT Subnet free ip - kolla_external_vip_address: "10.0.1.136" # OAM subnet free IP - neutron_external_interface: "ens2f0" # Data Interface - enable_neutron_provider_networks: "yes" Note: Only relevant fields are being shown in this query Also, for ironic following fields have been defined in globals.yml - enable_ironic: "yes" - enable_ironic_neutron_agent: "{{ enable_neutron | bool and enable_ironic | bool }}" - enable_horizon_ironic: "{{ enable_ironic | bool }}" - ironic_dnsmasq_interface: "*ens2f0*" # Data interface - ironic_dnsmasq_dhcp_range: "20.20.20.10,20.20.20.100" - ironic_dnsmasq_boot_file: "pxelinux.0" - ironic_cleaning_network: "public1" - ironic_dnsmasq_default_gateway: "20.20.20.1" After successful deployment, a flat provider network with the name public1 is being created in openstack using the below commands: - openstack network create public1 --provider-network-type flat --provider-physical-network physnet1 - openstack subnet create subnet1 --network public1 --subnet-range 20.20.20.0/24 --allocation-pool start=20.20.20.10,end=20.20.20.100 --ip-version=4 --gateway=20.20.20.1 --dhcp Issue/Queries: - Is the configuration done in globals.yml correct or is there anything else that needs to be done in order to separate control and data plane traffic? - Also I have set automated_cleaning as "true" in ironic-conductor conatiner settings.But after creating the baremetal node, we run "node manage" command which runs successfully. Running "*openstack baremetal node provide <node id>"* command powers on the machine, sets the boot mode on Network Boot but no DHCP request for that particular mac is obtained on the controller. Is there anything I am missing that needs to be done in order to make ironic work? Note: I have also verified that the nic is PXE enabled in system configuration setting Regards Anirudh Gupta

3 15

[cinder][nova] fsfreeze hooks issues with cinder snapshot/backup
by Arthur Outhenin-Chalandre 01 Sep '21

01 Sep '21

Hello, We are trying to trigger an fsfreeze via a cinder backup or snapshot. We confirmed that the fsfreeze hooks are actually called with a nova snapshot with `/var/log/qga-fsfreeze-hook.log` in the VM, but we can't achieve the same thing with a cinder backup/snapshot attached to the same instance. We are using Wallaby, libvirt, RBD for cinder volumes and RBD as well for cinder-backup. According to this (old) spec [0], cinder should call the `quiesce()` method in nova during backup/snapshot. We looked in the cinder code and couldn't find any clear evidence that this method is actually called by cinder (but we may have missed something). We added some debug messages on quiesce/can_quiesce/require_quiesce/... in `nova/virt/libvirt/driver.py` and they are never called with a cinder backup/snapshot in our setup while they are (and succeed) if we do a nova snapshot. We are starting to suspect that something is missing in cinder, but it could very well be a problem with our setup as well... Does someone use this feature or know if it should be working/implemented? [0]: https://wiki.openstack.org/wiki/Cinder/QuiescedSnapshotWithQemuGuestAgent#C… Cheers, -- Arthur Outhenin-Chalandre

4 4

[release][heat][ironic][requirements][swift][OpenStackSDK][vitrage][barbican][heat][horizon][ironic][neutron][QA] Cycle With Intermediary Unreleased Deliverables
by Herve Beraud 01 Sep '21

01 Sep '21

Hello teams Quick reminder that we'll need a release very soon for a number of deliverables following a cycle-with-intermediary release model but which have not done *any* release yet in the Xena cycle: ansible-role-atos-hsm ansible-role-lunasa-hsm ansible-role-thales-hsm heat-agents horizon ironic-prometheus-exporter ironic-python-agent-builder ironic-ui ovn-octavia-provider patrole python-openstackclient requirements swift tempest vitrage-dashboard vitrage Those should be released ASAP, and in all cases before the week of 13 September, 2021, so that we have a release to include in the final Xena release. Thanks for your attention. -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

4 6

Need some help to understand "Baremetal Provision Configuration" file
by wodel youchi 01 Sep '21

01 Sep '21

Hi, I am trying to deploy openstack Wallaby. I need some help to understand the meaning of this file "Baremetal Provision Configuration" Here is the example given in the documentation : First in : https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deploym… : Provision Baremetal Instances > - name: Controller > count: 3 > defaults: > networks: > - network: ctlplane > subnet: ctlplane-subnet > vif: true > - network: external > subnet: external_subnet > *- network: internalapi > subnet: internal_api_subnet01* > - network: storage > subnet: storage_subnet01 > - network: storagemgmt > subnet: storage_mgmt_subnet01 > - network: tenant > subnet: tenant_subnet01 > network_config: > template: /home/stack/nic-config/controller.j2 > default_route_network: > - external- name: Compute > count: 100 > defaults: > networks: > - network: ctlplane > subnet: ctlplane-subnet > vif: true > *- network: internalapi > subnet: internal_api_subnet02* > - network: tenant > subnet: tenant_subnet02 > - network: storage > subnet: storage_subnet02 > network_config: > template: /home/stack/nic-config/compute.j2 > > Second in : https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/provisi… : Baremetal Provision Configuration > - name: Controller > count: 1 > hostname_format: controller-%index% > ansible_playbooks: > - playbook: bm-deploy-playbook.yaml > defaults: > profile: control > networks: > - network: external > subnet: external_subnet > *- network: internal_api > subnet: internal_api_subnet01* > - network: storage > subnet: storage_subnet01 > - network: storage_mgmt > subnet: storage_mgmt_subnet01 > - network: tenant > subnet: tenant_subnet01 > network_config: > template: templates/multiple_nics/multiple_nics_dvr.j2 > default_route_network: > - external- name: Compute > count: 1 > hostname_format: compute-%index% > ansible_playbooks: > - playbook: bm-deploy-playbook.yaml > defaults: > profile: compute-leaf2 > networks: > *- network: internal_api > subnet: internal_api_subnet02* > - network: tenant > subnet: tenant_subnet02 > - network: storage > subnet: storage_subnet02 > network_config: > template: templates/multiple_nics/multiple_nics_dvr.j2 > > My questions : 1 - Does the name of the network have to match the name of the network (name_lower) in network_data.yaml? because there is an underscore missing in the first example 2 - What is the meaning of the numbers in the subnet name of the network "*internal_api_subnet01 *for controllers and *internal_api_subnet02 *for compute nodes" ? why a different number? What is the meaning of it? I have tried to create the *overcloud-baremetal-deployed.yaml* file several times, and every time I get errors here. Regards.

3 10

Re: Wallaby Magnum Issue
by Mohammed Naser 31 Aug '21

31 Aug '21

Please keep replies on list so others can help too. I don’t know how well tested the Swarm driver is at this point. I believe most Magnum users are using it for Kubernetes only. On Fri, Aug 20, 2021 at 1:05 AM Karera Tony <tonykarera(a)gmail.com> wrote: > Hello Naser, > > Please check below. > > openstack coe cluster template create swarm-cluster-template1 \ > --image fedora-atomic-latest \ > --external-network External_1700\ > --dns-nameserver 8.8.8.8 \ > --master-flavor m1.small \ > --flavor m1.small \ > --coe swarm > openstack coe cluster create swarm-cluster \ > --cluster-template swarm-cluster-template \ > --master-count 1 \ > --node-count 2 \ > --keypair Newkey > > Regards > > Tony Karera > > > > > On Fri, Aug 20, 2021 at 7:03 AM Mohammed Naser <mnaser(a)vexxhost.com> > wrote: > >> What does your cluster template and cluster create command look like? >> >> On Fri, Aug 20, 2021 at 12:59 AM Karera Tony <tonykarera(a)gmail.com> >> wrote: >> >>> Hello Wang, >>> >>> Thanks for the feedback. >>> >>> Unfortunately Octavia is not deployed in my environment (at least not >>> yet) and LB is not enabled on either the cluster template or the cluster >>> itself. >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Fri, Aug 20, 2021 at 6:52 AM feilong <feilong(a)catalyst.net.nz> wrote: >>> >>>> Hi Karera, >>>> >>>> It's probably a bug. If you do have Octavia deployed, can you try to >>>> not disable the LB and see how it goes? >>>> >>>> >>>> On 20/08/21 4:18 pm, Karera Tony wrote: >>>> >>>> Hello Team, >>>> >>>> I deployed Openstack Wallby on Ubuntu20 and enabled Magum, however when >>>> I create a cluster I get the error below. >>>> >>>> >>>> *Status Reason ERROR: Property error: : resources.api_lb.properties: : >>>> Property allowed_cidrs not assigned* >>>> Can someone advise on where I could be wrong. Btw, I disabled load >>>> balancer while creating the cluster. >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> -- >>>> Cheers & Best regards, >>>> ------------------------------------------------------------------------------ >>>> Feilong Wang (王飞龙) (he/him) >>>> Head of Research & Development >>>> >>>> Catalyst Cloud >>>> Aotearoa's own >>>> >>>> Mob: +64 21 0832 6348 | www.catalystcloud.nz >>>> Level 6, 150 Willis Street, Wellington 6011, New Zealand <https://www.google.com/maps/search/150+Willis+Street,+Wellington+6011,+New+…> >>>> >>>> CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. >>>> It may contain privileged, confidential or copyright information. If you are >>>> not the named recipient, any use, reliance upon, disclosure or copying of this >>>> email or its attachments is unauthorised. If you have received this email in >>>> error, please reply via email or call +64 21 0832 6348. >>>> ------------------------------------------------------------------------------ >>>> >>>> -- >> Mohammed Naser >> VEXXHOST, Inc. >> > -- Mohammed Naser VEXXHOST, Inc.

6 34

How to stop the whole system gracefully ?
by Tommy Sway 31 Aug '21

31 Aug '21

My system is installed using kola-ansible. The database was inconsistent due to the last unexpected shutdown, and the system could not start normally, which was very troublesome. I would like to ask what command should be used to shut down the entire system gracefully in an emergency ? Thanks.

2 3