From eblock at nde.ag Mon Jul 1 07:31:14 2019 From: eblock at nde.ag (Eugen Block) Date: Mon, 01 Jul 2019 07:31:14 +0000 Subject: glance image upload error [Errno 32] Broken pipe In-Reply-To: Message-ID: <20190701073114.Horde.OFwjjcHd__j9TawGaNnt3iK@webmail.nde.ag> Hi, have you checked if your glance services are still up and running? The error message indicates that it's not ceph but the glance endpoint that's not there. Regards, Eugen Zitat von Satish Patel : > I have installed opnestack-ansible and integrate glance with ceph > storage, first day when i upload image it works but today when i am > trying to upload image i am getting this error > > [root at ostack-infra-2-1-utility-container-c166f549 ~]# openstack image > create --file cirros-0.3.4-x86_64-disk.raw --container-format bare > --disk-format raw --public cirros-0.3.4-tmp > Error finding address for > http://172.28.8.9:9292/v2/images/8f3456aa-52fc-4b4a-8b11-dfbadb8e88ca/file: > [Errno 32] Broken pipe > > I do have enough space on ceph storage, i am not seeing any error on > glance logs also which help me. > > [root at ostack-infra-01-ceph-mon-container-692bea95 root]# ceph df detail > GLOBAL: > SIZE AVAIL RAW USED %RAW USED OBJECTS > 9314G 3526G 5787G 62.14 245k > POOLS: > NAME ID QUOTA OBJECTS QUOTA BYTES USED > %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW > USED > images 6 95367M N/A 22522M > 3.62 584G 2839 2839 2594k 36245 > 67568M > vms 7 N/A N/A 1912G > 76.58 584G 248494 242k 6567M 3363M > 5738G > volumes 8 N/A N/A 0 > 0 584G 0 0 0 0 > > backups 9 N/A N/A 0 > 0 584G 0 0 0 0 > > metrics 10 N/A N/A 0 > 0 584G 0 0 0 0 > From mark at stackhpc.com Mon Jul 1 08:10:34 2019 From: mark at stackhpc.com (Mark Goddard) Date: Mon, 1 Jul 2019 09:10:34 +0100 Subject: [kolla-ansible] migration In-Reply-To: References: Message-ID: It sounds like you got quite close to having this working. I'd suggest debugging this instance build failure. One difference with kolla is that we run libvirt inside a container. Have you stopped libvirt from running on the host? Mark On Sun, 30 Jun 2019 at 09:55, Ignazio Cassano wrote: > > Hi Mark, > let me to explain what I am trying. > I have a queens installation based on centos and pacemaker with some instances and heat stacks. > I would like to have another installation with same instances, projects, stacks ....I'd like to have same uuid for all objects (users,projects instances and so on, because it is controlled by a cloud management platform we wrote. > > I stopped controllers on old queens installation backupping the openstack database. > I installed the new kolla openstack queens on new three controllers with same addresses of the old intallation , vip as well. > One of the three controllers is also a kvm node on queens. > I stopped all containeres except rabbit,keepalive,rabbit,haproxy and mariadb. > I deleted al openstack db on mariadb container and I imported the old tables, changing the address of rabbit for pointing to the new rabbit cluster. > I restarded containers. > Changing the rabbit address on old kvm nodes, I can see the old virtual machines and I can open console on them. > I can see all networks (tenant and provider) of al installation, but when I try to create a new instance on the new kvm, it remains in buiding state. > Seems it cannot aquire an address. > Storage between old and new installation are shred on nfs NETAPP, so I can see cinder volumes. > I suppose db structure is different between a kolla installation and a manual instaltion !? > What is wrong ? > Thanks > Ignazio > > > > > Il giorno gio 27 giu 2019 alle ore 16:44 Mark Goddard ha scritto: >> >> On Thu, 27 Jun 2019 at 14:46, Ignazio Cassano wrote: >> > >> > Sorry, for my question. >> > It does not need to change anything because endpoints refer to haproxy vips. >> > So if your new glance works fine you change haproxy backends for glance. >> > Regards >> > Ignazio >> >> That's correct - only the haproxy backend needs to be updated. >> >> > >> > >> > Il giorno gio 27 giu 2019 alle ore 15:21 Ignazio Cassano ha scritto: >> >> >> >> Hello Mark, >> >> let me to verify if I understood your method. >> >> >> >> You have old controllers,haproxy,mariadb and nova computes. >> >> You installed three new controllers but kolla.ansible inventory contains old mariadb and old rabbit servers. >> >> You are deployng single service on new controllers staring with glance. >> >> When you deploy glance on new controllers, it changes the glance endpoint on old mariadb db ? >> >> Regards >> >> Ignazio >> >> >> >> Il giorno gio 27 giu 2019 alle ore 10:52 Mark Goddard ha scritto: >> >>> >> >>> On Wed, 26 Jun 2019 at 19:34, Ignazio Cassano wrote: >> >>> > >> >>> > Hello, >> >>> > Anyone have tried to migrate an existing openstack installation to kolla containers? >> >>> >> >>> Hi, >> >>> >> >>> I'm aware of two people currently working on that. Gregory Orange and >> >>> one of my colleagues, Pierre Riteau. Pierre is away currently, so I >> >>> hope he doesn't mind me quoting him from an email to Gregory. >> >>> >> >>> Mark >> >>> >> >>> "I am indeed working on a similar migration using Kolla Ansible with >> >>> Kayobe, starting from a non-containerised OpenStack deployment based >> >>> on CentOS RPMs. >> >>> Existing OpenStack services are deployed across several controller >> >>> nodes and all sit behind HAProxy, including for internal endpoints. >> >>> We have additional controller nodes that we use to deploy >> >>> containerised services. If you don't have the luxury of additional >> >>> nodes, it will be more difficult as you will need to avoid processes >> >>> clashing when listening on the same port. >> >>> >> >>> The method I am using resembles your second suggestion, however I am >> >>> deploying only one containerised service at a time, in order to >> >>> validate each of them independently. >> >>> I use the --tags option of kolla-ansible to restrict Ansible to >> >>> specific roles, and when I am happy with the resulting configuration I >> >>> update HAProxy to point to the new controllers. >> >>> >> >>> As long as the configuration matches, this should be completely >> >>> transparent for purely HTTP-based services like Glance. You need to be >> >>> more careful with services that include components listening for RPC, >> >>> such as Nova: if the new nova.conf is incorrect and you've deployed a >> >>> nova-conductor that uses it, you could get failed instances launches. >> >>> Some roles depend on others: if you are deploying the >> >>> neutron-openvswitch-agent, you need to run the openvswitch role as >> >>> well. >> >>> >> >>> I suggest starting with migrating Glance as it doesn't have any >> >>> internal services and is easy to validate. Note that properly >> >>> migrating Keystone requires keeping existing Fernet keys around, so >> >>> any token stays valid until the time it is expected to stop working >> >>> (which is fairly complex, see >> >>> https://bugs.launchpad.net/kolla-ansible/+bug/1809469). >> >>> >> >>> While initially I was using an approach similar to your first >> >>> suggestion, it can have side effects since Kolla Ansible uses these >> >>> variables when templating configuration. As an example, most services >> >>> will only have notifications enabled if enable_ceilometer is true. >> >>> >> >>> I've added existing control plane nodes to the Kolla Ansible inventory >> >>> as separate groups, which allows me to use the existing database and >> >>> RabbitMQ for the containerised services. >> >>> For example, instead of: >> >>> >> >>> [mariadb:children] >> >>> control >> >>> >> >>> you may have: >> >>> >> >>> [mariadb:children] >> >>> oldcontrol_db >> >>> >> >>> I still have to perform the migration of these underlying services to >> >>> the new control plane, I will let you know if there is any hurdle. >> >>> >> >>> A few random things to note: >> >>> >> >>> - if run on existing control plane hosts, the baremetal role removes >> >>> some packages listed in `redhat_pkg_removals` which can trigger the >> >>> removal of OpenStack dependencies using them! I've changed this >> >>> variable to an empty list. >> >>> - compare your existing deployment with a Kolla Ansible one to check >> >>> for differences in endpoints, configuration files, database users, >> >>> service users, etc. For Heat, Kolla uses the domain heat_user_domain, >> >>> while your existing deployment may use another one (and this is >> >>> hardcoded in the Kolla Heat image). Kolla Ansible uses the "service" >> >>> project while a couple of deployments I worked with were using >> >>> "services". This shouldn't matter, except there was a bug in Kolla >> >>> which prevented it from setting the roles correctly: >> >>> https://bugs.launchpad.net/kolla/+bug/1791896 (now fixed in latest >> >>> Rocky and Queens images) >> >>> - the ml2_conf.ini generated for Neutron generates physical network >> >>> names like physnet1, physnet2… you may want to override >> >>> bridge_mappings completely. >> >>> - although sometimes it could be easier to change your existing >> >>> deployment to match Kolla Ansible settings, rather than configure >> >>> Kolla Ansible to match your deployment." >> >>> >> >>> > Thanks >> >>> > Ignazio >> >>> > From bence.romsics at gmail.com Mon Jul 1 09:26:21 2019 From: bence.romsics at gmail.com (Bence Romsics) Date: Mon, 1 Jul 2019 11:26:21 +0200 Subject: [neutron] bug deputy report for week of 2019-06-24 Message-ID: Hi Everybody, These are the new bugs of last week: Critical: * https://bugs.launchpad.net/neutron/+bug/1834298 Neutron-fwaas python 3.7 job is broken Fix tested, needs +2 from neutron-fwaas cores: https://review.opendev.org/667736 * https://bugs.launchpad.net/neutron/+bug/1833902 Revert resize tests are failing in jobs with iptables_hybrid fw driver A nova bug breaking neutron gate. Fix merged: https://review.opendev.org/667035 High: * https://bugs.launchpad.net/neutron/+bug/1834257 dhcp-agent can overwhelm neutron server with dhcp_ready_on_ports RPC calls Fix in progress: https://review.opendev.org/659274 * https://bugs.launchpad.net/neutron/+bug/1834484 [QoS] qos_plugin._extend_port_resource_request is killing port retrieval performance Fix in progress: https://review.opendev.org/667981, https://review.opendev.org/667998 Medium: * https://bugs.launchpad.net/neutron/+bug/1834308 [DVR][DB] too many slow query during agent restart Needs more information to reproduce * https://bugs.launchpad.net/neutron/+bug/1834753 TC filter priority parameter in Pyroute is "prio" Fix in progress: https://review.opendev.org/668308 * https://bugs.launchpad.net/neutron/+bug/1834825 Rule to prevent SNAT for router's internal traffic is wrong Fix in progress: https://review.opendev.org/668378 RFE: * https://bugs.launchpad.net/neutron/+bug/1834174 [RFE] Add support for IPoIB interface driver * https://bugs.launchpad.net/neutron/+bug/1834176 [RFE] Add support for per-physnet interface driver Cheers, Bence (rubasov) From ignaziocassano at gmail.com Mon Jul 1 10:10:58 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Mon, 1 Jul 2019 12:10:58 +0200 Subject: [kolla-ansible] migration In-Reply-To: References: Message-ID: Hi Mark, the kolla environment has a new kvm host hosted on controller node. The kolla openstack can see instances on old kvm nodes and it can access them using vnc console porvided by dashboard, but cannot run new instances on new kvm host :-( Regards Ignazio Il giorno lun 1 lug 2019 alle ore 10:10 Mark Goddard ha scritto: > It sounds like you got quite close to having this working. I'd suggest > debugging this instance build failure. One difference with kolla is > that we run libvirt inside a container. Have you stopped libvirt from > running on the host? > Mark > > On Sun, 30 Jun 2019 at 09:55, Ignazio Cassano > wrote: > > > > Hi Mark, > > let me to explain what I am trying. > > I have a queens installation based on centos and pacemaker with some > instances and heat stacks. > > I would like to have another installation with same instances, projects, > stacks ....I'd like to have same uuid for all objects (users,projects > instances and so on, because it is controlled by a cloud management > platform we wrote. > > > > I stopped controllers on old queens installation backupping the > openstack database. > > I installed the new kolla openstack queens on new three controllers with > same addresses of the old intallation , vip as well. > > One of the three controllers is also a kvm node on queens. > > I stopped all containeres except rabbit,keepalive,rabbit,haproxy and > mariadb. > > I deleted al openstack db on mariadb container and I imported the old > tables, changing the address of rabbit for pointing to the new rabbit > cluster. > > I restarded containers. > > Changing the rabbit address on old kvm nodes, I can see the old virtual > machines and I can open console on them. > > I can see all networks (tenant and provider) of al installation, but > when I try to create a new instance on the new kvm, it remains in buiding > state. > > Seems it cannot aquire an address. > > Storage between old and new installation are shred on nfs NETAPP, so I > can see cinder volumes. > > I suppose db structure is different between a kolla installation and a > manual instaltion !? > > What is wrong ? > > Thanks > > Ignazio > > > > > > > > > > Il giorno gio 27 giu 2019 alle ore 16:44 Mark Goddard > ha scritto: > >> > >> On Thu, 27 Jun 2019 at 14:46, Ignazio Cassano > wrote: > >> > > >> > Sorry, for my question. > >> > It does not need to change anything because endpoints refer to > haproxy vips. > >> > So if your new glance works fine you change haproxy backends for > glance. > >> > Regards > >> > Ignazio > >> > >> That's correct - only the haproxy backend needs to be updated. > >> > >> > > >> > > >> > Il giorno gio 27 giu 2019 alle ore 15:21 Ignazio Cassano < > ignaziocassano at gmail.com> ha scritto: > >> >> > >> >> Hello Mark, > >> >> let me to verify if I understood your method. > >> >> > >> >> You have old controllers,haproxy,mariadb and nova computes. > >> >> You installed three new controllers but kolla.ansible inventory > contains old mariadb and old rabbit servers. > >> >> You are deployng single service on new controllers staring with > glance. > >> >> When you deploy glance on new controllers, it changes the glance > endpoint on old mariadb db ? > >> >> Regards > >> >> Ignazio > >> >> > >> >> Il giorno gio 27 giu 2019 alle ore 10:52 Mark Goddard < > mark at stackhpc.com> ha scritto: > >> >>> > >> >>> On Wed, 26 Jun 2019 at 19:34, Ignazio Cassano < > ignaziocassano at gmail.com> wrote: > >> >>> > > >> >>> > Hello, > >> >>> > Anyone have tried to migrate an existing openstack installation > to kolla containers? > >> >>> > >> >>> Hi, > >> >>> > >> >>> I'm aware of two people currently working on that. Gregory Orange > and > >> >>> one of my colleagues, Pierre Riteau. Pierre is away currently, so I > >> >>> hope he doesn't mind me quoting him from an email to Gregory. > >> >>> > >> >>> Mark > >> >>> > >> >>> "I am indeed working on a similar migration using Kolla Ansible with > >> >>> Kayobe, starting from a non-containerised OpenStack deployment based > >> >>> on CentOS RPMs. > >> >>> Existing OpenStack services are deployed across several controller > >> >>> nodes and all sit behind HAProxy, including for internal endpoints. > >> >>> We have additional controller nodes that we use to deploy > >> >>> containerised services. If you don't have the luxury of additional > >> >>> nodes, it will be more difficult as you will need to avoid processes > >> >>> clashing when listening on the same port. > >> >>> > >> >>> The method I am using resembles your second suggestion, however I am > >> >>> deploying only one containerised service at a time, in order to > >> >>> validate each of them independently. > >> >>> I use the --tags option of kolla-ansible to restrict Ansible to > >> >>> specific roles, and when I am happy with the resulting > configuration I > >> >>> update HAProxy to point to the new controllers. > >> >>> > >> >>> As long as the configuration matches, this should be completely > >> >>> transparent for purely HTTP-based services like Glance. You need to > be > >> >>> more careful with services that include components listening for > RPC, > >> >>> such as Nova: if the new nova.conf is incorrect and you've deployed > a > >> >>> nova-conductor that uses it, you could get failed instances > launches. > >> >>> Some roles depend on others: if you are deploying the > >> >>> neutron-openvswitch-agent, you need to run the openvswitch role as > >> >>> well. > >> >>> > >> >>> I suggest starting with migrating Glance as it doesn't have any > >> >>> internal services and is easy to validate. Note that properly > >> >>> migrating Keystone requires keeping existing Fernet keys around, so > >> >>> any token stays valid until the time it is expected to stop working > >> >>> (which is fairly complex, see > >> >>> https://bugs.launchpad.net/kolla-ansible/+bug/1809469). > >> >>> > >> >>> While initially I was using an approach similar to your first > >> >>> suggestion, it can have side effects since Kolla Ansible uses these > >> >>> variables when templating configuration. As an example, most > services > >> >>> will only have notifications enabled if enable_ceilometer is true. > >> >>> > >> >>> I've added existing control plane nodes to the Kolla Ansible > inventory > >> >>> as separate groups, which allows me to use the existing database and > >> >>> RabbitMQ for the containerised services. > >> >>> For example, instead of: > >> >>> > >> >>> [mariadb:children] > >> >>> control > >> >>> > >> >>> you may have: > >> >>> > >> >>> [mariadb:children] > >> >>> oldcontrol_db > >> >>> > >> >>> I still have to perform the migration of these underlying services > to > >> >>> the new control plane, I will let you know if there is any hurdle. > >> >>> > >> >>> A few random things to note: > >> >>> > >> >>> - if run on existing control plane hosts, the baremetal role removes > >> >>> some packages listed in `redhat_pkg_removals` which can trigger the > >> >>> removal of OpenStack dependencies using them! I've changed this > >> >>> variable to an empty list. > >> >>> - compare your existing deployment with a Kolla Ansible one to check > >> >>> for differences in endpoints, configuration files, database users, > >> >>> service users, etc. For Heat, Kolla uses the domain > heat_user_domain, > >> >>> while your existing deployment may use another one (and this is > >> >>> hardcoded in the Kolla Heat image). Kolla Ansible uses the "service" > >> >>> project while a couple of deployments I worked with were using > >> >>> "services". This shouldn't matter, except there was a bug in Kolla > >> >>> which prevented it from setting the roles correctly: > >> >>> https://bugs.launchpad.net/kolla/+bug/1791896 (now fixed in latest > >> >>> Rocky and Queens images) > >> >>> - the ml2_conf.ini generated for Neutron generates physical network > >> >>> names like physnet1, physnet2… you may want to override > >> >>> bridge_mappings completely. > >> >>> - although sometimes it could be easier to change your existing > >> >>> deployment to match Kolla Ansible settings, rather than configure > >> >>> Kolla Ansible to match your deployment." > >> >>> > >> >>> > Thanks > >> >>> > Ignazio > >> >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Mon Jul 1 10:38:13 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Mon, 1 Jul 2019 12:38:13 +0200 Subject: [kolla-ansible] migration In-Reply-To: References: Message-ID: PS I presume the problem is neutron, because instances on new kvm nodes remain in building state e do not aquire address. Probably the netron db imported from old openstack installation has some difrrences ....probably I must check defferences from old and new neutron services configuration files. Ignazio Il giorno lun 1 lug 2019 alle ore 10:10 Mark Goddard ha scritto: > It sounds like you got quite close to having this working. I'd suggest > debugging this instance build failure. One difference with kolla is > that we run libvirt inside a container. Have you stopped libvirt from > running on the host? > Mark > > On Sun, 30 Jun 2019 at 09:55, Ignazio Cassano > wrote: > > > > Hi Mark, > > let me to explain what I am trying. > > I have a queens installation based on centos and pacemaker with some > instances and heat stacks. > > I would like to have another installation with same instances, projects, > stacks ....I'd like to have same uuid for all objects (users,projects > instances and so on, because it is controlled by a cloud management > platform we wrote. > > > > I stopped controllers on old queens installation backupping the > openstack database. > > I installed the new kolla openstack queens on new three controllers with > same addresses of the old intallation , vip as well. > > One of the three controllers is also a kvm node on queens. > > I stopped all containeres except rabbit,keepalive,rabbit,haproxy and > mariadb. > > I deleted al openstack db on mariadb container and I imported the old > tables, changing the address of rabbit for pointing to the new rabbit > cluster. > > I restarded containers. > > Changing the rabbit address on old kvm nodes, I can see the old virtual > machines and I can open console on them. > > I can see all networks (tenant and provider) of al installation, but > when I try to create a new instance on the new kvm, it remains in buiding > state. > > Seems it cannot aquire an address. > > Storage between old and new installation are shred on nfs NETAPP, so I > can see cinder volumes. > > I suppose db structure is different between a kolla installation and a > manual instaltion !? > > What is wrong ? > > Thanks > > Ignazio > > > > > > > > > > Il giorno gio 27 giu 2019 alle ore 16:44 Mark Goddard > ha scritto: > >> > >> On Thu, 27 Jun 2019 at 14:46, Ignazio Cassano > wrote: > >> > > >> > Sorry, for my question. > >> > It does not need to change anything because endpoints refer to > haproxy vips. > >> > So if your new glance works fine you change haproxy backends for > glance. > >> > Regards > >> > Ignazio > >> > >> That's correct - only the haproxy backend needs to be updated. > >> > >> > > >> > > >> > Il giorno gio 27 giu 2019 alle ore 15:21 Ignazio Cassano < > ignaziocassano at gmail.com> ha scritto: > >> >> > >> >> Hello Mark, > >> >> let me to verify if I understood your method. > >> >> > >> >> You have old controllers,haproxy,mariadb and nova computes. > >> >> You installed three new controllers but kolla.ansible inventory > contains old mariadb and old rabbit servers. > >> >> You are deployng single service on new controllers staring with > glance. > >> >> When you deploy glance on new controllers, it changes the glance > endpoint on old mariadb db ? > >> >> Regards > >> >> Ignazio > >> >> > >> >> Il giorno gio 27 giu 2019 alle ore 10:52 Mark Goddard < > mark at stackhpc.com> ha scritto: > >> >>> > >> >>> On Wed, 26 Jun 2019 at 19:34, Ignazio Cassano < > ignaziocassano at gmail.com> wrote: > >> >>> > > >> >>> > Hello, > >> >>> > Anyone have tried to migrate an existing openstack installation > to kolla containers? > >> >>> > >> >>> Hi, > >> >>> > >> >>> I'm aware of two people currently working on that. Gregory Orange > and > >> >>> one of my colleagues, Pierre Riteau. Pierre is away currently, so I > >> >>> hope he doesn't mind me quoting him from an email to Gregory. > >> >>> > >> >>> Mark > >> >>> > >> >>> "I am indeed working on a similar migration using Kolla Ansible with > >> >>> Kayobe, starting from a non-containerised OpenStack deployment based > >> >>> on CentOS RPMs. > >> >>> Existing OpenStack services are deployed across several controller > >> >>> nodes and all sit behind HAProxy, including for internal endpoints. > >> >>> We have additional controller nodes that we use to deploy > >> >>> containerised services. If you don't have the luxury of additional > >> >>> nodes, it will be more difficult as you will need to avoid processes > >> >>> clashing when listening on the same port. > >> >>> > >> >>> The method I am using resembles your second suggestion, however I am > >> >>> deploying only one containerised service at a time, in order to > >> >>> validate each of them independently. > >> >>> I use the --tags option of kolla-ansible to restrict Ansible to > >> >>> specific roles, and when I am happy with the resulting > configuration I > >> >>> update HAProxy to point to the new controllers. > >> >>> > >> >>> As long as the configuration matches, this should be completely > >> >>> transparent for purely HTTP-based services like Glance. You need to > be > >> >>> more careful with services that include components listening for > RPC, > >> >>> such as Nova: if the new nova.conf is incorrect and you've deployed > a > >> >>> nova-conductor that uses it, you could get failed instances > launches. > >> >>> Some roles depend on others: if you are deploying the > >> >>> neutron-openvswitch-agent, you need to run the openvswitch role as > >> >>> well. > >> >>> > >> >>> I suggest starting with migrating Glance as it doesn't have any > >> >>> internal services and is easy to validate. Note that properly > >> >>> migrating Keystone requires keeping existing Fernet keys around, so > >> >>> any token stays valid until the time it is expected to stop working > >> >>> (which is fairly complex, see > >> >>> https://bugs.launchpad.net/kolla-ansible/+bug/1809469). > >> >>> > >> >>> While initially I was using an approach similar to your first > >> >>> suggestion, it can have side effects since Kolla Ansible uses these > >> >>> variables when templating configuration. As an example, most > services > >> >>> will only have notifications enabled if enable_ceilometer is true. > >> >>> > >> >>> I've added existing control plane nodes to the Kolla Ansible > inventory > >> >>> as separate groups, which allows me to use the existing database and > >> >>> RabbitMQ for the containerised services. > >> >>> For example, instead of: > >> >>> > >> >>> [mariadb:children] > >> >>> control > >> >>> > >> >>> you may have: > >> >>> > >> >>> [mariadb:children] > >> >>> oldcontrol_db > >> >>> > >> >>> I still have to perform the migration of these underlying services > to > >> >>> the new control plane, I will let you know if there is any hurdle. > >> >>> > >> >>> A few random things to note: > >> >>> > >> >>> - if run on existing control plane hosts, the baremetal role removes > >> >>> some packages listed in `redhat_pkg_removals` which can trigger the > >> >>> removal of OpenStack dependencies using them! I've changed this > >> >>> variable to an empty list. > >> >>> - compare your existing deployment with a Kolla Ansible one to check > >> >>> for differences in endpoints, configuration files, database users, > >> >>> service users, etc. For Heat, Kolla uses the domain > heat_user_domain, > >> >>> while your existing deployment may use another one (and this is > >> >>> hardcoded in the Kolla Heat image). Kolla Ansible uses the "service" > >> >>> project while a couple of deployments I worked with were using > >> >>> "services". This shouldn't matter, except there was a bug in Kolla > >> >>> which prevented it from setting the roles correctly: > >> >>> https://bugs.launchpad.net/kolla/+bug/1791896 (now fixed in latest > >> >>> Rocky and Queens images) > >> >>> - the ml2_conf.ini generated for Neutron generates physical network > >> >>> names like physnet1, physnet2… you may want to override > >> >>> bridge_mappings completely. > >> >>> - although sometimes it could be easier to change your existing > >> >>> deployment to match Kolla Ansible settings, rather than configure > >> >>> Kolla Ansible to match your deployment." > >> >>> > >> >>> > Thanks > >> >>> > Ignazio > >> >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From missile0407 at gmail.com Mon Jul 1 11:23:19 2019 From: missile0407 at gmail.com (Eddie Yen) Date: Mon, 1 Jul 2019 19:23:19 +0800 Subject: [kolla][ceph] Cache OSDs didn't stay in the root=cache after ceph deployment. Message-ID: Hi, I'm using stable/rocky to try ceph cache tiering. Now I'm facing a one issue. I chose one SSD to become cache tier disk. And set below options in globals.yml. ceph_enable_cache = "yes" ceph_target_max_byte= "" ceph_target_max_objects = "" ceph_cache_mode = "writeback" And the default OSD type is bluestore. It will bootstrap the cache disk and create another OSD container. And also create the root bucket called "cache". then set the cache rule to every cache pools. The problem is, that OSD didn't stay at "cache" bucket, it still stay at "default" bucket. That caused the services can't access to the Ceph normally. Especially deploying Gnocchi. When error occurred, I manually set that OSD to the cache bucket then re-deploy, and everything is normal now. But still a strange issue that it stay in the wrong bucket. Did I miss something during deployment? Or what can I do? Many thanks, Eddie. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry at openstack.org Mon Jul 1 11:33:08 2019 From: thierry at openstack.org (Thierry Carrez) Date: Mon, 1 Jul 2019 13:33:08 +0200 Subject: [Release-job-failures] Release of openstack/ansible-role-redhat-subscription failed In-Reply-To: References: Message-ID: <1b761937-ed22-9a5a-ccd8-823846cc3150@openstack.org> zuul at openstack.org wrote: > Build failed. > > - release-openstack-python http://logs.openstack.org/59/59f40f154a5302df1ee4230f35f87d261e0bf9eb/release/release-openstack-python/8b78b28/ : POST_FAILURE in 2m 09s > - announce-release announce-release : SKIPPED > - propose-update-constraints propose-update-constraints : SKIPPED Upload to PyPI at the end of the release was rejected due to: "The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information." Impact: ansible-role-redhat-subscription 1.0.3 was produced and published, but not uploaded to PyPI. Remediation: once the description is fixed, we'll have to request a 1.0.4 to fix the situation. -- Thierry Carrez (ttx) From mnaser at vexxhost.com Mon Jul 1 11:36:43 2019 From: mnaser at vexxhost.com (Mohammed Naser) Date: Mon, 1 Jul 2019 06:36:43 -0500 Subject: [kolla-ansible] migration In-Reply-To: References: Message-ID: You should check your cell mapping records inside Nova. They're probably not right of you moved your database and rabbit Sorry for top posting this is from a phone. On Mon., Jul. 1, 2019, 5:46 a.m. Ignazio Cassano, wrote: > PS > I presume the problem is neutron, because instances on new kvm nodes > remain in building state e do not aquire address. > Probably the netron db imported from old openstack installation has some > difrrences ....probably I must check defferences from old and new neutron > services configuration files. > Ignazio > > Il giorno lun 1 lug 2019 alle ore 10:10 Mark Goddard > ha scritto: > >> It sounds like you got quite close to having this working. I'd suggest >> debugging this instance build failure. One difference with kolla is >> that we run libvirt inside a container. Have you stopped libvirt from >> running on the host? >> Mark >> >> On Sun, 30 Jun 2019 at 09:55, Ignazio Cassano >> wrote: >> > >> > Hi Mark, >> > let me to explain what I am trying. >> > I have a queens installation based on centos and pacemaker with some >> instances and heat stacks. >> > I would like to have another installation with same instances, >> projects, stacks ....I'd like to have same uuid for all objects >> (users,projects instances and so on, because it is controlled by a cloud >> management platform we wrote. >> > >> > I stopped controllers on old queens installation backupping the >> openstack database. >> > I installed the new kolla openstack queens on new three controllers >> with same addresses of the old intallation , vip as well. >> > One of the three controllers is also a kvm node on queens. >> > I stopped all containeres except rabbit,keepalive,rabbit,haproxy and >> mariadb. >> > I deleted al openstack db on mariadb container and I imported the old >> tables, changing the address of rabbit for pointing to the new rabbit >> cluster. >> > I restarded containers. >> > Changing the rabbit address on old kvm nodes, I can see the old virtual >> machines and I can open console on them. >> > I can see all networks (tenant and provider) of al installation, but >> when I try to create a new instance on the new kvm, it remains in buiding >> state. >> > Seems it cannot aquire an address. >> > Storage between old and new installation are shred on nfs NETAPP, so I >> can see cinder volumes. >> > I suppose db structure is different between a kolla installation and a >> manual instaltion !? >> > What is wrong ? >> > Thanks >> > Ignazio >> > >> > >> > >> > >> > Il giorno gio 27 giu 2019 alle ore 16:44 Mark Goddard < >> mark at stackhpc.com> ha scritto: >> >> >> >> On Thu, 27 Jun 2019 at 14:46, Ignazio Cassano < >> ignaziocassano at gmail.com> wrote: >> >> > >> >> > Sorry, for my question. >> >> > It does not need to change anything because endpoints refer to >> haproxy vips. >> >> > So if your new glance works fine you change haproxy backends for >> glance. >> >> > Regards >> >> > Ignazio >> >> >> >> That's correct - only the haproxy backend needs to be updated. >> >> >> >> > >> >> > >> >> > Il giorno gio 27 giu 2019 alle ore 15:21 Ignazio Cassano < >> ignaziocassano at gmail.com> ha scritto: >> >> >> >> >> >> Hello Mark, >> >> >> let me to verify if I understood your method. >> >> >> >> >> >> You have old controllers,haproxy,mariadb and nova computes. >> >> >> You installed three new controllers but kolla.ansible inventory >> contains old mariadb and old rabbit servers. >> >> >> You are deployng single service on new controllers staring with >> glance. >> >> >> When you deploy glance on new controllers, it changes the glance >> endpoint on old mariadb db ? >> >> >> Regards >> >> >> Ignazio >> >> >> >> >> >> Il giorno gio 27 giu 2019 alle ore 10:52 Mark Goddard < >> mark at stackhpc.com> ha scritto: >> >> >>> >> >> >>> On Wed, 26 Jun 2019 at 19:34, Ignazio Cassano < >> ignaziocassano at gmail.com> wrote: >> >> >>> > >> >> >>> > Hello, >> >> >>> > Anyone have tried to migrate an existing openstack installation >> to kolla containers? >> >> >>> >> >> >>> Hi, >> >> >>> >> >> >>> I'm aware of two people currently working on that. Gregory Orange >> and >> >> >>> one of my colleagues, Pierre Riteau. Pierre is away currently, so I >> >> >>> hope he doesn't mind me quoting him from an email to Gregory. >> >> >>> >> >> >>> Mark >> >> >>> >> >> >>> "I am indeed working on a similar migration using Kolla Ansible >> with >> >> >>> Kayobe, starting from a non-containerised OpenStack deployment >> based >> >> >>> on CentOS RPMs. >> >> >>> Existing OpenStack services are deployed across several controller >> >> >>> nodes and all sit behind HAProxy, including for internal endpoints. >> >> >>> We have additional controller nodes that we use to deploy >> >> >>> containerised services. If you don't have the luxury of additional >> >> >>> nodes, it will be more difficult as you will need to avoid >> processes >> >> >>> clashing when listening on the same port. >> >> >>> >> >> >>> The method I am using resembles your second suggestion, however I >> am >> >> >>> deploying only one containerised service at a time, in order to >> >> >>> validate each of them independently. >> >> >>> I use the --tags option of kolla-ansible to restrict Ansible to >> >> >>> specific roles, and when I am happy with the resulting >> configuration I >> >> >>> update HAProxy to point to the new controllers. >> >> >>> >> >> >>> As long as the configuration matches, this should be completely >> >> >>> transparent for purely HTTP-based services like Glance. You need >> to be >> >> >>> more careful with services that include components listening for >> RPC, >> >> >>> such as Nova: if the new nova.conf is incorrect and you've >> deployed a >> >> >>> nova-conductor that uses it, you could get failed instances >> launches. >> >> >>> Some roles depend on others: if you are deploying the >> >> >>> neutron-openvswitch-agent, you need to run the openvswitch role as >> >> >>> well. >> >> >>> >> >> >>> I suggest starting with migrating Glance as it doesn't have any >> >> >>> internal services and is easy to validate. Note that properly >> >> >>> migrating Keystone requires keeping existing Fernet keys around, so >> >> >>> any token stays valid until the time it is expected to stop working >> >> >>> (which is fairly complex, see >> >> >>> https://bugs.launchpad.net/kolla-ansible/+bug/1809469). >> >> >>> >> >> >>> While initially I was using an approach similar to your first >> >> >>> suggestion, it can have side effects since Kolla Ansible uses these >> >> >>> variables when templating configuration. As an example, most >> services >> >> >>> will only have notifications enabled if enable_ceilometer is true. >> >> >>> >> >> >>> I've added existing control plane nodes to the Kolla Ansible >> inventory >> >> >>> as separate groups, which allows me to use the existing database >> and >> >> >>> RabbitMQ for the containerised services. >> >> >>> For example, instead of: >> >> >>> >> >> >>> [mariadb:children] >> >> >>> control >> >> >>> >> >> >>> you may have: >> >> >>> >> >> >>> [mariadb:children] >> >> >>> oldcontrol_db >> >> >>> >> >> >>> I still have to perform the migration of these underlying services >> to >> >> >>> the new control plane, I will let you know if there is any hurdle. >> >> >>> >> >> >>> A few random things to note: >> >> >>> >> >> >>> - if run on existing control plane hosts, the baremetal role >> removes >> >> >>> some packages listed in `redhat_pkg_removals` which can trigger the >> >> >>> removal of OpenStack dependencies using them! I've changed this >> >> >>> variable to an empty list. >> >> >>> - compare your existing deployment with a Kolla Ansible one to >> check >> >> >>> for differences in endpoints, configuration files, database users, >> >> >>> service users, etc. For Heat, Kolla uses the domain >> heat_user_domain, >> >> >>> while your existing deployment may use another one (and this is >> >> >>> hardcoded in the Kolla Heat image). Kolla Ansible uses the >> "service" >> >> >>> project while a couple of deployments I worked with were using >> >> >>> "services". This shouldn't matter, except there was a bug in Kolla >> >> >>> which prevented it from setting the roles correctly: >> >> >>> https://bugs.launchpad.net/kolla/+bug/1791896 (now fixed in latest >> >> >>> Rocky and Queens images) >> >> >>> - the ml2_conf.ini generated for Neutron generates physical network >> >> >>> names like physnet1, physnet2… you may want to override >> >> >>> bridge_mappings completely. >> >> >>> - although sometimes it could be easier to change your existing >> >> >>> deployment to match Kolla Ansible settings, rather than configure >> >> >>> Kolla Ansible to match your deployment." >> >> >>> >> >> >>> > Thanks >> >> >>> > Ignazio >> >> >>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sombrafam at gmail.com Mon Jul 1 11:44:30 2019 From: sombrafam at gmail.com (Erlon Cruz) Date: Mon, 1 Jul 2019 08:44:30 -0300 Subject: [cinder] Deprecating driver versions In-Reply-To: <59de8659-6a0b-62b5-8bee-ed0fdf622cf9@gmail.com> References: <20190628075012.ndwk52gabg2akqvx@localhost> <59de8659-6a0b-62b5-8bee-ed0fdf622cf9@gmail.com> Message-ID: Hi Jay, No problem. I just wanted to know if other vendors/maintainers shared the same problems and concerns we have and if we could have an uniform solutions across all drivers which is not the case. Erlon Em sáb, 29 de jun de 2019 às 15:19, Jay S. Bryant escreveu: > Erlon, > > I appreciate the goal here but I agree with Gorka here. > > The drivers are the vendor's responsibilities and they version them as > they wish. I think updating the devref some best practices > recommendations would be good and maybe come to agreement between the > cores on what the best practices are so that we can try to enforce it to > some extent through reviews. That is probably the best way forward. > > Jay > > On 6/28/2019 2:50 AM, Gorka Eguileor wrote: > > On 27/06, Erlon Cruz wrote: > >> Hey folks, > >> > >> Driver versions has being a source of a lot of confusions with > costumers. > >> Most of our drivers > >> have a version number and history that are updated as the developers > adds > >> new fixes and > >> features. Drivers also have a VERSION variable in the version class that > >> should be bumped by > >> developers. The problem with that is: > >> > >> - sometimes folks from the community just push patches on drivers, > and > >> its hard to bump > >> every vendor version correctly; > >> - that relies in the human factor to remember adding it, and usually > >> that fails; > >> - if we create a bugfix and bump the version, the backport to older > >> branches will carry the > >> version, which will not reflect the correct driver code; > >> > >> So, the solution I'm proposing for this is that we use the Cinder > >> versions[1] and remove all > >> version strings for drivers. Every new release we get a version. For > stable > >> versions, from time to > >> time the PTL bumps the stable version and we have an accurate ways to > >> describe the code. > >> If we need to backport and send something to the costumer, we can do the > >> backport, poke > >> the PTL, and he will generate another version which can be downloaded on > >> github or via PIP, > >> and present the version to our costumers. > >> > >> So, what are your thought around this? Anyone else has had problems with > >> that? What would > >> be the implications of removing the driver version strings? > >> > >> Erlon > >> > > Hi Erlon, > > > > I am personally against removing the drivers versions, as I find them > > convenient and think they are good practice. > > > > A possible solution for the driver versioning is for a driver to > > designate a minor version per OpenStack release and use the patch > > version to track changes. This way one can always backport a patch and > > will just need to increase the patch version in the backport patch. > > > > Maybe we can have this formally described in our devref. We tell > > driver developers they can do whatever they want with the versioning in > > master, but backports must not backport the version as it is and instead > > increase the patch version. > > > > What do you think? > > > > If I remember correctly there are some drivers that only increase the > > version once per release. > > > > Cheers, > > Gorka. > > > >> [1] https://releases.openstack.org/teams/cinder.html > >> [2] > >> > https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/solidfire.py#L237 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangtrinhnt at gmail.com Mon Jul 1 11:45:49 2019 From: dangtrinhnt at gmail.com (Trinh Nguyen) Date: Mon, 1 Jul 2019 20:45:49 +0900 Subject: [searchlight] Cannot hold meeting today Message-ID: Daer team, Unfortunately, I'm in the middle of job transition and cannot hold the meeting today. I will propose a new meeting time this week. Please do not hesitate to contact me if you have any questions. Bests, -- *Trinh Nguyen* *www.edlab.xyz * -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Mon Jul 1 11:50:42 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Mon, 1 Jul 2019 13:50:42 +0200 Subject: [kolla-ansible] migration In-Reply-To: References: Message-ID: I checked them and I modified for fitting to new installation thanks Ignazio Il giorno lun 1 lug 2019 alle ore 13:36 Mohammed Naser ha scritto: > You should check your cell mapping records inside Nova. They're probably > not right of you moved your database and rabbit > > Sorry for top posting this is from a phone. > > On Mon., Jul. 1, 2019, 5:46 a.m. Ignazio Cassano, < > ignaziocassano at gmail.com> wrote: > >> PS >> I presume the problem is neutron, because instances on new kvm nodes >> remain in building state e do not aquire address. >> Probably the netron db imported from old openstack installation has some >> difrrences ....probably I must check defferences from old and new neutron >> services configuration files. >> Ignazio >> >> Il giorno lun 1 lug 2019 alle ore 10:10 Mark Goddard >> ha scritto: >> >>> It sounds like you got quite close to having this working. I'd suggest >>> debugging this instance build failure. One difference with kolla is >>> that we run libvirt inside a container. Have you stopped libvirt from >>> running on the host? >>> Mark >>> >>> On Sun, 30 Jun 2019 at 09:55, Ignazio Cassano >>> wrote: >>> > >>> > Hi Mark, >>> > let me to explain what I am trying. >>> > I have a queens installation based on centos and pacemaker with some >>> instances and heat stacks. >>> > I would like to have another installation with same instances, >>> projects, stacks ....I'd like to have same uuid for all objects >>> (users,projects instances and so on, because it is controlled by a cloud >>> management platform we wrote. >>> > >>> > I stopped controllers on old queens installation backupping the >>> openstack database. >>> > I installed the new kolla openstack queens on new three controllers >>> with same addresses of the old intallation , vip as well. >>> > One of the three controllers is also a kvm node on queens. >>> > I stopped all containeres except rabbit,keepalive,rabbit,haproxy and >>> mariadb. >>> > I deleted al openstack db on mariadb container and I imported the old >>> tables, changing the address of rabbit for pointing to the new rabbit >>> cluster. >>> > I restarded containers. >>> > Changing the rabbit address on old kvm nodes, I can see the old >>> virtual machines and I can open console on them. >>> > I can see all networks (tenant and provider) of al installation, but >>> when I try to create a new instance on the new kvm, it remains in buiding >>> state. >>> > Seems it cannot aquire an address. >>> > Storage between old and new installation are shred on nfs NETAPP, so I >>> can see cinder volumes. >>> > I suppose db structure is different between a kolla installation and a >>> manual instaltion !? >>> > What is wrong ? >>> > Thanks >>> > Ignazio >>> > >>> > >>> > >>> > >>> > Il giorno gio 27 giu 2019 alle ore 16:44 Mark Goddard < >>> mark at stackhpc.com> ha scritto: >>> >> >>> >> On Thu, 27 Jun 2019 at 14:46, Ignazio Cassano < >>> ignaziocassano at gmail.com> wrote: >>> >> > >>> >> > Sorry, for my question. >>> >> > It does not need to change anything because endpoints refer to >>> haproxy vips. >>> >> > So if your new glance works fine you change haproxy backends for >>> glance. >>> >> > Regards >>> >> > Ignazio >>> >> >>> >> That's correct - only the haproxy backend needs to be updated. >>> >> >>> >> > >>> >> > >>> >> > Il giorno gio 27 giu 2019 alle ore 15:21 Ignazio Cassano < >>> ignaziocassano at gmail.com> ha scritto: >>> >> >> >>> >> >> Hello Mark, >>> >> >> let me to verify if I understood your method. >>> >> >> >>> >> >> You have old controllers,haproxy,mariadb and nova computes. >>> >> >> You installed three new controllers but kolla.ansible inventory >>> contains old mariadb and old rabbit servers. >>> >> >> You are deployng single service on new controllers staring with >>> glance. >>> >> >> When you deploy glance on new controllers, it changes the glance >>> endpoint on old mariadb db ? >>> >> >> Regards >>> >> >> Ignazio >>> >> >> >>> >> >> Il giorno gio 27 giu 2019 alle ore 10:52 Mark Goddard < >>> mark at stackhpc.com> ha scritto: >>> >> >>> >>> >> >>> On Wed, 26 Jun 2019 at 19:34, Ignazio Cassano < >>> ignaziocassano at gmail.com> wrote: >>> >> >>> > >>> >> >>> > Hello, >>> >> >>> > Anyone have tried to migrate an existing openstack installation >>> to kolla containers? >>> >> >>> >>> >> >>> Hi, >>> >> >>> >>> >> >>> I'm aware of two people currently working on that. Gregory Orange >>> and >>> >> >>> one of my colleagues, Pierre Riteau. Pierre is away currently, so >>> I >>> >> >>> hope he doesn't mind me quoting him from an email to Gregory. >>> >> >>> >>> >> >>> Mark >>> >> >>> >>> >> >>> "I am indeed working on a similar migration using Kolla Ansible >>> with >>> >> >>> Kayobe, starting from a non-containerised OpenStack deployment >>> based >>> >> >>> on CentOS RPMs. >>> >> >>> Existing OpenStack services are deployed across several controller >>> >> >>> nodes and all sit behind HAProxy, including for internal >>> endpoints. >>> >> >>> We have additional controller nodes that we use to deploy >>> >> >>> containerised services. If you don't have the luxury of additional >>> >> >>> nodes, it will be more difficult as you will need to avoid >>> processes >>> >> >>> clashing when listening on the same port. >>> >> >>> >>> >> >>> The method I am using resembles your second suggestion, however I >>> am >>> >> >>> deploying only one containerised service at a time, in order to >>> >> >>> validate each of them independently. >>> >> >>> I use the --tags option of kolla-ansible to restrict Ansible to >>> >> >>> specific roles, and when I am happy with the resulting >>> configuration I >>> >> >>> update HAProxy to point to the new controllers. >>> >> >>> >>> >> >>> As long as the configuration matches, this should be completely >>> >> >>> transparent for purely HTTP-based services like Glance. You need >>> to be >>> >> >>> more careful with services that include components listening for >>> RPC, >>> >> >>> such as Nova: if the new nova.conf is incorrect and you've >>> deployed a >>> >> >>> nova-conductor that uses it, you could get failed instances >>> launches. >>> >> >>> Some roles depend on others: if you are deploying the >>> >> >>> neutron-openvswitch-agent, you need to run the openvswitch role as >>> >> >>> well. >>> >> >>> >>> >> >>> I suggest starting with migrating Glance as it doesn't have any >>> >> >>> internal services and is easy to validate. Note that properly >>> >> >>> migrating Keystone requires keeping existing Fernet keys around, >>> so >>> >> >>> any token stays valid until the time it is expected to stop >>> working >>> >> >>> (which is fairly complex, see >>> >> >>> https://bugs.launchpad.net/kolla-ansible/+bug/1809469). >>> >> >>> >>> >> >>> While initially I was using an approach similar to your first >>> >> >>> suggestion, it can have side effects since Kolla Ansible uses >>> these >>> >> >>> variables when templating configuration. As an example, most >>> services >>> >> >>> will only have notifications enabled if enable_ceilometer is true. >>> >> >>> >>> >> >>> I've added existing control plane nodes to the Kolla Ansible >>> inventory >>> >> >>> as separate groups, which allows me to use the existing database >>> and >>> >> >>> RabbitMQ for the containerised services. >>> >> >>> For example, instead of: >>> >> >>> >>> >> >>> [mariadb:children] >>> >> >>> control >>> >> >>> >>> >> >>> you may have: >>> >> >>> >>> >> >>> [mariadb:children] >>> >> >>> oldcontrol_db >>> >> >>> >>> >> >>> I still have to perform the migration of these underlying >>> services to >>> >> >>> the new control plane, I will let you know if there is any hurdle. >>> >> >>> >>> >> >>> A few random things to note: >>> >> >>> >>> >> >>> - if run on existing control plane hosts, the baremetal role >>> removes >>> >> >>> some packages listed in `redhat_pkg_removals` which can trigger >>> the >>> >> >>> removal of OpenStack dependencies using them! I've changed this >>> >> >>> variable to an empty list. >>> >> >>> - compare your existing deployment with a Kolla Ansible one to >>> check >>> >> >>> for differences in endpoints, configuration files, database users, >>> >> >>> service users, etc. For Heat, Kolla uses the domain >>> heat_user_domain, >>> >> >>> while your existing deployment may use another one (and this is >>> >> >>> hardcoded in the Kolla Heat image). Kolla Ansible uses the >>> "service" >>> >> >>> project while a couple of deployments I worked with were using >>> >> >>> "services". This shouldn't matter, except there was a bug in Kolla >>> >> >>> which prevented it from setting the roles correctly: >>> >> >>> https://bugs.launchpad.net/kolla/+bug/1791896 (now fixed in >>> latest >>> >> >>> Rocky and Queens images) >>> >> >>> - the ml2_conf.ini generated for Neutron generates physical >>> network >>> >> >>> names like physnet1, physnet2… you may want to override >>> >> >>> bridge_mappings completely. >>> >> >>> - although sometimes it could be easier to change your existing >>> >> >>> deployment to match Kolla Ansible settings, rather than configure >>> >> >>> Kolla Ansible to match your deployment." >>> >> >>> >>> >> >>> > Thanks >>> >> >>> > Ignazio >>> >> >>> > >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Mon Jul 1 12:05:33 2019 From: hberaud at redhat.com (Herve Beraud) Date: Mon, 1 Jul 2019 14:05:33 +0200 Subject: [Release-job-failures] Release of openstack/ansible-role-redhat-subscription failed In-Reply-To: <1b761937-ed22-9a5a-ccd8-823846cc3150@openstack.org> References: <1b761937-ed22-9a5a-ccd8-823846cc3150@openstack.org> Message-ID: Hello, We have the same issue on podman. https://travis-ci.org/containers/python-podman/jobs/551905955 I don't know why this is happening there, we used ` long_description_content_type="text/markdown"` [1] so normally is not an issue here... maybe pypi team have changed things on their side and we need to take care of it. I will take a look. We also use pbr on podman. Maybe these changes (Read description file as utf-8) [2] have introduced some unexpected behaviours. When I try to build a dist locally everything seems to be good... I don't see any issues. [1] https://github.com/containers/python-podman/commit/d1cc16a69e858faf1c81cde67cb198af22740ba4 [2] https://github.com/openstack/pbr/commit/17f9439e9f9026dd3e8cae1e917a78e80195152c Le lun. 1 juil. 2019 à 13:39, Thierry Carrez a écrit : > zuul at openstack.org wrote: > > Build failed. > > > > - release-openstack-python > http://logs.openstack.org/59/59f40f154a5302df1ee4230f35f87d261e0bf9eb/release/release-openstack-python/8b78b28/ > : POST_FAILURE in 2m 09s > > - announce-release announce-release : SKIPPED > > - propose-update-constraints propose-update-constraints : SKIPPED > > Upload to PyPI at the end of the release was rejected due to: > > "The description failed to render in the default format of > reStructuredText. See https://pypi.org/help/#description-content-type > for more information." > > Impact: ansible-role-redhat-subscription 1.0.3 was produced and > published, but not uploaded to PyPI. > > Remediation: once the description is fixed, we'll have to request a > 1.0.4 to fix the situation. > > -- > Thierry Carrez (ttx) > > -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Mon Jul 1 12:31:16 2019 From: hberaud at redhat.com (Herve Beraud) Date: Mon, 1 Jul 2019 14:31:16 +0200 Subject: [Release-job-failures] Release of openstack/ansible-role-redhat-subscription failed In-Reply-To: References: <1b761937-ed22-9a5a-ccd8-823846cc3150@openstack.org> Message-ID: I think it's related to: - https://github.com/pypa/warehouse/pull/5835 - https://github.com/pypa/warehouse/issues/5890 Le lun. 1 juil. 2019 à 14:05, Herve Beraud a écrit : > Hello, > > We have the same issue on podman. > > https://travis-ci.org/containers/python-podman/jobs/551905955 > > I don't know why this is happening there, we used ` > long_description_content_type="text/markdown"` [1] so normally is not an > issue here... maybe pypi team have changed things on their side and we need > to take care of it. I will take a look. > > We also use pbr on podman. > > Maybe these changes (Read description file as utf-8) [2] have introduced > some unexpected behaviours. > > When I try to build a dist locally everything seems to be good... I don't > see any issues. > > [1] > https://github.com/containers/python-podman/commit/d1cc16a69e858faf1c81cde67cb198af22740ba4 > [2] > https://github.com/openstack/pbr/commit/17f9439e9f9026dd3e8cae1e917a78e80195152c > > Le lun. 1 juil. 2019 à 13:39, Thierry Carrez a > écrit : > >> zuul at openstack.org wrote: >> > Build failed. >> > >> > - release-openstack-python >> http://logs.openstack.org/59/59f40f154a5302df1ee4230f35f87d261e0bf9eb/release/release-openstack-python/8b78b28/ >> : POST_FAILURE in 2m 09s >> > - announce-release announce-release : SKIPPED >> > - propose-update-constraints propose-update-constraints : SKIPPED >> >> Upload to PyPI at the end of the release was rejected due to: >> >> "The description failed to render in the default format of >> reStructuredText. See https://pypi.org/help/#description-content-type >> for more information." >> >> Impact: ansible-role-redhat-subscription 1.0.3 was produced and >> published, but not uploaded to PyPI. >> >> Remediation: once the description is fixed, we'll have to request a >> 1.0.4 to fix the situation. >> >> -- >> Thierry Carrez (ttx) >> >> > > -- > Hervé Beraud > Senior Software Engineer > Red Hat - Openstack Oslo > irc: hberaud > -----BEGIN PGP SIGNATURE----- > > wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ > Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ > RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP > F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G > 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g > glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw > m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ > hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 > qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y > F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 > B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O > v6rDpkeNksZ9fFSyoY2o > =ECSj > -----END PGP SIGNATURE----- > > -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Mon Jul 1 12:32:07 2019 From: eblock at nde.ag (Eugen Block) Date: Mon, 01 Jul 2019 12:32:07 +0000 Subject: [kolla][ceph] Cache OSDs didn't stay in the root=cache after ceph deployment. In-Reply-To: Message-ID: <20190701123207.Horde.Y7WzBt0rnChxKZBSc3ktF0Z@webmail.nde.ag> Hi, although I'm not familiar with kolla I can comment on the ceph part. > The problem is, that OSD didn't stay at "cache" bucket, it still stay at > "default" bucket. I'm not sure how the deployment process with kolla works and what exactly is done here, but this might be caused by this option [1]: osd crush update on start Its default is "true". We ran into this some time ago and were wondering why the OSDs were in the wrong bucket everytime we restarted services. As I said, I don't know how exactly this would affect you, but you could set that config option to "false" and see if that still happens. Regards, Eugen [1] http://docs.ceph.com/docs/master/rados/operations/crush-map/ Zitat von Eddie Yen : > Hi, > > I'm using stable/rocky to try ceph cache tiering. > Now I'm facing a one issue. > > I chose one SSD to become cache tier disk. And set below options in > globals.yml. > ceph_enable_cache = "yes" > ceph_target_max_byte= "" > ceph_target_max_objects = "" > ceph_cache_mode = "writeback" > > And the default OSD type is bluestore. > > > It will bootstrap the cache disk and create another OSD container. > And also create the root bucket called "cache". then set the cache rule to > every cache pools. > The problem is, that OSD didn't stay at "cache" bucket, it still stay at > "default" bucket. > That caused the services can't access to the Ceph normally. Especially > deploying Gnocchi. > > When error occurred, I manually set that OSD to the cache bucket then > re-deploy, and everything is normal now. > But still a strange issue that it stay in the wrong bucket. > > Did I miss something during deployment? Or what can I do? > > > Many thanks, > Eddie. From skaplons at redhat.com Mon Jul 1 12:44:29 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Mon, 1 Jul 2019 14:44:29 +0200 Subject: [neutron][ci] Gate failure Message-ID: <7C407162-556C-4124-BAE2-F1FDBFC2D4F4@redhat.com> Hi, It looks that we (again) have some broken gate. This time it’s problem with neutron-tempest-plugin-designate-scenario job which is failing 100% times. See [1] for details. Patch [2] is proposed so now lets wait until it will be merged and You can than rebase Your neutron patches to make Zuul happy again :) [1] https://bugs.launchpad.net/devstack/+bug/1834849 [2] https://review.opendev.org/#/c/668447/ — Slawek Kaplonski Senior software engineer Red Hat From fungi at yuggoth.org Mon Jul 1 13:08:10 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 1 Jul 2019 13:08:10 +0000 Subject: [neutron][ci] Gate failure In-Reply-To: <7C407162-556C-4124-BAE2-F1FDBFC2D4F4@redhat.com> References: <7C407162-556C-4124-BAE2-F1FDBFC2D4F4@redhat.com> Message-ID: <20190701130809.4uhsjiojbbxms3ce@yuggoth.org> On 2019-07-01 14:44:29 +0200 (+0200), Slawek Kaplonski wrote: [...] > Patch [2] is proposed so now lets wait until it will be merged and > You can than rebase Your neutron patches to make Zuul happy again [...] There should be no need to rebase changes for Zuul to make use of a fix which merges to the branch. Just a "recheck" comment is all you need. Zuul always merges your change onto the latest branch state when starting new builds, so the *only* reason to rebase should be if you have a legitimate merge conflict preventing it from being tested at all. This is not new, this is the way Zuul has always worked. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From aschultz at redhat.com Mon Jul 1 13:52:44 2019 From: aschultz at redhat.com (Alex Schultz) Date: Mon, 1 Jul 2019 07:52:44 -0600 Subject: [Release-job-failures] Release of openstack/ansible-role-redhat-subscription failed In-Reply-To: <1b761937-ed22-9a5a-ccd8-823846cc3150@openstack.org> References: <1b761937-ed22-9a5a-ccd8-823846cc3150@openstack.org> Message-ID: On Mon, Jul 1, 2019 at 5:44 AM Thierry Carrez wrote: > zuul at openstack.org wrote: > > Build failed. > > > > - release-openstack-python > http://logs.openstack.org/59/59f40f154a5302df1ee4230f35f87d261e0bf9eb/release/release-openstack-python/8b78b28/ > : POST_FAILURE in 2m 09s > > - announce-release announce-release : SKIPPED > > - propose-update-constraints propose-update-constraints : SKIPPED > > Upload to PyPI at the end of the release was rejected due to: > > "The description failed to render in the default format of > reStructuredText. See https://pypi.org/help/#description-content-type > for more information." > > Impact: ansible-role-redhat-subscription 1.0.3 was produced and > published, but not uploaded to PyPI. > > Remediation: once the description is fixed, we'll have to request a > 1.0.4 to fix the situation. > > https://review.opendev.org/668471 > -- > Thierry Carrez (ttx) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Mon Jul 1 14:55:25 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Mon, 1 Jul 2019 16:55:25 +0200 Subject: [neutron][ci] Gate failure In-Reply-To: <20190701130809.4uhsjiojbbxms3ce@yuggoth.org> References: <7C407162-556C-4124-BAE2-F1FDBFC2D4F4@redhat.com> <20190701130809.4uhsjiojbbxms3ce@yuggoth.org> Message-ID: Hi, > On 1 Jul 2019, at 15:08, Jeremy Stanley wrote: > > On 2019-07-01 14:44:29 +0200 (+0200), Slawek Kaplonski wrote: > [...] >> Patch [2] is proposed so now lets wait until it will be merged and >> You can than rebase Your neutron patches to make Zuul happy again > [...] > > There should be no need to rebase changes for Zuul to make use of a > fix which merges to the branch. Just a "recheck" comment is all you > need. Zuul always merges your change onto the latest branch state > when starting new builds, so the *only* reason to rebase should be > if you have a legitimate merge conflict preventing it from being > tested at all. This is not new, this is the way Zuul has always > worked. So if I have fix for some issue in Neutron repo (lets say patch A), and other patch to neutron repo (patch B) which is failing because of this issue, I don’t need to rebase my failing patch B to include fix from patch A and to get +1 from Zuul? Am I understanding correct what You wrote? I know that it is like that in gate queue but I though that in check queue it works differently. > -- > Jeremy Stanley — Slawek Kaplonski Senior software engineer Red Hat From cboylan at sapwetik.org Mon Jul 1 15:03:01 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 01 Jul 2019 08:03:01 -0700 Subject: [neutron][ci] Gate failure In-Reply-To: References: <7C407162-556C-4124-BAE2-F1FDBFC2D4F4@redhat.com> <20190701130809.4uhsjiojbbxms3ce@yuggoth.org> Message-ID: On Mon, Jul 1, 2019, at 7:55 AM, Slawek Kaplonski wrote: > Hi, > > > On 1 Jul 2019, at 15:08, Jeremy Stanley wrote: > > > > On 2019-07-01 14:44:29 +0200 (+0200), Slawek Kaplonski wrote: > > [...] > >> Patch [2] is proposed so now lets wait until it will be merged and > >> You can than rebase Your neutron patches to make Zuul happy again > > [...] > > > > There should be no need to rebase changes for Zuul to make use of a > > fix which merges to the branch. Just a "recheck" comment is all you > > need. Zuul always merges your change onto the latest branch state > > when starting new builds, so the *only* reason to rebase should be > > if you have a legitimate merge conflict preventing it from being > > tested at all. This is not new, this is the way Zuul has always > > worked. > > So if I have fix for some issue in Neutron repo (lets say patch A), and > other patch to neutron repo (patch B) which is failing because of this > issue, I don’t need to rebase my failing patch B to include fix from > patch A and to get +1 from Zuul? Am I understanding correct what You > wrote? > I know that it is like that in gate queue but I though that in check > queue it works differently. You don't need to rebase or use depends on once change A has merged because Zuul will always merge your proposed changes (change B) to the target branch (which includes change A). If you want things to go green prior to change A merging you will need to rebase or use depends on. Clark From doka.ua at gmx.com Mon Jul 1 15:40:54 2019 From: doka.ua at gmx.com (Volodymyr Litovka) Date: Mon, 1 Jul 2019 18:40:54 +0300 Subject: [octavia] HTTP/2 support Message-ID: <2c5d7d87-ae8b-1ef1-4c00-bf542d5cb15e@gmx.com> Dear colleagues, - since haproxy supports HTTP/2 starting 1.9 (released on Dec 2018) for both frontends and backends - due to wide adoption of HTTP/2 (34% of all sites are now support HTTP/2 according to W3techs) - support for HTTP/2 is in all browsers and web-servers whether Octavia team has plans to add support for HTTP/2 as well? Thank you. -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison From skaplons at redhat.com Mon Jul 1 16:26:47 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Mon, 1 Jul 2019 18:26:47 +0200 Subject: [neutron][ci] Gate failure In-Reply-To: References: <7C407162-556C-4124-BAE2-F1FDBFC2D4F4@redhat.com> <20190701130809.4uhsjiojbbxms3ce@yuggoth.org> Message-ID: Hi, > On 1 Jul 2019, at 17:03, Clark Boylan wrote: > > On Mon, Jul 1, 2019, at 7:55 AM, Slawek Kaplonski wrote: >> Hi, >> >>> On 1 Jul 2019, at 15:08, Jeremy Stanley wrote: >>> >>> On 2019-07-01 14:44:29 +0200 (+0200), Slawek Kaplonski wrote: >>> [...] >>>> Patch [2] is proposed so now lets wait until it will be merged and >>>> You can than rebase Your neutron patches to make Zuul happy again >>> [...] >>> >>> There should be no need to rebase changes for Zuul to make use of a >>> fix which merges to the branch. Just a "recheck" comment is all you >>> need. Zuul always merges your change onto the latest branch state >>> when starting new builds, so the *only* reason to rebase should be >>> if you have a legitimate merge conflict preventing it from being >>> tested at all. This is not new, this is the way Zuul has always >>> worked. >> >> So if I have fix for some issue in Neutron repo (lets say patch A), and >> other patch to neutron repo (patch B) which is failing because of this >> issue, I don’t need to rebase my failing patch B to include fix from >> patch A and to get +1 from Zuul? Am I understanding correct what You >> wrote? >> I know that it is like that in gate queue but I though that in check >> queue it works differently. > > You don't need to rebase or use depends on once change A has merged because Zuul will always merge your proposed changes (change B) to the target branch (which includes change A). Thx a lot for explanation. I didn’t know that it is like that in check queue :) > > If you want things to go green prior to change A merging you will need to rebase or use depends on. > > Clark — Slawek Kaplonski Senior software engineer Red Hat From tpb at dyncloud.net Mon Jul 1 19:44:37 2019 From: tpb at dyncloud.net (Tom Barron) Date: Mon, 1 Jul 2019 15:44:37 -0400 Subject: [manila] no meeting July 4 Message-ID: <20190701194437.wvm2y6ggc7givbai@barron.net> We just agreed in freenode #openstack-manila to skip this week's Manila community meeting since several regular attendees including the chair are on holiday. We'll plan to meet as usual the following Thursday, 11 July, at 1500 UTC in #openstack-meeting-alt. Feel free to add to the agenda here: https://wiki.openstack.org/wiki/Manila/Meetings -- Tom Barron From rfolco at redhat.com Mon Jul 1 20:33:36 2019 From: rfolco at redhat.com (Rafael Folco) Date: Mon, 1 Jul 2019 17:33:36 -0300 Subject: [tripleo] TripleO CI Summary: Sprint 32 Message-ID: Greetings, The TripleO CI team has just completed Sprint 32 / Unified Sprint 11 (June 06 thru Jun 26). The following is a summary of completed work during this sprint cycle: - Created image and container build jobs on RHEL 7 in the internal instance of Software Factory as a prep work for RHEL8 jobs on RDO Software Factory. - Started creating RHEL8 jobs to build a periodic pipeline in the RDO Software Factory and provide feedback for CentOS8 coverage. - Promotion status: green on all branches at most of the sprint. The planned work for the next sprint [1] are: - Prepare the grounds for moving RDO on RHEL8 jobs from the internal instance of Software Factory to the RDO instance. This includes building a nodepool image w/ RHUI for base repos and enabling RHEL8 on tripleo-repos for delorean repos. - Get RDO on RHEL8 build jobs working and address dependency issues. - Continue the design document for a staging environment to test changes in the promoter server. This will benefit CI team with less breakages in the promoter server and also prepare the grounds for the multi-arch builds. The Ruck and Rover for this sprint are Arx Cruz (arxcruz) and Sagi Shnaidman (sshnaidm). Please direct questions or queries to them regarding CI status or issues in #tripleo, ideally to whomever has the ‘|ruck’ suffix on their nick. Ruck/rover notes are being tracked in etherpad [2]. Thanks, rfolco [1] https://tree.taiga.io/project/tripleo-ci-board/taskboard/unified-sprint-12 [2] https://etherpad.openstack.org/p/ruckroversprint12 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongbin034 at gmail.com Mon Jul 1 21:08:17 2019 From: hongbin034 at gmail.com (Hongbin Lu) Date: Mon, 1 Jul 2019 17:08:17 -0400 Subject: [Zun][Kata] Devstack plugin for installing Kata container Message-ID: Hi all, I have a patch [1] that adds support for installing kata Container in devstack. Right now, it configures kata container to run with Docker. In the future, I plan to add support for containerd's cri plugin, which basically allows running pods with kata container. Initially, OpenStack Zun will use this plugin to install Kata container, but I believe it would be beneficial for other projects. Appreciate if anyone interest to cast your feedback on the patch. [1] https://review.opendev.org/#/c/668490/ Best regards, Hongbin -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnsomor at gmail.com Mon Jul 1 21:33:24 2019 From: johnsomor at gmail.com (Michael Johnson) Date: Mon, 1 Jul 2019 14:33:24 -0700 Subject: [octavia] HTTP/2 support In-Reply-To: <2c5d7d87-ae8b-1ef1-4c00-bf542d5cb15e@gmx.com> References: <2c5d7d87-ae8b-1ef1-4c00-bf542d5cb15e@gmx.com> Message-ID: Hi Volodymyr, Yes, this is definitely something we are planning to do. There are a couple of reasons we have held off doing this: 1. You really want HAProxy 2.0 (also recently released) to get the full capabilities of HTTP/2 (See https://www.haproxy.com/blog/haproxy-2-0-and-beyond/#end-to-end-http-2). Prior to this, HTTP/2 connections from users were translated to HTTP/1.1 connections on the servers and/or had some open issues. 2. Another challenge is that none of the distributions (LTS versions) are shipping with HAProxy 1.9 or 2.0. Currently they all ship with 1.8. This means that operators would have to build images with a custom package or build of HAProxy in it. 3. This feature would bring in capabilities that only HAProxy 1.9/2.0 based amphora images would support. We have been talking in the weekly IRC and at the PTG about how we could handle "discovering" that an image has been build with a specific version of HAProxy prior to it being booted. I think the current leading idea is to tag the images in glance with a special tag/metadata that would allow us to query if there is a compatible image available for a feature a user is requesting. 4. No vendor provider drivers have requested or implemented HTTP/2 support in their drivers. 5. No one has come forward indicating that they can do the development work. (yet) As you can see we have a few issues to figure out and work on before HTTP/2 will be supported. So, definitely on the roadmap (updated), but I can't name which release will get it added. Michael On Mon, Jul 1, 2019 at 8:44 AM Volodymyr Litovka wrote: > > Dear colleagues, > > - since haproxy supports HTTP/2 starting 1.9 (released on Dec 2018) for > both frontends and backends > - due to wide adoption of HTTP/2 (34% of all sites are now support > HTTP/2 according to W3techs) > - support for HTTP/2 is in all browsers and web-servers > > whether Octavia team has plans to add support for HTTP/2 as well? > > Thank you. > > -- > Volodymyr Litovka > "Vision without Execution is Hallucination." -- Thomas Edison > > From missile0407 at gmail.com Tue Jul 2 00:44:13 2019 From: missile0407 at gmail.com (Eddie Yen) Date: Tue, 2 Jul 2019 08:44:13 +0800 Subject: [kolla][ceph] Cache OSDs didn't stay in the root=cache after ceph deployment. In-Reply-To: <20190701123207.Horde.Y7WzBt0rnChxKZBSc3ktF0Z@webmail.nde.ag> References: <20190701123207.Horde.Y7WzBt0rnChxKZBSc3ktF0Z@webmail.nde.ag> Message-ID: Hi Eugen, thanks for your reply first. I tested what you said, addeed "osd crush update on start = False" in the pre-deploy config file (/etc/kolla/config/ceph.conf) Then destroy & re-deploy again. Now the cache OSDs has stayed in the right bucket after ceph deployment. Really thanks for your advise, now everything works now. Appreciate, Eddie. Eugen Block 於 2019年7月1日 週一 下午8:40寫道： > Hi, > > although I'm not familiar with kolla I can comment on the ceph part. > > > The problem is, that OSD didn't stay at "cache" bucket, it still stay at > > "default" bucket. > > I'm not sure how the deployment process with kolla works and what > exactly is done here, but this might be caused by this option [1]: > > osd crush update on start > > Its default is "true". We ran into this some time ago and were > wondering why the OSDs were in the wrong bucket everytime we restarted > services. As I said, I don't know how exactly this would affect you, > but you could set that config option to "false" and see if that still > happens. > > > Regards, > Eugen > > [1] http://docs.ceph.com/docs/master/rados/operations/crush-map/ > > Zitat von Eddie Yen : > > > Hi, > > > > I'm using stable/rocky to try ceph cache tiering. > > Now I'm facing a one issue. > > > > I chose one SSD to become cache tier disk. And set below options in > > globals.yml. > > ceph_enable_cache = "yes" > > ceph_target_max_byte= "" > > ceph_target_max_objects = "" > > ceph_cache_mode = "writeback" > > > > And the default OSD type is bluestore. > > > > > > It will bootstrap the cache disk and create another OSD container. > > And also create the root bucket called "cache". then set the cache rule > to > > every cache pools. > > The problem is, that OSD didn't stay at "cache" bucket, it still stay at > > "default" bucket. > > That caused the services can't access to the Ceph normally. Especially > > deploying Gnocchi. > > > > When error occurred, I manually set that OSD to the cache bucket then > > re-deploy, and everything is normal now. > > But still a strange issue that it stay in the wrong bucket. > > > > Did I miss something during deployment? Or what can I do? > > > > > > Many thanks, > > Eddie. > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhang.lei.fly+os-discuss at gmail.com Tue Jul 2 04:47:59 2019 From: zhang.lei.fly+os-discuss at gmail.com (Jeffrey Zhang) Date: Tue, 2 Jul 2019 12:47:59 +0800 Subject: [kolla] Proposing yoctozepto as core In-Reply-To: References: <950bf345-da0f-dabc-31f1-fcb1711e36df@linaro.org> Message-ID: +1 On Fri, Jun 28, 2019 at 8:11 PM Gaëtan Trellu wrote: > +1, congrats yoctozeoto! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Tue Jul 2 07:19:29 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Tue, 2 Jul 2019 09:19:29 +0200 Subject: [neutron][ci] Gate failure In-Reply-To: <7C407162-556C-4124-BAE2-F1FDBFC2D4F4@redhat.com> References: <7C407162-556C-4124-BAE2-F1FDBFC2D4F4@redhat.com> Message-ID: <68593D1D-07E0-451D-8867-9FE0F92F1F73@redhat.com> Hi, Fix [1] is merged so You can now recheck Your patches if it failed on neutron-tempest-plugin-designate-scenario job earlier. > On 1 Jul 2019, at 14:44, Slawek Kaplonski wrote: > > Hi, > > It looks that we (again) have some broken gate. This time it’s problem with neutron-tempest-plugin-designate-scenario job which is failing 100% times. > See [1] for details. > Patch [2] is proposed so now lets wait until it will be merged and You can than rebase Your neutron patches to make Zuul happy again :) > > [1] https://bugs.launchpad.net/devstack/+bug/1834849 > [2] https://review.opendev.org/#/c/668447/ > > — > Slawek Kaplonski > Senior software engineer > Red Hat > [1] https://review.opendev.org/#/c/668447/ — Slawek Kaplonski Senior software engineer Red Hat From sneha.rai at hpe.com Tue Jul 2 04:14:10 2019 From: sneha.rai at hpe.com (RAI, SNEHA) Date: Tue, 2 Jul 2019 04:14:10 +0000 Subject: HPE 3PAR Cinder driver-Multiattach: Fails to detach second instance from volume Message-ID: Hi All, Is there a way to find in cinder how many instances are attached to a volume? Thanks & Regards, Sneha Rai From: RAI, SNEHA Sent: Monday, July 1, 2019 5:26 PM To: openstack-dev at lists.openstack.org Subject: HPE 3PAR Cinder driver-Multiattach: Fails to detach second instance from volume Hi Team, There is a bug on 3PAR Cinder driver https://bugs.launchpad.net/cinder/+bug/1834660. I am able to attach multiple instances to 3PAR volume but only the first instance gets detached successfully. For the second instance, volume goes into detaching status due to "Host does not exist" error. What is happening here is, the first detach call invokes _delete_3par_host() which removes the compute host entry from 3PAR which ideally should be done only when the last instance is to be detached. It would be great if someone could help me understand if this needs to be handled in driver code or nova would internally take care of it. Code changes done to support multiattach-https://review.opendev.org/#/c/659443 Thanks & Regards, Sneha Rai -------------- next part -------------- An HTML attachment was scrubbed... URL: From wangpeihuixyz at 126.com Tue Jul 2 04:20:55 2019 From: wangpeihuixyz at 126.com (Frank Wang) Date: Tue, 2 Jul 2019 12:20:55 +0800 (CST) Subject: Does networking-ovn support fwaas ? Message-ID: <79a5567a.38aa.16bb0ea04ef.Coremail.wangpeihuixyz@126.com> Hi all, I'm investigating ovn as the neutron backend recently, networking-ovn is a great project, but one thing confuses me, I'd like to know if networking-ovn support fwaas? Security protection for network level, not port leve like security group, I looked up lots of materials, didn't find a clue. Any comments would be appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aj at suse.com Tue Jul 2 11:06:09 2019 From: aj at suse.com (Andreas Jaeger) Date: Tue, 2 Jul 2019 13:06:09 +0200 Subject: [uc][tc][ops] reviving osops- repos In-Reply-To: References: <20190530205552.falsvxcegehtyuge@yuggoth.org> <20190531123501.tawgvqgsw6yle2nu@csail.mit.edu> <20190531164102.5lwt2jyxk24u3vdz@yuggoth.org> Message-ID: On 11/06/2019 13.55, Jean-Philippe Evrard wrote: >> Alternatively, I feel like a SIG (be it the Ops Docs SIG or a new >> "Operational tooling" SIG) would totally be a good idea to revive this. >> In that case we'd define the repository in [4]. >> >> My personal preference would be for a new SIG, but whoever is signing up >> to work on this should definitely have the final say. > > Agreed on having it inside OpenStack namespace, and code handled by a team/SIG/WG (with my preference being a SIG -- existing or not). When this team/SIG/WG retires, the repo would with it. It provides clean ownership, and clear cleanup when disbanding. Mohammed, is that consensus and actionable? Could you then update https://review.opendev.org/#/c/662300/ to reflect this, please? Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From mriedemos at gmail.com Tue Jul 2 13:26:33 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Tue, 2 Jul 2019 08:26:33 -0500 Subject: HPE 3PAR Cinder driver-Multiattach: Fails to detach second instance from volume In-Reply-To: References: Message-ID: <7cc537a6-0112-baeb-3645-961b6d1e8d73@gmail.com> On 7/1/2019 11:14 PM, RAI, SNEHA wrote: > Is there a way to find in cinder how many instances are attached to a > volume? Yes, the response to GET /v3/{project_id}/volumes/{volume_id} has an "attachments" parameter which is a list of dicts of information about the servers that are attached to the volume. Note that a single server can have multiple attachments to the volume while the server is being migrated across compute hosts, so to determine unique server attachments you'd have to distinguish by server ID in the attachments list. -- Thanks, Matt From mihalis68 at gmail.com Tue Jul 2 15:04:07 2019 From: mihalis68 at gmail.com (Chris Morgan) Date: Tue, 2 Jul 2019 11:04:07 -0400 Subject: [ops] Ops Meetups Team meeting 2019-7-2 Message-ID: Meeting minutes for todays meeting are linked below. Two key decisions from today that you might like to know about: 1. The team will attempt to run an ops day at the Shanghai summit if there is a) sufficient interest and b) space to hold it. To gauge the interest level we will run a twitter poll. We have found that the engagement we get via twitter is actually better than via mailing lists. 2. The current consensus for the next meetups is as follows: September, NYC (USA) - this is under preparation 2020 meetup #1 EU region 2020 meetup #2 APAC region Calls for hosting the 2020 meetups will be issued by the meetups team soon. Thanks Chris Morgan (on behalf of the openstack ops meetups team) Meeting ended Tue Jul 2 14:48:24 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) 10:48 AM Minutes: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_team.2019-07-02-14.17.html 10:48 AM Minutes (text): http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_team.2019-07-02-14.17.txt 10:48 AM Log: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_team.2019-07-02-14.17.log.html -- Chris Morgan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashlee at openstack.org Tue Jul 2 19:17:22 2019 From: ashlee at openstack.org (Ashlee Ferguson) Date: Tue, 2 Jul 2019 14:17:22 -0500 Subject: [Shanghai Summit] CFP Deadline TONIGHT Message-ID: <25C8C459-0A89-4955-8F62-97D31C5A3362@openstack.org> Hi everyone, The Shanghai Summit Call for Presentations [1] deadline is TONIGHT, July 2 at 11:59 pm PT (July 3, 2019 at 15:00 China Standard Time)! Submit your presentations, panels, and Hands-on Workshops for the Open Infrastructure Summit [2] by the end of today, and join the global community in Shanghai, November 4-6, 2019. Sessions will be presented in both Mandarin and English, so you may submit your presentation in either language. Tracks [4]: 5G, NFV & Edge AI, Machine Learning & HPC CI/CD Container Infrastructure Getting Started Hands-on Workshops Open Development Private & Hybrid Cloud Public Cloud Security Other Helpful Shanghai Summit & PTG Information-- * Register now [5] before the early bird registration deadline in early August (USD or RMB options available) * Apply for Travel Support [6] before August 8. More information here [7]. * Interested in sponsoring the Summit? [8]. * The content submission process for the Forum and Project Teams Gathering will be managed separately in the upcoming months. We look forward to your submissions! Cheers, Ashlee [1] https://cfp.openstack.org/ [2] https://www.openstack.org/summit/shanghai-2019/ [3] https://cfp.openstack.org/ [4] https://www.openstack.org/summit/shanghai-2019/summit-categories/ [5] https://www.openstack.org/summit/shanghai-2019/ [6] https://openstackfoundation.formstack.com/forms/travelsupportshanghai [7] https://www.openstack.org/summit/shanghai-2019/travel/ [8] https://www.openstack.org/summit/shanghai-2019/sponsors/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbragstad at gmail.com Tue Jul 2 19:34:50 2019 From: lbragstad at gmail.com (Lance Bragstad) Date: Tue, 2 Jul 2019 14:34:50 -0500 Subject: [keystone][oslo] oslo.limt implementation update Message-ID: Today in keystone's office hours, we went through a group code review of what's currently proposed for the oslo.limit library [0]. This is a summary of the action items that came out of that meeting. * We should implement a basic functional testing framework that exercises keystoneauth connections (used for pulling limit information for keystone). Otherwise, we'll be mocking things left and right in unit tests to get decent test coverage with the current keystoneauth code. * Investigate alternatives to globals for keystoneauth connections [1]. * Investigate adopting a keystoneauth-like way of loading enforcement models (similar to how ksa loads authentication plugins) [2]. * Figure out if we want to use endpoint_id or service name + region name for service configuration [3]. * Build out functional testing for flat enforcement * Implement strict-two-level enforcement model This existing rewrite was mostly stolen from John's patches to his fork oslo.limit [4]. Hopefully the current series moves things in that direction. Feel free to chime in if you have additional notes or comments. Lance [0] https://review.opendev.org/#/q/topic:rewrite+(status:open+OR+status:merged)+project:openstack/oslo.limit [1] https://bugs.launchpad.net/oslo.limit/+bug/1835103 [2] https://bugs.launchpad.net/oslo.limit/+bug/1835104 [3] https://bugs.launchpad.net/oslo.limit/+bug/1835106 [4] https://github.com/JohnGarbutt/oslo.limit/commit/a5b908046fd904c25b6cd15c65266c747774b5ab -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From lbragstad at gmail.com Tue Jul 2 19:37:35 2019 From: lbragstad at gmail.com (Lance Bragstad) Date: Tue, 2 Jul 2019 14:37:35 -0500 Subject: [keystone][oslo] oslo.limt implementation update In-Reply-To: References: Message-ID: <23745175-52fa-88e4-d0de-eaa9ddc37dbb@gmail.com> I forgot to add that the meeting was held in IRC. Logs are available if you're interested in following along [0]. [0] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2019-07-02.log.html#t2019-07-02T16:32:50 On 7/2/19 2:34 PM, Lance Bragstad wrote: > Today in keystone's office hours, we went through a group code review > of what's currently proposed for the oslo.limit library [0]. This is a > summary of the action items that came out of that meeting. > > * We should implement a basic functional testing framework that > exercises keystoneauth connections (used for pulling limit information > for keystone). Otherwise, we'll be mocking things left and right in > unit tests to get decent test coverage with the current keystoneauth code. > * Investigate alternatives to globals for keystoneauth connections [1]. > * Investigate adopting a keystoneauth-like way of loading enforcement > models (similar to how ksa loads authentication plugins) [2]. > * Figure out if we want to use endpoint_id or service name + region > name for service configuration [3]. > * Build out functional testing for flat enforcement > * Implement strict-two-level enforcement model > > This existing rewrite was mostly stolen from John's patches to his > fork oslo.limit [4]. Hopefully the current series moves things in that > direction. > > Feel free to chime in if you have additional notes or comments. > > Lance > > [0] > https://review.opendev.org/#/q/topic:rewrite+(status:open+OR+status:merged)+project:openstack/oslo.limit > [1] https://bugs.launchpad.net/oslo.limit/+bug/1835103 > [2] https://bugs.launchpad.net/oslo.limit/+bug/1835104 > [3] https://bugs.launchpad.net/oslo.limit/+bug/1835106 > [4] > https://github.com/JohnGarbutt/oslo.limit/commit/a5b908046fd904c25b6cd15c65266c747774b5ab -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From raubvogel at gmail.com Tue Jul 2 19:51:55 2019 From: raubvogel at gmail.com (Mauricio Tavares) Date: Tue, 2 Jul 2019 15:51:55 -0400 Subject: Instance (vm guest) not getting PCI card Message-ID: Newbie and easy questions: I have two cards, one in each stein (centos) compute node setup for kvm, which I want to be able to handle to a vm guest (instance). Following https://docs.openstack.org/nova/latest/admin/pci-passthrough.html, I 1. Setup both computer nodes to vt-t and iommu. 2. On the controller 2.1. Create a PCI alias based on the vendor and product ID alias = { "vendor_id":"19fg", "product_id":"4000", "device_type":"type-PF", "name":"testnic" } - The PCI address for the card is different on each compute node 2.2. Create a flavor, say, n1.large openstack flavor create n1.large --id auto --ram 8192 --disk 80 --vcpus 4 --property "pci_passthrough:alias"="testnic:1" 2.3. Restart openstack-nova-api 3. On each compute node 3.1. Create a PCI alias based on the vendor and product ID alias = { "vendor_id":"19fg", "product_id":"4000", "device_type":"type-PF", "name":"testnic" } 3.2. Create passthrough_whitelist entry passthrough_whitelist = { "vendor_id":"19fg", "product_id":"4000" } 3.3. Restart openstack-nova-compute 4. Create instance (vm guest) using the n1.large flavor. 5. Login to instance and discover dmesg and lspci does not list card 6. Do a "virsh dumpxml" for the instance on its compute node and discover there is no entry for the card listed in the xml file. I take nova would automagically do what I would if this was a kvm install, namely ensure card cannot be accessed/used by the host and then edit the guest xml file so it can see said card. Questions: Q1: If a device is sr-iov capable, do I have to use that or can I just pass the entire card to the vm guest? Q2: Is there anywhere I can look for clues to why is the libvirt xml file for the instance not being populated with the pci card info? So far I only looked in the controller node's nova_scheduler.log file. From smooney at redhat.com Tue Jul 2 20:36:32 2019 From: smooney at redhat.com (Sean Mooney) Date: Tue, 02 Jul 2019 21:36:32 +0100 Subject: Instance (vm guest) not getting PCI card In-Reply-To: References: Message-ID: On Tue, 2019-07-02 at 15:51 -0400, Mauricio Tavares wrote: > Newbie and easy questions: I have two cards, one in each stein > (centos) compute node setup for kvm, which I want to be able to handle > to a vm guest (instance). Following > https://docs.openstack.org/nova/latest/admin/pci-passthrough.html, I > > 1. Setup both computer nodes to vt-t and iommu. > 2. On the controller > 2.1. Create a PCI alias based on the vendor and product ID > alias = { "vendor_id":"19fg", "product_id":"4000", > "device_type":"type-PF", "name":"testnic" } the alias looks correcect. assuming you have it set in teh pci section https://docs.openstack.org/nova/latest/configuration/config.html#pci.alias then i should generate teh request of a pci device. the alias needs to be defiend in the nova.conf used by the api node and the compute node for it to work correctly but i assume that when you say its set on the contoler it set on the nova.conf the nova api is useing. > > - The PCI address for the card is different on each compute node > > 2.2. Create a flavor, say, n1.large > openstack flavor create n1.large --id auto --ram 8192 --disk 80 > --vcpus 4 --property "pci_passthrough:alias"="testnic:1" this is also correct > > 2.3. Restart openstack-nova-api > > 3. On each compute node > 3.1. Create a PCI alias based on the vendor and product ID > alias = { "vendor_id":"19fg", "product_id":"4000", > "device_type":"type-PF", "name":"testnic" } > > 3.2. Create passthrough_whitelist entry > passthrough_whitelist = { "vendor_id":"19fg", "product_id":"4000" } assuming this is set in the pci section it also looks correct https://docs.openstack.org/nova/latest/configuration/config.html#pci.passthrough_whitelist > > 3.3. Restart openstack-nova-compute > > 4. Create instance (vm guest) using the n1.large flavor. > > 5. Login to instance and discover dmesg and lspci does not list card > > 6. Do a "virsh dumpxml" for the instance on its compute node and > discover there is no entry for the card listed in the xml file. I take > nova would automagically do what I would if this was a kvm install, > namely ensure card cannot be accessed/used by the host and then edit > the guest xml file so it can see said card. yes you should have seen it in the xml and the card should have been passed through to the guest. > > Questions: > Q1: If a device is sr-iov capable, do I have to use that or can I just > pass the entire card to the vm guest? you can passthorugh the entire card to the guest yes. > Q2: Is there anywhere I can look for clues to why is the libvirt xml > file for the instance not being populated with the pci card info? So > far I only looked in the controller node's nova_scheduler.log file. there are several things to check. first i would check the nova compute agenet log and see if there are any tracebacks or errors second in the nova cell db, often called just nova or nova_cell1 (not nova_cell0) check the pci_devices table and see if the devices are listed. > From openstack at fried.cc Tue Jul 2 21:01:19 2019 From: openstack at fried.cc (Eric Fried) Date: Tue, 2 Jul 2019 16:01:19 -0500 Subject: [keystone][oslo] oslo.limt implementation update In-Reply-To: <23745175-52fa-88e4-d0de-eaa9ddc37dbb@gmail.com> References: <23745175-52fa-88e4-d0de-eaa9ddc37dbb@gmail.com> Message-ID: <2ad2cb15-b7dc-cf12-1179-8612a9801699@fried.cc> >> * We should implement a basic functional testing framework that >> exercises keystoneauth connections (used for pulling limit information >> for keystone). Otherwise, we'll be mocking things left and right in >> unit tests to get decent test coverage with the current keystoneauth code. I would be not at all offended if functional test fixtures for ksa artifacts were made generally available. efried . From kecarter at redhat.com Tue Jul 2 21:28:55 2019 From: kecarter at redhat.com (Kevin Carter) Date: Tue, 2 Jul 2019 16:28:55 -0500 Subject: [TripleO] The Transformation Squad meeting canceled - 4 July, 2019 Message-ID: Hello all, I wanted to let everyone know that the Transformation Squad team meeting will be canceled this coming Thursday; 4 July, 2019 @ 13:00UTC. We will continue with regular meetings on 11 July, 2019 @ 13:00UTC. If there are questions about this cancellation, or anything else, please let us know. Thank you and have a great week. -- Kevin Carter IRC: cloudnull -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Tue Jul 2 22:37:34 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 02 Jul 2019 15:37:34 -0700 Subject: [all][qa] Tempest jobs are swapping Message-ID: I've been working to bring up a new cloud as part of our nodepool resource set and one of the things we do to sanity check that is run a default tempest full job. The first time I ran tempest it failed because I hadn't configured swap on the test node and we ran out of memory. I added swap, reran things and tempest passed just fine. Our base jobs configure swap as a last ditch effort to avoid failing jobs unnecessarily but the ideal is to avoid swap entirely. In the past 8GB of memory has been plenty to run the tempest testsuite so I think something has changed here and I think we should be able to get us running back under 8GB of memory again. I bring this up because in recent weeks we've seen different groups attempt to reduce their resource footprint (which is good), but many of the approaches seem to ignore that making our jobs as quick and reliable as possible (eg don't use swap) will have a major impact. This is due to the way gating works where a failure requires we discard all results for subsequent changes in the gate, remove the change that failed, then re enqueue jobs for the changes after the failed change. On top of that the quicker our jobs run the quicker we return resources to the pool. How do we debug this? Devstack jobs actually do capture dstat data as well as memory specific information that can be used to identify resource hogs. Taking a recent tempest-full job's dstat log we can see that cinder-backup is using 785MB of memory all on its own [0] (scroll to the bottom). Devstack also captures memory usage of a larger set of processes in its peakmem_tracker log [1]. This includes RSS specifically which doesn't match up with dstat's number making me think dstat's number may be virtual memory and not resident memory. This peakmem_tracker log identifies other processes which we might look at for improving this situation. It would be great if the QA team and various projects could take a look at this to help improve the reliability and throughput of our testing. Thank you. [0] http://logs.openstack.org/81/665281/3/check/tempest-full/cf5e17e/controller/logs/screen-dstat.txt.gz [1] http://logs.openstack.org/81/665281/3/check/tempest-full/cf5e17e/controller/logs/screen-peakmem_tracker.txt.gz From mriedemos at gmail.com Tue Jul 2 23:18:50 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Tue, 2 Jul 2019 18:18:50 -0500 Subject: [all][qa] Tempest jobs are swapping In-Reply-To: References: Message-ID: On 7/2/2019 5:37 PM, Clark Boylan wrote: > Taking a recent tempest-full job's dstat log we can see that cinder-backup is using 785MB of memory all on its own [0] (scroll to the bottom). Oh hello.... https://review.opendev.org/#/c/651865/ -- Thanks, Matt From juliaashleykreger at gmail.com Tue Jul 2 23:35:51 2019 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Tue, 2 Jul 2019 16:35:51 -0700 Subject: [ironic] Moving to office hours as opposed to weekly meetings for the next month Message-ID: Greetings Everyone! This week, during the weekly meeting, we seemed to reach consensus that we would try taking a break from meetings[0] and moving to orienting around using the mailing list[1] and our etherpad "whiteboard" [2]. With this, we're going to want to re-evaluate in about a month. I suspect it would be a good time for us to have a "mid-cycle" style set of topical calls. I've gone ahead and created a poll to try and identify a couple days that might be ideal for contributors[3]. But in the mean time, we want to ensure that we have some times for office hours. The suggestion was also made during this week's meeting that we may want to make the office hours window a little larger to enable more discussion. So when will we have office hours? ---------------------------------- Ideally we'll start with two time windows. One to provide coverage to US and Europe friendly time zones, and another for APAC contributors. * I think 2-4 PM UTC on Mondays would be ideal. This translates to 7-9 AM US-Pacific or 10 AM to 12 PM US-Eastern. * We need to determine a time window that would be ideal for APAC contributors. I've created a poll to help facilitate discussion[4]. So what is Office Hours? ------------------------ Office hours are a time window when we expect some contributors to be on IRC and able to partake in higher bandwidth discussions. These times are not absolute. They can change and evolve, and that is the most important thing for us to keep in mind. -- If there are any questions, Please let me know! Otherwise I'll send a summary email out on next Monday. -Julia [0]: http://eavesdrop.openstack.org/meetings/ironic/2019/ironic.2019-07-01-15.00.log.html#l-123 [1]: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007038.html [2]: https://etherpad.openstack.org/p/IronicWhiteBoard [3]: https://doodle.com/poll/652gzta6svsda343 [4]: https://doodle.com/poll/2ta5vbskytpntmgv From matt at oliver.net.au Wed Jul 3 00:16:48 2019 From: matt at oliver.net.au (Matthew Oliver) Date: Wed, 3 Jul 2019 10:16:48 +1000 Subject: [PTL] update FC_SIG liaison list reminder Message-ID: Hey PTLs, I'm one of your friendly First Contact SIG (FC_SIG) representatives, and previously emailed you fine folk asking if you could take a look at the FC_SIG liaison list[0] and update it as needed. As mentioned in the PTL guide[1]. For those of you who have already done that thing, awesome, thanks so much. If you haven't had a chance yet it would be fantastic if you could! What is a FC_SIG liaison, they will be the person we contact or add to a review when a new contributor contacts us and wants to work on your project or we find a new patch from a new contributor and we want to make sure they get all the love they need to get them engaged. So what's important here is having the liaison's details, including timezone, up to date. Also note more then 1 liaison in more then 1 timezone would be preferred (if it can be manged), so we can connect new contributors with someone closer to them. If a project doesn't have a liaison then it'll default to the PTL, but we'd still rather have the PTL in the list with timezones as that makes it easier for us. So talk about it in your teams. Thanks, Matt [0] - https://wiki.openstack.org/wiki/CrossProjectLiaisons#First_Contact_SIG [1] - https://docs.openstack.org/project-team-guide/ptl.html#at-the-beginning-of-a-new-cycle -------------- next part -------------- An HTML attachment was scrubbed... URL: From li.canwei2 at zte.com.cn Wed Jul 3 02:34:52 2019 From: li.canwei2 at zte.com.cn (li.canwei2 at zte.com.cn) Date: Wed, 3 Jul 2019 10:34:52 +0800 (CST) Subject: =?UTF-8?B?W1dhdGNoZXJdIHRlYW0gbWVldGluZyDCoGF0IDA4OjAwIFVUQyB0b2RheQ==?= Message-ID: <201907031034525007996@zte.com.cn> Hi all, Watcher team will have a meeting at 08:00 UTC today in the #openstack-meeting-alt channel. The agenda is available on https://wiki.openstack.org/wiki/Watcher_Meeting_Agenda feel free to add any additional items. Thanks! Canwei Li -------------- next part -------------- An HTML attachment was scrubbed... URL: From berndbausch at gmail.com Wed Jul 3 03:22:23 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Wed, 3 Jul 2019 12:22:23 +0900 Subject: HPE 3PAR Cinder driver-Multiattach: Fails to detach second instance from volume In-Reply-To: References: Message-ID: <45419a88-ca76-0e16-a857-4d7f33717bcb@gmail.com> Yes, by determining the size of the attachments list. CLI: /openstack volume show /displays attachment details. API: The response for the volume details API contains an /attachments /parameter. On 7/2/2019 1:14 PM, RAI, SNEHA wrote: > Hi All, > > Is there a way to find in cinder how many instances are attached to a > volume? > > Thanks & Regards, > > Sneha Rai > > *From:* RAI, SNEHA > *Sent:* Monday, July 1, 2019 5:26 PM > *To:* openstack-dev at lists.openstack.org > *Subject:* HPE 3PAR Cinder driver-Multiattach: Fails to detach second > instance from volume > > Hi Team, > > There is a bug on 3PAR Cinder driver > https://bugs.launchpad.net/cinder/+bug/1834660 > . > > I am able to attach multiple instances to 3PAR volume but only the > first instance gets detached successfully. > > For the second instance, volume goes into detaching status due to > “Host does not exist” error. > > What is happening here is, the first detach call invokes > _delete_3par_host() which removes the compute host entry from 3PAR > which ideally should be done only when the last instance is to be > detached. > > It would be great if someone could help me understand if this needs to > be handled in driver code or nova would internally take care of it. > > Code changes done to support > multiattach-https://review.opendev.org/#/c/659443 > > Thanks & Regards, > > Sneha Rai > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rico.lin.guanyu at gmail.com Wed Jul 3 04:07:40 2019 From: rico.lin.guanyu at gmail.com (Rico Lin) Date: Wed, 3 Jul 2019 12:07:40 +0800 Subject: [Shanghai Summit] CFP Deadline TONIGHT In-Reply-To: <25C8C459-0A89-4955-8F62-97D31C5A3362@openstack.org> References: <25C8C459-0A89-4955-8F62-97D31C5A3362@openstack.org> Message-ID: Hi Ashlee, I notice the deadline has postponed to the end of July 7th. If that is correct, people who follow this mail should know about that. Ashlee Ferguson 於 2019年7月3日 週三，上午3:23寫道： > Hi everyone, > > The Shanghai Summit Call for Presentations [1] deadline is TONIGHT, July 2 > at 11:59 pm PT (July 3, 2019 at 15:00 China Standard Time)! > > Submit your presentations, panels, and Hands-on Workshops for the Open > Infrastructure Summit [2] by the end of today, and join the global > community in Shanghai, November 4-6, 2019. Sessions will be presented in > both Mandarin and English, so you may submit your presentation in either > language. > > Tracks [4]: > 5G, NFV & Edge > AI, Machine Learning & HPC > CI/CD > Container Infrastructure > Getting Started > Hands-on Workshops > Open Development > Private & Hybrid Cloud > Public Cloud > Security > > Other Helpful Shanghai Summit & PTG Information-- > > * Register now [5] before the early bird registration deadline in > early August (USD or RMB options available) > * Apply for Travel Support [6] before August 8. More information here [7]. > * Interested in sponsoring the Summit? [8]. > * The content submission process for the Forum and Project Teams Gathering > will be managed separately in the upcoming months. > > We look forward to your submissions! > > Cheers, > Ashlee > > [1] https://cfp.openstack.org/ > [2] https://www.openstack.org/summit/shanghai-2019/ > [3] https://cfp.openstack.org/ > [4] https://www.openstack.org/summit/shanghai-2019/summit-categories/ > [5] https://www.openstack.org/summit/shanghai-2019/ > [6] https://openstackfoundation.formstack.com/forms/travelsupportshanghai > [7] https://www.openstack.org/summit/shanghai-2019/travel/ > [8] https://www.openstack.org/summit/shanghai-2019/sponsors/ > > > > > -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimmy at openstack.org Wed Jul 3 04:15:43 2019 From: jimmy at openstack.org (Jimmy McArthur) Date: Tue, 02 Jul 2019 23:15:43 -0500 Subject: [OpenStack Foundation] [Shanghai Summit] CFP Deadline TONIGHT In-Reply-To: References: <25C8C459-0A89-4955-8F62-97D31C5A3362@openstack.org> Message-ID: <5D1C2BEF.1050204@openstack.org> The deadline is tonight, as far as comms are concerned. We tend to keep the CFP open for a few extra days for stragglers and such :) It's absolutely critical for us to get a final count of submissions, so the sooner the better. Cheers, Jimmy > Rico Lin > July 2, 2019 at 11:07 PM > Hi Ashlee, I notice the deadline has postponed to the end of July 7th. > If that is correct, people who follow this mail should know about that. > > -- > May The Force of OpenStack Be With You, > */Rico Lin > /*irc: ricolin > > > > _______________________________________________ > Foundation mailing list > Foundation at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/foundation > Ashlee Ferguson > July 2, 2019 at 2:17 PM > Hi everyone, > > The Shanghai Summit Call for Presentations [1] deadline is TONIGHT, > July 2 at 11:59 pm PT (July 3, 2019 at 15:00 China Standard Time)! > > Submit your presentations, panels, and Hands-on Workshops for the Open > Infrastructure Summit [2] by the end of today, and join the global > community in Shanghai, November 4-6, 2019. Sessions will be presented > in both Mandarin and English, so you may submit your presentation in > either language. > > Tracks [4]: > 5G, NFV & Edge > AI, Machine Learning & HPC > CI/CD > Container Infrastructure > Getting Started > Hands-on Workshops > Open Development > Private & Hybrid Cloud > Public Cloud > Security > > Other Helpful Shanghai Summit & PTG Information-- > > * Register now [5] before the early bird registration deadline in > early August (USD or RMB options available) > * Apply for Travel Support [6] before August 8. More information here [7]. > * Interested in sponsoring the Summit? [8]. > * The content submission process for the Forum and Project > Teams Gathering will be managed separately in the upcoming months. > > We look forward to your submissions! > > Cheers, > Ashlee > > [1] https://cfp.openstack.org/ > [2] https://www.openstack.org/summit/shanghai-2019/ > [3] https://cfp.openstack.org/ > [4] https://www.openstack.org/summit/shanghai-2019/summit-categories/ > [5] https://www.openstack.org/summit/shanghai-2019/ > [6] https://openstackfoundation.formstack.com/forms/travelsupportshanghai > [7] https://www.openstack.org/summit/shanghai-2019/travel/ > [8] https://www.openstack.org/summit/shanghai-2019/sponsors/ > > > > > _______________________________________________ > Foundation mailing list > Foundation at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/foundation -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Wed Jul 3 09:32:45 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 03 Jul 2019 18:32:45 +0900 Subject: [all][qa] Tempest jobs are swapping In-Reply-To: References: Message-ID: <16bb72de130.e132b245172524.593102227457274520@ghanshyammann.com> ---- On Wed, 03 Jul 2019 07:37:34 +0900 Clark Boylan wrote ---- > I've been working to bring up a new cloud as part of our nodepool resource set and one of the things we do to sanity check that is run a default tempest full job. The first time I ran tempest it failed because I hadn't configured swap on the test node and we ran out of memory. I added swap, reran things and tempest passed just fine. > > Our base jobs configure swap as a last ditch effort to avoid failing jobs unnecessarily but the ideal is to avoid swap entirely. In the past 8GB of memory has been plenty to run the tempest testsuite so I think something has changed here and I think we should be able to get us running back under 8GB of memory again. > > I bring this up because in recent weeks we've seen different groups attempt to reduce their resource footprint (which is good), but many of the approaches seem to ignore that making our jobs as quick and reliable as possible (eg don't use swap) will have a major impact. This is due to the way gating works where a failure requires we discard all results for subsequent changes in the gate, remove the change that failed, then re enqueue jobs for the changes after the failed change. On top of that the quicker our jobs run the quicker we return resources to the pool. > > How do we debug this? Devstack jobs actually do capture dstat data as well as memory specific information that can be used to identify resource hogs. Taking a recent tempest-full job's dstat log we can see that cinder-backup is using 785MB of memory all on its own [0] (scroll to the bottom). Devstack also captures memory usage of a larger set of processes in its peakmem_tracker log [1]. This includes RSS specifically which doesn't match up with dstat's number making me think dstat's number may be virtual memory and not resident memory. This peakmem_tracker log identifies other processes which we might look at for improving this situation. > > It would be great if the QA team and various projects could take a look at this to help improve the reliability and throughput of our testing. Thank you. Thanks, Clark for pointing this. We have faced the memory issue in fast also where some of the swift services were disabled. cinder-backup service is no doubt taking a lot of memory. As matt mentioned the patch of disabling the c-bak service in tempest-full, we need some job which can run c-bak tests on cinder as well on tempest side but not on other projects. There will be some improvement I except by splitting the integrated tests as per actual dependent services [3]. I need some time to prepare those template and propose if the situation improves. > > [0] http://logs.openstack.org/81/665281/3/check/tempest-full/cf5e17e/controller/logs/screen-dstat.txt.gz > [1] http://logs.openstack.org/81/665281/3/check/tempest-full/cf5e17e/controller/logs/screen-peakmem_tracker.txt.gz > [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html -gmann > From ignaziocassano at gmail.com Wed Jul 3 09:34:42 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Wed, 3 Jul 2019 11:34:42 +0200 Subject: [magnum][queens] Floating ip error Message-ID: Hello All, I've just install queens kolla openstack qith magnum but when I try to create a docker swarm cluster the magnum conductor reports: : : The Resource Type (Magnum::Optional::Neutron::LBaaS::FloatingIP) could not be found. I tried lbaas and floating ip works of magnum context. Does it requires octavia ? Regards Ignazio -------------- next part -------------- An HTML attachment was scrubbed... URL: From raubvogel at gmail.com Wed Jul 3 14:51:49 2019 From: raubvogel at gmail.com (Mauricio Tavares) Date: Wed, 3 Jul 2019 10:51:49 -0400 Subject: Instance (vm guest) not getting PCI card In-Reply-To: References: Message-ID: On Tue, Jul 2, 2019 at 4:36 PM Sean Mooney wrote: > > On Tue, 2019-07-02 at 15:51 -0400, Mauricio Tavares wrote: > > Newbie and easy questions: I have two cards, one in each stein > > (centos) compute node setup for kvm, which I want to be able to handle > > to a vm guest (instance). Following > > https://docs.openstack.org/nova/latest/admin/pci-passthrough.html, I > > > > 1. Setup both computer nodes to vt-t and iommu. > > 2. On the controller > > 2.1. Create a PCI alias based on the vendor and product ID > > alias = { "vendor_id":"19fg", "product_id":"4000", > > "device_type":"type-PF", "name":"testnic" } > the alias looks correcect. assuming you have it set in teh pci section > https://docs.openstack.org/nova/latest/configuration/config.html#pci.alias > then i should generate teh request of a pci device. > In fact my initial sed script is 'find the line beginning with "[pci]" and then append this underneath it'. I could probably do something more clever, or use ansible, but I was in a hurry. :) > the alias needs to be defiend in the nova.conf used by the api node and the compute node > for it to work correctly but i assume that when you say its set on the contoler it set on > the nova.conf the nova api is useing. Exactly; that would be step 3.1 further down. > > > > - The PCI address for the card is different on each compute node > > > > 2.2. Create a flavor, say, n1.large > > openstack flavor create n1.large --id auto --ram 8192 --disk 80 > > --vcpus 4 --property "pci_passthrough:alias"="testnic:1" > this is also correct > > > > > 2.3. Restart openstack-nova-api > > > > 3. On each compute node > > 3.1. Create a PCI alias based on the vendor and product ID > > alias = { "vendor_id":"19fg", "product_id":"4000", > > "device_type":"type-PF", "name":"testnic" } > > > > 3.2. Create passthrough_whitelist entry > > passthrough_whitelist = { "vendor_id":"19fg", "product_id":"4000" } > assuming this is set in the pci section it also looks correct > https://docs.openstack.org/nova/latest/configuration/config.html#pci.passthrough_whitelist > > I actually put the passthrough_whitelist entry just below the alias one, which is below the [pci] label in the nova.conf file. Make it easier for me to find them later on. > > 3.3. Restart openstack-nova-compute > > > > 4. Create instance (vm guest) using the n1.large flavor. > > > > 5. Login to instance and discover dmesg and lspci does not list card > > > > 6. Do a "virsh dumpxml" for the instance on its compute node and > > discover there is no entry for the card listed in the xml file. I take > > nova would automagically do what I would if this was a kvm install, > > namely ensure card cannot be accessed/used by the host and then edit > > the guest xml file so it can see said card. > yes you should have seen it in the xml and the card should have been passed through to the guest. > > > > Questions: > > Q1: If a device is sr-iov capable, do I have to use that or can I just > > pass the entire card to the vm guest? > you can passthorugh the entire card to the guest yes. > Now, when I ask for the list of pci devices available in the compute nodes, why are they listed as type-PF? I am a bit concerned because it feels like it will be anxiously trying to virtualize it instead of just leaving said card alone, which I would expect with type-PCI. > > > Q2: Is there anywhere I can look for clues to why is the libvirt xml > > file for the instance not being populated with the pci card info? So > > far I only looked in the controller node's nova_scheduler.log file. > there are several things to check. > first i would check the nova compute agenet log and see if there are any tracebacks or errors > second in the nova cell db, often called just nova or nova_cell1 (not nova_cell0) check the pci_devices > table and see if the devices are listed. > > > Well, logs I can understand (we are talking about the nova-compute.log, right?) but I guess this is where my completely cluelessness shows up in grand style: I do not know where to look for that in the database. Nor could figure out how to talk to the REST interface using curl other than getting a token. So, I did a kludgy workaround and got the pci pool associated with each node, say [{'count': 1, 'product_id': u'4000', u'dev_type': u'type-PF', 'numa_node': 0, 'vendor_id': u'19fg'}] My ASSumption (yes, I know what they say about them) here is that when the object defining the compute node is updated, the database entry associated with it gets fed the pci pool I am seeing. In other words, that is the list of pci devices openstack things the node has. I guess this is the time I have to sheepishly admit that while one of the nodes has a single card, the other one has two; they are identified by being 'numa_node': 0 and 'numa_node': 1. Hopefully that will not cause issues. Then, I compared it to the pci request made before the instance is created: (alias_name='testnic',count=1,is_new=,numa_policy='legacy',request_id=None,requester_id=,spec=[{dev_type='type-PF',product_id='4000',vendor_id='19fg'}]) Since they both match, they satisfied the pci passthrough filter test and the instance was allowed to be spawned. That is as far as I went. From ignaziocassano at gmail.com Wed Jul 3 14:58:49 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Wed, 3 Jul 2019 16:58:49 +0200 Subject: [queens][magnuma] kubernetes cluster Message-ID: Hi All, I just installed openstack kolla queens with magnum but trying to create a kubernetes cluster the master nodes does not terminate installation: it loops with the following message: curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5 Anyone can help ? Best Regards Ignazio -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Wed Jul 3 19:03:12 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Wed, 3 Jul 2019 21:03:12 +0200 Subject: [magnum][queens] kume master error in clout-init-output.log Message-ID: Hi All, I 've just installed openstack kolla queens and wen I try to run a kubernetes cluster, the master node reports the following in the cloud init log file: Cloud-init v. 0.7.9 running 'modules:config' at Wed, 03 Jul 2019 18:57:19 +0000. Up 20.40 seconds. + CA_FILE=/etc/pki/ca-trust/source/anchors/openstack-ca.pem + '[' -n '' ']' /var/lib/cloud/instance/scripts/part-007: line 57: /etc/etcd/etcd.conf: No such file or directory /var/lib/cloud/instance/scripts/part-007: line 70: /etc/etcd/etcd.conf: No such file or directory /var/lib/cloud/instance/scripts/part-007: line 86: /etc/etcd/etcd.conf: No such file or directory Cloud-init v. 0.7.9 running 'modules:final' at Wed, 03 Jul 2019 18:57:21 +0000. Up 21.93 seconds. 2019-07-03 18:59:23,121 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-007 [1] The image I am using is suggested in the openstack documentation: $ wget https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2 The volume driver is cinder and the docker storage driver is overlay. The docker volume size is 5 GB and THE VOLUME SIZE OF THE FLAVOR IS 20gb. Seems image is missing some packeages :-( Anyone could help me, please ? Ignazio -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.adams at intel.com Tue Jul 2 23:24:39 2019 From: eric.adams at intel.com (Adams, Eric) Date: Tue, 2 Jul 2019 23:24:39 +0000 Subject: [kata-dev] [Zun][Kata] Devstack plugin for installing Kata container In-Reply-To: References: Message-ID: Hongbin, That is great to hear. When it is fully integrated we can replace our somewhat hacky Zun / Kata instructions at https://github.com/kata-containers/documentation/blob/master/use-cases/zun_kata.md which just takes the previous Clear Containers work enabling on OpenStack Zun and replaces it with the Kata runtime. Thanks, Eric From: Hongbin Lu [mailto:hongbin034 at gmail.com] Sent: Monday, July 01, 2019 2:08 PM To: kata-dev at lists.katacontainers.io; OpenStack Discuss Subject: [kata-dev] [Zun][Kata] Devstack plugin for installing Kata container Hi all, I have a patch [1] that adds support for installing kata Container in devstack. Right now, it configures kata container to run with Docker. In the future, I plan to add support for containerd's cri plugin, which basically allows running pods with kata container. Initially, OpenStack Zun will use this plugin to install Kata container, but I believe it would be beneficial for other projects. Appreciate if anyone interest to cast your feedback on the patch. [1] https://review.opendev.org/#/c/668490/ Best regards, Hongbin -------------- next part -------------- An HTML attachment was scrubbed... URL: From martialmichel at datamachines.io Wed Jul 3 04:36:41 2019 From: martialmichel at datamachines.io (Martial Michel) Date: Wed, 3 Jul 2019 00:36:41 -0400 Subject: [Scientific] No Scientific SIG meeting Jul 3rd Message-ID: Sending this email to inform you that we will not be having a Scientific SIG meeting Today. Hopefully next week -- Martial -------------- next part -------------- An HTML attachment was scrubbed... URL: From martialmichel at datamachines.io Wed Jul 3 04:36:41 2019 From: martialmichel at datamachines.io (Martial Michel) Date: Wed, 3 Jul 2019 00:36:41 -0400 Subject: [Scientific] No Scientific SIG meeting Jul 3rd Message-ID: Sending this email to inform you that we will not be having a Scientific SIG meeting Today. Hopefully next week -- Martial -------------- next part -------------- An HTML attachment was scrubbed... URL: From dp at servinga.com Wed Jul 3 15:31:16 2019 From: dp at servinga.com (Denis Pascheka) Date: Wed, 3 Jul 2019 17:31:16 +0200 (CEST) Subject: [queens][magnuma] kubernetes cluster In-Reply-To: References: Message-ID: <765458860.27505.1562167876903@ox.servinga.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 65536 bytes Desc: not available URL: From ignaziocassano at gmail.com Wed Jul 3 15:49:25 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Wed, 3 Jul 2019 17:49:25 +0200 Subject: [queens][magnuma] kubernetes cluster In-Reply-To: <765458860.27505.1562167876903@ox.servinga.com> References: <765458860.27505.1562167876903@ox.servinga.com> Message-ID: Thanks Denis, but on kube master the port 8080 is not listening ....probably some services are not active :-( Il Mer 3 Lug 2019 17:31 Denis Pascheka ha scritto: > Hi Ignazio, > > in Queens there is an issue within Magnum which has been resolved in the > Rocky release. > Take a look at this file: > https://github.com/openstack/magnum/blob/stable/rocky/magnum/drivers/common/templates/kubernetes/fragments/wc-notify-master.sh. > > The execution of the curl command in row 16 needs to be escaped with an > backslash. You can achieve this by building your own magnum containers > and > adding an template override > to > it where you add your fixed/own wc-notify-master.sh script from the plugin > directory > . > > > Best Regards, > > *Denis Pascheka* > Cloud Architect > > t: +49 (69) 348 75 11 12 > m: +49 (170) 495 6364 > e: dp at servinga.com > servinga GmbH > Mainzer Landstr. 351-353 > 60326 Frankfurt > > > > * > www.servinga.com * > > Amtsgericht Frankfurt am Main - HRB 91418 - Geschäftsführer Adam Lakota, > Christian Lertes > > Ignazio Cassano hat am 3. Juli 2019 um 16:58 > geschrieben: > > > Hi All, > I just installed openstack kolla queens with magnum but trying to create a > kubernetes cluster the master nodes does not terminate installation: it > loops with the following message: > > curl --silent http://127.0.0.1:8080/healthz > + '[' ok = '' ']' > + sleep 5 > > Anyone can help ? > Best Regards > Ignazio > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: From ignaziocassano at gmail.com Wed Jul 3 16:37:54 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Wed, 3 Jul 2019 18:37:54 +0200 Subject: [queens][magnuma] kubernetes cluster In-Reply-To: <765458860.27505.1562167876903@ox.servinga.com> References: <765458860.27505.1562167876903@ox.servinga.com> Message-ID: Thanks Denis, but I think there is another problem: on kube muster port 8080 is not listening, probably some services are note started Regards Ignazio Il giorno mer 3 lug 2019 alle ore 17:31 Denis Pascheka ha scritto: > Hi Ignazio, > > in Queens there is an issue within Magnum which has been resolved in the > Rocky release. > Take a look at this file: > https://github.com/openstack/magnum/blob/stable/rocky/magnum/drivers/common/templates/kubernetes/fragments/wc-notify-master.sh. > > The execution of the curl command in row 16 needs to be escaped with an > backslash. You can achieve this by building your own magnum containers > and > adding an template override > to > it where you add your fixed/own wc-notify-master.sh script from the plugin > directory > . > > > Best Regards, > > *Denis Pascheka* > Cloud Architect > > t: +49 (69) 348 75 11 12 > m: +49 (170) 495 6364 > e: dp at servinga.com > servinga GmbH > Mainzer Landstr. 351-353 > 60326 Frankfurt > > > > * > www.servinga.com * > > Amtsgericht Frankfurt am Main - HRB 91418 - Geschäftsführer Adam Lakota, > Christian Lertes > > Ignazio Cassano hat am 3. Juli 2019 um 16:58 > geschrieben: > > > Hi All, > I just installed openstack kolla queens with magnum but trying to create a > kubernetes cluster the master nodes does not terminate installation: it > loops with the following message: > > curl --silent http://127.0.0.1:8080/healthz > + '[' ok = '' ']' > + sleep 5 > > Anyone can help ? > Best Regards > Ignazio > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: From feilong at catalyst.net.nz Thu Jul 4 02:30:20 2019 From: feilong at catalyst.net.nz (Feilong Wang) Date: Thu, 4 Jul 2019 14:30:20 +1200 Subject: [magnum][queens] kume master error in clout-init-output.log In-Reply-To: References: Message-ID: Hi Ignazio, Based on the error message you provided, the etcd cluster is not started successfully. You can run the script part-007 manually to debug what's the root cause. On 4/07/19 7:03 AM, Ignazio Cassano wrote: > Hi All, > I 've just installed openstack kolla queens and wen I try to run a > kubernetes cluster, > the master node reports the following in the cloud init log file: > Cloud-init v. 0.7.9 running 'modules:config' at Wed, 03 Jul 2019 > 18:57:19 +0000. Up 20.40 seconds. > + CA_FILE=/etc/pki/ca-trust/source/anchors/openstack-ca.pem > + '[' -n '' ']' > > /var/lib/cloud/instance/scripts/part-007: line 57: > /etc/etcd/etcd.conf: No such file or directory > /var/lib/cloud/instance/scripts/part-007: line 70: > /etc/etcd/etcd.conf: No such file or directory > /var/lib/cloud/instance/scripts/part-007: line 86: > /etc/etcd/etcd.conf: No such file or directory > Cloud-init v. 0.7.9 running 'modules:final' at Wed, 03 Jul 2019 > 18:57:21 +0000. Up 21.93 seconds. > 2019-07-03 18:59:23,121 - util.py[WARNING]: Failed running > /var/lib/cloud/instance/scripts/part-007 [1] > || > |The image I am using is suggested in the openstack documentation: $ > wget > https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2 > | > |The volume driver is cinder and the docker storage driver is overlay. | > |The docker volume size is 5 GB and THE VOLUME SIZE OF THE FLAVOR IS > 20gb. | > |Seems image is missing some packeages :-( | > |Anyone could help me, please ? | > |Ignazio | > || > || -- Cheers & Best regards, Feilong Wang (王飞龙) -------------------------------------------------------------------------- Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang at catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington -------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Thu Jul 4 03:46:18 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Thu, 4 Jul 2019 05:46:18 +0200 Subject: [magnum][queens] kume master error in clout-init-output.log In-Reply-To: References: Message-ID: Thanks. I'll try Ignazio Il Gio 4 Lug 2019 04:39 Feilong Wang ha scritto: > Hi Ignazio, > > Based on the error message you provided, the etcd cluster is not started > successfully. You can run the script part-007 manually to debug what's the > root cause. > > > On 4/07/19 7:03 AM, Ignazio Cassano wrote: > > Hi All, > I 've just installed openstack kolla queens and wen I try to run a > kubernetes cluster, > the master node reports the following in the cloud init log file: > Cloud-init v. 0.7.9 running 'modules:config' at Wed, 03 Jul 2019 18:57:19 > +0000. Up 20.40 seconds. > + CA_FILE=/etc/pki/ca-trust/source/anchors/openstack-ca.pem > + '[' -n '' ']' > > /var/lib/cloud/instance/scripts/part-007: line 57: /etc/etcd/etcd.conf: No > such file or directory > /var/lib/cloud/instance/scripts/part-007: line 70: /etc/etcd/etcd.conf: No > such file or directory > /var/lib/cloud/instance/scripts/part-007: line 86: /etc/etcd/etcd.conf: No > such file or directory > Cloud-init v. 0.7.9 running 'modules:final' at Wed, 03 Jul 2019 18:57:21 > +0000. Up 21.93 seconds. > 2019-07-03 18:59:23,121 - util.py[WARNING]: Failed running > /var/lib/cloud/instance/scripts/part-007 [1] > > The image I am using is suggested in the openstack documentation: > $ wget https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2 > > The volume driver is cinder and the docker storage driver is overlay. > > The docker volume size is 5 GB and THE VOLUME SIZE OF THE FLAVOR IS 20gb. > > > Seems image is missing some packeages :-( > > Anyone could help me, please ? > > Ignazio > > -- > Cheers & Best regards, > Feilong Wang (王飞龙) > -------------------------------------------------------------------------- > Senior Cloud Software Engineer > Tel: +64-48032246 > Email: flwang at catalyst.net.nz > Catalyst IT Limited > Level 6, Catalyst House, 150 Willis Street, Wellington > -------------------------------------------------------------------------- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Thu Jul 4 04:49:42 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Thu, 4 Jul 2019 06:49:42 +0200 Subject: [magnum][queens] kume master error in clout-init-output.log In-Reply-To: References: Message-ID: Hello, the problem is that my externet network uses http proxy for downloading package from internet The file /etc/sysconfig/heat-params containes the HTTP_PROXY variable I passed to the cluster template but it does not export it for part007 script and etcd is not downloaded. If I change /etc/sysconfig/heat-params and insert at the head set -a and then I run part007 script, it works because reads the HTTP_PROXY variable and can download etcd. This seems a bug. Regards Ignazio Il giorno gio 4 lug 2019 alle ore 04:39 Feilong Wang < feilong at catalyst.net.nz> ha scritto: > Hi Ignazio, > > Based on the error message you provided, the etcd cluster is not started > successfully. You can run the script part-007 manually to debug what's the > root cause. > > > On 4/07/19 7:03 AM, Ignazio Cassano wrote: > > Hi All, > I 've just installed openstack kolla queens and wen I try to run a > kubernetes cluster, > the master node reports the following in the cloud init log file: > Cloud-init v. 0.7.9 running 'modules:config' at Wed, 03 Jul 2019 18:57:19 > +0000. Up 20.40 seconds. > + CA_FILE=/etc/pki/ca-trust/source/anchors/openstack-ca.pem > + '[' -n '' ']' > > /var/lib/cloud/instance/scripts/part-007: line 57: /etc/etcd/etcd.conf: No > such file or directory > /var/lib/cloud/instance/scripts/part-007: line 70: /etc/etcd/etcd.conf: No > such file or directory > /var/lib/cloud/instance/scripts/part-007: line 86: /etc/etcd/etcd.conf: No > such file or directory > Cloud-init v. 0.7.9 running 'modules:final' at Wed, 03 Jul 2019 18:57:21 > +0000. Up 21.93 seconds. > 2019-07-03 18:59:23,121 - util.py[WARNING]: Failed running > /var/lib/cloud/instance/scripts/part-007 [1] > > The image I am using is suggested in the openstack documentation: > $ wget https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2 > > The volume driver is cinder and the docker storage driver is overlay. > > The docker volume size is 5 GB and THE VOLUME SIZE OF THE FLAVOR IS 20gb. > > > Seems image is missing some packeages :-( > > Anyone could help me, please ? > > Ignazio > > -- > Cheers & Best regards, > Feilong Wang (王飞龙) > -------------------------------------------------------------------------- > Senior Cloud Software Engineer > Tel: +64-48032246 > Email: flwang at catalyst.net.nz > Catalyst IT Limited > Level 6, Catalyst House, 150 Willis Street, Wellington > -------------------------------------------------------------------------- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Thu Jul 4 06:13:01 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Thu, 4 Jul 2019 08:13:01 +0200 Subject: [magnum][queens] kume master error in clout-init-output.log In-Reply-To: References: Message-ID: Hello Feilong, do you work actively in magnum project ? If yes, how we could solve this problem ? Must we open a bug ? Alternatives ? Regards Ignazio Il giorno gio 4 lug 2019 alle ore 04:39 Feilong Wang < feilong at catalyst.net.nz> ha scritto: > Hi Ignazio, > > Based on the error message you provided, the etcd cluster is not started > successfully. You can run the script part-007 manually to debug what's the > root cause. > > > On 4/07/19 7:03 AM, Ignazio Cassano wrote: > > Hi All, > I 've just installed openstack kolla queens and wen I try to run a > kubernetes cluster, > the master node reports the following in the cloud init log file: > Cloud-init v. 0.7.9 running 'modules:config' at Wed, 03 Jul 2019 18:57:19 > +0000. Up 20.40 seconds. > + CA_FILE=/etc/pki/ca-trust/source/anchors/openstack-ca.pem > + '[' -n '' ']' > > /var/lib/cloud/instance/scripts/part-007: line 57: /etc/etcd/etcd.conf: No > such file or directory > /var/lib/cloud/instance/scripts/part-007: line 70: /etc/etcd/etcd.conf: No > such file or directory > /var/lib/cloud/instance/scripts/part-007: line 86: /etc/etcd/etcd.conf: No > such file or directory > Cloud-init v. 0.7.9 running 'modules:final' at Wed, 03 Jul 2019 18:57:21 > +0000. Up 21.93 seconds. > 2019-07-03 18:59:23,121 - util.py[WARNING]: Failed running > /var/lib/cloud/instance/scripts/part-007 [1] > > The image I am using is suggested in the openstack documentation: > $ wget https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2 > > The volume driver is cinder and the docker storage driver is overlay. > > The docker volume size is 5 GB and THE VOLUME SIZE OF THE FLAVOR IS 20gb. > > > Seems image is missing some packeages :-( > > Anyone could help me, please ? > > Ignazio > > -- > Cheers & Best regards, > Feilong Wang (王飞龙) > -------------------------------------------------------------------------- > Senior Cloud Software Engineer > Tel: +64-48032246 > Email: flwang at catalyst.net.nz > Catalyst IT Limited > Level 6, Catalyst House, 150 Willis Street, Wellington > -------------------------------------------------------------------------- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Thu Jul 4 07:24:57 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 4 Jul 2019 09:24:57 +0200 Subject: [neutron][ci] Gate broken Message-ID: <545F4CFE-2DAE-4B73-A47A-23FEC08C3AEC@redhat.com> Hi, Currently we have broken our functional and fullstack jobs due to patch [1] merged in Devstack. So functional and fullstack jobs are finishing with RETRY_LIMIT now. Fix is already proposed in [2]. So if Your patch failed on those jobs now, please don’t recheck it until [2] will be merged. [1] https://review.opendev.org/#/c/619562/ [2] https://review.opendev.org/#/c/669067/ — Slawek Kaplonski Senior software engineer Red Hat From moreira.belmiro.email.lists at gmail.com Thu Jul 4 08:14:40 2019 From: moreira.belmiro.email.lists at gmail.com (Belmiro Moreira) Date: Thu, 4 Jul 2019 10:14:40 +0200 Subject: [nova][ironic] Lock-related performance issue with update_resources periodic job In-Reply-To: References: Message-ID: Hi, we do have the issue of ironic instances taking a lot of time to start being created (The same Jason described). This is because the resource tracker takes >30 minutes to cycle (~2500 ironic nodes in one nova-compute). Meanwhile operations are "queue" until it finish. To speed up the resource tracker we use: https://review.opendev.org/#/c/637225/ We are working in shard the nova-compute for ironic. I think that is the right way to go. Considering the experience described by Jason we now increased the "update_resources_interval" to 24h. Yes, the "queue" issue disappeared. We will report back if you find some weird unexpected consequence. Belmiro CERN On Tue, Jun 11, 2019 at 5:56 PM Jason Anderson wrote: > Hi Surya, > > On 5/13/19 3:15 PM, Surya Seetharaman wrote: > > We faced the same problem at CERN when we upgraded to rocky (we have ~2300 > nodes on a single compute) like Eric said, and we set the > [compute]resource_provider_association_refresh to a large value (this > definitely helps by stopping the syncing of traits/aggregates and provider > tree cache info stuff in terms of chattiness with placement) and inspite of > that it doesn't scale that well for us. We still find the periodic task > taking too much of time which causes the locking to hold up the claim for > instances in BUILD state (the exact same problem you described). While one > way to tackle this like you said is to set the "update_resources_interval" > to a higher value - we were not sure how much out of sync things would get > with placement, so it will be interesting to see how this spans out for you > - another way out would be to use multiple computes and spread the nodes > around (though this is also a pain to maintain IMHO) which is what we are > looking into presently. > > I wanted to let you know that we've been running this way in production > for a few weeks now and it's had a noticeable improvement: instances are no > longer sticking in the "Build" stage, pre-networking, for ages. We were > able to track the improvement by comparing the Nova conductor logs ("Took > {seconds} to build the instance" vs "Took {seconds} to spawn the instance > on the hypervisor"; the delta should be as small as possible and in our > case went from ~30 minutes to ~1 minute.) There have been a few cases where > a resource provider claim got "stuck", but in practice it has been so > infrequent that it potentially has other causes. As such, I can recommend > increasing the interval time significantly. Currently we have it set to 6 > hours. > > I have not yet looked in to bringing in the other Nova patches used at > CERN (and available in Stein). I did take a look at updating the locking > mechanism, but do not have work to show for this yet. > > Cheers, > > /Jason > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ruslanas at lpic.lt Thu Jul 4 08:30:09 2019 From: ruslanas at lpic.lt (=?UTF-8?Q?Ruslanas_G=C5=BEibovskis?=) Date: Thu, 4 Jul 2019 10:30:09 +0200 Subject: tripleO undercloud import failing in stein Message-ID: Hi team, I am using CentOS7 with repos installed from stein package (centos-openstack-stein) in CentOS7. I just enable rdo-trunk-stein-tested because undercloud install do not work from centos-openstack-stein due to unable to mount some path, found that it is known and was fixed in stein tested. so I succeeded to boot and build everything. but when I use "time openstack --debug overcloud node import setup/hosts.yaml" step, it stops at: HTTP POST https://undercloud_public_host:13989/v2/executions 201 Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: 55f14621-cabd-4783-a611-d5b822ad0833 Waiting for messages on queue 'tripleo' with no timeout. I can reach iDRAC over ssh, and I can boot Physical HW to PXE. Physical HW state do not change, it do not change power state. current state is poweroff of physical hosts I am adding. any ideas? more info can be found here: https://ask.openstack.org/en/question/122870/valueerror-no-json-object-could-be-decoded-in-tripleo-stein/ -- Ruslanas Gžibovskis +370 6030 7030 -------------- next part -------------- An HTML attachment was scrubbed... URL: From openstack at fried.cc Thu Jul 4 10:16:00 2019 From: openstack at fried.cc (Eric Fried) Date: Thu, 4 Jul 2019 05:16:00 -0500 Subject: [nova][ironic] Lock-related performance issue with update_resources periodic job In-Reply-To: References: Message-ID: <37D021C4-ED1B-4942-9C90-0A26FDE3DD76@fried.cc> > https://review.opendev.org/#/c/637225/ Ah heck, I had totally forgotten about that patch. If it's working for you, let me get it polished up and merged. We could probably justify backporting it too. Matt? efried > On Jul 4, 2019, at 03:14, Belmiro Moreira wrote: > > Hi, > we do have the issue of ironic instances taking a lot of time to start being created (The same Jason described). > This is because the resource tracker takes >30 minutes to cycle (~2500 ironic nodes in one nova-compute). Meanwhile operations are "queue" until it finish. > To speed up the resource tracker we use: https://review.opendev.org/#/c/637225/ > > We are working in shard the nova-compute for ironic. I think that is the right way to go. > > Considering the experience described by Jason we now increased the "update_resources_interval" to 24h. > Yes, the "queue" issue disappeared. > > We will report back if you find some weird unexpected consequence. > > Belmiro > CERN > >> On Tue, Jun 11, 2019 at 5:56 PM Jason Anderson wrote: >> Hi Surya, >> >>> On 5/13/19 3:15 PM, Surya Seetharaman wrote: >>> We faced the same problem at CERN when we upgraded to rocky (we have ~2300 nodes on a single compute) like Eric said, and we set the [compute]resource_provider_association_refresh to a large value (this definitely helps by stopping the syncing of traits/aggregates and provider tree cache info stuff in terms of chattiness with placement) and inspite of that it doesn't scale that well for us. We still find the periodic task taking too much of time which causes the locking to hold up the claim for instances in BUILD state (the exact same problem you described). While one way to tackle this like you said is to set the "update_resources_interval" to a higher value - we were not sure how much out of sync things would get with placement, so it will be interesting to see how this spans out for you - another way out would be to use multiple computes and spread the nodes around (though this is also a pain to maintain IMHO) which is what we are looking into presently. >> I wanted to let you know that we've been running this way in production for a few weeks now and it's had a noticeable improvement: instances are no longer sticking in the "Build" stage, pre-networking, for ages. We were able to track the improvement by comparing the Nova conductor logs ("Took {seconds} to build the instance" vs "Took {seconds} to spawn the instance on the hypervisor"; the delta should be as small as possible and in our case went from ~30 minutes to ~1 minute.) There have been a few cases where a resource provider claim got "stuck", but in practice it has been so infrequent that it potentially has other causes. As such, I can recommend increasing the interval time significantly. Currently we have it set to 6 hours. >> >> I have not yet looked in to bringing in the other Nova patches used at CERN (and available in Stein). I did take a look at updating the locking mechanism, but do not have work to show for this yet. >> >> Cheers, >> >> /Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.rosser at rd.bbc.co.uk Thu Jul 4 10:30:45 2019 From: jonathan.rosser at rd.bbc.co.uk (Jonathan Rosser) Date: Thu, 4 Jul 2019 11:30:45 +0100 Subject: [magnum][queens] kume master error in clout-init-output.log In-Reply-To: References: Message-ID: <6fa4b20a-5023-8d97-0b5f-fc148b409ca7@rd.bbc.co.uk> Ignazio, You will something like this applied to Queens magnum https://review.opendev.org/#/c/637390/ which corrects the issue you have identified with the proxy environment variables not being exported. I would also suggest that you apply another patch https://review.opendev.org/#/c/667284 otherwise the proxy settings in the cluster template will interfere with the operation of magnum-conductor. Depending on your network environment you will have further issues without reverting that code. I would recommend using config in kolla to setup any http proxy you need for the magnum-conductor service - completely separate from any end user proxy settings that come from the cluster template. Hope this helps, Jon. On 04/07/2019 05:49, Ignazio Cassano wrote: > Hello, > the problem is that my externet network uses http proxy for downloading > package from internet > The file /etc/sysconfig/heat-params containes the HTTP_PROXY variable I > passed to the cluster template but it does not export it for part007 script > and etcd is not downloaded. > If I change /etc/sysconfig/heat-params and insert at the head > set -a > > and then I run part007 script, it works because reads the HTTP_PROXY > variable and can download etcd. > > This seems a bug. > Regards > Ignazio > > > Il giorno gio 4 lug 2019 alle ore 04:39 Feilong Wang < > feilong at catalyst.net.nz> ha scritto: > >> Hi Ignazio, >> >> Based on the error message you provided, the etcd cluster is not started >> successfully. You can run the script part-007 manually to debug what's the >> root cause. >> >> >> On 4/07/19 7:03 AM, Ignazio Cassano wrote: >> >> Hi All, >> I 've just installed openstack kolla queens and wen I try to run a >> kubernetes cluster, >> the master node reports the following in the cloud init log file: >> Cloud-init v. 0.7.9 running 'modules:config' at Wed, 03 Jul 2019 18:57:19 >> +0000. Up 20.40 seconds. >> + CA_FILE=/etc/pki/ca-trust/source/anchors/openstack-ca.pem >> + '[' -n '' ']' >> >> /var/lib/cloud/instance/scripts/part-007: line 57: /etc/etcd/etcd.conf: No >> such file or directory >> /var/lib/cloud/instance/scripts/part-007: line 70: /etc/etcd/etcd.conf: No >> such file or directory >> /var/lib/cloud/instance/scripts/part-007: line 86: /etc/etcd/etcd.conf: No >> such file or directory >> Cloud-init v. 0.7.9 running 'modules:final' at Wed, 03 Jul 2019 18:57:21 >> +0000. Up 21.93 seconds. >> 2019-07-03 18:59:23,121 - util.py[WARNING]: Failed running >> /var/lib/cloud/instance/scripts/part-007 [1] >> >> The image I am using is suggested in the openstack documentation: >> $ wget https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2 >> >> The volume driver is cinder and the docker storage driver is overlay. >> >> The docker volume size is 5 GB and THE VOLUME SIZE OF THE FLAVOR IS 20gb. >> >> >> Seems image is missing some packeages :-( >> >> Anyone could help me, please ? >> >> Ignazio >> >> -- >> Cheers & Best regards, >> Feilong Wang (王飞龙) >> -------------------------------------------------------------------------- >> Senior Cloud Software Engineer >> Tel: +64-48032246 >> Email: flwang at catalyst.net.nz >> Catalyst IT Limited >> Level 6, Catalyst House, 150 Willis Street, Wellington >> -------------------------------------------------------------------------- >> >> > From kchamart at redhat.com Thu Jul 4 10:31:48 2019 From: kchamart at redhat.com (Kashyap Chamarthy) Date: Thu, 4 Jul 2019 12:31:48 +0200 Subject: [ops][nova] Quick show of hands: any use Intel (non-CMT) `perf` events? Message-ID: <20190704103148.GF19519@paraplu> Heya folks, While removing some dead code I was wondering if anyone here uses "non-CMT" (Cache Monitoring Technology) performance events events? I'm referring to the events here[0], besides the first three, which are CMT-related. Background ---------- The Intel CMT events (there are three of them) were deprecated during the Rocky release, in this[1] commit, and with this rationale: Upstream Linux kernel has deleted[*] the `perf` framework integration with Intel CMT (Cache Monitoring Technology; or "CQM" in Linux kernel parlance), because the feature was broken by design -- an incompatibility between Linux's `perf` infrastructure and Intel CMT hardware support. It was removed in upstream kernel version v4.14; but bear in mind that downstream Linux distributions with lower kernel versions than 4.14 have backported the said change. Nova supports monitoring of the above mentioned Intel CMT events (namely: 'cmt', 'mbm_local', and 'mbm_total') via the configuration attribute `[libvirt]/enabled_perf_events`. Given that the underlying Linux kernel infrastructure for Intel CMT is removed, we should remove support for it in Nova too. Otherwise enabling them in Nova, and updating to a Linux kernel 4.14 (or above) will result in instances failing to boot. To that end, deprecate support for the three Intel CMT events in "Rocky" release, with the intention to remove support for it in the upcoming "Stein" release. Note that we cannot deprecate / remove `enabled_perf_events` config attribute altogether -- since there are other[+] `perf` events besides Intel CMT. Whether anyone is using those other events with Nova is a good question to which we don't have an equally good answer for, if at all. Now we're removing[2] support for CMT events altogether. Question -------- What I'm wondering now is the answer to the last sentence in the above quoted commit: "Whether anyone is using those other events with Nova is a good question to which we don't have an equally good answer for, if at all". If we know that "no one" (as if we can tell for sure) is using them, we can get rid of more dead code. So, any operators using the non-CMT events from here[0]? [0] https://libvirt.org/formatdomain.html#elementsPerf [1] https://opendev.org/openstack/nova/commit/fc4794acc6 —libvirt: Deprecate support for monitoring Intel CMT `perf` events [2] https://review.opendev.org/669129 — libvirt: Remove support for Intel CMT `perf` event -- /kashyap From zhangbailin at inspur.com Thu Jul 4 10:45:29 2019 From: zhangbailin at inspur.com (=?gb2312?B?QnJpbiBaaGFuZyjVxbDZwdYp?=) Date: Thu, 4 Jul 2019 10:45:29 +0000 Subject: reply: [lists.openstack.org][nova] API updates week 19-26 Message-ID: <29c14b1113644f559b56a52c2db2dfb1@inspur.com> > Spec Ready for Review: 5. Add flavor group https://review.opendev.org/#/c/663563/ Brin Zhang Hello Everyone, Please find the Nova API updates of this week. API Related BP : ============ COMPLETED: 1. Support adding description while locking an instance: - https://blueprints.launchpad.net/nova/+spec/add-locked-reason Code Ready for Review: ------------------------------ 1. Add host and hypervisor_hostname flag to create server - Topic: https://review.opendev.org/#/q/topic:bp/add-host-and-hypervisor-hostname-fla g-to-create-server+(status:open+OR+status:merged) - Weekly Progress: patch is updated with review comment. ready for re-review. I will re-review it tomorrow. 2. Specifying az when restore shelved server - Topic: https://review.opendev.org/#/q/topic:bp/support-specifying-az-when-restore-s helved-server+(status:open+OR+status:merged) - Weekly Progress: Review comments is fixed and ready to re-review. 3. Nova API cleanup - Topic: https://review.opendev.org/#/c/666889/ - Weekly Progress: Code is up for review. A lot of files changed but should be ok to review. I have pushed a couple of patches for missing tests of previous microversions. 4. Detach and attach boot volumes: - Topic: https://review.openstack.org/#/q/topic:bp/detach-boot-volume+(status:open+OR +status:merged) - Weekly Progress: No Progress Spec Ready for Review: ----------------------------- 1. Nova API policy improvement - Spec: https://review.openstack.org/#/c/547850/ - PoC: https://review.openstack.org/#/q/topic:bp/policy-default-refresh+(status:ope n+OR+status:merged) - Weekly Progress: Under review and updates. 2. Support for changing deleted_on_termination after boot -Spec: https://review.openstack.org/#/c/580336/ - Weekly Progress: No update this week. 3. Support delete_on_termination in volume attach api -Spec: https://review.openstack.org/#/c/612949/ - Weekly Progress: No updates this week. 4. Add API ref guideline for body text - ~8 api-ref are left to fix. Previously approved Spec needs to be re-proposed for Train: --------------------------------------------------------------------------- 1. Servers Ips non-unique network names : - https://blueprints.launchpad.net/nova/+spec/servers-ips-non-unique-network-n ames - https://review.openstack.org/#/q/topic:bp/servers-ips-non-unique-network-nam es+(status:open+OR+status:merged) 2. Volume multiattach enhancements: - https://blueprints.launchpad.net/nova/+spec/volume-multiattach-enhancements - https://review.openstack.org/#/q/topic:bp/volume-multiattach-enhancements+(s tatus:open+OR+status:merged) Bugs: ==== No progress report in this week too. I will start the bug triage next week. NOTE- There might be some bug which is not tagged as 'api' or 'api-ref', those are not in the above list. Tag such bugs so that we can keep our eyes. -gmann From renat.akhmerov at gmail.com Thu Jul 4 10:52:16 2019 From: renat.akhmerov at gmail.com (Renat Akhmerov) Date: Thu, 4 Jul 2019 17:52:16 +0700 Subject: [Mistral] PTL is on vacation till July 17 In-Reply-To: <698d16c2-c7aa-4ffa-a6a9-d454a0cb8d55@Spark> References: <698d16c2-c7aa-4ffa-a6a9-d454a0cb8d55@Spark> Message-ID: Hi, Just letting you know that I’ll be on vacation till July 17. With all the urgent requests/issue please contact Dougal Matthews (d0gal), Oleg Ovcharuk (vgoleg) or Adriano Petrich (apetrich). Thanks Renat Akhmerov @Nokia -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Thu Jul 4 10:58:10 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 4 Jul 2019 12:58:10 +0200 Subject: [neutron][ci] Gate broken In-Reply-To: <545F4CFE-2DAE-4B73-A47A-23FEC08C3AEC@redhat.com> References: <545F4CFE-2DAE-4B73-A47A-23FEC08C3AEC@redhat.com> Message-ID: <390CD4A3-78D1-4C42-8F4F-DE6A2E2CFFCA@redhat.com> Hi, Fix is merged now. You can now recheck Your patches :) > On 4 Jul 2019, at 09:24, Slawek Kaplonski wrote: > > Hi, > > Currently we have broken our functional and fullstack jobs due to patch [1] merged in Devstack. > So functional and fullstack jobs are finishing with RETRY_LIMIT now. > Fix is already proposed in [2]. > So if Your patch failed on those jobs now, please don’t recheck it until [2] will be merged. > > [1] https://review.opendev.org/#/c/619562/ > [2] https://review.opendev.org/#/c/669067/ > > — > Slawek Kaplonski > Senior software engineer > Red Hat > — Slawek Kaplonski Senior software engineer Red Hat From ignaziocassano at gmail.com Thu Jul 4 11:18:14 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Thu, 4 Jul 2019 13:18:14 +0200 Subject: [magnum][queens] kume master error in clout-init-output.log In-Reply-To: <6fa4b20a-5023-8d97-0b5f-fc148b409ca7@rd.bbc.co.uk> References: <6fa4b20a-5023-8d97-0b5f-fc148b409ca7@rd.bbc.co.uk> Message-ID: Many thanks Ignazio Il giorno gio 4 lug 2019 alle ore 12:38 Jonathan Rosser < jonathan.rosser at rd.bbc.co.uk> ha scritto: > Ignazio, > > You will something like this applied to Queens magnum > https://review.opendev.org/#/c/637390/ which corrects the issue you have > identified with the proxy environment variables not being exported. > > I would also suggest that you apply another patch > https://review.opendev.org/#/c/667284 otherwise the proxy settings in > the cluster template will interfere with the operation of > magnum-conductor. Depending on your network environment you will have > further issues without reverting that code. > > I would recommend using config in kolla to setup any http proxy you need > for the magnum-conductor service - completely separate from any end user > proxy settings that come from the cluster template. > > Hope this helps, > Jon. > > On 04/07/2019 05:49, Ignazio Cassano wrote: > > Hello, > > the problem is that my externet network uses http proxy for downloading > > package from internet > > The file /etc/sysconfig/heat-params containes the HTTP_PROXY variable I > > passed to the cluster template but it does not export it for part007 > script > > and etcd is not downloaded. > > If I change /etc/sysconfig/heat-params and insert at the head > > set -a > > > > and then I run part007 script, it works because reads the HTTP_PROXY > > variable and can download etcd. > > > > This seems a bug. > > Regards > > Ignazio > > > > > > Il giorno gio 4 lug 2019 alle ore 04:39 Feilong Wang < > > feilong at catalyst.net.nz> ha scritto: > > > >> Hi Ignazio, > >> > >> Based on the error message you provided, the etcd cluster is not started > >> successfully. You can run the script part-007 manually to debug what's > the > >> root cause. > >> > >> > >> On 4/07/19 7:03 AM, Ignazio Cassano wrote: > >> > >> Hi All, > >> I 've just installed openstack kolla queens and wen I try to run a > >> kubernetes cluster, > >> the master node reports the following in the cloud init log file: > >> Cloud-init v. 0.7.9 running 'modules:config' at Wed, 03 Jul 2019 > 18:57:19 > >> +0000. Up 20.40 seconds. > >> + CA_FILE=/etc/pki/ca-trust/source/anchors/openstack-ca.pem > >> + '[' -n '' ']' > >> > >> /var/lib/cloud/instance/scripts/part-007: line 57: /etc/etcd/etcd.conf: > No > >> such file or directory > >> /var/lib/cloud/instance/scripts/part-007: line 70: /etc/etcd/etcd.conf: > No > >> such file or directory > >> /var/lib/cloud/instance/scripts/part-007: line 86: /etc/etcd/etcd.conf: > No > >> such file or directory > >> Cloud-init v. 0.7.9 running 'modules:final' at Wed, 03 Jul 2019 18:57:21 > >> +0000. Up 21.93 seconds. > >> 2019-07-03 18:59:23,121 - util.py[WARNING]: Failed running > >> /var/lib/cloud/instance/scripts/part-007 [1] > >> > >> The image I am using is suggested in the openstack documentation: > >> $ wget > https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2 > >> > >> The volume driver is cinder and the docker storage driver is overlay. > >> > >> The docker volume size is 5 GB and THE VOLUME SIZE OF THE FLAVOR IS > 20gb. > >> > >> > >> Seems image is missing some packeages :-( > >> > >> Anyone could help me, please ? > >> > >> Ignazio > >> > >> -- > >> Cheers & Best regards, > >> Feilong Wang (王飞龙) > >> > -------------------------------------------------------------------------- > >> Senior Cloud Software Engineer > >> Tel: +64-48032246 > >> Email: flwang at catalyst.net.nz > >> Catalyst IT Limited > >> Level 6, Catalyst House, 150 Willis Street, Wellington > >> > -------------------------------------------------------------------------- > >> > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ekuvaja at redhat.com Thu Jul 4 11:55:33 2019 From: ekuvaja at redhat.com (Erno Kuvaja) Date: Thu, 4 Jul 2019 12:55:33 +0100 Subject: [Glance] No meeting today 4th of July Message-ID: Hi all, We had no proposed agenda for today so Happy Independence Day for all our American fellows and we will have meeting again next Thu the 11th. Best, Erno jokke Kuvaja -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdent+os at anticdent.org Thu Jul 4 15:04:46 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Thu, 4 Jul 2019 16:04:46 +0100 (BST) Subject: [placement] [tc] PTL not going to summit? Message-ID: I've decided [1] that I'm going to resist going to tech conferences and summits in particular by air travel, unless it becomes an existential issue. For a variety of reasons described in [1] and the two other posts it links to. I know for some people that this will present some concerns about my efficacy as PTL of placement, so I thought I better mention it sooner than later so we can make some decisions on how we want to proceed: * Should I take myself out of the running for U PTL? In which case we need to get a shadow warmed up. * If I'm not there, but still PTL, should we still hold the usual regularly placement things (project update and onboarding, PTG things)? In which case we need to prepare chairs/moderators for those sessions. Given my positions on the exclusive properties of conferences, I'd prefer that we turn update and onboarding activities into asynchronous, document-oriented affairs that anyone can utilize at any time, not just those wanting and able to go to summit. Similarly, given how useful the placement pre-PTG was for Train, the very small amount of time spent discussing placement issues in person in Denver, and the team's strategy of focusing on a relatively small number of changes, I'm not certain we need "time" in Shanghai. We can spread that work out. Thoughts or questions from anyone? [1] https://anticdent.org/remote-maintainership.html -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From mark at stackhpc.com Thu Jul 4 15:16:41 2019 From: mark at stackhpc.com (Mark Goddard) Date: Thu, 4 Jul 2019 16:16:41 +0100 Subject: [placement] [tc] PTL not going to summit? In-Reply-To: References: Message-ID: On Thu, 4 Jul 2019 at 16:06, Chris Dent wrote: > > I've decided [1] that I'm going to resist going to tech conferences > and summits in particular by air travel, unless it becomes an > existential issue. For a variety of reasons described in [1] and the > two other posts it links to. > > I know for some people that this will present some concerns about my > efficacy as PTL of placement, so I thought I better mention it > sooner than later so we can make some decisions on how we want to > proceed: > > * Should I take myself out of the running for U PTL? In which case > we need to get a shadow warmed up. > > * If I'm not there, but still PTL, should we still hold the usual > regularly placement things (project update and onboarding, PTG > things)? In which case we need to prepare chairs/moderators for > those sessions. > > Given my positions on the exclusive properties of conferences, I'd > prefer that we turn update and onboarding activities into > asynchronous, document-oriented affairs that anyone can utilize at > any time, not just those wanting and able to go to summit. > > Similarly, given how useful the placement pre-PTG was for Train, the > very small amount of time spent discussing placement issues in > person in Denver, and the team's strategy of focusing on a > relatively small number of changes, I'm not certain we need "time" > in Shanghai. We can spread that work out. > > Thoughts or questions from anyone? > Thanks for speaking up about your feelings on this. I'm not going so far as to say I won't fly to conferences, but one intercontinental flight in a year for conferences seems like more than enough to me. I won't be attending the Shanghai summit, and environmental factors play a role in this decision (as will an imminent newborn), so I'll watch how this plays out with interest. Mark > > [1] https://anticdent.org/remote-maintainership.html > > -- > Chris Dent ٩◔̯◔۶ https://anticdent.org/ > freenode: cdent -------------- next part -------------- An HTML attachment was scrubbed... URL: From emilien at redhat.com Thu Jul 4 15:45:09 2019 From: emilien at redhat.com (Emilien Macchi) Date: Thu, 4 Jul 2019 11:45:09 -0400 Subject: [placement] [tc] PTL not going to summit? In-Reply-To: References: Message-ID: On Thu, Jul 4, 2019 at 11:19 AM Chris Dent wrote: > > I've decided [1] that I'm going to resist going to tech conferences > and summits in particular by air travel, unless it becomes an > existential issue. For a variety of reasons described in [1] and the > two other posts it links to. > > I know for some people that this will present some concerns about my > efficacy as PTL of placement, so I thought I better mention it > sooner than later so we can make some decisions on how we want to > proceed: > > * Should I take myself out of the running for U PTL? In which case > we need to get a shadow warmed up. > > * If I'm not there, but still PTL, should we still hold the usual > regularly placement things (project update and onboarding, PTG > things)? In which case we need to prepare chairs/moderators for > those sessions. > > Given my positions on the exclusive properties of conferences, I'd > prefer that we turn update and onboarding activities into > asynchronous, document-oriented affairs that anyone can utilize at > any time, not just those wanting and able to go to summit. > > Similarly, given how useful the placement pre-PTG was for Train, the > very small amount of time spent discussing placement issues in > person in Denver, and the team's strategy of focusing on a > relatively small number of changes, I'm not certain we need "time" > in Shanghai. We can spread that work out. > > Thoughts or questions from anyone? > > [1] https://anticdent.org/remote-maintainership.html > Thanks a ton Chris for speaking up on that topic. I also share your opinions in this blog. I've talked with a few TripleO contributors and a bunch of us won't go to China either (for different reasons). Instead, I think we'll try to make progress in our asynchronous collaboration and eventually organize a virtual meetup if needed. Also, in my humble opinion you shouldn't step out of PTL role just because you won't go to the next conference. I think it's part of the PTL role to find out how to make the collaboration happen without barriers, no matter where you are in the world. Thanks for all your hard work on Placement and I hope you'll make the right decision for you and the project. -- Emilien Macchi -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Thu Jul 4 16:27:43 2019 From: smooney at redhat.com (Sean Mooney) Date: Thu, 04 Jul 2019 17:27:43 +0100 Subject: [placement] [tc] PTL not going to summit? In-Reply-To: References: Message-ID: <2841fd06a30a9920a4cd6af5c5eee19863e6f27a.camel@redhat.com> On Thu, 2019-07-04 at 11:45 -0400, Emilien Macchi wrote: > On Thu, Jul 4, 2019 at 11:19 AM Chris Dent wrote: > > > > > I've decided [1] that I'm going to resist going to tech conferences > > and summits in particular by air travel, unless it becomes an > > existential issue. For a variety of reasons described in [1] and the > > two other posts it links to. > > > > I know for some people that this will present some concerns about my > > efficacy as PTL of placement, so I thought I better mention it > > sooner than later so we can make some decisions on how we want to > > proceed: > > > > * Should I take myself out of the running for U PTL? In which case > > we need to get a shadow warmed up. > > > > * If I'm not there, but still PTL, should we still hold the usual > > regularly placement things (project update and onboarding, PTG > > things)? In which case we need to prepare chairs/moderators for > > those sessions. > > > > Given my positions on the exclusive properties of conferences, I'd > > prefer that we turn update and onboarding activities into > > asynchronous, document-oriented affairs that anyone can utilize at > > any time, not just those wanting and able to go to summit. > > > > Similarly, given how useful the placement pre-PTG was for Train, the > > very small amount of time spent discussing placement issues in > > person in Denver, and the team's strategy of focusing on a > > relatively small number of changes, I'm not certain we need "time" > > in Shanghai. We can spread that work out. > > > > Thoughts or questions from anyone? > > > > [1] https://anticdent.org/remote-maintainership.html > > > > Thanks a ton Chris for speaking up on that topic. I also share your > opinions in this blog. > I've talked with a few TripleO contributors and a bunch of us won't go to > China either (for different reasons). > > Instead, I think we'll try to make progress in our asynchronous > collaboration and eventually organize a virtual meetup if needed. > Also, in my humble opinion you shouldn't step out of PTL role just because > you won't go to the next conference. I think it's part of the PTL role to > find out how to make the collaboration happen without barriers, no matter > where you are in the world. Thanks for all your hard work on Placement and > I hope you'll make the right decision for you and the project. on this point i also dont think you would be the firsts ptl to not by able to attend ptg in person. i rememebr one of the past cyborge PTLs was not able to attend in person i belive in the first denver ptg and instead joined over voice chat to listen and replied mainly over etherpad to the discussion. i have remote attened 2 kolla midcycle in the past too were similarly by have a listen only audio stream and replying on etherpad as things were bing disucssed i was able to follow then conversation so i personally would not see an issue with a PTL delegating project updates or onbodaing to a core memeber that is present or chossing to hold a virtual ptg meetup. i belive there is still value in having in face discussion but including remotees in those discussion is something that i think we should try for too. so i dont think it should be a requriement to be in person the be ptl once your poject can effectivly discuss and organise your work within itself and with other projects. From fungi at yuggoth.org Thu Jul 4 17:19:59 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 4 Jul 2019 17:19:59 +0000 Subject: [placement] [tc] PTL not going to summit? In-Reply-To: References: Message-ID: <20190704171959.kbdchzlj7c72exkj@yuggoth.org> On 2019-07-04 16:04:46 +0100 (+0100), Chris Dent wrote: > I've decided [1] that I'm going to resist going to tech conferences > and summits in particular by air travel, unless it becomes an > existential issue. For a variety of reasons described in [1] and the > two other posts it links to. > > I know for some people that this will present some concerns about my > efficacy as PTL of placement [...] As Sean notes in his reply, you'd be far from the first PTL (or TC member or other community leader for that matter) to not attend a summit/forum/PTG. I've never heard anyone suggest this was a hard requirement for OpenStack contributors, whether or not they're hold leadership roles. There are plenty of good reasons not to attend, up to and including simply not wanting to be there. I see no problem with that whatsoever. > Given my positions on the exclusive properties of conferences, I'd > prefer that we turn update and onboarding activities into > asynchronous, document-oriented affairs that anyone can utilize at > any time, not just those wanting and able to go to summit. [...] The project updates have served as an opportunity to get video content about projects distributed, since these are typically recorded and post-processed by professional videographers. There have been suggestions of doing the same for onboarding sessions, but this has not happened in the past due to the additional cost required. Producing your own recordings seems like a viable alternative to me, and has also been suggested by a number of folks in the past. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From feilong at catalyst.net.nz Thu Jul 4 02:28:35 2019 From: feilong at catalyst.net.nz (Feilong Wang) Date: Thu, 4 Jul 2019 14:28:35 +1200 Subject: [queens][magnuma] kubernetes cluster In-Reply-To: References: <765458860.27505.1562167876903@ox.servinga.com> Message-ID: Hi Ignazio, We fixed a lot of issues in Rocky and Stein, which some of them can't be easily backported to Queens. Magnum has a very loose dependency for other services, so I would suggest to use rocky or stein if it's possible for you. As for your issue, the error means your kube-apiserver didn't start successfully. You can take a look the cloud init log for more information. On 4/07/19 4:37 AM, Ignazio Cassano wrote: > Thanks Denis, but I think there is another problem: on kube muster > port 8080 is not listening, probably some services are note started > Regards > Ignazio > > Il giorno mer 3 lug 2019 alle ore 17:31 Denis Pascheka > > ha scritto: > > Hi Ignazio,  > > in Queens there is an issue within Magnum which has been resolved > in the Rocky release. > Take a look at this file:  > https://github.com/openstack/magnum/blob/stable/rocky/magnum/drivers/common/templates/kubernetes/fragments/wc-notify-master.sh. > > The execution of the curl command in row 16 needs to be escaped > with an backslash. You can achieve this by building your own > magnum containers >  and > adding an template override >  to > it where you add your fixed/own wc-notify-master.sh script from > the plugin directory > . > > > Best Regards, > > *Denis Pascheka* > Cloud Architect > > t: +49 (69) 348 75 11 12 > m: +49 (170) 495 6364 > e: dp at servinga.com > servinga GmbH > Mainzer Landstr. 351-353 > 60326 Frankfurt > > > > > > *www.servinga.com * > Amtsgericht Frankfurt am Main - HRB 91418 - Geschäftsführer Adam > Lakota, Christian Lertes > >> Ignazio Cassano > > hat am 3. Juli 2019 um 16:58 >> geschrieben: >> >> >> Hi All, >> I just installed openstack kolla queens with magnum but trying to >> create a kubernetes cluster the master nodes does not terminate >> installation: it loops with the following message: >> >> curl --silent http://127.0.0.1:8080/healthz >> + '[' ok = '' ']' >> + sleep 5 >> >> Anyone can help ? >> Best Regards >> Ignazio > >   > -- Cheers & Best regards, Feilong Wang (王飞龙) -------------------------------------------------------------------------- Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang at catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington -------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: From kklimonda at syntaxhighlighted.com Thu Jul 4 23:12:49 2019 From: kklimonda at syntaxhighlighted.com (Krzysztof Klimonda) Date: Fri, 5 Jul 2019 01:12:49 +0200 Subject: [horizon] Unable to boot instance from volume if image size doesn't fit in flavor root disk size Message-ID: <35A8A85A-AA78-49DF-A1AB-2AB4046438B9@syntaxhighlighted.com> Hi, In order to avoid creating spurious flavors just to match image size requirements, I’d like to be able to spawn instances from volume even if the flavor disk is too small - this is possible via CLI/API but Horizon doesn’t let me launch the instance unless either flavor root disk is sized correctly, or the root disk size is 0(?). Is that a bug in Horizon, or is there something that I’m missing? Regards, -Chris From massimo.sgaravatto at gmail.com Fri Jul 5 06:45:02 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Fri, 5 Jul 2019 08:45:02 +0200 Subject: [ops] [nova] [placement] Mismatch between allocations and instances Message-ID: I tried to check the allocations on each compute node of a Ocata cloud, using the command: curl -s ${PLACEMENT_ENDPOINT}/resource_providers/${UUID}/allocations -H "x-auth-token: $TOKEN" | python -m json.tool I found that, on a few compute nodes, there are some instances for which there is not a corresponding allocation. On another Rocky cloud, we had the opposite problem: there were allocations also for some instances that didn't exist anymore. And this caused problems since we were not able to use all the resources of the relevant compute nodes: we had to manually remove the fwrong" allocations to fix the problem ... I wonder why/how this problem can happen ... And how can we fix the issue ? Should we manually add the missing allocations / manually remove the wrong ones ? Thanks, Massimo -------------- next part -------------- An HTML attachment was scrubbed... URL: From e0ne at e0ne.info Fri Jul 5 06:57:59 2019 From: e0ne at e0ne.info (Ivan Kolodyazhny) Date: Fri, 5 Jul 2019 09:57:59 +0300 Subject: [horizon] Unable to boot instance from volume if image size doesn't fit in flavor root disk size In-Reply-To: <35A8A85A-AA78-49DF-A1AB-2AB4046438B9@syntaxhighlighted.com> References: <35A8A85A-AA78-49DF-A1AB-2AB4046438B9@syntaxhighlighted.com> Message-ID: Hi Chris, It sounds like a horizon bug (TBH, I didn't try to reproduce it yet), so feel free to file a bug in the Launchpad. Regards, Ivan Kolodyazhny, http://blog.e0ne.info/ On Fri, Jul 5, 2019 at 2:15 AM Krzysztof Klimonda < kklimonda at syntaxhighlighted.com> wrote: > Hi, > > In order to avoid creating spurious flavors just to match image size > requirements, I’d like to be able to spawn instances from volume even if > the flavor disk is too small - this is possible via CLI/API but Horizon > doesn’t let me launch the instance unless either flavor root disk is sized > correctly, or the root disk size is 0(?). > > Is that a bug in Horizon, or is there something that I’m missing? > > Regards, > -Chris > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at stackhpc.com Fri Jul 5 07:33:03 2019 From: mark at stackhpc.com (Mark Goddard) Date: Fri, 5 Jul 2019 08:33:03 +0100 Subject: [kolla][kayobe] vote: kayobe as a kolla deliverable In-Reply-To: References: Message-ID: On Thu, 20 Jun 2019 at 14:40, Mark Goddard wrote: > Hi, > > In the most recent kolla meeting [1] we discussed the possibility of > kayobe becoming a deliverable of the kolla project. This follows on > from discussion at the PTG and then on here [3]. > > The two options discussed are: > > 1. become a deliverable of the Kolla project > 2. become an official top level OpenStack project > > There has been some positive feedback about option 1 and no negative > feedback that I am aware of. I would therefore like to ask the kolla > community to vote on whether to include kayobe as a deliverable of the > kolla project. The electorate is the kolla-core and kolla-ansible core > teams, excluding me. The opinion of others in the community is also > welcome. > > If you have questions or feedback, please respond to this email. > > Once you have made a decision, please respond with your answer to the > following question: > > "Should kayobe become a deliverable of the kolla project?" (yes/no) > > This vote has been open for over two weeks, and has had a number of positive responses and no negative responses. I will therefore make the necessary changes to add kayobe as a deliverable of the kolla project. Thank you for your consideration. Thanks, > Mark > > [1] > http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.log.html#l-120 > [2] https://etherpad.openstack.org/p/kolla-train-ptg > [3] > http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006901.html > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at stackhpc.com Fri Jul 5 07:39:18 2019 From: mark at stackhpc.com (Mark Goddard) Date: Fri, 5 Jul 2019 08:39:18 +0100 Subject: [kolla] Proposing yoctozepto as core In-Reply-To: References: Message-ID: On Fri, 28 Jun 2019 at 12:50, Mark Goddard wrote: > Hi, > > I would like to propose adding Radosław Piliszek (yoctozepto) to > kolla-core and kolla-ansible-core. While he has only recently started > working upstream in the project, I feel that he has made some valuable > contributions already, particularly around improving and maintaining > CI. His reviews generally provide useful feedback, sometimes in > advance of Zuul! > > Core team - please vote, and consider this my +1. I will keep this > vote open for a week or until all cores have responded. > This has been open for a week, and has had only positive responses. Welcome to the team yoctozepto! In less happy news, I will be removing zhubingbing and Duong Ha-Quang from the kolla-core and kolla-ansible-core groups. Thanks to both for their contributions to the project over the years. Mark > Cheers, > Mark > -------------- next part -------------- An HTML attachment was scrubbed... URL: From e0ne at e0ne.info Fri Jul 5 07:51:41 2019 From: e0ne at e0ne.info (Ivan Kolodyazhny) Date: Fri, 5 Jul 2019 10:51:41 +0300 Subject: [horizon] PTL on vacation Message-ID: Hi team, I'll be on a vacation until July, 19th with a very limited internet connection. During the last weekly meeting [1] we agreed to skip the next two. I'll try to check my mail during the PTO but for any issues, I recommend to ask our great core team [2] via emails or IRC. [1] http://eavesdrop.openstack.org/meetings/horizon/2019/horizon.2019-07-03-15.08.log.html [2] https://review.opendev.org/#/admin/groups/43,members Regards, Ivan Kolodyazhny, http://blog.e0ne.info/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From raubvogel at gmail.com Fri Jul 5 11:58:09 2019 From: raubvogel at gmail.com (Mauricio Tavares) Date: Fri, 5 Jul 2019 07:58:09 -0400 Subject: [nova] type-PCI vs type-PF vs type-VF Message-ID: Quick questions: 1. Is there some official documentation comparing them? Only thing I found so far was https://mohamede.com/2019/02/07/pci-passthrough-type-pf-type-vf-and-type-pci/ 2. Am I correct to assume (I seem to do a lot of that; be afraid) that when nova starts populating the database entry for a given physical node (say a computer node or a ironic baremetal node) it looks through the pci list and decides what type of device (type-PCI, type-PF, or type-VF) it is? What is the criteria? 3. https://mohamede.com/2019/02/07/pci-passthrough-type-pf-type-vf-and-type-pci/ implies that nova sees the 3 different types, well, differently where it will try hard to virtualize type-PF and type-VF and really really wants me to setup sr-iov on them ( https://docs.openstack.org/nova/latest/admin/pci-passthrough.html seem to also want sr-iov) really . Is that correct? -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at ic.unicamp.br Thu Jul 4 21:58:17 2019 From: william at ic.unicamp.br (William Lima Reiznautt) Date: Thu, 04 Jul 2019 18:58:17 -0300 Subject: [nova] consoleauth token ttl Message-ID: <20190704185817.Horde.kZVzaR7qWwNvYreX3yw-Bae@webmail2.ic.unicamp.br> Hello folks, Some known about this configuration token_ttl on [consoleauth] section: Is it on second or minutes ? At documentation is not information. Sorry the question here. -- -- William Lima Reiznautt IC/UNICAMP - Telefone: +55 (19) 3521-5915 --- chavepgp: http://www.ic.unicamp.br/~william/william.pgp From arteom.lap at yandex.ru Fri Jul 5 06:19:53 2019 From: arteom.lap at yandex.ru (=?utf-8?B?0JvQsNC/0LjQvSDQkNGA0YLQtdC8?=) Date: Fri, 05 Jul 2019 09:19:53 +0300 Subject: [Mistral] Guaranteed notification delivery Message-ID: <13379051562307593@iva3-4a4d8f90d111.qloud-c.yandex.net> Good day, please look at https://blueprints.launchpad.net/mistral/+spec/guaranteed-notifies. I would like to know your opinion about this feature. From arteom.lap at yandex.ru Fri Jul 5 09:06:10 2019 From: arteom.lap at yandex.ru (=?utf-8?B?0JvQsNC/0LjQvSDQkNGA0YLQtdC8?=) Date: Fri, 05 Jul 2019 12:06:10 +0300 Subject: [Mistral] Guaranteed notification delivery Message-ID: <25961801562317570@sas2-a1efad875d04.qloud-c.yandex.net> Good day, please look at https://blueprints.launchpad.net/mistral/+spec/guaranteed-notifies. I would like to know your opinion about this feature. From cdent+os at anticdent.org Fri Jul 5 13:01:40 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Fri, 5 Jul 2019 14:01:40 +0100 (BST) Subject: [placement] update 19-26 Message-ID: HTML: https://anticdent.org/placement-update-19-26.html Pupdate 19-26. Next week is R-15, two weeks until Train milestone 2. # Most Important The [spec for nested magic](https://docs.openstack.org/placement/latest/specs/train/approved/2005575-nested-magic-1.html) merged and significant progress has been made in the implementation. That work is nearly ready to merge (see below), after a few more reviews. Once that happens one of our most important tasks will be experimenting with that code to make sure it fully addresses the uses cases, has proper documentation (including "how do I use this?"), and is properly evaluated for performance and maintainability. # What's Changed * The implementation for mappings in allocation candidates had a bug which Eric [found](https://review.opendev.org/668302) and fixed and then I realized there was a [tidier way to do it](https://review.opendev.org/668724). This then led to the `same_subtree` work needing to manage less information, because it was already there. * The spec for [Consumer Types](https://docs.openstack.org/placement/latest/specs/train/approved/2005473-support-consumer-types.html) merged and work [has started](https://review.opendev.org/669170). * We're using os-traits 0.15.0 now. * There's a [framework in place](https://review.opendev.org/665695) for nested resource provider peformance testing. We need to update the provider topology to reflect real world situations (more on that below). * The `root_required` query parameter on `GET /allocation_candidates` has been merged as [microversion 1.35](https://docs.openstack.org/placement/latest/placement-api-microversion-history.html#support-root-required-queryparam-on-get-allocation-candidates). * I've sent [an email](http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007527.html) announcing my intent to not go to the Shangai (or any other) summit, and what changes that could imply for how Placement does the PTG. # Specs/Features All placement specs have merged. Thanks to everyone for the frequent reviews and quick followups. We've been maintaining some good velocity. Some non-placement specs are listed in the Other section below. # Stories/Bugs (Numbers in () are the change since the last pupdate.) There are 23 (3) stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 (0) are [untagged](https://storyboard.openstack.org/#!/worklist/580). 2 (-2) are [bugs](https://storyboard.openstack.org/#!/worklist/574). 5 (0) are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 11 (0) are [rfes](https://storyboard.openstack.org/#!/worklist/594). 4 (1) are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 16 (0). * Placement related nova [in progress bugs](https://goo.gl/vzGGDQ) on launchpad: 4 (-1). # osc-placement osc-placement is currently behind by 11 microversions. * Add support for multiple member_of. # Main Themes ## Nested Magic These are the features required by modern nested resource provider use cases. We've merged mappings in allocation candidates and `root_required`. `same_subtree` and resourceless request groups are what's left and they are in: * Support `same_subtree` queryparam ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. * ## Cleanup Cleanup is an overarching theme related to improving documentation, performance and the maintainability of the code. The changes we are making this cycle are fairly complex to use and are fairly complex to write, so it is good that we're going to have plenty of time to clean and clarify all these things. As mentioned above, one of the important cleanup tasks that is not yet in progress is updating the [gabbit](https://opendev.org/openstack/placement/src/branch/master/gate/gabbits/nested-perfload.yaml) that creates the nested topology that's used in nested performance testing. The topology there is simple, unrealistic, and doesn't sufficiently exercise the several features that may be used during a query that desires a nested response. Recently I've been seeing that the `placement-perfload` job is giving results that vary between `N` and `N*2` (usually .5 and 1 seconds) and the difference that I can discern is the type of CPUs being presented by the host (same number of CPUs (8) but different type). This supports something we've been theorizing for a while: when dealing with large result sets we are CPU bound processing the several large result sets returned by the database. Further profiling required… Another cleanup that needs to start is satisfying the community wide goal of [PDF doc generation](https://storyboard.openstack.org/#!/story/2006110). # Other Placement Miscellaneous changes can be found in [the usual place](https://review.opendev.org/#/q/project:openstack/placement+status:open). There are three [os-traits changes](https://review.opendev.org/#/q/project:openstack/os-traits+status:open) being discussed. And one [os-resource-classes change](https://review.opendev.org/#/q/project:openstack/os-resource-classes+status:open). # Other Service Users New discoveries are added to the end. Merged stuff is removed. Anything that has had no activity in 4 weeks has been removed. * Nova: nova-manage: heal port allocations * nova-spec: Allow compute nodes to use DISK_GB from shared storage RP * Cyborg: Placement report * helm: WIP: add placement chart * Nova: Use OpenStack SDK for placement * Nova: Spec: Provider config YAML file * libvirt: report pmem namespaces resources by provider tree * Nova: Remove PlacementAPIConnectFailure handling from AggregateAPI * Nova: support move ops with qos ports * Nova: get_ksa_adapter: nix by-service-type confgrp hack * OSA: Add nova placement to placement migration * Nova: Defaults missing group_policy to 'none' * Blazar: Create placement client for each request * tempest: Define the Integrated-gate-networking gate template * tempest: Define the Integrated-gate-placement gate template * Nova: Restore RT.old_resources if ComputeNode.save() fails * Remove assumption of http error if consumer not exists * TripleO: Add new parameter NovaSchedulerLimitTenantsToPlacementAggregate * puppet-nova: Expose limit_tenants_to_placement_aggregate parameter * nova: Support filtering of hosts by forbidden aggregates * blazar: Send global_request_id for tracing calls * nova: Implement update_provider_tree for hyperv * watcher: Improve Compute Data Model * Nova: pre filter disable computes * Nova: Update HostState.\*\_allocation_ratio earlier # End This space left intentionally blank. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From cdent+os at anticdent.org Fri Jul 5 13:53:24 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Fri, 5 Jul 2019 14:53:24 +0100 (BST) Subject: [placement] db query analysis In-Reply-To: <07c6e111-d6a9-d175-7014-aff832c6e9c7@gmail.com> References: <07c6e111-d6a9-d175-7014-aff832c6e9c7@gmail.com> Message-ID: On Mon, 24 Jun 2019, Jay Pipes wrote: >> Related to that, I've started working on a nested-perfload at >> https://review.opendev.org/665695 > > Please note that there used to be fewer queries performed in the allocation > candidate and get resource provider functions. We replaced the giant SQL > statements with multiple smaller SQL statements to assist in debuggability > and tracing. Yeah, I recall that and think it was the right thing to do, but now we've reached a point where we've got limited choices for how to eke out some more performance, which we're going to need to do to be reasonable in > 10,000 node environments. Being me I'm inclined towards things like infinite resource classes, vector math, and "PUT IT ALL IN RAM !!!!!" but luckily its not up to just me. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From ralonsoh at redhat.com Fri Jul 5 16:29:55 2019 From: ralonsoh at redhat.com (Rodolfo Alonso) Date: Fri, 05 Jul 2019 17:29:55 +0100 Subject: [neutron][requirements] Pyroute2 stable/queens upper version (0.4.21) has a memory leak Message-ID: Hello folks: As reported in [1], we have found a memory leak in Pyroute stable/queens upper version (0.4.21). This memory leak is reproducible both with Python 2.7 and Python 3.6 (I didn't test 3.5 or 3.7). The script used is [2]. Using "pmap" to read the process memory map (specifically the "total" value), we can see an increase of several MB per minute. This problem is not present in version 0.5.2 (stable/rocky upper-requirements) I know that in stable releases, the policy established [3] is only to modify those external libraries in case of security related issues. This is not exactly a security breach but can tear down a server along the time. I submitted a patch to bump the version in stable/queens [4] and another one to test this change in the Neutron CI [5]. Is it possible to merge [4]? Regards. [1] https://bugs.launchpad.net/neutron/+bug/1835044 [2] http://paste.openstack.org/show/753759/ [3] https://docs.openstack.org/project-team-guide/stable-branches.html [4] https://review.opendev.org/#/c/668676/ [5] https://review.opendev.org/#/c/668677/ From smooney at redhat.com Fri Jul 5 16:36:35 2019 From: smooney at redhat.com (Sean Mooney) Date: Fri, 05 Jul 2019 17:36:35 +0100 Subject: [nova] type-PCI vs type-PF vs type-VF In-Reply-To: References: Message-ID: On Fri, 2019-07-05 at 07:58 -0400, Mauricio Tavares wrote: > Quick questions: > this is not really documented that well but https://bugs.launchpad.net/nova/+bug/1832169 has some context were i explaing why the 3 types exist. > 1. Is there some official documentation comparing them? Only thing I found > so far was > https://mohamede.com/2019/02/07/pci-passthrough-type-pf-type-vf-and-type-pci/ > > 2. Am I correct to assume (I seem to do a lot of that; be afraid) that when > nova starts populating the database entry for a given physical node (say a > computer node or a ironic baremetal node) it looks through the pci list and > decides what type of device (type-PCI, type-PF, or type-VF) it is? What is > the criteria? we determing the type in the _get_device_type function in the libvirt virt driver here https://github.com/openstack/nova/blob/e75598be31d849bbe09c905081be224f68210d32/nova/virt/libvirt/driver.py#L6093-L6130 the critia we use is interogating the pcie config space capablitys reported via libvirt. if the pci device has the "virt_functions" capablity meaning it can create virtual funcitons it si reported as type-PF as it can be used as an sriov PF if it has cap phys_function and an adress set it means it has a parent PF and therefor it is a VF all other deivce are starndard PCI device that do not support sriov and are mapped to type-PCI > > 3. > https://mohamede.com/2019/02/07/pci-passthrough-type-pf-type-vf-and-type-pci/ > implies that nova sees the 3 different types, well, differently where it > will try hard to virtualize type-PF and type-VF and really really wants me > to setup sr-iov on them ( > https://docs.openstack.org/nova/latest/admin/pci-passthrough.html seem to > also want sr-iov) really . Is that correct? you dont have to use the device for VF passthough but if it is capable of creting VF it will be lists as a PF regradelss of if VF have been created via sysfs or kernel args. if you modify the frimeware mode on your card such as puting it in datacenter bridging mode whic disables sriov it will change the pci config space entries will change and it will change form type-PF to type-PCI the example alias above was using an intel nic that supported the sriov so it used type-PF to do a pci passthough of the entire nic but if it was a gpu or other device that did not support sriov it would have used type-PCI. From iain.macdonnell at oracle.com Fri Jul 5 18:15:14 2019 From: iain.macdonnell at oracle.com (iain.macdonnell at oracle.com) Date: Fri, 5 Jul 2019 11:15:14 -0700 Subject: [nova] consoleauth token ttl In-Reply-To: <20190704185817.Horde.kZVzaR7qWwNvYreX3yw-Bae@webmail2.ic.unicamp.br> References: <20190704185817.Horde.kZVzaR7qWwNvYreX3yw-Bae@webmail2.ic.unicamp.br> Message-ID: <13a9fe53-078a-c6f7-a8db-f93bade9c27b@oracle.com> On 7/4/19 2:58 PM, William Lima Reiznautt wrote: > Some known about this configuration token_ttl on [consoleauth] section: > Is it on second or minutes ? > > At documentation is not information. Per https://docs.openstack.org/nova/stein/configuration/config.html#consoleauth "The lifetime of a console auth token (in seconds)." ~iain From mriedemos at gmail.com Fri Jul 5 20:14:34 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Fri, 5 Jul 2019 15:14:34 -0500 Subject: [ops] [nova] [placement] Mismatch between allocations and instances In-Reply-To: References: Message-ID: On 7/5/2019 1:45 AM, Massimo Sgaravatto wrote: > I tried to check the allocations on each compute node of a Ocata cloud, > using the command: > > curl -s ${PLACEMENT_ENDPOINT}/resource_providers/${UUID}/allocations -H > "x-auth-token: $TOKEN"  | python -m json.tool > Just FYI you can use osc-placement (openstack client plugin) for command line: https://docs.openstack.org/osc-placement/latest/index.html > I found that, on a few compute nodes, there are some instances for which > there is not a corresponding allocation. The heal_allocations command [1] might be able to find and fix these up for you. The bad news for you is that heal_allocations wasn't added until Rocky and you're on Ocata. The good news is you should be able to take the current version of the code from master (or stein) and run that in a container or virtual environment against your Ocata cloud (this would be particularly useful if you want to use the --dry-run or --instance options added in Train). You could also potentially backport those changes to your internal branch, or we could start a discussion upstream about backporting that tooling to stable branches - though going to Ocata might be a bit much at this point given Ocata and Pike are in extended maintenance mode [2]. As for *why* the instances on those nodes are missing allocations, it's hard to say without debugging things. The allocation and resource tracking code has changed quite a bit since Ocata (in Pike the scheduler started creating the allocations but the resource tracker in the compute service could still overwrite those allocations if you had older nodes during a rolling upgrade). My guess would be a migration failed or there was just a bug in Ocata where we didn't cleanup or allocate properly. Again, heal_allocations should add the missing allocation for you if you can setup the environment to run that command. > > On another Rocky cloud, we had the opposite problem: there were > allocations also for some instances that didn't exist anymore. > And this caused problems since we were not able to use all the resources > of the relevant compute nodes: we had to manually remove the fwrong" > allocations to fix the problem ... Yup, this could happen for different reasons, usually all due to known bugs for which you don't have the fix yet, e.g. [3][4], or something is failing during a migration and we aren't cleaning up properly (an unreported/not-yet-fixed bug). > > > I wonder why/how this problem can happen ... I mentioned some possibilities above - but I'm sure there are other bugs that have been fixed which I've omitted here, or things that aren't fixed yet, especially in failure scenarios (rollback/cleanup handling is hard). Note that your Ocata and Rocky cases could be different because since Queens (once all compute nodes are >=Queens) during resize, cold and live migration the migration record in nova holds the source node allocations during the migration so the actual *consumer* of the allocations for a provider in placement might not be an instance (server) record but actually a migration, so if you were looking for an allocation consumer by ID in nova using something like "openstack server show $consumer_id" it might return NotFound because the consumer is actually not an instance but a migration record and the allocation was leaked. > > And how can we fix the issue ? Should we manually add the missing > allocations / manually remove the wrong ones ? Coincidentally a thread related to this [5] re-surfaced a couple of weeks ago. I am not sure what Sylvain's progress is on that audit tool, but the linked bug in that email has some other operator scripts you could try for the case that there are leaked/orphaned allocations on compute nodes that no longer have instances. > > Thanks, Massimo > > [1] https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement [2] https://docs.openstack.org/project-team-guide/stable-branches.html [3] https://bugs.launchpad.net/nova/+bug/1825537 [4] https://bugs.launchpad.net/nova/+bug/1821594 [5] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007241.html -- Thanks, Matt From corey.bryant at canonical.com Fri Jul 5 22:00:50 2019 From: corey.bryant at canonical.com (Corey Bryant) Date: Fri, 5 Jul 2019 18:00:50 -0400 Subject: [goal][python3] Train unit tests weekly update (goal-10) Message-ID: This is the goal-10 weekly update for the "Update Python 3 test runtimes for Train" goal [1]. There are 10 weeks remaining for completion of Train community goals [2]. == What's the Goal? == To ensure (in the Train cycle) that all official OpenStack repositories with Python 3 unit tests are exclusively using the 'openstack-python3-train-jobs' Zuul template or one of its variants (e.g. 'openstack-python3-train-jobs-neutron') to run unit tests, and that tests are passing. This will ensure that all official projects are running py36 and py37 unit tests in Train. For complete details please see [1]. == Ongoing Work == Patches have been submitted for all applicable projects except for: OpenStack Charms, Puppet OpenStack, Quality Assurance, Release Management, tripleo. Open patches needing reviews: https://review.openstack.org/#/q/topic:python3-train+is:open Failing patches: https://review.openstack.org/#/q/topic:python3-train+status:open+(+label:Verified-1+OR+label:Verified-2+) Patch automation scripts needing review: https://review.opendev.org/#/c/666934 == Completed Work == Merged patches: https://review.openstack.org/#/q/topic:python3-train+is:merged == How can you help? == Please take a look at the failing patches and help fix any failing unit tests for your projects. Python 3.7 unit tests will be self-testing in Zuul. If you're interested in helping submit patches, please let me know. == Reference Material == [1] Goal description: https://governance.openstack.org/tc/goals /train/python3-updates.html [2] Train release schedule: https://releases.openstack.org/train/schedule.html (see R-5 for "Train Community Goals Completed") Storyboard: https://storyboard.openstack.org/#!/story/2005924 Porting to Python 3.7: https://docs.python.org/3/whatsnew/3.7.html#porting-to-python-3-7 Python Update Process: https://opendev.org/openstack/governance/src/branch/master/resolutions/20181024-python-update-process.rst Train runtimes: https://opendev.org/openstack/governance/src/branch/master/reference/runtimes/train.rst Thanks, Corey -------------- next part -------------- An HTML attachment was scrubbed... URL: From mriedemos at gmail.com Fri Jul 5 22:46:06 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Fri, 5 Jul 2019 17:46:06 -0500 Subject: [nova][ironic] Lock-related performance issue with update_resources periodic job In-Reply-To: <37D021C4-ED1B-4942-9C90-0A26FDE3DD76@fried.cc> References: <37D021C4-ED1B-4942-9C90-0A26FDE3DD76@fried.cc> Message-ID: On 7/4/2019 5:16 AM, Eric Fried wrote: > > https://review.opendev.org/#/c/637225/ > > Ah heck, I had totally forgotten about that patch. If it's working for > you, let me get it polished up and merged. > > We could probably justify backporting it too. Matt? > > efried Sure - get a bug opened for it, extra points if CERN can provide some before/after numbers with the patch applied to help justify it. From skimming the commit message, if the only side effect would be for sharing providers, which we don't really support yet, then backports seem OK. -- Thanks, Matt From openstack at fried.cc Fri Jul 5 22:57:51 2019 From: openstack at fried.cc (Eric Fried) Date: Fri, 5 Jul 2019 17:57:51 -0500 Subject: [nova][ironic] Lock-related performance issue with update_resources periodic job In-Reply-To: References: <37D021C4-ED1B-4942-9C90-0A26FDE3DD76@fried.cc> Message-ID: <07C5F51A-DCAB-4432-B556-49E1E15801AC@fried.cc> Bug already associated with patch. I'll work on this next week. efried > On Jul 5, 2019, at 17:46, Matt Riedemann wrote: > >> On 7/4/2019 5:16 AM, Eric Fried wrote: >> > https://review.opendev.org/#/c/637225/ >> Ah heck, I had totally forgotten about that patch. If it's working for you, let me get it polished up and merged. >> We could probably justify backporting it too. Matt? >> efried > > Sure - get a bug opened for it, extra points if CERN can provide some before/after numbers with the patch applied to help justify it. > > From skimming the commit message, if the only side effect would be for sharing providers, which we don't really support yet, then backports seem OK. > > -- > > Thanks, > > Matt > From tony at bakeyournoodle.com Fri Jul 5 23:04:20 2019 From: tony at bakeyournoodle.com (Tony Breeds) Date: Sat, 6 Jul 2019 09:04:20 +1000 Subject: [release] Release countdown for week R-14, July 8-12 Message-ID: <20190705230420.GA19305@thor.bakeyournoodle.com> Hi folks, Development Focus ----------------- The Train-2 milestone will happen in three weeks, on July 25. Train-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/train/schedule.html for details. General Information ------------------- Please remember that libraries need to be released at least once per milestone period. At milestone 2, the release team will propose releases for any library that has not been otherwise released since milestone 1. Other non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. At milestone-2 we also freeze the contents of the final release. If you have a new deliverable that should be included in the final release, you should make sure it has a deliverable file in https://opendev.org/openstack/releases/src/branch/master/deliverables/train . You should request a beta release (or intermediary release) for those new deliverables by milestone-2. We understand some may not be quite ready for a full release yet, but if you have something minimally viable to get released it would be good to do a 0.x release to exercise the release tooling for your deliverables. See the MembershipFreeze description for more details: https://releases.openstack.org/train/schedule.html#t-mf Upcoming Deadlines & Dates -------------------------- Train-2 Milestone: July 25 (R-12 week) Yours Tony. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From tony at bakeyournoodle.com Fri Jul 5 23:12:29 2019 From: tony at bakeyournoodle.com (Tony Breeds) Date: Sat, 6 Jul 2019 09:12:29 +1000 Subject: [all][release] PTL on Vacation July 6-14 Message-ID: <20190705231229.GB19305@thor.bakeyournoodle.com> Hi All, Just a very quick note to say I'll be AFK for the next week. The only visible change could be that stable releases are delayed a little. As always if it's urgent nothing stops the release team from approving things in my absence Yours Tony. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From colleen at gazlene.net Fri Jul 5 23:14:44 2019 From: colleen at gazlene.net (Colleen Murphy) Date: Fri, 05 Jul 2019 16:14:44 -0700 Subject: [keystone] Keystone Team Update - Week of 1 July 2019 Message-ID: <10503ca2-647b-45f6-ad43-a1bab0471e49@www.fastmail.com> # Keystone Team Update - Week of 1 July 2019 ## News ### Midcycle Planning If you have not already done so, please participate in the scheduling poll[1] and contribute topic ideas[2] for the virtual midcycle. We will select a date next week and start preparing the agenda. [1] https://doodle.com/poll/wr7ct4uhpw82sysg [2] https://etherpad.openstack.org/p/keystone-train-midcycle-topics ### Oslo.limit Walkthrough We used this week's office hours session to walk through the new proposals for the oslo.limit interface, which Lance nicely summarized[3]. [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007487.html ## Open Specs Train specs: https://bit.ly/2uZ2tRl Ongoing specs: https://bit.ly/2OyDLTh We still have three Train specs that needs to be updated or reviewed prior to the Milestone 2 deadline. ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 14 changes this week. ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 43 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ## Bugs This week we opened 5 new bugs and closed none. Bugs opened (5) Bug #1835299 (keystone:Low) opened by Colleen Murphy https://bugs.launchpad.net/keystone/+bug/1835299 Bug #1835303 (keystoneauth:Undecided) opened by Ben Nemec https://bugs.launchpad.net/keystoneauth/+bug/1835303 Bug #1835103 (oslo.limit:Low) opened by Lance Bragstad https://bugs.launchpad.net/oslo.limit/+bug/1835103 Bug #1835104 (oslo.limit:Low) opened by Lance Bragstad https://bugs.launchpad.net/oslo.limit/+bug/1835104 Bug #1835106 (oslo.limit:Low) opened by Lance Bragstad https://bugs.launchpad.net/oslo.limit/+bug/1835106 ## Milestone Outlook https://releases.openstack.org/train/schedule.html Spec freeze is in 3 weeks, closely followed by feature proposal freeze. If you are working on a feature, even if the spec is not approved yet, don't hesitate to propose some code ahead of the deadline. ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter From fungi at yuggoth.org Sat Jul 6 00:28:11 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Sat, 6 Jul 2019 00:28:11 +0000 Subject: [neutron][requirements] Pyroute2 stable/queens upper version (0.4.21) has a memory leak In-Reply-To: References: Message-ID: <20190706002811.oigm7z3iov6g3s2s@yuggoth.org> On 2019-07-05 17:29:55 +0100 (+0100), Rodolfo Alonso wrote: [...] > I know that in stable releases, the policy established [3] is only > to modify those external libraries in case of security related > issues. This is not exactly a security breach but can tear down a > server along the time. [...] > [3] https://docs.openstack.org/project-team-guide/stable-branches.html [...] You're referring to policy about backporting fixes for bugs in OpenStack software, and so necessitates patch-level version increases for the affected OpenStack components in upper-constraints.txt to make sure we test other software against that newer version. The policy so far regarding stable branch upper-constraints.txt entries for external dependencies of OpenStack has been to not change them even if they include known security vulnerabilities or other critical bugs, unless those bugs impact our ability to reliably test proposed changes to stable branches of OpenStack software for possible regressions. It's a common misconception, but that upper-constraints.txt file is purely a reflection of the (basically frozen in the case of stable branches) set of dependency versions from PyPI against which changes to our software are tested. It is not a good idea to deploy production environments from the PyPI packages corresponding to the versions listed there, for a variety of reasons (most important of which is that they aren't a security-supported distribution, nor can they ever even remotely become one). -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From skaplons at redhat.com Sat Jul 6 08:58:20 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Sat, 6 Jul 2019 10:58:20 +0200 Subject: [TripleO] Using TripleO standalone with ML2/OVS Message-ID: <8F8016EB-7898-4F66-BD30-998ABDB094FB@redhat.com> Hi, I was trying to use TripleO standalone for development work in the way how Emilien described it in [1] and indeed it works quite well. Thx Emilien. But now, I’m trying to find out the way how to deploy it with Neutron using ML2/OVS instead of default in TripleO ML2/OVN. And I still don’t know how to do it :/ I know it’s my fault but maybe someone can help me with this and tell me what exactly options I should change there to deploy it with other Neutron backend? Thx in advance for any help. [1] https://my1.fr/blog/developer-workflow-with-tripleo/ — Slawek Kaplonski Senior software engineer Red Hat From sam47priya at gmail.com Sat Jul 6 14:54:14 2019 From: sam47priya at gmail.com (Sam P) Date: Sat, 6 Jul 2019 23:54:14 +0900 Subject: [masakari] Run masakari-hostmonitor into Docker container In-Reply-To: References: <7666a4eae3522bcb14741108bf8a5994@incloudus.com> Message-ID: Hi Gaëtan, I have never tried it, but sounds interesting. Current masakari is not designed to run in the containers. However, except masakari monitors, most of masakari components can run on the containers. For masakari-hostmonitor, you don't need to run systemd daemon inside the container. Hoever, the code need to be changed slightly to use remote systemd on the host OS. ex: systemctl --host user_name at host_name command Or we also can use the method Mark shared in previous email. > I would not recommend running the systemd daemon in a container, but > you could potentially use the client to access a daemon running on the > host. E.g., for debian: > https://stackoverflow.com/questions/54079586/make-systemctl-work-from-inside-a-container-in-a-debian-stretch-image. > No doubt there will be various gotchas with this. > Are you planning to run pacemaker and corosync on the host? Since masakari-hostmonitor needs to detect any failures of the host, pacemaker and corosync need to run on the host OS. You may do the otherwise, but this is the simplest solution. --- Regards, Sampath On Fri, Jun 28, 2019 at 5:15 PM Mark Goddard wrote: > > On Thu, 27 Jun 2019 at 21:52, wrote: > > > > Hi, > > > > I'm integrating Masakari into Kolla and Kolla Ansible projects but I'm > > facing > > an issue related to masakari-hostmonitor. > > > > Based on masakari-monitors code[1], "systemctl status" command is used > > to check > > if pacemaker, pacemaker-remote and corosync are running. > > > > Having systemd running into Docker container is not the best solution. > > Does any > > of you has been able to run masakari-monitor into Docker container ? > > > > I would not recommend running the systemd daemon in a container, but > you could potentially use the client to access a daemon running on the > host. E.g., for debian: > https://stackoverflow.com/questions/54079586/make-systemctl-work-from-inside-a-container-in-a-debian-stretch-image. > No doubt there will be various gotchas with this. > > Are you planning to run pacemaker and corosync on the host? > > Mark > > > Thanks for your help. > > > > Gaëtan > > > > - [1] > > https://github.com/openstack/masakari-monitors/blob/26d558333d9731ca06da09b26fe6592c49c0ac8a/masakarimonitors/hostmonitor/host_handler/handle_host.py#L48 > > > From mnaser at vexxhost.com Sat Jul 6 17:38:32 2019 From: mnaser at vexxhost.com (Mohammed Naser) Date: Sat, 6 Jul 2019 13:38:32 -0400 Subject: [docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos Message-ID: Hi everyone, One of the issue that we recently ran into was the fact that there was some inconsistency about merging retirement of repositories inside governance without the code being fully removed. In order to avoid this, I've made a change to our governance repository which will enforce that no code exists in those retired repositories, however, this has surfaced that some repositories were retired with some stale files, some are smaller littler files, some are entire projects still. I have compiled a list for every team, with the repos that are not properly retired that have extra files (using this change which should eventually +1 once we fix it all: https://review.opendev.org/669549) [Documentation] openstack/api-site has extra files, please remove: .gitignore, .zuul.yaml, LICENSE, api-quick-start, api-ref, bindep.txt, common, doc-tools-check-languages.conf, firstapp, test-requirements.txt, tools, tox.ini, www [Documentation] openstack/faafo has extra files, please remove: .gitignore, CONTRIBUTING.rst, LICENSE, Vagrantfile, bin, contrib, doc, etc, faafo, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini [fuel] openstack/fuel-agent has extra files, please remove: .gitignore, LICENSE, MAINTAINERS, cloud-init-templates, contrib, debian, etc, fuel_agent, requirements.txt, run_tests.sh, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [fuel] openstack/fuel-astute has extra files, please remove: .gitignore, .rspec, .ruby-version, Gemfile, LICENSE, MAINTAINERS, Rakefile, astute.gemspec, astute.service, astute.sysconfig, bin, bindep.txt, debian, examples, lib, mcagents, run_tests.sh, spec, specs, tests [fuel] openstack/fuel-library has extra files, please remove: .gitignore, CHANGELOG, Gemfile, LICENSE, MAINTAINERS, Rakefile, debian, deployment, files, graphs, logs, specs, tests, utils [fuel] openstack/fuel-main has extra files, please remove: .gitignore, 00-debmirror.patch, LICENSE, MAINTAINERS, Makefile, config.mk, fuel-release, iso, mirror, packages, prepare-build-env.sh, report-changelog.sh, repos.mk, requirements-fuel-rpm.txt, requirements-rpm.txt, rules.mk, sandbox.mk, specs [fuel] openstack/fuel-menu has extra files, please remove: .gitignore, MAINTAINERS, MANIFEST.in, fuelmenu, run_tests.sh, setup.py, specs, test-requirements.txt, tox.ini [fuel] openstack/fuel-mirror has extra files, please remove: .gitignore, .mailmap, MAINTAINERS, perestroika, tox.ini [fuel] openstack/fuel-nailgun-agent has extra files, please remove: .gitignore, Gemfile, LICENSE, MAINTAINERS, Rakefile, agent, debian, nailgun-agent.cron, nailgun-agent.gemspec, run_tests.sh, specs [fuel] openstack/fuel-ostf has extra files, please remove: .gitignore, LICENSE, MAINTAINERS, MANIFEST.in, etc, fuel_health, fuel_plugin, ostf.service, pylintrc, requirements.txt, run_tests.sh, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [fuel] openstack/fuel-qa has extra files, please remove: .coveragerc, .gitignore, .pylintrc, .pylintrc_gerrit, MAINTAINERS, core, doc, fuel_tests, fuelweb_test, gates_tests, packages_tests, pytest.ini, run_system_test.py, run_tests.sh, system_test, tox.ini, utils [fuel] openstack/fuel-ui has extra files, please remove: .eslintignore, .eslintrc.yaml, .gitignore, LICENSE, MAINTAINERS, fixtures, gulp, gulpfile.js, karma.config.js, npm-shrinkwrap.json, package.json, run_real_plugin_tests.sh, run_real_plugin_tests_on_real_nailgun.sh, run_ui_func_tests.sh, specs, static, webpack.config.js [fuel] openstack/fuel-virtualbox has extra files, please remove: .gitignore, MAINTAINERS, actions, clean.sh, config.sh, contrib, drivers, dumpkeys.cache, functions, iso, launch.sh, launch_16GB.sh, launch_8GB.sh [fuel] openstack/fuel-web has extra files, please remove: .gitignore, LICENSE, MAINTAINERS, bin, build_docs.sh, debian, docs, nailgun, run_tests.sh, specs, systemd, tools, tox.ini [fuel] openstack/shotgun has extra files, please remove: .coveragerc, .gitignore, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, MAINTAINERS, MANIFEST.in, bin, etc, requirements.txt, setup.cfg, setup.py, shotgun, specs, test-requirements.txt, tox.ini [fuel] openstack/fuel-dev-tools has extra files, please remove: .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, MAINTAINERS, babel.cfg, contrib, doc, fuel_dev_tools, openstack-common.conf, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini, vagrant [fuel] openstack/fuel-devops has extra files, please remove: .coveragerc, .gitignore, .pylintrc, .pylintrc_gerrit, LICENSE, MAINTAINERS, bin, devops, doc, run_tests.sh, samples, setup.cfg, setup.py, test-requirements.txt, tox.ini [fuel] openstack/fuel-docs has extra files, please remove: .gitignore, Makefile, _images, _templates, common_conf.py, conf.py, devdocs, examples, glossary, index.rst, make.bat, plugindocs, requirements.txt, setup.cfg, setup.py, tox.ini, userdocs [fuel] openstack/fuel-nailgun-extension-cluster-upgrade has extra files, please remove: .coveragerc, .gitignore, AUTHORS, LICENSE, MANIFEST.in, bindep.txt, cluster_upgrade, conftest.py, nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [fuel] openstack/fuel-nailgun-extension-iac has extra files, please remove: .gitignore, LICENSE, MANIFEST.in, doc, fuel_external_git, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [fuel] openstack/fuel-nailgun-extension-converted-serializers has extra files, please remove: .coveragerc, .gitignore, LICENSE, MANIFEST.in, bindep.txt, conftest.py, converted_serializers, nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [fuel] openstack/fuel-octane has extra files, please remove: .coveragerc, .gitignore, .mailmap, Gemfile, Gemfile.lock, HACKING.rst, LICENSE, MAINTAINERS, MANIFEST.in, Rakefile, bindep.txt, deploy, deployment, docs, misc, octane, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tox.ini [fuel] openstack/fuel-upgrade has extra files, please remove: .gitignore [fuel] openstack/tuning-box has extra files, please remove: .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, MAINTAINERS, MANIFEST.in, TODO, alembic.ini, babel.cfg, bindep.txt, doc, examples, openstack-common.conf, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini, tuning_box [fuel] openstack/fuel-plugins has extra files, please remove: .gitignore, CHANGELOG.md, CONTRIBUTING.rst, HACKING.rst, LICENSE, MAINTAINERS, examples, fuel_plugin_builder, requirements.txt, run_tests.sh, setup.cfg, setup.py, test-requirements.txt, tox.ini [fuel] openstack/fuel-plugin-murano has extra files, please remove: .gitignore, LICENSE, components.yaml, deployment_scripts, deployment_tasks.yaml, docs, environment_config.yaml, functions.sh, metadata.yaml, node_roles.yaml, pre_build_hook, releasenotes, repositories, test-requirements.txt, tox.ini, volumes.yaml [fuel] openstack/fuel-plugin-murano-tests has extra files, please remove: .gitignore, murano_plugin_tests, openrc.default, requirements.txt, tox.ini, utils [fuel] openstack/fuel-specs has extra files, please remove: .gitignore, .testr.conf, LICENSE, doc, images, policy, requirements.txt, setup.cfg, setup.py, specs, tests, tools, tox.ini [fuel] openstack/fuel-stats has extra files, please remove: .gitignore, LICENSE, MAINTAINERS, MANIFEST.in, analytics, collector, migration, requirements.txt, setup.py, test-requirements.txt, tools, tox.ini [fuel] openstack/python-fuelclient has extra files, please remove: .gitignore, .testr.conf, MAINTAINERS, MANIFEST.in, fuelclient, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [Infrastructure] opendev/puppet-releasestatus has extra files, please remove: .gitignore [ironic] openstack/python-dracclient has extra files, please remove: .gitignore, CONTRIBUTING.rst, HACKING.rst, LICENSE, doc, dracclient, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini [neutron] openstack/networking-calico has extra files, please remove: .coveragerc, .gitignore, .mailmap, .testr.conf, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, RELEASING.md, babel.cfg, debian, devstack, doc, networking_calico, playbooks, requirements.txt, rpm, setup.cfg, setup.py, test-requirements.txt, tox.ini [neutron] openstack/networking-l2gw has extra files, please remove: .coveragerc, .gitignore, .testr.conf, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, bindep.txt, contrib, debian, devstack, doc, etc, lower-constraints.txt, networking_l2gw, openstack-common.conf, requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, tools, tox.ini [neutron] openstack/networking-l2gw-tempest-plugin has extra files, please remove: .gitignore, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, babel.cfg, contrib, doc, networking_l2gw_tempest_plugin, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini [neutron] openstack/networking-onos has extra files, please remove: .coveragerc, .gitignore, .mailmap, .pylintrc, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, PKG-INFO, TESTING.rst, babel.cfg, devstack, doc, etc, lower-constraints.txt, networking_onos, package, rally-jobs, releasenotes, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tools, tox.ini [neutron] openstack/neutron-vpnaas has extra files, please remove: .coveragerc, .gitignore, .mailmap, .pylintrc, .stestr.conf, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, TESTING.rst, babel.cfg, devstack, doc, etc, lower-constraints.txt, neutron_vpnaas, playbooks, rally-jobs, releasenotes, requirements.txt, setup.cfg, setup.py, test-requirements.txt, tools, tox.ini [OpenStack Charms] openstack/charm-ceph has extra files, please remove: .gitignore [OpenStackAnsible] openstack/openstack-ansible-os_monasca has extra files, please remove: tests, tox.ini [solum] openstack/solum-infra-guestagent has extra files, please remove: .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, config-generator, doc, etc, requirements.txt, setup.cfg, setup.py, solum_guestagent, test-requirements.txt, tox.ini I'd like to kindly ask the affected teams to help out with this, or any member of our community is more than welcome to push a change to those repos and work with the appropriate teams to help land it. Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser at vexxhost.com W. http://vexxhost.com From dtantsur at redhat.com Sat Jul 6 19:29:13 2019 From: dtantsur at redhat.com (Dmitry Tantsur) Date: Sat, 6 Jul 2019 21:29:13 +0200 Subject: [docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos In-Reply-To: References: Message-ID: <9b4d6acc-350f-0dbd-fb27-1c2f5023c6b4@redhat.com> On 7/6/19 7:38 PM, Mohammed Naser wrote: > Hi everyone, > > One of the issue that we recently ran into was the fact that there was > some inconsistency about merging retirement of repositories inside > governance without the code being fully removed. > > In order to avoid this, I've made a change to our governance > repository which will enforce that no code exists in those retired > repositories, however, this has surfaced that some repositories were > retired with some stale files, some are smaller littler files, some > are entire projects still. > > I have compiled a list for every team, with the repos that are not > properly retired that have extra files (using this change which should > eventually +1 once we fix it all: https://review.opendev.org/669549) > > > [ironic] openstack/python-dracclient has extra files, please remove: > .gitignore, CONTRIBUTING.rst, HACKING.rst, LICENSE, doc, dracclient, > requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini This project is not retired, why is it listed here? I also used to think it was not under any governance.. > > > I'd like to kindly ask the affected teams to help out with this, or > any member of our community is more than welcome to push a change to > those repos and work with the appropriate teams to help land it. > > Mohammed > From mnaser at vexxhost.com Sat Jul 6 19:39:04 2019 From: mnaser at vexxhost.com (Mohammed Naser) Date: Sat, 6 Jul 2019 15:39:04 -0400 Subject: [docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos In-Reply-To: <9b4d6acc-350f-0dbd-fb27-1c2f5023c6b4@redhat.com> References: <9b4d6acc-350f-0dbd-fb27-1c2f5023c6b4@redhat.com> Message-ID: On Sat, Jul 6, 2019 at 3:33 PM Dmitry Tantsur wrote: > > On 7/6/19 7:38 PM, Mohammed Naser wrote: > > Hi everyone, > > > > One of the issue that we recently ran into was the fact that there was > > some inconsistency about merging retirement of repositories inside > > governance without the code being fully removed. > > > > In order to avoid this, I've made a change to our governance > > repository which will enforce that no code exists in those retired > > repositories, however, this has surfaced that some repositories were > > retired with some stale files, some are smaller littler files, some > > are entire projects still. > > > > I have compiled a list for every team, with the repos that are not > > properly retired that have extra files (using this change which should > > eventually +1 once we fix it all: https://review.opendev.org/669549) > > > > > > > [ironic] openstack/python-dracclient has extra files, please remove: > > .gitignore, CONTRIBUTING.rst, HACKING.rst, LICENSE, doc, dracclient, > > requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini > > This project is not retired, why is it listed here? > > I also used to think it was not under any governance.. https://opendev.org/openstack/governance/commit/003d9d8247f2deea7a344c47d87b54f7457fe19d So, I'm not sure if in this case it has to be moved to x/python-dracclient or... > > > > > > > I'd like to kindly ask the affected teams to help out with this, or > > any member of our community is more than welcome to push a change to > > those repos and work with the appropriate teams to help land it. > > > > Mohammed > > > > -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser at vexxhost.com W. http://vexxhost.com From dtantsur at redhat.com Sat Jul 6 19:42:59 2019 From: dtantsur at redhat.com (Dmitry Tantsur) Date: Sat, 6 Jul 2019 21:42:59 +0200 Subject: [docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos In-Reply-To: References: <9b4d6acc-350f-0dbd-fb27-1c2f5023c6b4@redhat.com> Message-ID: <43e69a72-9c37-6ce9-6a87-178a0ea2edb5@redhat.com> On 7/6/19 9:39 PM, Mohammed Naser wrote: > On Sat, Jul 6, 2019 at 3:33 PM Dmitry Tantsur wrote: >> >> On 7/6/19 7:38 PM, Mohammed Naser wrote: >>> Hi everyone, >>> >>> One of the issue that we recently ran into was the fact that there was >>> some inconsistency about merging retirement of repositories inside >>> governance without the code being fully removed. >>> >>> In order to avoid this, I've made a change to our governance >>> repository which will enforce that no code exists in those retired >>> repositories, however, this has surfaced that some repositories were >>> retired with some stale files, some are smaller littler files, some >>> are entire projects still. >>> >>> I have compiled a list for every team, with the repos that are not >>> properly retired that have extra files (using this change which should >>> eventually +1 once we fix it all: https://review.opendev.org/669549) >> >>> >>> >>> [ironic] openstack/python-dracclient has extra files, please remove: >>> .gitignore, CONTRIBUTING.rst, HACKING.rst, LICENSE, doc, dracclient, >>> requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini >> >> This project is not retired, why is it listed here? >> >> I also used to think it was not under any governance.. > > https://opendev.org/openstack/governance/commit/003d9d8247f2deea7a344c47d87b54f7457fe19d > > So, I'm not sure if in this case it has to be moved to x/python-dracclient or... Yep, I guess it should have been moved. This library is maintained by Dell, not by the ironic team. > >>> >> >>> >>> I'd like to kindly ask the affected teams to help out with this, or >>> any member of our community is more than welcome to push a change to >>> those repos and work with the appropriate teams to help land it. >>> >>> Mohammed >>> >> >> > > From skaplons at redhat.com Sat Jul 6 19:59:38 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Sat, 6 Jul 2019 21:59:38 +0200 Subject: [docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos In-Reply-To: References: Message-ID: <04C9A55A-1ABA-4015-AECC-63E44E760E29@redhat.com> Hi, > On 6 Jul 2019, at 19:38, Mohammed Naser wrote: > > Hi everyone, > > One of the issue that we recently ran into was the fact that there was > some inconsistency about merging retirement of repositories inside > governance without the code being fully removed. > > In order to avoid this, I've made a change to our governance > repository which will enforce that no code exists in those retired > repositories, however, this has surfaced that some repositories were > retired with some stale files, some are smaller littler files, some > are entire projects still. > > I have compiled a list for every team, with the repos that are not > properly retired that have extra files (using this change which should > eventually +1 once we fix it all: https://review.opendev.org/669549) > > [Documentation] openstack/api-site has extra files, please remove: > .gitignore, .zuul.yaml, LICENSE, api-quick-start, api-ref, bindep.txt, > common, doc-tools-check-languages.conf, firstapp, > test-requirements.txt, tools, tox.ini, www > [Documentation] openstack/faafo has extra files, please remove: > .gitignore, CONTRIBUTING.rst, LICENSE, Vagrantfile, bin, contrib, doc, > etc, faafo, requirements.txt, setup.cfg, setup.py, > test-requirements.txt, tox.ini > > [fuel] openstack/fuel-agent has extra files, please remove: > .gitignore, LICENSE, MAINTAINERS, cloud-init-templates, contrib, > debian, etc, fuel_agent, requirements.txt, run_tests.sh, setup.cfg, > setup.py, specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-astute has extra files, please remove: > .gitignore, .rspec, .ruby-version, Gemfile, LICENSE, MAINTAINERS, > Rakefile, astute.gemspec, astute.service, astute.sysconfig, bin, > bindep.txt, debian, examples, lib, mcagents, run_tests.sh, spec, > specs, tests > [fuel] openstack/fuel-library has extra files, please remove: > .gitignore, CHANGELOG, Gemfile, LICENSE, MAINTAINERS, Rakefile, > debian, deployment, files, graphs, logs, specs, tests, utils > [fuel] openstack/fuel-main has extra files, please remove: .gitignore, > 00-debmirror.patch, LICENSE, MAINTAINERS, Makefile, config.mk, > fuel-release, iso, mirror, packages, prepare-build-env.sh, > report-changelog.sh, repos.mk, requirements-fuel-rpm.txt, > requirements-rpm.txt, rules.mk, sandbox.mk, specs > [fuel] openstack/fuel-menu has extra files, please remove: .gitignore, > MAINTAINERS, MANIFEST.in, fuelmenu, run_tests.sh, setup.py, specs, > test-requirements.txt, tox.ini > [fuel] openstack/fuel-mirror has extra files, please remove: > .gitignore, .mailmap, MAINTAINERS, perestroika, tox.ini > [fuel] openstack/fuel-nailgun-agent has extra files, please remove: > .gitignore, Gemfile, LICENSE, MAINTAINERS, Rakefile, agent, debian, > nailgun-agent.cron, nailgun-agent.gemspec, run_tests.sh, specs > [fuel] openstack/fuel-ostf has extra files, please remove: .gitignore, > LICENSE, MAINTAINERS, MANIFEST.in, etc, fuel_health, fuel_plugin, > ostf.service, pylintrc, requirements.txt, run_tests.sh, setup.cfg, > setup.py, specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-qa has extra files, please remove: .coveragerc, > .gitignore, .pylintrc, .pylintrc_gerrit, MAINTAINERS, core, doc, > fuel_tests, fuelweb_test, gates_tests, packages_tests, pytest.ini, > run_system_test.py, run_tests.sh, system_test, tox.ini, utils > [fuel] openstack/fuel-ui has extra files, please remove: > .eslintignore, .eslintrc.yaml, .gitignore, LICENSE, MAINTAINERS, > fixtures, gulp, gulpfile.js, karma.config.js, npm-shrinkwrap.json, > package.json, run_real_plugin_tests.sh, > run_real_plugin_tests_on_real_nailgun.sh, run_ui_func_tests.sh, specs, > static, webpack.config.js > [fuel] openstack/fuel-virtualbox has extra files, please remove: > .gitignore, MAINTAINERS, actions, clean.sh, config.sh, contrib, > drivers, dumpkeys.cache, functions, iso, launch.sh, launch_16GB.sh, > launch_8GB.sh > [fuel] openstack/fuel-web has extra files, please remove: .gitignore, > LICENSE, MAINTAINERS, bin, build_docs.sh, debian, docs, nailgun, > run_tests.sh, specs, systemd, tools, tox.ini > [fuel] openstack/shotgun has extra files, please remove: .coveragerc, > .gitignore, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, > MAINTAINERS, MANIFEST.in, bin, etc, requirements.txt, setup.cfg, > setup.py, shotgun, specs, test-requirements.txt, tox.ini > [fuel] openstack/fuel-dev-tools has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, > HACKING.rst, LICENSE, MAINTAINERS, babel.cfg, contrib, doc, > fuel_dev_tools, openstack-common.conf, requirements.txt, setup.cfg, > setup.py, test-requirements.txt, tox.ini, vagrant > [fuel] openstack/fuel-devops has extra files, please remove: > .coveragerc, .gitignore, .pylintrc, .pylintrc_gerrit, LICENSE, > MAINTAINERS, bin, devops, doc, run_tests.sh, samples, setup.cfg, > setup.py, test-requirements.txt, tox.ini > [fuel] openstack/fuel-docs has extra files, please remove: .gitignore, > Makefile, _images, _templates, common_conf.py, conf.py, devdocs, > examples, glossary, index.rst, make.bat, plugindocs, requirements.txt, > setup.cfg, setup.py, tox.ini, userdocs > [fuel] openstack/fuel-nailgun-extension-cluster-upgrade has extra > files, please remove: .coveragerc, .gitignore, AUTHORS, LICENSE, > MANIFEST.in, bindep.txt, cluster_upgrade, conftest.py, > nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, > specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-nailgun-extension-iac has extra files, please > remove: .gitignore, LICENSE, MANIFEST.in, doc, fuel_external_git, > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > tools, tox.ini > [fuel] openstack/fuel-nailgun-extension-converted-serializers has > extra files, please remove: .coveragerc, .gitignore, LICENSE, > MANIFEST.in, bindep.txt, conftest.py, converted_serializers, > nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, > specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-octane has extra files, please remove: > .coveragerc, .gitignore, .mailmap, Gemfile, Gemfile.lock, HACKING.rst, > LICENSE, MAINTAINERS, MANIFEST.in, Rakefile, bindep.txt, deploy, > deployment, docs, misc, octane, requirements.txt, setup.cfg, setup.py, > specs, test-requirements.txt, tox.ini > [fuel] openstack/fuel-upgrade has extra files, please remove: .gitignore > [fuel] openstack/tuning-box has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, > HACKING.rst, LICENSE, MAINTAINERS, MANIFEST.in, TODO, alembic.ini, > babel.cfg, bindep.txt, doc, examples, openstack-common.conf, > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > tools, tox.ini, tuning_box > [fuel] openstack/fuel-plugins has extra files, please remove: > .gitignore, CHANGELOG.md, CONTRIBUTING.rst, HACKING.rst, LICENSE, > MAINTAINERS, examples, fuel_plugin_builder, requirements.txt, > run_tests.sh, setup.cfg, setup.py, test-requirements.txt, tox.ini > [fuel] openstack/fuel-plugin-murano has extra files, please remove: > .gitignore, LICENSE, components.yaml, deployment_scripts, > deployment_tasks.yaml, docs, environment_config.yaml, functions.sh, > metadata.yaml, node_roles.yaml, pre_build_hook, releasenotes, > repositories, test-requirements.txt, tox.ini, volumes.yaml > [fuel] openstack/fuel-plugin-murano-tests has extra files, please > remove: .gitignore, murano_plugin_tests, openrc.default, > requirements.txt, tox.ini, utils > [fuel] openstack/fuel-specs has extra files, please remove: > .gitignore, .testr.conf, LICENSE, doc, images, policy, > requirements.txt, setup.cfg, setup.py, specs, tests, tools, tox.ini > [fuel] openstack/fuel-stats has extra files, please remove: > .gitignore, LICENSE, MAINTAINERS, MANIFEST.in, analytics, collector, > migration, requirements.txt, setup.py, test-requirements.txt, tools, > tox.ini > [fuel] openstack/python-fuelclient has extra files, please remove: > .gitignore, .testr.conf, MAINTAINERS, MANIFEST.in, fuelclient, > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > tools, tox.ini > > [Infrastructure] opendev/puppet-releasestatus has extra files, please > remove: .gitignore > > [ironic] openstack/python-dracclient has extra files, please remove: > .gitignore, CONTRIBUTING.rst, HACKING.rst, LICENSE, doc, dracclient, > requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini > > [neutron] openstack/networking-calico has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .testr.conf, .zuul.yaml, > CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, RELEASING.md, > babel.cfg, debian, devstack, doc, networking_calico, playbooks, > requirements.txt, rpm, setup.cfg, setup.py, test-requirements.txt, > tox.ini IIRC this isn’t neutron stadium project but some 3rd party project. Should it be under “neutron” tag here? > [neutron] openstack/networking-l2gw has extra files, please remove: > .coveragerc, .gitignore, .testr.conf, .zuul.yaml, CONTRIBUTING.rst, > HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, bindep.txt, contrib, > debian, devstack, doc, etc, lower-constraints.txt, networking_l2gw, > openstack-common.conf, requirements.txt, setup.cfg, setup.py, specs, > test-requirements.txt, tools, tox.ini This also isn’t stadium project (at least not listed in https://governance.openstack.org/tc/reference/projects/neutron.html). I also don’t think that networking-l2gw is retired project. I know that there are still some (even new) maintainers for it. > [neutron] openstack/networking-l2gw-tempest-plugin has extra files, > please remove: .gitignore, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, > LICENSE, babel.cfg, contrib, doc, networking_l2gw_tempest_plugin, > requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini > [neutron] openstack/networking-onos has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .pylintrc, .testr.conf, > CONTRIBUTING.rst, HACKING.rst, LICENSE, PKG-INFO, TESTING.rst, > babel.cfg, devstack, doc, etc, lower-constraints.txt, networking_onos, > package, rally-jobs, releasenotes, requirements.txt, setup.cfg, > setup Also 3rd party project. > .py, test-requirements.txt, tools, tox.ini > [neutron] openstack/neutron-vpnaas has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .pylintrc, .stestr.conf, > .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, TESTING.rst, > babel.cfg, devstack, doc, etc, lower-constraints.txt, neutron_vpnaas, > playbooks, rally-jobs, releasenotes, requirements.txt, setup.cfg, > setup.py, test-requirements.txt, tools, tox.ini Neutron-vpnaas isn’t retired IIRC. > > [OpenStack Charms] openstack/charm-ceph has extra files, please > remove: .gitignore > > [OpenStackAnsible] openstack/openstack-ansible-os_monasca has extra > files, please remove: tests, tox.ini > > [solum] openstack/solum-infra-guestagent has extra files, please > remove: .coveragerc, .gitignore, .mailmap, .testr.conf, > CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, > config-generator, doc, etc, requirements.txt, setup.cfg, setup.py, > solum_guestagent, test-requirements.txt, tox.ini > > I'd like to kindly ask the affected teams to help out with this, or > any member of our community is more than welcome to push a change to > those repos and work with the appropriate teams to help land it. > > Mohammed > > -- > Mohammed Naser — vexxhost > ----------------------------------------------------- > D. 514-316-8872 > D. 800-910-1726 ext. 200 > E. mnaser at vexxhost.com > W. http://vexxhost.com > — Slawek Kaplonski Senior software engineer Red Hat From bcafarel at redhat.com Sat Jul 6 20:00:03 2019 From: bcafarel at redhat.com (Bernard Cafarelli) Date: Sat, 6 Jul 2019 22:00:03 +0200 Subject: [docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos In-Reply-To: References: Message-ID: On Sat, 6 Jul 2019 at 19:44, Mohammed Naser wrote: > Hi everyone, > > One of the issue that we recently ran into was the fact that there was > some inconsistency about merging retirement of repositories inside > governance without the code being fully removed. > > In order to avoid this, I've made a change to our governance > repository which will enforce that no code exists in those retired > repositories, however, this has surfaced that some repositories were > retired with some stale files, some are smaller littler files, some > are entire projects still. > > I have compiled a list for every team, with the repos that are not > properly retired that have extra files (using this change which should > eventually +1 once we fix it all: https://review.opendev.org/669549) > > [Documentation] openstack/api-site has extra files, please remove: > .gitignore, .zuul.yaml, LICENSE, api-quick-start, api-ref, bindep.txt, > common, doc-tools-check-languages.conf, firstapp, > test-requirements.txt, tools, tox.ini, www > [Documentation] openstack/faafo has extra files, please remove: > .gitignore, CONTRIBUTING.rst, LICENSE, Vagrantfile, bin, contrib, doc, > etc, faafo, requirements.txt, setup.cfg, setup.py, > test-requirements.txt, tox.ini > > [fuel] openstack/fuel-agent has extra files, please remove: > .gitignore, LICENSE, MAINTAINERS, cloud-init-templates, contrib, > debian, etc, fuel_agent, requirements.txt, run_tests.sh, setup.cfg, > setup.py, specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-astute has extra files, please remove: > .gitignore, .rspec, .ruby-version, Gemfile, LICENSE, MAINTAINERS, > Rakefile, astute.gemspec, astute.service, astute.sysconfig, bin, > bindep.txt, debian, examples, lib, mcagents, run_tests.sh, spec, > specs, tests > [fuel] openstack/fuel-library has extra files, please remove: > .gitignore, CHANGELOG, Gemfile, LICENSE, MAINTAINERS, Rakefile, > debian, deployment, files, graphs, logs, specs, tests, utils > [fuel] openstack/fuel-main has extra files, please remove: .gitignore, > 00-debmirror.patch, LICENSE, MAINTAINERS, Makefile, config.mk, > fuel-release, iso, mirror, packages, prepare-build-env.sh, > report-changelog.sh, repos.mk, requirements-fuel-rpm.txt, > requirements-rpm.txt, rules.mk, sandbox.mk, specs > [fuel] openstack/fuel-menu has extra files, please remove: .gitignore, > MAINTAINERS, MANIFEST.in, fuelmenu, run_tests.sh, setup.py, specs, > test-requirements.txt, tox.ini > [fuel] openstack/fuel-mirror has extra files, please remove: > .gitignore, .mailmap, MAINTAINERS, perestroika, tox.ini > [fuel] openstack/fuel-nailgun-agent has extra files, please remove: > .gitignore, Gemfile, LICENSE, MAINTAINERS, Rakefile, agent, debian, > nailgun-agent.cron, nailgun-agent.gemspec, run_tests.sh, specs > [fuel] openstack/fuel-ostf has extra files, please remove: .gitignore, > LICENSE, MAINTAINERS, MANIFEST.in, etc, fuel_health, fuel_plugin, > ostf.service, pylintrc, requirements.txt, run_tests.sh, setup.cfg, > setup.py, specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-qa has extra files, please remove: .coveragerc, > .gitignore, .pylintrc, .pylintrc_gerrit, MAINTAINERS, core, doc, > fuel_tests, fuelweb_test, gates_tests, packages_tests, pytest.ini, > run_system_test.py, run_tests.sh, system_test, tox.ini, utils > [fuel] openstack/fuel-ui has extra files, please remove: > .eslintignore, .eslintrc.yaml, .gitignore, LICENSE, MAINTAINERS, > fixtures, gulp, gulpfile.js, karma.config.js, npm-shrinkwrap.json, > package.json, run_real_plugin_tests.sh, > run_real_plugin_tests_on_real_nailgun.sh, run_ui_func_tests.sh, specs, > static, webpack.config.js > [fuel] openstack/fuel-virtualbox has extra files, please remove: > .gitignore, MAINTAINERS, actions, clean.sh, config.sh, contrib, > drivers, dumpkeys.cache, functions, iso, launch.sh, launch_16GB.sh, > launch_8GB.sh > [fuel] openstack/fuel-web has extra files, please remove: .gitignore, > LICENSE, MAINTAINERS, bin, build_docs.sh, debian, docs, nailgun, > run_tests.sh, specs, systemd, tools, tox.ini > [fuel] openstack/shotgun has extra files, please remove: .coveragerc, > .gitignore, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, > MAINTAINERS, MANIFEST.in, bin, etc, requirements.txt, setup.cfg, > setup.py, shotgun, specs, test-requirements.txt, tox.ini > [fuel] openstack/fuel-dev-tools has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, > HACKING.rst, LICENSE, MAINTAINERS, babel.cfg, contrib, doc, > fuel_dev_tools, openstack-common.conf, requirements.txt, setup.cfg, > setup.py, test-requirements.txt, tox.ini, vagrant > [fuel] openstack/fuel-devops has extra files, please remove: > .coveragerc, .gitignore, .pylintrc, .pylintrc_gerrit, LICENSE, > MAINTAINERS, bin, devops, doc, run_tests.sh, samples, setup.cfg, > setup.py, test-requirements.txt, tox.ini > [fuel] openstack/fuel-docs has extra files, please remove: .gitignore, > Makefile, _images, _templates, common_conf.py, conf.py, devdocs, > examples, glossary, index.rst, make.bat, plugindocs, requirements.txt, > setup.cfg, setup.py, tox.ini, userdocs > [fuel] openstack/fuel-nailgun-extension-cluster-upgrade has extra > files, please remove: .coveragerc, .gitignore, AUTHORS, LICENSE, > MANIFEST.in, bindep.txt, cluster_upgrade, conftest.py, > nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, > specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-nailgun-extension-iac has extra files, please > remove: .gitignore, LICENSE, MANIFEST.in, doc, fuel_external_git, > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > tools, tox.ini > [fuel] openstack/fuel-nailgun-extension-converted-serializers has > extra files, please remove: .coveragerc, .gitignore, LICENSE, > MANIFEST.in, bindep.txt, conftest.py, converted_serializers, > nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, > specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-octane has extra files, please remove: > .coveragerc, .gitignore, .mailmap, Gemfile, Gemfile.lock, HACKING.rst, > LICENSE, MAINTAINERS, MANIFEST.in, Rakefile, bindep.txt, deploy, > deployment, docs, misc, octane, requirements.txt, setup.cfg, setup.py, > specs, test-requirements.txt, tox.ini > [fuel] openstack/fuel-upgrade has extra files, please remove: .gitignore > [fuel] openstack/tuning-box has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, > HACKING.rst, LICENSE, MAINTAINERS, MANIFEST.in, TODO, alembic.ini, > babel.cfg, bindep.txt, doc, examples, openstack-common.conf, > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > tools, tox.ini, tuning_box > [fuel] openstack/fuel-plugins has extra files, please remove: > .gitignore, CHANGELOG.md, CONTRIBUTING.rst, HACKING.rst, LICENSE, > MAINTAINERS, examples, fuel_plugin_builder, requirements.txt, > run_tests.sh, setup.cfg, setup.py, test-requirements.txt, tox.ini > [fuel] openstack/fuel-plugin-murano has extra files, please remove: > .gitignore, LICENSE, components.yaml, deployment_scripts, > deployment_tasks.yaml, docs, environment_config.yaml, functions.sh, > metadata.yaml, node_roles.yaml, pre_build_hook, releasenotes, > repositories, test-requirements.txt, tox.ini, volumes.yaml > [fuel] openstack/fuel-plugin-murano-tests has extra files, please > remove: .gitignore, murano_plugin_tests, openrc.default, > requirements.txt, tox.ini, utils > [fuel] openstack/fuel-specs has extra files, please remove: > .gitignore, .testr.conf, LICENSE, doc, images, policy, > requirements.txt, setup.cfg, setup.py, specs, tests, tools, tox.ini > [fuel] openstack/fuel-stats has extra files, please remove: > .gitignore, LICENSE, MAINTAINERS, MANIFEST.in, analytics, collector, > migration, requirements.txt, setup.py, test-requirements.txt, tools, > tox.ini > [fuel] openstack/python-fuelclient has extra files, please remove: > .gitignore, .testr.conf, MAINTAINERS, MANIFEST.in, fuelclient, > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > tools, tox.ini > > [Infrastructure] opendev/puppet-releasestatus has extra files, please > remove: .gitignore > > [ironic] openstack/python-dracclient has extra files, please remove: > .gitignore, CONTRIBUTING.rst, HACKING.rst, LICENSE, doc, dracclient, > requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini > > [neutron] openstack/networking-calico has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .testr.conf, .zuul.yaml, > CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, RELEASING.md, > babel.cfg, debian, devstack, doc, networking_calico, playbooks, > requirements.txt, rpm, setup.cfg, setup.py, test-requirements.txt, > tox.ini > [neutron] openstack/networking-l2gw has extra files, please remove: > .coveragerc, .gitignore, .testr.conf, .zuul.yaml, CONTRIBUTING.rst, > HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, bindep.txt, contrib, > debian, devstack, doc, etc, lower-constraints.txt, networking_l2gw, > openstack-common.conf, requirements.txt, setup.cfg, setup.py, specs, > test-requirements.txt, tools, tox.ini > [neutron] openstack/networking-l2gw-tempest-plugin has extra files, > please remove: .gitignore, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, > LICENSE, babel.cfg, contrib, doc, networking_l2gw_tempest_plugin, > requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini > [neutron] openstack/networking-onos has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .pylintrc, .testr.conf, > CONTRIBUTING.rst, HACKING.rst, LICENSE, PKG-INFO, TESTING.rst, > babel.cfg, devstack, doc, etc, lower-constraints.txt, networking_onos, > package, rally-jobs, releasenotes, requirements.txt, setup.cfg, > setup.py, test-requirements.txt, tools, tox.ini > [neutron] openstack/neutron-vpnaas has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .pylintrc, .stestr.conf, > .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, TESTING.rst, > babel.cfg, devstack, doc, etc, lower-constraints.txt, neutron_vpnaas, > playbooks, rally-jobs, releasenotes, requirements.txt, setup.cfg, > setup.py, test-requirements.txt, tools, tox.ini > At least for networking-l2gw* and neutron-vpnaas, I suppose this was caused by: https://opendev.org/openstack/governance/commit/20f95dd947d2f87519b4bb50fb188e6f71deae7c What it meant is that they are not anymore under neutron governance, but they were not retired (at least as far as I know). There were still some recent commits even if minimal activity, and discussion on team status for neutron-vpnaas. Not sure about networking-calico status though > > [OpenStack Charms] openstack/charm-ceph has extra files, please > remove: .gitignore > > [OpenStackAnsible] openstack/openstack-ansible-os_monasca has extra > files, please remove: tests, tox.ini > > [solum] openstack/solum-infra-guestagent has extra files, please > remove: .coveragerc, .gitignore, .mailmap, .testr.conf, > CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, > config-generator, doc, etc, requirements.txt, setup.cfg, setup.py, > solum_guestagent, test-requirements.txt, tox.ini > > I'd like to kindly ask the affected teams to help out with this, or > any member of our community is more than welcome to push a change to > those repos and work with the appropriate teams to help land it. > > Mohammed > > -- > Mohammed Naser — vexxhost > ----------------------------------------------------- > D. 514-316-8872 > D. 800-910-1726 ext. 200 > E. mnaser at vexxhost.com > W. http://vexxhost.com > > -- Bernard Cafarelli -------------- next part -------------- An HTML attachment was scrubbed... URL: From vungoctan252 at gmail.com Sun Jul 7 03:30:39 2019 From: vungoctan252 at gmail.com (Vu Tan) Date: Sun, 7 Jul 2019 10:30:39 +0700 Subject: how to install masakari on centos 7 Message-ID: Hi, I would like to use Masakari and I'm having trouble finding a step by step or other documentation to get started with. Which part should be installed on controller, which is should be on compute, and what is the prerequisite to install masakari, I have installed corosync and pacemaker on compute and controller nodes, , what else do I need to do ? step I have done so far: - installed corosync/pacemaker - install masakari on compute node on this github repo: https://github.com/openstack/masakari - add masakari in to mariadb here is my configuration file of masakari.conf, do you mind to take a look at it, if I have misconfigured anything? [DEFAULT] enabled_apis = masakari_api # Enable to specify listening IP other than default masakari_api_listen = controller # Enable to specify port other than default masakari_api_listen_port = 15868 debug = False auth_strategy=keystone [wsgi] # The paste configuration file path api_paste_config = /etc/masakari/api-paste.ini [keystone_authtoken] www_authenticate_uri = http://controller:5000 auth_url = http://controller:5000 auth_type = password project_domain_id = default user_domain_id = default project_name = service username = masakari password = P at ssword [database] connection = mysql+pymysql://masakari:P at ssword@controller/masakari -------------- next part -------------- An HTML attachment was scrubbed... URL: From vungoctan252 at gmail.com Sun Jul 7 04:08:07 2019 From: vungoctan252 at gmail.com (Vu Tan) Date: Sun, 7 Jul 2019 11:08:07 +0700 Subject: [masakari] how to install masakari on centos 7 Message-ID: Vu Tan 10:30 AM (35 minutes ago) to openstack-discuss Sorry, I resend this email because I realized that I lacked of prefix on this email's subject Hi, I would like to use Masakari and I'm having trouble finding a step by step or other documentation to get started with. Which part should be installed on controller, which is should be on compute, and what is the prerequisite to install masakari, I have installed corosync and pacemaker on compute and controller nodes, , what else do I need to do ? step I have done so far: - installed corosync/pacemaker - install masakari on compute node on this github repo: https://github.com/openstack/masakari - add masakari in to mariadb here is my configuration file of masakari.conf, do you mind to take a look at it, if I have misconfigured anything? [DEFAULT] enabled_apis = masakari_api # Enable to specify listening IP other than default masakari_api_listen = controller # Enable to specify port other than default masakari_api_listen_port = 15868 debug = False auth_strategy=keystone [wsgi] # The paste configuration file path api_paste_config = /etc/masakari/api-paste.ini [keystone_authtoken] www_authenticate_uri = http://controller:5000 auth_url = http://controller:5000 auth_type = password project_domain_id = default user_domain_id = default project_name = service username = masakari password = P at ssword [database] connection = mysql+pymysql://masakari:P at ssword@controller/masakari -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetan.trellu at incloudus.com Sun Jul 7 12:37:05 2019 From: gaetan.trellu at incloudus.com (=?ISO-8859-1?Q?Ga=EBtan_Trellu?=) Date: Sun, 07 Jul 2019 08:37:05 -0400 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: Message-ID: <09a3849b-786e-49ed-a197-5e13af0428bf@email.android.com> An HTML attachment was scrubbed... URL: From moguimar at redhat.com Sun Jul 7 13:09:10 2019 From: moguimar at redhat.com (Moises Guimaraes de Medeiros) Date: Sun, 7 Jul 2019 15:09:10 +0200 Subject: [oslo] oslo.config /castellan poster review Message-ID: Hi, This week I'll be presenting a poster about oslo.config's castellan driver at EuroPython. I'd like to ask y'all interested in the subject to take a look at my poster. I'm planning to print it this Tuesday and I still have some spare space to fit a bit more. The latest version is available at: https://ep2019.europython.eu/media/conference/slides/m7RV4BB-protecting-secrets-with-osloconfig-and-hashicorp-vault.pdf Thanks a lot! -- Moisés Guimarães Software Engineer Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Sun Jul 7 13:41:50 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Sun, 07 Jul 2019 22:41:50 +0900 Subject: [qa][ptg][nova][cinder][keystone][neutron][glance][swift][placement] How to make integrated-gate testing (tempest-full) more stable and fast In-Reply-To: <16afe9a95fa.ff3031a3146625.6650617153857325463@ghanshyammann.com> References: <16a86db6ccd.d787148123989.2198391414179782565@ghanshyammann.com> <16af8ac7fc7.fa901245105341.2925519493395080868@ghanshyammann.com> <16afe9a95fa.ff3031a3146625.6650617153857325463@ghanshyammann.com> Message-ID: <16bccab5abc.efb5a50b274261.309866536645601231@ghanshyammann.com> ---- On Tue, 28 May 2019 22:21:44 +0900 Ghanshyam Mann wrote ---- > ---- On Mon, 27 May 2019 18:43:35 +0900 Ghanshyam Mann wrote ---- > > ---- On Tue, 07 May 2019 07:06:23 +0900 Morgan Fainberg wrote ---- > > > > > > > > > On Sun, May 5, 2019 at 12:19 AM Ghanshyam Mann wrote: > > > > > > For the "Integrated-gate-identity", I have a slight worry that we might lose some coverage with this change. I am unsure of how varied the use of Keystone is outside of KeystoneMiddleware (i.e. token validation) consumption that all services perform, Heat (not part of the integrated gate) and it's usage of Trusts, and some newer emerging uses such as "look up limit data" (potentially in Train, would be covered by Nova). Worst case, we could run all the integrated tests for Keystone changes (at least initially) until we have higher confidence and minimize the tests once we have a clearer audit of how the services use Keystone. The changes would speed up/minimize the usage for the other services directly and Keystone can follow down the line. > > > I want to be as close to 100% sure we're not going to suddenly break everyone because of some change we land. Keystone fortunately and unfortunately sits below most other services in an OpenStack deployment and is heavily relied throughout almost every single request. > > > --Morgan > > > > > > Thanks Morgan. That was what we were worried during PTG discussion. I agree with your point about not to lose coverage and first get to know how Keystone is being used by each service. Let's keep running the all service tests for keystone gate as of now and later we can shorten the tests run based on the clarity of usage. > > We can disable the ssh validation for "Integrated-gate-identity" which keystone does not need to care about. This can save the keystone gate for ssh timeout failure. > > -gmann > > > > > -gmann > > > > > > > Current integrated-gate jobs (tempest-full) is not so stable for various bugs specially timeout. We tried > > > to improve it via filtering the slow tests in the separate tempest-slow job but the situation has not been improved much. > > > > > > We talked about the Ideas to make it more stable and fast for projects especially when failure is not > > > related to each project. We are planning to split the integrated-gate template (only tempest-full job as > > > first step) per related services. > > > > > > Idea: > > > - Run only dependent service tests on project gate. > > > - Tempest gate will keep running all the services tests as the integrated gate at a centeralized place without any change in the current job. > > > - Each project can run the below mentioned template. > > > - All below template will be defined and maintained by QA team. > > > > > > I would like to know each 6 services which run integrated-gate jobs > > > > > > 1."Integrated-gate-networking" (job to run on neutron gate) > > > Tests to run in this template: neutron APIs , nova APIs, keystone APIs ? All scenario currently running in tempest-full in the same way ( means non-slow and in serial) > > > Improvement for neutron gate: exlcude the cinder API tests, glance API tests, swift API tests, > > > > > > 2."Integrated-gate-storage" (job to run on cinder gate, glance gate) > > > Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs, Nova APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial) > > > Improvement for cinder, glance gate: excluded the neutron APIs tests, Keystone APIs tests > > > > > > 3. "Integrated-gate-object-storage" (job to run on swift gate) > > > Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial) > > > Improvement for swift gate: excluded the neutron APIs tests, - Keystone APIs tests, - Nova APIs tests. > > > Note: swift does not run integrated-gate as of now. > > > > > > 4. "Integrated-gate-compute" (job to run on Nova gate) > > > tests to run is : Nova APIs, Cinder APIs , Glance APIs ?, neutron APIs and All scenario currently running in tempest-full in same way ( means non-slow and in serial) > > > Improvement for Nova gate: excluded the swift APIs tests(not running in current job but in future, it might), Keystone API tests. > > > > > > 5. "Integrated-gate-identity" (job to run on keystone gate) > > > Tests to run is : all as all project use keystone, we might need to run all tests as it is running in integrated-gate. > > > But does keystone is being unsed differently by all services? if no then, is it enough to run only single service tests say Nova or neutron ? > > > > > > 6. "Integrated-gate-placement" (job to run on placement gate) > > > Tests to run in this template: Nova APIs tests, Neutron APIs tests + scenario tests + any new service depends on placement APIs > > > Improvement for placement gate: excluded the glance APIs tests, cinder APIs tests, swift APIs tests, keystone APIs tests > > > I have prepared the new template for integrated gate testing[1] and tested in DNM patch [2]. You can observe ~20 min less time on new jobs(except compute one). But the main thing is will improve the stability of gate. Once they are merged, I will propose the patch to replace those template on the projects gate. NOTE: Along with APIs tests, I have back listed the non-dependent scenario tests also. [1] https://review.opendev.org/#/q/topic:refactor-integrated-gate-testing+(status:open+OR+status:merged) [2] https://review.opendev.org/#/c/669313/ -gmann > > > Thoughts on this approach? > > > > > > The important point is we must not lose the coverage of integrated testing per project. So I would like to > > > get each project view if we are missing any dependency (proposed tests removal) in above proposed templates. > > > > > > - https://etherpad.openstack.org/p/qa-train-ptg > > > > > > -gmann > > > > > > > > > > > > From doug at doughellmann.com Sun Jul 7 14:21:07 2019 From: doug at doughellmann.com (Doug Hellmann) Date: Sun, 07 Jul 2019 10:21:07 -0400 Subject: [oslo] oslo.config /castellan poster review In-Reply-To: References: Message-ID: Moises Guimaraes de Medeiros writes: > Hi, > > This week I'll be presenting a poster about oslo.config's castellan driver > at EuroPython. I'd like to ask y'all interested in the subject to take a > look at my poster. I'm planning to print it this Tuesday and I still have > some spare space to fit a bit more. > > The latest version is available at: > > https://ep2019.europython.eu/media/conference/slides/m7RV4BB-protecting-secrets-with-osloconfig-and-hashicorp-vault.pdf > > Thanks a lot! > > -- > > Moisés Guimarães > > Software Engineer > > Red Hat > > That looks great, Moises, nice work! -- Doug From ruijing.guo at intel.com Mon Jul 8 00:47:28 2019 From: ruijing.guo at intel.com (Guo, Ruijing) Date: Mon, 8 Jul 2019 00:47:28 +0000 Subject: [Neutron] NUMA aware VxLAN Message-ID: <2EE296D083DF2940BF4EBB91D39BB89F40CC0832@SHSMSX104.ccr.corp.intel.com> Hi, Existing neutron ML2 support one VxLAN for tenant network. In NUMA case, VM 0 can be created in node 0 and VM 1 can be created in node 1 and VxLAN is in node 0. VM1 need to cross node, which cause some performance downgrade. Does someone have this performance issue? Does Neutron community have plan to enhance it? Thanks, -Ruijing -------------- next part -------------- An HTML attachment was scrubbed... URL: From aj at suse.com Mon Jul 8 05:24:54 2019 From: aj at suse.com (Andreas Jaeger) Date: Mon, 8 Jul 2019 07:24:54 +0200 Subject: [docs][tc] proper retirement of repos In-Reply-To: References: Message-ID: <442f7f8e-6ba0-393e-e99d-7d59dc1e4911@suse.com> On 06/07/2019 19.38, Mohammed Naser wrote: > [Documentation] openstack/api-site has extra files, please remove: > .gitignore, .zuul.yaml, LICENSE, api-quick-start, api-ref, bindep.txt, > common, doc-tools-check-languages.conf, firstapp, > test-requirements.txt, tools, tox.ini, www > [Documentation] openstack/faafo has extra files, please remove: > .gitignore, CONTRIBUTING.rst, LICENSE, Vagrantfile, bin, contrib, doc, > etc, faafo, requirements.txt, setup.cfg, setup.py, > test-requirements.txt, tox.ini These repos were not really retired, they were just removed out of docs ownership but need a new owner, it still contains files for developer.openstack.org. api-site is still active. I would propose to remove content we cannot maintain and move it back into docs if no other owner is found. Is there a better way to flag this in governance repo? Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF: Nils Brauckmann, Felix Imendörffer, Enrica Angelone, HRB 247165 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From aj at suse.com Mon Jul 8 05:28:05 2019 From: aj at suse.com (Andreas Jaeger) Date: Mon, 8 Jul 2019 07:28:05 +0200 Subject: [docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos In-Reply-To: References: Message-ID: On 06/07/2019 19.38, Mohammed Naser wrote: > Hi everyone, > > One of the issue that we recently ran into was the fact that there was > some inconsistency about merging retirement of repositories inside > governance without the code being fully removed. > > In order to avoid this, I've made a change to our governance > repository which will enforce that no code exists in those retired > repositories, however, this has surfaced that some repositories were > retired with some stale files, some are smaller littler files, some > are entire projects still. > [...] Fuel and some networking repos were not retired (see comments in this thread), instead they were moved out of governance. So, your check is too aggressive, it catches not only the RETIRED case but also the "still active but not under governance" case. AFAIK there's no differentiation in governance repo for these, Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF: Nils Brauckmann, Felix Imendörffer, Enrica Angelone, HRB 247165 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From ykarel at redhat.com Mon Jul 8 06:07:56 2019 From: ykarel at redhat.com (Yatin Karel) Date: Mon, 8 Jul 2019 11:37:56 +0530 Subject: [TripleO] Using TripleO standalone with ML2/OVS In-Reply-To: <8F8016EB-7898-4F66-BD30-998ABDB094FB@redhat.com> References: <8F8016EB-7898-4F66-BD30-998ABDB094FB@redhat.com> Message-ID: Hi Slawek, So from workflow perspective you just need to pass an environment file with correct set of parameters for ovs(resource_registry and parameter_defaults) to "openstack tripleo deploy" command in the end(to override the defaults) with -e . In CI multinode ovs jobs are running with environment files containing ovs specific parameters. For Multinode ovs scenario the environment file [1]. Standalone ovs work(creation of environment file and CI job) is WIP by Marios[1]. I just rechecked the job to see where it's stuck as last failures were unrelated to ovs. @Marios Andreou can share exact status and @Slawomir Kaplonski you might also help in clearing it. From job logs of WIP patch you can check command for reference[3]. You can use standalone environment files directly or use them as a reference and create custom environment files as per your use case. I have not deployed myself standalone with ovs or any other backend so can't tell exact parameters but if i try i will start with these references until someone guides for exact set of parameters someone uses for standlone ovs deployment. May be someone who have tried it will post on this mailing list. [1] https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario007-multinode-containers.yaml [2] https://review.opendev.org/#/q/topic:scenario007+(status:open+OR+status:merged) [3] http://logs.openstack.org/97/631497/18/check/tripleo-ci-centos-7-scenario007-standalone/a5ee2d3/logs/undercloud/home/zuul/standalone.sh.txt.gz Regards Yatin Karel On Sat, Jul 6, 2019 at 2:33 PM Slawek Kaplonski wrote: > > Hi, > > I was trying to use TripleO standalone for development work in the way how Emilien described it in [1] and indeed it works quite well. Thx Emilien. > But now, I’m trying to find out the way how to deploy it with Neutron using ML2/OVS instead of default in TripleO ML2/OVN. > And I still don’t know how to do it :/ > I know it’s my fault but maybe someone can help me with this and tell me what exactly options I should change there to deploy it with other Neutron backend? > Thx in advance for any help. > > [1] https://my1.fr/blog/developer-workflow-with-tripleo/ > > — > Slawek Kaplonski > Senior software engineer > Red Hat > > From sbauza at redhat.com Mon Jul 8 07:45:17 2019 From: sbauza at redhat.com (Sylvain Bauza) Date: Mon, 8 Jul 2019 09:45:17 +0200 Subject: [ops] [nova] [placement] Mismatch between allocations and instances In-Reply-To: References: Message-ID: On Fri, Jul 5, 2019 at 10:21 PM Matt Riedemann wrote: > On 7/5/2019 1:45 AM, Massimo Sgaravatto wrote: > > I tried to check the allocations on each compute node of a Ocata cloud, > > using the command: > > > > curl -s ${PLACEMENT_ENDPOINT}/resource_providers/${UUID}/allocations -H > > "x-auth-token: $TOKEN" | python -m json.tool > > > > Just FYI you can use osc-placement (openstack client plugin) for command > line: > > https://docs.openstack.org/osc-placement/latest/index.html > > > I found that, on a few compute nodes, there are some instances for which > > there is not a corresponding allocation. > > The heal_allocations command [1] might be able to find and fix these up > for you. The bad news for you is that heal_allocations wasn't added > until Rocky and you're on Ocata. The good news is you should be able to > take the current version of the code from master (or stein) and run that > in a container or virtual environment against your Ocata cloud (this > would be particularly useful if you want to use the --dry-run or > --instance options added in Train). You could also potentially backport > those changes to your internal branch, or we could start a discussion > upstream about backporting that tooling to stable branches - though > going to Ocata might be a bit much at this point given Ocata and Pike > are in extended maintenance mode [2]. > > As for *why* the instances on those nodes are missing allocations, it's > hard to say without debugging things. The allocation and resource > tracking code has changed quite a bit since Ocata (in Pike the scheduler > started creating the allocations but the resource tracker in the compute > service could still overwrite those allocations if you had older nodes > during a rolling upgrade). My guess would be a migration failed or there > was just a bug in Ocata where we didn't cleanup or allocate properly. > Again, heal_allocations should add the missing allocation for you if you > can setup the environment to run that command. > > > > > On another Rocky cloud, we had the opposite problem: there were > > allocations also for some instances that didn't exist anymore. > > And this caused problems since we were not able to use all the resources > > of the relevant compute nodes: we had to manually remove the fwrong" > > allocations to fix the problem ... > > Yup, this could happen for different reasons, usually all due to known > bugs for which you don't have the fix yet, e.g. [3][4], or something is > failing during a migration and we aren't cleaning up properly (an > unreported/not-yet-fixed bug). > > > > > > > I wonder why/how this problem can happen ... > > I mentioned some possibilities above - but I'm sure there are other bugs > that have been fixed which I've omitted here, or things that aren't > fixed yet, especially in failure scenarios (rollback/cleanup handling is > hard). > > Note that your Ocata and Rocky cases could be different because since > Queens (once all compute nodes are >=Queens) during resize, cold and > live migration the migration record in nova holds the source node > allocations during the migration so the actual *consumer* of the > allocations for a provider in placement might not be an instance > (server) record but actually a migration, so if you were looking for an > allocation consumer by ID in nova using something like "openstack server > show $consumer_id" it might return NotFound because the consumer is > actually not an instance but a migration record and the allocation was > leaked. > > > > > And how can we fix the issue ? Should we manually add the missing > > allocations / manually remove the wrong ones ? > > Coincidentally a thread related to this [5] re-surfaced a couple of > weeks ago. I am not sure what Sylvain's progress is on that audit tool, > but the linked bug in that email has some other operator scripts you > could try for the case that there are leaked/orphaned allocations on > compute nodes that no longer have instances. > > Yeah, I'm fighting off with the change due to some issues, but I'll hopefully upload the change by the next days. -Sylvain > > > Thanks, Massimo > > > > > > [1] https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement > [2] https://docs.openstack.org/project-team-guide/stable-branches.html > [3] https://bugs.launchpad.net/nova/+bug/1825537 > [4] https://bugs.launchpad.net/nova/+bug/1821594 > [5] > > http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007241.html > > -- > > Thanks, > > Matt > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.sgaravatto at gmail.com Mon Jul 8 07:46:01 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Mon, 8 Jul 2019 09:46:01 +0200 Subject: [ops] [nova] [placement] Mismatch between allocations and instances In-Reply-To: References: Message-ID: On Fri, Jul 5, 2019 at 10:18 PM Matt Riedemann wrote: > On 7/5/2019 1:45 AM, Massimo Sgaravatto wrote: > > I tried to check the allocations on each compute node of a Ocata cloud, > > using the command: > > > > curl -s ${PLACEMENT_ENDPOINT}/resource_providers/${UUID}/allocations -H > > "x-auth-token: $TOKEN" | python -m json.tool > > > > Just FYI you can use osc-placement (openstack client plugin) for command > line: > > https://docs.openstack.org/osc-placement/latest/index.html > > Ok, thanks In the Rocky cloud I had to manually install the python2-osc-placement package. At least for centos7 it is not required by python2-openstackclient > > I found that, on a few compute nodes, there are some instances for which > > there is not a corresponding allocation. > > The heal_allocations command [1] might be able to find and fix these up > for you. The bad news for you is that heal_allocations wasn't added > until Rocky and you're on Ocata. Since in 1 week the Ocata cloud we'll be migrated to Rocky, I can wait ... :-) Thanks a lot for your help ! Cheers, Massimo -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Mon Jul 8 08:46:37 2019 From: katonalala at gmail.com (Lajos Katona) Date: Mon, 8 Jul 2019 10:46:37 +0200 Subject: [docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos In-Reply-To: References: Message-ID: Hi, For networking-l2gw (and networking-l2gw-tempest-plugin) I can say that it is maintained (with not that great activity, but anyway far from rerirement). Is there anything I have to do to make this "activeness" visible from governance perspective? Lajos Bernard Cafarelli ezt írta (időpont: 2019. júl. 6., Szo, 22:05): > > > On Sat, 6 Jul 2019 at 19:44, Mohammed Naser wrote: > >> Hi everyone, >> >> One of the issue that we recently ran into was the fact that there was >> some inconsistency about merging retirement of repositories inside >> governance without the code being fully removed. >> >> In order to avoid this, I've made a change to our governance >> repository which will enforce that no code exists in those retired >> repositories, however, this has surfaced that some repositories were >> retired with some stale files, some are smaller littler files, some >> are entire projects still. >> >> I have compiled a list for every team, with the repos that are not >> properly retired that have extra files (using this change which should >> eventually +1 once we fix it all: https://review.opendev.org/669549) >> >> [Documentation] openstack/api-site has extra files, please remove: >> .gitignore, .zuul.yaml, LICENSE, api-quick-start, api-ref, bindep.txt, >> common, doc-tools-check-languages.conf, firstapp, >> test-requirements.txt, tools, tox.ini, www >> [Documentation] openstack/faafo has extra files, please remove: >> .gitignore, CONTRIBUTING.rst, LICENSE, Vagrantfile, bin, contrib, doc, >> etc, faafo, requirements.txt, setup.cfg, setup.py, >> test-requirements.txt, tox.ini >> >> [fuel] openstack/fuel-agent has extra files, please remove: >> .gitignore, LICENSE, MAINTAINERS, cloud-init-templates, contrib, >> debian, etc, fuel_agent, requirements.txt, run_tests.sh, setup.cfg, >> setup.py, specs, test-requirements.txt, tools, tox.ini >> [fuel] openstack/fuel-astute has extra files, please remove: >> .gitignore, .rspec, .ruby-version, Gemfile, LICENSE, MAINTAINERS, >> Rakefile, astute.gemspec, astute.service, astute.sysconfig, bin, >> bindep.txt, debian, examples, lib, mcagents, run_tests.sh, spec, >> specs, tests >> [fuel] openstack/fuel-library has extra files, please remove: >> .gitignore, CHANGELOG, Gemfile, LICENSE, MAINTAINERS, Rakefile, >> debian, deployment, files, graphs, logs, specs, tests, utils >> [fuel] openstack/fuel-main has extra files, please remove: .gitignore, >> 00-debmirror.patch, LICENSE, MAINTAINERS, Makefile, config.mk, >> fuel-release, iso, mirror, packages, prepare-build-env.sh, >> report-changelog.sh, repos.mk, requirements-fuel-rpm.txt, >> requirements-rpm.txt, rules.mk, sandbox.mk, specs >> [fuel] openstack/fuel-menu has extra files, please remove: .gitignore, >> MAINTAINERS, MANIFEST.in, fuelmenu, run_tests.sh, setup.py, specs, >> test-requirements.txt, tox.ini >> [fuel] openstack/fuel-mirror has extra files, please remove: >> .gitignore, .mailmap, MAINTAINERS, perestroika, tox.ini >> [fuel] openstack/fuel-nailgun-agent has extra files, please remove: >> .gitignore, Gemfile, LICENSE, MAINTAINERS, Rakefile, agent, debian, >> nailgun-agent.cron, nailgun-agent.gemspec, run_tests.sh, specs >> [fuel] openstack/fuel-ostf has extra files, please remove: .gitignore, >> LICENSE, MAINTAINERS, MANIFEST.in, etc, fuel_health, fuel_plugin, >> ostf.service, pylintrc, requirements.txt, run_tests.sh, setup.cfg, >> setup.py, specs, test-requirements.txt, tools, tox.ini >> [fuel] openstack/fuel-qa has extra files, please remove: .coveragerc, >> .gitignore, .pylintrc, .pylintrc_gerrit, MAINTAINERS, core, doc, >> fuel_tests, fuelweb_test, gates_tests, packages_tests, pytest.ini, >> run_system_test.py, run_tests.sh, system_test, tox.ini, utils >> [fuel] openstack/fuel-ui has extra files, please remove: >> .eslintignore, .eslintrc.yaml, .gitignore, LICENSE, MAINTAINERS, >> fixtures, gulp, gulpfile.js, karma.config.js, npm-shrinkwrap.json, >> package.json, run_real_plugin_tests.sh, >> run_real_plugin_tests_on_real_nailgun.sh, run_ui_func_tests.sh, specs, >> static, webpack.config.js >> [fuel] openstack/fuel-virtualbox has extra files, please remove: >> .gitignore, MAINTAINERS, actions, clean.sh, config.sh, contrib, >> drivers, dumpkeys.cache, functions, iso, launch.sh, launch_16GB.sh, >> launch_8GB.sh >> [fuel] openstack/fuel-web has extra files, please remove: .gitignore, >> LICENSE, MAINTAINERS, bin, build_docs.sh, debian, docs, nailgun, >> run_tests.sh, specs, systemd, tools, tox.ini >> [fuel] openstack/shotgun has extra files, please remove: .coveragerc, >> .gitignore, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, >> MAINTAINERS, MANIFEST.in, bin, etc, requirements.txt, setup.cfg, >> setup.py, shotgun, specs, test-requirements.txt, tox.ini >> [fuel] openstack/fuel-dev-tools has extra files, please remove: >> .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, >> HACKING.rst, LICENSE, MAINTAINERS, babel.cfg, contrib, doc, >> fuel_dev_tools, openstack-common.conf, requirements.txt, setup.cfg, >> setup.py, test-requirements.txt, tox.ini, vagrant >> [fuel] openstack/fuel-devops has extra files, please remove: >> .coveragerc, .gitignore, .pylintrc, .pylintrc_gerrit, LICENSE, >> MAINTAINERS, bin, devops, doc, run_tests.sh, samples, setup.cfg, >> setup.py, test-requirements.txt, tox.ini >> [fuel] openstack/fuel-docs has extra files, please remove: .gitignore, >> Makefile, _images, _templates, common_conf.py, conf.py, devdocs, >> examples, glossary, index.rst, make.bat, plugindocs, requirements.txt, >> setup.cfg, setup.py, tox.ini, userdocs >> [fuel] openstack/fuel-nailgun-extension-cluster-upgrade has extra >> files, please remove: .coveragerc, .gitignore, AUTHORS, LICENSE, >> MANIFEST.in, bindep.txt, cluster_upgrade, conftest.py, >> nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, >> specs, test-requirements.txt, tools, tox.ini >> [fuel] openstack/fuel-nailgun-extension-iac has extra files, please >> remove: .gitignore, LICENSE, MANIFEST.in, doc, fuel_external_git, >> requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, >> tools, tox.ini >> [fuel] openstack/fuel-nailgun-extension-converted-serializers has >> extra files, please remove: .coveragerc, .gitignore, LICENSE, >> MANIFEST.in, bindep.txt, conftest.py, converted_serializers, >> nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, >> specs, test-requirements.txt, tools, tox.ini >> [fuel] openstack/fuel-octane has extra files, please remove: >> .coveragerc, .gitignore, .mailmap, Gemfile, Gemfile.lock, HACKING.rst, >> LICENSE, MAINTAINERS, MANIFEST.in, Rakefile, bindep.txt, deploy, >> deployment, docs, misc, octane, requirements.txt, setup.cfg, setup.py, >> specs, test-requirements.txt, tox.ini >> [fuel] openstack/fuel-upgrade has extra files, please remove: .gitignore >> [fuel] openstack/tuning-box has extra files, please remove: >> .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, >> HACKING.rst, LICENSE, MAINTAINERS, MANIFEST.in, TODO, alembic.ini, >> babel.cfg, bindep.txt, doc, examples, openstack-common.conf, >> requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, >> tools, tox.ini, tuning_box >> [fuel] openstack/fuel-plugins has extra files, please remove: >> .gitignore, CHANGELOG.md, CONTRIBUTING.rst, HACKING.rst, LICENSE, >> MAINTAINERS, examples, fuel_plugin_builder, requirements.txt, >> run_tests.sh, setup.cfg, setup.py, test-requirements.txt, tox.ini >> [fuel] openstack/fuel-plugin-murano has extra files, please remove: >> .gitignore, LICENSE, components.yaml, deployment_scripts, >> deployment_tasks.yaml, docs, environment_config.yaml, functions.sh, >> metadata.yaml, node_roles.yaml, pre_build_hook, releasenotes, >> repositories, test-requirements.txt, tox.ini, volumes.yaml >> [fuel] openstack/fuel-plugin-murano-tests has extra files, please >> remove: .gitignore, murano_plugin_tests, openrc.default, >> requirements.txt, tox.ini, utils >> [fuel] openstack/fuel-specs has extra files, please remove: >> .gitignore, .testr.conf, LICENSE, doc, images, policy, >> requirements.txt, setup.cfg, setup.py, specs, tests, tools, tox.ini >> [fuel] openstack/fuel-stats has extra files, please remove: >> .gitignore, LICENSE, MAINTAINERS, MANIFEST.in, analytics, collector, >> migration, requirements.txt, setup.py, test-requirements.txt, tools, >> tox.ini >> [fuel] openstack/python-fuelclient has extra files, please remove: >> .gitignore, .testr.conf, MAINTAINERS, MANIFEST.in, fuelclient, >> requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, >> tools, tox.ini >> >> [Infrastructure] opendev/puppet-releasestatus has extra files, please >> remove: .gitignore >> >> [ironic] openstack/python-dracclient has extra files, please remove: >> .gitignore, CONTRIBUTING.rst, HACKING.rst, LICENSE, doc, dracclient, >> requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini >> >> [neutron] openstack/networking-calico has extra files, please remove: >> .coveragerc, .gitignore, .mailmap, .testr.conf, .zuul.yaml, >> CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, RELEASING.md, >> babel.cfg, debian, devstack, doc, networking_calico, playbooks, >> requirements.txt, rpm, setup.cfg, setup.py, test-requirements.txt, >> tox.ini >> [neutron] openstack/networking-l2gw has extra files, please remove: >> .coveragerc, .gitignore, .testr.conf, .zuul.yaml, CONTRIBUTING.rst, >> HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, bindep.txt, contrib, >> debian, devstack, doc, etc, lower-constraints.txt, networking_l2gw, >> openstack-common.conf, requirements.txt, setup.cfg, setup.py, specs, >> test-requirements.txt, tools, tox.ini >> [neutron] openstack/networking-l2gw-tempest-plugin has extra files, >> please remove: .gitignore, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, >> LICENSE, babel.cfg, contrib, doc, networking_l2gw_tempest_plugin, >> requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini >> [neutron] openstack/networking-onos has extra files, please remove: >> .coveragerc, .gitignore, .mailmap, .pylintrc, .testr.conf, >> CONTRIBUTING.rst, HACKING.rst, LICENSE, PKG-INFO, TESTING.rst, >> babel.cfg, devstack, doc, etc, lower-constraints.txt, networking_onos, >> package, rally-jobs, releasenotes, requirements.txt, setup.cfg, >> setup.py, test-requirements.txt, tools, tox.ini >> [neutron] openstack/neutron-vpnaas has extra files, please remove: >> .coveragerc, .gitignore, .mailmap, .pylintrc, .stestr.conf, >> .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, TESTING.rst, >> babel.cfg, devstack, doc, etc, lower-constraints.txt, neutron_vpnaas, >> playbooks, rally-jobs, releasenotes, requirements.txt, setup.cfg, >> setup.py, test-requirements.txt, tools, tox.ini >> > At least for networking-l2gw* and neutron-vpnaas, I suppose this was > caused by: > > https://opendev.org/openstack/governance/commit/20f95dd947d2f87519b4bb50fb188e6f71deae7c > What it meant is that they are not anymore under neutron governance, but > they were not retired (at least as far as I know). > There were still some recent commits even if minimal activity, and > discussion on team status for neutron-vpnaas. > > Not sure about networking-calico status though > > >> >> [OpenStack Charms] openstack/charm-ceph has extra files, please >> remove: .gitignore >> >> [OpenStackAnsible] openstack/openstack-ansible-os_monasca has extra >> files, please remove: tests, tox.ini >> >> [solum] openstack/solum-infra-guestagent has extra files, please >> remove: .coveragerc, .gitignore, .mailmap, .testr.conf, >> CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, >> config-generator, doc, etc, requirements.txt, setup.cfg, setup.py, >> solum_guestagent, test-requirements.txt, tox.ini >> >> I'd like to kindly ask the affected teams to help out with this, or >> any member of our community is more than welcome to push a change to >> those repos and work with the appropriate teams to help land it. >> >> Mohammed >> >> -- >> Mohammed Naser — vexxhost >> ----------------------------------------------------- >> D. 514-316-8872 >> D. 800-910-1726 ext. 200 >> E. mnaser at vexxhost.com >> W. http://vexxhost.com >> >> > > -- > Bernard Cafarelli > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcafarel at redhat.com Mon Jul 8 09:16:22 2019 From: bcafarel at redhat.com (Bernard Cafarelli) Date: Mon, 8 Jul 2019 11:16:22 +0200 Subject: [neutron] Bug deputy report (week starting on 2019-07-01) Message-ID: Hi Neutrinos, time for a new cycle of bug deputy rotation, which means I was on duty last week, checking bugs up to 1835663 included Quite a few bugs in that list worth a read and further discussion: * port status changing to UP when changing the network name (and possible fallout on DHCP issues) * API currently allowing to set gateway outside of the subnet * How to handle external dependencies recommended version bumps (pyroute bump in queens fixing a memory leak) * possible issue in IPv6 address renewal since we started cleaning the dnsmasq leases file Critical: * Wrong endpoints config with configure_auth_token_middleware - https://bugs.launchpad.net/bugs/1834849 Removal of devstack deprecated option broke designate scenario job Fix merged: https://review.opendev.org/668447 High: * [Queens] Memory leak in pyroute2 0.4.21 - https://bugs.launchpad.net/neutron/+bug/1835044 we need a newer pyroute version Patches on requirements with DNM neutron one for testing in progress: https://review.opendev.org/668676 and https://review.opendev.org/668677 openstack-discuss thread: http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007545.html * Bulk-created ports ignore binding_host_id property - https://bugs.launchpad.net/neutron/+bug/1835209 Issue on bulk port creation work Fix in progress: https://review.opendev.org/#/c/665516/ * flood flow in br-tun table22 incorrect - https://bugs.launchpad.net/neutron/+bug/1835163 Linked to https://bugs.launchpad.net/neutron/+bug/1834979 (see below), this can cause hard to trace l2pop and dhcp issues No owner Medium: * Add ipam.utils.check_gateway_invalid_in_subnet unit tests - https://bugs.launchpad.net/neutron/+bug/1835448 Shows proper usage after bug https://bugs.launchpad.net/neutron/+bug/1835344 (see in Opinion section) Review in progress: https://review.opendev.org/669210 * Port status becomes active after updating network/subnet - https://bugs.launchpad.net/neutron/+bug/1834979 The port of a turned off VM will show as UP after a network modification (reproducer with network name) No owner * Some L3 RPCs are time-consuming especially get_routers - https://bugs.launchpad.net/neutron/+bug/1835663 Probably worth discussing in performance meeting? Incomplete: * Restart dhcp-agent cause IPv6 vm can't renew IP address, will lost minutes or hours - https://bugs.launchpad.net/neutron/+bug/1835484 Cleaning IPv6 addresses from dnsmasq leases file ( https://bugs.launchpad.net/neutron/+bug/1722126) apparently causes VM to loose its address on renewal Waiting for detailed logs, but if ipv6 experts can chime in Opinion: * neutron doesn't check the validity of gateway_ip as a subnet had been created - https://bugs.launchpad.net/neutron/+bug/1835344 How to handle gateway IP outside of the subnet? Discussion in the bug and related review: https://review.opendev.org/669030 Duplicate: * QoS plugin slows down get_ports operation - https://bugs.launchpad.net/neutron/+bug/1835369 Was filled close to https://bugs.launchpad.net/bugs/1834484 Regards, -- Bernard Cafarelli -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcafarel at redhat.com Mon Jul 8 09:36:57 2019 From: bcafarel at redhat.com (Bernard Cafarelli) Date: Mon, 8 Jul 2019 11:36:57 +0200 Subject: [TripleO] Using TripleO standalone with ML2/OVS In-Reply-To: References: <8F8016EB-7898-4F66-BD30-998ABDB094FB@redhat.com> Message-ID: On Mon, 8 Jul 2019 at 08:11, Yatin Karel wrote: > Hi Slawek, > > So from workflow perspective you just need to pass an environment file > with correct set of parameters for ovs(resource_registry and > parameter_defaults) to "openstack tripleo deploy" command in the > end(to override the defaults) with -e . > > In CI multinode ovs jobs are running with environment files containing > ovs specific parameters. For Multinode ovs scenario the environment > file [1]. Standalone ovs work(creation of environment file and CI job) > is WIP by Marios[1]. I just rechecked the job to see where it's stuck > as last failures were unrelated to ovs. @Marios Andreou can share > exact status and @Slawomir Kaplonski you might also help in clearing > it. From job logs of WIP patch you can check command for reference[3]. > > You can use standalone environment files directly or use them as a > reference and create custom environment files as per your use case. > > I have not deployed myself standalone with ovs or any other backend so > can't tell exact parameters but if i try i will start with these > references until someone guides for exact set of parameters someone > uses for standlone ovs deployment. May be someone who have tried it > will post on this mailing list. > Not tested yet, but when the default was switched to OVN, ML2/OVS "default" environment files were updated and should work: https://github.com/openstack/tripleo-heat-templates/commit/6053eb196488a086449f5f2e4fe807825a16bd51#diff-6cac0d1de221a29d330377086cec8599 (before that switch, I know "/usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-standalone.yaml" was working fine to do an OVN standalone deployment) And if the neutron-ovs.yaml/ neutron-ovs-dvr.yaml files do not work out of the box, they sound worth fixing :) So you should be able to just add "-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml" to your openstack tripleo deploy command, and get a standalone ML2/OVS env. > [1] > https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario007-multinode-containers.yaml > [2] > https://review.opendev.org/#/q/topic:scenario007+(status:open+OR+status:merged) > [3] > http://logs.openstack.org/97/631497/18/check/tripleo-ci-centos-7-scenario007-standalone/a5ee2d3/logs/undercloud/home/zuul/standalone.sh.txt.gz > > Regards > Yatin Karel > > On Sat, Jul 6, 2019 at 2:33 PM Slawek Kaplonski > wrote: > > > > Hi, > > > > I was trying to use TripleO standalone for development work in the way > how Emilien described it in [1] and indeed it works quite well. Thx Emilien. > > But now, I’m trying to find out the way how to deploy it with Neutron > using ML2/OVS instead of default in TripleO ML2/OVN. > > And I still don’t know how to do it :/ > > I know it’s my fault but maybe someone can help me with this and tell me > what exactly options I should change there to deploy it with other Neutron > backend? > > Thx in advance for any help. > > > > [1] https://my1.fr/blog/developer-workflow-with-tripleo/ > > > > — > > Slawek Kaplonski > > Senior software engineer > > Red Hat > > > > > > -- Bernard Cafarelli -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Mon Jul 8 09:52:41 2019 From: hberaud at redhat.com (Herve Beraud) Date: Mon, 8 Jul 2019 11:52:41 +0200 Subject: [oslo] oslo.config /castellan poster review In-Reply-To: References: Message-ID: Hey Moises, Really interesting thanks! I'll take a look to your github POC. Le dim. 7 juil. 2019 à 16:22, Doug Hellmann a écrit : > Moises Guimaraes de Medeiros writes: > > > Hi, > > > > This week I'll be presenting a poster about oslo.config's castellan > driver > > at EuroPython. I'd like to ask y'all interested in the subject to take a > > look at my poster. I'm planning to print it this Tuesday and I still have > > some spare space to fit a bit more. > > > > The latest version is available at: > > > > > https://ep2019.europython.eu/media/conference/slides/m7RV4BB-protecting-secrets-with-osloconfig-and-hashicorp-vault.pdf > > > > Thanks a lot! > > > > -- > > > > Moisés Guimarães > > > > Software Engineer > > > > Red Hat > > > > > > That looks great, Moises, nice work! > > -- > Doug > > -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcafarel at redhat.com Mon Jul 8 10:20:26 2019 From: bcafarel at redhat.com (Bernard Cafarelli) Date: Mon, 8 Jul 2019 12:20:26 +0200 Subject: [neutron] Bug deputy report (week starting on 2019-07-01) In-Reply-To: References: Message-ID: Small late-morning update On Mon, 8 Jul 2019 at 11:16, Bernard Cafarelli wrote: > Hi Neutrinos, > > time for a new cycle of bug deputy rotation, which means I was on duty > last week, checking bugs up to 1835663 included > > Quite a few bugs in that list worth a read and further discussion: > * port status changing to UP when changing the network name (and possible > fallout on DHCP issues) > * API currently allowing to set gateway outside of the subnet > * How to handle external dependencies recommended version bumps (pyroute > bump in queens fixing a memory leak) > * possible issue in IPv6 address renewal since we started cleaning the > dnsmasq leases file > > Critical: > * Wrong endpoints config with configure_auth_token_middleware - > https://bugs.launchpad.net/bugs/1834849 > Removal of devstack deprecated option broke designate scenario job > Fix merged: https://review.opendev.org/668447 > > High: > * [Queens] Memory leak in pyroute2 0.4.21 - > https://bugs.launchpad.net/neutron/+bug/1835044 > we need a newer pyroute version > Patches on requirements with DNM neutron one for testing in progress: > https://review.opendev.org/668676 and https://review.opendev.org/668677 > openstack-discuss thread: > http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007545.html > Closed as Won't fix, requirements listed versions are for CI, not for packagers. On the packaging side, I know we have RDO fixes in progress, other packagers on Queens should find this bug relevant > * Bulk-created ports ignore binding_host_id property - > https://bugs.launchpad.net/neutron/+bug/1835209 > Issue on bulk port creation work > Fix in progress: https://review.opendev.org/#/c/665516/ > * flood flow in br-tun table22 incorrect - > https://bugs.launchpad.net/neutron/+bug/1835163 > Linked to https://bugs.launchpad.net/neutron/+bug/1834979 (see below), > this can cause hard to trace l2pop and dhcp issues > No owner > > Medium: > * Add ipam.utils.check_gateway_invalid_in_subnet unit tests - > https://bugs.launchpad.net/neutron/+bug/1835448 > Shows proper usage after bug > https://bugs.launchpad.net/neutron/+bug/1835344 (see in Opinion section) > Review in progress: https://review.opendev.org/669210 > * Port status becomes active after updating network/subnet - > https://bugs.launchpad.net/neutron/+bug/1834979 > The port of a turned off VM will show as UP after a network modification > (reproducer with network name) > No owner > * Some L3 RPCs are time-consuming especially get_routers - > https://bugs.launchpad.net/neutron/+bug/1835663 > Probably worth discussing in performance meeting? > > Incomplete: > * Restart dhcp-agent cause IPv6 vm can't renew IP address, will lost > minutes or hours - https://bugs.launchpad.net/neutron/+bug/1835484 > Cleaning IPv6 addresses from dnsmasq leases file ( > https://bugs.launchpad.net/neutron/+bug/1722126) apparently causes VM to > loose its address on renewal > Waiting for detailed logs, but if ipv6 experts can chime in > > Opinion: > * neutron doesn't check the validity of gateway_ip as a subnet had been > created - https://bugs.launchpad.net/neutron/+bug/1835344 > How to handle gateway IP outside of the subnet? > Discussion in the bug and related review: > https://review.opendev.org/669030 > > Duplicate: > * QoS plugin slows down get_ports operation - > https://bugs.launchpad.net/neutron/+bug/1835369 > Was filled close to https://bugs.launchpad.net/bugs/1834484 > > Regards, > -- > Bernard Cafarelli > -- Bernard Cafarelli -------------- next part -------------- An HTML attachment was scrubbed... URL: From jyotishri403 at gmail.com Mon Jul 8 10:46:17 2019 From: jyotishri403 at gmail.com (Jyoti Dahiwele) Date: Mon, 8 Jul 2019 16:16:17 +0530 Subject: Glusterfs support in stein Message-ID: Dear team, Do stein support glusterfs backend in cinder configuration? If yes plz share configuration document. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aj at suse.com Mon Jul 8 10:50:32 2019 From: aj at suse.com (Andreas Jaeger) Date: Mon, 8 Jul 2019 12:50:32 +0200 Subject: [docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos In-Reply-To: References: Message-ID: <8a2ec881-29d7-7d42-0400-5371e528fa8b@suse.com> On 06/07/2019 19.38, Mohammed Naser wrote: > [...] > [OpenStack Charms] openstack/charm-ceph has extra files, please > remove: .gitignore I would just whitelist this and be fine with it. The cost of adding this (2 changes to project-config, manual infra-root involvement to enable ACLs again plus one change to the repo) is not worth this single file. The README looks fine, this file is just cosmetics. Note: For repos that have full content, I would do those steps but not for just an extra .gitignore, Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From massimo.sgaravatto at gmail.com Mon Jul 8 12:13:13 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Mon, 8 Jul 2019 14:13:13 +0200 Subject: Glusterfs support in stein In-Reply-To: References: Message-ID: We also used it in the past but as far as I remember the gluster driver was removed in Ocata (we had therefore to migrate to the NFSdriver) Cheers, Massimo On Mon, Jul 8, 2019 at 12:50 PM Jyoti Dahiwele wrote: > Dear team, > > Do stein support glusterfs backend in cinder configuration? > If yes plz share configuration document. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.mcginnis at gmx.com Mon Jul 8 12:51:19 2019 From: sean.mcginnis at gmx.com (Sean McGinnis) Date: Mon, 8 Jul 2019 07:51:19 -0500 Subject: [cinder] Glusterfs support in stein In-Reply-To: References: Message-ID: <20190708125119.GA15668@sm-workstation> On Mon, Jul 08, 2019 at 02:13:13PM +0200, Massimo Sgaravatto wrote: > We also used it in the past but as far as I remember the gluster driver was > removed in Ocata (we had therefore to migrate to the NFSdriver) > > Cheers, Massimo > This is correct. The GlusterFS driver was marked as deprecated by its maintainers in the Newton release, then officially removed in Ocata. The driver can still be found in those stable branches, but would probably take a not insignificant amount of work to use in later releases. Sean From thierry at openstack.org Mon Jul 8 12:54:53 2019 From: thierry at openstack.org (Thierry Carrez) Date: Mon, 8 Jul 2019 14:54:53 +0200 Subject: [tripleo][charms][helm][kolla][ansible][puppet][chef] Deployment tools capabilities v0.1.0 Message-ID: Hi, deployment tools teams, As mentioned here last month[1], we are working to improve the information present on the deployment tools pages on the OpenStack website, and we need your help! After the Forum session on this topic in Denver[2], a workgroup worked on producing a set of base capabilities that can be asserted by the various deployment tools we have. You can find version 0.1.0 of those capabilities here: https://opendev.org/osf/openstack-map/src/branch/master/deployment_tools_capabilities.yaml As an example, I pushed a change that makes every deployment tool assert the capability to deploy keystone ("components:keystone" tag) at: https://review.opendev.org/#/c/669648/ Now it's your turn. Please have a look at the list of the capabilities above, and propose a change to add those that are relevant to your deployment tool in the following file: https://opendev.org/osf/openstack-map/src/branch/master/deployment_tools.yaml Capabilities are all of the form "category:tag" (components:keystone, starts-from:os-installed, technology:puppet...). Once all deployment projects have completed that task, we'll add the capabilities to the rendered page on the website and allow for basic searching for tools with matching capability. Now, capabilities go only so far in describing your deployment tool. I also encourage you to improve in the same file the "desc" field: that one is directly displayed on the site.Uuse it to describe in more details how your deployment tool actually works and what makes it unique, beyond basic capabilities tags. Please feel free to use this thread (or personal email) if you have questions on this. And thanks in advance for your help! [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006964.html [2] https://etherpad.openstack.org/p/DEN-deployment-tools-capabilities -- Thierry Carrez (ttx) From vungoctan252 at gmail.com Mon Jul 8 08:24:16 2019 From: vungoctan252 at gmail.com (Vu Tan) Date: Mon, 8 Jul 2019 15:24:16 +0700 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: <09a3849b-786e-49ed-a197-5e13af0428bf@email.android.com> References: <09a3849b-786e-49ed-a197-5e13af0428bf@email.android.com> Message-ID: Hi, Thanks a lot for your reply, I install pacemaker/corosync, masakari-api, maskari-engine on controller node, and I run masakari-api with this command: masakari-api, but I dont know whether the process is running like that or is it just hang there, here is what it shows when I run the command, I leave it there for a while but it does not change anything : [root at controller masakari]# masakari-api 2019-07-08 15:21:09.946 30250 INFO masakari.api.openstack [-] Loaded extensions: ['extensions', 'notifications', 'os-hosts', 'segments', 'versions'] 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config [-] The option "__file__" in conf is not known to auth_token 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config [-] The option "here" in conf is not known to auth_token 2019-07-08 15:21:09.960 30250 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. 2019-07-08 15:21:09.974 30250 INFO masakari.wsgi [-] masakari_api listening on 127.0.0.1:15868 2019-07-08 15:21:09.975 30250 INFO oslo_service.service [-] Starting 4 workers 2019-07-08 15:21:09.984 30274 INFO masakari.masakari_api.wsgi.server [-] (30274) wsgi starting up on http://127.0.0.1:15868 2019-07-08 15:21:09.985 30275 INFO masakari.masakari_api.wsgi.server [-] (30275) wsgi starting up on http://127.0.0.1:15868 2019-07-08 15:21:09.992 30277 INFO masakari.masakari_api.wsgi.server [-] (30277) wsgi starting up on http://127.0.0.1:15868 2019-07-08 15:21:09.994 30276 INFO masakari.masakari_api.wsgi.server [-] (30276) wsgi starting up on http://127.0.0.1:15868 On Sun, Jul 7, 2019 at 7:37 PM Gaëtan Trellu wrote: > Hi Vu Tan, > > Masakari documentation doesn't really exist... I had to figured some stuff > by myself to make it works into Kolla project. > > On controller nodes you need: > > - pacemaker > - corosync > - masakari-api (openstack/masakari repository) > - masakari- engine (openstack/masakari repository) > > On compute nodes you need: > > - pacemaker-remote (integrated to pacemaker cluster as a resource) > - masakari- hostmonitor (openstack/masakari-monitor repository) > - masakari-instancemonitor (openstack/masakari-monitor repository) > - masakari-processmonitor (openstack/masakari-monitor repository) > > For masakari-hostmonitor, the service needs to have access to systemctl > command (make sure you are not using sysvinit). > > For masakari-monitor, the masakari-monitor.conf is a bit different, you > will have to configure the [api] section properly. > > RabbitMQ needs to be configured (as transport_url) on masakari-api and > masakari-engine too. > > Please check this review[1], you will have masakari.conf and > masakari-monitor.conf configuration examples. > > [1] https://review.opendev.org/#/c/615715 > > Gaëtan > > On Jul 7, 2019 12:08 AM, Vu Tan wrote: > > > Vu Tan > 10:30 AM (35 minutes ago) > to openstack-discuss > Sorry, I resend this email because I realized that I lacked of prefix on > this email's subject > > > Hi, > > I would like to use Masakari and I'm having trouble finding a step by step > or other documentation to get started with. Which part should be installed > on controller, which is should be on compute, and what is the prerequisite > to install masakari, I have installed corosync and pacemaker on compute and > controller nodes, , what else do I need to do ? step I have done so far: > - installed corosync/pacemaker > - install masakari on compute node on this github repo: > https://github.com/openstack/masakari > - add masakari in to mariadb > here is my configuration file of masakari.conf, do you mind to take a look > at it, if I have misconfigured anything? > > [DEFAULT] > enabled_apis = masakari_api > > # Enable to specify listening IP other than default > masakari_api_listen = controller > # Enable to specify port other than default > masakari_api_listen_port = 15868 > debug = False > auth_strategy=keystone > > [wsgi] > # The paste configuration file path > api_paste_config = /etc/masakari/api-paste.ini > > [keystone_authtoken] > www_authenticate_uri = http://controller:5000 > auth_url = http://controller:5000 > auth_type = password > project_domain_id = default > user_domain_id = default > project_name = service > username = masakari > password = P at ssword > > [database] > connection = mysql+pymysql://masakari:P at ssword@controller/masakari > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vungoctan252 at gmail.com Mon Jul 8 10:08:46 2019 From: vungoctan252 at gmail.com (Vu Tan) Date: Mon, 8 Jul 2019 17:08:46 +0700 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: References: <09a3849b-786e-49ed-a197-5e13af0428bf@email.android.com> Message-ID: Hi Gaetan, I try to generate config file by using this command tox -egenconfig on top level of masakari but the output is error, is this masakari still in beta version ? [root at compute1 masakari-monitors]# tox -egenconfig genconfig create: /root/masakari-monitors/.tox/genconfig ERROR: InterpreterNotFound: python3 _____________________________________________________________ summary ______________________________________________________________ ERROR: genconfig: InterpreterNotFound: python3 On Mon, Jul 8, 2019 at 3:24 PM Vu Tan wrote: > Hi, > Thanks a lot for your reply, I install pacemaker/corosync, masakari-api, > maskari-engine on controller node, and I run masakari-api with this > command: masakari-api, but I dont know whether the process is running like > that or is it just hang there, here is what it shows when I run the > command, I leave it there for a while but it does not change anything : > [root at controller masakari]# masakari-api > 2019-07-08 15:21:09.946 30250 INFO masakari.api.openstack [-] Loaded > extensions: ['extensions', 'notifications', 'os-hosts', 'segments', > 'versions'] > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > [-] The option "__file__" in conf is not known to auth_token > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > [-] The option "here" in conf is not known to auth_token > 2019-07-08 15:21:09.960 30250 WARNING keystonemiddleware.auth_token [-] > AuthToken middleware is set with > keystone_authtoken.service_token_roles_required set to False. This is > backwards compatible but deprecated behaviour. Please set this to True. > 2019-07-08 15:21:09.974 30250 INFO masakari.wsgi [-] masakari_api > listening on 127.0.0.1:15868 > 2019-07-08 15:21:09.975 30250 INFO oslo_service.service [-] Starting 4 > workers > 2019-07-08 15:21:09.984 30274 INFO masakari.masakari_api.wsgi.server [-] > (30274) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.985 30275 INFO masakari.masakari_api.wsgi.server [-] > (30275) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.992 30277 INFO masakari.masakari_api.wsgi.server [-] > (30277) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.994 30276 INFO masakari.masakari_api.wsgi.server [-] > (30276) wsgi starting up on http://127.0.0.1:15868 > > On Sun, Jul 7, 2019 at 7:37 PM Gaëtan Trellu > wrote: > >> Hi Vu Tan, >> >> Masakari documentation doesn't really exist... I had to figured some >> stuff by myself to make it works into Kolla project. >> >> On controller nodes you need: >> >> - pacemaker >> - corosync >> - masakari-api (openstack/masakari repository) >> - masakari- engine (openstack/masakari repository) >> >> On compute nodes you need: >> >> - pacemaker-remote (integrated to pacemaker cluster as a resource) >> - masakari- hostmonitor (openstack/masakari-monitor repository) >> - masakari-instancemonitor (openstack/masakari-monitor repository) >> - masakari-processmonitor (openstack/masakari-monitor repository) >> >> For masakari-hostmonitor, the service needs to have access to systemctl >> command (make sure you are not using sysvinit). >> >> For masakari-monitor, the masakari-monitor.conf is a bit different, you >> will have to configure the [api] section properly. >> >> RabbitMQ needs to be configured (as transport_url) on masakari-api and >> masakari-engine too. >> >> Please check this review[1], you will have masakari.conf and >> masakari-monitor.conf configuration examples. >> >> [1] https://review.opendev.org/#/c/615715 >> >> Gaëtan >> >> On Jul 7, 2019 12:08 AM, Vu Tan wrote: >> >> >> Vu Tan >> 10:30 AM (35 minutes ago) >> to openstack-discuss >> Sorry, I resend this email because I realized that I lacked of prefix on >> this email's subject >> >> >> Hi, >> >> I would like to use Masakari and I'm having trouble finding a step by >> step or other documentation to get started with. Which part should be >> installed on controller, which is should be on compute, and what is the >> prerequisite to install masakari, I have installed corosync and pacemaker >> on compute and controller nodes, , what else do I need to do ? step I >> have done so far: >> - installed corosync/pacemaker >> - install masakari on compute node on this github repo: >> https://github.com/openstack/masakari >> - add masakari in to mariadb >> here is my configuration file of masakari.conf, do you mind to take a >> look at it, if I have misconfigured anything? >> >> [DEFAULT] >> enabled_apis = masakari_api >> >> # Enable to specify listening IP other than default >> masakari_api_listen = controller >> # Enable to specify port other than default >> masakari_api_listen_port = 15868 >> debug = False >> auth_strategy=keystone >> >> [wsgi] >> # The paste configuration file path >> api_paste_config = /etc/masakari/api-paste.ini >> >> [keystone_authtoken] >> www_authenticate_uri = http://controller:5000 >> auth_url = http://controller:5000 >> auth_type = password >> project_domain_id = default >> user_domain_id = default >> project_name = service >> username = masakari >> password = P at ssword >> >> [database] >> connection = mysql+pymysql://masakari:P at ssword@controller/masakari >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetan.trellu at incloudus.com Mon Jul 8 14:14:55 2019 From: gaetan.trellu at incloudus.com (gaetan.trellu at incloudus.com) Date: Mon, 08 Jul 2019 10:14:55 -0400 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: References: <09a3849b-786e-49ed-a197-5e13af0428bf@email.android.com> Message-ID: <144114a2e83d8e8e30579ddb0ae39e59@incloudus.com> Vu Tan, About "auth_token" error, you need "os_privileged_user_*" options into your masakari.conf for the API. As mentioned previously please have a look here to have an example of configuration working (for me at least): - masakari.conf: https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari.conf.j2 - masakari-monitor.conf: https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari-monitors.conf.j2 About your tox issue make sure you have Python3 installed. Gaëtan On 2019-07-08 06:08, Vu Tan wrote: > Hi Gaetan, > I try to generate config file by using this command tox -egenconfig on > top level of masakari but the output is error, is this masakari still > in beta version ? > [root at compute1 masakari-monitors]# tox -egenconfig > genconfig create: /root/masakari-monitors/.tox/genconfig > ERROR: InterpreterNotFound: python3 > _____________________________________________________________ summary > ______________________________________________________________ > ERROR: genconfig: InterpreterNotFound: python3 > > On Mon, Jul 8, 2019 at 3:24 PM Vu Tan wrote: > Hi, > Thanks a lot for your reply, I install pacemaker/corosync, > masakari-api, maskari-engine on controller node, and I run masakari-api > with this command: masakari-api, but I dont know whether the process is > running like that or is it just hang there, here is what it shows when > I run the command, I leave it there for a while but it does not change > anything : > [root at controller masakari]# masakari-api > 2019-07-08 15:21:09.946 30250 INFO masakari.api.openstack [-] Loaded > extensions: ['extensions', 'notifications', 'os-hosts', 'segments', > 'versions'] > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > [-] The option "__file__" in conf is not known to auth_token > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > [-] The option "here" in conf is not known to auth_token > 2019-07-08 15:21:09.960 30250 WARNING keystonemiddleware.auth_token [-] > AuthToken middleware is set with > keystone_authtoken.service_token_roles_required set to False. This is > backwards compatible but deprecated behaviour. Please set this to True. > 2019-07-08 15:21:09.974 30250 INFO masakari.wsgi [-] masakari_api > listening on 127.0.0.1:15868 > 2019-07-08 15:21:09.975 30250 INFO oslo_service.service [-] Starting 4 > workers > 2019-07-08 15:21:09.984 30274 INFO masakari.masakari_api.wsgi.server > [-] (30274) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.985 30275 INFO masakari.masakari_api.wsgi.server > [-] (30275) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.992 30277 INFO masakari.masakari_api.wsgi.server > [-] (30277) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.994 30276 INFO masakari.masakari_api.wsgi.server > [-] (30276) wsgi starting up on http://127.0.0.1:15868 > > On Sun, Jul 7, 2019 at 7:37 PM Gaëtan Trellu > wrote: > > Hi Vu Tan, > > Masakari documentation doesn't really exist... I had to figured some > stuff by myself to make it works into Kolla project. > > On controller nodes you need: > > - pacemaker > - corosync > - masakari-api (openstack/masakari repository) > - masakari- engine (openstack/masakari repository) > > On compute nodes you need: > > - pacemaker-remote (integrated to pacemaker cluster as a resource) > - masakari- hostmonitor (openstack/masakari-monitor repository) > - masakari-instancemonitor (openstack/masakari-monitor repository) > - masakari-processmonitor (openstack/masakari-monitor repository) > > For masakari-hostmonitor, the service needs to have access to systemctl > command (make sure you are not using sysvinit). > > For masakari-monitor, the masakari-monitor.conf is a bit different, you > will have to configure the [api] section properly. > > RabbitMQ needs to be configured (as transport_url) on masakari-api and > masakari-engine too. > > Please check this review[1], you will have masakari.conf and > masakari-monitor.conf configuration examples. > > [1] https://review.opendev.org/#/c/615715 > > Gaëtan > > On Jul 7, 2019 12:08 AM, Vu Tan wrote: > > VU TAN > > 10:30 AM (35 minutes ago) > > to openstack-discuss > > Sorry, I resend this email because I realized that I lacked of prefix > on this email's subject > > Hi, > > I would like to use Masakari and I'm having trouble finding a step by > step or other documentation to get started with. Which part should be > installed on controller, which is should be on compute, and what is the > prerequisite to install masakari, I have installed corosync and > pacemaker on compute and controller nodes, , what else do I need to do > ? step I have done so far: > - installed corosync/pacemaker > - install masakari on compute node on this github repo: > https://github.com/openstack/masakari > - add masakari in to mariadb > here is my configuration file of masakari.conf, do you mind to take a > look at it, if I have misconfigured anything? > > [DEFAULT] > enabled_apis = masakari_api > > # Enable to specify listening IP other than default > masakari_api_listen = controller > # Enable to specify port other than default > masakari_api_listen_port = 15868 > debug = False > auth_strategy=keystone > > [wsgi] > # The paste configuration file path > api_paste_config = /etc/masakari/api-paste.ini > > [keystone_authtoken] > www_authenticate_uri = http://controller:5000 > auth_url = http://controller:5000 > auth_type = password > project_domain_id = default > user_domain_id = default > project_name = service > username = masakari > password = P at ssword > > [database] > connection = mysql+pymysql://masakari:P at ssword@controller/masakari -------------- next part -------------- A non-text attachment was scrubbed... Name: blocked.gif Type: image/gif Size: 118 bytes Desc: not available URL: From vungoctan252 at gmail.com Mon Jul 8 14:21:16 2019 From: vungoctan252 at gmail.com (Vu Tan) Date: Mon, 8 Jul 2019 21:21:16 +0700 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: <144114a2e83d8e8e30579ddb0ae39e59@incloudus.com> References: <09a3849b-786e-49ed-a197-5e13af0428bf@email.android.com> <144114a2e83d8e8e30579ddb0ae39e59@incloudus.com> Message-ID: Hi Gaetan, Thanks for pinpoint this out, silly me that did not notice the simple "error InterpreterNotFound: python3". Thanks a lot, I appreciate it On Mon, Jul 8, 2019 at 9:15 PM wrote: > Vu Tan, > > About "auth_token" error, you need "os_privileged_user_*" options into > your masakari.conf for the API. > As mentioned previously please have a look here to have an example of > configuration working (for me at least): > > - masakari.conf: > > https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari.conf.j2 > - masakari-monitor.conf: > > https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari-monitors.conf.j2 > > About your tox issue make sure you have Python3 installed. > > Gaëtan > > On 2019-07-08 06:08, Vu Tan wrote: > > > Hi Gaetan, > > I try to generate config file by using this command tox -egenconfig on > > top level of masakari but the output is error, is this masakari still > > in beta version ? > > [root at compute1 masakari-monitors]# tox -egenconfig > > genconfig create: /root/masakari-monitors/.tox/genconfig > > ERROR: InterpreterNotFound: python3 > > _____________________________________________________________ summary > > ______________________________________________________________ > > ERROR: genconfig: InterpreterNotFound: python3 > > > > On Mon, Jul 8, 2019 at 3:24 PM Vu Tan wrote: > > Hi, > > Thanks a lot for your reply, I install pacemaker/corosync, > > masakari-api, maskari-engine on controller node, and I run masakari-api > > with this command: masakari-api, but I dont know whether the process is > > running like that or is it just hang there, here is what it shows when > > I run the command, I leave it there for a while but it does not change > > anything : > > [root at controller masakari]# masakari-api > > 2019-07-08 15:21:09.946 30250 INFO masakari.api.openstack [-] Loaded > > extensions: ['extensions', 'notifications', 'os-hosts', 'segments', > > 'versions'] > > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > > [-] The option "__file__" in conf is not known to auth_token > > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > > [-] The option "here" in conf is not known to auth_token > > 2019-07-08 15:21:09.960 30250 WARNING keystonemiddleware.auth_token [-] > > AuthToken middleware is set with > > keystone_authtoken.service_token_roles_required set to False. This is > > backwards compatible but deprecated behaviour. Please set this to True. > > 2019-07-08 15:21:09.974 30250 INFO masakari.wsgi [-] masakari_api > > listening on 127.0.0.1:15868 > > 2019-07-08 15:21:09.975 30250 INFO oslo_service.service [-] Starting 4 > > workers > > 2019-07-08 15:21:09.984 30274 INFO masakari.masakari_api.wsgi.server > > [-] (30274) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.985 30275 INFO masakari.masakari_api.wsgi.server > > [-] (30275) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.992 30277 INFO masakari.masakari_api.wsgi.server > > [-] (30277) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.994 30276 INFO masakari.masakari_api.wsgi.server > > [-] (30276) wsgi starting up on http://127.0.0.1:15868 > > > > On Sun, Jul 7, 2019 at 7:37 PM Gaëtan Trellu > > wrote: > > > > Hi Vu Tan, > > > > Masakari documentation doesn't really exist... I had to figured some > > stuff by myself to make it works into Kolla project. > > > > On controller nodes you need: > > > > - pacemaker > > - corosync > > - masakari-api (openstack/masakari repository) > > - masakari- engine (openstack/masakari repository) > > > > On compute nodes you need: > > > > - pacemaker-remote (integrated to pacemaker cluster as a resource) > > - masakari- hostmonitor (openstack/masakari-monitor repository) > > - masakari-instancemonitor (openstack/masakari-monitor repository) > > - masakari-processmonitor (openstack/masakari-monitor repository) > > > > For masakari-hostmonitor, the service needs to have access to systemctl > > command (make sure you are not using sysvinit). > > > > For masakari-monitor, the masakari-monitor.conf is a bit different, you > > will have to configure the [api] section properly. > > > > RabbitMQ needs to be configured (as transport_url) on masakari-api and > > masakari-engine too. > > > > Please check this review[1], you will have masakari.conf and > > masakari-monitor.conf configuration examples. > > > > [1] https://review.opendev.org/#/c/615715 > > > > Gaëtan > > > > On Jul 7, 2019 12:08 AM, Vu Tan wrote: > > > > VU TAN > > > > 10:30 AM (35 minutes ago) > > > > to openstack-discuss > > > > Sorry, I resend this email because I realized that I lacked of prefix > > on this email's subject > > > > Hi, > > > > I would like to use Masakari and I'm having trouble finding a step by > > step or other documentation to get started with. Which part should be > > installed on controller, which is should be on compute, and what is the > > prerequisite to install masakari, I have installed corosync and > > pacemaker on compute and controller nodes, , what else do I need to do > > ? step I have done so far: > > - installed corosync/pacemaker > > - install masakari on compute node on this github repo: > > https://github.com/openstack/masakari > > - add masakari in to mariadb > > here is my configuration file of masakari.conf, do you mind to take a > > look at it, if I have misconfigured anything? > > > > [DEFAULT] > > enabled_apis = masakari_api > > > > # Enable to specify listening IP other than default > > masakari_api_listen = controller > > # Enable to specify port other than default > > masakari_api_listen_port = 15868 > > debug = False > > auth_strategy=keystone > > > > [wsgi] > > # The paste configuration file path > > api_paste_config = /etc/masakari/api-paste.ini > > > > [keystone_authtoken] > > www_authenticate_uri = http://controller:5000 > > auth_url = http://controller:5000 > > auth_type = password > > project_domain_id = default > > user_domain_id = default > > project_name = service > > username = masakari > > password = P at ssword > > > > [database] > > connection = mysql+pymysql://masakari:P at ssword@controller/masakari -------------- next part -------------- An HTML attachment was scrubbed... URL: From sfinucan at redhat.com Mon Jul 8 16:04:15 2019 From: sfinucan at redhat.com (Stephen Finucane) Date: Mon, 08 Jul 2019 17:04:15 +0100 Subject: [docs] Retire openstack/docs-specs Message-ID: <79c35e0e14cca630361b0e8caa434618cfce9d24.camel@redhat.com> Hey, I'm proposing retiring docs-specs [1][2]. This hasn't been used in some time since most of our documentation now lives in individual project repositories and any work that would span these multiple projects would likely exist as a community goal, documented in those relevant repositories. I do not intend for the existing content to be removed - it simply won't be possible to add new specs. Let me know if anyone has a reason not to do this. If not, we'll get a move on with this next week. Stephen [1] https://opendev.org/openstack/docs-specs [2] https://review.opendev.org/#/c/668853/ From aj at suse.com Mon Jul 8 17:18:20 2019 From: aj at suse.com (Andreas Jaeger) Date: Mon, 8 Jul 2019 19:18:20 +0200 Subject: [docs] Retire openstack/docs-specs In-Reply-To: <79c35e0e14cca630361b0e8caa434618cfce9d24.camel@redhat.com> References: <79c35e0e14cca630361b0e8caa434618cfce9d24.camel@redhat.com> Message-ID: On 08/07/2019 18.04, Stephen Finucane wrote: > Hey, > > I'm proposing retiring docs-specs [1][2]. This hasn't been used in some > time since most of our documentation now lives in individual project > repositories and any work that would span these multiple projects would > likely exist as a community goal, documented in those relevant > repositories. I do not intend for the existing content to be removed - > it simply won't be possible to add new specs. Retirement means removing the content from the repo, see the Infra manual. We can keep [1] online for sure... Andreas > > Let me know if anyone has a reason not to do this. If not, we'll get a > move on with this next week. > > Stephen > > [1] https://opendev.org/openstack/docs-specs > [2] https://review.opendev.org/#/c/668853/ > > > -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF: Nils Brauckmann, Felix Imendörffer, Enrica Angelone, HRB 247165 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF: Nils Brauckmann, Felix Imendörffer, Enrica Angelone, HRB 247165 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From openstack at nemebean.com Mon Jul 8 20:29:22 2019 From: openstack at nemebean.com (Ben Nemec) Date: Mon, 8 Jul 2019 15:29:22 -0500 Subject: [oslo] oslo.config /castellan poster review In-Reply-To: References: Message-ID: <0a525f18-d457-5d06-8e22-ebae840e37b6@nemebean.com> On 7/7/19 9:21 AM, Doug Hellmann wrote: > Moises Guimaraes de Medeiros writes: > >> Hi, >> >> This week I'll be presenting a poster about oslo.config's castellan driver >> at EuroPython. I'd like to ask y'all interested in the subject to take a >> look at my poster. I'm planning to print it this Tuesday and I still have >> some spare space to fit a bit more. >> >> The latest version is available at: >> >> https://ep2019.europython.eu/media/conference/slides/m7RV4BB-protecting-secrets-with-osloconfig-and-hashicorp-vault.pdf >> >> Thanks a lot! >> >> -- >> >> Moisés Guimarães >> >> Software Engineer >> >> Red Hat >> >> > > That looks great, Moises, nice work! > +1. Thanks for putting it together! From colleen at gazlene.net Mon Jul 8 20:31:38 2019 From: colleen at gazlene.net (Colleen Murphy) Date: Mon, 08 Jul 2019 13:31:38 -0700 Subject: [keystone] Virtual Midcycle Planning In-Reply-To: <5480a911-8beb-46de-a326-fe5eea6802e5@www.fastmail.com> References: <5480a911-8beb-46de-a326-fe5eea6802e5@www.fastmail.com> Message-ID: On Tue, Jun 25, 2019, at 13:30, Colleen Murphy wrote: > Hi team, > > As discussed in today's meeting, we will be having a virtual midcycle > some time around milestone 2. We'll do two days with one three-hour > session (with breaks) each day. We will do this over a video conference > session, details of how to join will follow closer to the event. > > I've started a brainstorming etherpad: > > https://etherpad.openstack.org/p/keystone-train-midcycle-topics > > Please add discussion topics or hacking ideas to the etherpad and I > will try to sort them. > > We need to decide on when exactly to hold the midcycle. I've created a > doodle poll: > > https://doodle.com/poll/wr7ct4uhpw82sysg > > Please select times and days that you're available and then we'll try > to schedule two back-to-back days (or at least two days in the same > week) for the midcycle. > > Let me know if you have any questions or concerns. > > Colleen > > There were a few top contenders in the poll, we'll go with the following two days: Monday, July 22, 14:00-17:00 UTC Tuesday, July 23, 14:00-17:00 UTC (we will skip the weekly team meeting that would have been at 16:00) Agenda still TBD, will be posted on the topics etherpad[1]. [1] https://etherpad.openstack.org/p/keystone-train-midcycle-topics Colleen From rosmaita.fossdev at gmail.com Mon Jul 8 22:44:39 2019 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Mon, 8 Jul 2019 18:44:39 -0400 Subject: [cinder][stable-maint] July releases from the stable branches Message-ID: I've posted release patches for the stable branches: https://review.openstack.org/#/q/topic:cinderproject-july-2019 Notes on what is/isn't being released are on the etherpad: https://etherpad.openstack.org/p/cinder-releases-tracking I have a question about the semver for the cinder stable/stein release; it's noted on the patch: https://review.opendev.org/669771 cheers, brian From Kevin.Fox at pnnl.gov Tue Jul 9 00:17:47 2019 From: Kevin.Fox at pnnl.gov (Fox, Kevin M) Date: Tue, 9 Jul 2019 00:17:47 +0000 Subject: [Neutron] NUMA aware VxLAN In-Reply-To: <2EE296D083DF2940BF4EBB91D39BB89F40CC0832@SHSMSX104.ccr.corp.intel.com> References: <2EE296D083DF2940BF4EBB91D39BB89F40CC0832@SHSMSX104.ccr.corp.intel.com> Message-ID: <1A3C52DFCD06494D8528644858247BF01C3A2651@EX10MBOX03.pnnl.gov> I'm curious. A lot of network cards support offloaded vxlan traffic these days so the processor isn't doing much work. Is this issue really a problem? Thanks, Kevin ________________________________ From: Guo, Ruijing [ruijing.guo at intel.com] Sent: Sunday, July 07, 2019 5:47 PM To: openstack-dev at lists.openstack.org; openstack at lists.openstack.org Subject: [Neutron] NUMA aware VxLAN Hi, Existing neutron ML2 support one VxLAN for tenant network. In NUMA case, VM 0 can be created in node 0 and VM 1 can be created in node 1 and VxLAN is in node 0. VM1 need to cross node, which cause some performance downgrade. Does someone have this performance issue? Does Neutron community have plan to enhance it? Thanks, -Ruijing -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Tue Jul 9 01:14:29 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 09 Jul 2019 10:14:29 +0900 Subject: [docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos In-Reply-To: References: Message-ID: <16bd44bd9de.c4394238317337.7822051836860954924@ghanshyammann.com> ---- On Mon, 08 Jul 2019 17:46:37 +0900 Lajos Katona wrote ---- > Hi, > For networking-l2gw (and networking-l2gw-tempest-plugin) I can say that it is maintained (with not that great activity, but anyway far from rerirement).Is there anything I have to do to make this "activeness" visible from governance perspective? Lajos, networking-l2gw* had been retired around ~3 years back[1]. If networking-l2gw situation is active and fulfils the neutron stadium criteria, you can propose it back after discussing with neutron team. [1] https://review.opendev.org/#/c/392010/ -gmann > Lajos > Bernard Cafarelli ezt írta (időpont: 2019. júl. 6., Szo, 22:05): > > > On Sat, 6 Jul 2019 at 19:44, Mohammed Naser wrote: > Hi everyone, > > One of the issue that we recently ran into was the fact that there was > some inconsistency about merging retirement of repositories inside > governance without the code being fully removed. > > In order to avoid this, I've made a change to our governance > repository which will enforce that no code exists in those retired > repositories, however, this has surfaced that some repositories were > retired with some stale files, some are smaller littler files, some > are entire projects still. > > I have compiled a list for every team, with the repos that are not > properly retired that have extra files (using this change which should > eventually +1 once we fix it all: https://review.opendev.org/669549) > > [Documentation] openstack/api-site has extra files, please remove: > .gitignore, .zuul.yaml, LICENSE, api-quick-start, api-ref, bindep.txt, > common, doc-tools-check-languages.conf, firstapp, > test-requirements.txt, tools, tox.ini, www > [Documentation] openstack/faafo has extra files, please remove: > .gitignore, CONTRIBUTING.rst, LICENSE, Vagrantfile, bin, contrib, doc, > etc, faafo, requirements.txt, setup.cfg, setup.py, > test-requirements.txt, tox.ini > > [fuel] openstack/fuel-agent has extra files, please remove: > .gitignore, LICENSE, MAINTAINERS, cloud-init-templates, contrib, > debian, etc, fuel_agent, requirements.txt, run_tests.sh, setup.cfg, > setup.py, specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-astute has extra files, please remove: > .gitignore, .rspec, .ruby-version, Gemfile, LICENSE, MAINTAINERS, > Rakefile, astute.gemspec, astute.service, astute.sysconfig, bin, > bindep.txt, debian, examples, lib, mcagents, run_tests.sh, spec, > specs, tests > [fuel] openstack/fuel-library has extra files, please remove: > .gitignore, CHANGELOG, Gemfile, LICENSE, MAINTAINERS, Rakefile, > debian, deployment, files, graphs, logs, specs, tests, utils > [fuel] openstack/fuel-main has extra files, please remove: .gitignore, > 00-debmirror.patch, LICENSE, MAINTAINERS, Makefile, config.mk, > fuel-release, iso, mirror, packages, prepare-build-env.sh, > report-changelog.sh, repos.mk, requirements-fuel-rpm.txt, > requirements-rpm.txt, rules.mk, sandbox.mk, specs > [fuel] openstack/fuel-menu has extra files, please remove: .gitignore, > MAINTAINERS, MANIFEST.in, fuelmenu, run_tests.sh, setup.py, specs, > test-requirements.txt, tox.ini > [fuel] openstack/fuel-mirror has extra files, please remove: > .gitignore, .mailmap, MAINTAINERS, perestroika, tox.ini > [fuel] openstack/fuel-nailgun-agent has extra files, please remove: > .gitignore, Gemfile, LICENSE, MAINTAINERS, Rakefile, agent, debian, > nailgun-agent.cron, nailgun-agent.gemspec, run_tests.sh, specs > [fuel] openstack/fuel-ostf has extra files, please remove: .gitignore, > LICENSE, MAINTAINERS, MANIFEST.in, etc, fuel_health, fuel_plugin, > ostf.service, pylintrc, requirements.txt, run_tests.sh, setup.cfg, > setup.py, specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-qa has extra files, please remove: .coveragerc, > .gitignore, .pylintrc, .pylintrc_gerrit, MAINTAINERS, core, doc, > fuel_tests, fuelweb_test, gates_tests, packages_tests, pytest.ini, > run_system_test.py, run_tests.sh, system_test, tox.ini, utils > [fuel] openstack/fuel-ui has extra files, please remove: > .eslintignore, .eslintrc.yaml, .gitignore, LICENSE, MAINTAINERS, > fixtures, gulp, gulpfile.js, karma.config.js, npm-shrinkwrap.json, > package.json, run_real_plugin_tests.sh, > run_real_plugin_tests_on_real_nailgun.sh, run_ui_func_tests.sh, specs, > static, webpack.config.js > [fuel] openstack/fuel-virtualbox has extra files, please remove: > .gitignore, MAINTAINERS, actions, clean.sh, config.sh, contrib, > drivers, dumpkeys.cache, functions, iso, launch.sh, launch_16GB.sh, > launch_8GB.sh > [fuel] openstack/fuel-web has extra files, please remove: .gitignore, > LICENSE, MAINTAINERS, bin, build_docs.sh, debian, docs, nailgun, > run_tests.sh, specs, systemd, tools, tox.ini > [fuel] openstack/shotgun has extra files, please remove: .coveragerc, > .gitignore, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, > MAINTAINERS, MANIFEST.in, bin, etc, requirements.txt, setup.cfg, > setup.py, shotgun, specs, test-requirements.txt, tox.ini > [fuel] openstack/fuel-dev-tools has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, > HACKING.rst, LICENSE, MAINTAINERS, babel.cfg, contrib, doc, > fuel_dev_tools, openstack-common.conf, requirements.txt, setup.cfg, > setup.py, test-requirements.txt, tox.ini, vagrant > [fuel] openstack/fuel-devops has extra files, please remove: > .coveragerc, .gitignore, .pylintrc, .pylintrc_gerrit, LICENSE, > MAINTAINERS, bin, devops, doc, run_tests.sh, samples, setup.cfg, > setup.py, test-requirements.txt, tox.ini > [fuel] openstack/fuel-docs has extra files, please remove: .gitignore, > Makefile, _images, _templates, common_conf.py, conf.py, devdocs, > examples, glossary, index.rst, make.bat, plugindocs, requirements.txt, > setup.cfg, setup.py, tox.ini, userdocs > [fuel] openstack/fuel-nailgun-extension-cluster-upgrade has extra > files, please remove: .coveragerc, .gitignore, AUTHORS, LICENSE, > MANIFEST.in, bindep.txt, cluster_upgrade, conftest.py, > nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, > specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-nailgun-extension-iac has extra files, please > remove: .gitignore, LICENSE, MANIFEST.in, doc, fuel_external_git, > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > tools, tox.ini > [fuel] openstack/fuel-nailgun-extension-converted-serializers has > extra files, please remove: .coveragerc, .gitignore, LICENSE, > MANIFEST.in, bindep.txt, conftest.py, converted_serializers, > nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, > specs, test-requirements.txt, tools, tox.ini > [fuel] openstack/fuel-octane has extra files, please remove: > .coveragerc, .gitignore, .mailmap, Gemfile, Gemfile.lock, HACKING.rst, > LICENSE, MAINTAINERS, MANIFEST.in, Rakefile, bindep.txt, deploy, > deployment, docs, misc, octane, requirements.txt, setup.cfg, setup.py, > specs, test-requirements.txt, tox.ini > [fuel] openstack/fuel-upgrade has extra files, please remove: .gitignore > [fuel] openstack/tuning-box has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, > HACKING.rst, LICENSE, MAINTAINERS, MANIFEST.in, TODO, alembic.ini, > babel.cfg, bindep.txt, doc, examples, openstack-common.conf, > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > tools, tox.ini, tuning_box > [fuel] openstack/fuel-plugins has extra files, please remove: > .gitignore, CHANGELOG.md, CONTRIBUTING.rst, HACKING.rst, LICENSE, > MAINTAINERS, examples, fuel_plugin_builder, requirements.txt, > run_tests.sh, setup.cfg, setup.py, test-requirements.txt, tox.ini > [fuel] openstack/fuel-plugin-murano has extra files, please remove: > .gitignore, LICENSE, components.yaml, deployment_scripts, > deployment_tasks.yaml, docs, environment_config.yaml, functions.sh, > metadata.yaml, node_roles.yaml, pre_build_hook, releasenotes, > repositories, test-requirements.txt, tox.ini, volumes.yaml > [fuel] openstack/fuel-plugin-murano-tests has extra files, please > remove: .gitignore, murano_plugin_tests, openrc.default, > requirements.txt, tox.ini, utils > [fuel] openstack/fuel-specs has extra files, please remove: > .gitignore, .testr.conf, LICENSE, doc, images, policy, > requirements.txt, setup.cfg, setup.py, specs, tests, tools, tox.ini > [fuel] openstack/fuel-stats has extra files, please remove: > .gitignore, LICENSE, MAINTAINERS, MANIFEST.in, analytics, collector, > migration, requirements.txt, setup.py, test-requirements.txt, tools, > tox.ini > [fuel] openstack/python-fuelclient has extra files, please remove: > .gitignore, .testr.conf, MAINTAINERS, MANIFEST.in, fuelclient, > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > tools, tox.ini > > [Infrastructure] opendev/puppet-releasestatus has extra files, please > remove: .gitignore > > [ironic] openstack/python-dracclient has extra files, please remove: > .gitignore, CONTRIBUTING.rst, HACKING.rst, LICENSE, doc, dracclient, > requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini > > [neutron] openstack/networking-calico has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .testr.conf, .zuul.yaml, > CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, RELEASING.md, > babel.cfg, debian, devstack, doc, networking_calico, playbooks, > requirements.txt, rpm, setup.cfg, setup.py, test-requirements.txt, > tox.ini > [neutron] openstack/networking-l2gw has extra files, please remove: > .coveragerc, .gitignore, .testr.conf, .zuul.yaml, CONTRIBUTING.rst, > HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, bindep.txt, contrib, > debian, devstack, doc, etc, lower-constraints.txt, networking_l2gw, > openstack-common.conf, requirements.txt, setup.cfg, setup.py, specs, > test-requirements.txt, tools, tox.ini > [neutron] openstack/networking-l2gw-tempest-plugin has extra files, > please remove: .gitignore, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, > LICENSE, babel.cfg, contrib, doc, networking_l2gw_tempest_plugin, > requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini > [neutron] openstack/networking-onos has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .pylintrc, .testr.conf, > CONTRIBUTING.rst, HACKING.rst, LICENSE, PKG-INFO, TESTING.rst, > babel.cfg, devstack, doc, etc, lower-constraints.txt, networking_onos, > package, rally-jobs, releasenotes, requirements.txt, setup.cfg, > setup.py, test-requirements.txt, tools, tox.ini > [neutron] openstack/neutron-vpnaas has extra files, please remove: > .coveragerc, .gitignore, .mailmap, .pylintrc, .stestr.conf, > .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, TESTING.rst, > babel.cfg, devstack, doc, etc, lower-constraints.txt, neutron_vpnaas, > playbooks, rally-jobs, releasenotes, requirements.txt, setup.cfg, > setup.py, test-requirements.txt, tools, tox.ini > At least for networking-l2gw* and neutron-vpnaas, I suppose this was caused by:https://opendev.org/openstack/governance/commit/20f95dd947d2f87519b4bb50fb188e6f71deae7c > What it meant is that they are not anymore under neutron governance, but they were not retired (at least as far as I know).There were still some recent commits even if minimal activity, and discussion on team status for neutron-vpnaas. > Not sure about networking-calico status though > [OpenStack Charms] openstack/charm-ceph has extra files, please > remove: .gitignore > > [OpenStackAnsible] openstack/openstack-ansible-os_monasca has extra > files, please remove: tests, tox.ini > > [solum] openstack/solum-infra-guestagent has extra files, please > remove: .coveragerc, .gitignore, .mailmap, .testr.conf, > CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, > config-generator, doc, etc, requirements.txt, setup.cfg, setup.py, > solum_guestagent, test-requirements.txt, tox.ini > > I'd like to kindly ask the affected teams to help out with this, or > any member of our community is more than welcome to push a change to > those repos and work with the appropriate teams to help land it. > > Mohammed > > -- > Mohammed Naser — vexxhost > ----------------------------------------------------- > D. 514-316-8872 > D. 800-910-1726 ext. 200 > E. mnaser at vexxhost.com > W. http://vexxhost.com > > > > -- > Bernard Cafarelli > From Tushar.Patil at nttdata.com Tue Jul 9 01:18:03 2019 From: Tushar.Patil at nttdata.com (Patil, Tushar) Date: Tue, 9 Jul 2019 01:18:03 +0000 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: References: <09a3849b-786e-49ed-a197-5e13af0428bf@email.android.com> <144114a2e83d8e8e30579ddb0ae39e59@incloudus.com>, Message-ID: Hi Vu and Gaetan, Gaetan, thank you for helping out Vu in setting up masakari-monitors service. As a masakari team ,we have noticed there is a need to add proper documentation to help the community run Masakari services in their environment. We are working on adding proper documentation in this 'Train' cycle. Will send an email on this mailing list once the patches are uploaded on the gerrit so that you can give your feedback on the same. If you have any trouble in setting up Masakari, please let us know on this mailing list or join the bi-weekly IRC Masakari meeting on the #openstack-meeting IRC channel. The next meeting will be held on 16th July 2019 @0400 UTC. Regards, Tushar Patil ________________________________________ From: Vu Tan Sent: Monday, July 8, 2019 11:21:16 PM To: Gaëtan Trellu Cc: openstack-discuss at lists.openstack.org Subject: Re: [masakari] how to install masakari on centos 7 Hi Gaetan, Thanks for pinpoint this out, silly me that did not notice the simple "error InterpreterNotFound: python3". Thanks a lot, I appreciate it On Mon, Jul 8, 2019 at 9:15 PM > wrote: Vu Tan, About "auth_token" error, you need "os_privileged_user_*" options into your masakari.conf for the API. As mentioned previously please have a look here to have an example of configuration working (for me at least): - masakari.conf: https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari.conf.j2 - masakari-monitor.conf: https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari-monitors.conf.j2 About your tox issue make sure you have Python3 installed. Gaëtan On 2019-07-08 06:08, Vu Tan wrote: > Hi Gaetan, > I try to generate config file by using this command tox -egenconfig on > top level of masakari but the output is error, is this masakari still > in beta version ? > [root at compute1 masakari-monitors]# tox -egenconfig > genconfig create: /root/masakari-monitors/.tox/genconfig > ERROR: InterpreterNotFound: python3 > _____________________________________________________________ summary > ______________________________________________________________ > ERROR: genconfig: InterpreterNotFound: python3 > > On Mon, Jul 8, 2019 at 3:24 PM Vu Tan > wrote: > Hi, > Thanks a lot for your reply, I install pacemaker/corosync, > masakari-api, maskari-engine on controller node, and I run masakari-api > with this command: masakari-api, but I dont know whether the process is > running like that or is it just hang there, here is what it shows when > I run the command, I leave it there for a while but it does not change > anything : > [root at controller masakari]# masakari-api > 2019-07-08 15:21:09.946 30250 INFO masakari.api.openstack [-] Loaded > extensions: ['extensions', 'notifications', 'os-hosts', 'segments', > 'versions'] > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > [-] The option "__file__" in conf is not known to auth_token > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > [-] The option "here" in conf is not known to auth_token > 2019-07-08 15:21:09.960 30250 WARNING keystonemiddleware.auth_token [-] > AuthToken middleware is set with > keystone_authtoken.service_token_roles_required set to False. This is > backwards compatible but deprecated behaviour. Please set this to True. > 2019-07-08 15:21:09.974 30250 INFO masakari.wsgi [-] masakari_api > listening on 127.0.0.1:15868 > 2019-07-08 15:21:09.975 30250 INFO oslo_service.service [-] Starting 4 > workers > 2019-07-08 15:21:09.984 30274 INFO masakari.masakari_api.wsgi.server > [-] (30274) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.985 30275 INFO masakari.masakari_api.wsgi.server > [-] (30275) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.992 30277 INFO masakari.masakari_api.wsgi.server > [-] (30277) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.994 30276 INFO masakari.masakari_api.wsgi.server > [-] (30276) wsgi starting up on http://127.0.0.1:15868 > > On Sun, Jul 7, 2019 at 7:37 PM Gaëtan Trellu > > wrote: > > Hi Vu Tan, > > Masakari documentation doesn't really exist... I had to figured some > stuff by myself to make it works into Kolla project. > > On controller nodes you need: > > - pacemaker > - corosync > - masakari-api (openstack/masakari repository) > - masakari- engine (openstack/masakari repository) > > On compute nodes you need: > > - pacemaker-remote (integrated to pacemaker cluster as a resource) > - masakari- hostmonitor (openstack/masakari-monitor repository) > - masakari-instancemonitor (openstack/masakari-monitor repository) > - masakari-processmonitor (openstack/masakari-monitor repository) > > For masakari-hostmonitor, the service needs to have access to systemctl > command (make sure you are not using sysvinit). > > For masakari-monitor, the masakari-monitor.conf is a bit different, you > will have to configure the [api] section properly. > > RabbitMQ needs to be configured (as transport_url) on masakari-api and > masakari-engine too. > > Please check this review[1], you will have masakari.conf and > masakari-monitor.conf configuration examples. > > [1] https://review.opendev.org/#/c/615715 > > Gaëtan > > On Jul 7, 2019 12:08 AM, Vu Tan > wrote: > > VU TAN > > > 10:30 AM (35 minutes ago) > > to openstack-discuss > > Sorry, I resend this email because I realized that I lacked of prefix > on this email's subject > > Hi, > > I would like to use Masakari and I'm having trouble finding a step by > step or other documentation to get started with. Which part should be > installed on controller, which is should be on compute, and what is the > prerequisite to install masakari, I have installed corosync and > pacemaker on compute and controller nodes, , what else do I need to do > ? step I have done so far: > - installed corosync/pacemaker > - install masakari on compute node on this github repo: > https://github.com/openstack/masakari > - add masakari in to mariadb > here is my configuration file of masakari.conf, do you mind to take a > look at it, if I have misconfigured anything? > > [DEFAULT] > enabled_apis = masakari_api > > # Enable to specify listening IP other than default > masakari_api_listen = controller > # Enable to specify port other than default > masakari_api_listen_port = 15868 > debug = False > auth_strategy=keystone > > [wsgi] > # The paste configuration file path > api_paste_config = /etc/masakari/api-paste.ini > > [keystone_authtoken] > www_authenticate_uri = http://controller:5000 > auth_url = http://controller:5000 > auth_type = password > project_domain_id = default > user_domain_id = default > project_name = service > username = masakari > password = P at ssword > > [database] > connection = mysql+pymysql://masakari:P at ssword@controller/masakari Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. From smooney at redhat.com Tue Jul 9 01:43:35 2019 From: smooney at redhat.com (Sean Mooney) Date: Tue, 09 Jul 2019 02:43:35 +0100 Subject: [Neutron] NUMA aware VxLAN In-Reply-To: <1A3C52DFCD06494D8528644858247BF01C3A2651@EX10MBOX03.pnnl.gov> References: <2EE296D083DF2940BF4EBB91D39BB89F40CC0832@SHSMSX104.ccr.corp.intel.com> <1A3C52DFCD06494D8528644858247BF01C3A2651@EX10MBOX03.pnnl.gov> Message-ID: <3e50593dbc47186d482429affeec497c89fdfc0f.camel@redhat.com> On Tue, 2019-07-09 at 00:17 +0000, Fox, Kevin M wrote: > I'm curious. A lot of network cards support offloaded vxlan traffic these days so the processor isn't doing much work. > Is this issue really a problem? the issue is not really with the cpu overhead of tunnel decapsulation it is with the cross numa latency incurred when crossing the qpi bus so even with hardware accleration on the nic there is a perfomace degradation if you go across a numa nodes. the optimal solution is to have multiple nics, one per numa node and bond them then affinities the vms mac to the numa local bond peer such that no traffic has to travers the numa node but that is non trivaial to do and is not supported by openstack natively. you would have to have an agent of or cron job that actuly a did the tuning after a vm is spawned but it could be an interesting experiment if someone wanted to code it up. > > Thanks, > Kevin > ________________________________ > From: Guo, Ruijing [ruijing.guo at intel.com] > Sent: Sunday, July 07, 2019 5:47 PM > To: openstack-dev at lists.openstack.org; openstack at lists.openstack.org > Subject: [Neutron] NUMA aware VxLAN > > Hi, > > Existing neutron ML2 support one VxLAN for tenant network. In NUMA case, VM 0 can be created in node 0 and VM 1 can be > created in node 1 and VxLAN is in node 0. > > VM1 need to cross node, which cause some performance downgrade. Does someone have this performance issue? Does Neutron > community have plan to enhance it? nova has a spec called numa aware vswitchs https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/numa-aware-vswitches.html that allow you to declare teh numa affinity of tunnels and physnets on a host. this will not allow you to have multiple tunnel enpoint ips but it will allow you force instnace with a numa toploty to be colocated on the same numa node as the network backend. > Thanks, > -Ruijing From smooney at redhat.com Tue Jul 9 01:43:35 2019 From: smooney at redhat.com (Sean Mooney) Date: Tue, 09 Jul 2019 02:43:35 +0100 Subject: [Neutron] NUMA aware VxLAN In-Reply-To: <1A3C52DFCD06494D8528644858247BF01C3A2651@EX10MBOX03.pnnl.gov> References: <2EE296D083DF2940BF4EBB91D39BB89F40CC0832@SHSMSX104.ccr.corp.intel.com> <1A3C52DFCD06494D8528644858247BF01C3A2651@EX10MBOX03.pnnl.gov> Message-ID: <3e50593dbc47186d482429affeec497c89fdfc0f.camel@redhat.com> On Tue, 2019-07-09 at 00:17 +0000, Fox, Kevin M wrote: > I'm curious. A lot of network cards support offloaded vxlan traffic these days so the processor isn't doing much work. > Is this issue really a problem? the issue is not really with the cpu overhead of tunnel decapsulation it is with the cross numa latency incurred when crossing the qpi bus so even with hardware accleration on the nic there is a perfomace degradation if you go across a numa nodes. the optimal solution is to have multiple nics, one per numa node and bond them then affinities the vms mac to the numa local bond peer such that no traffic has to travers the numa node but that is non trivaial to do and is not supported by openstack natively. you would have to have an agent of or cron job that actuly a did the tuning after a vm is spawned but it could be an interesting experiment if someone wanted to code it up. > > Thanks, > Kevin > ________________________________ > From: Guo, Ruijing [ruijing.guo at intel.com] > Sent: Sunday, July 07, 2019 5:47 PM > To: openstack-dev at lists.openstack.org; openstack at lists.openstack.org > Subject: [Neutron] NUMA aware VxLAN > > Hi, > > Existing neutron ML2 support one VxLAN for tenant network. In NUMA case, VM 0 can be created in node 0 and VM 1 can be > created in node 1 and VxLAN is in node 0. > > VM1 need to cross node, which cause some performance downgrade. Does someone have this performance issue? Does Neutron > community have plan to enhance it? nova has a spec called numa aware vswitchs https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/numa-aware-vswitches.html that allow you to declare teh numa affinity of tunnels and physnets on a host. this will not allow you to have multiple tunnel enpoint ips but it will allow you force instnace with a numa toploty to be colocated on the same numa node as the network backend. > Thanks, > -Ruijing From satish.txt at gmail.com Tue Jul 9 02:42:45 2019 From: satish.txt at gmail.com (Satish Patel) Date: Mon, 8 Jul 2019 22:42:45 -0400 Subject: neutron netns arp issue Message-ID: Hello, I am deploying openstack-ansible with octavia and i can see neutron created network for lb-mgmt-net which also created dhcp namespace for that network which is in vlan27 so far everything good so for testing i have created vm and it didn't get IP address so i have started troubleshooting and i found my namespace sending arp request but not getting reply back. [root at ostack-infra-2-2 ~]# ip netns exec qdhcp-2b94d9df-dd49-45b5-a992-63fee27bfa77 ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ns-5604eec1-20 at if132: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:c2:b3:4d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.27.12.3/21 brd 172.27.15.255 scope global ns-5604eec1-20 valid_lft forever preferred_lft forever inet 169.254.169.254/16 brd 169.254.255.255 scope global ns-5604eec1-20 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fec2:b34d/64 scope link valid_lft forever preferred_lft forever This is my linuxbridge [root at ostack-infra-2-2 ~]# brctl show brq2b94d9df-dd bridge name bridge id STP enabled interfaces brq2b94d9df-dd 8000.16d25dbea2cc no br-vlan.27 tap5604eec1-20 on same controller node "ostack-infra-2-2" i have br-lbaas network which has same VLAN 27 subnet IP. now when i ping from dhcp-namespace to outside host on same vlan 27, i can see ARP going out and remote host replying back but my reply coming on br-lbaas interface. [root at ostack-infra-2-2 ~]# ip netns exec qdhcp-2b94d9df-dd49-45b5-a992-63fee27bfa77 ping 172.27.8.4 PING 172.27.8.4 (172.27.8.4) 56(84) bytes of data. >From 172.27.12.3 icmp_seq=1 Destination Host Unreachable >From 172.27.12.3 icmp_seq=2 Destination Host Unreachable on other terminal i am running tcpdump on br-lbaas and i am seeing remote host ARP reply coming on that interface but not going to br-vlan.27 which neutron created. [root at ostack-infra-2-2 network-scripts]# tcpdump -i br-lbaas -nn tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on br-lbaas, link-type EN10MB (Ethernet), capture size 262144 bytes 22:41:38.920858 ARP, Reply 172.27.8.4 is-at 32:7c:a1:91:79:7c, length 46 22:41:39.922167 ARP, Reply 172.27.8.4 is-at 32:7c:a1:91:79:7c, length 46 22:41:40.924052 ARP, Reply 172.27.8.4 is-at 32:7c:a1:91:79:7c, length 46 Do you think i can't create two same subnet bridge on same host? From amotoki at gmail.com Tue Jul 9 04:58:05 2019 From: amotoki at gmail.com (Akihiro Motoki) Date: Tue, 9 Jul 2019 13:58:05 +0900 Subject: [docs][fuel][infra][ironic][neutron][charms][openstack-ansible][solum][tc] proper retirement of repos In-Reply-To: <16bd44bd9de.c4394238317337.7822051836860954924@ghanshyammann.com> References: <16bd44bd9de.c4394238317337.7822051836860954924@ghanshyammann.com> Message-ID: Regarding networking repositories, the neutron team evaluates activities of individual projects and include/push out them from the neutron governance depending on their situations. Thus, retirement from some repository does not mean the retirement of the repository. Development on such repository can continue after exclusion from the neutron governance. AFAIK, in case of networking-l2gw, the development is still active but it does not satisfy the criteria of the neutron stadium like integrated tests or community goal satisfaction due to the lack of development resources. This is the reason that networking-l2gw *repository* is not retired as TC expects although it is not under the TC governance. IMHO it looks better to move networking-l2-gw repository from "openstack" namespace to "x" namespace. (Of course the other option is the neutron team consider the inclusion of networking-l2gw again but it would take time even if we go to this route.) Thanks, Akihiro Motoki (irc: amotoki) On Tue, Jul 9, 2019 at 10:19 AM Ghanshyam Mann wrote: > > ---- On Mon, 08 Jul 2019 17:46:37 +0900 Lajos Katona wrote ---- > > Hi, > > For networking-l2gw (and networking-l2gw-tempest-plugin) I can say that it is maintained (with not that great activity, but anyway far from rerirement).Is there anything I have to do to make this "activeness" visible from governance perspective? > > Lajos, > networking-l2gw* had been retired around ~3 years back[1]. If networking-l2gw situation is active and fulfils the neutron stadium criteria, you can propose it back after discussing with neutron team. > > [1] https://review.opendev.org/#/c/392010/ > > -gmann > > > Lajos > > Bernard Cafarelli ezt írta (időpont: 2019. júl. 6., Szo, 22:05): > > > > > > On Sat, 6 Jul 2019 at 19:44, Mohammed Naser wrote: > > Hi everyone, > > > > One of the issue that we recently ran into was the fact that there was > > some inconsistency about merging retirement of repositories inside > > governance without the code being fully removed. > > > > In order to avoid this, I've made a change to our governance > > repository which will enforce that no code exists in those retired > > repositories, however, this has surfaced that some repositories were > > retired with some stale files, some are smaller littler files, some > > are entire projects still. > > > > I have compiled a list for every team, with the repos that are not > > properly retired that have extra files (using this change which should > > eventually +1 once we fix it all: https://review.opendev.org/669549) > > > > [Documentation] openstack/api-site has extra files, please remove: > > .gitignore, .zuul.yaml, LICENSE, api-quick-start, api-ref, bindep.txt, > > common, doc-tools-check-languages.conf, firstapp, > > test-requirements.txt, tools, tox.ini, www > > [Documentation] openstack/faafo has extra files, please remove: > > .gitignore, CONTRIBUTING.rst, LICENSE, Vagrantfile, bin, contrib, doc, > > etc, faafo, requirements.txt, setup.cfg, setup.py, > > test-requirements.txt, tox.ini > > > > [fuel] openstack/fuel-agent has extra files, please remove: > > .gitignore, LICENSE, MAINTAINERS, cloud-init-templates, contrib, > > debian, etc, fuel_agent, requirements.txt, run_tests.sh, setup.cfg, > > setup.py, specs, test-requirements.txt, tools, tox.ini > > [fuel] openstack/fuel-astute has extra files, please remove: > > .gitignore, .rspec, .ruby-version, Gemfile, LICENSE, MAINTAINERS, > > Rakefile, astute.gemspec, astute.service, astute.sysconfig, bin, > > bindep.txt, debian, examples, lib, mcagents, run_tests.sh, spec, > > specs, tests > > [fuel] openstack/fuel-library has extra files, please remove: > > .gitignore, CHANGELOG, Gemfile, LICENSE, MAINTAINERS, Rakefile, > > debian, deployment, files, graphs, logs, specs, tests, utils > > [fuel] openstack/fuel-main has extra files, please remove: .gitignore, > > 00-debmirror.patch, LICENSE, MAINTAINERS, Makefile, config.mk, > > fuel-release, iso, mirror, packages, prepare-build-env.sh, > > report-changelog.sh, repos.mk, requirements-fuel-rpm.txt, > > requirements-rpm.txt, rules.mk, sandbox.mk, specs > > [fuel] openstack/fuel-menu has extra files, please remove: .gitignore, > > MAINTAINERS, MANIFEST.in, fuelmenu, run_tests.sh, setup.py, specs, > > test-requirements.txt, tox.ini > > [fuel] openstack/fuel-mirror has extra files, please remove: > > .gitignore, .mailmap, MAINTAINERS, perestroika, tox.ini > > [fuel] openstack/fuel-nailgun-agent has extra files, please remove: > > .gitignore, Gemfile, LICENSE, MAINTAINERS, Rakefile, agent, debian, > > nailgun-agent.cron, nailgun-agent.gemspec, run_tests.sh, specs > > [fuel] openstack/fuel-ostf has extra files, please remove: .gitignore, > > LICENSE, MAINTAINERS, MANIFEST.in, etc, fuel_health, fuel_plugin, > > ostf.service, pylintrc, requirements.txt, run_tests.sh, setup.cfg, > > setup.py, specs, test-requirements.txt, tools, tox.ini > > [fuel] openstack/fuel-qa has extra files, please remove: .coveragerc, > > .gitignore, .pylintrc, .pylintrc_gerrit, MAINTAINERS, core, doc, > > fuel_tests, fuelweb_test, gates_tests, packages_tests, pytest.ini, > > run_system_test.py, run_tests.sh, system_test, tox.ini, utils > > [fuel] openstack/fuel-ui has extra files, please remove: > > .eslintignore, .eslintrc.yaml, .gitignore, LICENSE, MAINTAINERS, > > fixtures, gulp, gulpfile.js, karma.config.js, npm-shrinkwrap.json, > > package.json, run_real_plugin_tests.sh, > > run_real_plugin_tests_on_real_nailgun.sh, run_ui_func_tests.sh, specs, > > static, webpack.config.js > > [fuel] openstack/fuel-virtualbox has extra files, please remove: > > .gitignore, MAINTAINERS, actions, clean.sh, config.sh, contrib, > > drivers, dumpkeys.cache, functions, iso, launch.sh, launch_16GB.sh, > > launch_8GB.sh > > [fuel] openstack/fuel-web has extra files, please remove: .gitignore, > > LICENSE, MAINTAINERS, bin, build_docs.sh, debian, docs, nailgun, > > run_tests.sh, specs, systemd, tools, tox.ini > > [fuel] openstack/shotgun has extra files, please remove: .coveragerc, > > .gitignore, .testr.conf, CONTRIBUTING.rst, HACKING.rst, LICENSE, > > MAINTAINERS, MANIFEST.in, bin, etc, requirements.txt, setup.cfg, > > setup.py, shotgun, specs, test-requirements.txt, tox.ini > > [fuel] openstack/fuel-dev-tools has extra files, please remove: > > .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, > > HACKING.rst, LICENSE, MAINTAINERS, babel.cfg, contrib, doc, > > fuel_dev_tools, openstack-common.conf, requirements.txt, setup.cfg, > > setup.py, test-requirements.txt, tox.ini, vagrant > > [fuel] openstack/fuel-devops has extra files, please remove: > > .coveragerc, .gitignore, .pylintrc, .pylintrc_gerrit, LICENSE, > > MAINTAINERS, bin, devops, doc, run_tests.sh, samples, setup.cfg, > > setup.py, test-requirements.txt, tox.ini > > [fuel] openstack/fuel-docs has extra files, please remove: .gitignore, > > Makefile, _images, _templates, common_conf.py, conf.py, devdocs, > > examples, glossary, index.rst, make.bat, plugindocs, requirements.txt, > > setup.cfg, setup.py, tox.ini, userdocs > > [fuel] openstack/fuel-nailgun-extension-cluster-upgrade has extra > > files, please remove: .coveragerc, .gitignore, AUTHORS, LICENSE, > > MANIFEST.in, bindep.txt, cluster_upgrade, conftest.py, > > nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, > > specs, test-requirements.txt, tools, tox.ini > > [fuel] openstack/fuel-nailgun-extension-iac has extra files, please > > remove: .gitignore, LICENSE, MANIFEST.in, doc, fuel_external_git, > > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > > tools, tox.ini > > [fuel] openstack/fuel-nailgun-extension-converted-serializers has > > extra files, please remove: .coveragerc, .gitignore, LICENSE, > > MANIFEST.in, bindep.txt, conftest.py, converted_serializers, > > nailgun-test-settings.yaml, requirements.txt, setup.cfg, setup.py, > > specs, test-requirements.txt, tools, tox.ini > > [fuel] openstack/fuel-octane has extra files, please remove: > > .coveragerc, .gitignore, .mailmap, Gemfile, Gemfile.lock, HACKING.rst, > > LICENSE, MAINTAINERS, MANIFEST.in, Rakefile, bindep.txt, deploy, > > deployment, docs, misc, octane, requirements.txt, setup.cfg, setup.py, > > specs, test-requirements.txt, tox.ini > > [fuel] openstack/fuel-upgrade has extra files, please remove: .gitignore > > [fuel] openstack/tuning-box has extra files, please remove: > > .coveragerc, .gitignore, .mailmap, .testr.conf, CONTRIBUTING.rst, > > HACKING.rst, LICENSE, MAINTAINERS, MANIFEST.in, TODO, alembic.ini, > > babel.cfg, bindep.txt, doc, examples, openstack-common.conf, > > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > > tools, tox.ini, tuning_box > > [fuel] openstack/fuel-plugins has extra files, please remove: > > .gitignore, CHANGELOG.md, CONTRIBUTING.rst, HACKING.rst, LICENSE, > > MAINTAINERS, examples, fuel_plugin_builder, requirements.txt, > > run_tests.sh, setup.cfg, setup.py, test-requirements.txt, tox.ini > > [fuel] openstack/fuel-plugin-murano has extra files, please remove: > > .gitignore, LICENSE, components.yaml, deployment_scripts, > > deployment_tasks.yaml, docs, environment_config.yaml, functions.sh, > > metadata.yaml, node_roles.yaml, pre_build_hook, releasenotes, > > repositories, test-requirements.txt, tox.ini, volumes.yaml > > [fuel] openstack/fuel-plugin-murano-tests has extra files, please > > remove: .gitignore, murano_plugin_tests, openrc.default, > > requirements.txt, tox.ini, utils > > [fuel] openstack/fuel-specs has extra files, please remove: > > .gitignore, .testr.conf, LICENSE, doc, images, policy, > > requirements.txt, setup.cfg, setup.py, specs, tests, tools, tox.ini > > [fuel] openstack/fuel-stats has extra files, please remove: > > .gitignore, LICENSE, MAINTAINERS, MANIFEST.in, analytics, collector, > > migration, requirements.txt, setup.py, test-requirements.txt, tools, > > tox.ini > > [fuel] openstack/python-fuelclient has extra files, please remove: > > .gitignore, .testr.conf, MAINTAINERS, MANIFEST.in, fuelclient, > > requirements.txt, setup.cfg, setup.py, specs, test-requirements.txt, > > tools, tox.ini > > > > [Infrastructure] opendev/puppet-releasestatus has extra files, please > > remove: .gitignore > > > > [ironic] openstack/python-dracclient has extra files, please remove: > > .gitignore, CONTRIBUTING.rst, HACKING.rst, LICENSE, doc, dracclient, > > requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini > > > > [neutron] openstack/networking-calico has extra files, please remove: > > .coveragerc, .gitignore, .mailmap, .testr.conf, .zuul.yaml, > > CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, RELEASING.md, > > babel.cfg, debian, devstack, doc, networking_calico, playbooks, > > requirements.txt, rpm, setup.cfg, setup.py, test-requirements.txt, > > tox.ini > > [neutron] openstack/networking-l2gw has extra files, please remove: > > .coveragerc, .gitignore, .testr.conf, .zuul.yaml, CONTRIBUTING.rst, > > HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, bindep.txt, contrib, > > debian, devstack, doc, etc, lower-constraints.txt, networking_l2gw, > > openstack-common.conf, requirements.txt, setup.cfg, setup.py, specs, > > test-requirements.txt, tools, tox.ini > > [neutron] openstack/networking-l2gw-tempest-plugin has extra files, > > please remove: .gitignore, .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, > > LICENSE, babel.cfg, contrib, doc, networking_l2gw_tempest_plugin, > > requirements.txt, setup.cfg, setup.py, test-requirements.txt, tox.ini > > [neutron] openstack/networking-onos has extra files, please remove: > > .coveragerc, .gitignore, .mailmap, .pylintrc, .testr.conf, > > CONTRIBUTING.rst, HACKING.rst, LICENSE, PKG-INFO, TESTING.rst, > > babel.cfg, devstack, doc, etc, lower-constraints.txt, networking_onos, > > package, rally-jobs, releasenotes, requirements.txt, setup.cfg, > > setup.py, test-requirements.txt, tools, tox.ini > > [neutron] openstack/neutron-vpnaas has extra files, please remove: > > .coveragerc, .gitignore, .mailmap, .pylintrc, .stestr.conf, > > .zuul.yaml, CONTRIBUTING.rst, HACKING.rst, LICENSE, TESTING.rst, > > babel.cfg, devstack, doc, etc, lower-constraints.txt, neutron_vpnaas, > > playbooks, rally-jobs, releasenotes, requirements.txt, setup.cfg, > > setup.py, test-requirements.txt, tools, tox.ini > > At least for networking-l2gw* and neutron-vpnaas, I suppose this was caused by:https://opendev.org/openstack/governance/commit/20f95dd947d2f87519b4bb50fb188e6f71deae7c > > What it meant is that they are not anymore under neutron governance, but they were not retired (at least as far as I know).There were still some recent commits even if minimal activity, and discussion on team status for neutron-vpnaas. > > Not sure about networking-calico status though > > [OpenStack Charms] openstack/charm-ceph has extra files, please > > remove: .gitignore > > > > [OpenStackAnsible] openstack/openstack-ansible-os_monasca has extra > > files, please remove: tests, tox.ini > > > > [solum] openstack/solum-infra-guestagent has extra files, please > > remove: .coveragerc, .gitignore, .mailmap, .testr.conf, > > CONTRIBUTING.rst, HACKING.rst, LICENSE, MANIFEST.in, babel.cfg, > > config-generator, doc, etc, requirements.txt, setup.cfg, setup.py, > > solum_guestagent, test-requirements.txt, tox.ini > > > > I'd like to kindly ask the affected teams to help out with this, or > > any member of our community is more than welcome to push a change to > > those repos and work with the appropriate teams to help land it. > > > > Mohammed > > > > -- > > Mohammed Naser — vexxhost > > ----------------------------------------------------- > > D. 514-316-8872 > > D. 800-910-1726 ext. 200 > > E. mnaser at vexxhost.com > > W. http://vexxhost.com > > > > > > > > -- > > Bernard Cafarelli > > > > From dharmendra.kushwaha at india.nec.com Tue Jul 9 06:34:26 2019 From: dharmendra.kushwaha at india.nec.com (Dharmendra Kushwaha) Date: Tue, 9 Jul 2019 06:34:26 +0000 Subject: [Tacker] Proposing changes in Tacker core team Message-ID: Hello Team, I am proposing below changes in Team: I would like to propose Hiroyuki Jo to join Tacker core team. Hiroyuki Jo have lead multiple valuable features level activities like affinity policy, VDU-healing, and VNF reservation [1] in Rocky & Stein cycle, and made it sure to be completed timely. And currently working on VNF packages [2] and ETSI NFV-SOL specification support [3]. Hiroyuki has a good understanding of NFV and Tacker project, and helping team by providing sensible reviews. I believe it is a good addition in Tacker core team, and Tacker project will benefit from this nomination. On the other hand, I wanted to thank to Bharath Thiruveedula for his great & valuable contribution in the project. He helped a lot to make Tacker better in early days. But now he doesn't seem to be active in project and he decided to step-down from core team. Whenever you will decide to come back to the project, I will be happy to add you in core-team. Core-Team, Please respond with your +1/-1. If no objection, I will do these changes in next week. [1] https://review.opendev.org/#/q/project:openstack/tacker-specs+owner:%22Hiroyuki+Jo+%253Chiroyuki.jo.mt%2540hco.ntt.co.jp%253E%22 [2] https://blueprints.launchpad.net/tacker/+spec/tosca-csar-mgmt-driver [3] https://blueprints.launchpad.net/tacker/+spec/support-etsi-nfv-specs Thanks & Regards Dharmendra Kushwaha From ekuvaja at redhat.com Tue Jul 9 11:41:10 2019 From: ekuvaja at redhat.com (Erno Kuvaja) Date: Tue, 9 Jul 2019 12:41:10 +0100 Subject: [Glance][PTL] Vacation time Message-ID: Hi all, I'll be away for couple of weeks. I'll be back Wed 24th of July. - Erno "jokke" Kuvaja -------------- next part -------------- An HTML attachment was scrubbed... URL: From skylightcoder at gmail.com Tue Jul 9 12:46:02 2019 From: skylightcoder at gmail.com (=?UTF-8?B?R8O2a2hhbiBJxZ5JSw==?=) Date: Tue, 9 Jul 2019 15:46:02 +0300 Subject: [Nova] Instances can't be started after compute nodes unexpectedly shut down because of power outage Message-ID: Hi folks, Because of power outage, Most of our compute nodes unexpectedly shut down and now I can not start our instances. Error message is "Failed to get "write" lock another process using the image?". Instances Power status is No State. Full error log is http://paste.openstack.org/show/754107/. My environment is OpenStack Pike on Ubuntu 16.04 LTS servers and Instances are on a nfs shared storage. Nova version is 16.1.6.dev2. qemu version is 2.10.1. libvirt version is 3.6.0. I saw a commit [1], but it doesn't solve this problem. There are important instances on my environment. How can I rescue my instances? What would you suggest ? Thanks, Gökhan [1] https://review.opendev.org/#/c/509774/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbedyk at suse.de Tue Jul 9 11:45:14 2019 From: wbedyk at suse.de (Witek Bedyk) Date: Tue, 9 Jul 2019 13:45:14 +0200 Subject: [monasca] Virtual Midcycle Meeting scheduling Message-ID: <40121ffb-3306-ef6b-b2ba-9d75e2a1e130@suse.de> Hello, as discussed in the last team meeting, we will hold a virtual Midcycle Meeting. The goal is to sync on the progress of the development and update the stories if needed. I plan with 2 hours meeting. Please select the times which work best for you: https://doodle.com/poll/zszfxakcbfm6sdha Please fill in the topics you would like to discuss or update on in the etherpad: https://etherpad.openstack.org/p/monasca-train-midcycle Thanks Witek From jungleboyj at gmail.com Tue Jul 9 14:31:45 2019 From: jungleboyj at gmail.com (Jay Bryant) Date: Tue, 9 Jul 2019 09:31:45 -0500 Subject: [cinder][stable-maint] July releases from the stable branches In-Reply-To: References: Message-ID: On 7/8/2019 5:44 PM, Brian Rosmaita wrote: > I've posted release patches for the stable branches: > https://review.openstack.org/#/q/topic:cinderproject-july-2019 > > Notes on what is/isn't being released are on the etherpad: > https://etherpad.openstack.org/p/cinder-releases-tracking > > I have a question about the semver for the cinder stable/stein release; > it's noted on the patch: https://review.opendev.org/669771 > > cheers, > brian Brian, Thanks for putting this together.  Changes look good to me! Jay From jim at jimrollenhagen.com Tue Jul 9 14:36:15 2019 From: jim at jimrollenhagen.com (Jim Rollenhagen) Date: Tue, 9 Jul 2019 10:36:15 -0400 Subject: [tc] Assuming control of GitHub organizations In-Reply-To: <9a4fdd56-7af8-c432-6ade-d52c163e8b8c@openstack.org> References: <10270aaf-9f4e-80b0-8e40-760d7c52dc0d@ham.ie> <9a4fdd56-7af8-c432-6ade-d52c163e8b8c@openstack.org> Message-ID: On Thu, Jun 27, 2019 at 8:55 AM Thierry Carrez wrote: > Graham Hayes wrote: > > On 27/06/2019 09:55, Thierry Carrez wrote: > >> I have been considering our GitHub presence as a downstream "code > >> marketing" property, a sort of front-end or entry point into the > >> OpenStack universe for outsiders. As such, I'd consider it much closer > >> to openstack.org/software than to opendev.org/openstack. > >> > >> So one way to do this would be to ask Foundation staff to maintain this > >> code marketing property, taking care of aligning message with the > >> content at openstack.org/software (which is driven from the > >> osf/openstack-map repository). > >> > >> If we handle it at TC-level my fear is that we would duplicate work > >> around things like project descriptions and what is pinned, and end up > >> with slightly different messages. > > > > I am not as concerned about this, the TC should be setting out our > > viewpoint for the project, and if this is in conflict with the message > > from the foundation, we have plenty of avenues to raise it. > > How about the TC controls which repo is replicated where (and which ones > are pinned etc), but we import the descriptions from the openstack-map > repo? > > That would keep control on the TC side but avoid duplication of effort. > In my experience it's already difficult to get projects to update > descriptions in one place, so two... > This seems reasonable to me. > > Also, who is volunteering for setting up the replication, and then > keeping track of things as they evolve ? > I'm up for it, can we get a couple more? I'd like to get this going soon. // jim > > -- > Thierry Carrez (ttx) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juliaashleykreger at gmail.com Tue Jul 9 15:20:45 2019 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Tue, 9 Jul 2019 08:20:45 -0700 Subject: [ironic][edge] L3/DHCP-Less deployments Message-ID: Greetings everyone! Over the last few weeks, I've had numerous discussions with contributors who have expressed interest in the DHCP-less deployment method specification document[0]. It seems many parties are interested! I think the path forward at this time is to get everyone talking about where and how they can help push this feature forward. If those interested could propose time windows when they are available, I'll try to find a mutual time window where we can try to get everyone on IRC or on a call and try to map out the next steps together. Sound good? -Julia [0]: https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/L3-based-deployment.html From mriedemos at gmail.com Tue Jul 9 18:13:14 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Tue, 9 Jul 2019 13:13:14 -0500 Subject: [watcher] nova cdm builder performance optimizations - summary Message-ID: <2d3cea37-d4ba-d435-9b46-ce711fac55cc@gmail.com> I wanted to summarize a series of changes which have improved the performance of the NovaClusterDataModel builder for audits across single and multiple cells (in the CERN case) by a factor of 20-30%. There were initially three changes involved (in order): 1. https://review.opendev.org/#/c/659688/ - Optimize NovaClusterDataModelCollector.add_instance_node Reports on that patch alone said it fixed a regression introduced in Stein with scoped audits: "I checked this patch on the my test environment on the stable/stein branch. I have more than 1000 virtual servers (some real, some dummy). Previously, in the stable/rocky branch, the time to build a cluster was about 15-20 minutes, in the Stein branch there was a regression and the time increased to 90 minutes. After this patch, the build time is only 2 minutes." That change was backported to stable/stein. 2. - https://review.opendev.org/#/c/661121/ - Optimize hypervisor API calls (which requires https://review.opendev.org/#/c/659886/) As noted that change requires a patch to python-novaclient if you are looking to backport the change. We can't backport that upstream because of the python-novaclient dependency since it would require bumping the minimum required version of the library on a stable branch which is against stable branch policy (minimum version of library dependencies are more or less frozen on stable branches). That change also requires configuring watcher with: [nova_client] api_version = 2.53 # or greater; train now requires at least 2.56 3. - https://review.opendev.org/#/c/662089/ - Optimize NovaHelper.get_compute_node_by_hostname This optimizes code used to build/update the nova CDM during notification processing and also fixes a bug about looking up the compute service properly. After those three changes were merged, Corne Lukken (Dantali0n) started doing scale and performance testing with and without the changes in a CERN 5-cell test cluster. Corne identified a regression for which Canwei Li determined the root cause and chenker fixed: 4. https://review.opendev.org/#/c/668100/ - Reduce the query time of the instances when call get_instance_list() With that fix applied Corne reported the overall improvement of 20-30% when building the nova CDM during an audit in various scenarios. The actual performance numbers will be sent later as part of a thesis Corne is working on. I want to thank Dantali0n, licanwei and chenker for all of their help with this series of improvements. -- Thanks, Matt From emilien at redhat.com Tue Jul 9 19:37:41 2019 From: emilien at redhat.com (Emilien Macchi) Date: Tue, 9 Jul 2019 15:37:41 -0400 Subject: [tripleo] [cisco] [bigswitch] [midonet] Composable Services for Neutron plugins Message-ID: With that thread I'm hoping to reach out some of the folks who were involved in TripleO services for some of the Neutron plugins like Cisco, Bigswitch and Midonet. Some of them still use ExtraConfig interface which isn't super ideal. I'm willing to help convert them into composable services but I'll need help on the reviews and more importantly the actual testing. If you're in CC and can help, good please let me know. If you're not involved anymore, please give me a name if possible. If you were involved and know the plugin doesn't need maintenance, let me know as well, it'll save me time. I started with the Cisco UCSM: https://review.opendev.org/669931 -- Emilien Macchi -------------- next part -------------- An HTML attachment was scrubbed... URL: From stig.openstack at telfer.org Tue Jul 9 20:45:56 2019 From: stig.openstack at telfer.org (Stig Telfer) Date: Tue, 9 Jul 2019 21:45:56 +0100 Subject: [scientific-sig] IRC today Message-ID: <552CEE47-1355-4E01-8A5D-50BB6D133A6C@telfer.org> Hi All - We have a Scientific SIG IRC meeting coming up at 2100 UTC (about 15 minutes time) in channel #openstack-meeting. Everyone is welcome. Today’s agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_July_9th_2019 If you’d like to add anything to the agenda, come along to the meeting and bring it up. Cheers, Stig From openstack at fried.cc Tue Jul 9 21:31:12 2019 From: openstack at fried.cc (Eric Fried) Date: Tue, 9 Jul 2019 16:31:12 -0500 Subject: [nova][ironic] Lock-related performance issue with update_resources periodic job In-Reply-To: <07C5F51A-DCAB-4432-B556-49E1E15801AC@fried.cc> References: <37D021C4-ED1B-4942-9C90-0A26FDE3DD76@fried.cc> <07C5F51A-DCAB-4432-B556-49E1E15801AC@fried.cc> Message-ID: <7883d905-69df-117c-c78d-8a5667e6c941@fried.cc> >>>> https://review.opendev.org/#/c/637225/ >>> Ah heck, I had totally forgotten about that patch. If it's working for you, let me get it polished up and merged. This is polished and ready for review, merge, backport. efried . From zhengzhenyulixi at gmail.com Wed Jul 10 01:49:04 2019 From: zhengzhenyulixi at gmail.com (Zhenyu Zheng) Date: Wed, 10 Jul 2019 09:49:04 +0800 Subject: [watcher] nova cdm builder performance optimizations - summary In-Reply-To: <2d3cea37-d4ba-d435-9b46-ce711fac55cc@gmail.com> References: <2d3cea37-d4ba-d435-9b46-ce711fac55cc@gmail.com> Message-ID: Thanks alot for the summary, this could be very helpful, we will have a test on these :) On Wed, Jul 10, 2019 at 2:29 AM Matt Riedemann wrote: > I wanted to summarize a series of changes which have improved the > performance of the NovaClusterDataModel builder for audits across single > and multiple cells (in the CERN case) by a factor of 20-30%. > > There were initially three changes involved (in order): > > 1. https://review.opendev.org/#/c/659688/ - Optimize > NovaClusterDataModelCollector.add_instance_node > > Reports on that patch alone said it fixed a regression introduced in > Stein with scoped audits: > > "I checked this patch on the my test environment on the stable/stein > branch. I have more than 1000 virtual servers (some real, some dummy). > Previously, in the stable/rocky branch, the time to build a cluster was > about 15-20 minutes, in the Stein branch there was a regression and the > time increased to 90 minutes. After this patch, the build time is only 2 > minutes." > > That change was backported to stable/stein. > > 2. - https://review.opendev.org/#/c/661121/ - Optimize hypervisor API > calls (which requires https://review.opendev.org/#/c/659886/) > > As noted that change requires a patch to python-novaclient if you are > looking to backport the change. We can't backport that upstream because > of the python-novaclient dependency since it would require bumping the > minimum required version of the library on a stable branch which is > against stable branch policy (minimum version of library dependencies > are more or less frozen on stable branches). > > That change also requires configuring watcher with: > > [nova_client] > api_version = 2.53 # or greater; train now requires at least 2.56 > > 3. - https://review.opendev.org/#/c/662089/ - Optimize > NovaHelper.get_compute_node_by_hostname > > This optimizes code used to build/update the nova CDM during > notification processing and also fixes a bug about looking up the > compute service properly. > > After those three changes were merged, Corne Lukken (Dantali0n) started > doing scale and performance testing with and without the changes in a > CERN 5-cell test cluster. Corne identified a regression for which Canwei > Li determined the root cause and chenker fixed: > > 4. https://review.opendev.org/#/c/668100/ - Reduce the query time of the > instances when call get_instance_list() > > With that fix applied Corne reported the overall improvement of 20-30% > when building the nova CDM during an audit in various scenarios. The > actual performance numbers will be sent later as part of a thesis Corne > is working on. > > I want to thank Dantali0n, licanwei and chenker for all of their help > with this series of improvements. > > -- > > Thanks, > > Matt > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sungn2 at lenovo.com Wed Jul 10 02:20:22 2019 From: sungn2 at lenovo.com (Guannan GN2 Sun) Date: Wed, 10 Jul 2019 02:20:22 +0000 Subject: [devstack] Deploy issue on Ubuntu 16.04. Message-ID: Hi team, I meet some problem when deploying devstack(from https://github.com/openstack/devstack master branch) on Ubuntu 16.04. It seems something is wrong with placement-api error message as following: curl -g -k --noproxy '*' -s -o /dev/null -w '%{http_code}%' http://10.240.24.138/placement [[503 == 503]] [ERROR] /opt/stack/devstack/lib/placement:156 placement-api did not start However when I check its status using "systemctl status devstack at placement-api", it is active and running. I also change to "stein" branch and try to deploy again, but still meet the same problem. Does someone meet similar issue before or could someone help me to debug this issue? Below is my local.conf file. Thank you! local.conf: #################################################### [[local|localrc]] # Credentials ADMIN_PASSWORD=password DATABASE_PASSWORD=password RABBIT_PASSWORD=password SERVICE_PASSWORD=password SERVICE_TOKEN=password SWIFT_HASH=password SWIFT_TEMPURL_KEY=password GIT_BASE=${GIT_BASE:-https://git.openstack.org} # A clean install every time RECLONE=yes # Enable Ironic plugin IRONIC_USING_PLUGIN=true enable_plugin ironic git://github.com/openstack/ironic # Enable Tempest enable_service tempest # Disable nova novnc service, ironic does not support it anyway. disable_service n-novnc # Enable Swift for the direct deploy interface. enable_service s-proxy enable_service s-object enable_service s-container enable_service s-account enable_service placement-api enable_service placement-client # Disable Horizon disable_service horizon # Disable Cinder disable_service cinder c-sch c-api c-vol # Swift temp URL's are required for the direct deploy interface SWIFT_ENABLE_TEMPURLS=True # Tempest related options BUILD_TIMEOUT=3000 IRONIC_CALLBACK_TIMEOUT=3000 POWER_TIMEOUT=600 SERVICE_TIMEOUT=600 TEMPEST_PLUGINS+=' /opt/stack/ironic-tempest-plugin' # Ironic related options IRONIC_IS_HARDWARE=true VIRT_DRIVER=ironic IRONIC_HW_NODE_CPU=1 IRONIC_HW_NODE_RAM=4096 IRONIC_HW_NODE_DISK=20 IRONIC_BAREMETAL_BASIC_OPS=True DEFAULT_INSTANCE_TYPE=baremetal # Enable additional hardware types, if needed. IRONIC_ENABLED_HARDWARE_TYPES=ipmi,fake-hardware,xclarity # Don't forget that many hardware types require enabling of additional # interfaces, most often power and management: IRONIC_ENABLED_MANAGEMENT_INTERFACES=ipmitool,fake,xclarity IRONIC_ENABLED_POWER_INTERFACES=ipmitool,fake,xclarity # The 'ipmi' hardware type's default deploy interface is 'iscsi'. # This would change the default to 'direct': #IRONIC_DEFAULT_DEPLOY_INTERFACE=direct # Change this to alter the default driver for nodes created by devstack. # This driver should be in the enabled list above. IRONIC_DEPLOY_DRIVER=ipmi # The parameters below represent the minimum possible values to create # functional nodes. #IRONIC_VM_SPECS_RAM=1280 #IRONIC_VM_SPECS_DISK=10 # Size of the ephemeral partition in GB. Use 0 for no ephemeral partition. IRONIC_VM_EPHEMERAL_DISK=0 # By default, DevStack creates a 10.0.0.0/24 network for instances. # If this overlaps with the hosts network, you may adjust with the # following. PUBLIC_NETWORK_GATEWAY=10.240.24.1 FLOATING_RANGE=10.240.24.0/24 FIXED_RANGE=10.0.0.0/24 HOST_IP=10.240.24.138 SERVICE_HOST=$HOST_IP FIXED_NETWORK_SIZE=256 # Neutron options Q_USE_PROVIDERNET_FOR_PUBLIC=True PUBLIC_INTERFACE=ens9 OVS_PHYSICAL_BRIDGE=br-ens9 PUBLIC_BRIDGE=br-ens9 OVS_BRIDGE_MAPPINGS=public:br-ens9 ALLOCATION_POOL=start=10.0.0.30,end=10.0.0.100 # Log all output to files LOGFILE=/opt/stack/devstack.log LOGDIR=/opt/stack/logs IRONIC_VM_LOG_DIR=/opt/stack/ironic-bm-logs #################################################### Best Regards, Guannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From sekharvajjula at gmail.com Wed Jul 10 06:38:59 2019 From: sekharvajjula at gmail.com (chandra sekhar) Date: Wed, 10 Jul 2019 09:38:59 +0300 Subject: [ironic][edge] L3/DHCP-Less deployments In-Reply-To: References: Message-ID: Hi Julia, I am based in Finland . Anytime from 8:00 to 19:00 (EEST) is good for me. Regards Chandra R. On Tue, 9 Jul 2019, 18:33 Julia Kreger, wrote: > Greetings everyone! > > Over the last few weeks, I've had numerous discussions with > contributors who have expressed interest in the DHCP-less deployment > method specification document[0]. > > It seems many parties are interested! I think the path forward at this > time is to get everyone talking about where and how they can help push > this feature forward. > > If those interested could propose time windows when they are > available, I'll try to find a mutual time window where we can try to > get everyone on IRC or on a call and try to map out the next steps > together. > > Sound good? > > -Julia > > [0]: > https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/L3-based-deployment.html > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdent+os at anticdent.org Wed Jul 10 06:50:37 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Wed, 10 Jul 2019 07:50:37 +0100 (BST) Subject: [devstack] Deploy issue on Ubuntu 16.04. In-Reply-To: References: Message-ID: On Wed, 10 Jul 2019, Guannan GN2 Sun wrote: > I meet some problem when deploying devstack(from > https://github.com/openstack/devstack master branch) on Ubuntu > 16.04. It seems something is wrong with placement-api error > message as following: > > > curl -g -k --noproxy '*' -s -o /dev/null -w '%{http_code}%' http://10.240.24.138/placement > > [[503 == 503]] > > [ERROR] /opt/stack/devstack/lib/placement:156 placement-api did not start > > > However when I check its status using "systemctl status > devstack at placement-api", it is active and running. I also change > to "stein" branch and try to deploy again, but still meet the same > problem. Does someone meet similar issue before or could someone > help me to debug this issue? Below is my local.conf file. Thank > you! Look in /etc/apache2/sites-enabled/placement-api.conf (and any other files in the same directory) and make sure you only have one proxy configuration for the connection between apache2 and the uwsgi process that is running under systemd. What's could be happening is that though you have placement running, apache is trying to talk to the wrong thing. You can either: * clean up the placement-api.conf file so that it only has one entry and restart apache2 * unstack.sh, remove the files in /etc/apache2/sites-enabled and /etc/apaceh2/sites-available, and rerun stack.sh This happens when there are repeated runs of stack.sh on the same host with insufficient cleanup between. This probably means there's a bit of a bug in lib/placement that we could fix so that it cleans up after itself better. I hope this is helpful. If this wasn't it and you figure out what was causing it please post the solution. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From josephine.seifert at secustack.com Wed Jul 10 07:45:51 2019 From: josephine.seifert at secustack.com (Josephine Seifert) Date: Wed, 10 Jul 2019 09:45:51 +0200 Subject: [image-encryption][nova][cinder][glance][barbican]Image Encryption popup team meeting Message-ID: <9033c7e9-fc98-c6b3-b8b3-7a6d4a7df4d0@secustack.com> Hello, starting with next week, there will be a weekly meeting on *Monday, 13 UTC *[1]**for the Image Encryption popup team. Please feel free to join us :) Greetings Josephine [1] http://eavesdrop.openstack.org/#Image_Encryption_Popup-Team_Meeting -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean-philippe at evrard.me Wed Jul 10 08:15:10 2019 From: jean-philippe at evrard.me (Jean-Philippe Evrard) Date: Wed, 10 Jul 2019 10:15:10 +0200 Subject: [tripleo][openstack-ansible] Integrating ansible-role-collect-logs in OSA In-Reply-To: References: Message-ID: <71d568ee47ce516a5a4cab1422290da2be1baff6.camel@evrard.me> On Fri, 2019-06-28 at 16:30 +0530, Chandan kumar wrote: > With os_tempest project, TripleO and Openstack Ansible projects > started collaborating together to reuse the tools > developed by each other to avoid duplicates and enable more > collaboration. ... And that's amazing! > In TripleO, we have ansible-role-collect-logs[1.] role for the same > and in OSA we have logs_collect.sh[2.] script for > the same. But once the logs gets collected at each other projects, It > is very hard to navigate and find out where > is the respective files. Agreed. > By Keeping the same structure at all places > It would be easier. Agreed again > So moving ahead what we are going to do: > * Refactor collect-logs role to pass defaults list of files at one > place It seems the role is doing a lot of things, but things can be conditionally triggered. Wondering if this role shouldn't be split into multiple roles... But that's a different story. > * Pass the list of different logs files based on deployment tools I think this doesn't need to be in the role. Make the role the simplest as possible, and flexible enough to get passed the list of things to log/not log, and the destination. OSA can pass a list of files it wants to gather. But isn't that what the role already does? Or did I misunderstood? > * Put system/containers related commands at one place Can we simply rely on ansible inventory, and running a play with the role (targetting all) would gather logs for all systems (aio, multinodes, containers), each system could go into their own directory (log folder would be /{{ inventory_hostname }}/...) for example: aio1/ aio1-nova/ machine2/ It simple enough. But I am happy to see a different approach. > * Replace the collect_logs.sh script with playbook in OSA and replace > it. :thumbsup: > Thanks for reading, We are looking for the comments on the above > suggestion. Thanks for tackling that up! I am looking forward a simple common file gathering :) If you need to do changes in the role (to implement new features), maybe it can help you if I give you a prio list :) What I am _generally_ looking for in the logs: - The ara reports - The tempest report - The /etc/openstack_deploy/ - The /var/log/ for containers/hosts What I am _sometimes_ looking for in the logs (so less important/more contextual for me): - ram/disk usage per host - NICs details - cpu features (but I am not sure we really need this anymore) - host details (generally zuul does that for me, so no need to implement something there) Regards, Jean-Philippe Evrard (evrardjp) From zigo at debian.org Wed Jul 10 09:08:01 2019 From: zigo at debian.org (Thomas Goirand) Date: Wed, 10 Jul 2019 11:08:01 +0200 Subject: [horizon] django-pyscss unmaintained? Message-ID: <97088153-2339-4719-c08b-a78f8fd8090d@debian.org> Hi, As I'm maintaining python-django-pyscss in Debian, I've noticed that there's no commit since 2015 on the Github repository, and no pypi release either. I've sent a new pull request here: https://github.com/fusionbox/django-pyscss/pull/45 so that pyscss continues to pass unit tests with Django 2.2, but I have very little hope for it to be merged, as the previous pull request for Django 1.11 was never merged. Is it time to take over the project? Can someone from the Horizon team get in touch with the previous maintainer? Cheers, Thomas Goirand (zigo) From rico.lin.guanyu at gmail.com Wed Jul 10 09:18:37 2019 From: rico.lin.guanyu at gmail.com (Rico Lin) Date: Wed, 10 Jul 2019 17:18:37 +0800 Subject: [auto-scaling] Auto-scaling SIG update Message-ID: It has been two months after Summit and PTG [3], and we now have collected some use cases [1], and general documents for auto-scaling [2] (Great thanks to Joseph Davis and Adam Spiers). *Use Cases needed* We need more use cases from all, for autoscaling on/with OpenStack. For example, autoscaling k8s on OpenStack is already a thing, so IMO we need to put some words in use cases too. *Help to improve documents* Please check our new document and help to improve it if you think of any. *Join us* We have IRC channel and meeting so please join us in irc: #openstack-auto-scaling so we can hear you and you can help us. *Add request to our StoryBoard* If you have any feature requests or WIP plan, you can use our storyboard[4] as root trace for them all like request for `Integrate Monasca and Senlin`. *Open for bug?* One question I would like to bring to the team is should we open our storyboard[4] for bug collection? We have some good hands from multiple teams, who can help with bugs. And like feature requests, a top-level story for a bug should be very helpful, but what we need from the reporter in order to get better information about that bug, and also what part should we help? IMO we can make sure bug are well documented, trace bug progress (from top-level view), and help to raise attention(in ML and in events). Would like to hear more opinions on this. *PTG* I already sign up for half-day PTG schedule(as the team still under developing, I would not expect that we need more for now), so I see you all there if you will be there too! [1] https://docs.openstack.org/auto-scaling-sig/latest/use-cases.html [2] https://docs.openstack.org/auto-scaling-sig/latest/theory-of-auto-scaling.html [3] https://etherpad.openstack.org/p/DEN-auto-scaling-SIG [4] https://storyboard.openstack.org/#!/project/openstack/auto-scaling-sig [5] https://storyboard.openstack.org/#!/story/2005627 -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry at openstack.org Wed Jul 10 10:01:05 2019 From: thierry at openstack.org (Thierry Carrez) Date: Wed, 10 Jul 2019 12:01:05 +0200 Subject: [tc] Assuming control of GitHub organizations In-Reply-To: References: <10270aaf-9f4e-80b0-8e40-760d7c52dc0d@ham.ie> <9a4fdd56-7af8-c432-6ade-d52c163e8b8c@openstack.org> Message-ID: Jim Rollenhagen wrote: > On Thu, Jun 27, 2019 at 8:55 AM Thierry Carrez > wrote: > >> How about the TC controls which repo is replicated where (and which ones >> are pinned etc), but we import the descriptions from the openstack-map >> repo? >> >> That would keep control on the TC side but avoid duplication of effort. >> In my experience it's already difficult to get projects to update >> descriptions in one place, so two... > > This seems reasonable to me. > > >> Also, who is volunteering for setting up the replication, and then >> keeping track of things as they evolve ? > > I'm up for it, can we get a couple more? I'd like to get this going soon. Count me in. -- Thierry Carrez (ttx) From smooney at redhat.com Wed Jul 10 11:00:46 2019 From: smooney at redhat.com (Sean Mooney) Date: Wed, 10 Jul 2019 12:00:46 +0100 Subject: [devstack] Deploy issue on Ubuntu 16.04. In-Reply-To: References: Message-ID: On Wed, 2019-07-10 at 07:50 +0100, Chris Dent wrote: > On Wed, 10 Jul 2019, Guannan GN2 Sun wrote: > > > I meet some problem when deploying devstack(from > > https://github.com/openstack/devstack master branch) on Ubuntu > > 16.04. It seems something is wrong with placement-api error > > message as following: > > > > > > curl -g -k --noproxy '*' -s -o /dev/null -w '%{http_code}%' http://10.240.24.138/placement > > > > [[503 == 503]] > > > > [ERROR] /opt/stack/devstack/lib/placement:156 placement-api did not start > > > > > > However when I check its status using "systemctl status > > devstack at placement-api", it is active and running. I also change > > to "stein" branch and try to deploy again, but still meet the same > > problem. Does someone meet similar issue before or could someone > > help me to debug this issue? Below is my local.conf file. Thank > > you! > what cdent said below is also true but just wanted to highlight that master of devstack is not intendeted to be run with ubuntu 16.04 we move to 18.04 some thime ago upstream so if there have been any deviations between 16.04 and 18.04 dont expect master of devestack to support both. > Look in /etc/apache2/sites-enabled/placement-api.conf (and any other > files in the same directory) and make sure you only have one proxy > configuration for the connection between apache2 and the uwsgi > process that is running under systemd. > > What's could be happening is that though you have placement running, > apache is trying to talk to the wrong thing. > > You can either: > > * clean up the placement-api.conf file so that it only has one entry > and restart apache2 > > * unstack.sh, remove the files in /etc/apache2/sites-enabled and > /etc/apaceh2/sites-available, and rerun stack.sh > > This happens when there are repeated runs of stack.sh on the same > host with insufficient cleanup between. This probably means there's > a bit of a bug in lib/placement that we could fix so that it cleans > up after itself better. > > I hope this is helpful. If this wasn't it and you figure out what > was causing it please post the solution. > From cdent+os at anticdent.org Wed Jul 10 12:51:19 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Wed, 10 Jul 2019 13:51:19 +0100 (BST) Subject: [placement] mascot/logo live Message-ID: https://www.openstack.org/project-mascots has been updated to have the Placement mascot/logo. Here's a sample: https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619c416ff4bd19e1c087f27a43eea/www-images-prod/project-mascots/Placement/OpenStack_Project_Placement_vertical.jpg Thanks very much to the Foundation and their designers for making this happen. In case you missed the discussion [1] about this a while back, we know it looks a bit cranky. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/thread.html#7170 -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From james.page at canonical.com Wed Jul 10 13:02:30 2019 From: james.page at canonical.com (James Page) Date: Wed, 10 Jul 2019 14:02:30 +0100 Subject: [nova-lxd] retiring nova-lxd Message-ID: Hi All I’m slightly sad to announce that we’re retiring the LXD driver for Nova aka “nova-lxd”. Developing a driver for Nova for container based machines has been a fun and technically challenging ride over the last four years but we’ve never really seen any level of serious production deployment; as a result we’ve decided that it’s time to call it a day for nova-lxd. I’d like to thank all of the key contributors for their efforts over the years - specifically Chuck Short, Paul Hummer, Chris MacNaughton, Sahid Orentino and Alex Kavanaugh who have led or contributed to the development of the driver over its lifetime. I’ll be raising a review to leave a note for future followers as to the fate of nova-lxd. If anyone else would like to continue development of the driver they are more than welcome to revert my commit and become a part of the development team! We’ll continue to support our current set of stable branches for another ~12 months. Note that development of LXD and the pylxd Python module continues; its just the integration of OpenStack with LXD that we’re ceasing development of. Regards James -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetan.trellu at incloudus.com Wed Jul 10 13:06:58 2019 From: gaetan.trellu at incloudus.com (=?ISO-8859-1?Q?Ga=EBtan_Trellu?=) Date: Wed, 10 Jul 2019 09:06:58 -0400 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: Message-ID: An HTML attachment was scrubbed... URL: From juliaashleykreger at gmail.com Wed Jul 10 14:17:39 2019 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Wed, 10 Jul 2019 07:17:39 -0700 Subject: [ironic] Shanghai PTG/Forum Message-ID: Greetings fellow ironic contributors! I've created an initial planning/discussion etherpad for the upcoming Shanghai summit[0]. I need a count of contributors that intend to attend with-in the next few weeks. If you plan on or intend to attend, please indicate so on the etherpad. I look forward to ideas and items for discussion! -Julia [0]: https://etherpad.openstack.org/p/PVG-Ironic-Planning From johnsomor at gmail.com Wed Jul 10 15:21:48 2019 From: johnsomor at gmail.com (Michael Johnson) Date: Wed, 10 Jul 2019 08:21:48 -0700 Subject: [nova-lxd] retiring nova-lxd In-Reply-To: References: Message-ID: James, I am sorry to hear this. This last year I implemented a Proof-of-Concept using nova-lxd for the Octavia Amphora instances. There are some rough edges, but we were successful with our tempest tests passing[1][2]. LXD was the most straight forward path for the Octavia team to take for a container implementation. Thank you for the work and effort that went into nova-lxd, Michael [1] https://review.opendev.org/636066 [2] https://review.opendev.org/636069 On Wed, Jul 10, 2019 at 6:04 AM James Page wrote: > > Hi All > > > I’m slightly sad to announce that we’re retiring the LXD driver for Nova aka “nova-lxd”. > > > Developing a driver for Nova for container based machines has been a fun and technically challenging ride over the last four years but we’ve never really seen any level of serious production deployment; as a result we’ve decided that it’s time to call it a day for nova-lxd. > > > I’d like to thank all of the key contributors for their efforts over the years - specifically Chuck Short, Paul Hummer, Chris MacNaughton, Sahid Orentino and Alex Kavanaugh who have led or contributed to the development of the driver over its lifetime. > > > I’ll be raising a review to leave a note for future followers as to the fate of nova-lxd. If anyone else would like to continue development of the driver they are more than welcome to revert my commit and become a part of the development team! > > > We’ll continue to support our current set of stable branches for another ~12 months. > > > Note that development of LXD and the pylxd Python module continues; its just the integration of OpenStack with LXD that we’re ceasing development of. > > > Regards > > > James > > From fungi at yuggoth.org Wed Jul 10 15:24:40 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 10 Jul 2019 15:24:40 +0000 Subject: [horizon] django-pyscss unmaintained? In-Reply-To: <97088153-2339-4719-c08b-a78f8fd8090d@debian.org> References: <97088153-2339-4719-c08b-a78f8fd8090d@debian.org> Message-ID: <20190710152440.sxxentyyjwh6bvky@yuggoth.org> On 2019-07-10 11:08:01 +0200 (+0200), Thomas Goirand wrote: > As I'm maintaining python-django-pyscss in Debian, I've noticed that > there's no commit since 2015 on the Github repository, and no pypi > release either. [...] For that matter, the pyScss upstream maintainer seems to have not merged anything new or commented on new issues and pull requests for roughly a year at this point. It's worth considering that both might be abandoned now. How intrinsic is Sass to Horizon vs just straight CSS3? Enough that it would be less work for the Horizon team to take over the Python ecosystem tools around it than to replace the bits of Sass in use? Or are there newer tools which accomplish the same thing now and that's why the old ones are falling into disuse? -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From miguel at mlavalle.com Wed Jul 10 16:02:57 2019 From: miguel at mlavalle.com (Miguel Lavalle) Date: Wed, 10 Jul 2019 11:02:57 -0500 Subject: [openstack-dev] [neutron] [ptg] Start planning for the PTG in Shanghai Message-ID: Dear Neutrinos, We need to help the Foundation to plan for the the PTG in Shanghai. To that end, I have created an etherpad to start collecting the names of the team members who plan to attend and also topics to be discussed during our meetings: https://etherpad.openstack.org/p/Shanghai-Neutron-Planning I need to respond back to the Foundation's survey by August 11th. So it will be very helpful if all of you who plan to be in Shanghai add your names and irc nicks to the "Attendees" section no later than August 4th. This deadline doesn't apply to the proposed topics section. For that, we have more time. Regards Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: From openstack at fried.cc Wed Jul 10 16:05:24 2019 From: openstack at fried.cc (Eric Fried) Date: Wed, 10 Jul 2019 11:05:24 -0500 Subject: [nova-lxd] retiring nova-lxd In-Reply-To: References: Message-ID: <88feef3d-f86e-7a0e-78e7-298c6ad6fb40@fried.cc> > I’m slightly sad to announce that we’re retiring the LXD driver for Nova > aka “nova-lxd”. How does this impact the fate of the following series? https://review.opendev.org/667975 Add support for 'initenv' elements https://review.opendev.org/667976 Add support for cloud-init on LXC instances efried . From smooney at redhat.com Wed Jul 10 16:15:15 2019 From: smooney at redhat.com (Sean Mooney) Date: Wed, 10 Jul 2019 17:15:15 +0100 Subject: [nova-lxd] retiring nova-lxd In-Reply-To: <88feef3d-f86e-7a0e-78e7-298c6ad6fb40@fried.cc> References: <88feef3d-f86e-7a0e-78e7-298c6ad6fb40@fried.cc> Message-ID: On Wed, 2019-07-10 at 11:05 -0500, Eric Fried wrote: > > I’m slightly sad to announce that we’re retiring the LXD driver for Nova > > aka “nova-lxd”. > > How does this impact the fate of the following series? > > https://review.opendev.org/667975 Add support for 'initenv' elements > https://review.opendev.org/667976 Add support for cloud-init on LXC > instances it should have no impact that is for lxc container supprot via libvirt nova-lxd is an out of tree driver managing lxc container via lxd. so those patchse should be able to proceed unaffected by nova-lxd's retirement > > efried > . > > From thierry at openstack.org Wed Jul 10 16:37:10 2019 From: thierry at openstack.org (Thierry Carrez) Date: Wed, 10 Jul 2019 18:37:10 +0200 Subject: [tc] agenda for Technical Committee Meeting 11 July 2019 @ 1400 UTC Message-ID: TC Members, Our next meeting will be this Thursday, 11 July at 1400 UTC in #openstack-tc. I'll be your chair and we'll rely on mugsie to keep the GIF game level high. This email contains the agenda for the meeting, based on the content of the wiki [0]. If you will not be able to attend, please include your name in the "Apologies for Absence" section of the wiki page [0]. [0] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee * Follow up on past action items ** Health check changes: asettle to update community (done), fungi to update wiki (done), mugise to update yaml file with liasons and mnaser to update the tooling ** Help-most-needed list: AlanClark and zaneb to update investment opportunities document [0] https://etherpad.openstack.org/p/2019-upstream-investment-opportunities-refactor [1] https://review.opendev.org/#/c/657447/ ** Goal selection: lbragstad to prune the community-goals etherpad [2] https://etherpad.openstack.org/p/community-goals ** Pop-up teams: ttx to define pop-up teams [3] https://review.opendev.org/#/c/661356/ [4] https://review.opendev.org/#/c/661983/5 ** SIG governance: tc-members to review goal, popup, and SIG project etherpad [5] https://etherpad.openstack.org/p/explain-team-formate-differentiate ** Review PTL Guide: asettle ttx ricolin to sync up and review the PTL section of the project teams guide to improve the PTL experience [6] https://review.opendev.org/#/c/665699/ * Active initiatives ** Python 3: mnaser to sync up with swift team on python3 migration and mugsie to sync with dhellmann or release-team to find the code for the proposal bot ** Forum follow-up: ttx to organise Milestone 2 forum meeting with tc-members ** Make goal selection a two-step process (needs reviews at https://review.opendev.org/#/c/667932/) * Update on U release naming process * What are retired repos ? Regards, -- Thierry Carrez (ttx) From whayutin at redhat.com Wed Jul 10 18:12:26 2019 From: whayutin at redhat.com (Wesley Hayutin) Date: Wed, 10 Jul 2019 12:12:26 -0600 Subject: [tripleo][openstack-ansible] Integrating ansible-role-collect-logs in OSA In-Reply-To: <71d568ee47ce516a5a4cab1422290da2be1baff6.camel@evrard.me> References: <71d568ee47ce516a5a4cab1422290da2be1baff6.camel@evrard.me> Message-ID: On Wed, Jul 10, 2019 at 2:23 AM Jean-Philippe Evrard < jean-philippe at evrard.me> wrote: > On Fri, 2019-06-28 at 16:30 +0530, Chandan kumar wrote: > > With os_tempest project, TripleO and Openstack Ansible projects > > started collaborating together to reuse the tools > > developed by each other to avoid duplicates and enable more > > collaboration. > > ... And that's amazing! > Good stuff :) Agreed. > > > In TripleO, we have ansible-role-collect-logs[1.] role for the same > > and in OSA we have logs_collect.sh[2.] script for > > the same. But once the logs gets collected at each other projects, It > > is very hard to navigate and find out where > > is the respective files. > > Agreed. > > > By Keeping the same structure at all places > > It would be easier. > > Agreed again > > > So moving ahead what we are going to do: > > * Refactor collect-logs role to pass defaults list of files at one > > place > > It seems the role is doing a lot of things, but things can be > conditionally triggered. Wondering if this role shouldn't be split into > multiple roles... But that's a different story. > > > * Pass the list of different logs files based on deployment tools > > I think this doesn't need to be in the role. Make the role the simplest > as possible, and flexible enough to get passed the list of things to > log/not log, and the destination. OSA can pass a list of files it wants > to gather. But isn't that what the role already does? Or did I > misunderstood? > The TripleO team passes various config files to the collect roles depending on what the needs and requirements are. Some of these config files are public some are not. upstream config https://github.com/openstack/tripleo-ci/blob/master/toci-quickstart/config/collect-logs.yml default config https://github.com/openstack/ansible-role-collect-logs/blob/master/defaults/main.yml These are of course just passed in as extra-config. I think each project would want to define their own list of files and maintain it in their own project. WDYT? > > > * Put system/containers related commands at one place > > Can we simply rely on ansible inventory, and running a play with the > role (targetting all) would gather logs for all systems (aio, > multinodes, containers), each system > could go into their own directory (log folder would be /{{ > inventory_hostname }}/...) for example: > > aio1/ > aio1-nova/ > machine2/ > > It simple enough. But I am happy to see a different approach. > Yes, this is exactly how it works today. We don't break out which files should be collect for each host, but that is just our preference. > > > * Replace the collect_logs.sh script with playbook in OSA and replace > > it. > > :thumbsup: > > > Thanks for reading, We are looking for the comments on the above > > suggestion. > > Thanks for tackling that up! > I am looking forward a simple common file gathering :) > > If you need to do changes in the role (to implement new features), > maybe it can help you if I give you a prio list :) > > What I am _generally_ looking for in the logs: > - The ara reports > - The tempest report > - The /etc/openstack_deploy/ > - The /var/log/ for containers/hosts > Agree, having a table of contents in the footer is nice for users as well. https://github.com/openstack/tripleo-ci/blob/master/docs/tripleo-quickstart-logs.html Which is added by infra via https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/manifests/static.pp > > What I am _sometimes_ looking for in the logs (so less important/more > contextual for me): > - ram/disk usage per host > - NICs details > - cpu features (but I am not sure we really need this anymore) > - host details (generally zuul does that for me, so no need to > implement something there) > > AFAICT, if we were to organize the role more aggressively via the tasks we can easily enable or disable features as needed per project. The majority of the work would around the reorganization to better suit various projects. Any thoughts on additional work that I am not seeing? Thanks for responding! I know our team is very excited about the continued collaboration with other upstream projects, so thanks!! > Regards, > Jean-Philippe Evrard (evrardjp) > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vungoctan252 at gmail.com Tue Jul 9 15:25:59 2019 From: vungoctan252 at gmail.com (Vu Tan) Date: Tue, 9 Jul 2019 22:25:59 +0700 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: References: <09a3849b-786e-49ed-a197-5e13af0428bf@email.android.com> <144114a2e83d8e8e30579ddb0ae39e59@incloudus.com> Message-ID: Thank Patil Tushar, I hope it will be available soon On Tue, Jul 9, 2019 at 8:18 AM Patil, Tushar wrote: > Hi Vu and Gaetan, > > Gaetan, thank you for helping out Vu in setting up masakari-monitors > service. > > As a masakari team ,we have noticed there is a need to add proper > documentation to help the community run Masakari services in their > environment. We are working on adding proper documentation in this 'Train' > cycle. > > Will send an email on this mailing list once the patches are uploaded on > the gerrit so that you can give your feedback on the same. > > If you have any trouble in setting up Masakari, please let us know on this > mailing list or join the bi-weekly IRC Masakari meeting on the > #openstack-meeting IRC channel. The next meeting will be held on 16th July > 2019 @0400 UTC. > > Regards, > Tushar Patil > > ________________________________________ > From: Vu Tan > Sent: Monday, July 8, 2019 11:21:16 PM > To: Gaëtan Trellu > Cc: openstack-discuss at lists.openstack.org > Subject: Re: [masakari] how to install masakari on centos 7 > > Hi Gaetan, > Thanks for pinpoint this out, silly me that did not notice the simple > "error InterpreterNotFound: python3". Thanks a lot, I appreciate it > > On Mon, Jul 8, 2019 at 9:15 PM gaetan.trellu at incloudus.com>> wrote: > Vu Tan, > > About "auth_token" error, you need "os_privileged_user_*" options into > your masakari.conf for the API. > As mentioned previously please have a look here to have an example of > configuration working (for me at least): > > - masakari.conf: > > https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari.conf.j2 > - masakari-monitor.conf: > > https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari-monitors.conf.j2 > > About your tox issue make sure you have Python3 installed. > > Gaëtan > > On 2019-07-08 06:08, Vu Tan wrote: > > > Hi Gaetan, > > I try to generate config file by using this command tox -egenconfig on > > top level of masakari but the output is error, is this masakari still > > in beta version ? > > [root at compute1 masakari-monitors]# tox -egenconfig > > genconfig create: /root/masakari-monitors/.tox/genconfig > > ERROR: InterpreterNotFound: python3 > > _____________________________________________________________ summary > > ______________________________________________________________ > > ERROR: genconfig: InterpreterNotFound: python3 > > > > On Mon, Jul 8, 2019 at 3:24 PM Vu Tan vungoctan252 at gmail.com>> wrote: > > Hi, > > Thanks a lot for your reply, I install pacemaker/corosync, > > masakari-api, maskari-engine on controller node, and I run masakari-api > > with this command: masakari-api, but I dont know whether the process is > > running like that or is it just hang there, here is what it shows when > > I run the command, I leave it there for a while but it does not change > > anything : > > [root at controller masakari]# masakari-api > > 2019-07-08 15:21:09.946 30250 INFO masakari.api.openstack [-] Loaded > > extensions: ['extensions', 'notifications', 'os-hosts', 'segments', > > 'versions'] > > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > > [-] The option "__file__" in conf is not known to auth_token > > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > > [-] The option "here" in conf is not known to auth_token > > 2019-07-08 15:21:09.960 30250 WARNING keystonemiddleware.auth_token [-] > > AuthToken middleware is set with > > keystone_authtoken.service_token_roles_required set to False. This is > > backwards compatible but deprecated behaviour. Please set this to True. > > 2019-07-08 15:21:09.974 30250 INFO masakari.wsgi [-] masakari_api > > listening on 127.0.0.1:15868 > > 2019-07-08 15:21:09.975 30250 INFO oslo_service.service [-] Starting 4 > > workers > > 2019-07-08 15:21:09.984 30274 INFO masakari.masakari_api.wsgi.server > > [-] (30274) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.985 30275 INFO masakari.masakari_api.wsgi.server > > [-] (30275) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.992 30277 INFO masakari.masakari_api.wsgi.server > > [-] (30277) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.994 30276 INFO masakari.masakari_api.wsgi.server > > [-] (30276) wsgi starting up on http://127.0.0.1:15868 > > > > On Sun, Jul 7, 2019 at 7:37 PM Gaëtan Trellu > > > wrote: > > > > Hi Vu Tan, > > > > Masakari documentation doesn't really exist... I had to figured some > > stuff by myself to make it works into Kolla project. > > > > On controller nodes you need: > > > > - pacemaker > > - corosync > > - masakari-api (openstack/masakari repository) > > - masakari- engine (openstack/masakari repository) > > > > On compute nodes you need: > > > > - pacemaker-remote (integrated to pacemaker cluster as a resource) > > - masakari- hostmonitor (openstack/masakari-monitor repository) > > - masakari-instancemonitor (openstack/masakari-monitor repository) > > - masakari-processmonitor (openstack/masakari-monitor repository) > > > > For masakari-hostmonitor, the service needs to have access to systemctl > > command (make sure you are not using sysvinit). > > > > For masakari-monitor, the masakari-monitor.conf is a bit different, you > > will have to configure the [api] section properly. > > > > RabbitMQ needs to be configured (as transport_url) on masakari-api and > > masakari-engine too. > > > > Please check this review[1], you will have masakari.conf and > > masakari-monitor.conf configuration examples. > > > > [1] https://review.opendev.org/#/c/615715 > > > > Gaëtan > > > > On Jul 7, 2019 12:08 AM, Vu Tan vungoctan252 at gmail.com>> wrote: > > > > VU TAN > > > > > 10:30 AM (35 minutes ago) > > > > to openstack-discuss > > > > Sorry, I resend this email because I realized that I lacked of prefix > > on this email's subject > > > > Hi, > > > > I would like to use Masakari and I'm having trouble finding a step by > > step or other documentation to get started with. Which part should be > > installed on controller, which is should be on compute, and what is the > > prerequisite to install masakari, I have installed corosync and > > pacemaker on compute and controller nodes, , what else do I need to do > > ? step I have done so far: > > - installed corosync/pacemaker > > - install masakari on compute node on this github repo: > > https://github.com/openstack/masakari > > - add masakari in to mariadb > > here is my configuration file of masakari.conf, do you mind to take a > > look at it, if I have misconfigured anything? > > > > [DEFAULT] > > enabled_apis = masakari_api > > > > # Enable to specify listening IP other than default > > masakari_api_listen = controller > > # Enable to specify port other than default > > masakari_api_listen_port = 15868 > > debug = False > > auth_strategy=keystone > > > > [wsgi] > > # The paste configuration file path > > api_paste_config = /etc/masakari/api-paste.ini > > > > [keystone_authtoken] > > www_authenticate_uri = http://controller:5000 > > auth_url = http://controller:5000 > > auth_type = password > > project_domain_id = default > > user_domain_id = default > > project_name = service > > username = masakari > > password = P at ssword > > > > [database] > > connection = mysql+pymysql://masakari:P at ssword@controller/masakari > Disclaimer: This email and any attachments are sent in strictest > confidence for the sole use of the addressee and may contain legally > privileged, confidential, and proprietary data. If you are not the intended > recipient, please advise the sender by replying promptly to this email and > then delete and destroy this email and any attachments without any further > use, copying or forwarding. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vungoctan252 at gmail.com Wed Jul 10 07:12:49 2019 From: vungoctan252 at gmail.com (Vu Tan) Date: Wed, 10 Jul 2019 14:12:49 +0700 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: References: <09a3849b-786e-49ed-a197-5e13af0428bf@email.android.com> <144114a2e83d8e8e30579ddb0ae39e59@incloudus.com> Message-ID: Hi Gaetan, I follow you the guide you gave me, but the problem still persist, can you please take a look at my configuration to see what is wrong or what is missing in my config ? the error: 2019-07-10 14:08:46.876 17292 WARNING keystonemiddleware._common.config [-] The option "__file__" in conf is not known to auth_token 2019-07-10 14:08:46.876 17292 WARNING keystonemiddleware._common.config [-] The option "here" in conf is not known to auth_token 2019-07-10 14:08:46.882 17292 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_ the config: [DEFAULT] enabled_apis = masakari_api log_dir = /var/log/kolla/masakari state_path = /var/lib/masakari os_user_domain_name = default os_project_domain_name = default os_privileged_user_tenant = service os_privileged_user_auth_url = http://controller:5000/v3 os_privileged_user_name = nova os_privileged_user_password = P at ssword masakari_api_listen = controller masakari_api_listen_port = 15868 debug = False auth_strategy=keystone [wsgi] # The paste configuration file path api_paste_config = /etc/masakari/api-paste.ini [keystone_authtoken] www_authenticate_uri = http://controller:5000 auth_url = http://controller:5000 auth_type = password project_domain_id = default project_domain_name = default user_domain_name = default user_domain_id = default project_name = service username = masakari password = P at ssword region_name = RegionOne [oslo_middleware] enable_proxy_headers_parsing = True [database] connection = mysql+pymysql://masakari:P at ssword@controller/masakari On Tue, Jul 9, 2019 at 10:25 PM Vu Tan wrote: > Thank Patil Tushar, I hope it will be available soon > > On Tue, Jul 9, 2019 at 8:18 AM Patil, Tushar > wrote: > >> Hi Vu and Gaetan, >> >> Gaetan, thank you for helping out Vu in setting up masakari-monitors >> service. >> >> As a masakari team ,we have noticed there is a need to add proper >> documentation to help the community run Masakari services in their >> environment. We are working on adding proper documentation in this 'Train' >> cycle. >> >> Will send an email on this mailing list once the patches are uploaded on >> the gerrit so that you can give your feedback on the same. >> >> If you have any trouble in setting up Masakari, please let us know on >> this mailing list or join the bi-weekly IRC Masakari meeting on the >> #openstack-meeting IRC channel. The next meeting will be held on 16th July >> 2019 @0400 UTC. >> >> Regards, >> Tushar Patil >> >> ________________________________________ >> From: Vu Tan >> Sent: Monday, July 8, 2019 11:21:16 PM >> To: Gaëtan Trellu >> Cc: openstack-discuss at lists.openstack.org >> Subject: Re: [masakari] how to install masakari on centos 7 >> >> Hi Gaetan, >> Thanks for pinpoint this out, silly me that did not notice the simple >> "error InterpreterNotFound: python3". Thanks a lot, I appreciate it >> >> On Mon, Jul 8, 2019 at 9:15 PM > gaetan.trellu at incloudus.com>> wrote: >> Vu Tan, >> >> About "auth_token" error, you need "os_privileged_user_*" options into >> your masakari.conf for the API. >> As mentioned previously please have a look here to have an example of >> configuration working (for me at least): >> >> - masakari.conf: >> >> https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari.conf.j2 >> - masakari-monitor.conf: >> >> https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari-monitors.conf.j2 >> >> About your tox issue make sure you have Python3 installed. >> >> Gaëtan >> >> On 2019-07-08 06:08, Vu Tan wrote: >> >> > Hi Gaetan, >> > I try to generate config file by using this command tox -egenconfig on >> > top level of masakari but the output is error, is this masakari still >> > in beta version ? >> > [root at compute1 masakari-monitors]# tox -egenconfig >> > genconfig create: /root/masakari-monitors/.tox/genconfig >> > ERROR: InterpreterNotFound: python3 >> > _____________________________________________________________ summary >> > ______________________________________________________________ >> > ERROR: genconfig: InterpreterNotFound: python3 >> > >> > On Mon, Jul 8, 2019 at 3:24 PM Vu Tan > vungoctan252 at gmail.com>> wrote: >> > Hi, >> > Thanks a lot for your reply, I install pacemaker/corosync, >> > masakari-api, maskari-engine on controller node, and I run masakari-api >> > with this command: masakari-api, but I dont know whether the process is >> > running like that or is it just hang there, here is what it shows when >> > I run the command, I leave it there for a while but it does not change >> > anything : >> > [root at controller masakari]# masakari-api >> > 2019-07-08 15:21:09.946 30250 INFO masakari.api.openstack [-] Loaded >> > extensions: ['extensions', 'notifications', 'os-hosts', 'segments', >> > 'versions'] >> > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config >> > [-] The option "__file__" in conf is not known to auth_token >> > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config >> > [-] The option "here" in conf is not known to auth_token >> > 2019-07-08 15:21:09.960 30250 WARNING keystonemiddleware.auth_token [-] >> > AuthToken middleware is set with >> > keystone_authtoken.service_token_roles_required set to False. This is >> > backwards compatible but deprecated behaviour. Please set this to True. >> > 2019-07-08 15:21:09.974 30250 INFO masakari.wsgi [-] masakari_api >> > listening on 127.0.0.1:15868 >> > 2019-07-08 15:21:09.975 30250 INFO oslo_service.service [-] Starting 4 >> > workers >> > 2019-07-08 15:21:09.984 30274 INFO masakari.masakari_api.wsgi.server >> > [-] (30274) wsgi starting up on http://127.0.0.1:15868 >> > 2019-07-08 15:21:09.985 30275 INFO masakari.masakari_api.wsgi.server >> > [-] (30275) wsgi starting up on http://127.0.0.1:15868 >> > 2019-07-08 15:21:09.992 30277 INFO masakari.masakari_api.wsgi.server >> > [-] (30277) wsgi starting up on http://127.0.0.1:15868 >> > 2019-07-08 15:21:09.994 30276 INFO masakari.masakari_api.wsgi.server >> > [-] (30276) wsgi starting up on http://127.0.0.1:15868 >> > >> > On Sun, Jul 7, 2019 at 7:37 PM Gaëtan Trellu >> > > >> wrote: >> > >> > Hi Vu Tan, >> > >> > Masakari documentation doesn't really exist... I had to figured some >> > stuff by myself to make it works into Kolla project. >> > >> > On controller nodes you need: >> > >> > - pacemaker >> > - corosync >> > - masakari-api (openstack/masakari repository) >> > - masakari- engine (openstack/masakari repository) >> > >> > On compute nodes you need: >> > >> > - pacemaker-remote (integrated to pacemaker cluster as a resource) >> > - masakari- hostmonitor (openstack/masakari-monitor repository) >> > - masakari-instancemonitor (openstack/masakari-monitor repository) >> > - masakari-processmonitor (openstack/masakari-monitor repository) >> > >> > For masakari-hostmonitor, the service needs to have access to systemctl >> > command (make sure you are not using sysvinit). >> > >> > For masakari-monitor, the masakari-monitor.conf is a bit different, you >> > will have to configure the [api] section properly. >> > >> > RabbitMQ needs to be configured (as transport_url) on masakari-api and >> > masakari-engine too. >> > >> > Please check this review[1], you will have masakari.conf and >> > masakari-monitor.conf configuration examples. >> > >> > [1] https://review.opendev.org/#/c/615715 >> > >> > Gaëtan >> > >> > On Jul 7, 2019 12:08 AM, Vu Tan > vungoctan252 at gmail.com>> wrote: >> > >> > VU TAN > >> > >> > 10:30 AM (35 minutes ago) >> > >> > to openstack-discuss >> > >> > Sorry, I resend this email because I realized that I lacked of prefix >> > on this email's subject >> > >> > Hi, >> > >> > I would like to use Masakari and I'm having trouble finding a step by >> > step or other documentation to get started with. Which part should be >> > installed on controller, which is should be on compute, and what is the >> > prerequisite to install masakari, I have installed corosync and >> > pacemaker on compute and controller nodes, , what else do I need to do >> > ? step I have done so far: >> > - installed corosync/pacemaker >> > - install masakari on compute node on this github repo: >> > https://github.com/openstack/masakari >> > - add masakari in to mariadb >> > here is my configuration file of masakari.conf, do you mind to take a >> > look at it, if I have misconfigured anything? >> > >> > [DEFAULT] >> > enabled_apis = masakari_api >> > >> > # Enable to specify listening IP other than default >> > masakari_api_listen = controller >> > # Enable to specify port other than default >> > masakari_api_listen_port = 15868 >> > debug = False >> > auth_strategy=keystone >> > >> > [wsgi] >> > # The paste configuration file path >> > api_paste_config = /etc/masakari/api-paste.ini >> > >> > [keystone_authtoken] >> > www_authenticate_uri = http://controller:5000 >> > auth_url = http://controller:5000 >> > auth_type = password >> > project_domain_id = default >> > user_domain_id = default >> > project_name = service >> > username = masakari >> > password = P at ssword >> > >> > [database] >> > connection = mysql+pymysql://masakari:P at ssword@controller/masakari >> Disclaimer: This email and any attachments are sent in strictest >> confidence for the sole use of the addressee and may contain legally >> privileged, confidential, and proprietary data. If you are not the intended >> recipient, please advise the sender by replying promptly to this email and >> then delete and destroy this email and any attachments without any further >> use, copying or forwarding. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbedyk at suse.de Wed Jul 10 10:26:27 2019 From: wbedyk at suse.de (Witek Bedyk) Date: Wed, 10 Jul 2019 12:26:27 +0200 Subject: [auto-scaling] Auto-scaling SIG update In-Reply-To: References: Message-ID: Hi Rico, thanks for putting this together. > *Open for bug?* > One question I would like to bring to the team is should we open our > storyboard[4] for bug collection? We have some good hands from multiple > teams, who can help with bugs. And like feature requests, a top-level > story for a bug should be very helpful, but what we need from the > reporter in order to get better information about that bug, and also > what part should we help? > IMO we can make sure bug are well documented, trace bug progress (from > top-level view), and help to raise attention(in ML and in events). Would > like to hear more opinions on this. I think it's a good idea. Most bugs will refer to individual projects and code changes there. But StoryBoard has a nice ability to collect tasks for several projects in one story. Auto-scaling related bugs in individual projects could add an additional task in auto-scaling project. That way the internal project work will gain more general context and better visibility. Cheers Witek From laurentfdumont at gmail.com Wed Jul 10 19:08:34 2019 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Wed, 10 Jul 2019 15:08:34 -0400 Subject: openstack-sdk API - Get all flavors for all projects. Message-ID: Hi everyone, I'm trying to map the relationship between projects, host-aggregate and computes. I'm trying to find a way to use the Openstack Python SDK in order to list all existing flavors and the "access_project_ids" seems to be the best way to see which project is associated with the flavor. That said, it seems that the python SDK does not expose the value when querying using "get_flavor". Anyone know of way of getting the information from the SDK? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From james.slagle at gmail.com Wed Jul 10 20:17:38 2019 From: james.slagle at gmail.com (James Slagle) Date: Wed, 10 Jul 2019 16:17:38 -0400 Subject: [TripleO] Scaling node counts with only Ansible (N=1) Message-ID: There's been a fair amount of recent work around simplifying our Heat templates and migrating the software configuration part of our deployment entirely to Ansible. As part of this effort, it became apparent that we could render much of the data that we need out of Heat in a way that is generic per node, and then have Ansible render the node specific data during config-download runtime. To illustrate the point, consider when we specify ComputeCount:10 in our templates, that much of the work that Heat is doing across those 10 sets of resources for each Compute node is duplication. However, it's been necessary so that Heat can render data structures such as list of IP's, lists of hostnames, contents of /etc/hosts files, etc etc etc. If all that was driven by Ansible using host facts, then Heat doesn't need to do those 10 sets of resources to begin with. The goal is to get to a point where we can deploy the Heat stack with a count of 1 for each role, and then deploy any number of nodes per role using Ansible. To that end, I've been referring to this effort as N=1. The value in this work is that it directly addresses our scaling issues with Heat (by just deploying a much smaller stack). Obviously we'd still be relying heavily on Ansible to scale to the required levels, but I feel that is much better understood challenge at this point in the evolution of configuration tools. With the patches that we've been working on recently, I've got a POC running where I can deploy additional compute nodes with just Ansible. This is done by just adding the additional nodes to the Ansible inventory with a small set of facts to include IP addresses on each enabled network and a hostname. These patches are at https://review.opendev.org/#/q/topic:bp/reduce-deployment-resources and reviews/feedback are welcome. Other points: - Baremetal provisioning and port creation are presently handled by Heat. With the ongoing efforts to migrate baremetal provisioning out of Heat (nova-less deploy), I think these efforts are very complimentary. Eventually, we get to a point where Heat is not actually creating any other OpenStack API resources. For now, the patches only work when using pre-provisioned nodes. - We need to consider how we'd manage the Ansible inventory going forward if we open up an interface for operators to manipulate it directly. That's something we'd want to manage and preserve (version control) as it's critical data for the deployment. Given the progress that we've made with the POC, my sense is that we'll keep pushing in this overall direction. I'd like to get some feedback on the approach. We have an etherpad we are using to track some of the work at a high level: https://etherpad.openstack.org/p/tripleo-reduce-deployment-resources I'll be adding some notes on how I setup the POC to that etherpad if others would like to try it out. -- -- James Slagle -- From colleen at gazlene.net Wed Jul 10 22:15:45 2019 From: colleen at gazlene.net (Colleen Murphy) Date: Wed, 10 Jul 2019 15:15:45 -0700 Subject: [keystone] Shanghai PTG planning Message-ID: <60277b1f-9453-4abd-a3c4-a9117885e56f@www.fastmail.com> Hi team, The foundation has asked us to let them know whether we'll be needing PTG space in Shanghai. The keystone team usually has good attendance at PTGs and makes good use of the time, but I know this next one may be hard for some people to attend so I don't want to automatically assume we'll use it this time. If you wish to attend the next PTG and have a semi-reasonable amount of confidence that you may be able to, please add your name to the etherpad: https://etherpad.openstack.org/p/keystone-shanghai-ptg Please do this by Monday, August 5 so that I have time to respond to the foundation by August 11. Feel free to also start brainstorming topics for the PTG (no deadline on this). Colleen From openstack at fried.cc Wed Jul 10 22:44:57 2019 From: openstack at fried.cc (Eric Fried) Date: Wed, 10 Jul 2019 17:44:57 -0500 Subject: [nova][ptg] Shanghai attendance Message-ID: <1b6fcd6a-b3c6-d99b-fb20-466a8733115d@fried.cc> We (nova) need to let the foundation know about our planned presence at the Shanghai PTG for the U release. To that end, I've seeded an etherpad [1]. If you are a contributor or operator with an interest in nova, please add your name to the Attendance section, even (especially) if you know you are not attending. I need this information by the beginning of August, even if it's tentative. Thanks! efried [1] https://etherpad.openstack.org/p/nova-shanghai-ptg From mriedemos at gmail.com Thu Jul 11 00:06:58 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Wed, 10 Jul 2019 19:06:58 -0500 Subject: [nova][ptg] Shanghai attendance In-Reply-To: <1b6fcd6a-b3c6-d99b-fb20-466a8733115d@fried.cc> References: <1b6fcd6a-b3c6-d99b-fb20-466a8733115d@fried.cc> Message-ID: On 7/10/2019 5:44 PM, Eric Fried wrote: > We (nova) need to let the foundation know about our planned presence at > the Shanghai PTG for the U release. To that end, I've seeded an etherpad > [1]. If you are a contributor or operator with an interest in nova, > please add your name to the Attendance section, even (especially) if you > know you are not attending. I need this information by the beginning of > August, even if it's tentative. > > Thanks! > > efried > > [1]https://etherpad.openstack.org/p/nova-shanghai-ptg Any idea when the early bird pricing ends? Also, will there be a separate discount for the PTG like there has been in the past or is it all a flat combined rate now? This is more a question for Foundation-y people reading this. -- Thanks, Matt From allison at openstack.org Thu Jul 11 00:11:00 2019 From: allison at openstack.org (Allison Price) Date: Wed, 10 Jul 2019 19:11:00 -0500 Subject: [nova][ptg] Shanghai attendance In-Reply-To: References: <1b6fcd6a-b3c6-d99b-fb20-466a8733115d@fried.cc> Message-ID: Hey Matt, Early bird pricing is going to end in early August - we are publishing the exact date soon. There will be a contributor discount that will include OpenStack ATCs and AUCs, but there will not be a separate PTG discount. We will have more information in the upcoming days, but let us know if you have questions in the meantime. Thanks! Allison > On Jul 10, 2019, at 7:06 PM, Matt Riedemann wrote: > > On 7/10/2019 5:44 PM, Eric Fried wrote: >> We (nova) need to let the foundation know about our planned presence at >> the Shanghai PTG for the U release. To that end, I've seeded an etherpad >> [1]. If you are a contributor or operator with an interest in nova, >> please add your name to the Attendance section, even (especially) if you >> know you are not attending. I need this information by the beginning of >> August, even if it's tentative. >> Thanks! >> efried >> [1]https://etherpad.openstack.org/p/nova-shanghai-ptg > > Any idea when the early bird pricing ends? > > Also, will there be a separate discount for the PTG like there has been in the past or is it all a flat combined rate now? > > This is more a question for Foundation-y people reading this. > > -- > > Thanks, > > Matt > From mriedemos at gmail.com Thu Jul 11 00:15:00 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Wed, 10 Jul 2019 19:15:00 -0500 Subject: [nova][ptg] Shanghai attendance In-Reply-To: References: <1b6fcd6a-b3c6-d99b-fb20-466a8733115d@fried.cc> Message-ID: On 7/10/2019 7:11 PM, Allison Price wrote: > Early bird pricing is going to end in early August - we are publishing the exact date soon. There will be a contributor discount that will include OpenStack ATCs and AUCs, but there will not be a separate PTG discount. > > We will have more information in the upcoming days, but let us know if you have questions in the meantime. Thanks. I just compared the PTG and Summit registrations and it looks like they are the same - $231 for standard access and that gets you into both events. If it's a combined ticket I'm assuming any ATC discount would just apply to that (both summit and PTG). I'm also assuming "but there will not be a separate PTG discount" just means there is no discount for attending the last PTG in Denver. -- Thanks, Matt From emccormick at cirrusseven.com Thu Jul 11 00:17:47 2019 From: emccormick at cirrusseven.com (Erik McCormick) Date: Wed, 10 Jul 2019 20:17:47 -0400 Subject: [nova][ptg] Shanghai attendance In-Reply-To: References: <1b6fcd6a-b3c6-d99b-fb20-466a8733115d@fried.cc> Message-ID: On Wed, Jul 10, 2019, 8:08 PM Matt Riedemann wrote: > On 7/10/2019 5:44 PM, Eric Fried wrote: > > We (nova) need to let the foundation know about our planned presence at > > the Shanghai PTG for the U release. To that end, I've seeded an etherpad > > [1]. If you are a contributor or operator with an interest in nova, > > please add your name to the Attendance section, even (especially) if you > > know you are not attending. I need this information by the beginning of > > August, even if it's tentative. > > > > Thanks! > > > > efried > > > > [1]https://etherpad.openstack.org/p/nova-shanghai-ptg > > Any idea when the early bird pricing ends? > > Also, will there be a separate discount for the PTG like there has been > in the past or is it all a flat combined rate now? > > This is more a question for Foundation-y people reading this. > Attendance in Denver PTG (last Train Hotel) should get you into this summit gratis, no? I've been assuming codes will appear at some point ;) -- > > Thanks, > > Matt > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kendall at openstack.org Thu Jul 11 00:38:56 2019 From: kendall at openstack.org (Kendall Waters) Date: Wed, 10 Jul 2019 19:38:56 -0500 Subject: [nova][ptg] Shanghai attendance In-Reply-To: References: <1b6fcd6a-b3c6-d99b-fb20-466a8733115d@fried.cc> Message-ID: Yes, that is correct, the standard access will grant you access to both the Summit and PTG and the contributor discount can be applied to that ticket. To clarify, there is no discount for attending the last PTG. Since we have co-located the Summit and PTG, we are no longer offering a PTG attendee discount to the Summit like we have in the past. Cheers, Kendall > On Jul 10, 2019, at 7:17 PM, Erik McCormick wrote: > > > > On Wed, Jul 10, 2019, 8:08 PM Matt Riedemann > wrote: > On 7/10/2019 5:44 PM, Eric Fried wrote: > > We (nova) need to let the foundation know about our planned presence at > > the Shanghai PTG for the U release. To that end, I've seeded an etherpad > > [1]. If you are a contributor or operator with an interest in nova, > > please add your name to the Attendance section, even (especially) if you > > know you are not attending. I need this information by the beginning of > > August, even if it's tentative. > > > > Thanks! > > > > efried > > > > [1]https://etherpad.openstack.org/p/nova-shanghai-ptg > > Any idea when the early bird pricing ends? > > Also, will there be a separate discount for the PTG like there has been > in the past or is it all a flat combined rate now? > > This is more a question for Foundation-y people reading this. > > Attendance in Denver PTG (last Train Hotel) should get you into this summit gratis, no? I've been assuming codes will appear at some point ;) > > -- > > Thanks, > > Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From emccormick at cirrusseven.com Thu Jul 11 01:24:31 2019 From: emccormick at cirrusseven.com (Erik McCormick) Date: Wed, 10 Jul 2019 21:24:31 -0400 Subject: [nova][ptg] Shanghai attendance In-Reply-To: References: <1b6fcd6a-b3c6-d99b-fb20-466a8733115d@fried.cc> Message-ID: On Wed, Jul 10, 2019, 8:38 PM Kendall Waters wrote: > Yes, that is correct, the standard access will grant you access to both > the Summit and PTG and the contributor discount can be applied to that > ticket. > > To clarify, there is no discount for attending the last PTG. Since we have > co-located the Summit and PTG, we are no longer offering a PTG attendee > discount to the Summit like we have in the past. > The PTG I was referring to was the last one at the Renaissance. That was supposed to come with free admission to the next 2 summits (Denver and Shanghai). Hopefully that is still being honored? > Cheers, > Kendall > > > On Jul 10, 2019, at 7:17 PM, Erik McCormick > wrote: > > > > On Wed, Jul 10, 2019, 8:08 PM Matt Riedemann wrote: > >> On 7/10/2019 5:44 PM, Eric Fried wrote: >> > We (nova) need to let the foundation know about our planned presence at >> > the Shanghai PTG for the U release. To that end, I've seeded an etherpad >> > [1]. If you are a contributor or operator with an interest in nova, >> > please add your name to the Attendance section, even (especially) if you >> > know you are not attending. I need this information by the beginning of >> > August, even if it's tentative. >> > >> > Thanks! >> > >> > efried >> > >> > [1]https://etherpad.openstack.org/p/nova-shanghai-ptg >> >> Any idea when the early bird pricing ends? >> >> Also, will there be a separate discount for the PTG like there has been >> in the past or is it all a flat combined rate now? >> >> This is more a question for Foundation-y people reading this. >> > > Attendance in Denver PTG (last Train Hotel) should get you into this > summit gratis, no? I've been assuming codes will appear at some point ;) > > -- >> >> Thanks, >> >> Matt > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Thu Jul 11 03:03:48 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 11 Jul 2019 03:03:48 +0000 Subject: [nova][ptg] Shanghai attendance In-Reply-To: References: <1b6fcd6a-b3c6-d99b-fb20-466a8733115d@fried.cc> Message-ID: <20190711030347.33gkdi2ncuxzi2ok@yuggoth.org> On 2019-07-10 21:24:31 -0400 (-0400), Erik McCormick wrote: [...] > The PTG I was referring to was the last one at the Renaissance. That was > supposed to come with free admission to the next 2 summits (Denver and > Shanghai). Hopefully that is still being honored? [...] Your list has an off-by-one error. ;) That second "Denver" PTG at the Renaissance Stapleton Hotel was September 2018. The next two summits following it were November 2018 in Berlin and April 2019 in Denver. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From emccormick at cirrusseven.com Thu Jul 11 03:21:21 2019 From: emccormick at cirrusseven.com (Erik McCormick) Date: Wed, 10 Jul 2019 23:21:21 -0400 Subject: [nova][ptg] Shanghai attendance In-Reply-To: <20190711030347.33gkdi2ncuxzi2ok@yuggoth.org> References: <1b6fcd6a-b3c6-d99b-fb20-466a8733115d@fried.cc> <20190711030347.33gkdi2ncuxzi2ok@yuggoth.org> Message-ID: On Wed, Jul 10, 2019, 11:05 PM Jeremy Stanley wrote: > On 2019-07-10 21:24:31 -0400 (-0400), Erik McCormick wrote: > [...] > > The PTG I was referring to was the last one at the Renaissance. That was > > supposed to come with free admission to the next 2 summits (Denver and > > Shanghai). Hopefully that is still being honored? > [...] > > Your list has an off-by-one error. ;) > > That second "Denver" PTG at the Renaissance Stapleton Hotel was > September 2018. The next two summits following it were November 2018 > in Berlin and April 2019 in Denver. > -- > Jeremy Stanley > Wow. These cold meds have truly addled my brain. Thanks for the correction. I sleep now. -Erik > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sungn2 at lenovo.com Thu Jul 11 06:03:54 2019 From: sungn2 at lenovo.com (Guannan GN2 Sun) Date: Thu, 11 Jul 2019 06:03:54 +0000 Subject: [devstack] Deploy issue on Ubuntu 16.04. (Guannan) Message-ID: <594856dca0114bfaa4596dedf13c68a7@lenovo.com> Thanks Chric and Sean, I run unstack.sh, remove the files in /etc/apache2/sites-enabled and /etc/apaceh2/sites-available, and rerun stack.sh, that works for me! Although I'm still run master branch of devstack on Ubuntu 16.04, as Sean say, I may move to Ubuntu 18.04 later. Thank you! Best Regards, Guannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From manuel.sb at garvan.org.au Thu Jul 11 06:15:46 2019 From: manuel.sb at garvan.org.au (Manuel Sopena Ballesteros) Date: Thu, 11 Jul 2019 06:15:46 +0000 Subject: trying to understand steal time with cpu pinning Message-ID: <9D8A2486E35F0941A60430473E29F15B017EB2889D@MXDB1.ad.garvan.unsw.edu.au> Dear Openstack community, Please correct me if I am wrong. As far as I understand `steal time > 0` means that the hypervisor has replaced a vcpu with a different one on the physical cpu. Also, cpu pinning allocates a vcpu to a physical cpu permanently. I have a vm setup with cpu pinning and numa affinity and realized, that cpu steal time is between 1% and 0%. Why is that? Thank you very much NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From skylightcoder at gmail.com Thu Jul 11 06:58:23 2019 From: skylightcoder at gmail.com (=?UTF-8?B?R8O2a2hhbiBJxZ5JSw==?=) Date: Thu, 11 Jul 2019 09:58:23 +0300 Subject: [Nova] Instances can't be started after compute nodes unexpectedly shut down because of power outage In-Reply-To: References: Message-ID: Can anyone help me please ? I can no't rescue my instances yet :( Thanks, Gökhan Gökhan IŞIK , 9 Tem 2019 Sal, 15:46 tarihinde şunu yazdı: > Hi folks, > Because of power outage, Most of our compute nodes unexpectedly > shut down and now I can not start our instances. Error message is "Failed > to get "write" lock another process using the image?". Instances Power > status is No State. Full error log is > http://paste.openstack.org/show/754107/. My environment is OpenStack Pike > on Ubuntu 16.04 LTS servers and Instances are on a nfs shared storage. Nova > version is 16.1.6.dev2. qemu version is 2.10.1. libvirt version is 3.6.0. I > saw a commit [1], but it doesn't solve this problem. > There are important instances on my environment. How can I rescue my > instances? What would you suggest ? > > Thanks, > Gökhan > > [1] https://review.opendev.org/#/c/509774/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jyotishri403 at gmail.com Thu Jul 11 07:22:20 2019 From: jyotishri403 at gmail.com (Jyoti Dahiwele) Date: Thu, 11 Jul 2019 12:52:20 +0530 Subject: [cinder] Glusterfs support in stein In-Reply-To: <20190708125119.GA15668@sm-workstation> References: <20190708125119.GA15668@sm-workstation> Message-ID: Thank you for the clarification. On Mon, 8 Jul 2019, 18:21 Sean McGinnis, wrote: > On Mon, Jul 08, 2019 at 02:13:13PM +0200, Massimo Sgaravatto wrote: > > We also used it in the past but as far as I remember the gluster driver > was > > removed in Ocata (we had therefore to migrate to the NFSdriver) > > > > Cheers, Massimo > > > > This is correct. The GlusterFS driver was marked as deprecated by its > maintainers in the Newton release, then officially removed in Ocata. > > The driver can still be found in those stable branches, but would probably > take > a not insignificant amount of work to use in later releases. > > Sean > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.teckelmann at bertelsmann.de Thu Jul 11 07:28:26 2019 From: ralf.teckelmann at bertelsmann.de (Teckelmann, Ralf, NMU-OIP) Date: Thu, 11 Jul 2019 07:28:26 +0000 Subject: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' Message-ID: Good Morning everyone, We like to have FWaaS enabled for a Stein-based OpenStack installation. Using linuxbridges we are not able to use FWaaS_v2, because it only seems to work with ovs. We thus tried FWaaS (v1) following https://docs.openstack.org/openstack-ansible-os_neutron/latest/configure-network-services.html#firewall-service-optional . However, all we get from it is (1). Are we missing a point or is FWaaS_V1 just not supported in Stein anymore? If so, this would mean for a setup Stein+Linuxbridges no FWaaS is actually available, right? (1) grep firewall /var/log/neutron/neutron-server.log 2019-07-05 10:10:55.693 29793 ERROR neutron_lib.utils.runtime NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' 2019-07-05 10:10:55.694 29793 ERROR neutron.manager [req-394624b6-e638-45ec-be7c-ce86793fdbc4 - - - - -] Plugin 'firewall' not found. 2019-07-05 10:11:00.046 29979 INFO neutron.manager [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Loading Plugin: firewall 2019-07-05 10:11:00.046 29979 ERROR neutron_lib.utils.runtime [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Error loading class by alias: NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' Best regards, Ralf T. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdent+os at anticdent.org Thu Jul 11 07:59:27 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Thu, 11 Jul 2019 08:59:27 +0100 (BST) Subject: [placement][ptg] Shanghai attendance Message-ID: As with nova [1] and keystone [2], placement needs to decide what kind of presence it would like to have at the PTG portion of the event in Shanghai in November, and let the Foundation know. I've already expressed that I don't intend to go [3] and in that message asked for feedback from the rest of the placement team on whether we need to do any placement work there, given the value we got out of the virtual pre-PTG in April. Let me ask again: Do we need a presence at the PTG in Shanghai? I've created an etherpad where we can track who will be there for sure and who will not, and start making note of relevant topics. https://etherpad.openstack.org/p/placement-shanghai-ptg [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007640.html [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007639.html [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007527.html -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From skaplons at redhat.com Thu Jul 11 08:04:02 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 11 Jul 2019 10:04:02 +0200 Subject: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' In-Reply-To: References: Message-ID: <28A1866C-F2C6-4BAC-BE73-467AD22B4805@redhat.com> Hi, FWaaS v1 was deprecated since some time and was removed completely in Stein release. > On 11 Jul 2019, at 09:28, Teckelmann, Ralf, NMU-OIP wrote: > > Good Morning everyone, > > We like to have FWaaS enabled for a Stein-based OpenStack installation. > Using linuxbridges we are not able to use FWaaS_v2, because it only seems to work with ovs. > > We thus tried FWaaS (v1) following https://docs.openstack.org/openstack-ansible-os_neutron/latest/configure-network-services.html#firewall-service-optional . > However, all we get from it is (1). > > Are we missing a point or is FWaaS_V1 just not supported in Stein anymore? > If so, this would mean for a setup Stein+Linuxbridges no FWaaS is actually available, right? > > (1) > grep firewall /var/log/neutron/neutron-server.log > 2019-07-05 10:10:55.693 29793 ERROR neutron_lib.utils.runtime NoMatches: No'neutron.service_plugins' driver found, looking for 'firewall' > 2019-07-05 10:10:55.694 29793 ERROR neutron.manager [req-394624b6-e638-45ec-be7c-ce86793fdbc4 - - - - -] Plugin 'firewall' not found. > 2019-07-05 10:11:00.046 29979 INFO neutron.manager [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Loading Plugin: firewall > 2019-07-05 10:11:00.046 29979 ERROR neutron_lib.utils.runtime [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Error loading class by alias: NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' > > Best regards, > > Ralf T. — Slawek Kaplonski Senior software engineer Red Hat From ignaziocassano at gmail.com Thu Jul 11 09:01:09 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Thu, 11 Jul 2019 11:01:09 +0200 Subject: [queens][nova] nova host-evacuate errot Message-ID: Hello All, on ocata when I poweroff a node with active instance , doing a nova host-evacuate works fine and instances are restartd on an active node. On queens it does non evacuate instances but nova-api reports for each instance the following: 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is in task_state powering-off So it poweroff all instance on the failed node but does not start them on active nodes What is changed ? Ignazio -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.teckelmann at bertelsmann.de Thu Jul 11 09:23:54 2019 From: ralf.teckelmann at bertelsmann.de (Teckelmann, Ralf, NMU-OIP) Date: Thu, 11 Jul 2019 09:23:54 +0000 Subject: AW: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' In-Reply-To: <28A1866C-F2C6-4BAC-BE73-467AD22B4805@redhat.com> References: , <28A1866C-F2C6-4BAC-BE73-467AD22B4805@redhat.com> Message-ID: Hello Slawek, Thank your for your fast response. This means in regard of a Stein-Deployment with Linuxbridges no Perimeter-Firewall is offered anymore. Are there plans to remedy this deficiency in the next releases? Cheers, Ralf T. ________________________________ Von: Slawek Kaplonski Gesendet: Donnerstag, 11. Juli 2019 10:04:02 An: Teckelmann, Ralf, NMU-OIP Cc: openstack-discuss at lists.openstack.org Betreff: Re: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' Hi, FWaaS v1 was deprecated since some time and was removed completely in Stein release. > On 11 Jul 2019, at 09:28, Teckelmann, Ralf, NMU-OIP wrote: > > Good Morning everyone, > > We like to have FWaaS enabled for a Stein-based OpenStack installation. > Using linuxbridges we are not able to use FWaaS_v2, because it only seems to work with ovs. > > We thus tried FWaaS (v1) following https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_openstack-2Dansible-2Dos-5Fneutron_latest_configure-2Dnetwork-2Dservices.html-23firewall-2Dservice-2Doptional&d=DwIFaQ&c=vo2ie5TPcLdcgWuLVH4y8lsbGPqIayH3XbK3gK82Oco&r=WXex93lsaiQ-z7CeZkHv93lzt4fdCRIPXloSPQEU7CM&m=mRJxK4Dne35uMLvIxZWOXNeMxXzMcUTsQQd1yrgQ7kM&s=9KmdvZINwdij6mV-kMqE6S94CMiK4z8yO1b7cfXNhv8&e= . > However, all we get from it is (1). > > Are we missing a point or is FWaaS_V1 just not supported in Stein anymore? > If so, this would mean for a setup Stein+Linuxbridges no FWaaS is actually available, right? > > (1) > grep firewall /var/log/neutron/neutron-server.log > 2019-07-05 10:10:55.693 29793 ERROR neutron_lib.utils.runtime NoMatches: No'neutron.service_plugins' driver found, looking for 'firewall' > 2019-07-05 10:10:55.694 29793 ERROR neutron.manager [req-394624b6-e638-45ec-be7c-ce86793fdbc4 - - - - -] Plugin 'firewall' not found. > 2019-07-05 10:11:00.046 29979 INFO neutron.manager [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Loading Plugin: firewall > 2019-07-05 10:11:00.046 29979 ERROR neutron_lib.utils.runtime [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Error loading class by alias: NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' > > Best regards, > > Ralf T. — Slawek Kaplonski Senior software engineer Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Thu Jul 11 09:27:49 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Thu, 11 Jul 2019 11:27:49 +0200 Subject: [queens][nova] nova host-evacuate errot In-Reply-To: References: Message-ID: I am sorry. For simulating an host crash I used a wrong procedure. Using "echo 'c' > /proc/sysrq-trigger" all work fine Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano < ignaziocassano at gmail.com> ha scritto: > Hello All, > on ocata when I poweroff a node with active instance , doing a nova > host-evacuate works fine > and instances are restartd on an active node. > On queens it does non evacuate instances but nova-api reports for each > instance the following: > > 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi > [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 > c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: > Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is > in task_state powering-off > > So it poweroff all instance on the failed node but does not start them on > active nodes > > What is changed ? > Ignazio > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jyotishri403 at gmail.com Thu Jul 11 09:37:19 2019 From: jyotishri403 at gmail.com (Jyoti Dahiwele) Date: Thu, 11 Jul 2019 15:07:19 +0530 Subject: root disk for instance Message-ID: Dear Team, Please clear me my following doubts. When I use image from source option and mini flavor to create an instace, from which storage pool instance will get root disk ? From cinder or glance? -------------- next part -------------- An HTML attachment was scrubbed... URL: From manulachathurika at gmail.com Thu Jul 11 09:41:03 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Thu, 11 Jul 2019 15:11:03 +0530 Subject: g-placement / g-api didn't start in devstack Message-ID: I'm installing DevStack in Ubuntu 18.04. I'm using stable/stein branch for the installation. When I running the stach.sh I'm getting following error. I have updated the timeout to 300 seconds as well. But still I'm not able to run the DevStack. Could you please help me on this issue? Is there specific python version do I need to install. In my PC I have python version 3.6.8. ++:: curl -g -k --noproxy '*' -s -o /dev/null -w '%{http_code}' http://192.168.9.10/image +:: [[ 503 == 503 ]] +:: sleep 1 +functions:wait_for_service:431 rval=124 +functions:wait_for_service:436 time_stop wait_for_service +functions-common:time_stop:2317 local name +functions-common:time_stop:2318 local end_time +functions-common:time_stop:2319 local elapsed_time +functions-common:time_stop:2320 local total +functions-common:time_stop:2321 local start_time +functions-common:time_stop:2323 name=wait_for_service +functions-common:time_stop:2324 start_time=1562654135525 +functions-common:time_stop:2326 [[ -z 1562654135525 ]] ++functions-common:time_stop:2329 date +%s%3N +functions-common:time_stop:2329 end_time=1562654195603 +functions-common:time_stop:2330 elapsed_time=60078 +functions-common:time_stop:2331 total=93 +functions-common:time_stop:2333 _TIME_START[$name]= +functions-common:time_stop:2334 _TIME_TOTAL[$name]=60171 +functions:wait_for_service:437 return 124 +lib/glance:start_glance:353 die 353 'g-api did not start' +functions-common:die:195 local exitcode=0 [Call Trace] ./stack.sh:1261:start_glance /opt/stack/devstack/lib/glance:353:die [ERROR] /opt/stack/devstack/lib/glance:353 g-api did not start Error on exit Traceback (most recent call last): File "/opt/stack/devstack/tools/worlddump.py", line 255, in sys.exit(main()) File "/opt/stack/devstack/tools/worlddump.py", line 239, in main Thanks ! -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayachander.it at gmail.com Thu Jul 11 09:48:21 2019 From: jayachander.it at gmail.com (Jay See) Date: Thu, 11 Jul 2019 11:48:21 +0200 Subject: [queens][nova] nova host-evacuate errot In-Reply-To: References: Message-ID: Hi Ignazio, I am trying to evacuate the compute host on older version (mitaka). Could please share the process you followed. I am not able to succeed with openstack live-migration fails with error message (this is known issue in older versions) and nova live-ligration - nothing happens even after initiating VM migration. It is almost 4 days. ~Jay. On Thu, Jul 11, 2019 at 11:31 AM Ignazio Cassano wrote: > I am sorry. > For simulating an host crash I used a wrong procedure. > Using "echo 'c' > /proc/sysrq-trigger" all work fine > > Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano < > ignaziocassano at gmail.com> ha scritto: > >> Hello All, >> on ocata when I poweroff a node with active instance , doing a nova >> host-evacuate works fine >> and instances are restartd on an active node. >> On queens it does non evacuate instances but nova-api reports for each >> instance the following: >> >> 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi >> [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 >> c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: >> Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is >> in task_state powering-off >> >> So it poweroff all instance on the failed node but does not start them on >> active nodes >> >> What is changed ? >> Ignazio >> >> >> -- ​ P *SAVE PAPER – Please do not print this e-mail unless absolutely necessary.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Thu Jul 11 09:52:27 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Thu, 11 Jul 2019 11:52:27 +0200 Subject: [queens][nova] nova host-evacuate errot In-Reply-To: References: Message-ID: Hi Jay, would you like to evacuate a failed compute node or evacuate a running compute node ? Ignazio Il giorno gio 11 lug 2019 alle ore 11:48 Jay See ha scritto: > Hi Ignazio, > > I am trying to evacuate the compute host on older version (mitaka). > Could please share the process you followed. I am not able to succeed with > openstack live-migration fails with error message (this is known issue in > older versions) and nova live-ligration - nothing happens even after > initiating VM migration. It is almost 4 days. > > ~Jay. > > On Thu, Jul 11, 2019 at 11:31 AM Ignazio Cassano > wrote: > >> I am sorry. >> For simulating an host crash I used a wrong procedure. >> Using "echo 'c' > /proc/sysrq-trigger" all work fine >> >> Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano < >> ignaziocassano at gmail.com> ha scritto: >> >>> Hello All, >>> on ocata when I poweroff a node with active instance , doing a nova >>> host-evacuate works fine >>> and instances are restartd on an active node. >>> On queens it does non evacuate instances but nova-api reports for each >>> instance the following: >>> >>> 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi >>> [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 >>> c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: >>> Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is >>> in task_state powering-off >>> >>> So it poweroff all instance on the failed node but does not start them >>> on active nodes >>> >>> What is changed ? >>> Ignazio >>> >>> >>> > > -- > ​ > P *SAVE PAPER – Please do not print this e-mail unless absolutely > necessary.* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Thu Jul 11 09:53:26 2019 From: eblock at nde.ag (Eugen Block) Date: Thu, 11 Jul 2019 09:53:26 +0000 Subject: root disk for instance In-Reply-To: Message-ID: <20190711095326.Horde._eRtJ1r8ufNJjxxy9JnnVe-@webmail.nde.ag> Hi, it's always glance that serves the images, it just depends on how you decide to create the instance, ephemeral or persistent disks. You can find more information about storage concepts in [1]. If I'm not completely wrong, since Newton release the default in the Horizon settings is to create an instance from volume, so it would be a persistent disk managed by cinder (the volume persists after the instance has been deleted, this is also configurable). The image is downloaded from glance into a volume on your volume server. If you change the Horizon behavior or if you launch an instance from the cli you'd get an ephemeral disk by nova, depending on your storage backend this would be a local copy of the image on the compute node(s) or something related in your storage backend, e.g. an rbd object in ceph. Does this clear it up a bit? Regards, Eugen [1] https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html Zitat von Jyoti Dahiwele : > Dear Team, > > Please clear me my following doubts. > When I use image from source option and mini flavor to create an instace, > from which storage pool instance will get root disk ? From cinder or glance? From jayachander.it at gmail.com Thu Jul 11 09:57:34 2019 From: jayachander.it at gmail.com (Jay See) Date: Thu, 11 Jul 2019 11:57:34 +0200 Subject: [queens][nova] nova host-evacuate errot In-Reply-To: References: Message-ID: Hi , I have tried on a failed compute node which is in power off state now. I have tried on a running compute node, no errors. But nothing happens. On running compute node - Disabled the compute service and tried migration also. May be I might have not followed proper steps. Just wanted to know the steps you have followed. Otherwise, I was planning to manual migration also if possible. ~Jay. On Thu, Jul 11, 2019 at 11:52 AM Ignazio Cassano wrote: > Hi Jay, > would you like to evacuate a failed compute node or evacuate a running > compute node ? > > Ignazio > > Il giorno gio 11 lug 2019 alle ore 11:48 Jay See > ha scritto: > >> Hi Ignazio, >> >> I am trying to evacuate the compute host on older version (mitaka). >> Could please share the process you followed. I am not able to succeed >> with openstack live-migration fails with error message (this is known issue >> in older versions) and nova live-ligration - nothing happens even after >> initiating VM migration. It is almost 4 days. >> >> ~Jay. >> >> On Thu, Jul 11, 2019 at 11:31 AM Ignazio Cassano < >> ignaziocassano at gmail.com> wrote: >> >>> I am sorry. >>> For simulating an host crash I used a wrong procedure. >>> Using "echo 'c' > /proc/sysrq-trigger" all work fine >>> >>> Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano < >>> ignaziocassano at gmail.com> ha scritto: >>> >>>> Hello All, >>>> on ocata when I poweroff a node with active instance , doing a nova >>>> host-evacuate works fine >>>> and instances are restartd on an active node. >>>> On queens it does non evacuate instances but nova-api reports for each >>>> instance the following: >>>> >>>> 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi >>>> [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 >>>> c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: >>>> Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is >>>> in task_state powering-off >>>> >>>> So it poweroff all instance on the failed node but does not start them >>>> on active nodes >>>> >>>> What is changed ? >>>> Ignazio >>>> >>>> >>>> >> >> -- >> ​ >> P *SAVE PAPER – Please do not print this e-mail unless absolutely >> necessary.* >> > -- ​ P *SAVE PAPER – Please do not print this e-mail unless absolutely necessary.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdent+os at anticdent.org Thu Jul 11 09:59:24 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Thu, 11 Jul 2019 10:59:24 +0100 (BST) Subject: g-placement / g-api didn't start in devstack In-Reply-To: References: Message-ID: On Thu, 11 Jul 2019, Manula Thantriwatte wrote: > I'm installing DevStack in Ubuntu 18.04. I'm using stable/stein branch for > the installation. When I running the stach.sh I'm getting following error. > I have updated the timeout to 300 seconds as well. But still I'm not able > to run the DevStack. Could you please help me on this issue? Is there > specific python version do I need to install. In my PC I have python > version 3.6.8. You might be seeing the same issue as described in http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007616.html so you might try the suggestions in there. (It might also be something else). -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From skaplons at redhat.com Thu Jul 11 10:30:49 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 11 Jul 2019 12:30:49 +0200 Subject: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' In-Reply-To: References: <28A1866C-F2C6-4BAC-BE73-467AD22B4805@redhat.com> Message-ID: Hi, AFAICT there is no many still active developers of neutron-fwaas project and I don’t know about such plans currently. > On 11 Jul 2019, at 11:23, Teckelmann, Ralf, NMU-OIP wrote: > > Hello Slawek, > > Thank your for your fast response. > This means in regard of a Stein-Deployment with Linuxbridges no Perimeter-Firewall is offered anymore. > Are there plans to remedy this deficiency in the next releases? > > Cheers, > > Ralf T. > Von: Slawek Kaplonski > Gesendet: Donnerstag, 11. Juli 2019 10:04:02 > An: Teckelmann, Ralf, NMU-OIP > Cc: openstack-discuss at lists.openstack.org > Betreff: Re: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' > > Hi, > > FWaaS v1 was deprecated since some time and was removed completely in Stein release. > > > On 11 Jul 2019, at 09:28, Teckelmann, Ralf, NMU-OIP wrote: > > > > Good Morning everyone, > > > > We like to have FWaaS enabled for a Stein-based OpenStack installation. > > Using linuxbridges we are not able to use FWaaS_v2, because it only seems to work with ovs. > > > > We thus tried FWaaS (v1) following https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_openstack-2Dansible-2Dos-5Fneutron_latest_configure-2Dnetwork-2Dservices.html-23firewall-2Dservice-2Doptional&d=DwIFaQ&c=vo2ie5TPcLdcgWuLVH4y8lsbGPqIayH3XbK3gK82Oco&r=WXex93lsaiQ-z7CeZkHv93lzt4fdCRIPXloSPQEU7CM&m=mRJxK4Dne35uMLvIxZWOXNeMxXzMcUTsQQd1yrgQ7kM&s=9KmdvZINwdij6mV-kMqE6S94CMiK4z8yO1b7cfXNhv8&e= . > > However, all we get from it is (1). > > > > Are we missing a point or is FWaaS_V1 just not supported in Stein anymore? > > If so, this would mean for a setup Stein+Linuxbridges no FWaaS is actually available, right? > > > > (1) > > grep firewall /var/log/neutron/neutron-server.log > > 2019-07-05 10:10:55.693 29793 ERROR neutron_lib.utils.runtime NoMatches: No'neutron.service_plugins' driver found, looking for 'firewall' > > 2019-07-05 10:10:55.694 29793 ERROR neutron.manager [req-394624b6-e638-45ec-be7c-ce86793fdbc4 - - - - -] Plugin 'firewall' not found. > > 2019-07-05 10:11:00.046 29979 INFO neutron.manager [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Loading Plugin: firewall > > 2019-07-05 10:11:00.046 29979 ERROR neutron_lib.utils.runtime [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Error loading class by alias: NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' > > > > Best regards, > > > > Ralf T. > > — > Slawek Kaplonski > Senior software engineer > Red Hat > — Slawek Kaplonski Senior software engineer Red Hat From gmann at ghanshyammann.com Thu Jul 11 10:41:49 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Thu, 11 Jul 2019 19:41:49 +0900 Subject: [nova] API updates week 19-28 Message-ID: <16be09ffa52.ccb8c25789487.3681219850412168320@ghanshyammann.com> Hello Everyone, Please find the Nova API updates of this week. API Related BP : ============ COMPLETED: 1. Support adding description while locking an instance: - https://blueprints.launchpad.net/nova/+spec/add-locked-reason Code Ready for Review: ------------------------------ 1. Add host and hypervisor_hostname flag to create server - Topic: https://review.opendev.org/#/q/topic:bp/add-host-and-hypervisor-hostname-flag-to-create-server+(status:open+OR+status:merged) - Weekly Progress: matt is +2 on this. Need another core reviewer. 2. Specifying az when restore shelved server - Topic: https://review.opendev.org/#/q/topic:bp/support-specifying-az-when-restore-shelved-server+(status:open+OR+status:merged) - Weekly Progress: Brin updated the patch after review comments. Ready for another round of review. 3. Nova API cleanup - Topic: https://review.opendev.org/#/c/666889/ - Weekly Progress: Code is up for review. Specs are merged and code in-progress: ------------------------------ ------------------ 4. Nova API policy improvement - Topic: https://review.openstack.org/#/q/topic:bp/policy-default-refresh+(status:open+OR+status:merged) - Weekly Progress: Spec is merged. Started the testing coverage of the existing policies with little refactoring on John PoC. 5. Detach and attach boot volumes: - Topic: https://review.openstack.org/#/q/topic:bp/detach-boot-volume+(status:open+OR+status:merged) - Weekly Progress: No Progress. Patches are in merge conflict. Spec Ready for Review: ----------------------------- 1. Support for changing deleted_on_termination after boot -Spec: https://review.openstack.org/#/c/580336/ - Weekly Progress: No update this week. Pending on Lee Yarwood proposal after PTG discussion. 3. Support delete_on_termination in volume attach api -Spec: https://review.openstack.org/#/c/612949/ - Weekly Progress: No updates this week. matt recommend to merging this with 580336 which is pending on Lee Yarwood proposal. Previously approved Spec needs to be re-proposed for Train: --------------------------------------------------------------------------- 1. Servers Ips non-unique network names : - https://blueprints.launchpad.net/nova/+spec/servers-ips-non-unique-network-names - https://review.openstack.org/#/q/topic:bp/servers-ips-non-unique-network-names+(status:open+OR+status:merged) - I remember I planned this to re-propose but could not get time. If anyone would like to help on this please repropose. otherwise I will start this in U cycle. 2. Volume multiattach enhancements: - https://blueprints.launchpad.net/nova/+spec/volume-multiattach-enhancements - https://review.openstack.org/#/q/topic:bp/volume-multiattach-enhancements+(status:open+OR+status:merged) - This also need volutneer - http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007411.html Others: 1. Add API ref guideline for body text - Need another +2 on this - https://review.opendev.org/#/c/668234/ - after that, 2 api-ref are left to fix. Bugs: ==== No progress report in this week. NOTE- There might be some bug which is not tagged as 'api' or 'api-ref', those are not in the above list. Tag such bugs so that we can keep our eyes. -gmann From aheczko at mirantis.com Thu Jul 11 10:55:25 2019 From: aheczko at mirantis.com (Adam Heczko) Date: Thu, 11 Jul 2019 12:55:25 +0200 Subject: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' In-Reply-To: References: <28A1866C-F2C6-4BAC-BE73-467AD22B4805@redhat.com> Message-ID: Hi Ralf, WDYM saying 'no Perimeter-Firewall is offered anymore'? OpenStack with OVS ML2 provides a security groups, which is considered a 'perimeter firewall'. On Thu, Jul 11, 2019 at 12:35 PM Slawek Kaplonski wrote: > Hi, > > AFAICT there is no many still active developers of neutron-fwaas project > and I don’t know about such plans currently. > > > On 11 Jul 2019, at 11:23, Teckelmann, Ralf, NMU-OIP < > ralf.teckelmann at bertelsmann.de> wrote: > > > > Hello Slawek, > > > > Thank your for your fast response. > > This means in regard of a Stein-Deployment with Linuxbridges no > Perimeter-Firewall is offered anymore. > > Are there plans to remedy this deficiency in the next releases? > > > > Cheers, > > > > Ralf T. > > Von: Slawek Kaplonski > > Gesendet: Donnerstag, 11. Juli 2019 10:04:02 > > An: Teckelmann, Ralf, NMU-OIP > > Cc: openstack-discuss at lists.openstack.org > > Betreff: Re: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' > driver found, looking for 'firewall' > > > > Hi, > > > > FWaaS v1 was deprecated since some time and was removed completely in > Stein release. > > > > > On 11 Jul 2019, at 09:28, Teckelmann, Ralf, NMU-OIP < > ralf.teckelmann at bertelsmann.de> wrote: > > > > > > Good Morning everyone, > > > > > > We like to have FWaaS enabled for a Stein-based OpenStack installation. > > > Using linuxbridges we are not able to use FWaaS_v2, because it only > seems to work with ovs. > > > > > > We thus tried FWaaS (v1) following > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_openstack-2Dansible-2Dos-5Fneutron_latest_configure-2Dnetwork-2Dservices.html-23firewall-2Dservice-2Doptional&d=DwIFaQ&c=vo2ie5TPcLdcgWuLVH4y8lsbGPqIayH3XbK3gK82Oco&r=WXex93lsaiQ-z7CeZkHv93lzt4fdCRIPXloSPQEU7CM&m=mRJxK4Dne35uMLvIxZWOXNeMxXzMcUTsQQd1yrgQ7kM&s=9KmdvZINwdij6mV-kMqE6S94CMiK4z8yO1b7cfXNhv8&e= > . > > > However, all we get from it is (1). > > > > > > Are we missing a point or is FWaaS_V1 just not supported in Stein > anymore? > > > If so, this would mean for a setup Stein+Linuxbridges no FWaaS is > actually available, right? > > > > > > (1) > > > grep firewall /var/log/neutron/neutron-server.log > > > 2019-07-05 10:10:55.693 29793 ERROR neutron_lib.utils.runtime > NoMatches: No'neutron.service_plugins' driver found, looking for 'firewall' > > > 2019-07-05 10:10:55.694 29793 ERROR neutron.manager > [req-394624b6-e638-45ec-be7c-ce86793fdbc4 - - - - -] Plugin 'firewall' not > found. > > > 2019-07-05 10:11:00.046 29979 INFO neutron.manager > [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Loading Plugin: > firewall > > > 2019-07-05 10:11:00.046 29979 ERROR neutron_lib.utils.runtime > [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Error loading class by > alias: NoMatches: No 'neutron.service_plugins' driver found, looking for > 'firewall' > > > > > > Best regards, > > > > > > Ralf T. > > > > — > > Slawek Kaplonski > > Senior software engineer > > Red Hat > > > > — > Slawek Kaplonski > Senior software engineer > Red Hat > > > -- Adam Heczko Principal Security Architect @ Mirantis Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssbarnea at redhat.com Thu Jul 11 11:01:25 2019 From: ssbarnea at redhat.com (Sorin Sbarnea) Date: Thu, 11 Jul 2019 12:01:25 +0100 Subject: [doc8] future development and maintenance Message-ID: <9D091EF4-60F2-490C-BF76-AAD7A68FB8A1@redhat.com> It seems that the doc8 project is lacking some love, regardless the fact that is used by >90 projects from opendev.org. https://review.opendev.org/#/q/project:x/doc8 Last merge and release was more than 2 years ago and no reviews were performed either. I think it would be in our interest to assure that doc8 maintenance continues and that we can keep it usable. I would like to propose extenting the list of cores from the current 4 ones that I already listed in CC with 3 more, so we can effectively make a change that gets merged and later released (anyone willing to help?) If current cores agree, I would be happy to help with maintenance. I estimate that the effort needed would likely be less than 1h/month in longer term. If there is a desire to move it to github/travis, I would not mind either. Thanks Sorin Sbarnea Red Hat TripleO CI From ralf.teckelmann at bertelsmann.de Thu Jul 11 11:13:14 2019 From: ralf.teckelmann at bertelsmann.de (Teckelmann, Ralf, NMU-OIP) Date: Thu, 11 Jul 2019 11:13:14 +0000 Subject: AW: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' In-Reply-To: References: <28A1866C-F2C6-4BAC-BE73-467AD22B4805@redhat.com> Message-ID: Hello Adam, You may missed the part „in regard of a Stein-Deployment with Linuxbridges” of my question. So OVS is not relevant, as I understand the mutual exclusion of linux bridges and ovs. Cheers, Ralf T. Von: Adam Heczko Gesendet: Donnerstag, 11. Juli 2019 12:55 An: Slawek Kaplonski Cc: Teckelmann, Ralf, NMU-OIP ; openstack-discuss at lists.openstack.org Betreff: Re: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' Hi Ralf, WDYM saying 'no Perimeter-Firewall is offered anymore'? OpenStack with OVS ML2 provides a security groups, which is considered a 'perimeter firewall'. On Thu, Jul 11, 2019 at 12:35 PM Slawek Kaplonski > wrote: Hi, AFAICT there is no many still active developers of neutron-fwaas project and I don’t know about such plans currently. > On 11 Jul 2019, at 11:23, Teckelmann, Ralf, NMU-OIP > wrote: > > Hello Slawek, > > Thank your for your fast response. > This means in regard of a Stein-Deployment with Linuxbridges no Perimeter-Firewall is offered anymore. > Are there plans to remedy this deficiency in the next releases? > > Cheers, > > Ralf T. > Von: Slawek Kaplonski > > Gesendet: Donnerstag, 11. Juli 2019 10:04:02 > An: Teckelmann, Ralf, NMU-OIP > Cc: openstack-discuss at lists.openstack.org > Betreff: Re: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' > > Hi, > > FWaaS v1 was deprecated since some time and was removed completely in Stein release. > > > On 11 Jul 2019, at 09:28, Teckelmann, Ralf, NMU-OIP > wrote: > > > > Good Morning everyone, > > > > We like to have FWaaS enabled for a Stein-based OpenStack installation. > > Using linuxbridges we are not able to use FWaaS_v2, because it only seems to work with ovs. > > > > We thus tried FWaaS (v1) following https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_openstack-2Dansible-2Dos-5Fneutron_latest_configure-2Dnetwork-2Dservices.html-23firewall-2Dservice-2Doptional&d=DwIFaQ&c=vo2ie5TPcLdcgWuLVH4y8lsbGPqIayH3XbK3gK82Oco&r=WXex93lsaiQ-z7CeZkHv93lzt4fdCRIPXloSPQEU7CM&m=mRJxK4Dne35uMLvIxZWOXNeMxXzMcUTsQQd1yrgQ7kM&s=9KmdvZINwdij6mV-kMqE6S94CMiK4z8yO1b7cfXNhv8&e= . > > However, all we get from it is (1). > > > > Are we missing a point or is FWaaS_V1 just not supported in Stein anymore? > > If so, this would mean for a setup Stein+Linuxbridges no FWaaS is actually available, right? > > > > (1) > > grep firewall /var/log/neutron/neutron-server.log > > 2019-07-05 10:10:55.693 29793 ERROR neutron_lib.utils.runtime NoMatches: No'neutron.service_plugins' driver found, looking for 'firewall' > > 2019-07-05 10:10:55.694 29793 ERROR neutron.manager [req-394624b6-e638-45ec-be7c-ce86793fdbc4 - - - - -] Plugin 'firewall' not found. > > 2019-07-05 10:11:00.046 29979 INFO neutron.manager [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Loading Plugin: firewall > > 2019-07-05 10:11:00.046 29979 ERROR neutron_lib.utils.runtime [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Error loading class by alias: NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' > > > > Best regards, > > > > Ralf T. > > — > Slawek Kaplonski > Senior software engineer > Red Hat > — Slawek Kaplonski Senior software engineer Red Hat -- Adam Heczko Principal Security Architect @ Mirantis Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetan.trellu at incloudus.com Thu Jul 11 11:14:47 2019 From: gaetan.trellu at incloudus.com (=?ISO-8859-1?Q?Ga=EBtan_Trellu?=) Date: Thu, 11 Jul 2019 07:14:47 -0400 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: Message-ID: <35400f83-c29d-475a-8d36-d56b3cf16d30@email.android.com> An HTML attachment was scrubbed... URL: From skaplons at redhat.com Thu Jul 11 11:22:12 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 11 Jul 2019 13:22:12 +0200 Subject: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' In-Reply-To: References: <28A1866C-F2C6-4BAC-BE73-467AD22B4805@redhat.com> Message-ID: Hi, Security groups are supported by both Linuxbridge and OVS agents. But this is different solution than FWaaS. Security groups are applied on port’s level, not on router. > On 11 Jul 2019, at 13:13, Teckelmann, Ralf, NMU-OIP wrote: > > Hello Adam, > > You may missed the part „in regard of a Stein-Deployment with Linuxbridges” of my question. > So OVS is not relevant, as I understand the mutual exclusion of linux bridges and ovs. > > Cheers, > > Ralf T. > > Von: Adam Heczko > Gesendet: Donnerstag, 11. Juli 2019 12:55 > An: Slawek Kaplonski > Cc: Teckelmann, Ralf, NMU-OIP ; openstack-discuss at lists.openstack.org > Betreff: Re: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' > > Hi Ralf, WDYM saying 'no Perimeter-Firewall is offered anymore'? > OpenStack with OVS ML2 provides a security groups, which is considered a 'perimeter firewall'. > > On Thu, Jul 11, 2019 at 12:35 PM Slawek Kaplonski wrote: > Hi, > > AFAICT there is no many still active developers of neutron-fwaas project and I don’t know about such plans currently. > > > On 11 Jul 2019, at 11:23, Teckelmann, Ralf, NMU-OIP wrote: > > > > Hello Slawek, > > > > Thank your for your fast response. > > This means in regard of a Stein-Deployment with Linuxbridges no Perimeter-Firewall is offered anymore. > > Are there plans to remedy this deficiency in the next releases? > > > > Cheers, > > > > Ralf T. > > Von: Slawek Kaplonski > > Gesendet: Donnerstag, 11. Juli 2019 10:04:02 > > An: Teckelmann, Ralf, NMU-OIP > > Cc: openstack-discuss at lists.openstack.org > > Betreff: Re: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' > > > > Hi, > > > > FWaaS v1 was deprecated since some time and was removed completely in Stein release. > > > > > On 11 Jul 2019, at 09:28, Teckelmann, Ralf, NMU-OIP wrote: > > > > > > Good Morning everyone, > > > > > > We like to have FWaaS enabled for a Stein-based OpenStack installation. > > > Using linuxbridges we are not able to use FWaaS_v2, because it only seems to work with ovs. > > > > > > We thus tried FWaaS (v1) following https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_openstack-2Dansible-2Dos-5Fneutron_latest_configure-2Dnetwork-2Dservices.html-23firewall-2Dservice-2Doptional&d=DwIFaQ&c=vo2ie5TPcLdcgWuLVH4y8lsbGPqIayH3XbK3gK82Oco&r=WXex93lsaiQ-z7CeZkHv93lzt4fdCRIPXloSPQEU7CM&m=mRJxK4Dne35uMLvIxZWOXNeMxXzMcUTsQQd1yrgQ7kM&s=9KmdvZINwdij6mV-kMqE6S94CMiK4z8yO1b7cfXNhv8&e= . > > > However, all we get from it is (1). > > > > > > Are we missing a point or is FWaaS_V1 just not supported in Stein anymore? > > > If so, this would mean for a setup Stein+Linuxbridges no FWaaS is actually available, right? > > > > > > (1) > > > grep firewall /var/log/neutron/neutron-server.log > > > 2019-07-05 10:10:55.693 29793 ERROR neutron_lib.utils.runtime NoMatches: No'neutron.service_plugins' driver found, looking for 'firewall' > > > 2019-07-05 10:10:55.694 29793 ERROR neutron.manager [req-394624b6-e638-45ec-be7c-ce86793fdbc4 - - - - -] Plugin 'firewall' not found. > > > 2019-07-05 10:11:00.046 29979 INFO neutron.manager [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Loading Plugin: firewall > > > 2019-07-05 10:11:00.046 29979 ERROR neutron_lib.utils.runtime [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Error loading class by alias: NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' > > > > > > Best regards, > > > > > > Ralf T. > > > > — > > Slawek Kaplonski > > Senior software engineer > > Red Hat > > > > — > Slawek Kaplonski > Senior software engineer > Red Hat > > > > > -- > Adam Heczko > Principal Security Architect @ Mirantis Inc. — Slawek Kaplonski Senior software engineer Red Hat From tomas.bredar at gmail.com Thu Jul 11 11:26:25 2019 From: tomas.bredar at gmail.com (=?UTF-8?B?VG9tw6HFoSBCcmVkw6Fy?=) Date: Thu, 11 Jul 2019 13:26:25 +0200 Subject: [tripleo][cinder][netapp] Message-ID: Hi community, I'm trying to define multiple NetApp storage backends via Tripleo installer. According to [1] the puppet manifest supports multiple backends. The current templates [2] [3] support only single backend. Does anyone know how to define multiple netapp backends in the tripleo-heat environment files / templates? You help is appreciated. Tomas [1] https://opendev.org/openstack/puppet-cinder/src/branch/stable/queens/manifests/backend/netapp.pp [2] https://opendev.org/openstack/tripleo-heat-templates/src/branch/stable/queens/environments/cinder-netapp-config.yaml [3] https://opendev.org/openstack/tripleo-heat-templates/src/branch/stable/queens/puppet/services/cinder-backend-netapp.yaml -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Thu Jul 11 11:33:13 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Thu, 11 Jul 2019 13:33:13 +0200 Subject: [queens][nova] nova host-evacuate errot In-Reply-To: References: Message-ID: Ok Jay, let me to describe my environment. I have an openstack made up of 3 controllers nodes ad several compute nodes. The controller nodes services are controlled by pacemaker and the compute nodes services are controlled by remote pacemaker. My hardware is Dell so I am using ipmi fencing device . I wrote a service controlled by pacemaker: this service controls if a compude node fails and for avoiding split brains if a compute node does nod respond on the management network and on storage network the stonith poweroff the node and then execute a nova host-evacuate. Anycase to have a simulation before writing the service I described above you can do as follows: connect on one compute node where some virtual machines are running run the command: echo 'c' > /proc/sysrq-trigger (it stops immediately the node like in case of failure) On a controller node run: nova host-evacuate "name of failed compute node" Instances running on the failed compute node should be restarted on another compute node Ignazio Il giorno gio 11 lug 2019 alle ore 11:57 Jay See ha scritto: > Hi , > > I have tried on a failed compute node which is in power off state now. > I have tried on a running compute node, no errors. But nothing happens. > On running compute node - Disabled the compute service and tried migration > also. > > May be I might have not followed proper steps. Just wanted to know the > steps you have followed. Otherwise, I was planning to manual migration also > if possible. > ~Jay. > > On Thu, Jul 11, 2019 at 11:52 AM Ignazio Cassano > wrote: > >> Hi Jay, >> would you like to evacuate a failed compute node or evacuate a running >> compute node ? >> >> Ignazio >> >> Il giorno gio 11 lug 2019 alle ore 11:48 Jay See < >> jayachander.it at gmail.com> ha scritto: >> >>> Hi Ignazio, >>> >>> I am trying to evacuate the compute host on older version (mitaka). >>> Could please share the process you followed. I am not able to succeed >>> with openstack live-migration fails with error message (this is known issue >>> in older versions) and nova live-ligration - nothing happens even after >>> initiating VM migration. It is almost 4 days. >>> >>> ~Jay. >>> >>> On Thu, Jul 11, 2019 at 11:31 AM Ignazio Cassano < >>> ignaziocassano at gmail.com> wrote: >>> >>>> I am sorry. >>>> For simulating an host crash I used a wrong procedure. >>>> Using "echo 'c' > /proc/sysrq-trigger" all work fine >>>> >>>> Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano < >>>> ignaziocassano at gmail.com> ha scritto: >>>> >>>>> Hello All, >>>>> on ocata when I poweroff a node with active instance , doing a nova >>>>> host-evacuate works fine >>>>> and instances are restartd on an active node. >>>>> On queens it does non evacuate instances but nova-api reports for each >>>>> instance the following: >>>>> >>>>> 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi >>>>> [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 >>>>> c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: >>>>> Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is >>>>> in task_state powering-off >>>>> >>>>> So it poweroff all instance on the failed node but does not start them >>>>> on active nodes >>>>> >>>>> What is changed ? >>>>> Ignazio >>>>> >>>>> >>>>> >>> >>> -- >>> ​ >>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>> necessary.* >>> >> > > -- > ​ > P *SAVE PAPER – Please do not print this e-mail unless absolutely > necessary.* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aheczko at mirantis.com Thu Jul 11 11:44:49 2019 From: aheczko at mirantis.com (Adam Heczko) Date: Thu, 11 Jul 2019 13:44:49 +0200 Subject: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' In-Reply-To: References: <28A1866C-F2C6-4BAC-BE73-467AD22B4805@redhat.com> Message-ID: Exacly Slawek. Ralph I was referring to the sentence 'Perimeter-Firewall' OpenStack provides a Perimeter-Firewall and that is a Security Groups. https://docs.openstack.org/nova/queens/admin/security-groups.html SG (Security Groups) is something different than FWaaS. Though FWaaS to some degree could also provide a SG functionality, as it can bind to AFAIK and Neutron port. On Thu, Jul 11, 2019 at 1:22 PM Slawek Kaplonski wrote: > Hi, > > Security groups are supported by both Linuxbridge and OVS agents. But this > is different solution than FWaaS. Security groups are applied on port’s > level, not on router. > > > On 11 Jul 2019, at 13:13, Teckelmann, Ralf, NMU-OIP < > ralf.teckelmann at bertelsmann.de> wrote: > > > > Hello Adam, > > > > You may missed the part „in regard of a Stein-Deployment with > Linuxbridges” of my question. > > So OVS is not relevant, as I understand the mutual exclusion of linux > bridges and ovs. > > > > Cheers, > > > > Ralf T. > > > > Von: Adam Heczko > > Gesendet: Donnerstag, 11. Juli 2019 12:55 > > An: Slawek Kaplonski > > Cc: Teckelmann, Ralf, NMU-OIP ; > openstack-discuss at lists.openstack.org > > Betreff: Re: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' > driver found, looking for 'firewall' > > > > Hi Ralf, WDYM saying 'no Perimeter-Firewall is offered anymore'? > > OpenStack with OVS ML2 provides a security groups, which is considered a > 'perimeter firewall'. > > > > On Thu, Jul 11, 2019 at 12:35 PM Slawek Kaplonski > wrote: > > Hi, > > > > AFAICT there is no many still active developers of neutron-fwaas project > and I don’t know about such plans currently. > > > > > On 11 Jul 2019, at 11:23, Teckelmann, Ralf, NMU-OIP < > ralf.teckelmann at bertelsmann.de> wrote: > > > > > > Hello Slawek, > > > > > > Thank your for your fast response. > > > This means in regard of a Stein-Deployment with Linuxbridges no > Perimeter-Firewall is offered anymore. > > > Are there plans to remedy this deficiency in the next releases? > > > > > > Cheers, > > > > > > Ralf T. > > > Von: Slawek Kaplonski > > > Gesendet: Donnerstag, 11. Juli 2019 10:04:02 > > > An: Teckelmann, Ralf, NMU-OIP > > > Cc: openstack-discuss at lists.openstack.org > > > Betreff: Re: FWaaS in Stein - NoMatches: No 'neutron.service_plugins' > driver found, looking for 'firewall' > > > > > > Hi, > > > > > > FWaaS v1 was deprecated since some time and was removed completely in > Stein release. > > > > > > > On 11 Jul 2019, at 09:28, Teckelmann, Ralf, NMU-OIP < > ralf.teckelmann at bertelsmann.de> wrote: > > > > > > > > Good Morning everyone, > > > > > > > > We like to have FWaaS enabled for a Stein-based OpenStack > installation. > > > > Using linuxbridges we are not able to use FWaaS_v2, because it only > seems to work with ovs. > > > > > > > > We thus tried FWaaS (v1) following > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_openstack-2Dansible-2Dos-5Fneutron_latest_configure-2Dnetwork-2Dservices.html-23firewall-2Dservice-2Doptional&d=DwIFaQ&c=vo2ie5TPcLdcgWuLVH4y8lsbGPqIayH3XbK3gK82Oco&r=WXex93lsaiQ-z7CeZkHv93lzt4fdCRIPXloSPQEU7CM&m=mRJxK4Dne35uMLvIxZWOXNeMxXzMcUTsQQd1yrgQ7kM&s=9KmdvZINwdij6mV-kMqE6S94CMiK4z8yO1b7cfXNhv8&e= > . > > > > However, all we get from it is (1). > > > > > > > > Are we missing a point or is FWaaS_V1 just not supported in Stein > anymore? > > > > If so, this would mean for a setup Stein+Linuxbridges no FWaaS is > actually available, right? > > > > > > > > (1) > > > > grep firewall /var/log/neutron/neutron-server.log > > > > 2019-07-05 10:10:55.693 29793 ERROR neutron_lib.utils.runtime > NoMatches: No'neutron.service_plugins' driver found, looking for 'firewall' > > > > 2019-07-05 10:10:55.694 29793 ERROR neutron.manager > [req-394624b6-e638-45ec-be7c-ce86793fdbc4 - - - - -] Plugin 'firewall' not > found. > > > > 2019-07-05 10:11:00.046 29979 INFO neutron.manager > [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Loading Plugin: > firewall > > > > 2019-07-05 10:11:00.046 29979 ERROR neutron_lib.utils.runtime > [req-e86af4f4-afae-46d7-ac5e-51585a12083b - - - - -] Error loading class by > alias: NoMatches: No 'neutron.service_plugins' driver found, looking for > 'firewall' > > > > > > > > Best regards, > > > > > > > > Ralf T. > > > > > > — > > > Slawek Kaplonski > > > Senior software engineer > > > Red Hat > > > > > > > — > > Slawek Kaplonski > > Senior software engineer > > Red Hat > > > > > > > > > > -- > > Adam Heczko > > Principal Security Architect @ Mirantis Inc. > > — > Slawek Kaplonski > Senior software engineer > Red Hat > > -- Adam Heczko Principal Security Architect @ Mirantis Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Thu Jul 11 12:03:28 2019 From: smooney at redhat.com (Sean Mooney) Date: Thu, 11 Jul 2019 13:03:28 +0100 Subject: trying to understand steal time with cpu pinning In-Reply-To: <9D8A2486E35F0941A60430473E29F15B017EB2889D@MXDB1.ad.garvan.unsw.edu.au> References: <9D8A2486E35F0941A60430473E29F15B017EB2889D@MXDB1.ad.garvan.unsw.edu.au> Message-ID: <9d112059e4772bccb037907fa984044be069640f.camel@redhat.com> On Thu, 2019-07-11 at 06:15 +0000, Manuel Sopena Ballesteros wrote: > Dear Openstack community, > > Please correct me if I am wrong. > > As far as I understand `steal time > 0` means that the hypervisor has replaced a vcpu with a different one on the > physical cpu. > Also, cpu pinning allocates a vcpu to a physical cpu permanently. > > I have a vm setup with cpu pinning and numa affinity and realized, that cpu steal time is between 1% and 0%. > > Why is that? there are 2 ways that this can happen. 1.) you are not setting hw:emulator_thread_policy to move the qemu emulator threads to a different core. 2.) you have host system process or kernel threads that are stealing guest cpu time like vhost threads. you can prevent 2 using systemd or take teh blunt hammer approch and use the kernel isolcpus parmater but that generally should only be used for realtime systems. openstack does not prevent other host process form running on the vcpu_pin_set, that is left to the operator/installer/os to do. > > Thank you very much > NOTICE > Please consider the environment before printing this email. This message and any attachments are intended for the > addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended > recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this > message in error please notify us at once by return email and then delete both messages. We accept no liability for > the distribution of viruses or similar in electronic communications. This notice should not be removed. Technically disclaimer like ^ are considered poor mailing list etiquette and are not legally enforceable. if you can disable it for the openstack mailing list that would be for the best. the discloser,copy and distribution parts in partcalar can never be honoured by a publicly archived mailing list so this just adds noise to the list. From emilien at redhat.com Thu Jul 11 12:35:20 2019 From: emilien at redhat.com (Emilien Macchi) Date: Thu, 11 Jul 2019 08:35:20 -0400 Subject: [tripleo][cinder][netapp] In-Reply-To: References: Message-ID: On Thu, Jul 11, 2019 at 7:32 AM Tomáš Bredár wrote: > Hi community, > > I'm trying to define multiple NetApp storage backends via Tripleo > installer. > According to [1] the puppet manifest supports multiple backends. > The current templates [2] [3] support only single backend. > Does anyone know how to define multiple netapp backends in the > tripleo-heat environment files / templates? > We don't support that via the templates that you linked, however if you follow this manual you should be able to configure multiple NetApp backends: https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/cinder_custom_backend.html Let us know how it worked! -- Emilien Macchi -------------- next part -------------- An HTML attachment was scrubbed... URL: From jyotishri403 at gmail.com Thu Jul 11 14:21:04 2019 From: jyotishri403 at gmail.com (Jyoti Dahiwele) Date: Thu, 11 Jul 2019 19:51:04 +0530 Subject: root disk for instance In-Reply-To: <20190711095326.Horde._eRtJ1r8ufNJjxxy9JnnVe-@webmail.nde.ag> References: <20190711095326.Horde._eRtJ1r8ufNJjxxy9JnnVe-@webmail.nde.ag> Message-ID: Thanks for your reply. I'm referring this link https://docs.openstack.org/mitaka/admin-guide/compute-images-instances.html to understand about root disk. In this it is saying root disk will come from compute node. What is the location of root disk on compute node? If I want to keep all my vms on shared storage . How to configure it ? Or If I want to keep all my vms on cinder volume. What will be the configuration for it on nova and cinder? On Thu, 11 Jul 2019, 15:24 Eugen Block, wrote: > Hi, > > it's always glance that serves the images, it just depends on how you > decide to create the instance, ephemeral or persistent disks. You can > find more information about storage concepts in [1]. > > If I'm not completely wrong, since Newton release the default in the > Horizon settings is to create an instance from volume, so it would be > a persistent disk managed by cinder (the volume persists after the > instance has been deleted, this is also configurable). The image is > downloaded from glance into a volume on your volume server. > > If you change the Horizon behavior or if you launch an instance from > the cli you'd get an ephemeral disk by nova, depending on your storage > backend this would be a local copy of the image on the compute node(s) > or something related in your storage backend, e.g. an rbd object in > ceph. > > Does this clear it up a bit? > > Regards, > Eugen > > [1] > > https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html > > > Zitat von Jyoti Dahiwele : > > > Dear Team, > > > > Please clear me my following doubts. > > When I use image from source option and mini flavor to create an > instace, > > from which storage pool instance will get root disk ? From cinder or > glance? > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.bishop at beyondhosting.net Thu Jul 11 14:23:42 2019 From: tyler.bishop at beyondhosting.net (Tyler Bishop) Date: Thu, 11 Jul 2019 10:23:42 -0400 Subject: root disk for instance In-Reply-To: References: <20190711095326.Horde._eRtJ1r8ufNJjxxy9JnnVe-@webmail.nde.ag> Message-ID: /var/lib/qemu/instances/instance-hash/blah blah > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Thu Jul 11 14:48:27 2019 From: eblock at nde.ag (Eugen Block) Date: Thu, 11 Jul 2019 14:48:27 +0000 Subject: root disk for instance In-Reply-To: References: <20190711095326.Horde._eRtJ1r8ufNJjxxy9JnnVe-@webmail.nde.ag> Message-ID: <20190711144827.Horde.tglCLyI8aGqVcHo23OPMuxK@webmail.nde.ag> > I'm referring this link > https://docs.openstack.org/mitaka/admin-guide/compute-images-instances.html > to understand about root disk. In this it is saying root disk will come > from compute node. What is the location of root disk on compute node? First, use a later release, Mitaka is quite old, Stein is the current release. If you use a default configuration without any specific backend for your instances they will be located on the compute nodes in /var/lib/nova/instances/. The respective base images would reside in /var/lib/nova/instances/_base, so your compute nodes should have sufficient disk space. > If I want to keep all my vms on shared storage . How to configure it ? That's up to you, there are several backends supported, so you'll have to choose one. Many people including me use Ceph as storage backend for Glance, Cinder and Nova. > Or If I want to keep all my vms on cinder volume. What will be the > configuration for it on nova and cinder? I recommend to set up a lab environment where you can learn to set up OpenStack, then play around and test different backends if required. The general configuration requirements are covered in the docs [1]. If you don't want to configure every single service you can follow a deployment guide [2], but that will require skills in ansible, juju or tripleO. I'd recommend the manual way, that way you learn the basics and how the different components interact. [1] https://docs.openstack.org/stein/install/ [2] https://docs.openstack.org/stein/deploy/ Zitat von Jyoti Dahiwele : > Thanks for your reply. > I'm referring this link > https://docs.openstack.org/mitaka/admin-guide/compute-images-instances.html > to understand about root disk. In this it is saying root disk will come > from compute node. What is the location of root disk on compute node? > > If I want to keep all my vms on shared storage . How to configure it ? > > Or If I want to keep all my vms on cinder volume. What will be the > configuration for it on nova and cinder? > > > On Thu, 11 Jul 2019, 15:24 Eugen Block, wrote: > >> Hi, >> >> it's always glance that serves the images, it just depends on how you >> decide to create the instance, ephemeral or persistent disks. You can >> find more information about storage concepts in [1]. >> >> If I'm not completely wrong, since Newton release the default in the >> Horizon settings is to create an instance from volume, so it would be >> a persistent disk managed by cinder (the volume persists after the >> instance has been deleted, this is also configurable). The image is >> downloaded from glance into a volume on your volume server. >> >> If you change the Horizon behavior or if you launch an instance from >> the cli you'd get an ephemeral disk by nova, depending on your storage >> backend this would be a local copy of the image on the compute node(s) >> or something related in your storage backend, e.g. an rbd object in >> ceph. >> >> Does this clear it up a bit? >> >> Regards, >> Eugen >> >> [1] >> >> https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html >> >> >> Zitat von Jyoti Dahiwele : >> >> > Dear Team, >> > >> > Please clear me my following doubts. >> > When I use image from source option and mini flavor to create an >> instace, >> > from which storage pool instance will get root disk ? From cinder or >> glance? >> >> >> >> >> From akekane at redhat.com Thu Jul 11 15:04:18 2019 From: akekane at redhat.com (Abhishek Kekane) Date: Thu, 11 Jul 2019 20:34:18 +0530 Subject: [glance] Train: Milestone 2 review priorities Message-ID: Dear Reviewers/Developers, Train milestone two is just two weeks away from now. I have created one etherpad [1] which lists the priority patches for glance, glance_store, python-glanceclient and glance-specs which will be good to have merged before Train 2 milestone. Feel free to add if you think any patch which I have missed is good to have during this milestone 2. Request all reviewers to review these patches so that we can have smooth milestone 2 release. Highlight of this milestone 2 release is that we are planning to roll out glance-store version 1.0 which will officially mark the multiple-stores feature of glance as a stable feature, [1] https://etherpad.openstack.org/p/Glance-Train-MileStone-2-Release-Plan Thanks & Best Regards, Abhishek Kekane -------------- next part -------------- An HTML attachment was scrubbed... URL: From grant at civo.com Thu Jul 11 15:07:49 2019 From: grant at civo.com (Grant Morley) Date: Thu, 11 Jul 2019 16:07:49 +0100 Subject: Cinder issue with DataCore Message-ID: Hi All, We are trying to test DataCore storage backend for cinder ( running on Queens ). We have everything installed and have the cinder config all setup. However whenever we try and start the "cinder-volume" service, we get the following error: 2019-07-11 12:30:06.977 1909533 ERROR cinder.volume.manager File "/openstack/venvs/cinder-17.1.2/lib/python2.7/site-packages/cinder/volume/manager.py", line 456, in init_host 2019-07-11 12:30:06.977 1909533 ERROR cinder.volume.manager self.driver.do_setup(ctxt) 2019-07-11 12:30:06.977 1909533 ERROR cinder.volume.manager   File "/openstack/venvs/cinder-17.1.2/lib/python2.7/site-packages/cinder/volume/drivers/datacore/iscsi.py", line 83, in do_setup 2019-07-11 12:30:06.977 1909533 ERROR cinder.volume.manager super(ISCSIVolumeDriver, self).do_setup(context) 2019-07-11 12:30:06.977 1909533 ERROR cinder.volume.manager   File "/openstack/venvs/cinder-17.1.2/lib/python2.7/site-packages/cinder/volume/drivers/datacore/driver.py", line 116, in do_setup 2019-07-11 12:30:06.977 1909533 ERROR cinder.volume.manager self.configuration.datacore_api_timeout) 2019-07-11 12:30:06.977 1909533 ERROR cinder.volume.manager   File "/openstack/venvs/cinder-17.1.2/lib/python2.7/site-packages/cinder/volume/drivers/datacore/api.py", line 176, in __init__ 2019-07-11 12:30:06.977 1909533 ERROR cinder.volume.manager raise datacore_exceptions.DataCoreException(msg) 2019-07-11 12:30:06.977 1909533 ERROR cinder.volume.manager DataCoreException: Failed to import websocket-client python module. Please, ensure the module is installed. We have the "websocket-client" installed also: pip freeze | grep websocket websocket-client==0.44.0 The datacore libraries also appear to be available in our venvs dirs: ls /openstack/venvs/cinder-17.1.2/lib/python2.7/site-packages/cinder/volume/drivers/datacore api.py  driver.py  exception.py  fc.py  __init__.py  iscsi.py passwd.py  utils.py We are a bit stumped at the moment and wondered if anyone knew what might be causing the error? We have managed to get Ceph and SolidFire working fine. Regards, -- Grant Morley Cloud Lead, Civo Ltd www.civo.com | Signup for an account! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jyotishri403 at gmail.com Thu Jul 11 15:22:45 2019 From: jyotishri403 at gmail.com (Jyoti Dahiwele) Date: Thu, 11 Jul 2019 20:52:45 +0530 Subject: root disk for instance In-Reply-To: <20190711144827.Horde.tglCLyI8aGqVcHo23OPMuxK@webmail.nde.ag> References: <20190711095326.Horde._eRtJ1r8ufNJjxxy9JnnVe-@webmail.nde.ag> <20190711144827.Horde.tglCLyI8aGqVcHo23OPMuxK@webmail.nde.ag> Message-ID: Thanks, I'll check it out. On Thu, 11 Jul 2019, 20:18 Eugen Block, wrote: > > I'm referring this link > > > https://docs.openstack.org/mitaka/admin-guide/compute-images-instances.html > > to understand about root disk. In this it is saying root disk will come > > from compute node. What is the location of root disk on compute node? > > First, use a later release, Mitaka is quite old, Stein is the current > release. > > If you use a default configuration without any specific backend for > your instances they will be located on the compute nodes in > /var/lib/nova/instances/. The respective base images would reside in > /var/lib/nova/instances/_base, so your compute nodes should have > sufficient disk space. > > > If I want to keep all my vms on shared storage . How to configure it ? > > That's up to you, there are several backends supported, so you'll have > to choose one. Many people including me use Ceph as storage backend > for Glance, Cinder and Nova. > > > Or If I want to keep all my vms on cinder volume. What will be the > > configuration for it on nova and cinder? > > I recommend to set up a lab environment where you can learn to set up > OpenStack, then play around and test different backends if required. > The general configuration requirements are covered in the docs [1]. If > you don't want to configure every single service you can follow a > deployment guide [2], but that will require skills in ansible, juju or > tripleO. I'd recommend the manual way, that way you learn the basics > and how the different components interact. > > [1] https://docs.openstack.org/stein/install/ > [2] https://docs.openstack.org/stein/deploy/ > > > Zitat von Jyoti Dahiwele : > > > Thanks for your reply. > > I'm referring this link > > > https://docs.openstack.org/mitaka/admin-guide/compute-images-instances.html > > to understand about root disk. In this it is saying root disk will come > > from compute node. What is the location of root disk on compute node? > > > > If I want to keep all my vms on shared storage . How to configure it ? > > > > Or If I want to keep all my vms on cinder volume. What will be the > > configuration for it on nova and cinder? > > > > > > On Thu, 11 Jul 2019, 15:24 Eugen Block, wrote: > > > >> Hi, > >> > >> it's always glance that serves the images, it just depends on how you > >> decide to create the instance, ephemeral or persistent disks. You can > >> find more information about storage concepts in [1]. > >> > >> If I'm not completely wrong, since Newton release the default in the > >> Horizon settings is to create an instance from volume, so it would be > >> a persistent disk managed by cinder (the volume persists after the > >> instance has been deleted, this is also configurable). The image is > >> downloaded from glance into a volume on your volume server. > >> > >> If you change the Horizon behavior or if you launch an instance from > >> the cli you'd get an ephemeral disk by nova, depending on your storage > >> backend this would be a local copy of the image on the compute node(s) > >> or something related in your storage backend, e.g. an rbd object in > >> ceph. > >> > >> Does this clear it up a bit? > >> > >> Regards, > >> Eugen > >> > >> [1] > >> > >> > https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html > >> > >> > >> Zitat von Jyoti Dahiwele : > >> > >> > Dear Team, > >> > > >> > Please clear me my following doubts. > >> > When I use image from source option and mini flavor to create an > >> instace, > >> > from which storage pool instance will get root disk ? From cinder or > >> glance? > >> > >> > >> > >> > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jungleboyj at gmail.com Thu Jul 11 15:42:35 2019 From: jungleboyj at gmail.com (Jay Bryant) Date: Thu, 11 Jul 2019 10:42:35 -0500 Subject: [cinder] Shanghai PTG/Forum Attendance ... Message-ID: <49416a91-b7d4-6b58-a7d9-33c6b6fc2f5a@gmail.com> All, We have been asked to get an idea of how many people are planning to attend the PTG and/or Forum for Cinder.  I have created an etherpad to start collecting topics as well as a list of people who are planning to attend [1].  If you think you may be in Shanghai or are in a timezone that can participate remotely, please update the etherpad. For fun, I have also created a Twitter Poll [2].  If you want to vote there to indicate attendance/non-attendance, please do so! Thanks! Jay (irc: jungleboyj) [1] https://etherpad.openstack.org/p/cinder-shanghai-ptg-planning [2] https://twitter.com/jungleboyj/status/1149045308271333378 From sfinucan at redhat.com Thu Jul 11 16:10:29 2019 From: sfinucan at redhat.com (Stephen Finucane) Date: Thu, 11 Jul 2019 17:10:29 +0100 Subject: [doc8] future development and maintenance In-Reply-To: <9D091EF4-60F2-490C-BF76-AAD7A68FB8A1@redhat.com> References: <9D091EF4-60F2-490C-BF76-AAD7A68FB8A1@redhat.com> Message-ID: <273715c4f2d4933232fc6b26cb982609daa1a2c7.camel@redhat.com> On Thu, 2019-07-11 at 12:01 +0100, Sorin Sbarnea wrote: > It seems that the doc8 project is lacking some love, regardless the > fact that is used by >90 projects from opendev.org. > > https://review.opendev.org/#/q/project:x/doc8 > > Last merge and release was more than 2 years ago and no reviews were > performed either. > > I think it would be in our interest to assure that doc8 maintenance > continues and that we can keep it usable. > > I would like to propose extenting the list of cores from the current > 4 ones that I already listed in CC with 3 more, so we can effectively > make a change that gets merged and later released (anyone willing to > help?) > > If current cores agree, I would be happy to help with maintenance. I > estimate that the effort needed would likely be less than 1h/month in > longer term. If there is a desire to move it to github/travis, I > would not mind either. I'd be tentatively interested in helping out here, though it's not as if I don't already have a lot on my plate :) I wonder if this might have better success outside of OpenStack, perhaps in the sphinx-doc or sphinx-contrib GitHub repo? Stephen > Thanks > Sorin Sbarnea > Red Hat TripleO CI From emilien at redhat.com Thu Jul 11 16:11:23 2019 From: emilien at redhat.com (Emilien Macchi) Date: Thu, 11 Jul 2019 12:11:23 -0400 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: References: Message-ID: On Wed, Jul 10, 2019 at 4:24 PM James Slagle wrote: > There's been a fair amount of recent work around simplifying our Heat > templates and migrating the software configuration part of our > deployment entirely to Ansible. > > As part of this effort, it became apparent that we could render much > of the data that we need out of Heat in a way that is generic per > node, and then have Ansible render the node specific data during > config-download runtime. > > To illustrate the point, consider when we specify ComputeCount:10 in > our templates, that much of the work that Heat is doing across those > 10 sets of resources for each Compute node is duplication. However, > it's been necessary so that Heat can render data structures such as > list of IP's, lists of hostnames, contents of /etc/hosts files, etc > etc etc. If all that was driven by Ansible using host facts, then Heat > doesn't need to do those 10 sets of resources to begin with. > > The goal is to get to a point where we can deploy the Heat stack with > a count of 1 for each role, and then deploy any number of nodes per > role using Ansible. To that end, I've been referring to this effort as > N=1. > > The value in this work is that it directly addresses our scaling > issues with Heat (by just deploying a much smaller stack). Obviously > we'd still be relying heavily on Ansible to scale to the required > levels, but I feel that is much better understood challenge at this > point in the evolution of configuration tools. > > With the patches that we've been working on recently, I've got a POC > running where I can deploy additional compute nodes with just Ansible. > This is done by just adding the additional nodes to the Ansible > inventory with a small set of facts to include IP addresses on each > enabled network and a hostname. > > These patches are at > https://review.opendev.org/#/q/topic:bp/reduce-deployment-resources > and reviews/feedback are welcome. > This is a fabulous proposal in my opinion. I've added (and will continue to add) TODO ideas in the etherpad. Anyone willing to help, please ping us if needed. Another point, somewhat related: I took the opportunity of this work to reduce the complexity around the number of hieradata files. I would like to investigate if we can generate one data file which would be loaded by both Puppet and Ansible for doing the configuration management. I'll create a separated thread on that effort very soon. > Other points: > > - Baremetal provisioning and port creation are presently handled by > Heat. With the ongoing efforts to migrate baremetal provisioning out > of Heat (nova-less deploy), I think these efforts are very > complimentary. Eventually, we get to a point where Heat is not > actually creating any other OpenStack API resources. For now, the > patches only work when using pre-provisioned nodes. > > - We need to consider how we'd manage the Ansible inventory going > forward if we open up an interface for operators to manipulate it > directly. That's something we'd want to manage and preserve (version > control) as it's critical data for the deployment. > > Given the progress that we've made with the POC, my sense is that > we'll keep pushing in this overall direction. I'd like to get some > feedback on the approach. We have an etherpad we are using to track > some of the work at a high level: > > https://etherpad.openstack.org/p/tripleo-reduce-deployment-resources > > I'll be adding some notes on how I setup the POC to that etherpad if > others would like to try it out. > > -- > -- James Slagle > -- > > -- Emilien Macchi -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianyrchoi at gmail.com Thu Jul 11 16:41:05 2019 From: ianyrchoi at gmail.com (Ian Y. Choi) Date: Fri, 12 Jul 2019 01:41:05 +0900 Subject: [I18n][ptg] Shanghai PTG/Forum Attendance Message-ID: <64124295-1a95-598c-f929-fe546cbecad8@gmail.com> Hello, Like other official teams, I18n team has been also asked to get an idea of how many people are planning to attend the PTG and/or Forum. I have created an Etherpad to start collecting topics as well as a list of people who are planning to attend [1], and if you think you may be in Shanghai or are in a timezone that can participate remotely, please update the Etherpad. Note that this upcoming PTG/Forum is very Asia-timezone friendly - I highly encourage people in Asia countries (including China, surely) to participate offline or remotely. Thanks! With many thanks, /Ian [1] https://etherpad.openstack.org/p/i18n-shanghai-ptg-planning From emilien at redhat.com Thu Jul 11 17:52:29 2019 From: emilien at redhat.com (Emilien Macchi) Date: Thu, 11 Jul 2019 13:52:29 -0400 Subject: [tripleo] Roadmap to simplify TripleO Message-ID: Even though TripleO is well known for its maturity, it also has a reputation of being complex when it comes to the number of tools that it uses. Somewhat related to the efforts that James is leading with "Scaling TripleO" [1] [2], I would like to formalize our joint efforts to make TripleO simpler in the future. Some work has been done over the last releases already and yet we have seen net benefits; however we still have challenges ahead of us. - With no UI anymore, do we really need an API? - How can we reduce the number of languages in TripleO? ... and make Python + Ansible the main ones. - How can we reduce our dependencies? I created a document which explains the problems and propose some solutions: https://docs.google.com/document/d/1vY9rsccgp7NHFXpLtCFTHQm14F15Tij7lhn5X_P14Ys For those who can't or don't want Google Doc, I've put together the notes into etherpad [3] and I'll take care of making sure it's updated at last at the beginning until we sort things out. Feel free to be involved: - comment or suggest edits if you have feedback / ideas - sign-up if you're interested to contribute For now I expect this document to be a clear place what our plan is but I wouldn't be surprised if in the future we break it down into specs / blueprints / etc for better tracking. Thanks, [1] https://docs.google.com/document/d/12tPc4NC5fo8ytGuFZ4DSZXXyzes1x3U7oYz9neaPP_o/edit [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007638.html [3] https://etherpad.openstack.org/p/tripleo-simplification -- Emilien Macchi -------------- next part -------------- An HTML attachment was scrubbed... URL: From openstack at fried.cc Thu Jul 11 17:58:15 2019 From: openstack at fried.cc (Eric Fried) Date: Thu, 11 Jul 2019 12:58:15 -0500 Subject: [placement][ptg] Shanghai attendance In-Reply-To: References: Message-ID: <11798408-977b-ecdd-2730-4e7dfb16f38d@fried.cc> Chris- > asked for feedback from the rest of the placement team on whether we > need to do any placement work there, given the value we got out of > the virtual pre-PTG in April. > > Let me ask again: Do we need a presence at the PTG in Shanghai? The email-based virtual pre-PTG was extremely productive. We should definitely do it again. However, I feel like that last hour on Saturday saw more design progress than weeks worth of emails or spec reviews could have accomplished. To me, this is the value of the in-person meetups. It sucks that we can't involve everybody in the discussions - but we can rapidly crystallize an idea enough to produce a coherent spec that *can* then involve everybody. Having said that, I'm really not coming up with any major design topics that would benefit from such a meetup this time around. I feel like what we've accomplished in Train sets us up for a cycle or two of refinement (perf/docs/refactor/tech-debt) rather than feature work. I suppose we'll see what shakes out on the > https://etherpad.openstack.org/p/placement-shanghai-ptg efried . From donny at fortnebula.com Thu Jul 11 18:06:00 2019 From: donny at fortnebula.com (Donny Davis) Date: Thu, 11 Jul 2019 14:06:00 -0400 Subject: [Nova] Instances can't be started after compute nodes unexpectedly shut down because of power outage In-Reply-To: References: Message-ID: Can you ssh to the hypervisor and run virsh list to make sure your instances are in fact down? On Thu, Jul 11, 2019 at 3:02 AM Gökhan IŞIK wrote: > Can anyone help me please ? I can no't rescue my instances yet :( > > Thanks, > Gökhan > > Gökhan IŞIK , 9 Tem 2019 Sal, 15:46 tarihinde > şunu yazdı: > >> Hi folks, >> Because of power outage, Most of our compute nodes unexpectedly >> shut down and now I can not start our instances. Error message is "Failed >> to get "write" lock another process using the image?". Instances Power >> status is No State. Full error log is >> http://paste.openstack.org/show/754107/. My environment is OpenStack >> Pike on Ubuntu 16.04 LTS servers and Instances are on a nfs shared storage. >> Nova version is 16.1.6.dev2. qemu version is 2.10.1. libvirt version is >> 3.6.0. I saw a commit [1], but it doesn't solve this problem. >> There are important instances on my environment. How can I rescue my >> instances? What would you suggest ? >> >> Thanks, >> Gökhan >> >> [1] https://review.opendev.org/#/c/509774/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Thu Jul 11 18:15:55 2019 From: smooney at redhat.com (Sean Mooney) Date: Thu, 11 Jul 2019 19:15:55 +0100 Subject: [placement][ptg] Shanghai attendance In-Reply-To: <11798408-977b-ecdd-2730-4e7dfb16f38d@fried.cc> References: <11798408-977b-ecdd-2730-4e7dfb16f38d@fried.cc> Message-ID: <98d51bb7ad7d34268927e0fdb47fb8f31c6af6cb.camel@redhat.com> On Thu, 2019-07-11 at 12:58 -0500, Eric Fried wrote: > Chris- > > > asked for feedback from the rest of the placement team on whether we > > need to do any placement work there, given the value we got out of > > the virtual pre-PTG in April. > > > > Let me ask again: Do we need a presence at the PTG in Shanghai? > > The email-based virtual pre-PTG was extremely productive. We should > definitely do it again. > > However, I feel like that last hour on Saturday saw more design progress > than weeks worth of emails or spec reviews could have accomplished. To > me, this is the value of the in-person meetups. It sucks that we can't > involve everybody in the discussions - but we can rapidly crystallize an > idea enough to produce a coherent spec that *can* then involve everybody. well if ye do a virtual email based pre-ptg again why not also continue that virtual concept and consider a google hangout or somehting to allow realtime discussion with video/etherpad. i personally found the email discussion hard to follow vs an etherpad or gerrit review per topic but it did have pluses too. > > Having said that, I'm really not coming up with any major design topics > that would benefit from such a meetup this time around. I feel like what > we've accomplished in Train sets us up for a cycle or two of refinement > (perf/docs/refactor/tech-debt) rather than feature work. I suppose we'll > see what shakes out on the > > > https://etherpad.openstack.org/p/placement-shanghai-ptg > > efried > . > From skylightcoder at gmail.com Thu Jul 11 18:39:11 2019 From: skylightcoder at gmail.com (=?UTF-8?B?R8O2a2hhbiBJxZ5JSw==?=) Date: Thu, 11 Jul 2019 21:39:11 +0300 Subject: [Nova] Instances can't be started after compute nodes unexpectedly shut down because of power outage In-Reply-To: References: Message-ID: I run virsh list --all command and output is below: root at compute06:~# virsh list --all Id Name State ---------------------------------------------------- - instance-000012f9 shut off - instance-000013b6 shut off - instance-000016fb shut off - instance-0000190a shut off - instance-00001a8a shut off - instance-00001e05 shut off - instance-0000202a shut off - instance-00002135 shut off - instance-00002141 shut off - instance-000021b6 shut off - instance-000021ec shut off - instance-000023db shut off - instance-00002ad7 shut off And also when I try start instances with virsh , output is below: root at compute06:~# virsh start instance-0000219b error: Failed to start domain instance-000012f9 error: internal error: process exited while connecting to monitor: 2019-07-11T18:36:34.229534Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on: char device redirected to /dev/pts/3 (label charserial0) 2019-07-11T18:36:34.243395Z qemu-system-x86_64: -drive file=/var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=ignore: Failed to get "write" lock Is another process using the image? Thanks, Gökhan Donny Davis , 11 Tem 2019 Per, 21:06 tarihinde şunu yazdı: > Can you ssh to the hypervisor and run virsh list to make sure your > instances are in fact down? > > On Thu, Jul 11, 2019 at 3:02 AM Gökhan IŞIK > wrote: > >> Can anyone help me please ? I can no't rescue my instances yet :( >> >> Thanks, >> Gökhan >> >> Gökhan IŞIK , 9 Tem 2019 Sal, 15:46 tarihinde >> şunu yazdı: >> >>> Hi folks, >>> Because of power outage, Most of our compute nodes unexpectedly >>> shut down and now I can not start our instances. Error message is "Failed >>> to get "write" lock another process using the image?". Instances Power >>> status is No State. Full error log is >>> http://paste.openstack.org/show/754107/. My environment is OpenStack >>> Pike on Ubuntu 16.04 LTS servers and Instances are on a nfs shared storage. >>> Nova version is 16.1.6.dev2. qemu version is 2.10.1. libvirt version is >>> 3.6.0. I saw a commit [1], but it doesn't solve this problem. >>> There are important instances on my environment. How can I rescue my >>> instances? What would you suggest ? >>> >>> Thanks, >>> Gökhan >>> >>> [1] https://review.opendev.org/#/c/509774/ >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayachander.it at gmail.com Thu Jul 11 18:51:30 2019 From: jayachander.it at gmail.com (Jay See) Date: Thu, 11 Jul 2019 20:51:30 +0200 Subject: [queens][nova] nova host-evacuate errot In-Reply-To: References: Message-ID: Thanks for explanation Ignazio. I have tried same same by trying to put the compute node on a failure (echo 'c' > /proc/sysrq-trigger ). Compute node was stuck and I was not able connect to it. All the VMs are now in Error state. Running the host-evacaute was successful on controller node, but now I am not able to use the VMs. Because they are all in error state now. root at h004:~$ nova host-evacuate h017 +--------------------------------------+-------------------+---------------+ | Server UUID | Evacuate Accepted | Error Message | +--------------------------------------+-------------------+---------------+ | f3545f7d-b85e-49ee-b407-333a4c5b5ab9 | True | | | 9094494b-cfa3-459b-8d51-d9aae0ea9636 | True | | | abe7075b-ac22-4168-bf3d-d302ba37d80e | True | | | c9919371-5f2e-4155-a01a-5f41d9c8b0e7 | True | | | ffd983bb-851e-4314-9d1d-375303c278f3 | True | | +--------------------------------------+-------------------+---------------+ Now I have restarted the compute node manually , now I am able to connect to the compute node but VMs are still in Error state. 1. Any ideas, how to recover the VMs? 2. Are there any other methods to evacuate, as this method seems to be not working in mitaka version. ~Jay. On Thu, Jul 11, 2019 at 1:33 PM Ignazio Cassano wrote: > Ok Jay, > let me to describe my environment. > I have an openstack made up of 3 controllers nodes ad several compute > nodes. > The controller nodes services are controlled by pacemaker and the compute > nodes services are controlled by remote pacemaker. > My hardware is Dell so I am using ipmi fencing device . > I wrote a service controlled by pacemaker: > this service controls if a compude node fails and for avoiding split > brains if a compute node does nod respond on the management network and on > storage network the stonith poweroff the node and then execute a nova > host-evacuate. > > Anycase to have a simulation before writing the service I described above > you can do as follows: > > connect on one compute node where some virtual machines are running > run the command: echo 'c' > /proc/sysrq-trigger (it stops immediately the > node like in case of failure) > On a controller node run: nova host-evacuate "name of failed compute node" > Instances running on the failed compute node should be restarted on > another compute node > > > Ignazio > > Il giorno gio 11 lug 2019 alle ore 11:57 Jay See > ha scritto: > >> Hi , >> >> I have tried on a failed compute node which is in power off state now. >> I have tried on a running compute node, no errors. But nothing happens. >> On running compute node - Disabled the compute service and tried >> migration also. >> >> May be I might have not followed proper steps. Just wanted to know the >> steps you have followed. Otherwise, I was planning to manual migration also >> if possible. >> ~Jay. >> >> On Thu, Jul 11, 2019 at 11:52 AM Ignazio Cassano < >> ignaziocassano at gmail.com> wrote: >> >>> Hi Jay, >>> would you like to evacuate a failed compute node or evacuate a running >>> compute node ? >>> >>> Ignazio >>> >>> Il giorno gio 11 lug 2019 alle ore 11:48 Jay See < >>> jayachander.it at gmail.com> ha scritto: >>> >>>> Hi Ignazio, >>>> >>>> I am trying to evacuate the compute host on older version (mitaka). >>>> Could please share the process you followed. I am not able to succeed >>>> with openstack live-migration fails with error message (this is known issue >>>> in older versions) and nova live-ligration - nothing happens even after >>>> initiating VM migration. It is almost 4 days. >>>> >>>> ~Jay. >>>> >>>> On Thu, Jul 11, 2019 at 11:31 AM Ignazio Cassano < >>>> ignaziocassano at gmail.com> wrote: >>>> >>>>> I am sorry. >>>>> For simulating an host crash I used a wrong procedure. >>>>> Using "echo 'c' > /proc/sysrq-trigger" all work fine >>>>> >>>>> Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano < >>>>> ignaziocassano at gmail.com> ha scritto: >>>>> >>>>>> Hello All, >>>>>> on ocata when I poweroff a node with active instance , doing a nova >>>>>> host-evacuate works fine >>>>>> and instances are restartd on an active node. >>>>>> On queens it does non evacuate instances but nova-api reports for >>>>>> each instance the following: >>>>>> >>>>>> 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi >>>>>> [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 >>>>>> c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: >>>>>> Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is >>>>>> in task_state powering-off >>>>>> >>>>>> So it poweroff all instance on the failed node but does not start >>>>>> them on active nodes >>>>>> >>>>>> What is changed ? >>>>>> Ignazio >>>>>> >>>>>> >>>>>> >>>> >>>> -- >>>> ​ >>>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>>> necessary.* >>>> >>> >> >> -- >> ​ >> P *SAVE PAPER – Please do not print this e-mail unless absolutely >> necessary.* >> > -- ​ P *SAVE PAPER – Please do not print this e-mail unless absolutely necessary.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From donny at fortnebula.com Thu Jul 11 19:11:02 2019 From: donny at fortnebula.com (Donny Davis) Date: Thu, 11 Jul 2019 15:11:02 -0400 Subject: [Nova] Instances can't be started after compute nodes unexpectedly shut down because of power outage In-Reply-To: References: Message-ID: Well that is interesting. If you look in your libvirt config directory (/etc/libvirt on Centos) you can get a little more info on what is being used for locking. Maybe strace can shed some light on it. Try something like strace -ttt -f qemu-img info /var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk On Thu, Jul 11, 2019 at 2:39 PM Gökhan IŞIK wrote: > I run virsh list --all command and output is below: > > root at compute06:~# virsh list --all > Id Name State > ---------------------------------------------------- > - instance-000012f9 shut off > - instance-000013b6 shut off > - instance-000016fb shut off > - instance-0000190a shut off > - instance-00001a8a shut off > - instance-00001e05 shut off > - instance-0000202a shut off > - instance-00002135 shut off > - instance-00002141 shut off > - instance-000021b6 shut off > - instance-000021ec shut off > - instance-000023db shut off > - instance-00002ad7 shut off > > And also when I try start instances with virsh , output is below: > > root at compute06:~# virsh start instance-0000219b > error: Failed to start domain instance-000012f9 > error: internal error: process exited while connecting to monitor: > 2019-07-11T18:36:34.229534Z qemu-system-x86_64: -chardev > pty,id=charserial0,logfile=/dev/fdset/2,logappend=on: char device > redirected to /dev/pts/3 (label charserial0) > 2019-07-11T18:36:34.243395Z qemu-system-x86_64: -drive > file=/var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=ignore: > Failed to get "write" lock > Is another process using the image? > > Thanks, > Gökhan > > Donny Davis , 11 Tem 2019 Per, 21:06 tarihinde şunu > yazdı: > >> Can you ssh to the hypervisor and run virsh list to make sure your >> instances are in fact down? >> >> On Thu, Jul 11, 2019 at 3:02 AM Gökhan IŞIK >> wrote: >> >>> Can anyone help me please ? I can no't rescue my instances yet :( >>> >>> Thanks, >>> Gökhan >>> >>> Gökhan IŞIK , 9 Tem 2019 Sal, 15:46 tarihinde >>> şunu yazdı: >>> >>>> Hi folks, >>>> Because of power outage, Most of our compute nodes unexpectedly >>>> shut down and now I can not start our instances. Error message is "Failed >>>> to get "write" lock another process using the image?". Instances Power >>>> status is No State. Full error log is >>>> http://paste.openstack.org/show/754107/. My environment is OpenStack >>>> Pike on Ubuntu 16.04 LTS servers and Instances are on a nfs shared storage. >>>> Nova version is 16.1.6.dev2. qemu version is 2.10.1. libvirt version is >>>> 3.6.0. I saw a commit [1], but it doesn't solve this problem. >>>> There are important instances on my environment. How can I rescue my >>>> instances? What would you suggest ? >>>> >>>> Thanks, >>>> Gökhan >>>> >>>> [1] https://review.opendev.org/#/c/509774/ >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From skylightcoder at gmail.com Thu Jul 11 19:23:37 2019 From: skylightcoder at gmail.com (=?UTF-8?B?R8O2a2hhbiBJxZ5JSw==?=) Date: Thu, 11 Jul 2019 22:23:37 +0300 Subject: [Nova] Instances can't be started after compute nodes unexpectedly shut down because of power outage In-Reply-To: References: Message-ID: In [1] it says "Image locking is added and enabled by default. Multiple QEMU processes cannot write to the same image as long as the host supports OFD or posix locking, unless options are specified otherwise." May be need to do something on nova side. I run this command and get same error. Output is in http://paste.openstack.org/show/754311/ İf I run qemu-img info instance-0000219b with -U , it doesn't give any errors. [1] https://wiki.qemu.org/ChangeLog/2.10 Donny Davis , 11 Tem 2019 Per, 22:11 tarihinde şunu yazdı: > Well that is interesting. If you look in your libvirt config directory > (/etc/libvirt on Centos) you can get a little more info on what is being > used for locking. > > Maybe strace can shed some light on it. Try something like > > strace -ttt -f qemu-img info > /var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk > > > > > > On Thu, Jul 11, 2019 at 2:39 PM Gökhan IŞIK > wrote: > >> I run virsh list --all command and output is below: >> >> root at compute06:~# virsh list --all >> Id Name State >> ---------------------------------------------------- >> - instance-000012f9 shut off >> - instance-000013b6 shut off >> - instance-000016fb shut off >> - instance-0000190a shut off >> - instance-00001a8a shut off >> - instance-00001e05 shut off >> - instance-0000202a shut off >> - instance-00002135 shut off >> - instance-00002141 shut off >> - instance-000021b6 shut off >> - instance-000021ec shut off >> - instance-000023db shut off >> - instance-00002ad7 shut off >> >> And also when I try start instances with virsh , output is below: >> >> root at compute06:~# virsh start instance-0000219b >> error: Failed to start domain instance-000012f9 >> error: internal error: process exited while connecting to monitor: >> 2019-07-11T18:36:34.229534Z qemu-system-x86_64: -chardev >> pty,id=charserial0,logfile=/dev/fdset/2,logappend=on: char device >> redirected to /dev/pts/3 (label charserial0) >> 2019-07-11T18:36:34.243395Z qemu-system-x86_64: -drive >> file=/var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=ignore: >> Failed to get "write" lock >> Is another process using the image? >> >> Thanks, >> Gökhan >> >> Donny Davis , 11 Tem 2019 Per, 21:06 tarihinde >> şunu yazdı: >> >>> Can you ssh to the hypervisor and run virsh list to make sure your >>> instances are in fact down? >>> >>> On Thu, Jul 11, 2019 at 3:02 AM Gökhan IŞIK >>> wrote: >>> >>>> Can anyone help me please ? I can no't rescue my instances yet :( >>>> >>>> Thanks, >>>> Gökhan >>>> >>>> Gökhan IŞIK , 9 Tem 2019 Sal, 15:46 tarihinde >>>> şunu yazdı: >>>> >>>>> Hi folks, >>>>> Because of power outage, Most of our compute nodes unexpectedly >>>>> shut down and now I can not start our instances. Error message is "Failed >>>>> to get "write" lock another process using the image?". Instances Power >>>>> status is No State. Full error log is >>>>> http://paste.openstack.org/show/754107/. My environment is OpenStack >>>>> Pike on Ubuntu 16.04 LTS servers and Instances are on a nfs shared storage. >>>>> Nova version is 16.1.6.dev2. qemu version is 2.10.1. libvirt version is >>>>> 3.6.0. I saw a commit [1], but it doesn't solve this problem. >>>>> There are important instances on my environment. How can I rescue my >>>>> instances? What would you suggest ? >>>>> >>>>> Thanks, >>>>> Gökhan >>>>> >>>>> [1] https://review.opendev.org/#/c/509774/ >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From vungoctan252 at gmail.com Thu Jul 11 04:32:06 2019 From: vungoctan252 at gmail.com (Vu Tan) Date: Thu, 11 Jul 2019 11:32:06 +0700 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: References: Message-ID: I know it's just a warning, just take a look at this image: [image: image.png] it's just hang there forever, and in the log show what I have shown to you On Wed, Jul 10, 2019 at 8:07 PM Gaëtan Trellu wrote: > This is just a warning, not an error. > > On Jul 10, 2019 3:12 AM, Vu Tan wrote: > > Hi Gaetan, > I follow you the guide you gave me, but the problem still persist, can you > please take a look at my configuration to see what is wrong or what is > missing in my config ? > the error: > 2019-07-10 14:08:46.876 17292 WARNING keystonemiddleware._common.config > [-] The option "__file__" in conf is not known to auth_token > 2019-07-10 14:08:46.876 17292 WARNING keystonemiddleware._common.config > [-] The option "here" in conf is not known to auth_token > 2019-07-10 14:08:46.882 17292 WARNING keystonemiddleware.auth_token [-] > AuthToken middleware is set with keystone_authtoken.service_ > > the config: > > [DEFAULT] > enabled_apis = masakari_api > log_dir = /var/log/kolla/masakari > state_path = /var/lib/masakari > os_user_domain_name = default > os_project_domain_name = default > os_privileged_user_tenant = service > os_privileged_user_auth_url = http://controller:5000/v3 > os_privileged_user_name = nova > os_privileged_user_password = P at ssword > masakari_api_listen = controller > masakari_api_listen_port = 15868 > debug = False > auth_strategy=keystone > > [wsgi] > # The paste configuration file path > api_paste_config = /etc/masakari/api-paste.ini > > [keystone_authtoken] > www_authenticate_uri = http://controller:5000 > auth_url = http://controller:5000 > auth_type = password > project_domain_id = default > project_domain_name = default > user_domain_name = default > user_domain_id = default > project_name = service > username = masakari > password = P at ssword > region_name = RegionOne > > [oslo_middleware] > enable_proxy_headers_parsing = True > > [database] > connection = mysql+pymysql://masakari:P at ssword@controller/masakari > > > > On Tue, Jul 9, 2019 at 10:25 PM Vu Tan wrote: > > Thank Patil Tushar, I hope it will be available soon > > On Tue, Jul 9, 2019 at 8:18 AM Patil, Tushar > wrote: > > Hi Vu and Gaetan, > > Gaetan, thank you for helping out Vu in setting up masakari-monitors > service. > > As a masakari team ,we have noticed there is a need to add proper > documentation to help the community run Masakari services in their > environment. We are working on adding proper documentation in this 'Train' > cycle. > > Will send an email on this mailing list once the patches are uploaded on > the gerrit so that you can give your feedback on the same. > > If you have any trouble in setting up Masakari, please let us know on this > mailing list or join the bi-weekly IRC Masakari meeting on the > #openstack-meeting IRC channel. The next meeting will be held on 16th July > 2019 @0400 UTC. > > Regards, > Tushar Patil > > ________________________________________ > From: Vu Tan > Sent: Monday, July 8, 2019 11:21:16 PM > To: Gaëtan Trellu > Cc: openstack-discuss at lists.openstack.org > Subject: Re: [masakari] how to install masakari on centos 7 > > Hi Gaetan, > Thanks for pinpoint this out, silly me that did not notice the simple > "error InterpreterNotFound: python3". Thanks a lot, I appreciate it > > On Mon, Jul 8, 2019 at 9:15 PM gaetan.trellu at incloudus.com>> wrote: > Vu Tan, > > About "auth_token" error, you need "os_privileged_user_*" options into > your masakari.conf for the API. > As mentioned previously please have a look here to have an example of > configuration working (for me at least): > > - masakari.conf: > > https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari.conf.j2 > - masakari-monitor.conf: > > https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari-monitors.conf.j2 > > About your tox issue make sure you have Python3 installed. > > Gaëtan > > On 2019-07-08 06:08, Vu Tan wrote: > > > Hi Gaetan, > > I try to generate config file by using this command tox -egenconfig on > > top level of masakari but the output is error, is this masakari still > > in beta version ? > > [root at compute1 masakari-monitors]# tox -egenconfig > > genconfig create: /root/masakari-monitors/.tox/genconfig > > ERROR: InterpreterNotFound: python3 > > _____________________________________________________________ summary > > ______________________________________________________________ > > ERROR: genconfig: InterpreterNotFound: python3 > > > > On Mon, Jul 8, 2019 at 3:24 PM Vu Tan vungoctan252 at gmail.com>> wrote: > > Hi, > > Thanks a lot for your reply, I install pacemaker/corosync, > > masakari-api, maskari-engine on controller node, and I run masakari-api > > with this command: masakari-api, but I dont know whether the process is > > running like that or is it just hang there, here is what it shows when > > I run the command, I leave it there for a while but it does not change > > anything : > > [root at controller masakari]# masakari-api > > 2019-07-08 15:21:09.946 30250 INFO masakari.api.openstack [-] Loaded > > extensions: ['extensions', 'notifications', 'os-hosts', 'segments', > > 'versions'] > > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > > [-] The option "__file__" in conf is not known to auth_token > > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > > [-] The option "here" in conf is not known to auth_token > > 2019-07-08 15:21:09.960 30250 WARNING keystonemiddleware.auth_token [-] > > AuthToken middleware is set with > > keystone_authtoken.service_token_roles_required set to False. This is > > backwards compatible but deprecated behaviour. Please set this to True. > > 2019-07-08 15:21:09.974 30250 INFO masakari.wsgi [-] masakari_api > > listening on 127.0.0.1:15868 > > 2019-07-08 15:21:09.975 30250 INFO oslo_service.service [-] Starting 4 > > workers > > 2019-07-08 15:21:09.984 30274 INFO masakari.masakari_api.wsgi.server > > [-] (30274) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.985 30275 INFO masakari.masakari_api.wsgi.server > > [-] (30275) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.992 30277 INFO masakari.masakari_api.wsgi.server > > [-] (30277) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.994 30276 INFO masakari.masakari_api.wsgi.server > > [-] (30276) wsgi starting up on http://127.0.0.1:15868 > > > > On Sun, Jul 7, 2019 at 7:37 PM Gaëtan Trellu > > > wrote: > > > > Hi Vu Tan, > > > > Masakari documentation doesn't really exist... I had to figured some > > stuff by myself to make it works into Kolla project. > > > > On controller nodes you need: > > > > - pacemaker > > - corosync > > - masakari-api (openstack/masakari repository) > > - masakari- engine (openstack/masakari repository) > > > > On compute nodes you need: > > > > - pacemaker-remote (integrated to pacemaker cluster as a resource) > > - masakari- hostmonitor (openstack/masakari-monitor repository) > > - masakari-instancemonitor (openstack/masakari-monitor repository) > > - masakari-processmonitor (openstack/masakari-monitor repository) > > > > For masakari-hostmonitor, the service needs to have access to systemctl > > command (make sure you are not using sysvinit). > > > > For masakari-monitor, the masakari-monitor.conf is a bit different, you > > will have to configure the [api] section properly. > > > > RabbitMQ needs to be configured (as transport_url) on masakari-api and > > masakari-engine too. > > > > Please check this review[1], you will have masakari.conf and > > masakari-monitor.conf configuration examples. > > > > [1] https://review.opendev.org/#/c/615715 > > > > Gaëtan > > > > On Jul 7, 2019 12:08 AM, Vu Tan vungoctan252 at gmail.com>> wrote: > > > > VU TAN > > > > > 10:30 AM (35 minutes ago) > > > > to openstack-discuss > > > > Sorry, I resend this email because I realized that I lacked of prefix > > on this email's subject > > > > Hi, > > > > I would like to use Masakari and I'm having trouble finding a step by > > step or other documentation to get started with. Which part should be > > installed on controller, which is should be on compute, and what is the > > prerequisite to install masakari, I have installed corosync and > > pacemaker on compute and controller nodes, , what else do I need to do > > ? step I have done so far: > > - installed corosync/pacemaker > > - install masakari on compute node on this github repo: > > https://github.com/openstack/masakari > > - add masakari in to mariadb > > here is my configuration file of masakari.conf, do you mind to take a > > look at it, if I have misconfigured anything? > > > > [DEFAULT] > > enabled_apis = masakari_api > > > > # Enable to specify listening IP other than default > > masakari_api_listen = controller > > # Enable to specify port other than default > > masakari_api_listen_port = 15868 > > debug = False > > auth_strategy=keystone > > > > [wsgi] > > # The paste configuration file path > > api_paste_config = /etc/masakari/api-paste.ini > > > > [keystone_authtoken] > > www_authenticate_uri = http://controller:5000 > > auth_url = http://controller:5000 > > auth_type = password > > project_domain_id = default > > user_domain_id = default > > project_name = service > > username = masakari > > password = P at ssword > > > > [database] > > connection = mysql+pymysql://masakari:P at ssword@controller/masakari > Disclaimer: This email and any attachments are sent in strictest > confidence for the sole use of the addressee and may contain legally > privileged, confidential, and proprietary data. If you are not the intended > recipient, please advise the sender by replying promptly to this email and > then delete and destroy this email and any attachments without any further > use, copying or forwarding. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 2369 bytes Desc: not available URL: From manuel.sb at garvan.org.au Thu Jul 11 06:15:00 2019 From: manuel.sb at garvan.org.au (Manuel Sopena Ballesteros) Date: Thu, 11 Jul 2019 06:15:00 +0000 Subject: trying to understand steal time with cpu pinning Message-ID: <9D8A2486E35F0941A60430473E29F15B017EB28866@MXDB1.ad.garvan.unsw.edu.au> Dear Openstack community, Please correct me if I am wrong. As far as I understand `steal time > 0` means that the hypervisor has replaced a vcpu with a different one on the physical cpu. Also, cpu pinning allocates a vcpu to a physical cpu permanently. I have a vm setup with cpu pinning and numa affinity and realized, that cpu steal time is between 1% and 0%. [cid:image002.png at 01D53803.C9C9F190] Why is that? Thank you very much NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 108451 bytes Desc: image002.png URL: From ignaziocassano at gmail.com Thu Jul 11 08:10:39 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Thu, 11 Jul 2019 10:10:39 +0200 Subject: [queens][magnuma] kubernetes cluster In-Reply-To: References: <765458860.27505.1562167876903@ox.servinga.com> Message-ID: Hi Feilong, we are going to upgrade our openstack installations asap. Thanks Ignazio Il giorno gio 4 lug 2019 alle ore 19:45 Feilong Wang < feilong at catalyst.net.nz> ha scritto: > Hi Ignazio, > > We fixed a lot of issues in Rocky and Stein, which some of them can't be > easily backported to Queens. Magnum has a very loose dependency for other > services, so I would suggest to use rocky or stein if it's possible for you. > > As for your issue, the error means your kube-apiserver didn't start > successfully. You can take a look the cloud init log for more information. > > > On 4/07/19 4:37 AM, Ignazio Cassano wrote: > > Thanks Denis, but I think there is another problem: on kube muster port > 8080 is not listening, probably some services are note started > Regards > Ignazio > > Il giorno mer 3 lug 2019 alle ore 17:31 Denis Pascheka > ha scritto: > >> Hi Ignazio, >> >> in Queens there is an issue within Magnum which has been resolved in the >> Rocky release. >> Take a look at this file: >> https://github.com/openstack/magnum/blob/stable/rocky/magnum/drivers/common/templates/kubernetes/fragments/wc-notify-master.sh. >> >> The execution of the curl command in row 16 needs to be escaped with an >> backslash. You can achieve this by building your own magnum containers >> and >> adding an template override >> to >> it where you add your fixed/own wc-notify-master.sh script from the plugin >> directory >> . >> >> >> Best Regards, >> >> *Denis Pascheka* >> Cloud Architect >> >> t: +49 (69) 348 75 11 12 >> m: +49 (170) 495 6364 >> e: dp at servinga.com >> servinga GmbH >> Mainzer Landstr. 351-353 >> 60326 Frankfurt >> >> >> >> * >> www.servinga.com * >> >> Amtsgericht Frankfurt am Main - HRB 91418 - Geschäftsführer Adam Lakota, >> Christian Lertes >> >> Ignazio Cassano hat am 3. Juli 2019 um 16:58 >> geschrieben: >> >> >> Hi All, >> I just installed openstack kolla queens with magnum but trying to create >> a kubernetes cluster the master nodes does not terminate installation: it >> loops with the following message: >> >> curl --silent http://127.0.0.1:8080/healthz >> + '[' ok = '' ']' >> + sleep 5 >> >> Anyone can help ? >> Best Regards >> Ignazio >> >> >> >> > -- > Cheers & Best regards, > Feilong Wang (王飞龙) > -------------------------------------------------------------------------- > Senior Cloud Software Engineer > Tel: +64-48032246 > Email: flwang at catalyst.net.nz > Catalyst IT Limited > Level 6, Catalyst House, 150 Willis Street, Wellington > -------------------------------------------------------------------------- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: noname Type: image/png Size: 65536 bytes Desc: not available URL: From donny at fortnebula.com Thu Jul 11 20:10:43 2019 From: donny at fortnebula.com (Donny Davis) Date: Thu, 11 Jul 2019 16:10:43 -0400 Subject: [Nova] Instances can't be started after compute nodes unexpectedly shut down because of power outage In-Reply-To: References: Message-ID: You surely want to leave locking turned on. You may want to ask qemu-devel about the locking of a image file and how it works. This isn't really an Openstack issue, seems to be a layer below. Depending on how mission critical your VM's are, you could probably work around it by just passing in --force-share into the command openstack is trying to run. I cannot recommend this path, the best way is to find out how you remove the lock. On Thu, Jul 11, 2019 at 3:23 PM Gökhan IŞIK wrote: > In [1] it says "Image locking is added and enabled by default. Multiple > QEMU processes cannot write to the same image as long as the host supports > OFD or posix locking, unless options are specified otherwise." May be need > to do something on nova side. > > I run this command and get same error. Output is in > http://paste.openstack.org/show/754311/ > > İf I run qemu-img info instance-0000219b with -U , it doesn't give any > errors. > > [1] https://wiki.qemu.org/ChangeLog/2.10 > > Donny Davis , 11 Tem 2019 Per, 22:11 tarihinde şunu > yazdı: > >> Well that is interesting. If you look in your libvirt config directory >> (/etc/libvirt on Centos) you can get a little more info on what is being >> used for locking. >> >> Maybe strace can shed some light on it. Try something like >> >> strace -ttt -f qemu-img info >> /var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk >> >> >> >> >> >> On Thu, Jul 11, 2019 at 2:39 PM Gökhan IŞIK >> wrote: >> >>> I run virsh list --all command and output is below: >>> >>> root at compute06:~# virsh list --all >>> Id Name State >>> ---------------------------------------------------- >>> - instance-000012f9 shut off >>> - instance-000013b6 shut off >>> - instance-000016fb shut off >>> - instance-0000190a shut off >>> - instance-00001a8a shut off >>> - instance-00001e05 shut off >>> - instance-0000202a shut off >>> - instance-00002135 shut off >>> - instance-00002141 shut off >>> - instance-000021b6 shut off >>> - instance-000021ec shut off >>> - instance-000023db shut off >>> - instance-00002ad7 shut off >>> >>> And also when I try start instances with virsh , output is below: >>> >>> root at compute06:~# virsh start instance-0000219b >>> error: Failed to start domain instance-000012f9 >>> error: internal error: process exited while connecting to monitor: >>> 2019-07-11T18:36:34.229534Z qemu-system-x86_64: -chardev >>> pty,id=charserial0,logfile=/dev/fdset/2,logappend=on: char device >>> redirected to /dev/pts/3 (label charserial0) >>> 2019-07-11T18:36:34.243395Z qemu-system-x86_64: -drive >>> file=/var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=ignore: >>> Failed to get "write" lock >>> Is another process using the image? >>> >>> Thanks, >>> Gökhan >>> >>> Donny Davis , 11 Tem 2019 Per, 21:06 tarihinde >>> şunu yazdı: >>> >>>> Can you ssh to the hypervisor and run virsh list to make sure your >>>> instances are in fact down? >>>> >>>> On Thu, Jul 11, 2019 at 3:02 AM Gökhan IŞIK >>>> wrote: >>>> >>>>> Can anyone help me please ? I can no't rescue my instances yet :( >>>>> >>>>> Thanks, >>>>> Gökhan >>>>> >>>>> Gökhan IŞIK , 9 Tem 2019 Sal, 15:46 >>>>> tarihinde şunu yazdı: >>>>> >>>>>> Hi folks, >>>>>> Because of power outage, Most of our compute nodes unexpectedly >>>>>> shut down and now I can not start our instances. Error message is "Failed >>>>>> to get "write" lock another process using the image?". Instances Power >>>>>> status is No State. Full error log is >>>>>> http://paste.openstack.org/show/754107/. My environment is OpenStack >>>>>> Pike on Ubuntu 16.04 LTS servers and Instances are on a nfs shared storage. >>>>>> Nova version is 16.1.6.dev2. qemu version is 2.10.1. libvirt version is >>>>>> 3.6.0. I saw a commit [1], but it doesn't solve this problem. >>>>>> There are important instances on my environment. How can I rescue my >>>>>> instances? What would you suggest ? >>>>>> >>>>>> Thanks, >>>>>> Gökhan >>>>>> >>>>>> [1] https://review.opendev.org/#/c/509774/ >>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.mcginnis at gmx.com Thu Jul 11 20:16:11 2019 From: sean.mcginnis at gmx.com (Sean McGinnis) Date: Thu, 11 Jul 2019 15:16:11 -0500 Subject: Cinder issue with DataCore In-Reply-To: References: Message-ID: <20190711201611.GA26823@sm-workstation> On Thu, Jul 11, 2019 at 04:07:49PM +0100, Grant Morley wrote: > Hi All, > > We are trying to test DataCore storage backend for cinder ( running on > Queens ). We have everything installed and have the cinder config all setup. > However whenever we try and start the "cinder-volume" service, we get the > following error: > Just a word of warning so you don't have any bad surprises later - the DataCore driver was no longer being maintained so it was deprecated in the Rocky release and removed in Stein. If I remember correctly, they actually stopped running third party CI to test their driver in Queens, but we didn't catch it in time to mark it deprecated in that release. It could very well be fine for Queens, I just don't have any data showing that for sure. > > We have the "websocket-client" installed also: > > pip freeze | grep websocket > websocket-client==0.44.0 Make sure you have it installed in your venv. Try this with: /openstack/venvs/cinder-17.1.2/lib/python2.7/bin/pip freeze | grep websocket > > The datacore libraries also appear to be available in our venvs dirs: > > ls /openstack/venvs/cinder-17.1.2/lib/python2.7/site-packages/cinder/volume/drivers/datacore > api.py  driver.py  exception.py  fc.py  __init__.py  iscsi.py passwd.py  > utils.py > > We are a bit stumped at the moment and wondered if anyone knew what might be > causing the error? We have managed to get Ceph and SolidFire working fine. > > Regards, > > -- > > > Grant Morley > Cloud Lead, Civo Ltd > www.civo.com | Signup for an account! > From cdent+os at anticdent.org Thu Jul 11 21:25:50 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Thu, 11 Jul 2019 22:25:50 +0100 (BST) Subject: [placement][ptg] Shanghai attendance In-Reply-To: <11798408-977b-ecdd-2730-4e7dfb16f38d@fried.cc> References: <11798408-977b-ecdd-2730-4e7dfb16f38d@fried.cc> Message-ID: On Thu, 11 Jul 2019, Eric Fried wrote: >> asked for feedback from the rest of the placement team on whether we >> need to do any placement work there, given the value we got out of >> the virtual pre-PTG in April. >> >> Let me ask again: Do we need a presence at the PTG in Shanghai? > > The email-based virtual pre-PTG was extremely productive. We should > definitely do it again. > > However, I feel like that last hour on Saturday saw more design progress > than weeks worth of emails or spec reviews could have accomplished. To > me, this is the value of the in-person meetups. It sucks that we can't > involve everybody in the discussions - but we can rapidly crystallize an > idea enough to produce a coherent spec that *can* then involve everybody. I don't dispute that, and if sufficient people are there, then I hope that such conversations can happen. However, if it's just that hour that is useful, then we don't need to have placement attend in a formal fashion. Spare rooms and a little forethought plus serendipity will do the trick. This is especially the case for Shanghai if the same thing that was true in Denver remains so: most people will be engaged with other meetings and projects [1]. > Having said that, I'm really not coming up with any major design topics > that would benefit from such a meetup this time around. I feel like what > we've accomplished in Train sets us up for a cycle or two of refinement > (perf/docs/refactor/tech-debt) rather than feature work. Yes. It might even be possible to call it mature and close to done. That's what it is if nobody is clamoring for features. [1] BTW: I think this is a good thing. If placement requires three days of 30 people who only work on that talking at one another, we're doing placement completely wrong. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From cdent+os at anticdent.org Thu Jul 11 21:32:30 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Thu, 11 Jul 2019 22:32:30 +0100 (BST) Subject: [placement][ptg] Shanghai attendance In-Reply-To: <98d51bb7ad7d34268927e0fdb47fb8f31c6af6cb.camel@redhat.com> References: <11798408-977b-ecdd-2730-4e7dfb16f38d@fried.cc> <98d51bb7ad7d34268927e0fdb47fb8f31c6af6cb.camel@redhat.com> Message-ID: On Thu, 11 Jul 2019, Sean Mooney wrote: > well if ye do a virtual email based pre-ptg again why not also continue that virtual > concept and consider a google hangout or somehting to allow realtime discussion with video/etherpad. > i personally found the email discussion hard to follow vs an etherpad or gerrit review per topic > but it did have pluses too. Let's work this out closer to the time. If it's what people want we can certainly do it. I'd vote against it, but definitely not block it. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From fungi at yuggoth.org Thu Jul 11 21:55:48 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 11 Jul 2019 21:55:48 +0000 Subject: [placement][ptg] Shanghai attendance In-Reply-To: References: <11798408-977b-ecdd-2730-4e7dfb16f38d@fried.cc> <98d51bb7ad7d34268927e0fdb47fb8f31c6af6cb.camel@redhat.com> Message-ID: <20190711215548.n2idwnmtzdzov6pf@yuggoth.org> On 2019-07-11 22:32:30 +0100 (+0100), Chris Dent wrote: > On Thu, 11 Jul 2019, Sean Mooney wrote: > > > well if ye do a virtual email based pre-ptg again why not also > > continue that virtual concept and consider a google hangout or > > somehting to allow realtime discussion with video/etherpad. i > > personally found the email discussion hard to follow vs an > > etherpad or gerrit review per topic but it did have pluses too. > > Let's work this out closer to the time. If it's what people want we > can certainly do it. I'd vote against it, but definitely not block > it. Or pop some popcorn and sit back... they're going to have a devil of a time getting to Google Hangouts from behind the GFWoC. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From donny at fortnebula.com Thu Jul 11 21:56:00 2019 From: donny at fortnebula.com (Donny Davis) Date: Thu, 11 Jul 2019 17:56:00 -0400 Subject: [Nova] Instances can't be started after compute nodes unexpectedly shut down because of power outage In-Reply-To: References: Message-ID: Of course you can also always just pull the disk images from the vm folders, merge them back with the base file, upload to glance and then relaunch the instances. You can give this method a spin with the lowest risk to your instances https://medium.com/@kumar_pravin/qemu-merge-snapshot-and-backing-file-into-standalone-disk-c8d3a2b17c0e On Thu, Jul 11, 2019 at 4:10 PM Donny Davis wrote: > You surely want to leave locking turned on. > > You may want to ask qemu-devel about the locking of a image file and how > it works. This isn't really an Openstack issue, seems to be a layer below. > > Depending on how mission critical your VM's are, you could probably work > around it by just passing in --force-share into the command openstack is > trying to run. > > I cannot recommend this path, the best way is to find out how you remove > the lock. > > > > > > > On Thu, Jul 11, 2019 at 3:23 PM Gökhan IŞIK > wrote: > >> In [1] it says "Image locking is added and enabled by default. Multiple >> QEMU processes cannot write to the same image as long as the host supports >> OFD or posix locking, unless options are specified otherwise." May be need >> to do something on nova side. >> >> I run this command and get same error. Output is in >> http://paste.openstack.org/show/754311/ >> >> İf I run qemu-img info instance-0000219b with -U , it doesn't give any >> errors. >> >> [1] https://wiki.qemu.org/ChangeLog/2.10 >> >> Donny Davis , 11 Tem 2019 Per, 22:11 tarihinde >> şunu yazdı: >> >>> Well that is interesting. If you look in your libvirt config directory >>> (/etc/libvirt on Centos) you can get a little more info on what is being >>> used for locking. >>> >>> Maybe strace can shed some light on it. Try something like >>> >>> strace -ttt -f qemu-img info >>> /var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk >>> >>> >>> >>> >>> >>> On Thu, Jul 11, 2019 at 2:39 PM Gökhan IŞIK >>> wrote: >>> >>>> I run virsh list --all command and output is below: >>>> >>>> root at compute06:~# virsh list --all >>>> Id Name State >>>> ---------------------------------------------------- >>>> - instance-000012f9 shut off >>>> - instance-000013b6 shut off >>>> - instance-000016fb shut off >>>> - instance-0000190a shut off >>>> - instance-00001a8a shut off >>>> - instance-00001e05 shut off >>>> - instance-0000202a shut off >>>> - instance-00002135 shut off >>>> - instance-00002141 shut off >>>> - instance-000021b6 shut off >>>> - instance-000021ec shut off >>>> - instance-000023db shut off >>>> - instance-00002ad7 shut off >>>> >>>> And also when I try start instances with virsh , output is below: >>>> >>>> root at compute06:~# virsh start instance-0000219b >>>> error: Failed to start domain instance-000012f9 >>>> error: internal error: process exited while connecting to monitor: >>>> 2019-07-11T18:36:34.229534Z qemu-system-x86_64: -chardev >>>> pty,id=charserial0,logfile=/dev/fdset/2,logappend=on: char device >>>> redirected to /dev/pts/3 (label charserial0) >>>> 2019-07-11T18:36:34.243395Z qemu-system-x86_64: -drive >>>> file=/var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=ignore: >>>> Failed to get "write" lock >>>> Is another process using the image? >>>> >>>> Thanks, >>>> Gökhan >>>> >>>> Donny Davis , 11 Tem 2019 Per, 21:06 tarihinde >>>> şunu yazdı: >>>> >>>>> Can you ssh to the hypervisor and run virsh list to make sure your >>>>> instances are in fact down? >>>>> >>>>> On Thu, Jul 11, 2019 at 3:02 AM Gökhan IŞIK >>>>> wrote: >>>>> >>>>>> Can anyone help me please ? I can no't rescue my instances yet :( >>>>>> >>>>>> Thanks, >>>>>> Gökhan >>>>>> >>>>>> Gökhan IŞIK , 9 Tem 2019 Sal, 15:46 >>>>>> tarihinde şunu yazdı: >>>>>> >>>>>>> Hi folks, >>>>>>> Because of power outage, Most of our compute nodes unexpectedly >>>>>>> shut down and now I can not start our instances. Error message is "Failed >>>>>>> to get "write" lock another process using the image?". Instances Power >>>>>>> status is No State. Full error log is >>>>>>> http://paste.openstack.org/show/754107/. My environment is >>>>>>> OpenStack Pike on Ubuntu 16.04 LTS servers and Instances are on a nfs >>>>>>> shared storage. Nova version is 16.1.6.dev2. qemu version is 2.10.1. >>>>>>> libvirt version is 3.6.0. I saw a commit [1], but it doesn't solve this >>>>>>> problem. >>>>>>> There are important instances on my environment. How can I rescue my >>>>>>> instances? What would you suggest ? >>>>>>> >>>>>>> Thanks, >>>>>>> Gökhan >>>>>>> >>>>>>> [1] https://review.opendev.org/#/c/509774/ >>>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Thu Jul 11 23:33:09 2019 From: smooney at redhat.com (Sean Mooney) Date: Fri, 12 Jul 2019 00:33:09 +0100 Subject: [placement][ptg] Shanghai attendance In-Reply-To: <20190711215548.n2idwnmtzdzov6pf@yuggoth.org> References: <11798408-977b-ecdd-2730-4e7dfb16f38d@fried.cc> <98d51bb7ad7d34268927e0fdb47fb8f31c6af6cb.camel@redhat.com> <20190711215548.n2idwnmtzdzov6pf@yuggoth.org> Message-ID: On Thu, 2019-07-11 at 21:55 +0000, Jeremy Stanley wrote: > On 2019-07-11 22:32:30 +0100 (+0100), Chris Dent wrote: > > On Thu, 11 Jul 2019, Sean Mooney wrote: > > > > > well if ye do a virtual email based pre-ptg again why not also > > > continue that virtual concept and consider a google hangout or > > > somehting to allow realtime discussion with video/etherpad. i > > > personally found the email discussion hard to follow vs an > > > etherpad or gerrit review per topic but it did have pluses too. > > > > Let's work this out closer to the time. If it's what people want we > > can certainly do it. I'd vote against it, but definitely not block > > it. well it was more of a suggestion that if a.) hallway chats/ spare rooms was not enough and b.) realtime "virtual" face 2 face time was needed for some reason an online meeting could proably be tried a a fall back. if its not needed great. > > Or pop some popcorn and sit back... they're going to have a devil of > a time getting to Google Hangouts from behind the GFWoC. hehe ya i was actully thinking of still before the ptg. hangout during the ptg praobly wont be much of an option. From gagehugo at gmail.com Thu Jul 11 23:42:59 2019 From: gagehugo at gmail.com (Gage Hugo) Date: Thu, 11 Jul 2019 18:42:59 -0500 Subject: [security sig] Weekly Newsletter July 11th 2019 Message-ID: #Week of: 11 July 2019 - Security SIG Meeting Info: http://eavesdrop.openstack.org/#Security_SIG_meeting - Weekly on Thursday at 1500 UTC in #openstack-meeting - Agenda: https://etherpad.openstack.org/p/security-agenda - https://security.openstack.org/ - https://wiki.openstack.org/wiki/Security-SIG #Meeting Notes - Summary: http://eavesdrop.openstack.org/meetings/security/2019/security.2019-07-11-15.00.html - Image Encryption Pop-Up Team Meeting - The image encryption pop-up team has an official meeting! See http://eavesdrop.openstack.org/#Image_Encryption_Popup-Team_Meeting - Syntribos still in use - It appears that there is an occasional update to the Syntribos project (outside of maintenance patches). There will be another email send out to see if this project is still being used. - Security Guide Cleanup - nickthetait has been looking into cleaning up & updating the OpenStack Security Guide, yay! - Shanghai PTG Attendance - We have been asked to check how many people are planning on attending the PTG/Forum for the Security SIG. There will be another email sent out for this. # VMT Reports - A full list of publicly marked security issues can be found here: https://bugs.launchpad.net/ossa/ - No new public security bugs this week -------------- next part -------------- An HTML attachment was scrubbed... URL: From gagehugo at gmail.com Thu Jul 11 23:47:06 2019 From: gagehugo at gmail.com (Gage Hugo) Date: Thu, 11 Jul 2019 18:47:06 -0500 Subject: [Security SIG] Shanghai PTG Attendence Message-ID: Hello everyone, We have been asked to look into how many are planning on attending the PTG/Forum for the Security SIG. I have created an etherpad to collect a list of people who are planning on attending[0]. If you are interested in Security or the SIG, please update the etherpad. Thanks! [0] https://etherpad.openstack.org/p/security-shanghai-ptg-planning -------------- next part -------------- An HTML attachment was scrubbed... URL: From gagehugo at gmail.com Thu Jul 11 23:53:43 2019 From: gagehugo at gmail.com (Gage Hugo) Date: Thu, 11 Jul 2019 18:53:43 -0500 Subject: [Syntribos] Does anyone still use this? Message-ID: The Security SIG has recently been looking into updating several sites/documentation and one part we discussed the last couple meetings was the security tools listings[0]. One of the projects listed there, Syntribos, doesn't appear to have much activity[1] outside of zuul/python/docs maintenance and the idea of retiring the project was mentioned. However, there does seem to be an occasional bug-fix submitted, so we are sending this email out to see if anyone is still utilizing Syntribos. If so, please either respond here, reach out to us in the #openstack-security irc channel, or fill out the section in the Security SIG Agenda etherpad[2]. Thanks! [0] https://security.openstack.org/#security-tool-development [1] https://review.opendev.org/#/q/project:openstack/syntribos [2] https://etherpad.openstack.org/p/security-agenda -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Fri Jul 12 04:12:35 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Fri, 12 Jul 2019 06:12:35 +0200 Subject: [queens][nova] nova host-evacuate errot In-Reply-To: References: Message-ID: Jay, for recovering vm state use the command nova reset-state.... nova help reset-state to check the command requested parameters. Ad far as evacuation la concerned, how many compute nodes do gli have ? Instance live migration works? Are gli using shared cinder storage? Ignazio Il Gio 11 Lug 2019 20:51 Jay See ha scritto: > Thanks for explanation Ignazio. > > I have tried same same by trying to put the compute node on a failure > (echo 'c' > /proc/sysrq-trigger ). Compute node was stuck and I was not > able connect to it. > All the VMs are now in Error state. > > Running the host-evacaute was successful on controller node, but now I am > not able to use the VMs. Because they are all in error state now. > > root at h004:~$ nova host-evacuate h017 > > +--------------------------------------+-------------------+---------------+ > | Server UUID | Evacuate Accepted | Error Message > | > > +--------------------------------------+-------------------+---------------+ > | f3545f7d-b85e-49ee-b407-333a4c5b5ab9 | True | > | > | 9094494b-cfa3-459b-8d51-d9aae0ea9636 | True | > | > | abe7075b-ac22-4168-bf3d-d302ba37d80e | True | > | > | c9919371-5f2e-4155-a01a-5f41d9c8b0e7 | True | > | > | ffd983bb-851e-4314-9d1d-375303c278f3 | True | > | > > +--------------------------------------+-------------------+---------------+ > > Now I have restarted the compute node manually , now I am able to connect > to the compute node but VMs are still in Error state. > 1. Any ideas, how to recover the VMs? > 2. Are there any other methods to evacuate, as this method seems to be not > working in mitaka version. > > ~Jay. > > On Thu, Jul 11, 2019 at 1:33 PM Ignazio Cassano > wrote: > >> Ok Jay, >> let me to describe my environment. >> I have an openstack made up of 3 controllers nodes ad several compute >> nodes. >> The controller nodes services are controlled by pacemaker and the compute >> nodes services are controlled by remote pacemaker. >> My hardware is Dell so I am using ipmi fencing device . >> I wrote a service controlled by pacemaker: >> this service controls if a compude node fails and for avoiding split >> brains if a compute node does nod respond on the management network and on >> storage network the stonith poweroff the node and then execute a nova >> host-evacuate. >> >> Anycase to have a simulation before writing the service I described above >> you can do as follows: >> >> connect on one compute node where some virtual machines are running >> run the command: echo 'c' > /proc/sysrq-trigger (it stops immediately the >> node like in case of failure) >> On a controller node run: nova host-evacuate "name of failed compute >> node" >> Instances running on the failed compute node should be restarted on >> another compute node >> >> >> Ignazio >> >> Il giorno gio 11 lug 2019 alle ore 11:57 Jay See < >> jayachander.it at gmail.com> ha scritto: >> >>> Hi , >>> >>> I have tried on a failed compute node which is in power off state now. >>> I have tried on a running compute node, no errors. But nothing happens. >>> On running compute node - Disabled the compute service and tried >>> migration also. >>> >>> May be I might have not followed proper steps. Just wanted to know the >>> steps you have followed. Otherwise, I was planning to manual migration also >>> if possible. >>> ~Jay. >>> >>> On Thu, Jul 11, 2019 at 11:52 AM Ignazio Cassano < >>> ignaziocassano at gmail.com> wrote: >>> >>>> Hi Jay, >>>> would you like to evacuate a failed compute node or evacuate a running >>>> compute node ? >>>> >>>> Ignazio >>>> >>>> Il giorno gio 11 lug 2019 alle ore 11:48 Jay See < >>>> jayachander.it at gmail.com> ha scritto: >>>> >>>>> Hi Ignazio, >>>>> >>>>> I am trying to evacuate the compute host on older version (mitaka). >>>>> Could please share the process you followed. I am not able to succeed >>>>> with openstack live-migration fails with error message (this is known issue >>>>> in older versions) and nova live-ligration - nothing happens even after >>>>> initiating VM migration. It is almost 4 days. >>>>> >>>>> ~Jay. >>>>> >>>>> On Thu, Jul 11, 2019 at 11:31 AM Ignazio Cassano < >>>>> ignaziocassano at gmail.com> wrote: >>>>> >>>>>> I am sorry. >>>>>> For simulating an host crash I used a wrong procedure. >>>>>> Using "echo 'c' > /proc/sysrq-trigger" all work fine >>>>>> >>>>>> Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano < >>>>>> ignaziocassano at gmail.com> ha scritto: >>>>>> >>>>>>> Hello All, >>>>>>> on ocata when I poweroff a node with active instance , doing a nova >>>>>>> host-evacuate works fine >>>>>>> and instances are restartd on an active node. >>>>>>> On queens it does non evacuate instances but nova-api reports for >>>>>>> each instance the following: >>>>>>> >>>>>>> 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi >>>>>>> [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 >>>>>>> c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: >>>>>>> Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is >>>>>>> in task_state powering-off >>>>>>> >>>>>>> So it poweroff all instance on the failed node but does not start >>>>>>> them on active nodes >>>>>>> >>>>>>> What is changed ? >>>>>>> Ignazio >>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> -- >>>>> ​ >>>>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>>>> necessary.* >>>>> >>>> >>> >>> -- >>> ​ >>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>> necessary.* >>> >> > > -- > ​ > P *SAVE PAPER – Please do not print this e-mail unless absolutely > necessary.* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jyotishri403 at gmail.com Fri Jul 12 04:16:01 2019 From: jyotishri403 at gmail.com (Jyoti Dahiwele) Date: Fri, 12 Jul 2019 09:46:01 +0530 Subject: Flow of instance creation Message-ID: Dear Everyone, Can you explain me the flow of instance creation in stein ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From skylightcoder at gmail.com Fri Jul 12 07:46:31 2019 From: skylightcoder at gmail.com (=?UTF-8?B?R8O2a2hhbiBJxZ5JSw==?=) Date: Fri, 12 Jul 2019 10:46:31 +0300 Subject: [Nova] Instances can't be started after compute nodes unexpectedly shut down because of power outage In-Reply-To: References: Message-ID: Awesome, thanks! Donny, I followed below steps and rescue my instance. 1. Find instance id and compute host root at infra1-utility-container-50bcf920:~# openstack server show 1d2e8a39-97ee-4ce7-a612-1b50f90cc51e -c id -c OS-EXT-SRV-ATTR:hypervisor_hostname +-------------------------------------+--------------------------------------+ | Field | Value | +-------------------------------------+--------------------------------------+ | OS-EXT-SRV-ATTR:hypervisor_hostname | compute06 | | id | 1d2e8a39-97ee-4ce7-a612-1b50f90cc51e | +-------------------------------------+--------------------------------------+ 2. Find image and backing image file on compute host root at compute06:~# qemu-img info -U --backing-chain /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk image: /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk file format: qcow2 virtual size: 160G (171798691840 bytes) disk size: 32G cluster_size: 65536 backing file: /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false image: /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323 file format: raw virtual size: 160G (171798691840 bytes) disk size: 18G 3. Copy image and backing image file root at compute06:~# cp /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk master root at compute06:~# cp /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323 new-master 4. Rebase the image file that was backed off the original file so that it uses the new file i.e new-master then commit those changes back to original file master back into the new base new-master root at compute06:~# qemu-img rebase -b new-master -U master root at compute06:~# qemu-img commit master root at compute06:~# qemu-img info new-master 5. Convert raw image to qcow2 root at compute06:~# qemu-img convert -f raw -O qcow2 new-master new-master.qcow2 6. Time to upload glance and then launch instance from this image :) Thanks, Gökhan. Donny Davis , 12 Tem 2019 Cum, 00:56 tarihinde şunu yazdı: > Of course you can also always just pull the disk images from the vm > folders, merge them back with the base file, upload to glance and then > relaunch the instances. > > You can give this method a spin with the lowest risk to your instances > > > https://medium.com/@kumar_pravin/qemu-merge-snapshot-and-backing-file-into-standalone-disk-c8d3a2b17c0e > > > > > > On Thu, Jul 11, 2019 at 4:10 PM Donny Davis wrote: > >> You surely want to leave locking turned on. >> >> You may want to ask qemu-devel about the locking of a image file and how >> it works. This isn't really an Openstack issue, seems to be a layer below. >> >> Depending on how mission critical your VM's are, you could probably work >> around it by just passing in --force-share into the command openstack is >> trying to run. >> >> I cannot recommend this path, the best way is to find out how you remove >> the lock. >> >> >> >> >> >> >> On Thu, Jul 11, 2019 at 3:23 PM Gökhan IŞIK >> wrote: >> >>> In [1] it says "Image locking is added and enabled by default. Multiple >>> QEMU processes cannot write to the same image as long as the host supports >>> OFD or posix locking, unless options are specified otherwise." May be need >>> to do something on nova side. >>> >>> I run this command and get same error. Output is in >>> http://paste.openstack.org/show/754311/ >>> >>> İf I run qemu-img info instance-0000219b with -U , it doesn't give any >>> errors. >>> >>> [1] https://wiki.qemu.org/ChangeLog/2.10 >>> >>> Donny Davis , 11 Tem 2019 Per, 22:11 tarihinde >>> şunu yazdı: >>> >>>> Well that is interesting. If you look in your libvirt config directory >>>> (/etc/libvirt on Centos) you can get a little more info on what is being >>>> used for locking. >>>> >>>> Maybe strace can shed some light on it. Try something like >>>> >>>> strace -ttt -f qemu-img info >>>> /var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Jul 11, 2019 at 2:39 PM Gökhan IŞIK >>>> wrote: >>>> >>>>> I run virsh list --all command and output is below: >>>>> >>>>> root at compute06:~# virsh list --all >>>>> Id Name State >>>>> ---------------------------------------------------- >>>>> - instance-000012f9 shut off >>>>> - instance-000013b6 shut off >>>>> - instance-000016fb shut off >>>>> - instance-0000190a shut off >>>>> - instance-00001a8a shut off >>>>> - instance-00001e05 shut off >>>>> - instance-0000202a shut off >>>>> - instance-00002135 shut off >>>>> - instance-00002141 shut off >>>>> - instance-000021b6 shut off >>>>> - instance-000021ec shut off >>>>> - instance-000023db shut off >>>>> - instance-00002ad7 shut off >>>>> >>>>> And also when I try start instances with virsh , output is below: >>>>> >>>>> root at compute06:~# virsh start instance-0000219b >>>>> error: Failed to start domain instance-000012f9 >>>>> error: internal error: process exited while connecting to monitor: >>>>> 2019-07-11T18:36:34.229534Z qemu-system-x86_64: -chardev >>>>> pty,id=charserial0,logfile=/dev/fdset/2,logappend=on: char device >>>>> redirected to /dev/pts/3 (label charserial0) >>>>> 2019-07-11T18:36:34.243395Z qemu-system-x86_64: -drive >>>>> file=/var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=ignore: >>>>> Failed to get "write" lock >>>>> Is another process using the image? >>>>> >>>>> Thanks, >>>>> Gökhan >>>>> >>>>> Donny Davis , 11 Tem 2019 Per, 21:06 tarihinde >>>>> şunu yazdı: >>>>> >>>>>> Can you ssh to the hypervisor and run virsh list to make sure your >>>>>> instances are in fact down? >>>>>> >>>>>> On Thu, Jul 11, 2019 at 3:02 AM Gökhan IŞIK >>>>>> wrote: >>>>>> >>>>>>> Can anyone help me please ? I can no't rescue my instances yet :( >>>>>>> >>>>>>> Thanks, >>>>>>> Gökhan >>>>>>> >>>>>>> Gökhan IŞIK , 9 Tem 2019 Sal, 15:46 >>>>>>> tarihinde şunu yazdı: >>>>>>> >>>>>>>> Hi folks, >>>>>>>> Because of power outage, Most of our compute nodes unexpectedly >>>>>>>> shut down and now I can not start our instances. Error message is "Failed >>>>>>>> to get "write" lock another process using the image?". Instances Power >>>>>>>> status is No State. Full error log is >>>>>>>> http://paste.openstack.org/show/754107/. My environment is >>>>>>>> OpenStack Pike on Ubuntu 16.04 LTS servers and Instances are on a nfs >>>>>>>> shared storage. Nova version is 16.1.6.dev2. qemu version is 2.10.1. >>>>>>>> libvirt version is 3.6.0. I saw a commit [1], but it doesn't solve this >>>>>>>> problem. >>>>>>>> There are important instances on my environment. How can I rescue >>>>>>>> my instances? What would you suggest ? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Gökhan >>>>>>>> >>>>>>>> [1] https://review.opendev.org/#/c/509774/ >>>>>>>> >>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From grant at civo.com Fri Jul 12 08:28:06 2019 From: grant at civo.com (Grant Morley) Date: Fri, 12 Jul 2019 09:28:06 +0100 Subject: Cinder issue with DataCore In-Reply-To: <20190711201611.GA26823@sm-workstation> References: <20190711201611.GA26823@sm-workstation> Message-ID: Hi Sean, Thanks for that. Luckily we can give the device back and we haven't paid anything for it either. I'll stop trying to get it working now then :) Regards, On 11/07/2019 21:16, Sean McGinnis wrote: > On Thu, Jul 11, 2019 at 04:07:49PM +0100, Grant Morley wrote: >> Hi All, >> >> We are trying to test DataCore storage backend for cinder ( running on >> Queens ). We have everything installed and have the cinder config all setup. >> However whenever we try and start the "cinder-volume" service, we get the >> following error: >> > Just a word of warning so you don't have any bad surprises later - the DataCore > driver was no longer being maintained so it was deprecated in the Rocky release > and removed in Stein. > > If I remember correctly, they actually stopped running third party CI to test > their driver in Queens, but we didn't catch it in time to mark it deprecated in > that release. It could very well be fine for Queens, I just don't have any data > showing that for sure. > >> We have the "websocket-client" installed also: >> >> pip freeze | grep websocket >> websocket-client==0.44.0 > Make sure you have it installed in your venv. Try this with: > > /openstack/venvs/cinder-17.1.2/lib/python2.7/bin/pip freeze | grep websocket > >> The datacore libraries also appear to be available in our venvs dirs: >> >> ls /openstack/venvs/cinder-17.1.2/lib/python2.7/site-packages/cinder/volume/drivers/datacore >> api.py  driver.py  exception.py  fc.py  __init__.py  iscsi.py passwd.py >> utils.py >> >> We are a bit stumped at the moment and wondered if anyone knew what might be >> causing the error? We have managed to get Ceph and SolidFire working fine. >> >> Regards, >> >> -- >> >> >> Grant Morley >> Cloud Lead, Civo Ltd >> www.civo.com | Signup for an account! >> -- Grant Morley Cloud Lead, Civo Ltd www.civo.com | Signup for an account! -------------- next part -------------- An HTML attachment was scrubbed... URL: From sfinucan at redhat.com Fri Jul 12 09:12:37 2019 From: sfinucan at redhat.com (Stephen Finucane) Date: Fri, 12 Jul 2019 10:12:37 +0100 Subject: [nova][ec2] Removal of unused utility methods from 'nova.api.ec2' Message-ID: <4f01bfc7bd76a37e7bdb1630e6a1ee8fc767cec5.camel@redhat.com> I have a couple of patches up to remove what looks like a large chunk of unused ec2 API code from nova. The patches start at [1] and notes are provided in earlier revisions of those reviews ([2], [3], [4]) about why I'm able to remove some things and not others. I have a DNM patch proposed against the ec2-api [5] to ensure that things really don't break there but in case someone is maintaining something outside of opendev.org (I'd have found it on codesearch.openstack.org) that really needs these, you need to speak up on the review asap. If not, we'll get going with this. Stephen PS: To be clear, ec2-api *will still work*. None of the things we're removing are used by that project or we wouldn't be removing them :) [1] https://review.opendev.org/#/c/662501/ [2] https://review.opendev.org/#/c/662501/1/nova/api/ec2/ec2utils.py [3] https://review.opendev.org/#/c/662502/1/nova/objects/ec2.py [4] https://review.opendev.org/#/c/662503/1/nova/api/ec2/cloud.py [5] https://review.opendev.org/#/c/663386/ From jayachander.it at gmail.com Fri Jul 12 09:26:00 2019 From: jayachander.it at gmail.com (Jay See) Date: Fri, 12 Jul 2019 11:26:00 +0200 Subject: [queens][nova] nova host-evacuate errot In-Reply-To: References: Message-ID: Ignazio, One instance is stuck in error state not able to recover it. All other instances are running now. root at h004:~$ nova reset-state --all-tenants my-instance-1-2 Reset state for server my-instance-1-2 succeeded; new state is error I have several compute nodes (14). I am not sure what is gli? Live migration is not working, i have tried it was not throwing any errors. But nothing seems to happen. I am not completely sure, I haven't heard about gli before. (This setup is deployed by someone else). ~Jay. On Fri, Jul 12, 2019 at 6:12 AM Ignazio Cassano wrote: > Jay, for recovering vm state use the command nova reset-state.... > > nova help reset-state to check the command requested parameters. > > Ad far as evacuation la concerned, how many compute nodes do gli have ? > Instance live migration works? > Are gli using shared cinder storage? > Ignazio > > Il Gio 11 Lug 2019 20:51 Jay See ha scritto: > >> Thanks for explanation Ignazio. >> >> I have tried same same by trying to put the compute node on a failure >> (echo 'c' > /proc/sysrq-trigger ). Compute node was stuck and I was not >> able connect to it. >> All the VMs are now in Error state. >> >> Running the host-evacaute was successful on controller node, but now I am >> not able to use the VMs. Because they are all in error state now. >> >> root at h004:~$ nova host-evacuate h017 >> >> +--------------------------------------+-------------------+---------------+ >> | Server UUID | Evacuate Accepted | Error >> Message | >> >> +--------------------------------------+-------------------+---------------+ >> | f3545f7d-b85e-49ee-b407-333a4c5b5ab9 | True | >> | >> | 9094494b-cfa3-459b-8d51-d9aae0ea9636 | True | >> | >> | abe7075b-ac22-4168-bf3d-d302ba37d80e | True | >> | >> | c9919371-5f2e-4155-a01a-5f41d9c8b0e7 | True | >> | >> | ffd983bb-851e-4314-9d1d-375303c278f3 | True | >> | >> >> +--------------------------------------+-------------------+---------------+ >> >> Now I have restarted the compute node manually , now I am able to connect >> to the compute node but VMs are still in Error state. >> 1. Any ideas, how to recover the VMs? >> 2. Are there any other methods to evacuate, as this method seems to be >> not working in mitaka version. >> >> ~Jay. >> >> On Thu, Jul 11, 2019 at 1:33 PM Ignazio Cassano >> wrote: >> >>> Ok Jay, >>> let me to describe my environment. >>> I have an openstack made up of 3 controllers nodes ad several compute >>> nodes. >>> The controller nodes services are controlled by pacemaker and the >>> compute nodes services are controlled by remote pacemaker. >>> My hardware is Dell so I am using ipmi fencing device . >>> I wrote a service controlled by pacemaker: >>> this service controls if a compude node fails and for avoiding split >>> brains if a compute node does nod respond on the management network and on >>> storage network the stonith poweroff the node and then execute a nova >>> host-evacuate. >>> >>> Anycase to have a simulation before writing the service I described >>> above you can do as follows: >>> >>> connect on one compute node where some virtual machines are running >>> run the command: echo 'c' > /proc/sysrq-trigger (it stops immediately >>> the node like in case of failure) >>> On a controller node run: nova host-evacuate "name of failed compute >>> node" >>> Instances running on the failed compute node should be restarted on >>> another compute node >>> >>> >>> Ignazio >>> >>> Il giorno gio 11 lug 2019 alle ore 11:57 Jay See < >>> jayachander.it at gmail.com> ha scritto: >>> >>>> Hi , >>>> >>>> I have tried on a failed compute node which is in power off state now. >>>> I have tried on a running compute node, no errors. But nothing happens. >>>> On running compute node - Disabled the compute service and tried >>>> migration also. >>>> >>>> May be I might have not followed proper steps. Just wanted to know the >>>> steps you have followed. Otherwise, I was planning to manual migration also >>>> if possible. >>>> ~Jay. >>>> >>>> On Thu, Jul 11, 2019 at 11:52 AM Ignazio Cassano < >>>> ignaziocassano at gmail.com> wrote: >>>> >>>>> Hi Jay, >>>>> would you like to evacuate a failed compute node or evacuate a running >>>>> compute node ? >>>>> >>>>> Ignazio >>>>> >>>>> Il giorno gio 11 lug 2019 alle ore 11:48 Jay See < >>>>> jayachander.it at gmail.com> ha scritto: >>>>> >>>>>> Hi Ignazio, >>>>>> >>>>>> I am trying to evacuate the compute host on older version (mitaka). >>>>>> Could please share the process you followed. I am not able to succeed >>>>>> with openstack live-migration fails with error message (this is known issue >>>>>> in older versions) and nova live-ligration - nothing happens even after >>>>>> initiating VM migration. It is almost 4 days. >>>>>> >>>>>> ~Jay. >>>>>> >>>>>> On Thu, Jul 11, 2019 at 11:31 AM Ignazio Cassano < >>>>>> ignaziocassano at gmail.com> wrote: >>>>>> >>>>>>> I am sorry. >>>>>>> For simulating an host crash I used a wrong procedure. >>>>>>> Using "echo 'c' > /proc/sysrq-trigger" all work fine >>>>>>> >>>>>>> Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano < >>>>>>> ignaziocassano at gmail.com> ha scritto: >>>>>>> >>>>>>>> Hello All, >>>>>>>> on ocata when I poweroff a node with active instance , doing a >>>>>>>> nova host-evacuate works fine >>>>>>>> and instances are restartd on an active node. >>>>>>>> On queens it does non evacuate instances but nova-api reports for >>>>>>>> each instance the following: >>>>>>>> >>>>>>>> 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi >>>>>>>> [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 >>>>>>>> c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: >>>>>>>> Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is >>>>>>>> in task_state powering-off >>>>>>>> >>>>>>>> So it poweroff all instance on the failed node but does not start >>>>>>>> them on active nodes >>>>>>>> >>>>>>>> What is changed ? >>>>>>>> Ignazio >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> -- >>>>>> ​ >>>>>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>>>>> necessary.* >>>>>> >>>>> >>>> >>>> -- >>>> ​ >>>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>>> necessary.* >>>> >>> >> >> -- >> ​ >> P *SAVE PAPER – Please do not print this e-mail unless absolutely >> necessary.* >> > -- ​ P *SAVE PAPER – Please do not print this e-mail unless absolutely necessary.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Fri Jul 12 09:52:49 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Fri, 12 Jul 2019 11:52:49 +0200 Subject: [queens][nova] nova host-evacuate errot In-Reply-To: References: Message-ID: Sorry ...the question was : how many compute nodes do you have ? instead of how many compute nodes do gli have... Anycase; Did you configured cinder ? Il giorno ven 12 lug 2019 alle ore 11:26 Jay See ha scritto: > Ignazio, > > One instance is stuck in error state not able to recover it. All other > instances are running now. > > root at h004:~$ nova reset-state --all-tenants my-instance-1-2 > Reset state for server my-instance-1-2 succeeded; new state is error > > I have several compute nodes (14). I am not sure what is gli? > Live migration is not working, i have tried it was not throwing any > errors. But nothing seems to happen. > I am not completely sure, I haven't heard about gli before. (This setup is > deployed by someone else). > > ~Jay. > > On Fri, Jul 12, 2019 at 6:12 AM Ignazio Cassano > wrote: > >> Jay, for recovering vm state use the command nova reset-state.... >> >> nova help reset-state to check the command requested parameters. >> >> Ad far as evacuation la concerned, how many compute nodes do gli have ? >> Instance live migration works? >> Are gli using shared cinder storage? >> Ignazio >> >> Il Gio 11 Lug 2019 20:51 Jay See ha scritto: >> >>> Thanks for explanation Ignazio. >>> >>> I have tried same same by trying to put the compute node on a failure >>> (echo 'c' > /proc/sysrq-trigger ). Compute node was stuck and I was not >>> able connect to it. >>> All the VMs are now in Error state. >>> >>> Running the host-evacaute was successful on controller node, but now I >>> am not able to use the VMs. Because they are all in error state now. >>> >>> root at h004:~$ nova host-evacuate h017 >>> >>> +--------------------------------------+-------------------+---------------+ >>> | Server UUID | Evacuate Accepted | Error >>> Message | >>> >>> +--------------------------------------+-------------------+---------------+ >>> | f3545f7d-b85e-49ee-b407-333a4c5b5ab9 | True | >>> | >>> | 9094494b-cfa3-459b-8d51-d9aae0ea9636 | True | >>> | >>> | abe7075b-ac22-4168-bf3d-d302ba37d80e | True | >>> | >>> | c9919371-5f2e-4155-a01a-5f41d9c8b0e7 | True | >>> | >>> | ffd983bb-851e-4314-9d1d-375303c278f3 | True | >>> | >>> >>> +--------------------------------------+-------------------+---------------+ >>> >>> Now I have restarted the compute node manually , now I am able to >>> connect to the compute node but VMs are still in Error state. >>> 1. Any ideas, how to recover the VMs? >>> 2. Are there any other methods to evacuate, as this method seems to be >>> not working in mitaka version. >>> >>> ~Jay. >>> >>> On Thu, Jul 11, 2019 at 1:33 PM Ignazio Cassano < >>> ignaziocassano at gmail.com> wrote: >>> >>>> Ok Jay, >>>> let me to describe my environment. >>>> I have an openstack made up of 3 controllers nodes ad several compute >>>> nodes. >>>> The controller nodes services are controlled by pacemaker and the >>>> compute nodes services are controlled by remote pacemaker. >>>> My hardware is Dell so I am using ipmi fencing device . >>>> I wrote a service controlled by pacemaker: >>>> this service controls if a compude node fails and for avoiding split >>>> brains if a compute node does nod respond on the management network and on >>>> storage network the stonith poweroff the node and then execute a nova >>>> host-evacuate. >>>> >>>> Anycase to have a simulation before writing the service I described >>>> above you can do as follows: >>>> >>>> connect on one compute node where some virtual machines are running >>>> run the command: echo 'c' > /proc/sysrq-trigger (it stops immediately >>>> the node like in case of failure) >>>> On a controller node run: nova host-evacuate "name of failed compute >>>> node" >>>> Instances running on the failed compute node should be restarted on >>>> another compute node >>>> >>>> >>>> Ignazio >>>> >>>> Il giorno gio 11 lug 2019 alle ore 11:57 Jay See < >>>> jayachander.it at gmail.com> ha scritto: >>>> >>>>> Hi , >>>>> >>>>> I have tried on a failed compute node which is in power off state now. >>>>> I have tried on a running compute node, no errors. But nothing happens. >>>>> On running compute node - Disabled the compute service and tried >>>>> migration also. >>>>> >>>>> May be I might have not followed proper steps. Just wanted to know the >>>>> steps you have followed. Otherwise, I was planning to manual migration also >>>>> if possible. >>>>> ~Jay. >>>>> >>>>> On Thu, Jul 11, 2019 at 11:52 AM Ignazio Cassano < >>>>> ignaziocassano at gmail.com> wrote: >>>>> >>>>>> Hi Jay, >>>>>> would you like to evacuate a failed compute node or evacuate a >>>>>> running compute node ? >>>>>> >>>>>> Ignazio >>>>>> >>>>>> Il giorno gio 11 lug 2019 alle ore 11:48 Jay See < >>>>>> jayachander.it at gmail.com> ha scritto: >>>>>> >>>>>>> Hi Ignazio, >>>>>>> >>>>>>> I am trying to evacuate the compute host on older version (mitaka). >>>>>>> Could please share the process you followed. I am not able to >>>>>>> succeed with openstack live-migration fails with error message (this is >>>>>>> known issue in older versions) and nova live-ligration - nothing happens >>>>>>> even after initiating VM migration. It is almost 4 days. >>>>>>> >>>>>>> ~Jay. >>>>>>> >>>>>>> On Thu, Jul 11, 2019 at 11:31 AM Ignazio Cassano < >>>>>>> ignaziocassano at gmail.com> wrote: >>>>>>> >>>>>>>> I am sorry. >>>>>>>> For simulating an host crash I used a wrong procedure. >>>>>>>> Using "echo 'c' > /proc/sysrq-trigger" all work fine >>>>>>>> >>>>>>>> Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano < >>>>>>>> ignaziocassano at gmail.com> ha scritto: >>>>>>>> >>>>>>>>> Hello All, >>>>>>>>> on ocata when I poweroff a node with active instance , doing a >>>>>>>>> nova host-evacuate works fine >>>>>>>>> and instances are restartd on an active node. >>>>>>>>> On queens it does non evacuate instances but nova-api reports for >>>>>>>>> each instance the following: >>>>>>>>> >>>>>>>>> 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi >>>>>>>>> [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 >>>>>>>>> c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: >>>>>>>>> Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is >>>>>>>>> in task_state powering-off >>>>>>>>> >>>>>>>>> So it poweroff all instance on the failed node but does not start >>>>>>>>> them on active nodes >>>>>>>>> >>>>>>>>> What is changed ? >>>>>>>>> Ignazio >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> ​ >>>>>>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>>>>>> necessary.* >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> ​ >>>>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>>>> necessary.* >>>>> >>>> >>> >>> -- >>> ​ >>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>> necessary.* >>> >> > > -- > ​ > P *SAVE PAPER – Please do not print this e-mail unless absolutely > necessary.* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdent+os at anticdent.org Fri Jul 12 10:00:01 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Fri, 12 Jul 2019 11:00:01 +0100 (BST) Subject: [placement] update 19-27 Message-ID: HTML: https://anticdent.org/placement-update-19-27.html Pupdate 19-27 is here and now. # Most Important Of the features we planned to do this cycle, all are done save one: consumer types (in progress, see below). This means we have a good opportunity to focus on documentation, performance, and improving the codebase for maintainability. You do not need permission to work on these things. If you find a problem and know how to fix it, fix it. If you are not sure about the solution, please discuss it on this email list or in the `#openstack-placement` IRC channel. This also means we're in a good position to help review changes that use placement in other projects. The Foundation needs to know how much, if any, Placement time will be needed in Shanghai. I started a [thread](http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007654.html) and an [etherpad](https://etherpad.openstack.org/p/placement-shanghai-ptg). # What's Changed * The `same_subtree` query parameter has merged as [microversion 1.36](https://docs.openstack.org/placement/latest/placement-api-microversion-history.html#support-same-subtree-queryparam-on-get-allocation-candidates). This enables a form of nested provider affinity: "these providers must all share the same ancestor". * The placement projects have been updated (pending merge) to match the [Python 3 test runtimes for Train](https://governance.openstack.org/tc/goals/train/python3-updates.html) community wide goal. Since the projects have been Python3-enabled from the start, this was mostly a matter of aligning configurations with community-wide norms. When Python 3.8 becomes available we should get on that sooner than later to catch issues early. # Specs/Features All placement specs have merged. Thanks to everyone for the frequent reviews and quick followups. Some non-placement specs are listed in the Other section below. # Stories/Bugs (Numbers in () are the change since the last pupdate.) There are 23 (0) stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 (0) are [untagged](https://storyboard.openstack.org/#!/worklist/580). 3 (1) are [bugs](https://storyboard.openstack.org/#!/worklist/574). 5 (0) are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 11 (0) are [rfes](https://storyboard.openstack.org/#!/worklist/594). 4 (0) are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 16 (0). * Placement related nova [in progress bugs](https://goo.gl/vzGGDQ) on launchpad: 4 (0). # osc-placement osc-placement is currently behind by 11 microversions. * Add support for multiple member_of. This is stuck and needs input from users of the tool. # Main Themes ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. * This is currently the migration to add the necessary table and column. ## Cleanup Cleanup is an overarching theme related to improving documentation, performance and the maintainability of the code. The changes we are making this cycle are fairly complex to use and are fairly complex to write, so it is good that we're going to have plenty of time to clean and clarify all these things. As mentioned last week, one of the important cleanup tasks that is not yet in progress is updating the [gabbit](https://opendev.org/openstack/placement/src/branch/master/gate/gabbits/nested-perfload.yaml) that creates the nested topology that's used in nested performance testing. The topology there is simple, unrealistic, and doesn't sufficiently exercise the several features that may be used during a query that desires a nested response. This needs to be someone who is more closely related to real world use of nested than me. efried? gibi? Another cleanup that needs to start is satisfying the community wide goal of [PDF doc generation](https://storyboard.openstack.org/#!/story/2006110). Anyone know if there is a cookbook for this? # Other Placement Miscellaneous changes can be found in [the usual place](https://review.opendev.org/#/q/project:openstack/placement+status:open). There are three [os-traits changes](https://review.opendev.org/#/q/project:openstack/os-traits+status:open) being discussed. And one [os-resource-classes change](https://review.opendev.org/#/q/project:openstack/os-resource-classes+status:open). # Other Service Users New discoveries are added to the end. Merged stuff is removed. Anything that has had no activity in 4 weeks has been removed. * Nova: nova-manage: heal port allocations * nova-spec: Allow compute nodes to use DISK_GB from shared storage RP * Cyborg: Placement report * helm: add placement chart * Nova: Use OpenStack SDK for placement * Nova: Spec: Provider config YAML file * libvirt: report pmem namespaces resources by provider tree * Nova: Remove PlacementAPIConnectFailure handling from AggregateAPI * Nova: support move ops with qos ports * Nova: get_ksa_adapter: nix by-service-type confgrp hack * OSA: Add nova placement to placement migration * Nova: Defaults missing group_policy to 'none' * Blazar: Create placement client for each request * tempest: Define the Integrated-gate-placement gate template * Nova: Restore RT.old_resources if ComputeNode.save() fails * nova: Support filtering of hosts by forbidden aggregates * blazar: Send global_request_id for tracing calls * Nova: Update HostState.\*\_allocation_ratio earlier * neutron: segments: fix rp inventory update * Nova: WIP: Add a placement audit command * OSA: Add placement client for neutron * zun: [WIP] Use placement for unified resource management # End A colleague suggested yesterday that the universe doesn't have an over subscription problem, rather there's localized contention, and what we really have is a placement problem. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From jayachander.it at gmail.com Fri Jul 12 10:48:07 2019 From: jayachander.it at gmail.com (Jay See) Date: Fri, 12 Jul 2019 12:48:07 +0200 Subject: [queens][nova] nova host-evacuate errot In-Reply-To: References: Message-ID: Yes, cinder is running. root at h017:~$ service --status-all | grep cinder [ + ] cinder-volume On Fri, Jul 12, 2019 at 11:53 AM Ignazio Cassano wrote: > Sorry ...the question was : how many compute nodes do you have ? > instead of how many compute nodes do gli have... > > > Anycase; > Did you configured cinder ? > > Il giorno ven 12 lug 2019 alle ore 11:26 Jay See > ha scritto: > >> Ignazio, >> >> One instance is stuck in error state not able to recover it. All other >> instances are running now. >> >> root at h004:~$ nova reset-state --all-tenants my-instance-1-2 >> Reset state for server my-instance-1-2 succeeded; new state is error >> >> I have several compute nodes (14). I am not sure what is gli? >> Live migration is not working, i have tried it was not throwing any >> errors. But nothing seems to happen. >> I am not completely sure, I haven't heard about gli before. (This setup >> is deployed by someone else). >> >> ~Jay. >> >> On Fri, Jul 12, 2019 at 6:12 AM Ignazio Cassano >> wrote: >> >>> Jay, for recovering vm state use the command nova reset-state.... >>> >>> nova help reset-state to check the command requested parameters. >>> >>> Ad far as evacuation la concerned, how many compute nodes do gli have ? >>> Instance live migration works? >>> Are gli using shared cinder storage? >>> Ignazio >>> >>> Il Gio 11 Lug 2019 20:51 Jay See ha scritto: >>> >>>> Thanks for explanation Ignazio. >>>> >>>> I have tried same same by trying to put the compute node on a failure >>>> (echo 'c' > /proc/sysrq-trigger ). Compute node was stuck and I was not >>>> able connect to it. >>>> All the VMs are now in Error state. >>>> >>>> Running the host-evacaute was successful on controller node, but now I >>>> am not able to use the VMs. Because they are all in error state now. >>>> >>>> root at h004:~$ nova host-evacuate h017 >>>> >>>> +--------------------------------------+-------------------+---------------+ >>>> | Server UUID | Evacuate Accepted | Error >>>> Message | >>>> >>>> +--------------------------------------+-------------------+---------------+ >>>> | f3545f7d-b85e-49ee-b407-333a4c5b5ab9 | True | >>>> | >>>> | 9094494b-cfa3-459b-8d51-d9aae0ea9636 | True | >>>> | >>>> | abe7075b-ac22-4168-bf3d-d302ba37d80e | True | >>>> | >>>> | c9919371-5f2e-4155-a01a-5f41d9c8b0e7 | True | >>>> | >>>> | ffd983bb-851e-4314-9d1d-375303c278f3 | True | >>>> | >>>> >>>> +--------------------------------------+-------------------+---------------+ >>>> >>>> Now I have restarted the compute node manually , now I am able to >>>> connect to the compute node but VMs are still in Error state. >>>> 1. Any ideas, how to recover the VMs? >>>> 2. Are there any other methods to evacuate, as this method seems to be >>>> not working in mitaka version. >>>> >>>> ~Jay. >>>> >>>> On Thu, Jul 11, 2019 at 1:33 PM Ignazio Cassano < >>>> ignaziocassano at gmail.com> wrote: >>>> >>>>> Ok Jay, >>>>> let me to describe my environment. >>>>> I have an openstack made up of 3 controllers nodes ad several compute >>>>> nodes. >>>>> The controller nodes services are controlled by pacemaker and the >>>>> compute nodes services are controlled by remote pacemaker. >>>>> My hardware is Dell so I am using ipmi fencing device . >>>>> I wrote a service controlled by pacemaker: >>>>> this service controls if a compude node fails and for avoiding split >>>>> brains if a compute node does nod respond on the management network and on >>>>> storage network the stonith poweroff the node and then execute a nova >>>>> host-evacuate. >>>>> >>>>> Anycase to have a simulation before writing the service I described >>>>> above you can do as follows: >>>>> >>>>> connect on one compute node where some virtual machines are running >>>>> run the command: echo 'c' > /proc/sysrq-trigger (it stops immediately >>>>> the node like in case of failure) >>>>> On a controller node run: nova host-evacuate "name of failed compute >>>>> node" >>>>> Instances running on the failed compute node should be restarted on >>>>> another compute node >>>>> >>>>> >>>>> Ignazio >>>>> >>>>> Il giorno gio 11 lug 2019 alle ore 11:57 Jay See < >>>>> jayachander.it at gmail.com> ha scritto: >>>>> >>>>>> Hi , >>>>>> >>>>>> I have tried on a failed compute node which is in power off state now. >>>>>> I have tried on a running compute node, no errors. But >>>>>> nothing happens. >>>>>> On running compute node - Disabled the compute service and tried >>>>>> migration also. >>>>>> >>>>>> May be I might have not followed proper steps. Just wanted to know >>>>>> the steps you have followed. Otherwise, I was planning to manual migration >>>>>> also if possible. >>>>>> ~Jay. >>>>>> >>>>>> On Thu, Jul 11, 2019 at 11:52 AM Ignazio Cassano < >>>>>> ignaziocassano at gmail.com> wrote: >>>>>> >>>>>>> Hi Jay, >>>>>>> would you like to evacuate a failed compute node or evacuate a >>>>>>> running compute node ? >>>>>>> >>>>>>> Ignazio >>>>>>> >>>>>>> Il giorno gio 11 lug 2019 alle ore 11:48 Jay See < >>>>>>> jayachander.it at gmail.com> ha scritto: >>>>>>> >>>>>>>> Hi Ignazio, >>>>>>>> >>>>>>>> I am trying to evacuate the compute host on older version (mitaka). >>>>>>>> Could please share the process you followed. I am not able to >>>>>>>> succeed with openstack live-migration fails with error message (this is >>>>>>>> known issue in older versions) and nova live-ligration - nothing happens >>>>>>>> even after initiating VM migration. It is almost 4 days. >>>>>>>> >>>>>>>> ~Jay. >>>>>>>> >>>>>>>> On Thu, Jul 11, 2019 at 11:31 AM Ignazio Cassano < >>>>>>>> ignaziocassano at gmail.com> wrote: >>>>>>>> >>>>>>>>> I am sorry. >>>>>>>>> For simulating an host crash I used a wrong procedure. >>>>>>>>> Using "echo 'c' > /proc/sysrq-trigger" all work fine >>>>>>>>> >>>>>>>>> Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano < >>>>>>>>> ignaziocassano at gmail.com> ha scritto: >>>>>>>>> >>>>>>>>>> Hello All, >>>>>>>>>> on ocata when I poweroff a node with active instance , doing a >>>>>>>>>> nova host-evacuate works fine >>>>>>>>>> and instances are restartd on an active node. >>>>>>>>>> On queens it does non evacuate instances but nova-api reports for >>>>>>>>>> each instance the following: >>>>>>>>>> >>>>>>>>>> 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi >>>>>>>>>> [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 >>>>>>>>>> c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: >>>>>>>>>> Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is >>>>>>>>>> in task_state powering-off >>>>>>>>>> >>>>>>>>>> So it poweroff all instance on the failed node but does not start >>>>>>>>>> them on active nodes >>>>>>>>>> >>>>>>>>>> What is changed ? >>>>>>>>>> Ignazio >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> ​ >>>>>>>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>>>>>>> necessary.* >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> ​ >>>>>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>>>>> necessary.* >>>>>> >>>>> >>>> >>>> -- >>>> ​ >>>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>>> necessary.* >>>> >>> >> >> -- >> ​ >> P *SAVE PAPER – Please do not print this e-mail unless absolutely >> necessary.* >> > -- ​ P *SAVE PAPER – Please do not print this e-mail unless absolutely necessary.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Fri Jul 12 11:18:01 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Fri, 12 Jul 2019 13:18:01 +0200 Subject: [queens][nova] nova host-evacuate errot In-Reply-To: References: Message-ID: Ok. But your virtual machine are using a root volume on cinder or are ephemeral ? Anycase when you try a live migration, look at nova compute log on the kvm node where the instance is migrate from Il giorno ven 12 lug 2019 alle ore 12:48 Jay See ha scritto: > Yes, cinder is running. > > root at h017:~$ service --status-all | grep cinder > [ + ] cinder-volume > > On Fri, Jul 12, 2019 at 11:53 AM Ignazio Cassano > wrote: > >> Sorry ...the question was : how many compute nodes do you have ? >> instead of how many compute nodes do gli have... >> >> >> Anycase; >> Did you configured cinder ? >> >> Il giorno ven 12 lug 2019 alle ore 11:26 Jay See < >> jayachander.it at gmail.com> ha scritto: >> >>> Ignazio, >>> >>> One instance is stuck in error state not able to recover it. All other >>> instances are running now. >>> >>> root at h004:~$ nova reset-state --all-tenants my-instance-1-2 >>> Reset state for server my-instance-1-2 succeeded; new state is error >>> >>> I have several compute nodes (14). I am not sure what is gli? >>> Live migration is not working, i have tried it was not throwing any >>> errors. But nothing seems to happen. >>> I am not completely sure, I haven't heard about gli before. (This setup >>> is deployed by someone else). >>> >>> ~Jay. >>> >>> On Fri, Jul 12, 2019 at 6:12 AM Ignazio Cassano < >>> ignaziocassano at gmail.com> wrote: >>> >>>> Jay, for recovering vm state use the command nova reset-state.... >>>> >>>> nova help reset-state to check the command requested parameters. >>>> >>>> Ad far as evacuation la concerned, how many compute nodes do gli have >>>> ? >>>> Instance live migration works? >>>> Are gli using shared cinder storage? >>>> Ignazio >>>> >>>> Il Gio 11 Lug 2019 20:51 Jay See ha scritto: >>>> >>>>> Thanks for explanation Ignazio. >>>>> >>>>> I have tried same same by trying to put the compute node on a failure >>>>> (echo 'c' > /proc/sysrq-trigger ). Compute node was stuck and I was not >>>>> able connect to it. >>>>> All the VMs are now in Error state. >>>>> >>>>> Running the host-evacaute was successful on controller node, but now I >>>>> am not able to use the VMs. Because they are all in error state now. >>>>> >>>>> root at h004:~$ nova host-evacuate h017 >>>>> >>>>> +--------------------------------------+-------------------+---------------+ >>>>> | Server UUID | Evacuate Accepted | Error >>>>> Message | >>>>> >>>>> +--------------------------------------+-------------------+---------------+ >>>>> | f3545f7d-b85e-49ee-b407-333a4c5b5ab9 | True | >>>>> | >>>>> | 9094494b-cfa3-459b-8d51-d9aae0ea9636 | True | >>>>> | >>>>> | abe7075b-ac22-4168-bf3d-d302ba37d80e | True | >>>>> | >>>>> | c9919371-5f2e-4155-a01a-5f41d9c8b0e7 | True | >>>>> | >>>>> | ffd983bb-851e-4314-9d1d-375303c278f3 | True | >>>>> | >>>>> >>>>> +--------------------------------------+-------------------+---------------+ >>>>> >>>>> Now I have restarted the compute node manually , now I am able to >>>>> connect to the compute node but VMs are still in Error state. >>>>> 1. Any ideas, how to recover the VMs? >>>>> 2. Are there any other methods to evacuate, as this method seems to be >>>>> not working in mitaka version. >>>>> >>>>> ~Jay. >>>>> >>>>> On Thu, Jul 11, 2019 at 1:33 PM Ignazio Cassano < >>>>> ignaziocassano at gmail.com> wrote: >>>>> >>>>>> Ok Jay, >>>>>> let me to describe my environment. >>>>>> I have an openstack made up of 3 controllers nodes ad several compute >>>>>> nodes. >>>>>> The controller nodes services are controlled by pacemaker and the >>>>>> compute nodes services are controlled by remote pacemaker. >>>>>> My hardware is Dell so I am using ipmi fencing device . >>>>>> I wrote a service controlled by pacemaker: >>>>>> this service controls if a compude node fails and for avoiding split >>>>>> brains if a compute node does nod respond on the management network and on >>>>>> storage network the stonith poweroff the node and then execute a nova >>>>>> host-evacuate. >>>>>> >>>>>> Anycase to have a simulation before writing the service I described >>>>>> above you can do as follows: >>>>>> >>>>>> connect on one compute node where some virtual machines are running >>>>>> run the command: echo 'c' > /proc/sysrq-trigger (it stops immediately >>>>>> the node like in case of failure) >>>>>> On a controller node run: nova host-evacuate "name of failed compute >>>>>> node" >>>>>> Instances running on the failed compute node should be restarted on >>>>>> another compute node >>>>>> >>>>>> >>>>>> Ignazio >>>>>> >>>>>> Il giorno gio 11 lug 2019 alle ore 11:57 Jay See < >>>>>> jayachander.it at gmail.com> ha scritto: >>>>>> >>>>>>> Hi , >>>>>>> >>>>>>> I have tried on a failed compute node which is in power off state >>>>>>> now. >>>>>>> I have tried on a running compute node, no errors. But >>>>>>> nothing happens. >>>>>>> On running compute node - Disabled the compute service and tried >>>>>>> migration also. >>>>>>> >>>>>>> May be I might have not followed proper steps. Just wanted to know >>>>>>> the steps you have followed. Otherwise, I was planning to manual migration >>>>>>> also if possible. >>>>>>> ~Jay. >>>>>>> >>>>>>> On Thu, Jul 11, 2019 at 11:52 AM Ignazio Cassano < >>>>>>> ignaziocassano at gmail.com> wrote: >>>>>>> >>>>>>>> Hi Jay, >>>>>>>> would you like to evacuate a failed compute node or evacuate a >>>>>>>> running compute node ? >>>>>>>> >>>>>>>> Ignazio >>>>>>>> >>>>>>>> Il giorno gio 11 lug 2019 alle ore 11:48 Jay See < >>>>>>>> jayachander.it at gmail.com> ha scritto: >>>>>>>> >>>>>>>>> Hi Ignazio, >>>>>>>>> >>>>>>>>> I am trying to evacuate the compute host on older version (mitaka). >>>>>>>>> Could please share the process you followed. I am not able to >>>>>>>>> succeed with openstack live-migration fails with error message (this is >>>>>>>>> known issue in older versions) and nova live-ligration - nothing happens >>>>>>>>> even after initiating VM migration. It is almost 4 days. >>>>>>>>> >>>>>>>>> ~Jay. >>>>>>>>> >>>>>>>>> On Thu, Jul 11, 2019 at 11:31 AM Ignazio Cassano < >>>>>>>>> ignaziocassano at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> I am sorry. >>>>>>>>>> For simulating an host crash I used a wrong procedure. >>>>>>>>>> Using "echo 'c' > /proc/sysrq-trigger" all work fine >>>>>>>>>> >>>>>>>>>> Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano < >>>>>>>>>> ignaziocassano at gmail.com> ha scritto: >>>>>>>>>> >>>>>>>>>>> Hello All, >>>>>>>>>>> on ocata when I poweroff a node with active instance , doing a >>>>>>>>>>> nova host-evacuate works fine >>>>>>>>>>> and instances are restartd on an active node. >>>>>>>>>>> On queens it does non evacuate instances but nova-api reports >>>>>>>>>>> for each instance the following: >>>>>>>>>>> >>>>>>>>>>> 2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi >>>>>>>>>>> [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 >>>>>>>>>>> c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: >>>>>>>>>>> Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is >>>>>>>>>>> in task_state powering-off >>>>>>>>>>> >>>>>>>>>>> So it poweroff all instance on the failed node but does not >>>>>>>>>>> start them on active nodes >>>>>>>>>>> >>>>>>>>>>> What is changed ? >>>>>>>>>>> Ignazio >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> ​ >>>>>>>>> P *SAVE PAPER – Please do not print this e-mail unless >>>>>>>>> absolutely necessary.* >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> ​ >>>>>>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>>>>>> necessary.* >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> ​ >>>>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>>>> necessary.* >>>>> >>>> >>> >>> -- >>> ​ >>> P *SAVE PAPER – Please do not print this e-mail unless absolutely >>> necessary.* >>> >> > > -- > ​ > P *SAVE PAPER – Please do not print this e-mail unless absolutely > necessary.* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssbarnea at redhat.com Fri Jul 12 11:27:04 2019 From: ssbarnea at redhat.com (Sorin Sbarnea) Date: Fri, 12 Jul 2019 12:27:04 +0100 Subject: [doc8] future development and maintenance In-Reply-To: <273715c4f2d4933232fc6b26cb982609daa1a2c7.camel@redhat.com> References: <9D091EF4-60F2-490C-BF76-AAD7A68FB8A1@redhat.com> <273715c4f2d4933232fc6b26cb982609daa1a2c7.camel@redhat.com> Message-ID: As 2/4 emails returned as invalid accounts, my hopes to hear back are getting lower and lower (tried few other channels too). I also believe that doc8 may be better suited outside OpenStack and especially under GitHub due to extra visibility to other developers. So temporary I created a fork at https://github.com/pycontribs/doc8 when I enabled travis and merged few fixes. If we do not hear back I propose to use ^ for future development and maintenance. If Sphinx organization wants to adopt it, even better, it can be transferred without loosing any tickets/PRs/... To be able to make a new release on https://pypi.org/project/doc8/ from Travis, I would need to have the "pycontribs" user added as maintainer, or I could always attempt to publish it as "doc9" but I would prefer to avoid a schism. /sorin > On 11 Jul 2019, at 17:10, Stephen Finucane wrote: > > On Thu, 2019-07-11 at 12:01 +0100, Sorin Sbarnea wrote: >> It seems that the doc8 project is lacking some love, regardless the >> fact that is used by >90 projects from opendev.org. >> >> https://review.opendev.org/#/q/project:x/doc8 >> >> Last merge and release was more than 2 years ago and no reviews were >> performed either. >> >> I think it would be in our interest to assure that doc8 maintenance >> continues and that we can keep it usable. >> >> I would like to propose extenting the list of cores from the current >> 4 ones that I already listed in CC with 3 more, so we can effectively >> make a change that gets merged and later released (anyone willing to >> help?) >> >> If current cores agree, I would be happy to help with maintenance. I >> estimate that the effort needed would likely be less than 1h/month in >> longer term. If there is a desire to move it to github/travis, I >> would not mind either. > > I'd be tentatively interested in helping out here, though it's not as > if I don't already have a lot on my plate :) > > I wonder if this might have better success outside of OpenStack, > perhaps in the sphinx-doc or sphinx-contrib GitHub repo? > > Stephen > >> Thanks >> Sorin Sbarnea >> Red Hat TripleO CI From jose.castro.leon at cern.ch Fri Jul 12 13:03:14 2019 From: jose.castro.leon at cern.ch (Jose Castro Leon) Date: Fri, 12 Jul 2019 13:03:14 +0000 Subject: [Manila] CephFS deferred deletion Message-ID: Dear all, Lately, one of our clients stored 300k files in a manila cephfs share. Then he deleted the share in Manila. This event make the driver unresponsive for several hours until all the data was removed in the cluster. We had a quick look at the code in manila [1] and the deletion is done first by calling the following api calls in the ceph bindings (delete_volume[1] and then purge_volume[2]). The first call moves the directory to a volumes_deleted directory. The second call does a deletion in depth of all the contents of that directory. The last operation is the one that trigger the issue. We had a similar issue in the past in Cinder. There, Arne proposed to do a deferred deletion of volumes. I think we could do the same in Manila for the cephfs driver. The idea is to continue to call to the delete_volume. And then inside a periodic task in the driver, asynchronously it will get the contents of that directory and trigger the purge command. I can propose the change and contribute with the code, but before going to deep I would like to know if there is a reason of having a singleton for the volume_client connection. If I compare with cinder code the connection is established and closed in each operation with the backend. If you are not the maintainer, could you please point me to he/she? I can post it in the mailing list if you prefer Cheers Jose Castro Leon CERN Cloud Infrastructure [1] https://github.com/openstack/manila/blob/master/manila/share/drivers/cephfs/driver.py#L260-L267 [2] https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L700-L734 [2] https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L736-L790 PS: The issue was triggered by one of our clients in kubernetes using the Manila CSI driver From tomas.bredar at gmail.com Fri Jul 12 13:06:55 2019 From: tomas.bredar at gmail.com (=?UTF-8?B?VG9tw6HFoSBCcmVkw6Fy?=) Date: Fri, 12 Jul 2019 15:06:55 +0200 Subject: [tripleo][cinder][netapp] In-Reply-To: References: Message-ID: Hi Emilien! Thanks for your help. Yes with this I am able to define multiple stanzas in cinder.conf. However netapp driver needs a .conf file with the nfs shares listed in it. Defining multiple configuration files with nfs share details in each is not possible with the manual you've sent nor with the templates in my first email. I'm wondering if it's possible to define a second backend by creating another service, for example "OS::TripleO::Services::CinderBackendNetApp2" ? Tomas št 11. 7. 2019 o 14:35 Emilien Macchi napísal(a): > On Thu, Jul 11, 2019 at 7:32 AM Tomáš Bredár > wrote: > >> Hi community, >> >> I'm trying to define multiple NetApp storage backends via Tripleo >> installer. >> According to [1] the puppet manifest supports multiple backends. >> The current templates [2] [3] support only single backend. >> Does anyone know how to define multiple netapp backends in the >> tripleo-heat environment files / templates? >> > > We don't support that via the templates that you linked, however if you > follow this manual you should be able to configure multiple NetApp backends: > > https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/cinder_custom_backend.html > > Let us know how it worked! > -- > Emilien Macchi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tpb at dyncloud.net Fri Jul 12 13:15:40 2019 From: tpb at dyncloud.net (Tom Barron) Date: Fri, 12 Jul 2019 09:15:40 -0400 Subject: [Manila] CephFS deferred deletion In-Reply-To: References: Message-ID: <20190712131540.3eqvltysfix6eivd@barron.net> On 12/07/19 13:03 +0000, Jose Castro Leon wrote: >Dear all, > >Lately, one of our clients stored 300k files in a manila cephfs share. >Then he deleted the share in Manila. This event make the driver >unresponsive for several hours until all the data was removed in the >cluster. > >We had a quick look at the code in manila [1] and the deletion is done >first by calling the following api calls in the ceph bindings >(delete_volume[1] and then purge_volume[2]). The first call moves the >directory to a volumes_deleted directory. The second call does a >deletion in depth of all the contents of that directory. > >The last operation is the one that trigger the issue. > >We had a similar issue in the past in Cinder. There, Arne proposed to >do a deferred deletion of volumes. I think we could do the same in >Manila for the cephfs driver. > >The idea is to continue to call to the delete_volume. And then inside a >periodic task in the driver, asynchronously it will get the contents of >that directory and trigger the purge command. > >I can propose the change and contribute with the code, but before going >to deep I would like to know if there is a reason of having a singleton >for the volume_client connection. If I compare with cinder code the >connection is established and closed in each operation with the >backend. > >If you are not the maintainer, could you please point me to he/she? >I can post it in the mailing list if you prefer > >Cheers >Jose Castro Leon >CERN Cloud Infrastructure > >[1] >https://github.com/openstack/manila/blob/master/manila/share/drivers/cephfs/driver.py#L260-L267 > > >[2] >https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L700-L734 > > >[2] >https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L736-L790 > > >PS: The issue was triggered by one of our clients in kubernetes using >the Manila CSI driver Hi Jose, Let's get this fixed since there's a lot of interest in Manila CSI driver and I think we can expect more batched deletes with it than we have had historically. I've copied Ramana Raja and Patrick Donnelly since they will be able to answer your question about the singleton volume_client connection more authoritatively than I can. Thanks for volunteering to propose a review to deal with this issue! -- Tom Barron From tpb at dyncloud.net Fri Jul 12 13:25:07 2019 From: tpb at dyncloud.net (Tom Barron) Date: Fri, 12 Jul 2019 09:25:07 -0400 Subject: [manila][ptg] Manila PTG planning Message-ID: <20190712132507.de6cuud5jmnx2okd@barron.net> >From our weekly community meetings [1] we know there are five or six people expecting to participate in the Shanghai PTG this coming November. But those meetings are at 1500 UTC -- not the best time for folks in Asia -- so let's check via email as well. Please update the Manila Shanghai PTG Planning etherpad [2] to indicate if you plan (or are trying) to attend the PTG in person or remotely. There's also a section for Topic Proposals, so we can start adding to it now as well. Thanks! -- Tom Barron [1] https://wiki.openstack.org/wiki/Manila/Meetings [2] https://etherpad.openstack.org/p/manila-shanghai-ptg-planning From dpeacock at redhat.com Fri Jul 12 13:45:48 2019 From: dpeacock at redhat.com (David Peacock) Date: Fri, 12 Jul 2019 09:45:48 -0400 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: References: Message-ID: Hi James, On Wed, Jul 10, 2019 at 4:20 PM James Slagle wrote: > There's been a fair amount of recent work around simplifying our Heat > templates and migrating the software configuration part of our > deployment entirely to Ansible. > > As part of this effort, it became apparent that we could render much > of the data that we need out of Heat in a way that is generic per > node, and then have Ansible render the node specific data during > config-download runtime. > I find this endeavour very exciting. Do you have any early indications of performance gains that you can share? Cheers, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From km.giuseppesannino at gmail.com Fri Jul 12 13:57:48 2019 From: km.giuseppesannino at gmail.com (Giuseppe Sannino) Date: Fri, 12 Jul 2019 15:57:48 +0200 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host Message-ID: Hi community, I need your help ,tips, advices. *> Environment <* I have deployed Openstack "Stein" using the latest kolla-ansible on the following deployment topology: 1) OS Controller running as VM on a "cloud" location 2) OS Compute running on a baremetal server remotely (wrt OS Controller) location 3) Network node running on the Compute host As per the above info, Controller and compute run on two different networks. Kolla-Ansible is not really designed for such scenario but after manipulating the globals.yml and the inventory files (basically I had to move node specific network settings from the globals to the inventory file), eventually the deployment works fine. *> Problem <* I have no specific issue working with this deployment except the following: "SSH connection to the VM is quite slow". It takes around 20 seconds for me to log into the VM (Ubuntu, CentOS, whatever). *> Observations <* - Except for the slowness during the SSH login, I don't have any further specific issue working with this envirorment - With the Network on the Compute I can turn the OS controller off with no impact on the VM. Still the connection is slow - I tried different type of images (Ubuntu, CentOS, Windows) always with the same result. - SSH connection is slow even if I try to login into the VM within the IP Namespace >From the ssh -vvv, I can see that the authentication gets stuck here: debug1: Authentication succeeded (publickey). Authenticated to ***** debug1: channel 0: new [client-session] debug3: ssh_session2_open: channel_new: 0 debug2: channel 0: send open debug3: send packet: type 90 debug1: Requesting no-more-sessions at openssh.com debug3: send packet: type 80 debug1: Entering interactive session. debug1: pledge: network >>>>> 10 to 15 seconds later debug3: receive packet: type 80 debug1: client_input_global_request: rtype hostkeys-00 at openssh.com want_reply 0 debug3: receive packet: type 91 debug2: callback start debug2: fd 3 setting TCP_NODELAY debug3: ssh_packet_set_tos: set IP_TOS 0x10 debug2: client_session2_setup: id 0 debug2: channel 0: request pty-req confirm 1 Have you ever experienced such issue ? Any suggestion? Many thanks /Giuseppe -------------- next part -------------- An HTML attachment was scrubbed... URL: From abishop at redhat.com Fri Jul 12 14:11:03 2019 From: abishop at redhat.com (Alan Bishop) Date: Fri, 12 Jul 2019 07:11:03 -0700 Subject: [tripleo][cinder][netapp] In-Reply-To: References: Message-ID: On Fri, Jul 12, 2019 at 6:09 AM Tomáš Bredár wrote: > Hi Emilien! > > Thanks for your help. Yes with this I am able to define multiple stanzas > in cinder.conf. However netapp driver needs a .conf file with the nfs > shares listed in it. Defining multiple configuration files with nfs share > details in each is not possible with the manual you've sent nor with the > templates in my first email. > Hi Tomas, When deploying a single backend, the tripleo template takes care of generating the nfs shares file (actually, puppet-cinder generates the file, but it's triggered by tripleo). But when you use the custom backend method that Emilien pointed you to use, then you are responsible for supplying all the pieces for the backend(s) to function correctly. This means you will need to generate the nfs shares file on the host (controller), and then bind mount the file using CinderVolumeOptVolumes so that the shares file on the host is visible to the cinder-volume process running in a container. I'm wondering if it's possible to define a second backend by creating > another service, for example "OS::TripleO::Services::CinderBackendNetApp2" ? > Sorry, this won't work. TripleO will trying to deploy two completely separate instances of the cinder-volume service, but the two deployments will step all over each other. There has been a long standing goal of enhancing tripleo so that it can deploy multiple instances of a cinder backend, but it's a complex task that will require non-trivial changes to tripleo. Alan Tomas > > št 11. 7. 2019 o 14:35 Emilien Macchi napísal(a): > >> On Thu, Jul 11, 2019 at 7:32 AM Tomáš Bredár >> wrote: >> >>> Hi community, >>> >>> I'm trying to define multiple NetApp storage backends via Tripleo >>> installer. >>> According to [1] the puppet manifest supports multiple backends. >>> The current templates [2] [3] support only single backend. >>> Does anyone know how to define multiple netapp backends in the >>> tripleo-heat environment files / templates? >>> >> >> We don't support that via the templates that you linked, however if you >> follow this manual you should be able to configure multiple NetApp backends: >> >> https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/cinder_custom_backend.html >> >> Let us know how it worked! >> -- >> Emilien Macchi >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From haleyb.dev at gmail.com Fri Jul 12 14:23:18 2019 From: haleyb.dev at gmail.com (Brian Haley) Date: Fri, 12 Jul 2019 10:23:18 -0400 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: References: Message-ID: On 7/12/19 9:57 AM, Giuseppe Sannino wrote: > Hi community, > I need your help ,tips, advices. > > > *> Environment <* > I have deployed Openstack "Stein" using the latest kolla-ansible on the > following deployment topology: > > 1) OS Controller running as VM on a "cloud" location > 2) OS Compute running on a baremetal server remotely (wrt OS Controller) > location > 3) Network node running on the Compute host > > As per the above info, Controller and compute run on two different networks. > > Kolla-Ansible is not really designed for such scenario but after > manipulating the globals.yml and the inventory files (basically I had to > move node specific network settings from the globals to the inventory > file), eventually the deployment works fine. > > > *> Problem <* > I have no specific issue working with this deployment except the following: > > "SSH connection to the VM is quite slow". > > It takes around 20 seconds for me to log into the VM (Ubuntu, CentOS, > whatever). But once logged-in things are OK? For example, an scp stalls the same way, but the transfer is fast? > *> Observations <* > > * Except for the slowness during the SSH login, I don't have any > further specific issue working with this envirorment > * With the Network on the Compute I can turn the OS controller off > with no impact on the VM. Still the connection is slow > * I tried different type of images (Ubuntu, CentOS, Windows) always > with the same result. > * SSH connection is slow even if I try to login into the VM within the > IP Namespace > > From the ssh -vvv, I can see that the authentication gets stuck here: > > debug1: Authentication succeeded (publickey). > Authenticated to ***** > debug1: channel 0: new [client-session] > debug3: ssh_session2_open: channel_new: 0 > debug2: channel 0: send open > debug3: send packet: type 90 > debug1: Requesting no-more-sessions at openssh.com > > debug3: send packet: type 80 > debug1: Entering interactive session. > debug1: pledge: network > > >>>>> 10 to 15 seconds later What is sshd doing at this time? Have you tried enabling debug or running tcpdump when a new connection is attempted? At first glance I'd say it's a DNS issue since it eventually succeeds, the logs would help to point in a direction. -Brian > debug3: receive packet: type 80 > debug1: client_input_global_request: rtype hostkeys-00 at openssh.com > want_reply 0 > debug3: receive packet: type 91 > debug2: callback start > debug2: fd 3 setting TCP_NODELAY > debug3: ssh_packet_set_tos: set IP_TOS 0x10 > debug2: client_session2_setup: id 0 > debug2: channel 0: request pty-req confirm 1 > > > Have you ever experienced such issue ? > Any suggestion? > > Many thanks > > /Giuseppe > > > From mark at stackhpc.com Fri Jul 12 15:35:50 2019 From: mark at stackhpc.com (Mark Goddard) Date: Fri, 12 Jul 2019 16:35:50 +0100 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: References: Message-ID: On Fri, 12 Jul 2019 at 15:24, Brian Haley wrote: > > > On 7/12/19 9:57 AM, Giuseppe Sannino wrote: > > Hi community, > > I need your help ,tips, advices. > > > > > > *> Environment <* > > I have deployed Openstack "Stein" using the latest kolla-ansible on the > > following deployment topology: > > > > 1) OS Controller running as VM on a "cloud" location > > 2) OS Compute running on a baremetal server remotely (wrt OS Controller) > > location > > 3) Network node running on the Compute host > > > > As per the above info, Controller and compute run on two different > networks. > > > > Kolla-Ansible is not really designed for such scenario but after > > manipulating the globals.yml and the inventory files (basically I had to > > move node specific network settings from the globals to the inventory > > file), eventually the deployment works fine. > > > > > > *> Problem <* > > I have no specific issue working with this deployment except the > following: > > > > "SSH connection to the VM is quite slow". > > > > It takes around 20 seconds for me to log into the VM (Ubuntu, CentOS, > > whatever). > > But once logged-in things are OK? For example, an scp stalls the same > way, but the transfer is fast? > > > *> Observations <* > > > > * Except for the slowness during the SSH login, I don't have any > > further specific issue working with this envirorment > > * With the Network on the Compute I can turn the OS controller off > > with no impact on the VM. Still the connection is slow > > * I tried different type of images (Ubuntu, CentOS, Windows) always > > with the same result. > > * SSH connection is slow even if I try to login into the VM within the > > IP Namespace > > > > From the ssh -vvv, I can see that the authentication gets stuck here: > > > > debug1: Authentication succeeded (publickey). > > Authenticated to ***** > > debug1: channel 0: new [client-session] > > debug3: ssh_session2_open: channel_new: 0 > > debug2: channel 0: send open > > debug3: send packet: type 90 > > debug1: Requesting no-more-sessions at openssh.com > > > > debug3: send packet: type 80 > > debug1: Entering interactive session. > > debug1: pledge: network > > > > >>>>> 10 to 15 seconds later > > What is sshd doing at this time? Have you tried enabling debug or > running tcpdump when a new connection is attempted? At first glance I'd > say it's a DNS issue since it eventually succeeds, the logs would help > to point in a direction. > +1 - ~30s timeout on SSH login is normally a DNS issue. > -Brian > > > > debug3: receive packet: type 80 > > debug1: client_input_global_request: rtype hostkeys-00 at openssh.com > > want_reply 0 > > debug3: receive packet: type 91 > > debug2: callback start > > debug2: fd 3 setting TCP_NODELAY > > debug3: ssh_packet_set_tos: set IP_TOS 0x10 > > debug2: client_session2_setup: id 0 > > debug2: channel 0: request pty-req confirm 1 > > > > > > Have you ever experienced such issue ? > > Any suggestion? > > > > Many thanks > > > > /Giuseppe > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Fri Jul 12 15:40:58 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Fri, 12 Jul 2019 17:40:58 +0200 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: References: Message-ID: <01AA2BA1-3EEC-4142-BCDB-F9861707102E@redhat.com> Hi, I suspect some problems with names resolving. Can You check if You have also such delay when doing e.g. “sudo” commands after You ssh to the instance? > On 12 Jul 2019, at 16:23, Brian Haley wrote: > > > > On 7/12/19 9:57 AM, Giuseppe Sannino wrote: >> Hi community, >> I need your help ,tips, advices. >> *> Environment <* >> I have deployed Openstack "Stein" using the latest kolla-ansible on the following deployment topology: >> 1) OS Controller running as VM on a "cloud" location >> 2) OS Compute running on a baremetal server remotely (wrt OS Controller) location >> 3) Network node running on the Compute host >> As per the above info, Controller and compute run on two different networks. >> Kolla-Ansible is not really designed for such scenario but after manipulating the globals.yml and the inventory files (basically I had to move node specific network settings from the globals to the inventory file), eventually the deployment works fine. >> *> Problem <* >> I have no specific issue working with this deployment except the following: >> "SSH connection to the VM is quite slow". >> It takes around 20 seconds for me to log into the VM (Ubuntu, CentOS, whatever). > > But once logged-in things are OK? For example, an scp stalls the same way, but the transfer is fast? > >> *> Observations <* >> * Except for the slowness during the SSH login, I don't have any >> further specific issue working with this envirorment >> * With the Network on the Compute I can turn the OS controller off >> with no impact on the VM. Still the connection is slow >> * I tried different type of images (Ubuntu, CentOS, Windows) always >> with the same result. >> * SSH connection is slow even if I try to login into the VM within the >> IP Namespace >> From the ssh -vvv, I can see that the authentication gets stuck here: >> debug1: Authentication succeeded (publickey). >> Authenticated to ***** >> debug1: channel 0: new [client-session] >> debug3: ssh_session2_open: channel_new: 0 >> debug2: channel 0: send open >> debug3: send packet: type 90 >> debug1: Requesting no-more-sessions at openssh.com >> debug3: send packet: type 80 >> debug1: Entering interactive session. >> debug1: pledge: network >> >>>>> 10 to 15 seconds later > > What is sshd doing at this time? Have you tried enabling debug or running tcpdump when a new connection is attempted? At first glance I'd say it's a DNS issue since it eventually succeeds, the logs would help to point in a direction. > > -Brian > > >> debug3: receive packet: type 80 >> debug1: client_input_global_request: rtype hostkeys-00 at openssh.com want_reply 0 >> debug3: receive packet: type 91 >> debug2: callback start >> debug2: fd 3 setting TCP_NODELAY >> debug3: ssh_packet_set_tos: set IP_TOS 0x10 >> debug2: client_session2_setup: id 0 >> debug2: channel 0: request pty-req confirm 1 >> Have you ever experienced such issue ? >> Any suggestion? >> Many thanks >> /Giuseppe — Slawek Kaplonski Senior software engineer Red Hat From thiagocmartinsc at gmail.com Fri Jul 12 15:42:46 2019 From: thiagocmartinsc at gmail.com (=?UTF-8?B?TWFydGlueCAtIOOCuOOCp+ODvOODoOOCug==?=) Date: Fri, 12 Jul 2019 11:42:46 -0400 Subject: [nova-lxd] retiring nova-lxd In-Reply-To: References: Message-ID: Oh, that's very sad news... I was about to deploy a relatively big OpenStack Cloud (for ~500 LXD Instances) running on nova-lxd but now, I'll cancel it. No more bare-metal cloud?! :-( On Wed, 10 Jul 2019 at 09:08, James Page wrote: > Hi All > > I’m slightly sad to announce that we’re retiring the LXD driver for Nova > aka “nova-lxd”. > > Developing a driver for Nova for container based machines has been a fun > and technically challenging ride over the last four years but we’ve never > really seen any level of serious production deployment; as a result we’ve > decided that it’s time to call it a day for nova-lxd. > > I’d like to thank all of the key contributors for their efforts over the > years - specifically Chuck Short, Paul Hummer, Chris MacNaughton, Sahid > Orentino and Alex Kavanaugh who have led or contributed to the development > of the driver over its lifetime. > > I’ll be raising a review to leave a note for future followers as to the > fate of nova-lxd. If anyone else would like to continue development of the > driver they are more than welcome to revert my commit and become a part of > the development team! > > We’ll continue to support our current set of stable branches for another > ~12 months. > > Note that development of LXD and the pylxd Python module continues; its > just the integration of OpenStack with LXD that we’re ceasing development > of. > > Regards > > James > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangtrinhnt at gmail.com Fri Jul 12 16:11:32 2019 From: dangtrinhnt at gmail.com (Trinh Nguyen) Date: Sat, 13 Jul 2019 01:11:32 +0900 Subject: OpenInfra Days Vietnam 2019 (Hanoi) - Call For Presentations In-Reply-To: References: Message-ID: Hello teams, Less than 2 months and the OpenInfra Days Vietnam will happen. We love to hear the voices of Kata, StarlingX, Airship, Zuul, and other OpenStack projects team. Please join us in one day event in Hanoi this August 24th. - *Event's website:* http://day.vietopeninfra.org/ - *Call for Presentation (extended deadline 30th July):* https://forms.gle/iiRBxxyRv1mGFbgi7 - *Buy tickets:* https://ticketbox.vn/event/vietnam-openinfra-days-2019-75375 Tell me if you have any questions. Yours, Trinh On Tue, May 21, 2019 at 2:36 PM Trinh Nguyen wrote: > Hello, > > Hope you're doing well :) > > The OpenInfra Days Vietnam 2019 [1] is looking for speakers in many > different topics (e.g., container, CI, deployment, edge computing, etc.). > If you would love to have a taste of Hanoi, the capital of Vietnam, please > join us this one-day event and submit your presentation [2]. > > *- Date:* 24 AUGUST 2019 > *- Location:* INTERCONTINENTAL HANOI LANDMARK72, HANOI, VIETNAM > > Especially this time, we're honored to have the Upstream Institute > Training [3] hosted by the OpenStack Foundation on the next day (25 August > 2019). > > [1] http://day.vietopeninfra.org/ > [2] https://forms.gle/iiRBxxyRv1mGFbgi7 > [3] > https://docs.openstack.org/upstream-training/upstream-training-content.html > > See you in Hanoi! > > Bests, > > On behalf of the VietOpenInfra Group. > > -- > *Trinh Nguyen* > *www.edlab.xyz * > > -- *Trinh Nguyen* *www.edlab.xyz * -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssbarnea at redhat.com Fri Jul 12 16:37:03 2019 From: ssbarnea at redhat.com (Sorin Sbarnea) Date: Fri, 12 Jul 2019 17:37:03 +0100 Subject: [tripleo] tripleo-ci team investigating lowering the time check jobs are waiting Message-ID: <26F98673-AC01-4C61-BCA5-8AA28937D5E7@redhat.com> Tripleo CI team is trying to find ways to lower the resource loads on upstream zuul by identifying jobs running in check or gate which should have being skipped. This is an exercise in tuning which jobs launch against the various TripleO gerrit repos and not a complete overhaul of the coverage. There have been some obvious mistakes made in the zuul configurations in the past, we are looking to resolve all these issues, but also wanted to reach out to the broader community for more input. Since yesterday we started to track the queue lengths for both tripleo-check and tripleo-gate [1], so soon we will be able to see some trends. We have also been monitoring the number of jobs launched in both check and gate and their pass rates [2][3] Even if you only have some doubts about some jobs, it would be useful to report them to us so we can investigate further. Report on: https://etherpad.openstack.org/p/tripleo-jobs-tunning [1] http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&fullscreen&panelId=398 [2] check http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&fullscreen&panelId=157 [3] gate http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&fullscreen&panelId=307 Thanks, Sorin Sbarnea TripleO CI From james.slagle at gmail.com Fri Jul 12 16:43:13 2019 From: james.slagle at gmail.com (James Slagle) Date: Fri, 12 Jul 2019 12:43:13 -0400 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: References: Message-ID: On Fri, Jul 12, 2019 at 9:46 AM David Peacock wrote: > > Hi James, > > On Wed, Jul 10, 2019 at 4:20 PM James Slagle wrote: >> >> There's been a fair amount of recent work around simplifying our Heat >> templates and migrating the software configuration part of our >> deployment entirely to Ansible. >> >> As part of this effort, it became apparent that we could render much >> of the data that we need out of Heat in a way that is generic per >> node, and then have Ansible render the node specific data during >> config-download runtime. > > > I find this endeavour very exciting. Do you have any early indications of performance gains that you can share? No hard numbers yet, but I can say that I can get to the Ansible stage of the deployment with any number of nodes with an undercloud that just meets the minimum requirements. This is significant because previously we could not get to this stage without first deploying a huge Heat stack which required a lot of physical resources, tuning, tweaking, or going the undercloud minion route. Also, it's less about performance and more about scale. Certainly the Heat stack operation will be much faster as the number of nodes in the deployment increases. The stack operation time will in fact be constant in relation to the number of nodes in the deployment. It will depend on the number of *roles*, but typically those are ~< 5 per deployment, and the most I've seen is 12. The total work done by Ansible does increase as we move more logic into roles and tasks. However, I expect the total Ansible run time to be roughly equivalent to what we have today since the sum of all that Ansible applies is roughly equal. In terms of scale however, it allows us to move beyond the ~300 node limit we're at today. And it keeps the Heat performance constant as opposed to increasing with the node count. -- -- James Slagle -- From donny at fortnebula.com Fri Jul 12 19:37:05 2019 From: donny at fortnebula.com (Donny Davis) Date: Fri, 12 Jul 2019 15:37:05 -0400 Subject: [Nova] Instances can't be started after compute nodes unexpectedly shut down because of power outage In-Reply-To: References: Message-ID: How is the recovery coming along Gökhan? I am curious to hear. On Fri, Jul 12, 2019 at 3:46 AM Gökhan IŞIK wrote: > Awesome, thanks! Donny, > I followed below steps and rescue my instance. > > 1. > > Find instance id and compute host > > root at infra1-utility-container-50bcf920:~# openstack server show 1d2e8a39-97ee-4ce7-a612-1b50f90cc51e -c id -c OS-EXT-SRV-ATTR:hypervisor_hostname > +-------------------------------------+--------------------------------------+ > | Field | Value | > +-------------------------------------+--------------------------------------+ > | OS-EXT-SRV-ATTR:hypervisor_hostname | compute06 | > | id | 1d2e8a39-97ee-4ce7-a612-1b50f90cc51e | > +-------------------------------------+--------------------------------------+ > > > 2. > > Find image and backing image file on compute host > > root at compute06:~# qemu-img info -U --backing-chain /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk > image: /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk > file format: qcow2 > virtual size: 160G (171798691840 bytes) > disk size: 32G > cluster_size: 65536 > backing file: /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323 > Format specific information: > compat: 1.1 > lazy refcounts: false > refcount bits: 16 > corrupt: false > image: /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323 > file format: raw > virtual size: 160G (171798691840 bytes) > disk size: 18G > > > > 3. Copy image and backing image file > > > root at compute06:~# cp /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk master > root at compute06:~# cp /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323 new-master > > > 4. > > Rebase the image file that was backed off the original file so that > it uses the new file i.e new-master then commit those changes back to > original file master back into the new base new-master > > root at compute06:~# qemu-img rebase -b new-master -U master > > root at compute06:~# qemu-img commit master > > root at compute06:~# qemu-img info new-master > > > > > 5. > > Convert raw image to qcow2 > > root at compute06:~# qemu-img convert -f raw -O qcow2 new-master new-master.qcow2 > > > 6. Time to upload glance and then launch instance from this image :) > > > Thanks, > Gökhan. > > Donny Davis , 12 Tem 2019 Cum, 00:56 tarihinde şunu > yazdı: > >> Of course you can also always just pull the disk images from the vm >> folders, merge them back with the base file, upload to glance and then >> relaunch the instances. >> >> You can give this method a spin with the lowest risk to your instances >> >> >> https://medium.com/@kumar_pravin/qemu-merge-snapshot-and-backing-file-into-standalone-disk-c8d3a2b17c0e >> >> >> >> >> >> On Thu, Jul 11, 2019 at 4:10 PM Donny Davis wrote: >> >>> You surely want to leave locking turned on. >>> >>> You may want to ask qemu-devel about the locking of a image file and how >>> it works. This isn't really an Openstack issue, seems to be a layer below. >>> >>> Depending on how mission critical your VM's are, you could probably work >>> around it by just passing in --force-share into the command openstack is >>> trying to run. >>> >>> I cannot recommend this path, the best way is to find out how you remove >>> the lock. >>> >>> >>> >>> >>> >>> >>> On Thu, Jul 11, 2019 at 3:23 PM Gökhan IŞIK >>> wrote: >>> >>>> In [1] it says "Image locking is added and enabled by default. >>>> Multiple QEMU processes cannot write to the same image as long as the host >>>> supports OFD or posix locking, unless options are specified otherwise." May >>>> be need to do something on nova side. >>>> >>>> I run this command and get same error. Output is in >>>> http://paste.openstack.org/show/754311/ >>>> >>>> İf I run qemu-img info instance-0000219b with -U , it doesn't give any >>>> errors. >>>> >>>> [1] https://wiki.qemu.org/ChangeLog/2.10 >>>> >>>> Donny Davis , 11 Tem 2019 Per, 22:11 tarihinde >>>> şunu yazdı: >>>> >>>>> Well that is interesting. If you look in your libvirt config directory >>>>> (/etc/libvirt on Centos) you can get a little more info on what is being >>>>> used for locking. >>>>> >>>>> Maybe strace can shed some light on it. Try something like >>>>> >>>>> strace -ttt -f qemu-img info >>>>> /var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Jul 11, 2019 at 2:39 PM Gökhan IŞIK >>>>> wrote: >>>>> >>>>>> I run virsh list --all command and output is below: >>>>>> >>>>>> root at compute06:~# virsh list --all >>>>>> Id Name State >>>>>> ---------------------------------------------------- >>>>>> - instance-000012f9 shut off >>>>>> - instance-000013b6 shut off >>>>>> - instance-000016fb shut off >>>>>> - instance-0000190a shut off >>>>>> - instance-00001a8a shut off >>>>>> - instance-00001e05 shut off >>>>>> - instance-0000202a shut off >>>>>> - instance-00002135 shut off >>>>>> - instance-00002141 shut off >>>>>> - instance-000021b6 shut off >>>>>> - instance-000021ec shut off >>>>>> - instance-000023db shut off >>>>>> - instance-00002ad7 shut off >>>>>> >>>>>> And also when I try start instances with virsh , output is below: >>>>>> >>>>>> root at compute06:~# virsh start instance-0000219b >>>>>> error: Failed to start domain instance-000012f9 >>>>>> error: internal error: process exited while connecting to monitor: >>>>>> 2019-07-11T18:36:34.229534Z qemu-system-x86_64: -chardev >>>>>> pty,id=charserial0,logfile=/dev/fdset/2,logappend=on: char device >>>>>> redirected to /dev/pts/3 (label charserial0) >>>>>> 2019-07-11T18:36:34.243395Z qemu-system-x86_64: -drive >>>>>> file=/var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=ignore: >>>>>> Failed to get "write" lock >>>>>> Is another process using the image? >>>>>> >>>>>> Thanks, >>>>>> Gökhan >>>>>> >>>>>> Donny Davis , 11 Tem 2019 Per, 21:06 tarihinde >>>>>> şunu yazdı: >>>>>> >>>>>>> Can you ssh to the hypervisor and run virsh list to make sure your >>>>>>> instances are in fact down? >>>>>>> >>>>>>> On Thu, Jul 11, 2019 at 3:02 AM Gökhan IŞIK >>>>>>> wrote: >>>>>>> >>>>>>>> Can anyone help me please ? I can no't rescue my instances yet :( >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Gökhan >>>>>>>> >>>>>>>> Gökhan IŞIK , 9 Tem 2019 Sal, 15:46 >>>>>>>> tarihinde şunu yazdı: >>>>>>>> >>>>>>>>> Hi folks, >>>>>>>>> Because of power outage, Most of our compute nodes unexpectedly >>>>>>>>> shut down and now I can not start our instances. Error message is "Failed >>>>>>>>> to get "write" lock another process using the image?". Instances Power >>>>>>>>> status is No State. Full error log is >>>>>>>>> http://paste.openstack.org/show/754107/. My environment is >>>>>>>>> OpenStack Pike on Ubuntu 16.04 LTS servers and Instances are on a nfs >>>>>>>>> shared storage. Nova version is 16.1.6.dev2. qemu version is 2.10.1. >>>>>>>>> libvirt version is 3.6.0. I saw a commit [1], but it doesn't solve this >>>>>>>>> problem. >>>>>>>>> There are important instances on my environment. How can I rescue >>>>>>>>> my instances? What would you suggest ? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Gökhan >>>>>>>>> >>>>>>>>> [1] https://review.opendev.org/#/c/509774/ >>>>>>>>> >>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hjensas at redhat.com Fri Jul 12 19:59:34 2019 From: hjensas at redhat.com (Harald =?ISO-8859-1?Q?Jens=E5s?=) Date: Fri, 12 Jul 2019 21:59:34 +0200 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: References: Message-ID: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> On Wed, 2019-07-10 at 16:17 -0400, James Slagle wrote: > There's been a fair amount of recent work around simplifying our Heat > templates and migrating the software configuration part of our > deployment entirely to Ansible. > > As part of this effort, it became apparent that we could render much > of the data that we need out of Heat in a way that is generic per > node, and then have Ansible render the node specific data during > config-download runtime. > > To illustrate the point, consider when we specify ComputeCount:10 in > our templates, that much of the work that Heat is doing across those > 10 sets of resources for each Compute node is duplication. However, > it's been necessary so that Heat can render data structures such as > list of IP's, lists of hostnames, contents of /etc/hosts files, etc > etc etc. If all that was driven by Ansible using host facts, then > Heat > doesn't need to do those 10 sets of resources to begin with. > > The goal is to get to a point where we can deploy the Heat stack with > a count of 1 for each role, and then deploy any number of nodes per > role using Ansible. To that end, I've been referring to this effort > as > N=1. > > The value in this work is that it directly addresses our scaling > issues with Heat (by just deploying a much smaller stack). Obviously > we'd still be relying heavily on Ansible to scale to the required > levels, but I feel that is much better understood challenge at this > point in the evolution of configuration tools. > > With the patches that we've been working on recently, I've got a POC > running where I can deploy additional compute nodes with just > Ansible. > This is done by just adding the additional nodes to the Ansible > inventory with a small set of facts to include IP addresses on each > enabled network and a hostname. > > These patches are at > https://review.opendev.org/#/q/topic:bp/reduce-deployment-resources > and reviews/feedback are welcome. > > Other points: > > - Baremetal provisioning and port creation are presently handled by > Heat. With the ongoing efforts to migrate baremetal provisioning out > of Heat (nova-less deploy), I think these efforts are very > complimentary. Eventually, we get to a point where Heat is not > actually creating any other OpenStack API resources. For now, the > patches only work when using pre-provisioned nodes. > I've said this before, but I think we should turn this nova-less around. Now with nova-less we create a bunch of servers, and write up the parameters file to use the deployed-server approach. Effectively we still neet to have the resource group in heat making a server resource for every server. Creating the fake server resource is fast, because Heat does'nt call Nova,Ironic to create any resources. But the stack is equally big, with a stack for every node. i.e not N=1. What you are doing here, is essentially to say we don't create a resource group that then creates N number of role stacks, one for each overcloud node. You are creating a single generic "server" definition per Role. So we drop the resource group and create OS::Triple::{{Role}}.Server 1-time (once). To me it's backwards to push a large struct with properties for N=many nodes into the creation of that stack. Currently the puppet/role-role.yaml creates all the network ports etc. As you only want to create it once, it instead could simply output the UUID of the networks+subnets. These are identical for all servers in the role. So we end up with a small heat stack. Once the stack is created we could use that generic "server" role data to feed into something (ansible?, python?, mistral?) that calls metalsmith to build the servers, then create ports for each server in neutron, one port for each network+subnet defined in the role. Then feed that output into the json (hieradata) that is pushed to each node and used during service configuration, all the things we need to configure network interfaces, /etc/hosts and so on. We need a way to keep track of which ports belong to wich node, but I guess something simple like using the node's ironic UUID in either the name, description or tag field of the neutron port will work. There is also the extra filed in Ironic which is json type, so we could place a map of network->port_uuid in there as well. Another idea I've been pondering is if we put credentials on the overcloud nodes so that the node itself could make the call to neutron on the undercloud to create ports in neutron. I.e we just push the UUID of the correct network and subnet where the resource should be created, and let the overcloud node do the create. The problem with this is that we wouldn't have a way to build the /etc/hosts and probably other things that include ips etc for all the nodes. Maby if all the nodes was part of an etcd cluster, and pushed it's data there? I think the creation of the actual Networks and Subnets can be left in heat, it's typically 5-6 networks and 5-6 subnets so it's not a lot of resources. Even in a large DCN deployment having 50-100 subnets per network or even 50-100 networks I think this is'nt a problem. > - We need to consider how we'd manage the Ansible inventory going > forward if we open up an interface for operators to manipulate it > directly. That's something we'd want to manage and preserve (version > control) as it's critical data for the deployment. > > Given the progress that we've made with the POC, my sense is that > we'll keep pushing in this overall direction. I'd like to get some > feedback on the approach. We have an etherpad we are using to track > some of the work at a high level: > > https://etherpad.openstack.org/p/tripleo-reduce-deployment-resources > > I'll be adding some notes on how I setup the POC to that etherpad if > others would like to try it out. > From corey.bryant at canonical.com Fri Jul 12 20:57:29 2019 From: corey.bryant at canonical.com (Corey Bryant) Date: Fri, 12 Jul 2019 16:57:29 -0400 Subject: [goal][python3] Train unit tests weekly update (goal-9) Message-ID: This is the goal-9 weekly update for the "Update Python 3 test runtimes for Train" goal [1]. There are 9 weeks remaining for completion of Train community goals [2]. == What's the Goal? == To ensure (in the Train cycle) that all official OpenStack repositories with Python 3 unit tests are exclusively using the 'openstack-python3-train-jobs' Zuul template or one of its variants (e.g. 'openstack-python3-train-jobs-neutron') to run unit tests, and that tests are passing. This will ensure that all official projects are running py36 and py37 unit tests in Train. For complete details please see [1]. == Ongoing Work == Patches have been submitted for all applicable projects except for one, OpenStack Charms. I hope to have those submitted next week. Open patches needing reviews: https://review.openstack.org/#/q/topic:python3-train+is:open Failing patches: https://review.openstack.org/#/q/topic:python3-train+status:open+(+label:Verified-1+OR+label:Verified-2+) Patch automation scripts needing review: https://review.opendev.org/#/c/666934 == Completed Work == Merged patches: https://review.openstack.org/#/q/topic:python3-train+is:merged == How can you help? == Please take a look at the failing patches and help fix any failing unit tests for your projects. Python 3.7 unit tests will be self-testing in Zuul. == Reference Material == [1] Goal description: https://governance.openstack.org/tc/goals /train/python3-updates.html [2] Train release schedule: https://releases.openstack.org/train/schedule.html (see R-5 for "Train Community Goals Completed") Storyboard: https://storyboard.openstack.org/#!/story/2005924 Porting to Python 3.7: https://docs.python.org/3/whatsnew/3.7.html#porting-to-python-3-7 Python Update Process: https://opendev.org/openstack/governance/src/branch/master/resolutions/20181024-python-update-process.rst Train runtimes: https://opendev.org/openstack/governance/src/branch/master/reference/runtimes/train.rst Thanks, Corey -------------- next part -------------- An HTML attachment was scrubbed... URL: From colleen at gazlene.net Sat Jul 13 00:24:32 2019 From: colleen at gazlene.net (Colleen Murphy) Date: Fri, 12 Jul 2019 17:24:32 -0700 Subject: [keystone] Keystone Team Update - Week of 8 July 2019 Message-ID: <4f0e2f82-1580-4074-8814-d336f918a339@www.fastmail.com> # Keystone Team Update - Week of 8 July 2019 ## News ### Midcycle Planning We've set dates for the week of Milestone 2 for our virtual midcycle[1]. Topic suggestion is still open[2] and I will try to propose a draft agenda early next week. You can help by adding votes to topics you think are especially wortwhile in the etherpad. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007593.html [2] https://etherpad.openstack.org/p/keystone-train-midcycle-topics ### PTG Planning It's also already time to start thinking about the PTG[3]. If you have an inkling of whether you will be able to attend the PTG in Shanghai in November and would participate in the keystone room, please add your name to the planning etherpad[4]. [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007639.html [4] https://etherpad.openstack.org/p/keystone-shanghai-ptg ### Handling Defunct APIs While cleaning up deprecated config options related to PKI tokens[5], we found that these config options affect the OS-SIMPLE-CERT API and discussed[6][7] how best to handle the changing API without needing a version change. If you have thoughts on this matter, let us know in the review. [5] https://review.opendev.org/659434 [6] http://eavesdrop.openstack.org/meetings/keystone/2019/keystone.2019-07-09-16.00.log.html#l-99 [7] http://eavesdrop.openstack.org/irclogs/%23openstack-sdks/%23openstack-sdks.2019-07-11.log.html#t2019-07-11T16:18:01 ### External Auth Prior to federated authentication, we supported external authentication using the REMOTE_USER HTTPD variable. Now that we have federated authentication available to us, we discussed whether it's time to discourage the use of regular external authentication[8]. If you are using external authentication or have an opinion about it, please join in the discussion with us[9]. [8] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2019-07-09.log.html#t2019-07-09T23:42:19 [9] https://review.opendev.org/669959 ## Open Specs Train specs: https://bit.ly/2uZ2tRl Ongoing specs: https://bit.ly/2OyDLTh Since we do not have anyone who can commit to the implementation of the "Expose root domain as assignment target" spec[10] for Train, the latest patchset proposes it to the backlog rather than to the Train cycle. [10] https://review.opendev.org/661837 ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 10 changes this week. ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 46 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ## Bugs This week we opened 1 new bugs and closed 1. Bugs opened (1) Bug #1836390 (oslo.policy:Undecided) opened by Vadym Markov https://bugs.launchpad.net/oslo.policy/+bug/1836390 Bugs fixed (1) Bug #1811771 (keystone:Low) fixed by Vishakha Agarwal https://bugs.launchpad.net/keystone/+bug/1811771 Although we did not open many new bugs this week, the backlog of unconfirmed and untriaged bugs has been slowly growing[11], please help by reproducing and triaging new bugs. [11] https://bugs.launchpad.net/keystone/+bugs?search=Search&field.status=New ## Milestone Outlook https://releases.openstack.org/train/schedule.html Milestone 2 is in two weeks, which means spec freeze is coming up followed closely by feature proposal freeze. ## Shout-outs Big thanks to Guang Yee for taking on an overhaul of the tokenless authentication documentation[12] which has been in severe need of an update and cleanup. [12] https://review.opendev.org/669790 ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter From ianyrchoi at gmail.com Sat Jul 13 01:42:57 2019 From: ianyrchoi at gmail.com (Ian Y. Choi) Date: Sat, 13 Jul 2019 10:42:57 +0900 Subject: OpenInfra Days Vietnam 2019 (Hanoi) - Call For Presentations In-Reply-To: References: Message-ID: (Copying to OpenStack & Community mailing lists) Trinh Nguyen wrote on 7/13/2019 1:11 AM: > Hello teams, > > Less than 2 months and the OpenInfra Days Vietnam will happen. We love > to hear the voices of Kata, StarlingX, Airship, Zuul, and other > OpenStack projects team. Please join us in one day event in Hanoi this > August 24th. > > * *Event's website:* http://day.vietopeninfra.org/ > * *Call for Presentation (extended deadline 30th July):* > https://forms.gle/iiRBxxyRv1mGFbgi7 > * *Buy tickets:* > https://ticketbox.vn/event/vietnam-openinfra-days-2019-75375 > > Tell me if you have any questions. > > Yours, > Trinh > > > On Tue, May 21, 2019 at 2:36 PM Trinh Nguyen > wrote: > > Hello, > > Hope you're doing well :) > > The OpenInfra Days Vietnam 2019 [1] is looking for speakers in > many different topics (e.g., container, CI, deployment, edge > computing, etc.). If you would love to have a taste of Hanoi, the > capital of Vietnam, please join us this one-day event and submit > your presentation [2]. > > *- Date:* 24 AUGUST 2019 > *- Location:* INTERCONTINENTAL HANOI LANDMARK72, HANOI, VIETNAM > > Especially this time, we're honored to have the Upstream Institute > Training [3] hosted by the OpenStack Foundation on the next day > (25 August 2019). > > [1] http://day.vietopeninfra.org/ > [2] https://forms.gle/iiRBxxyRv1mGFbgi7 > [3] > https://docs.openstack.org/upstream-training/upstream-training-content.html > > See you in Hanoi! > > Bests, > > On behalf of the VietOpenInfra Group. > > -- > *Trinh Nguyen* > _www.edlab.xyz _ > > > > -- > *Trinh Nguyen* > _www.edlab.xyz _ > From amotoki at gmail.com Sat Jul 13 08:19:18 2019 From: amotoki at gmail.com (Akihiro Motoki) Date: Sat, 13 Jul 2019 17:19:18 +0900 Subject: [all] [ptls] [tc] [nova] [neutron] [tripleo] Volunteers that know TeX for PDF community goal In-Reply-To: <80e2e8550fd39cf9e224e24b4e6ad806acdc9e16.camel@redhat.com> References: <20190624155629.GA26343@sinanju.localdomain> <80e2e8550fd39cf9e224e24b4e6ad806acdc9e16.camel@redhat.com> Message-ID: On Wed, Jun 26, 2019 at 6:24 PM Stephen Finucane wrote: > > On Mon, 2019-06-24 at 16:36 -0700, Michael Johnson wrote: > > 4. We should document how to ignore or re-order the docs. We have an > > internal API reference that comes through as the first section, but is > > of little use to anyone outside the developers. It is also confusing > > as the actual Octavia API-REF link doesn't render. > > I think this happens because it renders pages in the order that it > encounters them. If you ensure there's a table of contents on the index > page of each subsection ('/user', '/admin', '/config', ...), and that > there's a top-level table of contents linking to each of these from > your 'master_doc' as defined in 'conf.py (typically 'index') then > things _should_ render in the correct order. These table of contents > _could_ be hidden, though I haven't tested that yet. > > I plan to rework the nova docs according to the above...as soon as I > get the darn thing building. I found time to explore TOC related stuffs a bit deeper for the neutron document. I share my experiences below. BTW, it looks better to prepare some place to share our knowledge, etherpad? P.S. I still have a trouble in sample config files (relative paths and literalinclude), but it is a different topic. It would be nice if someone can dive into this topic. Let's start the detail on TOC. [TOC structure] It seems the HTML and LaTeX builders handle toctree in a bit different manners. The following are what I hit. I noted solutions/workarounds for individual items, but another solution is to use a separate top page (See "Separate top page" below). The following style like [1] is used commonly in our OpenStack project documents. This is useful for HTML documents but it is not for PDF documents :-( Installation Guide ------------------ .. toctree:: :maxdepth: 2 install/index leads to the following TOC in a PDF doc. It does not sound nice :p 1 Installation Guide 1.1 Networking service Installation Guide 2 Networking Guide 2.1 OpenStack Networking Guide To avoid this, we need to use toctree without sections in the top page. [Search page] "search" page works only for HTML pages. It is meaningless in PDF docs. The straight-forward solution is to use "only" directive. .. only:: html * :ref:`search` [URLs in toctree] > > It is also confusing > > as the actual Octavia API-REF link doesn't render. It seems the latex builder does not consider a direct link in toctree like below. In case of the neutron doc, I created a new rst file to point the API reference [2], but it is just a workaround. .. toctree :maxdepth: 2 ... API Reference ... [Separate top page] You can also have a separate top page (master_doc) for PDF document [3]. We usually specify a file specified in 'master_doc' (default: 'index') to 'startdocname' but we can use a different one. latex_documents = [ ('pdf-index', 'neutron.tex', u'Neutron Documentation', u'Neutron development team', 'manual'), ] [TOC level] By default, the first two levels are shown in TOC. If you would like to show deeper levels, you need to set 'tocdepth'. To show the fourth level, the following needs to be configured in doc/source/conf.py [4] (The number starts from 0, so 3 means the fourth level): latex_elements = { 'preamble': r'\setcounter{tocdepth}{3}', } [Stop generating the index] It seems the latex builder always tries to generate the module index by default, but if we do not load all modules in a document the index would be partial and not useful. To disable this, the following configuration in doc/source/conf.py would work: latex_elements = { 'makeindex': '', 'printindex': '', } [1] https://opendev.org/openstack/neutron/blame/commit/9d4161f955681c83bf121d84f1bc335cef758fce/doc/source/index.rst#L37-L44 [2] https://review.opendev.org/#/c/667345/6/doc/source/reference/rest-api.rst [3] https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_documents [4] 'preamble' in https://www.sphinx-doc.org/en/master/latex.html#the-latex-elements-configuration-setting -- Akihiro Motoki (irc:amotoki) From gmann at ghanshyammann.com Sat Jul 13 12:16:32 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Sat, 13 Jul 2019 21:16:32 +0900 Subject: [dev][release][qa] patrole stable/stein is created by mistake ? Message-ID: <16beb436c21.c9d98154147017.455401003873410397@ghanshyammann.com> Hi Release team, Today I noticed while doing patrole review that stable/stain has been created for patrole which is wrong. Patrole is branchless[1] and I remember I have not requested the stable branch while releasing the patrole. Is it created by mistakenly ? or intentional? [1] https://docs.openstack.org/patrole/latest/overview.html#release-versioning -gmann From gmann at ghanshyammann.com Sat Jul 13 12:19:35 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Sat, 13 Jul 2019 21:19:35 +0900 Subject: [dev][release][qa] patrole stable/stein is created by mistake ? In-Reply-To: <16beb436c21.c9d98154147017.455401003873410397@ghanshyammann.com> References: <16beb436c21.c9d98154147017.455401003873410397@ghanshyammann.com> Message-ID: <16beb463457.1193bd7a4147042.5135658897011450644@ghanshyammann.com> ---- On Sat, 13 Jul 2019 21:16:32 +0900 Ghanshyam Mann wrote ---- > Hi Release team, > > Today I noticed while doing patrole review that stable/stain has been created for patrole which is wrong. > Patrole is branchless[1] and I remember I have not requested the stable branch while releasing the patrole. > > Is it created by mistakenly ? or intentional? I found the patch which created this - https://review.opendev.org/#/c/650173/1. Can we revert that but that patch has more changes? -gmann > > [1] https://docs.openstack.org/patrole/latest/overview.html#release-versioning > > -gmann > From aj at suse.com Sat Jul 13 19:08:32 2019 From: aj at suse.com (Andreas Jaeger) Date: Sat, 13 Jul 2019 21:08:32 +0200 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? Message-ID: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> The website developer.openstack.org has the tagline "Development resources for OpenStack clouds" and hosts development and API information, especially: * An index page * The OpenStack api-guide * The document "Writing Your First OpenStack Application" ("firstapp guide") * The individual api-guides and api-references which are published from the individual projects like Nova or Keystone. The index page, the OpenStack api-guide and the document "Writing your first OpenStack Application" are hosted in the api-site repository which was formerly part of the docs team but is now orphaned. Let's look at the content hosted in api-site repository * The index page needs little updates * The OpenStack Api-Guide: This is a small guide that needs little maintenance (when projects create/retire API versions) * Writing your first OpenStack Application: This was never finished and only covers a few programming language, it's dead. With moving the repo out of docs (see https://review.opendev.org/485249 for change), the hope was that somebody took it over - but this did not happen. We have an official entry point for OpenStack's development resources and it is served by api-site repo and I suggest we look at the situation and solve it fully. I see the following options: 1) Retiring developer.openstack.org completely, this would mean we would host the api-guides and api-references on docs.openstack.org (perhaps with moving them into doc/source). If we go down this road, we need to discuss what this means (redirects) and what to do with the Api-Guide and the FirstApp guide. 2) Fully revitialize the repo and have it owned by an official team or SIG (this means reverting parts of https://review.opendev.org/485249/) 3) Retire the document "Writing your first OpenStack Application", and unretire api-site and have it owned by some official team/SIG. Any other options? What shall we do? Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF: Nils Brauckmann, Felix Imendörffer, Enrica Angelone, HRB 247165 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From james.slagle at gmail.com Sat Jul 13 20:19:13 2019 From: james.slagle at gmail.com (James Slagle) Date: Sat, 13 Jul 2019 16:19:13 -0400 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> Message-ID: On Fri, Jul 12, 2019 at 3:59 PM Harald Jensås wrote: > I've said this before, but I think we should turn this nova-less > around. Now with nova-less we create a bunch of servers, and write up > the parameters file to use the deployed-server approach. Effectively we > still neet to have the resource group in heat making a server resource > for every server. Creating the fake server resource is fast, because > Heat does'nt call Nova,Ironic to create any resources. But the stack is > equally big, with a stack for every node. i.e not N=1. > > What you are doing here, is essentially to say we don't create a > resource group that then creates N number of role stacks, one for each > overcloud node. You are creating a single generic "server" definition > per Role. So we drop the resource group and create > OS::Triple::{{Role}}.Server 1-time (once). To me it's backwards to push > a large struct with properties for N=many nodes into the creation of > that stack. I'm not entirely following what you're saying is backwards. What I've proposed is that we *don't* have any node specific data in the stack. It sounds like you're saying the way we do it today is backwards. It's correct that what's been proposed with metalsmith currently still requires the full ResourceGroup with a member for each node. With the template changes I'm proposing, that wouldn't be required, so we could actually do the Heat stack first, then metalsmith. > > Currently the puppet/role-role.yaml creates all the network ports etc. > As you only want to create it once, it instead could simply output the > UUID of the networks+subnets. These are identical for all servers in > the role. So we end up with a small heat stack. > > Once the stack is created we could use that generic "server" role data > to feed into something (ansible?, python?, mistral?) that calls > metalsmith to build the servers, then create ports for each server in > neutron, one port for each network+subnet defined in the role. Then > feed that output into the json (hieradata) that is pushed to each node > and used during service configuration, all the things we need to > configure network interfaces, /etc/hosts and so on. We need a way to > keep track of which ports belong to wich node, but I guess something > simple like using the node's ironic UUID in either the name, > description or tag field of the neutron port will work. There is also > the extra filed in Ironic which is json type, so we could place a map > of network->port_uuid in there as well. It won't matter whether we do baremetal provisioning before or after the Heat stack. Heat won't care, as it won't have any expectation to create any servers or that they are already created. We can define where we end up calling the metalsmith piece as it should be independent of the Heat stack if we make these template changes. > Another idea I've been pondering is if we put credentials on the > overcloud nodes so that the node itself could make the call to neutron > on the undercloud to create ports in neutron. I.e we just push the UUID > of the correct network and subnet where the resource should be created, > and let the overcloud node do the create. The problem with this is that I don't think it would be a good idea to have undercloud credentials on the overcloud nodes. > I think the creation of the actual Networks and Subnets can be left in > heat, it's typically 5-6 networks and 5-6 subnets so it's not a lot of > resources. Even in a large DCN deployment having 50-100 subnets per > network or even 50-100 networks I think this is'nt a problem. Agreed, I'm not specifically proposing we move those pieces at this time. -- -- James Slagle -- From sean.mcginnis at gmx.com Sat Jul 13 20:38:28 2019 From: sean.mcginnis at gmx.com (Sean McGinnis) Date: Sat, 13 Jul 2019 15:38:28 -0500 Subject: [dev][release][qa] patrole stable/stein is created by mistake ? In-Reply-To: <16beb463457.1193bd7a4147042.5135658897011450644@ghanshyammann.com> References: <16beb436c21.c9d98154147017.455401003873410397@ghanshyammann.com> <16beb463457.1193bd7a4147042.5135658897011450644@ghanshyammann.com> Message-ID: <20190713203828.GA29711@sm-workstation> On Sat, Jul 13, 2019 at 09:19:35PM +0900, Ghanshyam Mann wrote: > ---- On Sat, 13 Jul 2019 21:16:32 +0900 Ghanshyam Mann wrote ---- > > Hi Release team, > > > > Today I noticed while doing patrole review that stable/stain has been created for patrole which is wrong. > > Patrole is branchless[1] and I remember I have not requested the stable branch while releasing the patrole. > > > > Is it created by mistakenly ? or intentional? > > I found the patch which created this - https://review.opendev.org/#/c/650173/1. > > Can we revert that but that patch has more changes? > > -gmann > We can't really just revert it. If you would like to change this, please update the patrole release type to tempest-plugin or move it to be an independent release deliverable if it is not actually cycle based. You can also remove the branching information with that change, then after that is gone and there isn't a risk of recreating it, the infra team may be able to assist in deleting the existing branch. Sean From gmann at ghanshyammann.com Sun Jul 14 00:26:10 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Sun, 14 Jul 2019 09:26:10 +0900 Subject: [dev][release][qa] patrole stable/stein is created by mistake ? In-Reply-To: <20190713203828.GA29711@sm-workstation> References: <16beb436c21.c9d98154147017.455401003873410397@ghanshyammann.com> <16beb463457.1193bd7a4147042.5135658897011450644@ghanshyammann.com> <20190713203828.GA29711@sm-workstation> Message-ID: <16beddf697c.fd5e3e64150304.4626321901014847129@ghanshyammann.com> ---- On Sun, 14 Jul 2019 05:38:28 +0900 Sean McGinnis wrote ---- > On Sat, Jul 13, 2019 at 09:19:35PM +0900, Ghanshyam Mann wrote: > > ---- On Sat, 13 Jul 2019 21:16:32 +0900 Ghanshyam Mann wrote ---- > > > Hi Release team, > > > > > > Today I noticed while doing patrole review that stable/stain has been created for patrole which is wrong. > > > Patrole is branchless[1] and I remember I have not requested the stable branch while releasing the patrole. > > > > > > Is it created by mistakenly ? or intentional? > > > > I found the patch which created this - https://review.opendev.org/#/c/650173/1. > > > > Can we revert that but that patch has more changes? > > > > -gmann > > > > We can't really just revert it. If you would like to change this, please update > the patrole release type to tempest-plugin or move it to be an independent > release deliverable if it is not actually cycle based. > > You can also remove the branching information with that change, then after that > is gone and there isn't a risk of recreating it, the infra team may be able to > assist in deleting the existing branch. Release model for patrole is 'cycle-with-intermediary' which is right, we do not need to change that. We just need to remove the stable/stein and branch information which was added for the only stein. I will push the change. Can we have a tag which clearly says the branchless and no-stable branch nature for deliverables? 'tempest-plugins' was introduced for a different purpose. new tag can be applicable for other deliverables also (current or in future). That can help to avoid these errrors. -gmann > > Sean > > From masha.atakova at mail.com Sun Jul 14 12:12:32 2019 From: masha.atakova at mail.com (Mary Atakova) Date: Sun, 14 Jul 2019 15:12:32 +0300 Subject: Horizon and nova-cli inconsistency in live migration parameters Message-ID: <4d5629c4-e342-0bb6-4dd1-dd9d22d2e099@mail.com> Hi everyone, I think I have found an inconsistency between nova cli and Horizon in the way they send live migration commands to nova API. I'm not sure where this needs to be fixed, so please advise on that. The situation: I try to live-migrate a VM, and I do it through nova cli and through Horizon, and nova-api logs contain different data depending on the way I do it. 1. nova cli I run the command: nova live-migration 17359460-d23c-4acc-a9b1-5cf117b54430 and it shows up in nova-api logs as: {"os-migrateLive": {"block_migration": "auto", "host": null}} 2. horizon I run live migration via Horizon with unchecked Disk Overcommit and unchecked Block Migration. It shows up in nova-api logs as: {"os-migrateLive": {"disk_over_commit": false, "block_migration": false, "host": null}} This difference can lead to different parameters provided for live migration when block_migration: auto is translated into block_migration: true. My environment: Openstack Stein CentOS Linux release 7.6.1810 nova versions: # yum list installed | grep nova openstack-nova-api.noarch       1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-common.noarch    1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-conductor.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-console.noarch   1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-novncproxy.noarch openstack-nova-placement-api.noarch openstack-nova-scheduler.noarch 1:19.0.1-1.el7 @centos-openstack-stein python2-nova.noarch             1:19.0.1-1.el7 @centos-openstack-stein python2-novaclient.noarch       1:13.0.0-1.el7 @centos-openstack-stein horizon version: # yum list installed | grep dashboard openstack-dashboard.noarch         1:15.1.0-1.el7 @centos-openstack-stein openstack-dashboard-theme.noarch   1:15.1.0-1.el7 @centos-openstack-stein Thank you in advance for your attention Best regards, Mary From masha.atakova at mail.com Sun Jul 14 12:35:29 2019 From: masha.atakova at mail.com (Mary Atakova) Date: Sun, 14 Jul 2019 15:35:29 +0300 Subject: [nova] [horizon] Horizon and nova-cli inconsistency in live migration parameters Message-ID: <98059178-00c5-8bc0-7bbf-bf5c55c53397@mail.com> Hi everyone, I think I have found an inconsistency between nova cli and Horizon in the way they send live migration commands to nova API. I'm not sure where this needs to be fixed, so please advise on that. The situation: I try to live-migrate a VM, and I do it through nova cli and through Horizon, and nova-api logs contain different data depending on the way I do it. 1. nova cli I run the command: nova live-migration 17359460-d23c-4acc-a9b1-5cf117b54430 and it shows up in nova-api logs as: {"os-migrateLive": {"block_migration": "auto", "host": null}} 2. horizon I run live migration via Horizon with unchecked Disk Overcommit and unchecked Block Migration. It shows up in nova-api logs as: {"os-migrateLive": {"disk_over_commit": false, "block_migration": false, "host": null}} This difference can lead to different parameters provided for live migration when block_migration: auto is translated into block_migration: true. My environment: Openstack Stein CentOS Linux release 7.6.1810 nova versions: # yum list installed | grep nova openstack-nova-api.noarch       1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-common.noarch    1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-conductor.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-console.noarch   1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-novncproxy.noarch openstack-nova-placement-api.noarch openstack-nova-scheduler.noarch 1:19.0.1-1.el7 @centos-openstack-stein python2-nova.noarch             1:19.0.1-1.el7 @centos-openstack-stein python2-novaclient.noarch       1:13.0.0-1.el7 @centos-openstack-stein horizon version: # yum list installed | grep dashboard openstack-dashboard.noarch         1:15.1.0-1.el7 @centos-openstack-stein openstack-dashboard-theme.noarch   1:15.1.0-1.el7 @centos-openstack-stein Thank you in advance for your attention Best regards, Mary -------------- next part -------------- An HTML attachment was scrubbed... URL: From bansalnehal26 at gmail.com Fri Jul 12 05:00:31 2019 From: bansalnehal26 at gmail.com (Nehal Bansal) Date: Fri, 12 Jul 2019 10:30:31 +0530 Subject: [Ceilometer] [gnocchi] Regarding problem in installation of gnocchi-api Message-ID: Hi, I am trying to install Ceilometer service with Openstack queens release. On installing gnocchi-api with apt-get install gnocchi-api I get the following error: Registering service and endpoints for gnocchi with type metric at http://192.168.133.81:8041 Failed to discover available identity versions when contacting http://127.0.0.1:35357/v3/. Attempting to parse version from URL. Unable to establish connection to http://127.0.0.1:35357/v3/auth/tokens: HTTPConnectionPool(host='127.0.0.1', port=35357): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) Warning - data is empty Failed to discover available identity versions when contacting http://127.0.0.1:35357/v3/. Attempting to parse version from URL. Unable to establish connection to http://127.0.0.1:35357/v3/auth/tokens: HTTPConnectionPool(host='127.0.0.1', port=35357): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) dpkg: error processing package gnocchi-api (--configure): subprocess installed post-installation script returned error exit status 1 Errors were encountered while processing: gnocchi-api E: Sub-process /usr/bin/dpkg returned an error code (1) Kindly, let me know how to proceed from here. Regards, Nehal -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Sun Jul 14 22:34:13 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Sun, 14 Jul 2019 22:34:13 +0000 Subject: [tc][ptl][infra] Resolution: Mandatory Repository Retirement Message-ID: <20190714223412.wc34j47lvgd3kykm@yuggoth.org> When an effort ceases to be an official part (deliverable) of OpenStack with the intent to continue development outside OpenStack's technical governance, the maintainers of that codebase may end up later abandoning it or making other choices at odds with OpenStack policies with no clear indication to consumers. Some folks may have been using it when it was still part of OpenStack, and so may continue to have expectations which are no longer in sync with reality. To solve this, a resolution has been proposed asserting that software which is removed from OpenStack's technical governance in the future can only be continued as a fork, so that its original Git repository in the "openstack" namespace on OpenDev can be retired with a clear message letting consumers of that source code know what has happened and where it has gone. Some breathing room is allowed, as the initial draft includes measures to allow any deliverable repositories which move out of OpenStack prior to the end of the current development cycle (Train) to be renamed into new OpenDev namespaces with a redirect before the new policy goes into effect. The text of the proposed resolution can be found here: https://review.opendev.org/670741 Please follow up on the above review with any comments/concerns, or you can express them as a reply to this ML thread, or in private to me or any other TC members. Thanks for reading! -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From berndbausch at gmail.com Sun Jul 14 22:39:42 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Mon, 15 Jul 2019 07:39:42 +0900 Subject: [Ceilometer] [gnocchi] Regarding problem in installation of gnocchi-api In-Reply-To: References: Message-ID: <1539d343-766b-33fb-742c-1c2b7e9f9469@gmail.com> Either Keystone is not running, but that would cause other problems in the cloud. In short, the cloud would not function at all. Or the URL http://127.0.0.1:35357 in the Gnocchi configuration is incorrect. 127.0.0.1 (localhost) is an unusual address, to say the least; try 192.168.133.81 instead. Also ensure that Keystone listens at port 35357 and correct it if not. Bernd. On 7/12/2019 2:00 PM, Nehal Bansal wrote: > I am trying to install Ceilometer service with Openstack queens > release. On installing gnocchi-api with apt-get install gnocchi-api I > get the following error: > > Registering service and endpoints for gnocchi with type metric at > http://192.168.133.81:8041 > Failed to discover available identity versions when contacting > http://127.0.0.1:35357/v3/. Attempting to parse version from URL. > Unable to establish connection to > http://127.0.0.1:35357/v3/auth/tokens: > HTTPConnectionPool(host='127.0.0.1', port=35357): Max retries exceeded > with url: /v3/auth/tokens (Caused by > NewConnectionError(' 0x7f0bf2672250>: Failed to establish a new connection: [Errno 111] > Connection refused',)) > > Kindly, let me know how to proceed from here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangtrinhnt at gmail.com Mon Jul 15 01:23:01 2019 From: dangtrinhnt at gmail.com (Trinh Nguyen) Date: Mon, 15 Jul 2019 10:23:01 +0900 Subject: [telemetry] Looking for new meeting time Message-ID: Hi team, I've changed job and could not afford doing the meeting at 02:00 UTC on Thursday anymore. My possible time slot is from 13:00-15:00 UTC Tuesday-Thursday. Please propose a better time frame for our team meeting. Sincerely, -- *Trinh Nguyen* *www.edlab.xyz * -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangtrinhnt at gmail.com Mon Jul 15 01:26:55 2019 From: dangtrinhnt at gmail.com (Trinh Nguyen) Date: Mon, 15 Jul 2019 10:26:55 +0900 Subject: [searchlight] Team meeting today Message-ID: Hi team, Train-2 milestone is coming. Let's get together for a team meeting today if possible. There are a couple things we need to finish. Yours, -- *Trinh Nguyen* *www.edlab.xyz * -------------- next part -------------- An HTML attachment was scrubbed... URL: From anlin.kong at gmail.com Mon Jul 15 04:04:55 2019 From: anlin.kong at gmail.com (Lingxian Kong) Date: Mon, 15 Jul 2019 16:04:55 +1200 Subject: [telemetry] Looking for new meeting time In-Reply-To: References: Message-ID: congratulations first :-) I'm on UTC+12, so meeting during 13:00-15:00 UTC is hard for me :-( Best regards, Lingxian Kong Catalyst Cloud On Mon, Jul 15, 2019 at 1:23 PM Trinh Nguyen wrote: > Hi team, > > I've changed job and could not afford doing the meeting at 02:00 UTC on > Thursday anymore. My possible time slot is from 13:00-15:00 UTC > Tuesday-Thursday. > > Please propose a better time frame for our team meeting. > > Sincerely, > > -- > *Trinh Nguyen* > *www.edlab.xyz * > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From noonedeadpunk at ya.ru Mon Jul 15 05:28:17 2019 From: noonedeadpunk at ya.ru (=?UTF-8?B?0JTQvNC40YLRgNC40Lkg0KDQsNCx0L7RgtGP0LPQvtCy?=) Date: Mon, 15 Jul 2019 08:28:17 +0300 Subject: [Ceilometer] [gnocchi] Regarding problem in installation of gnocchi-api In-Reply-To: References: Message-ID: Hi, This looks like packaging problem while running postinst hook since endpoint is hardcoded in pkgos_get_id(). As a workaround I may advice you to configure gnocchi as wsgi application: https://gnocchi.xyz/operating.html#running-api-as-a-wsgi-application or just install service using pip (I guess you'll need 2.x version for Queens). пт, 12 июл. 2019 г. в 08:00, Nehal Bansal : > Hi, > I am trying to install Ceilometer service with Openstack queens release. > On installing gnocchi-api with apt-get install gnocchi-api I get the > following error: > > Registering service and endpoints for gnocchi with type metric at > http://192.168.133.81:8041 > Failed to discover available identity versions when contacting > http://127.0.0.1:35357/v3/. Attempting to parse version from URL. > Unable to establish connection to http://127.0.0.1:35357/v3/auth/tokens: > HTTPConnectionPool(host='127.0.0.1', port=35357): Max retries exceeded with > url: /v3/auth/tokens (Caused by > NewConnectionError(' 0x7f0bf2672250>: Failed to establish a new connection: [Errno 111] > Connection refused',)) > Warning - data is empty > Failed to discover available identity versions when contacting > http://127.0.0.1:35357/v3/. Attempting to parse version from URL. > Unable to establish connection to http://127.0.0.1:35357/v3/auth/tokens: > HTTPConnectionPool(host='127.0.0.1', port=35357): Max retries exceeded with > url: /v3/auth/tokens (Caused by > NewConnectionError(' 0x7fa1c8b0cb90>: Failed to establish a new connection: [Errno 111] > Connection refused',)) > dpkg: error processing package gnocchi-api (--configure): > subprocess installed post-installation script returned error exit status 1 > Errors were encountered while processing: > gnocchi-api > E: Sub-process /usr/bin/dpkg returned an error code (1) > > Kindly, let me know how to proceed from here. > > Regards, > Nehal > -- -- С Уважением, Дмитрий -------------- next part -------------- An HTML attachment was scrubbed... URL: From rony.khan at brilliant.com.bd Mon Jul 15 06:26:16 2019 From: rony.khan at brilliant.com.bd (Md. Farhad Hasan Khan) Date: Mon, 15 Jul 2019 12:26:16 +0600 Subject: OpenStack Neutron IPv6 Message-ID: <03c201d53ad6$3555de90$a0019bb0$@brilliant.com.bd> Hi, How to configure IPv6 in OpenStack Rocky neutron? Please help me. Thanks & B'Rgds, Rony -------------- next part -------------- An HTML attachment was scrubbed... URL: From ruslanas at lpic.lt Mon Jul 15 06:42:27 2019 From: ruslanas at lpic.lt (=?UTF-8?Q?Ruslanas_G=C5=BEibovskis?=) Date: Mon, 15 Jul 2019 08:42:27 +0200 Subject: OpenStack Neutron IPv6 In-Reply-To: <03c201d53ad6$3555de90$a0019bb0$@brilliant.com.bd> References: <03c201d53ad6$3555de90$a0019bb0$@brilliant.com.bd> Message-ID: What do you want to configure with IPv6? Provider network? Local network? VIM? Provider and Local networks can be configured using horizon very easily, just in a drop down specify IPv6. I think it do not require any additional actions, unlease during cloud setup you specified another way. On Mon, 15 Jul 2019 at 08:30, Md. Farhad Hasan Khan < rony.khan at brilliant.com.bd> wrote: > Hi, > > How to configure IPv6 in OpenStack Rocky neutron? Please help me. > > > > Thanks & B’Rgds, > > Rony > > > > -- Ruslanas Gžibovskis +370 6030 7030 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sungn2 at lenovo.com Mon Jul 15 06:55:16 2019 From: sungn2 at lenovo.com (Guannan GN2 Sun) Date: Mon, 15 Jul 2019 06:55:16 +0000 Subject: [ironic] Using ironic in a devstack environment to deploy OS on bare-metal host. Message-ID: <8a82c02e5ae64fc79e02e10629a65556@lenovo.com> Hi team, I met some problem when using ironic in a devstack environment to deploy OS on bare-metal host. By now I can successfully boot into the tiny os and set ip, also I can ping the ip from my devstack server. However, when the tiny os running some configuration, it will failed after about 5 minutes. I found some log information in "/var/log/ironic-python-agent.log" in the tiny os that may relate to the problem: "DEBUG oslo_concurrency.processutils [-] u'iscsistart -f' failed. Not Retrying. execute /usr/local/lib/python2.7/site-packages/oslo_concurrency/processutils.py:457" "DEBUG root [-] No iscsi connection detected. Skipping iscsi. Error: [Errno 2] No such file or directory _check_for_iscsi /usr/local/lib/python2.7/site-packages/ironic_python_agent/hardware.py:107" "WARNING root [-] The root device was not detected in 27 seconds: DeviceNotFound: Error finding the disk or partition device to deploy the image onto: No suitable device was found for deployment - root device hints were not provided and all found block devices are smaller than 4294967296B." "WARNING root [-] Can't find field vendor for device usb0 in device class net: IOError: [Errno 2] No such file or directory: '/sys/class/net/usb0/device/vendor'" "WARNING root [-] Can't find field vendor for device tunl0 in device class net: IOError: [Errno 2] No such file or directory: '/sys/class/net/tunl0/device/vendor'" "DEBUG root [-] No Mellanox devices found evaluate_hardware_support /usr/local/lib/python2.7/site-packages/ironic_python_agent/hardware_managers/mlnx.py:84" And when it deploy faild and I run command "openstack baremetal node show server-51", it will show following information on 'last_error' field: "Asynchronous exception: Node failed to deploy. Exception: Connection to agent failed: Failed to connect to the agent running on node f8794658-bc69-40cb-9673-be75f78ced21 for invoking command iscsi.start_iscsi_target. Error: ('Connection aborted.', BadStatusLine("''",)) for node" I'm trying to debug, but have not find the root cause, please give me your advice if someone have experience on it. Thank you! Best Regards, Guannan -------------- next part -------------- An HTML attachment was scrubbed... URL: From manulachathurika at gmail.com Mon Jul 15 08:44:11 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Mon, 15 Jul 2019 14:14:11 +0530 Subject: ImportError: No module named django.core.wsgi - DevStack Message-ID: Hi All, I have successfully install DevStack in Ubuntu 18.04. But when I'm accessing the dashboard I'm getting 500 error. What I'm getting in horizon_error.log is, 2019-07-15 14:10:43.218296 mod_wsgi (pid=31763): Target WSGI script '/opt/stack/horizon/openstack_dashboard/wsgi.py' cannot be loaded as Python module. 2019-07-15 14:10:43.218323 mod_wsgi (pid=31763): Exception occurred processing WSGI script '/opt/stack/horizon/openstack_dashboard/wsgi.py'. 2019-07-15 14:10:43.218349 Traceback (most recent call last): 2019-07-15 14:10:43.218370 File "/opt/stack/horizon/openstack_dashboard/wsgi.py", line 21, in 2019-07-15 14:10:43.218401 from django.core.wsgi import get_wsgi_application 2019-07-15 14:10:43.218422 ImportError: No module named django.core.wsgi Python Version is : 3.6.8 Django version is : 2.0.5 I tried with uninstalling Djanago and reinstalling it. But it didn't work for me. Can someone help me on how to resole this issue ? Thanks ! -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hjensas at redhat.com Mon Jul 15 09:12:46 2019 From: hjensas at redhat.com (Harald =?ISO-8859-1?Q?Jens=E5s?=) Date: Mon, 15 Jul 2019 11:12:46 +0200 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> Message-ID: On Sat, 2019-07-13 at 16:19 -0400, James Slagle wrote: > On Fri, Jul 12, 2019 at 3:59 PM Harald Jensås > wrote: > > I've said this before, but I think we should turn this nova-less > > around. Now with nova-less we create a bunch of servers, and write > > up > > the parameters file to use the deployed-server approach. > > Effectively we > > still neet to have the resource group in heat making a server > > resource > > for every server. Creating the fake server resource is fast, > > because > > Heat does'nt call Nova,Ironic to create any resources. But the > > stack is > > equally big, with a stack for every node. i.e not N=1. > > > > What you are doing here, is essentially to say we don't create a > > resource group that then creates N number of role stacks, one for > > each > > overcloud node. You are creating a single generic "server" > > definition > > per Role. So we drop the resource group and create > > OS::Triple::{{Role}}.Server 1-time (once). To me it's backwards to > > push > > a large struct with properties for N=many nodes into the creation > > of > > that stack. > > I'm not entirely following what you're saying is backwards. What I've > proposed is that we *don't* have any node specific data in the stack. > It sounds like you're saying the way we do it today is backwards. > What I mean to say is that I think the way we are integrating nova-less by first deploying the servers, to then provide the data to Heat to create the resource groups as we do today becomes backwards when your work on N=1 is introduced. > It's correct that what's been proposed with metalsmith currently > still > requires the full ResourceGroup with a member for each node. With the > template changes I'm proposing, that wouldn't be required, so we > could > actually do the Heat stack first, then metalsmith. > Yes, this is what I think we should do. Especially if your changes here removes the resource group entirely. It makes more sense to create the stack, and once that is created we can do deployment, scaling etc without updating the stack again. > > Currently the puppet/role-role.yaml creates all the network ports > > etc. > > As you only want to create it once, it instead could simply output > > the > > UUID of the networks+subnets. These are identical for all servers > > in > > the role. So we end up with a small heat stack. > > > > Once the stack is created we could use that generic "server" role > > data > > to feed into something (ansible?, python?, mistral?) that calls > > metalsmith to build the servers, then create ports for each server > > in > > neutron, one port for each network+subnet defined in the role. Then > > feed that output into the json (hieradata) that is pushed to each > > node > > and used during service configuration, all the things we need to > > configure network interfaces, /etc/hosts and so on. We need a way > > to > > keep track of which ports belong to wich node, but I guess > > something > > simple like using the node's ironic UUID in either the name, > > description or tag field of the neutron port will work. There is > > also > > the extra filed in Ironic which is json type, so we could place a > > map > > of network->port_uuid in there as well. > > It won't matter whether we do baremetal provisioning before or after > the Heat stack. Heat won't care, as it won't have any expectation to > create any servers or that they are already created. We can define > where we end up calling the metalsmith piece as it should be > independent of the Heat stack if we make these template changes. > This is true. But, in your previous mail in this thread you wrote: """ Other points: - Baremetal provisioning and port creation are presently handled by Heat. With the ongoing efforts to migrate baremetal provisioning out of Heat (nova-less deploy), I think these efforts are very complimentary. Eventually, we get to a point where Heat is not actually creating any other OpenStack API resources. For now, the patches only work when using pre-provisioned nodes. """ IMO "baremetal provision and port creation" fit together. (I read the above statement so as well.) Currently nova-less creates the ctlplane port and provision the baremetal node. If we want to do both baremetal provisioning and port creation togheter (I think this makes sense), we have to do it after the stack has created the networks. What I envision is to have one method that creates all the ports, ctlplane + composable networks in a unified way. Today these are created differently, the ctlplane port is part of the server resource (or metalsmith in nova-less case) and the other ports are created by heat. > > I think the creation of the actual Networks and Subnets can be left > > in > > heat, it's typically 5-6 networks and 5-6 subnets so it's not a lot > > of > > resources. Even in a large DCN deployment having 50-100 subnets per > > network or even 50-100 networks I think this is'nt a problem. > > Agreed, I'm not specifically proposing we move those pieces at this > time. > +1 From thierry at openstack.org Mon Jul 15 09:50:05 2019 From: thierry at openstack.org (Thierry Carrez) Date: Mon, 15 Jul 2019 11:50:05 +0200 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> Message-ID: Andreas Jaeger wrote: > [...] > I see the following options: > > 1) Retiring developer.openstack.org completely, this would mean we would > host the api-guides and api-references on docs.openstack.org (perhaps > with moving them into doc/source). If we go down this road, we need to > discuss what this means (redirects) and what to do with the Api-Guide > and the FirstApp guide. > > 2) Fully revitialize the repo and have it owned by an official team or > SIG (this means reverting parts of https://review.opendev.org/485249/) > > 3) Retire the document "Writing your first OpenStack Application", and > unretire api-site and have it owned by some official team/SIG. > > Any other options? What shall we do? Thanks Andreas for raising this. As an extra data point, my long-term plan was to have SDKs and CLIs properly listed in the Software pages under SDKs[1], including third-party ones in their own subtab, all driven from the osf/openstack-map repository[2]. With that in mind, I think it would make sense to look into retiring developer.openstack.org, and move docs to docs.openstack.org. We could also revive https://www.openstack.org/appdev/ and use it as the base landing page to direct application-side people to the various pieces. [1] https://www.openstack.org/software/project-navigator/sdks [2] https://opendev.org/osf/openstack-map/ -- Thierry Carrez (ttx) From thierry at openstack.org Mon Jul 15 10:30:26 2019 From: thierry at openstack.org (Thierry Carrez) Date: Mon, 15 Jul 2019 12:30:26 +0200 Subject: [dev][release][qa] patrole stable/stein is created by mistake ? In-Reply-To: <16beddf697c.fd5e3e64150304.4626321901014847129@ghanshyammann.com> References: <16beb436c21.c9d98154147017.455401003873410397@ghanshyammann.com> <16beb463457.1193bd7a4147042.5135658897011450644@ghanshyammann.com> <20190713203828.GA29711@sm-workstation> <16beddf697c.fd5e3e64150304.4626321901014847129@ghanshyammann.com> Message-ID: <0a85c82c-f312-a8a9-34af-f5b812c8f2b3@openstack.org> Ghanshyam Mann wrote: > ---- On Sun, 14 Jul 2019 05:38:28 +0900 Sean McGinnis wrote ---- > > On Sat, Jul 13, 2019 at 09:19:35PM +0900, Ghanshyam Mann wrote: > > > ---- On Sat, 13 Jul 2019 21:16:32 +0900 Ghanshyam Mann wrote ---- > > > > Hi Release team, > > > > > > > > Today I noticed while doing patrole review that stable/stain has been created for patrole which is wrong. > > > > Patrole is branchless[1] and I remember I have not requested the stable branch while releasing the patrole. > > > > > > > > Is it created by mistakenly ? or intentional? > > > > > > I found the patch which created this - https://review.opendev.org/#/c/650173/1. > > > > > > Can we revert that but that patch has more changes? > > > > > > -gmann > > > > > > > We can't really just revert it. If you would like to change this, please update > > the patrole release type to tempest-plugin or move it to be an independent > > release deliverable if it is not actually cycle based. > > > > You can also remove the branching information with that change, then after that > > is gone and there isn't a risk of recreating it, the infra team may be able to > > assist in deleting the existing branch. > > Release model for patrole is 'cycle-with-intermediary' which is right, we do not need > to change that. We just need to remove the stable/stein and branch information which > was added for the only stein. I will push the change. > > Can we have a tag which clearly says the branchless and no-stable branch nature for deliverables? > > 'tempest-plugins' was introduced for a different purpose. new tag can be applicable for other > deliverables also (current or in future). That can help to avoid these errrors. Creating a new release model or a new deliverable type sounds a bit overkill. I think the simpler would be to add a new value to the existing "stable-branch-type" key. Like "stable-branch-type: none" and then set that value for all tempest plugins, tempest itself and patrole. See https://review.opendev.org/670808 as a strawman. -- Thierry Carrez (ttx) From tomas.bredar at gmail.com Mon Jul 15 11:10:49 2019 From: tomas.bredar at gmail.com (=?UTF-8?B?VG9tw6HFoSBCcmVkw6Fy?=) Date: Mon, 15 Jul 2019 13:10:49 +0200 Subject: [tripleo][cinder][netapp] In-Reply-To: References: Message-ID: Hi Alan! Thanks for the pointers. For now I'm going with a single backend with two NFS shares, so I'll use the tripleo templates for netapp. For the future, could you point me to the right template / puppet-cinder code which can create multiple nfs share files for me? Or should I create my own puppet manifest? Thanks again. Tomas pi 12. 7. 2019 o 16:11 Alan Bishop napísal(a): > > On Fri, Jul 12, 2019 at 6:09 AM Tomáš Bredár > wrote: > >> Hi Emilien! >> >> Thanks for your help. Yes with this I am able to define multiple stanzas >> in cinder.conf. However netapp driver needs a .conf file with the nfs >> shares listed in it. Defining multiple configuration files with nfs share >> details in each is not possible with the manual you've sent nor with the >> templates in my first email. >> > > Hi Tomas, > > When deploying a single backend, the tripleo template takes care of > generating the nfs shares file (actually, puppet-cinder generates the file, > but it's triggered by tripleo). But when you use the custom backend method > that Emilien pointed you to use, then you are responsible for supplying all > the pieces for the backend(s) to function correctly. This means you will > need to generate the nfs shares file on the host (controller), and then > bind mount the file using CinderVolumeOptVolumes so that the shares file on > the host is visible to the cinder-volume process running in a container. > > I'm wondering if it's possible to define a second backend by creating >> another service, for example "OS::TripleO::Services::CinderBackendNetApp2" ? >> > > Sorry, this won't work. TripleO will trying to deploy two completely > separate instances of the cinder-volume service, but the two deployments > will step all over each other. There has been a long standing goal of > enhancing tripleo so that it can deploy multiple instances of a cinder > backend, but it's a complex task that will require non-trivial changes to > tripleo. > > Alan > > Tomas >> >> št 11. 7. 2019 o 14:35 Emilien Macchi napísal(a): >> >>> On Thu, Jul 11, 2019 at 7:32 AM Tomáš Bredár >>> wrote: >>> >>>> Hi community, >>>> >>>> I'm trying to define multiple NetApp storage backends via Tripleo >>>> installer. >>>> According to [1] the puppet manifest supports multiple backends. >>>> The current templates [2] [3] support only single backend. >>>> Does anyone know how to define multiple netapp backends in the >>>> tripleo-heat environment files / templates? >>>> >>> >>> We don't support that via the templates that you linked, however if you >>> follow this manual you should be able to configure multiple NetApp backends: >>> >>> https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/cinder_custom_backend.html >>> >>> Let us know how it worked! >>> -- >>> Emilien Macchi >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Mon Jul 15 12:30:56 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Mon, 15 Jul 2019 14:30:56 +0200 Subject: [neutron] Bug deputy - week of July 8th Message-ID: Hi, I was bug deputy last week. It was relatively easy week. Below is my summary of what happened there: *High* https://bugs.launchpad.net/neutron/+bug/1835731 - DVR router with configured external_network_bridge option don't have external connectivity, patch proposed https://review.opendev.org/#/c/669640/ https://bugs.launchpad.net/neutron/+bug/1836023 - OVS agent "hangs" while processing trusted ports - fix proposed, I added loadimact tag to it, https://bugs.launchpad.net/neutron/+bug/1836095 - Improve "OVSFirewallDriver.process_trusted_ports" - fix also in progress - this is “child” of 1836023 https://bugs.launchpad.net/neutron/+bug/1836015 - [neutron-fwaas]firewall goup status is inactive when updating policy in fwg - fix proposed already *Medium* https://bugs.launchpad.net/neutron/+bug/1835914 - Test test_show_network_segment_range failing - gate failure issue, set to medium as it is not happening very often - we need volunteer for this one, https://bugs.launchpad.net/neutron/+bug/1836028 - Functional Test script results in an authentication error, fix in progress: https://review.opendev.org/#/c/670021/ https://bugs.launchpad.net/neutron/+bug/1836037 - Routed provider networks nova inventory update fails - fix in progress: https://review.opendev.org/670105 https://bugs.launchpad.net/neutron/+bug/1836253 - Sometimes InstanceMetada API returns 404 due to invalid InstaceID returned by _get_instance_and_tenant_id() - we need volunteer for this https://bugs.launchpad.net/neutron/+bug/1836565 - Functional test test_keepalived_state_change_notification may fail do to race condition - fix in progress: https://review.opendev.org/670815 *Low* https://bugs.launchpad.net/neutron/+bug/1836263 - [DOCS] doc: PUT /ports/{port_id} updates selectively - we need volunteer for this one *Incomplete* https://bugs.launchpad.net/neutron/+bug/1835848 - Check the list of ports, fixed IP is empty and still connected devices are displayed - waiting for some more info from reporter https://bugs.launchpad.net/neutron/+bug/1775644 - Neutron fwaas v2 group port binding failed - waiting for info from reporter *Old bugs revived recently* https://bugs.launchpad.net/neutron/+bug/1806032 - shim API extension proposed by Bence was merged long time ago but nothing else happens since then, maybe we should get back to this? — Slawek Kaplonski Senior software engineer Red Hat From abishop at redhat.com Mon Jul 15 13:56:05 2019 From: abishop at redhat.com (Alan Bishop) Date: Mon, 15 Jul 2019 06:56:05 -0700 Subject: [tripleo][cinder][netapp] In-Reply-To: References: Message-ID: On Mon, Jul 15, 2019 at 4:11 AM Tomáš Bredár wrote: > Hi Alan! > Thanks for the pointers. For now I'm going with a single backend with two > NFS shares, so I'll use the tripleo templates for netapp. > For the future, could you point me to the right template / puppet-cinder > code which can create multiple nfs share files for me? Or should I create > my own puppet manifest? > Hi Tomas, The puppet-cinder code that renders the shares config file is [1], and the data comes from puppet-tripleo [2]. This is puppet hiera data, and the value is bound to the CindeNetappNfsShares tripleo parameter [3]. So you should be able to deploy a single backend that accesses multiple shares by adding something like this to your tripleo deployment. parameter_defaults: CinderNetappNfsShares: 'host_1:/path/to/share_1,host_2:/path/to/share_2' [1] https://opendev.org/openstack/puppet-cinder/src/branch/stable/queens/manifests/backend/netapp.pp#L280 [2] https://opendev.org/openstack/puppet-tripleo/src/branch/stable/queens/manifests/profile/base/cinder/volume/netapp.pp#L38 [3] https://opendev.org/openstack/tripleo-heat-templates/src/branch/stable/queens/puppet/services/cinder-backend-netapp.yaml#L137 Alan Thanks again. > > Tomas > > pi 12. 7. 2019 o 16:11 Alan Bishop napísal(a): > >> >> On Fri, Jul 12, 2019 at 6:09 AM Tomáš Bredár >> wrote: >> >>> Hi Emilien! >>> >>> Thanks for your help. Yes with this I am able to define multiple stanzas >>> in cinder.conf. However netapp driver needs a .conf file with the nfs >>> shares listed in it. Defining multiple configuration files with nfs share >>> details in each is not possible with the manual you've sent nor with the >>> templates in my first email. >>> >> >> Hi Tomas, >> >> When deploying a single backend, the tripleo template takes care of >> generating the nfs shares file (actually, puppet-cinder generates the file, >> but it's triggered by tripleo). But when you use the custom backend method >> that Emilien pointed you to use, then you are responsible for supplying all >> the pieces for the backend(s) to function correctly. This means you will >> need to generate the nfs shares file on the host (controller), and then >> bind mount the file using CinderVolumeOptVolumes so that the shares file on >> the host is visible to the cinder-volume process running in a container. >> >> I'm wondering if it's possible to define a second backend by creating >>> another service, for example "OS::TripleO::Services::CinderBackendNetApp2" ? >>> >> >> Sorry, this won't work. TripleO will trying to deploy two completely >> separate instances of the cinder-volume service, but the two deployments >> will step all over each other. There has been a long standing goal of >> enhancing tripleo so that it can deploy multiple instances of a cinder >> backend, but it's a complex task that will require non-trivial changes to >> tripleo. >> >> Alan >> >> Tomas >>> >>> št 11. 7. 2019 o 14:35 Emilien Macchi napísal(a): >>> >>>> On Thu, Jul 11, 2019 at 7:32 AM Tomáš Bredár >>>> wrote: >>>> >>>>> Hi community, >>>>> >>>>> I'm trying to define multiple NetApp storage backends via Tripleo >>>>> installer. >>>>> According to [1] the puppet manifest supports multiple backends. >>>>> The current templates [2] [3] support only single backend. >>>>> Does anyone know how to define multiple netapp backends in the >>>>> tripleo-heat environment files / templates? >>>>> >>>> >>>> We don't support that via the templates that you linked, however if you >>>> follow this manual you should be able to configure multiple NetApp backends: >>>> >>>> https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/cinder_custom_backend.html >>>> >>>> Let us know how it worked! >>>> -- >>>> Emilien Macchi >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From raubvogel at gmail.com Mon Jul 15 14:19:19 2019 From: raubvogel at gmail.com (Mauricio Tavares) Date: Mon, 15 Jul 2019 10:19:19 -0400 Subject: Configuring the network interfaces of an instance Message-ID: If I have an instance (vm guest) with more than one network interface, how can I tell it during the install process to use a specific one? -------------- next part -------------- An HTML attachment was scrubbed... URL: From alifshit at redhat.com Mon Jul 15 14:24:36 2019 From: alifshit at redhat.com (Artom Lifshitz) Date: Mon, 15 Jul 2019 10:24:36 -0400 Subject: [nova][neutron][CI] test_server_connectivity_cold_migration_revert failing again Message-ID: Looks like we merged [1] too soon, and despite external events handling during migration revert now being handled correctly in Nova with the merger of [2], there's something else broken, perhaps at the Neutron level, that prevents test_server_connectivity_cold_migration_revert from consistently passing. Until we figure out what that "something" is, I've proposed to skip the test in tempest [3] and filed a bug for it [4]. The bug is filed under Neutron, but that's more of a placeholder based on preliminary guessing. We can easily change it to Nova or some other component once we understand better what's going on. [1] https://review.opendev.org/#/c/663405/ [2] https://review.opendev.org/#/c/667177/ [3] https://review.opendev.org/#/c/670848/1 [4] https://bugs.launchpad.net/neutron/+bug/1836595 From smooney at redhat.com Mon Jul 15 14:28:33 2019 From: smooney at redhat.com (Sean Mooney) Date: Mon, 15 Jul 2019 15:28:33 +0100 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> Message-ID: <116050ecdd0b7c5ecbe728d914b67d3f0770a2ea.camel@redhat.com> On Mon, 2019-07-15 at 11:50 +0200, Thierry Carrez wrote: > Andreas Jaeger wrote: > > [...] > > I see the following options: > > > > 1) Retiring developer.openstack.org completely, this would mean we would > > host the api-guides and api-references on docs.openstack.org (perhaps > > with moving them into doc/source). If we go down this road, we need to > > discuss what this means (redirects) and what to do with the Api-Guide > > and the FirstApp guide. > > > > 2) Fully revitialize the repo and have it owned by an official team or > > SIG (this means reverting parts of https://review.opendev.org/485249/) > > > > 3) Retire the document "Writing your first OpenStack Application", and > > unretire api-site and have it owned by some official team/SIG. > > > > Any other options? What shall we do? > > Thanks Andreas for raising this. > > As an extra data point, my long-term plan was to have SDKs and CLIs > properly listed in the Software pages under SDKs[1], including > third-party ones in their own subtab, all driven from the > osf/openstack-map repository[2]. > > With that in mind, I think it would make sense to look into retiring > developer.openstack.org, i use https://developer.openstack.org/api-ref/compute/ almost daily so unless we host the api ref somwhere else and put redirect in place i would hope we can keep this inplace. if we move it under docs like the config stuff https://docs.openstack.org/nova/latest/configuration/config.html or somewhere else that is fine but i fine it very useful to be able to link the rendered api docs to people on irc that ask questions. i can obviosly point peple to github https://github.com/openstack/nova/blob/master/api-ref/source/servers.inc but unlike the configs the api ref is much less readable with out rendering it with sphinx > and move docs to docs.openstack.org. We could > also revive https://www.openstack.org/appdev/ and use it as the base > landing page to direct application-side people to the various pieces. > > [1] https://www.openstack.org/software/project-navigator/sdks > [2] https://opendev.org/osf/openstack-map/ > From km.giuseppesannino at gmail.com Mon Jul 15 14:29:47 2019 From: km.giuseppesannino at gmail.com (Giuseppe Sannino) Date: Mon, 15 Jul 2019 16:29:47 +0200 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: <01AA2BA1-3EEC-4142-BCDB-F9861707102E@redhat.com> References: <01AA2BA1-3EEC-4142-BCDB-F9861707102E@redhat.com> Message-ID: Hi! first of all, thanks for the fast replies. I do appreciate that. I did some more test trying to figure out the issue. - Set UseDNS to "no" in sshd_config => Issue persists - Installed and configured Telnet => Telnet login is slow as well >From the "top" or "auth.log"nothing specific popped up. I can sshd taking some cpu for a short while but nothing more than that. Once logged in the VM is not too slow. CLI doesn't get stuck or similar. One thing worthwhile to mention, it seems like the writing throughput on the disk is a bit slow: 67MB/s wrt around 318MB/s of another VM running on a "datacenter" Openstack installation. The Cinder Volume docker is running on the Compute Host and Cinder is using the filesystem as backend. BR /Giuseppe On Fri, 12 Jul 2019 at 17:41, Slawek Kaplonski wrote: > Hi, > > I suspect some problems with names resolving. Can You check if You have > also such delay when doing e.g. “sudo” commands after You ssh to the > instance? > > > On 12 Jul 2019, at 16:23, Brian Haley wrote: > > > > > > > > On 7/12/19 9:57 AM, Giuseppe Sannino wrote: > >> Hi community, > >> I need your help ,tips, advices. > >> *> Environment <* > >> I have deployed Openstack "Stein" using the latest kolla-ansible on the > following deployment topology: > >> 1) OS Controller running as VM on a "cloud" location > >> 2) OS Compute running on a baremetal server remotely (wrt OS > Controller) location > >> 3) Network node running on the Compute host > >> As per the above info, Controller and compute run on two different > networks. > >> Kolla-Ansible is not really designed for such scenario but after > manipulating the globals.yml and the inventory files (basically I had to > move node specific network settings from the globals to the inventory > file), eventually the deployment works fine. > >> *> Problem <* > >> I have no specific issue working with this deployment except the > following: > >> "SSH connection to the VM is quite slow". > >> It takes around 20 seconds for me to log into the VM (Ubuntu, CentOS, > whatever). > > > > But once logged-in things are OK? For example, an scp stalls the same > way, but the transfer is fast? > > > >> *> Observations <* > >> * Except for the slowness during the SSH login, I don't have any > >> further specific issue working with this envirorment > >> * With the Network on the Compute I can turn the OS controller off > >> with no impact on the VM. Still the connection is slow > >> * I tried different type of images (Ubuntu, CentOS, Windows) always > >> with the same result. > >> * SSH connection is slow even if I try to login into the VM within the > >> IP Namespace > >> From the ssh -vvv, I can see that the authentication gets stuck here: > >> debug1: Authentication succeeded (publickey). > >> Authenticated to ***** > >> debug1: channel 0: new [client-session] > >> debug3: ssh_session2_open: channel_new: 0 > >> debug2: channel 0: send open > >> debug3: send packet: type 90 > >> debug1: Requesting no-more-sessions at openssh.com no-more-sessions at openssh.com> > >> debug3: send packet: type 80 > >> debug1: Entering interactive session. > >> debug1: pledge: network > >> >>>>> 10 to 15 seconds later > > > > What is sshd doing at this time? Have you tried enabling debug or > running tcpdump when a new connection is attempted? At first glance I'd > say it's a DNS issue since it eventually succeeds, the logs would help to > point in a direction. > > > > -Brian > > > > > >> debug3: receive packet: type 80 > >> debug1: client_input_global_request: rtype hostkeys-00 at openssh.com > want_reply 0 > >> debug3: receive packet: type 91 > >> debug2: callback start > >> debug2: fd 3 setting TCP_NODELAY > >> debug3: ssh_packet_set_tos: set IP_TOS 0x10 > >> debug2: client_session2_setup: id 0 > >> debug2: channel 0: request pty-req confirm 1 > >> Have you ever experienced such issue ? > >> Any suggestion? > >> Many thanks > >> /Giuseppe > > — > Slawek Kaplonski > Senior software engineer > Red Hat > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomas.bredar at gmail.com Mon Jul 15 14:33:50 2019 From: tomas.bredar at gmail.com (=?UTF-8?B?VG9tw6HFoSBCcmVkw6Fy?=) Date: Mon, 15 Jul 2019 16:33:50 +0200 Subject: [tripleo][cinder][netapp] In-Reply-To: References: Message-ID: Hi Alan, Yes, this is something I was able to achieve. Sorry I think I didn't express myself clearly. My question was how to correctly create multiple configuration files? po 15. 7. 2019 o 15:56 Alan Bishop napísal(a): > > On Mon, Jul 15, 2019 at 4:11 AM Tomáš Bredár > wrote: > >> Hi Alan! >> Thanks for the pointers. For now I'm going with a single backend with two >> NFS shares, so I'll use the tripleo templates for netapp. >> For the future, could you point me to the right template / puppet-cinder >> code which can create multiple nfs share files for me? Or should I create >> my own puppet manifest? >> > > Hi Tomas, > > The puppet-cinder code that renders the shares config file is [1], and the > data comes from puppet-tripleo [2]. This is puppet hiera data, and the > value is bound to the CindeNetappNfsShares tripleo parameter [3]. > > So you should be able to deploy a single backend that accesses multiple > shares by adding something like this to your tripleo deployment. > > parameter_defaults: > CinderNetappNfsShares: 'host_1:/path/to/share_1,host_2:/path/to/share_2' > > [1] > https://opendev.org/openstack/puppet-cinder/src/branch/stable/queens/manifests/backend/netapp.pp#L280 > [2] > https://opendev.org/openstack/puppet-tripleo/src/branch/stable/queens/manifests/profile/base/cinder/volume/netapp.pp#L38 > [3] > https://opendev.org/openstack/tripleo-heat-templates/src/branch/stable/queens/puppet/services/cinder-backend-netapp.yaml#L137 > > Alan > > Thanks again. >> >> Tomas >> >> pi 12. 7. 2019 o 16:11 Alan Bishop napísal(a): >> >>> >>> On Fri, Jul 12, 2019 at 6:09 AM Tomáš Bredár >>> wrote: >>> >>>> Hi Emilien! >>>> >>>> Thanks for your help. Yes with this I am able to define multiple >>>> stanzas in cinder.conf. However netapp driver needs a .conf file with the >>>> nfs shares listed in it. Defining multiple configuration files with nfs >>>> share details in each is not possible with the manual you've sent nor with >>>> the templates in my first email. >>>> >>> >>> Hi Tomas, >>> >>> When deploying a single backend, the tripleo template takes care of >>> generating the nfs shares file (actually, puppet-cinder generates the file, >>> but it's triggered by tripleo). But when you use the custom backend method >>> that Emilien pointed you to use, then you are responsible for supplying all >>> the pieces for the backend(s) to function correctly. This means you will >>> need to generate the nfs shares file on the host (controller), and then >>> bind mount the file using CinderVolumeOptVolumes so that the shares file on >>> the host is visible to the cinder-volume process running in a container. >>> >>> I'm wondering if it's possible to define a second backend by creating >>>> another service, for example "OS::TripleO::Services::CinderBackendNetApp2" ? >>>> >>> >>> Sorry, this won't work. TripleO will trying to deploy two completely >>> separate instances of the cinder-volume service, but the two deployments >>> will step all over each other. There has been a long standing goal of >>> enhancing tripleo so that it can deploy multiple instances of a cinder >>> backend, but it's a complex task that will require non-trivial changes to >>> tripleo. >>> >>> Alan >>> >>> Tomas >>>> >>>> št 11. 7. 2019 o 14:35 Emilien Macchi napísal(a): >>>> >>>>> On Thu, Jul 11, 2019 at 7:32 AM Tomáš Bredár >>>>> wrote: >>>>> >>>>>> Hi community, >>>>>> >>>>>> I'm trying to define multiple NetApp storage backends via Tripleo >>>>>> installer. >>>>>> According to [1] the puppet manifest supports multiple backends. >>>>>> The current templates [2] [3] support only single backend. >>>>>> Does anyone know how to define multiple netapp backends in the >>>>>> tripleo-heat environment files / templates? >>>>>> >>>>> >>>>> We don't support that via the templates that you linked, however if >>>>> you follow this manual you should be able to configure multiple NetApp >>>>> backends: >>>>> >>>>> https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/cinder_custom_backend.html >>>>> >>>>> Let us know how it worked! >>>>> -- >>>>> Emilien Macchi >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Mon Jul 15 16:18:57 2019 From: smooney at redhat.com (Sean Mooney) Date: Mon, 15 Jul 2019 17:18:57 +0100 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: References: <01AA2BA1-3EEC-4142-BCDB-F9861707102E@redhat.com> Message-ID: <9f46833c423d5e87a19e93ad77f7999e44cc5268.camel@redhat.com> On Mon, 2019-07-15 at 16:29 +0200, Giuseppe Sannino wrote: > Hi! > first of all, thanks for the fast replies. I do appreciate that. > > I did some more test trying to figure out the issue. > - Set UseDNS to "no" in sshd_config => Issue persists > - Installed and configured Telnet => Telnet login is slow as well > > From the "top" or "auth.log"nothing specific popped up. I can sshd taking > some cpu for a short while but nothing more than that. > > Once logged in the VM is not too slow. CLI doesn't get stuck or similar. > One thing worthwhile to mention, it seems like the writing throughput on > the disk is a bit slow: 67MB/s wrt around 318MB/s of another VM running on > a "datacenter" Openstack installation. unless you see iowait in the guest its likely not related to the disk speed. you might be able to improve the disk performace by changeing the chache mode but unless you are seeing io wait that is just an optimisation to try later. when you are logged into the vm have you tried ssh again via localhost to determin if the long login time is related to the network or the vm. if its related to the network it will be fast over localhost if its related to the vm, e.g. because of disk, cpu load, memory load or ssh server configuration then the local ssh will be slow. > > The Cinder Volume docker is running on the Compute Host and Cinder is using > the filesystem as backend. > > BR > /Giuseppe > > > > > > > > > > > > > > > > > > > > On Fri, 12 Jul 2019 at 17:41, Slawek Kaplonski wrote: > > > Hi, > > > > I suspect some problems with names resolving. Can You check if You have > > also such delay when doing e.g. “sudo” commands after You ssh to the > > instance? > > > > > On 12 Jul 2019, at 16:23, Brian Haley wrote: > > > > > > > > > > > > On 7/12/19 9:57 AM, Giuseppe Sannino wrote: > > > > Hi community, > > > > I need your help ,tips, advices. > > > > *> Environment <* > > > > I have deployed Openstack "Stein" using the latest kolla-ansible on the > > > > following deployment topology: > > > > 1) OS Controller running as VM on a "cloud" location > > > > 2) OS Compute running on a baremetal server remotely (wrt OS > > > > Controller) location > > > > 3) Network node running on the Compute host > > > > As per the above info, Controller and compute run on two different > > > > networks. > > > > Kolla-Ansible is not really designed for such scenario but after > > > > manipulating the globals.yml and the inventory files (basically I had to > > move node specific network settings from the globals to the inventory > > file), eventually the deployment works fine. > > > > *> Problem <* > > > > I have no specific issue working with this deployment except the > > > > following: > > > > "SSH connection to the VM is quite slow". > > > > It takes around 20 seconds for me to log into the VM (Ubuntu, CentOS, > > > > whatever). > > > > > > But once logged-in things are OK? For example, an scp stalls the same > > > > way, but the transfer is fast? > > > > > > > *> Observations <* > > > > * Except for the slowness during the SSH login, I don't have any > > > > further specific issue working with this envirorment > > > > * With the Network on the Compute I can turn the OS controller off > > > > with no impact on the VM. Still the connection is slow > > > > * I tried different type of images (Ubuntu, CentOS, Windows) always > > > > with the same result. > > > > * SSH connection is slow even if I try to login into the VM within the > > > > IP Namespace > > > > From the ssh -vvv, I can see that the authentication gets stuck here: > > > > debug1: Authentication succeeded (publickey). > > > > Authenticated to ***** > > > > debug1: channel 0: new [client-session] > > > > debug3: ssh_session2_open: channel_new: 0 > > > > debug2: channel 0: send open > > > > debug3: send packet: type 90 > > > > debug1: Requesting no-more-sessions at openssh.com > > > no-more-sessions at openssh.com> > > > > debug3: send packet: type 80 > > > > debug1: Entering interactive session. > > > > debug1: pledge: network > > > > > > > > > 10 to 15 seconds later > > > > > > What is sshd doing at this time? Have you tried enabling debug or > > > > running tcpdump when a new connection is attempted? At first glance I'd > > say it's a DNS issue since it eventually succeeds, the logs would help to > > point in a direction. > > > > > > -Brian > > > > > > > > > > debug3: receive packet: type 80 > > > > debug1: client_input_global_request: rtype hostkeys-00 at openssh.com > > > > want_reply 0 > > > > debug3: receive packet: type 91 > > > > debug2: callback start > > > > debug2: fd 3 setting TCP_NODELAY > > > > debug3: ssh_packet_set_tos: set IP_TOS 0x10 > > > > debug2: client_session2_setup: id 0 > > > > debug2: channel 0: request pty-req confirm 1 > > > > Have you ever experienced such issue ? > > > > Any suggestion? > > > > Many thanks > > > > /Giuseppe > > > > — > > Slawek Kaplonski > > Senior software engineer > > Red Hat > > > > From fungi at yuggoth.org Mon Jul 15 16:29:36 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 15 Jul 2019 16:29:36 +0000 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: References: <01AA2BA1-3EEC-4142-BCDB-F9861707102E@redhat.com> Message-ID: <20190715162936.qowsv7aq2j6bqriq@yuggoth.org> On 2019-07-15 16:29:47 +0200 (+0200), Giuseppe Sannino wrote: [...] > Once logged in the VM is not too slow. CLI doesn't get stuck or similar. > One thing worthwhile to mention, it seems like the writing throughput on > the disk is a bit slow: 67MB/s wrt around 318MB/s of another VM running on > a "datacenter" Openstack installation. [...] Have you checked dmesg in the guest instance to see if there is any I/O problem reported by the kernel? The login process will block on updating /var/log/wtmp or similar, so if writes to whatever backing store that lives on are delayed, that can explain the symptom. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From km.giuseppesannino at gmail.com Mon Jul 15 16:34:36 2019 From: km.giuseppesannino at gmail.com (Giuseppe Sannino) Date: Mon, 15 Jul 2019 18:34:36 +0200 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: <9f46833c423d5e87a19e93ad77f7999e44cc5268.camel@redhat.com> References: <01AA2BA1-3EEC-4142-BCDB-F9861707102E@redhat.com> <9f46833c423d5e87a19e93ad77f7999e44cc5268.camel@redhat.com> Message-ID: Hi Sean, the ssh to localhost is slow as well. "telnet localhost" is also slow. /Giuseppe On Mon, 15 Jul 2019 at 18:18, Sean Mooney wrote: > On Mon, 2019-07-15 at 16:29 +0200, Giuseppe Sannino wrote: > > Hi! > > first of all, thanks for the fast replies. I do appreciate that. > > > > I did some more test trying to figure out the issue. > > - Set UseDNS to "no" in sshd_config => Issue persists > > - Installed and configured Telnet => Telnet login is slow as well > > > > From the "top" or "auth.log"nothing specific popped up. I can sshd taking > > some cpu for a short while but nothing more than that. > > > > Once logged in the VM is not too slow. CLI doesn't get stuck or similar. > > One thing worthwhile to mention, it seems like the writing throughput on > > the disk is a bit slow: 67MB/s wrt around 318MB/s of another VM running > on > > a "datacenter" Openstack installation. > unless you see iowait in the guest its likely not related to the disk > speed. > you might be able to improve the disk performace by changeing the chache > mode > but unless you are seeing io wait that is just an optimisation to try > later. > > when you are logged into the vm have you tried ssh again via localhost to > determin if the long login time is related to the network or the vm. > > if its related to the network it will be fast over localhost > if its related to the vm, e.g. because of disk, cpu load, memory load or > ssh server configuration > then the local ssh will be slow. > > > > > The Cinder Volume docker is running on the Compute Host and Cinder is > using > > the filesystem as backend. > > > > BR > > /Giuseppe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 12 Jul 2019 at 17:41, Slawek Kaplonski > wrote: > > > > > Hi, > > > > > > I suspect some problems with names resolving. Can You check if You have > > > also such delay when doing e.g. “sudo” commands after You ssh to the > > > instance? > > > > > > > On 12 Jul 2019, at 16:23, Brian Haley wrote: > > > > > > > > > > > > > > > > On 7/12/19 9:57 AM, Giuseppe Sannino wrote: > > > > > Hi community, > > > > > I need your help ,tips, advices. > > > > > *> Environment <* > > > > > I have deployed Openstack "Stein" using the latest kolla-ansible > on the > > > > > > following deployment topology: > > > > > 1) OS Controller running as VM on a "cloud" location > > > > > 2) OS Compute running on a baremetal server remotely (wrt OS > > > > > > Controller) location > > > > > 3) Network node running on the Compute host > > > > > As per the above info, Controller and compute run on two different > > > > > > networks. > > > > > Kolla-Ansible is not really designed for such scenario but after > > > > > > manipulating the globals.yml and the inventory files (basically I had > to > > > move node specific network settings from the globals to the inventory > > > file), eventually the deployment works fine. > > > > > *> Problem <* > > > > > I have no specific issue working with this deployment except the > > > > > > following: > > > > > "SSH connection to the VM is quite slow". > > > > > It takes around 20 seconds for me to log into the VM (Ubuntu, > CentOS, > > > > > > whatever). > > > > > > > > But once logged-in things are OK? For example, an scp stalls the > same > > > > > > way, but the transfer is fast? > > > > > > > > > *> Observations <* > > > > > * Except for the slowness during the SSH login, I don't have any > > > > > further specific issue working with this envirorment > > > > > * With the Network on the Compute I can turn the OS controller off > > > > > with no impact on the VM. Still the connection is slow > > > > > * I tried different type of images (Ubuntu, CentOS, Windows) > always > > > > > with the same result. > > > > > * SSH connection is slow even if I try to login into the VM > within the > > > > > IP Namespace > > > > > From the ssh -vvv, I can see that the authentication gets stuck > here: > > > > > debug1: Authentication succeeded (publickey). > > > > > Authenticated to ***** > > > > > debug1: channel 0: new [client-session] > > > > > debug3: ssh_session2_open: channel_new: 0 > > > > > debug2: channel 0: send open > > > > > debug3: send packet: type 90 > > > > > debug1: Requesting no-more-sessions at openssh.com > > > > > no-more-sessions at openssh.com> > > > > > debug3: send packet: type 80 > > > > > debug1: Entering interactive session. > > > > > debug1: pledge: network > > > > > > > > > > 10 to 15 seconds later > > > > > > > > What is sshd doing at this time? Have you tried enabling debug or > > > > > > running tcpdump when a new connection is attempted? At first glance > I'd > > > say it's a DNS issue since it eventually succeeds, the logs would help > to > > > point in a direction. > > > > > > > > -Brian > > > > > > > > > > > > > debug3: receive packet: type 80 > > > > > debug1: client_input_global_request: rtype hostkeys-00 at openssh.com > > > > > > want_reply 0 > > > > > debug3: receive packet: type 91 > > > > > debug2: callback start > > > > > debug2: fd 3 setting TCP_NODELAY > > > > > debug3: ssh_packet_set_tos: set IP_TOS 0x10 > > > > > debug2: client_session2_setup: id 0 > > > > > debug2: channel 0: request pty-req confirm 1 > > > > > Have you ever experienced such issue ? > > > > > Any suggestion? > > > > > Many thanks > > > > > /Giuseppe > > > > > > — > > > Slawek Kaplonski > > > Senior software engineer > > > Red Hat > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From km.giuseppesannino at gmail.com Mon Jul 15 16:43:58 2019 From: km.giuseppesannino at gmail.com (Giuseppe Sannino) Date: Mon, 15 Jul 2019 18:43:58 +0200 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: <20190715162936.qowsv7aq2j6bqriq@yuggoth.org> References: <01AA2BA1-3EEC-4142-BCDB-F9861707102E@redhat.com> <20190715162936.qowsv7aq2j6bqriq@yuggoth.org> Message-ID: Ciao Jeremy, dmesg reports no error on the guest. syslog and auth.log look clean as well. /G On Mon, 15 Jul 2019 at 18:30, Jeremy Stanley wrote: > On 2019-07-15 16:29:47 +0200 (+0200), Giuseppe Sannino wrote: > [...] > > Once logged in the VM is not too slow. CLI doesn't get stuck or similar. > > One thing worthwhile to mention, it seems like the writing throughput on > > the disk is a bit slow: 67MB/s wrt around 318MB/s of another VM running > on > > a "datacenter" Openstack installation. > [...] > > Have you checked dmesg in the guest instance to see if there is any > I/O problem reported by the kernel? The login process will block on > updating /var/log/wtmp or similar, so if writes to whatever backing > store that lives on are delayed, that can explain the symptom. > -- > Jeremy Stanley > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aschultz at redhat.com Mon Jul 15 16:45:59 2019 From: aschultz at redhat.com (Alex Schultz) Date: Mon, 15 Jul 2019 10:45:59 -0600 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: References: <01AA2BA1-3EEC-4142-BCDB-F9861707102E@redhat.com> <9f46833c423d5e87a19e93ad77f7999e44cc5268.camel@redhat.com> Message-ID: On Mon, Jul 15, 2019 at 10:40 AM Giuseppe Sannino < km.giuseppesannino at gmail.com> wrote: > Hi Sean, > the ssh to localhost is slow as well. > "telnet localhost" is also slow. > > Are you having dns issues? Historically if you have UseDNS set to true and your dns servers are bad it can just be slow to connect as it tries to do the reverse lookup. > /Giuseppe > > On Mon, 15 Jul 2019 at 18:18, Sean Mooney wrote: > >> On Mon, 2019-07-15 at 16:29 +0200, Giuseppe Sannino wrote: >> > Hi! >> > first of all, thanks for the fast replies. I do appreciate that. >> > >> > I did some more test trying to figure out the issue. >> > - Set UseDNS to "no" in sshd_config => Issue persists >> > - Installed and configured Telnet => Telnet login is slow as well >> > >> > From the "top" or "auth.log"nothing specific popped up. I can sshd >> taking >> > some cpu for a short while but nothing more than that. >> > >> > Once logged in the VM is not too slow. CLI doesn't get stuck or similar. >> > One thing worthwhile to mention, it seems like the writing throughput on >> > the disk is a bit slow: 67MB/s wrt around 318MB/s of another VM >> running on >> > a "datacenter" Openstack installation. >> unless you see iowait in the guest its likely not related to the disk >> speed. >> you might be able to improve the disk performace by changeing the chache >> mode >> but unless you are seeing io wait that is just an optimisation to try >> later. >> >> when you are logged into the vm have you tried ssh again via localhost to >> determin if the long login time is related to the network or the vm. >> >> if its related to the network it will be fast over localhost >> if its related to the vm, e.g. because of disk, cpu load, memory load or >> ssh server configuration >> then the local ssh will be slow. >> >> > >> > The Cinder Volume docker is running on the Compute Host and Cinder is >> using >> > the filesystem as backend. >> > >> > BR >> > /Giuseppe >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > On Fri, 12 Jul 2019 at 17:41, Slawek Kaplonski >> wrote: >> > >> > > Hi, >> > > >> > > I suspect some problems with names resolving. Can You check if You >> have >> > > also such delay when doing e.g. “sudo” commands after You ssh to the >> > > instance? >> > > >> > > > On 12 Jul 2019, at 16:23, Brian Haley wrote: >> > > > >> > > > >> > > > >> > > > On 7/12/19 9:57 AM, Giuseppe Sannino wrote: >> > > > > Hi community, >> > > > > I need your help ,tips, advices. >> > > > > *> Environment <* >> > > > > I have deployed Openstack "Stein" using the latest kolla-ansible >> on the >> > > >> > > following deployment topology: >> > > > > 1) OS Controller running as VM on a "cloud" location >> > > > > 2) OS Compute running on a baremetal server remotely (wrt OS >> > > >> > > Controller) location >> > > > > 3) Network node running on the Compute host >> > > > > As per the above info, Controller and compute run on two different >> > > >> > > networks. >> > > > > Kolla-Ansible is not really designed for such scenario but after >> > > >> > > manipulating the globals.yml and the inventory files (basically I had >> to >> > > move node specific network settings from the globals to the inventory >> > > file), eventually the deployment works fine. >> > > > > *> Problem <* >> > > > > I have no specific issue working with this deployment except the >> > > >> > > following: >> > > > > "SSH connection to the VM is quite slow". >> > > > > It takes around 20 seconds for me to log into the VM (Ubuntu, >> CentOS, >> > > >> > > whatever). >> > > > >> > > > But once logged-in things are OK? For example, an scp stalls the >> same >> > > >> > > way, but the transfer is fast? >> > > > >> > > > > *> Observations <* >> > > > > * Except for the slowness during the SSH login, I don't have any >> > > > > further specific issue working with this envirorment >> > > > > * With the Network on the Compute I can turn the OS controller >> off >> > > > > with no impact on the VM. Still the connection is slow >> > > > > * I tried different type of images (Ubuntu, CentOS, Windows) >> always >> > > > > with the same result. >> > > > > * SSH connection is slow even if I try to login into the VM >> within the >> > > > > IP Namespace >> > > > > From the ssh -vvv, I can see that the authentication gets stuck >> here: >> > > > > debug1: Authentication succeeded (publickey). >> > > > > Authenticated to ***** >> > > > > debug1: channel 0: new [client-session] >> > > > > debug3: ssh_session2_open: channel_new: 0 >> > > > > debug2: channel 0: send open >> > > > > debug3: send packet: type 90 >> > > > > debug1: Requesting no-more-sessions at openssh.com > > > >> > > no-more-sessions at openssh.com> >> > > > > debug3: send packet: type 80 >> > > > > debug1: Entering interactive session. >> > > > > debug1: pledge: network >> > > > > > > > > > 10 to 15 seconds later >> > > > >> > > > What is sshd doing at this time? Have you tried enabling debug or >> > > >> > > running tcpdump when a new connection is attempted? At first glance >> I'd >> > > say it's a DNS issue since it eventually succeeds, the logs would >> help to >> > > point in a direction. >> > > > >> > > > -Brian >> > > > >> > > > >> > > > > debug3: receive packet: type 80 >> > > > > debug1: client_input_global_request: rtype >> hostkeys-00 at openssh.com >> > > >> > > want_reply 0 >> > > > > debug3: receive packet: type 91 >> > > > > debug2: callback start >> > > > > debug2: fd 3 setting TCP_NODELAY >> > > > > debug3: ssh_packet_set_tos: set IP_TOS 0x10 >> > > > > debug2: client_session2_setup: id 0 >> > > > > debug2: channel 0: request pty-req confirm 1 >> > > > > Have you ever experienced such issue ? >> > > > > Any suggestion? >> > > > > Many thanks >> > > > > /Giuseppe >> > > >> > > — >> > > Slawek Kaplonski >> > > Senior software engineer >> > > Red Hat >> > > >> > > >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From km.giuseppesannino at gmail.com Mon Jul 15 17:05:13 2019 From: km.giuseppesannino at gmail.com (Giuseppe Sannino) Date: Mon, 15 Jul 2019 19:05:13 +0200 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: References: <01AA2BA1-3EEC-4142-BCDB-F9861707102E@redhat.com> <9f46833c423d5e87a19e93ad77f7999e44cc5268.camel@redhat.com> Message-ID: Hi Alex, yeah, it was the first suspect also based on the various research on internet. I currently have the "useDNS" set to no but still the issue persists. /G On Mon, 15 Jul 2019 at 18:46, Alex Schultz wrote: > > On Mon, Jul 15, 2019 at 10:40 AM Giuseppe Sannino < > km.giuseppesannino at gmail.com> wrote: > >> Hi Sean, >> the ssh to localhost is slow as well. >> "telnet localhost" is also slow. >> >> > Are you having dns issues? Historically if you have UseDNS set to true and > your dns servers are bad it can just be slow to connect as it tries to do > the reverse lookup. > > >> /Giuseppe >> >> On Mon, 15 Jul 2019 at 18:18, Sean Mooney wrote: >> >>> On Mon, 2019-07-15 at 16:29 +0200, Giuseppe Sannino wrote: >>> > Hi! >>> > first of all, thanks for the fast replies. I do appreciate that. >>> > >>> > I did some more test trying to figure out the issue. >>> > - Set UseDNS to "no" in sshd_config => Issue persists >>> > - Installed and configured Telnet => Telnet login is slow as well >>> > >>> > From the "top" or "auth.log"nothing specific popped up. I can sshd >>> taking >>> > some cpu for a short while but nothing more than that. >>> > >>> > Once logged in the VM is not too slow. CLI doesn't get stuck or >>> similar. >>> > One thing worthwhile to mention, it seems like the writing throughput >>> on >>> > the disk is a bit slow: 67MB/s wrt around 318MB/s of another VM >>> running on >>> > a "datacenter" Openstack installation. >>> unless you see iowait in the guest its likely not related to the disk >>> speed. >>> you might be able to improve the disk performace by changeing the chache >>> mode >>> but unless you are seeing io wait that is just an optimisation to try >>> later. >>> >>> when you are logged into the vm have you tried ssh again via localhost to >>> determin if the long login time is related to the network or the vm. >>> >>> if its related to the network it will be fast over localhost >>> if its related to the vm, e.g. because of disk, cpu load, memory load or >>> ssh server configuration >>> then the local ssh will be slow. >>> >>> > >>> > The Cinder Volume docker is running on the Compute Host and Cinder is >>> using >>> > the filesystem as backend. >>> > >>> > BR >>> > /Giuseppe >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > On Fri, 12 Jul 2019 at 17:41, Slawek Kaplonski >>> wrote: >>> > >>> > > Hi, >>> > > >>> > > I suspect some problems with names resolving. Can You check if You >>> have >>> > > also such delay when doing e.g. “sudo” commands after You ssh to the >>> > > instance? >>> > > >>> > > > On 12 Jul 2019, at 16:23, Brian Haley >>> wrote: >>> > > > >>> > > > >>> > > > >>> > > > On 7/12/19 9:57 AM, Giuseppe Sannino wrote: >>> > > > > Hi community, >>> > > > > I need your help ,tips, advices. >>> > > > > *> Environment <* >>> > > > > I have deployed Openstack "Stein" using the latest kolla-ansible >>> on the >>> > > >>> > > following deployment topology: >>> > > > > 1) OS Controller running as VM on a "cloud" location >>> > > > > 2) OS Compute running on a baremetal server remotely (wrt OS >>> > > >>> > > Controller) location >>> > > > > 3) Network node running on the Compute host >>> > > > > As per the above info, Controller and compute run on two >>> different >>> > > >>> > > networks. >>> > > > > Kolla-Ansible is not really designed for such scenario but after >>> > > >>> > > manipulating the globals.yml and the inventory files (basically I >>> had to >>> > > move node specific network settings from the globals to the inventory >>> > > file), eventually the deployment works fine. >>> > > > > *> Problem <* >>> > > > > I have no specific issue working with this deployment except the >>> > > >>> > > following: >>> > > > > "SSH connection to the VM is quite slow". >>> > > > > It takes around 20 seconds for me to log into the VM (Ubuntu, >>> CentOS, >>> > > >>> > > whatever). >>> > > > >>> > > > But once logged-in things are OK? For example, an scp stalls the >>> same >>> > > >>> > > way, but the transfer is fast? >>> > > > >>> > > > > *> Observations <* >>> > > > > * Except for the slowness during the SSH login, I don't have any >>> > > > > further specific issue working with this envirorment >>> > > > > * With the Network on the Compute I can turn the OS controller >>> off >>> > > > > with no impact on the VM. Still the connection is slow >>> > > > > * I tried different type of images (Ubuntu, CentOS, Windows) >>> always >>> > > > > with the same result. >>> > > > > * SSH connection is slow even if I try to login into the VM >>> within the >>> > > > > IP Namespace >>> > > > > From the ssh -vvv, I can see that the authentication gets stuck >>> here: >>> > > > > debug1: Authentication succeeded (publickey). >>> > > > > Authenticated to ***** >>> > > > > debug1: channel 0: new [client-session] >>> > > > > debug3: ssh_session2_open: channel_new: 0 >>> > > > > debug2: channel 0: send open >>> > > > > debug3: send packet: type 90 >>> > > > > debug1: Requesting no-more-sessions at openssh.com >> > > >>> > > no-more-sessions at openssh.com> >>> > > > > debug3: send packet: type 80 >>> > > > > debug1: Entering interactive session. >>> > > > > debug1: pledge: network >>> > > > > > > > > > 10 to 15 seconds later >>> > > > >>> > > > What is sshd doing at this time? Have you tried enabling debug or >>> > > >>> > > running tcpdump when a new connection is attempted? At first glance >>> I'd >>> > > say it's a DNS issue since it eventually succeeds, the logs would >>> help to >>> > > point in a direction. >>> > > > >>> > > > -Brian >>> > > > >>> > > > >>> > > > > debug3: receive packet: type 80 >>> > > > > debug1: client_input_global_request: rtype >>> hostkeys-00 at openssh.com >>> > > >>> > > want_reply 0 >>> > > > > debug3: receive packet: type 91 >>> > > > > debug2: callback start >>> > > > > debug2: fd 3 setting TCP_NODELAY >>> > > > > debug3: ssh_packet_set_tos: set IP_TOS 0x10 >>> > > > > debug2: client_session2_setup: id 0 >>> > > > > debug2: channel 0: request pty-req confirm 1 >>> > > > > Have you ever experienced such issue ? >>> > > > > Any suggestion? >>> > > > > Many thanks >>> > > > > /Giuseppe >>> > > >>> > > — >>> > > Slawek Kaplonski >>> > > Senior software engineer >>> > > Red Hat >>> > > >>> > > >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Mon Jul 15 17:26:52 2019 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Mon, 15 Jul 2019 19:26:52 +0200 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: References: <01AA2BA1-3EEC-4142-BCDB-F9861707102E@redhat.com> <9f46833c423d5e87a19e93ad77f7999e44cc5268.camel@redhat.com> Message-ID: For completeness - always also ensure that 'GSSAPIAuthentication' is set to 'no' because in default config it might require DNS lookups too. (Obviously you can run GSSAPIAuthentication and avoid DNS lookups by configuring GSSAPI appropriately ;-) ). Kind regards, Radek pon., 15 lip 2019 o 19:14 Giuseppe Sannino napisał(a): > Hi Alex, > yeah, it was the first suspect also based on the various research on > internet. > I currently have the "useDNS" set to no but still the issue persists. > > /G > > On Mon, 15 Jul 2019 at 18:46, Alex Schultz wrote: > >> >> On Mon, Jul 15, 2019 at 10:40 AM Giuseppe Sannino < >> km.giuseppesannino at gmail.com> wrote: >> >>> Hi Sean, >>> the ssh to localhost is slow as well. >>> "telnet localhost" is also slow. >>> >>> >> Are you having dns issues? Historically if you have UseDNS set to true >> and your dns servers are bad it can just be slow to connect as it tries to >> do the reverse lookup. >> >> >>> /Giuseppe >>> >>> On Mon, 15 Jul 2019 at 18:18, Sean Mooney wrote: >>> >>>> On Mon, 2019-07-15 at 16:29 +0200, Giuseppe Sannino wrote: >>>> > Hi! >>>> > first of all, thanks for the fast replies. I do appreciate that. >>>> > >>>> > I did some more test trying to figure out the issue. >>>> > - Set UseDNS to "no" in sshd_config => Issue persists >>>> > - Installed and configured Telnet => Telnet login is slow as well >>>> > >>>> > From the "top" or "auth.log"nothing specific popped up. I can sshd >>>> taking >>>> > some cpu for a short while but nothing more than that. >>>> > >>>> > Once logged in the VM is not too slow. CLI doesn't get stuck or >>>> similar. >>>> > One thing worthwhile to mention, it seems like the writing throughput >>>> on >>>> > the disk is a bit slow: 67MB/s wrt around 318MB/s of another VM >>>> running on >>>> > a "datacenter" Openstack installation. >>>> unless you see iowait in the guest its likely not related to the disk >>>> speed. >>>> you might be able to improve the disk performace by changeing the >>>> chache mode >>>> but unless you are seeing io wait that is just an optimisation to try >>>> later. >>>> >>>> when you are logged into the vm have you tried ssh again via localhost >>>> to >>>> determin if the long login time is related to the network or the vm. >>>> >>>> if its related to the network it will be fast over localhost >>>> if its related to the vm, e.g. because of disk, cpu load, memory load >>>> or ssh server configuration >>>> then the local ssh will be slow. >>>> >>>> > >>>> > The Cinder Volume docker is running on the Compute Host and Cinder is >>>> using >>>> > the filesystem as backend. >>>> > >>>> > BR >>>> > /Giuseppe >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > On Fri, 12 Jul 2019 at 17:41, Slawek Kaplonski >>>> wrote: >>>> > >>>> > > Hi, >>>> > > >>>> > > I suspect some problems with names resolving. Can You check if You >>>> have >>>> > > also such delay when doing e.g. “sudo” commands after You ssh to the >>>> > > instance? >>>> > > >>>> > > > On 12 Jul 2019, at 16:23, Brian Haley >>>> wrote: >>>> > > > >>>> > > > >>>> > > > >>>> > > > On 7/12/19 9:57 AM, Giuseppe Sannino wrote: >>>> > > > > Hi community, >>>> > > > > I need your help ,tips, advices. >>>> > > > > *> Environment <* >>>> > > > > I have deployed Openstack "Stein" using the latest >>>> kolla-ansible on the >>>> > > >>>> > > following deployment topology: >>>> > > > > 1) OS Controller running as VM on a "cloud" location >>>> > > > > 2) OS Compute running on a baremetal server remotely (wrt OS >>>> > > >>>> > > Controller) location >>>> > > > > 3) Network node running on the Compute host >>>> > > > > As per the above info, Controller and compute run on two >>>> different >>>> > > >>>> > > networks. >>>> > > > > Kolla-Ansible is not really designed for such scenario but after >>>> > > >>>> > > manipulating the globals.yml and the inventory files (basically I >>>> had to >>>> > > move node specific network settings from the globals to the >>>> inventory >>>> > > file), eventually the deployment works fine. >>>> > > > > *> Problem <* >>>> > > > > I have no specific issue working with this deployment except the >>>> > > >>>> > > following: >>>> > > > > "SSH connection to the VM is quite slow". >>>> > > > > It takes around 20 seconds for me to log into the VM (Ubuntu, >>>> CentOS, >>>> > > >>>> > > whatever). >>>> > > > >>>> > > > But once logged-in things are OK? For example, an scp stalls the >>>> same >>>> > > >>>> > > way, but the transfer is fast? >>>> > > > >>>> > > > > *> Observations <* >>>> > > > > * Except for the slowness during the SSH login, I don't have >>>> any >>>> > > > > further specific issue working with this envirorment >>>> > > > > * With the Network on the Compute I can turn the OS controller >>>> off >>>> > > > > with no impact on the VM. Still the connection is slow >>>> > > > > * I tried different type of images (Ubuntu, CentOS, Windows) >>>> always >>>> > > > > with the same result. >>>> > > > > * SSH connection is slow even if I try to login into the VM >>>> within the >>>> > > > > IP Namespace >>>> > > > > From the ssh -vvv, I can see that the authentication gets stuck >>>> here: >>>> > > > > debug1: Authentication succeeded (publickey). >>>> > > > > Authenticated to ***** >>>> > > > > debug1: channel 0: new [client-session] >>>> > > > > debug3: ssh_session2_open: channel_new: 0 >>>> > > > > debug2: channel 0: send open >>>> > > > > debug3: send packet: type 90 >>>> > > > > debug1: Requesting no-more-sessions at openssh.com >>> > > >>>> > > no-more-sessions at openssh.com> >>>> > > > > debug3: send packet: type 80 >>>> > > > > debug1: Entering interactive session. >>>> > > > > debug1: pledge: network >>>> > > > > > > > > > 10 to 15 seconds later >>>> > > > >>>> > > > What is sshd doing at this time? Have you tried enabling debug or >>>> > > >>>> > > running tcpdump when a new connection is attempted? At first >>>> glance I'd >>>> > > say it's a DNS issue since it eventually succeeds, the logs would >>>> help to >>>> > > point in a direction. >>>> > > > >>>> > > > -Brian >>>> > > > >>>> > > > >>>> > > > > debug3: receive packet: type 80 >>>> > > > > debug1: client_input_global_request: rtype >>>> hostkeys-00 at openssh.com >>>> > > >>>> > > want_reply 0 >>>> > > > > debug3: receive packet: type 91 >>>> > > > > debug2: callback start >>>> > > > > debug2: fd 3 setting TCP_NODELAY >>>> > > > > debug3: ssh_packet_set_tos: set IP_TOS 0x10 >>>> > > > > debug2: client_session2_setup: id 0 >>>> > > > > debug2: channel 0: request pty-req confirm 1 >>>> > > > > Have you ever experienced such issue ? >>>> > > > > Any suggestion? >>>> > > > > Many thanks >>>> > > > > /Giuseppe >>>> > > >>>> > > — >>>> > > Slawek Kaplonski >>>> > > Senior software engineer >>>> > > Red Hat >>>> > > >>>> > > >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From amy at demarco.com Mon Jul 15 17:52:04 2019 From: amy at demarco.com (Amy Marrich) Date: Mon, 15 Jul 2019 12:52:04 -0500 Subject: [Diversity] Grace Hopper Lead Co-Mentor(s) needed Message-ID: OpenStack has participated in Open Source Day at the Grace Hopper Conference for the last 5 years and this year it is in Orlando on October 3rd. Due to unforeseen circumstances we are one lead mentor short to participate this year and are looking for someone willing to help lead the session. The Anita Borg Institute will provide a ticket for the entire conference, the downside is you or your company will be responsible for your travel, lodging and food. This year CityNetwork is supplying the VM infrastructure for us to use. The session submitted is as follows: Participants will sign up for the necessary accounts needed to contribute to OpenStack and install Git and Gerrit. They will configure their systems and learn how to file bugs and review code within the Open Dev infrastructure. Due to time and resource requirements the groups will be given a virtual machine running Devstack in which to create their own theme for the Horizon Dashboard. The group will decide on which humanitarian effort they wish to support but suggestions will be provided for those groups that can not decide. The group will also work as a team to locate the necessary documentation on the OpenStack site if they have not done so in preparation. The selected Project Manager will lead these discussions as well as making sure efforts are staying on track to meet the deadline of the demonstrations. The Developers in the group will work to locate the necessary files and directories on the VM in order to complete their task. They will be in charge of creating/changing the necessary files to create a new theme utilizing SCSS and restarting any services needed to implement. The Graphic Designer if present will create any necessary graphics for the theme and will either scp them themselves to the virtual machine or provide to the developer. If there is no designated graphic designer the PM and devs will divide this duty up. After the themes are designed relevant files will be scped back down to the participants machines and they will be able to commit their code for review in a sandbox environment for other groups to review. If you are interested in participating please let me know by July 20th as we need to send additional information to the OSD folks on July 21st. Any questions please reach out on the lists, IRC, or privately! Thanks, Amy (spotz) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsneddon at redhat.com Mon Jul 15 18:25:34 2019 From: dsneddon at redhat.com (Dan Sneddon) Date: Mon, 15 Jul 2019 11:25:34 -0700 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> Message-ID: On Mon, Jul 15, 2019 at 2:13 AM Harald Jensås wrote: > On Sat, 2019-07-13 at 16:19 -0400, James Slagle wrote: > > On Fri, Jul 12, 2019 at 3:59 PM Harald Jensås > > wrote: > > > I've said this before, but I think we should turn this nova-less > > > around. Now with nova-less we create a bunch of servers, and write > > > up > > > the parameters file to use the deployed-server approach. > > > Effectively we > > > still neet to have the resource group in heat making a server > > > resource > > > for every server. Creating the fake server resource is fast, > > > because > > > Heat does'nt call Nova,Ironic to create any resources. But the > > > stack is > > > equally big, with a stack for every node. i.e not N=1. > > > > > > What you are doing here, is essentially to say we don't create a > > > resource group that then creates N number of role stacks, one for > > > each > > > overcloud node. You are creating a single generic "server" > > > definition > > > per Role. So we drop the resource group and create > > > OS::Triple::{{Role}}.Server 1-time (once). To me it's backwards to > > > push > > > a large struct with properties for N=many nodes into the creation > > > of > > > that stack. > > > > I'm not entirely following what you're saying is backwards. What I've > > proposed is that we *don't* have any node specific data in the stack. > > It sounds like you're saying the way we do it today is backwards. > > > > What I mean to say is that I think the way we are integrating nova-less > by first deploying the servers, to then provide the data to Heat to > create the resource groups as we do today becomes backwards when your > work on N=1 is introduced. > > > > It's correct that what's been proposed with metalsmith currently > > still > > requires the full ResourceGroup with a member for each node. With the > > template changes I'm proposing, that wouldn't be required, so we > > could > > actually do the Heat stack first, then metalsmith. > > > > Yes, this is what I think we should do. Especially if your changes here > removes the resource group entirely. It makes more sense to create the > stack, and once that is created we can do deployment, scaling etc > without updating the stack again. > > > > Currently the puppet/role-role.yaml creates all the network ports > > > etc. > > > As you only want to create it once, it instead could simply output > > > the > > > UUID of the networks+subnets. These are identical for all servers > > > in > > > the role. So we end up with a small heat stack. > > > > > > Once the stack is created we could use that generic "server" role > > > data > > > to feed into something (ansible?, python?, mistral?) that calls > > > metalsmith to build the servers, then create ports for each server > > > in > > > neutron, one port for each network+subnet defined in the role. Then > > > feed that output into the json (hieradata) that is pushed to each > > > node > > > and used during service configuration, all the things we need to > > > configure network interfaces, /etc/hosts and so on. We need a way > > > to > > > keep track of which ports belong to wich node, but I guess > > > something > > > simple like using the node's ironic UUID in either the name, > > > description or tag field of the neutron port will work. There is > > > also > > > the extra filed in Ironic which is json type, so we could place a > > > map > > > of network->port_uuid in there as well. > > > > It won't matter whether we do baremetal provisioning before or after > > the Heat stack. Heat won't care, as it won't have any expectation to > > create any servers or that they are already created. We can define > > where we end up calling the metalsmith piece as it should be > > independent of the Heat stack if we make these template changes. > > > > This is true. But, in your previous mail in this thread you wrote: > > """ > Other points: > > - Baremetal provisioning and port creation are presently handled by > Heat. With the ongoing efforts to migrate baremetal provisioning out > of Heat (nova-less deploy), I think these efforts are very > complimentary. Eventually, we get to a point where Heat is not > actually creating any other OpenStack API resources. For now, the > patches only work when using pre-provisioned nodes. > """ > > IMO "baremetal provision and port creation" fit together. (I read the > above statement so as well.) Currently nova-less creates the ctlplane > port and provision the baremetal node. If we want to do both baremetal > provisioning and port creation togheter (I think this makes sense), we > have to do it after the stack has created the networks. > > What I envision is to have one method that creates all the ports, > ctlplane + composable networks in a unified way. Today these are > created differently, the ctlplane port is part of the server resource > (or metalsmith in nova-less case) and the other ports are created by > heat. > This is my main question about this proposal. When TripleO was in its infancy, there wasn't a mechanism to create Neutron ports separately from the server, so we created a Nova Server resource that specified which network the port was on (originally there was only one port created, now we create additional ports in Neutron). This can be seen in the puppet/-role.yaml file, for example: resources: Controller: type: OS::TripleO::ControllerServer deletion_policy: {get_param: ServerDeletionPolicy} metadata: os-collect-config: command: {get_param: ConfigCommand} splay: {get_param: ConfigCollectSplay} properties: [...] networks: - if: - ctlplane_fixed_ip_set - network: ctlplane subnet: {get_param: ControllerControlPlaneSubnet} fixed_ip: yaql: expression: $.data.where(not isEmpty($)).first() data: - get_param: [ControllerIPs, 'ctlplane', {get_param: NodeIndex}] - network: ctlplane subnet: {get_param: ControllerControlPlaneSubnet} This has the side-effect that the ports are created by Nova calling Neutron rather than by Heat calling Neutron for port creation. We have maintained this mechanism even in the latest versions of THT for backwards compatibility. This would all be easier if we were creating the Neutron ctlplane port and then assigning it to the server, but that breaks backwards-compatibility. How would the creation of the ctlplane port be handled in this proposal? If metalsmith is creating the ctlplane port, do we still need a separate Server resource for every node? If so, I imagine it would have a much smaller stack than what we currently create for each server. If not, would metalsmith create a port on the ctlplane as part of the provisioning steps, and then pass this port back? We still need to be able to support fixed IPs for ctlplane ports, so we need to be able to pass a specific IP to metalsmith. > > > I think the creation of the actual Networks and Subnets can be left > > > in > > > heat, it's typically 5-6 networks and 5-6 subnets so it's not a lot > > > of > > > resources. Even in a large DCN deployment having 50-100 subnets per > > > network or even 50-100 networks I think this is'nt a problem. > > > > Agreed, I'm not specifically proposing we move those pieces at this > > time. > > > > +1 > > > > > -- Dan Sneddon | Senior Principal Software Engineer dsneddon at redhat.com | redhat.com/cloud dsneddon:irc | @dxs:twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From doug at doughellmann.com Mon Jul 15 18:37:05 2019 From: doug at doughellmann.com (Doug Hellmann) Date: Mon, 15 Jul 2019 14:37:05 -0400 Subject: [dev][release][qa] patrole stable/stein is created by mistake ? In-Reply-To: <0a85c82c-f312-a8a9-34af-f5b812c8f2b3@openstack.org> References: <16beb436c21.c9d98154147017.455401003873410397@ghanshyammann.com> <16beb463457.1193bd7a4147042.5135658897011450644@ghanshyammann.com> <20190713203828.GA29711@sm-workstation> <16beddf697c.fd5e3e64150304.4626321901014847129@ghanshyammann.com> <0a85c82c-f312-a8a9-34af-f5b812c8f2b3@openstack.org> Message-ID: <717CD206-8A3A-438F-A52B-22396CFA9BAD@doughellmann.com> > On Jul 15, 2019, at 6:30 AM, Thierry Carrez wrote: > > Ghanshyam Mann wrote: >> ---- On Sun, 14 Jul 2019 05:38:28 +0900 Sean McGinnis wrote ---- >> > On Sat, Jul 13, 2019 at 09:19:35PM +0900, Ghanshyam Mann wrote: >> > > ---- On Sat, 13 Jul 2019 21:16:32 +0900 Ghanshyam Mann wrote ---- >> > > > Hi Release team, >> > > > >> > > > Today I noticed while doing patrole review that stable/stain has been created for patrole which is wrong. >> > > > Patrole is branchless[1] and I remember I have not requested the stable branch while releasing the patrole. >> > > > >> > > > Is it created by mistakenly ? or intentional? >> > > >> > > I found the patch which created this - https://review.opendev.org/#/c/650173/1. >> > > >> > > Can we revert that but that patch has more changes? >> > > >> > > -gmann >> > > >> > >> > We can't really just revert it. If you would like to change this, please update >> > the patrole release type to tempest-plugin or move it to be an independent >> > release deliverable if it is not actually cycle based. >> > >> > You can also remove the branching information with that change, then after that >> > is gone and there isn't a risk of recreating it, the infra team may be able to >> > assist in deleting the existing branch. >> Release model for patrole is 'cycle-with-intermediary' which is right, we do not need >> to change that. We just need to remove the stable/stein and branch information which >> was added for the only stein. I will push the change. >> Can we have a tag which clearly says the branchless and no-stable branch nature for deliverables? >> 'tempest-plugins' was introduced for a different purpose. new tag can be applicable for other >> deliverables also (current or in future). That can help to avoid these errrors. > > Creating a new release model or a new deliverable type sounds a bit overkill. > > I think the simpler would be to add a new value to the existing "stable-branch-type" key. Like "stable-branch-type: none" and then set that value for all tempest plugins, tempest itself and patrole. > > See https://review.opendev.org/670808 as a strawman. > > -- > Thierry Carrez (ttx) If the project isn’t creating stable branches each cycle, how is it cycle-based? What’s wrong with moving it to use the independent release model, which was created for cases like this? Doug -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnaser at vexxhost.com Mon Jul 15 21:46:13 2019 From: mnaser at vexxhost.com (Mohammed Naser) Date: Mon, 15 Jul 2019 17:46:13 -0400 Subject: [tc] Weekly update Message-ID: Hi everyone, This is the weekly update for what happened inside the OpenStack TC, you can get more information by checking for changes in the openstack/governance repository. # Retired projects - release-schedule-generator (from releases) - tripleo-ansible-roles (from tripleo) - docs-specs (from docs) # New projects - octavia-diskimage-retrofit (under charms) - kayobe (under kolla) # General changes - The "U" release will be based in the China region as per the upcoming summit. - The documentation team is starting a transition to a SIG - Few changes to the PDF goal proposal patches alongside storyboard links - Typo fixes (we can't spell liaison apparently!) - Jeremy Stanley has volunteered to become TC liaison for the image encryption popup team - Each project has been assigned 2 TC liaisons to provide assistance within our community. Thanks for tuning in :) Regards, Mohammed From sbaker at redhat.com Mon Jul 15 22:26:10 2019 From: sbaker at redhat.com (Steve Baker) Date: Tue, 16 Jul 2019 10:26:10 +1200 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> Message-ID: <6bdebb51-0b3c-2888-a691-720bd5ac039a@redhat.com> On 15/07/19 9:12 PM, Harald Jensås wrote: > On Sat, 2019-07-13 at 16:19 -0400, James Slagle wrote: >> On Fri, Jul 12, 2019 at 3:59 PM Harald Jensås >> wrote: >>> I've said this before, but I think we should turn this nova-less >>> around. Now with nova-less we create a bunch of servers, and write >>> up >>> the parameters file to use the deployed-server approach. >>> Effectively we >>> still neet to have the resource group in heat making a server >>> resource >>> for every server. Creating the fake server resource is fast, >>> because >>> Heat does'nt call Nova,Ironic to create any resources. But the >>> stack is >>> equally big, with a stack for every node. i.e not N=1. >>> >>> What you are doing here, is essentially to say we don't create a >>> resource group that then creates N number of role stacks, one for >>> each >>> overcloud node. You are creating a single generic "server" >>> definition >>> per Role. So we drop the resource group and create >>> OS::Triple::{{Role}}.Server 1-time (once). To me it's backwards to >>> push >>> a large struct with properties for N=many nodes into the creation >>> of >>> that stack. >> I'm not entirely following what you're saying is backwards. What I've >> proposed is that we *don't* have any node specific data in the stack. >> It sounds like you're saying the way we do it today is backwards. >> > What I mean to say is that I think the way we are integrating nova-less > by first deploying the servers, to then provide the data to Heat to > create the resource groups as we do today becomes backwards when your > work on N=1 is introduced. > > >> It's correct that what's been proposed with metalsmith currently >> still >> requires the full ResourceGroup with a member for each node. With the >> template changes I'm proposing, that wouldn't be required, so we >> could >> actually do the Heat stack first, then metalsmith. >> > Yes, this is what I think we should do. Especially if your changes here > removes the resource group entirely. It makes more sense to create the > stack, and once that is created we can do deployment, scaling etc > without updating the stack again. I think this is something we can move towards after James has finished this work. It would probably mean deprecating "openstack overcloud node provision" and providing some other way of running the baremetal provisioning in isolation after the heat stack operation, like an equivalent to "openstack overcloud deploy --config-download-only" From mark.kirkwood at catalyst.net.nz Mon Jul 15 23:21:44 2019 From: mark.kirkwood at catalyst.net.nz (Mark Kirkwood) Date: Tue, 16 Jul 2019 11:21:44 +1200 Subject: [devstack] [trove] Enabling other datastores, plugin variable design less than tidy Message-ID: Hi, I'm doing some work with the Trove Postgres datastore. So the first thing I did was attempt to get Devstack to set up a stack with Postgres instead of Mysql. Reading the docs, it seemed that all I needed to do was this: $ cat local.conf [[local|localrc]] ADMIN_PASSWORD=password DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD TROVE_DATASTORE_TYPE=postgresql TROVE_DATASTORE_VERSION=9.6 TROVE_DATASTORE_PACKAGE=postgresql-9.6 enable_plugin trove https://opendev.org/openstack/trove Wrong! After watching the plugin try to build a Mysql guest image of version 9.6 (!), I realized that more reading of the plugin source was required. So iteration 2 (or maybe it was 3...lol), of my local.conf is: [[local|localrc]] ADMIN_PASSWORD=password DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD TROVE_DATASTORE_TYPE=postgresql TROVE_DATASTORE_VERSION=9.6 TROVE_DATASTORE_PACKAGE=postgresql-9.6 SERVICE_TYPE=$TROVE_DATASTORE_TYPE DATASTORE_VERSION=$TROVE_DATASTORE_VERSION VM=/opt/stack/images/ubuntu_postgresql/ubuntu_postgresql enable_plugin trove https://opendev.org/openstack/trove This works. However, it seems like those last 3 variable substitutions should not be required. i.e: - SERVICE_TYPE should not exist (we should use TROVE_DATASTORE_TYPE) - DATASTORE_VERSION should not exist (we should use TROVE_DATASTORE_VERSION) - VM should be constructed out of DISTRO and TROVE_DATASTORE_TYPE Thoughts? regards Mark From abishop at redhat.com Mon Jul 15 23:44:58 2019 From: abishop at redhat.com (Alan Bishop) Date: Mon, 15 Jul 2019 16:44:58 -0700 Subject: [tripleo][cinder][netapp] In-Reply-To: References: Message-ID: On Mon, Jul 15, 2019 at 7:34 AM Tomáš Bredár wrote: > Hi Alan, > Yes, this is something I was able to achieve. Sorry I think I didn't > express myself clearly. My question was how to correctly create multiple > configuration files? > Hi Tomas, Sorry, but I'm not aware of any existing template or puppet module you could use to create multiple share config files. However, if you're willing to get your hands dirty and spend time experimenting, there are a couple of tripleo facilities you might be able to use to get the job done. One approach would be to use an "extra config" hook [1], in which you could execute a script that generates the contents of your share config files (e.g. a bash script that echos data to a file). Tripleo's extraconfig/services directory [2] might provide some ideas. For example, you could create a template that defines a new, minimal tripleo "service" whose host_prep_tasks (essentially a list of ansible tasks) create the files. [1] http://tripleo.org/install/advanced_deployment/extra_config.html [2] https://github.com/openstack/tripleo-heat-templates/tree/stable/queens/extraconfig/services Unfortunately, it's been quite a while since I dabbled in these areas, so I can't point you to a concise example. Maybe a tripleo expert can provide better guidance. Alan > po 15. 7. 2019 o 15:56 Alan Bishop napísal(a): > >> >> On Mon, Jul 15, 2019 at 4:11 AM Tomáš Bredár >> wrote: >> >>> Hi Alan! >>> Thanks for the pointers. For now I'm going with a single backend with >>> two NFS shares, so I'll use the tripleo templates for netapp. >>> For the future, could you point me to the right template / puppet-cinder >>> code which can create multiple nfs share files for me? Or should I create >>> my own puppet manifest? >>> >> >> Hi Tomas, >> >> The puppet-cinder code that renders the shares config file is [1], and >> the data comes from puppet-tripleo [2]. This is puppet hiera data, and the >> value is bound to the CindeNetappNfsShares tripleo parameter [3]. >> >> So you should be able to deploy a single backend that accesses multiple >> shares by adding something like this to your tripleo deployment. >> >> parameter_defaults: >> CinderNetappNfsShares: 'host_1:/path/to/share_1,host_2:/path/to/share_2' >> >> [1] >> https://opendev.org/openstack/puppet-cinder/src/branch/stable/queens/manifests/backend/netapp.pp#L280 >> [2] >> https://opendev.org/openstack/puppet-tripleo/src/branch/stable/queens/manifests/profile/base/cinder/volume/netapp.pp#L38 >> [3] >> https://opendev.org/openstack/tripleo-heat-templates/src/branch/stable/queens/puppet/services/cinder-backend-netapp.yaml#L137 >> >> Alan >> >> Thanks again. >>> >>> Tomas >>> >>> pi 12. 7. 2019 o 16:11 Alan Bishop napísal(a): >>> >>>> >>>> On Fri, Jul 12, 2019 at 6:09 AM Tomáš Bredár >>>> wrote: >>>> >>>>> Hi Emilien! >>>>> >>>>> Thanks for your help. Yes with this I am able to define multiple >>>>> stanzas in cinder.conf. However netapp driver needs a .conf file with the >>>>> nfs shares listed in it. Defining multiple configuration files with nfs >>>>> share details in each is not possible with the manual you've sent nor with >>>>> the templates in my first email. >>>>> >>>> >>>> Hi Tomas, >>>> >>>> When deploying a single backend, the tripleo template takes care of >>>> generating the nfs shares file (actually, puppet-cinder generates the file, >>>> but it's triggered by tripleo). But when you use the custom backend method >>>> that Emilien pointed you to use, then you are responsible for supplying all >>>> the pieces for the backend(s) to function correctly. This means you will >>>> need to generate the nfs shares file on the host (controller), and then >>>> bind mount the file using CinderVolumeOptVolumes so that the shares file on >>>> the host is visible to the cinder-volume process running in a container. >>>> >>>> I'm wondering if it's possible to define a second backend by creating >>>>> another service, for example "OS::TripleO::Services::CinderBackendNetApp2" ? >>>>> >>>> >>>> Sorry, this won't work. TripleO will trying to deploy two completely >>>> separate instances of the cinder-volume service, but the two deployments >>>> will step all over each other. There has been a long standing goal of >>>> enhancing tripleo so that it can deploy multiple instances of a cinder >>>> backend, but it's a complex task that will require non-trivial changes to >>>> tripleo. >>>> >>>> Alan >>>> >>>> Tomas >>>>> >>>>> št 11. 7. 2019 o 14:35 Emilien Macchi napísal(a): >>>>> >>>>>> On Thu, Jul 11, 2019 at 7:32 AM Tomáš Bredár >>>>>> wrote: >>>>>> >>>>>>> Hi community, >>>>>>> >>>>>>> I'm trying to define multiple NetApp storage backends via Tripleo >>>>>>> installer. >>>>>>> According to [1] the puppet manifest supports multiple backends. >>>>>>> The current templates [2] [3] support only single backend. >>>>>>> Does anyone know how to define multiple netapp backends in the >>>>>>> tripleo-heat environment files / templates? >>>>>>> >>>>>> >>>>>> We don't support that via the templates that you linked, however if >>>>>> you follow this manual you should be able to configure multiple NetApp >>>>>> backends: >>>>>> >>>>>> https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/cinder_custom_backend.html >>>>>> >>>>>> Let us know how it worked! >>>>>> -- >>>>>> Emilien Macchi >>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Tue Jul 16 03:42:00 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 16 Jul 2019 12:42:00 +0900 Subject: [dev][release][qa] patrole stable/stein is created by mistake ? In-Reply-To: <0a85c82c-f312-a8a9-34af-f5b812c8f2b3@openstack.org> References: <16beb436c21.c9d98154147017.455401003873410397@ghanshyammann.com> <16beb463457.1193bd7a4147042.5135658897011450644@ghanshyammann.com> <20190713203828.GA29711@sm-workstation> <16beddf697c.fd5e3e64150304.4626321901014847129@ghanshyammann.com> <0a85c82c-f312-a8a9-34af-f5b812c8f2b3@openstack.org> Message-ID: <16bf8df6c0c.e19126b3193710.3862555704510892753@ghanshyammann.com> ---- On Mon, 15 Jul 2019 19:30:26 +0900 Thierry Carrez wrote ---- > Ghanshyam Mann wrote: > > ---- On Sun, 14 Jul 2019 05:38:28 +0900 Sean McGinnis wrote ---- > > > On Sat, Jul 13, 2019 at 09:19:35PM +0900, Ghanshyam Mann wrote: > > > > ---- On Sat, 13 Jul 2019 21:16:32 +0900 Ghanshyam Mann wrote ---- > > > > > Hi Release team, > > > > > > > > > > Today I noticed while doing patrole review that stable/stain has been created for patrole which is wrong. > > > > > Patrole is branchless[1] and I remember I have not requested the stable branch while releasing the patrole. > > > > > > > > > > Is it created by mistakenly ? or intentional? > > > > > > > > I found the patch which created this - https://review.opendev.org/#/c/650173/1. > > > > > > > > Can we revert that but that patch has more changes? > > > > > > > > -gmann > > > > > > > > > > We can't really just revert it. If you would like to change this, please update > > > the patrole release type to tempest-plugin or move it to be an independent > > > release deliverable if it is not actually cycle based. > > > > > > You can also remove the branching information with that change, then after that > > > is gone and there isn't a risk of recreating it, the infra team may be able to > > > assist in deleting the existing branch. > > > > Release model for patrole is 'cycle-with-intermediary' which is right, we do not need > > to change that. We just need to remove the stable/stein and branch information which > > was added for the only stein. I will push the change. > > > > Can we have a tag which clearly says the branchless and no-stable branch nature for deliverables? > > > > 'tempest-plugins' was introduced for a different purpose. new tag can be applicable for other > > deliverables also (current or in future). That can help to avoid these errrors. > > Creating a new release model or a new deliverable type sounds a bit > overkill. > > I think the simpler would be to add a new value to the existing > "stable-branch-type" key. Like "stable-branch-type: none" and then set > that value for all tempest plugins, tempest itself and patrole. Thanks, that will work perfectly in Tempest, and its plugins cases. -gmann > > See https://review.opendev.org/670808 as a strawman. > > -- > Thierry Carrez (ttx) > > From gmann at ghanshyammann.com Tue Jul 16 03:51:45 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 16 Jul 2019 12:51:45 +0900 Subject: [dev][release][qa] patrole stable/stein is created by mistake ? In-Reply-To: <717CD206-8A3A-438F-A52B-22396CFA9BAD@doughellmann.com> References: <16beb436c21.c9d98154147017.455401003873410397@ghanshyammann.com> <16beb463457.1193bd7a4147042.5135658897011450644@ghanshyammann.com> <20190713203828.GA29711@sm-workstation> <16beddf697c.fd5e3e64150304.4626321901014847129@ghanshyammann.com> <0a85c82c-f312-a8a9-34af-f5b812c8f2b3@openstack.org> <717CD206-8A3A-438F-A52B-22396CFA9BAD@doughellmann.com> Message-ID: <16bf8e85983.b41e1d9d193754.93039526410042655@ghanshyammann.com> ---- On Tue, 16 Jul 2019 03:37:05 +0900 Doug Hellmann wrote ---- > > > On Jul 15, 2019, at 6:30 AM, Thierry Carrez wrote: > Ghanshyam Mann wrote: > ---- On Sun, 14 Jul 2019 05:38:28 +0900 Sean McGinnis wrote ---- > > On Sat, Jul 13, 2019 at 09:19:35PM +0900, Ghanshyam Mann wrote: > > > ---- On Sat, 13 Jul 2019 21:16:32 +0900 Ghanshyam Mann wrote ---- > > > > Hi Release team, > > > > > > > > Today I noticed while doing patrole review that stable/stain has been created for patrole which is wrong. > > > > Patrole is branchless[1] and I remember I have not requested the stable branch while releasing the patrole. > > > > > > > > Is it created by mistakenly ? or intentional? > > > > > > I found the patch which created this - https://review.opendev.org/#/c/650173/1. > > > > > > Can we revert that but that patch has more changes? > > > > > > -gmann > > > > > > > We can't really just revert it. If you would like to change this, please update > > the patrole release type to tempest-plugin or move it to be an independent > > release deliverable if it is not actually cycle based. > > > > You can also remove the branching information with that change, then after that > > is gone and there isn't a risk of recreating it, the infra team may be able to > > assist in deleting the existing branch. > Release model for patrole is 'cycle-with-intermediary' which is right, we do not need > to change that. We just need to remove the stable/stein and branch information which > was added for the only stein. I will push the change. > Can we have a tag which clearly says the branchless and no-stable branch nature for deliverables? > 'tempest-plugins' was introduced for a different purpose. new tag can be applicable for other > deliverables also (current or in future). That can help to avoid these errrors. > > Creating a new release model or a new deliverable type sounds a bit overkill. > > I think the simpler would be to add a new value to the existing "stable-branch-type" key. Like "stable-branch-type: none" and then set that value for all tempest plugins, tempest itself and patrole. > > See https://review.opendev.org/670808 as a strawman. > > -- > Thierry Carrez (ttx) > If the project isn’t creating stable branches each cycle, how is it cycle-based? What’s wrong with moving it to use the independent release model, which was created for cases like this? Tempest and all Tempest plugins are "cycle-with-intermediary" because we want a particular release tag per OpenStack release. That is what we decided in ML discussion while deciding the release model for tempest plugins [1]. With independent release mode, we face the issue of not having the compatible versions of all Tempest plugins which user need to run on against their production cloud. [1] http://lists.openstack.org/pipermail/openstack-dev/2018-June/131837.html > Doug > From dtantsur at redhat.com Tue Jul 16 07:15:50 2019 From: dtantsur at redhat.com (Dmitry Tantsur) Date: Tue, 16 Jul 2019 09:15:50 +0200 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: <6bdebb51-0b3c-2888-a691-720bd5ac039a@redhat.com> References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> <6bdebb51-0b3c-2888-a691-720bd5ac039a@redhat.com> Message-ID: <49e44a0f-84c9-195a-99d5-b843e4045454@redhat.com> On 7/16/19 12:26 AM, Steve Baker wrote: > > On 15/07/19 9:12 PM, Harald Jensås wrote: >> On Sat, 2019-07-13 at 16:19 -0400, James Slagle wrote: >>> On Fri, Jul 12, 2019 at 3:59 PM Harald Jensås >>> wrote: >>>> I've said this before, but I think we should turn this nova-less >>>> around. Now with nova-less we create a bunch of servers, and write >>>> up >>>> the parameters file to use the deployed-server approach. >>>> Effectively we >>>> still neet to have the resource group in heat making a server >>>> resource >>>> for every server. Creating the fake server resource is fast, >>>> because >>>> Heat does'nt call Nova,Ironic to create any resources. But the >>>> stack is >>>> equally big, with a stack for every node. i.e not N=1. >>>> >>>> What you are doing here, is essentially to say we don't create a >>>> resource group that then creates N number of role stacks, one for >>>> each >>>> overcloud node. You are creating a single generic "server" >>>> definition >>>> per Role. So we drop the resource group and create >>>> OS::Triple::{{Role}}.Server 1-time (once). To me it's backwards to >>>> push >>>> a large struct with properties for N=many nodes into the creation >>>> of >>>> that stack. >>> I'm not entirely following what you're saying is backwards. What I've >>> proposed is that we *don't* have any node specific data in the stack. >>> It sounds like you're saying the way we do it today is backwards. >>> >> What I mean to say is that I think the way we are integrating nova-less >> by first deploying the servers, to then provide the data to Heat to >> create the resource groups as we do today becomes backwards when your >> work on N=1 is introduced. >> >> >>> It's correct that what's been proposed with metalsmith currently >>> still >>> requires the full ResourceGroup with a member for each node. With the >>> template changes I'm proposing, that wouldn't be required, so we >>> could >>> actually do the Heat stack first, then metalsmith. >>> >> Yes, this is what I think we should do. Especially if your changes here >> removes the resource group entirely. It makes more sense to create the >> stack, and once that is created we can do deployment, scaling etc >> without updating the stack again. > > I think this is something we can move towards after James has finished this > work. It would probably mean deprecating "openstack overcloud node provision" > and providing some other way of running the baremetal provisioning in isolation > after the heat stack operation, like an equivalent to "openstack overcloud > deploy --config-download-only" > > I'm very much against on deprecating "openstack overcloud node provision", it's one of the reasons of this whole effort. I'm equally -2 on making the bare metal provisioning depending on heat in any way for the same reason. Dmitry From rraja at redhat.com Tue Jul 16 07:19:37 2019 From: rraja at redhat.com (Ramana Venkatesh Raja) Date: Tue, 16 Jul 2019 12:49:37 +0530 Subject: [Manila] CephFS deferred deletion In-Reply-To: <20190712131540.3eqvltysfix6eivd@barron.net> References: <20190712131540.3eqvltysfix6eivd@barron.net> Message-ID: Re-sending the email as it didn't get posted in the ML. On Fri, Jul 12, 2019 at 6:45 PM Tom Barron wrote: > > On 12/07/19 13:03 +0000, Jose Castro Leon wrote: > >Dear all, > > > >Lately, one of our clients stored 300k files in a manila cephfs share. > >Then he deleted the share in Manila. This event make the driver > >unresponsive for several hours until all the data was removed in the > >cluster. > > > >We had a quick look at the code in manila [1] and the deletion is done > >first by calling the following api calls in the ceph bindings > >(delete_volume[1] and then purge_volume[2]). The first call moves the > >directory to a volumes_deleted directory. The second call does a > >deletion in depth of all the contents of that directory. > > > >The last operation is the one that trigger the issue. > > > >We had a similar issue in the past in Cinder. There, Arne proposed to > >do a deferred deletion of volumes. I think we could do the same in > >Manila for the cephfs driver. > > > >The idea is to continue to call to the delete_volume. And then inside a > >periodic task in the driver, asynchronously it will get the contents of > >that directory and trigger the purge command. > > > >I can propose the change and contribute with the code, but before going > >to deep I would like to know if there is a reason of having a singleton > >for the volume_client connection. If I compare with cinder code the > >connection is established and closed in each operation with the > >backend. > > > >If you are not the maintainer, could you please point me to he/she? > >I can post it in the mailing list if you prefer > > > >Cheers > >Jose Castro Leon > >CERN Cloud Infrastructure > > > >[1] > >https://github.com/openstack/manila/blob/master/manila/share/drivers/cephfs/driver.py#L260-L267 > > > > > >[2] > >https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L700-L734 > > > > > >[2] > >https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L736-L790 > > > > > >PS: The issue was triggered by one of our clients in kubernetes using > >the Manila CSI driver > > Hi Jose, > > Let's get this fixed since there's a lot of interest in Manila CSI > driver and I think we can expect more batched deletes with it than we > have had historically. The plan is to have manila's CephFS driver use the ceph-mgr's new volumes module, https://github.com/ceph/ceph/blob/master/src/pybind/mgr/volumes/module.py to create/delete manila groups/shares/snapshots, authorize/de-authorize access to the shares. Manila shares, essentially CephFS subdirectories with a specific data layout and quota, are referred to as FS subvolumes, and Ceph filesystems as FS volumes in the ceph-mgr volumes module. The ceph-mgr volumes modules is under active development. The latest Ceph CSI (v1.1.0) release is the first consumer of this module. The Ceph CSI issues CLI calls to the ceph-mgr to manage the lifecycle of the FS subvolumes, https://github.com/ceph/ceph-csi/pull/400 We're implementing the asynchronous purge of FS subvolumes in the ceph-mgr module. The PR is close to being merged, https://github.com/ceph/ceph/pull/28003/ https://github.com/ceph/ceph/pull/28003/commits/483a2141fe8c9a58bc25a544412cdf5b047ad772 http://tracker.ceph.com/issues/40036 Issuing the `ceph fs subvolume rm` command in the Ceph CSI driver (and later in the manila driver) will move the FS subvolume to a trash directory, whose contents will be asynchronously purged by a set of worker threads. > > I've copied Ramana Raja and Patrick Donnelly since they will be able > to answer your question about the singleton volume_client connection > more authoritatively than I can. Currently, in the mgr-volumes module we establish and close connection to a FS volume (a Ceph filesystem) for each FS subvolume (CephFS subdirectory within the filesystem) operation, https://github.com/ceph/ceph/pull/28082/commits/8d29816f0f3db6c7d287bbb7469db77c9de701d1#diff-cfd3b6f517caccc18f7f066395e8a4bdR174 Instead, we want to maintain a connection to a FS volume and perform operations on its subvolumes, until the FS volume is deleted. This would reduce the time taken to perform subvolume operations, important in CSI work loads (and in OpenStack workloads?). The code is in review, https://github.com/ceph/ceph/pull/28003/commits/5c41e949af9acabd612b0644de0603e374b4b42a Thanks, Ramana > > Thanks for volunteering to propose a review to deal with this issue! > > -- Tom Barron > From dsneddon at redhat.com Tue Jul 16 07:34:56 2019 From: dsneddon at redhat.com (Dan Sneddon) Date: Tue, 16 Jul 2019 00:34:56 -0700 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: <49e44a0f-84c9-195a-99d5-b843e4045454@redhat.com> References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> <6bdebb51-0b3c-2888-a691-720bd5ac039a@redhat.com> <49e44a0f-84c9-195a-99d5-b843e4045454@redhat.com> Message-ID: On Tue, Jul 16, 2019 at 12:19 AM Dmitry Tantsur wrote: > On 7/16/19 12:26 AM, Steve Baker wrote: > > > > On 15/07/19 9:12 PM, Harald Jensås wrote: > >> On Sat, 2019-07-13 at 16:19 -0400, James Slagle wrote: > >>> On Fri, Jul 12, 2019 at 3:59 PM Harald Jensås > >>> wrote: > >>>> I've said this before, but I think we should turn this nova-less > >>>> around. Now with nova-less we create a bunch of servers, and write > >>>> up > >>>> the parameters file to use the deployed-server approach. > >>>> Effectively we > >>>> still neet to have the resource group in heat making a server > >>>> resource > >>>> for every server. Creating the fake server resource is fast, > >>>> because > >>>> Heat does'nt call Nova,Ironic to create any resources. But the > >>>> stack is > >>>> equally big, with a stack for every node. i.e not N=1. > >>>> > >>>> What you are doing here, is essentially to say we don't create a > >>>> resource group that then creates N number of role stacks, one for > >>>> each > >>>> overcloud node. You are creating a single generic "server" > >>>> definition > >>>> per Role. So we drop the resource group and create > >>>> OS::Triple::{{Role}}.Server 1-time (once). To me it's backwards to > >>>> push > >>>> a large struct with properties for N=many nodes into the creation > >>>> of > >>>> that stack. > >>> I'm not entirely following what you're saying is backwards. What I've > >>> proposed is that we *don't* have any node specific data in the stack. > >>> It sounds like you're saying the way we do it today is backwards. > >>> > >> What I mean to say is that I think the way we are integrating nova-less > >> by first deploying the servers, to then provide the data to Heat to > >> create the resource groups as we do today becomes backwards when your > >> work on N=1 is introduced. > >> > >> > >>> It's correct that what's been proposed with metalsmith currently > >>> still > >>> requires the full ResourceGroup with a member for each node. With the > >>> template changes I'm proposing, that wouldn't be required, so we > >>> could > >>> actually do the Heat stack first, then metalsmith. > >>> > >> Yes, this is what I think we should do. Especially if your changes here > >> removes the resource group entirely. It makes more sense to create the > >> stack, and once that is created we can do deployment, scaling etc > >> without updating the stack again. > > > > I think this is something we can move towards after James has finished > this > > work. It would probably mean deprecating "openstack overcloud node > provision" > > and providing some other way of running the baremetal provisioning in > isolation > > after the heat stack operation, like an equivalent to "openstack > overcloud > > deploy --config-download-only" > > > > > > I'm very much against on deprecating "openstack overcloud node provision", > it's > one of the reasons of this whole effort. I'm equally -2 on making the bare > metal > provisioning depending on heat in any way for the same reason. > > Dmitry > > My concerns about network ports boil down to technical debt with Heat. It would be great if we can make the individual nodes completely independent of Heat, and somehow migrate from the old Heat-based definition for upgrades. -- Dan Sneddon -------------- next part -------------- An HTML attachment was scrubbed... URL: From bdobreli at redhat.com Tue Jul 16 09:26:11 2019 From: bdobreli at redhat.com (Bogdan Dobrelya) Date: Tue, 16 Jul 2019 11:26:11 +0200 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> <6bdebb51-0b3c-2888-a691-720bd5ac039a@redhat.com> <49e44a0f-84c9-195a-99d5-b843e4045454@redhat.com> Message-ID: <91b5d9aa-3210-1645-f0c7-adf91b84d007@redhat.com> On 16.07.2019 9:34, Dan Sneddon wrote: > On Tue, Jul 16, 2019 at 12:19 AM Dmitry Tantsur > wrote: > > On 7/16/19 12:26 AM, Steve Baker wrote: > > > > On 15/07/19 9:12 PM, Harald Jensås wrote: > >> On Sat, 2019-07-13 at 16:19 -0400, James Slagle wrote: > >>> On Fri, Jul 12, 2019 at 3:59 PM Harald Jensås > > > >>> wrote: > >>>> I've said this before, but I think we should turn this nova-less > >>>> around. Now with nova-less we create a bunch of servers, and write > >>>> up > >>>> the parameters file to use the deployed-server approach. > >>>> Effectively we > >>>> still neet to have the resource group in heat making a server > >>>> resource > >>>> for every server. Creating the fake server resource is fast, > >>>> because > >>>> Heat does'nt call Nova,Ironic to create any resources. But the > >>>> stack is > >>>> equally big, with a stack for every node. i.e not N=1. > >>>> > >>>> What you are doing here, is essentially to say we don't create a > >>>> resource group that then creates N number of role stacks, one for > >>>> each > >>>> overcloud node. You are creating a single generic "server" > >>>> definition > >>>> per Role. So we drop the resource group and create > >>>> OS::Triple::{{Role}}.Server 1-time (once). To me it's backwards to > >>>> push > >>>> a large struct with properties for N=many nodes into the creation > >>>> of > >>>> that stack. > >>> I'm not entirely following what you're saying is backwards. > What I've > >>> proposed is that we *don't* have any node specific data in the > stack. > >>> It sounds like you're saying the way we do it today is backwards. > >>> > >> What I mean to say is that I think the way we are integrating > nova-less > >> by first deploying the servers, to then provide the data to Heat to > >> create the resource groups as we do today becomes backwards when > your > >> work on N=1 is introduced. > >> > >> > >>> It's correct that what's been proposed with metalsmith currently > >>> still > >>> requires the full ResourceGroup with a member for each node. > With the > >>> template changes I'm proposing, that wouldn't be required, so we > >>> could > >>> actually do the Heat stack first, then metalsmith. > >>> > >> Yes, this is what I think we should do. Especially if your > changes here > >> removes the resource group entirely. It makes more sense to > create the > >> stack, and once that is created we can do deployment, scaling etc > >> without updating the stack again. > > > > I think this is something we can move towards after James has > finished this > > work. It would probably mean deprecating "openstack overcloud > node provision" > > and providing some other way of running the baremetal > provisioning in isolation > > after the heat stack operation, like an equivalent to "openstack > overcloud > > deploy --config-download-only" > > > > > > I'm very much against on deprecating "openstack overcloud node > provision", it's > one of the reasons of this whole effort. I'm equally -2 on making > the bare metal > provisioning depending on heat in any way for the same reason. > > Dmitry > > > My concerns about network ports boil down to technical debt with Heat. > It would be great if we can make the individual nodes completely > independent of Heat, and somehow migrate from the old Heat-based > definition for upgrades. As it was earlier mentioned in the thread, we'll highly likely need some external data store to migrate/upgrade things out of Heat smoothly. That probably should be etcd? I don't think a clever ansible inventory could handle that fully replacing such a data store. > > -- > Dan Sneddon -- Best regards, Bogdan Dobrelya, Irc #bogdando From dharmendra.kushwaha at india.nec.com Tue Jul 16 09:29:34 2019 From: dharmendra.kushwaha at india.nec.com (Dharmendra Kushwaha) Date: Tue, 16 Jul 2019 09:29:34 +0000 Subject: [Tacker] Proposing changes in Tacker core team In-Reply-To: References: Message-ID: Welcome Hiroyuki Jo in Tacker Core-Team. :) Thanks & Regards Dharmendra Kushwaha ________________________________________ From: Dharmendra Kushwaha Sent: Tuesday, July 9, 2019 12:04 PM To: openstack-discuss at lists.openstack.org Subject: [Tacker] Proposing changes in Tacker core team Hello Team, I am proposing below changes in Team: I would like to propose Hiroyuki Jo to join Tacker core team. Hiroyuki Jo have lead multiple valuable features level activities like affinity policy, VDU-healing, and VNF reservation [1] in Rocky & Stein cycle, and made it sure to be completed timely. And currently working on VNF packages [2] and ETSI NFV-SOL specification support [3]. Hiroyuki has a good understanding of NFV and Tacker project, and helping team by providing sensible reviews. I believe it is a good addition in Tacker core team, and Tacker project will benefit from this nomination. On the other hand, I wanted to thank to Bharath Thiruveedula for his great & valuable contribution in the project. He helped a lot to make Tacker better in early days. But now he doesn't seem to be active in project and he decided to step-down from core team. Whenever you will decide to come back to the project, I will be happy to add you in core-team. Core-Team, Please respond with your +1/-1. If no objection, I will do these changes in next week. [1] https://review.opendev.org/#/q/project:openstack/tacker-specs+owner:%22Hiroyuki+Jo+%253Chiroyuki.jo.mt%2540hco.ntt.co.jp%253E%22 [2] https://blueprints.launchpad.net/tacker/+spec/tosca-csar-mgmt-driver [3] https://blueprints.launchpad.net/tacker/+spec/support-etsi-nfv-specs Thanks & Regards Dharmendra Kushwaha From manulachathurika at gmail.com Tue Jul 16 10:01:59 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Tue, 16 Jul 2019 15:31:59 +0530 Subject: ImportError: No module named django.core.wsgi - DevStack - Stable/Stein - Ubuntu 18.04 In-Reply-To: References: Message-ID: Hi All, I have successfully install DevStack in Ubuntu 18.04. But when I'm accessing the dashboard I'm getting 500 error. What I'm getting in horizon_error.log is, 2019-07-15 14:10:43.218296 mod_wsgi (pid=31763): Target WSGI script '/opt/stack/horizon/openstack_dashboard/wsgi.py' cannot be loaded as Python module. 2019-07-15 14:10:43.218323 mod_wsgi (pid=31763): Exception occurred processing WSGI script '/opt/stack/horizon/openstack_dashboard/wsgi.py'. 2019-07-15 14:10:43.218349 Traceback (most recent call last): 2019-07-15 14:10:43.218370 File "/opt/stack/horizon/openstack_dashboard/wsgi.py", line 21, in 2019-07-15 14:10:43.218401 from django.core.wsgi import get_wsgi_application 2019-07-15 14:10:43.218422 ImportError: No module named django.core.wsgi Python Version is : 3.6.8 Django version is : 2.0.5 I tried with uninstalling Djanago and reinstalling it. But it didn't work for me. Can someone help me on how to resole this issue ? Thanks ! -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.sgaravatto at gmail.com Tue Jul 16 10:38:03 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Tue, 16 Jul 2019 12:38:03 +0200 Subject: Problems running db_sync for nova (Ocata --> Pike) Message-ID: Hi We are trying to update our Ocata cloud to Pike (this is the first step: we will go through Ocata --> Pike --> Queens --> Rocky) but we have a problem with nova-manage db sync: [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db sync" nova ERROR: Could not access cell0. Has the nova_api database been created? Has the nova_cell0 database been created? Has "nova-manage api_db sync" been run? Has "nova-manage cell_v2 map_cell0" been run? Is [api_database]/connection set in nova.conf? Is the cell0 database connection URL correct? Error: "Database schema file with version 390 doesn't exist." We have these settings in nova.conf: [database] connection = mysql+pymysql://nova_prod:xyz at 192.168.60.10:6306/nova_prod [api_database] connection = mysql+pymysql:// nova_api_prod:xyz at 192.168.60.10:6306/nova_api_prod I can't see problems accessing the databases. i.e. these commands work: mysql -u nova_api_prod -pxyz -h 192.168.60.10 -P 6306 nova_api_prod mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod_cell0 If I try to rerun nova-manage cell_v2 map_cell0: [root at cld-ctrl-01 ~]# nova-manage cell_v2 map_cell0 Cell0 is already setup [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage api_db version" nova 45 [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db version" nova 362 [root at cld-ctrl-01 ~]# nova-manage cell_v2 list_cells +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ | Name | UUID | Transport URL | Database Connection | +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ | cell0 | 00000000-0000-0000-0000-000000000000 | none:/// | mysql+pymysql://nova_prod:****@ 192.168.60.10:6306/nova_prod_cell0 | | cell1 | 5e42faa0-710b-4967-bb42-fcf53602c96e | rabbit://openstack_prod:****@192.168.60.183:5672 | mysql+pymysql://nova_prod:****@192.168.60.10:6306/nova_prod | +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ Following some doc we also tried to specify the --database_connection / --database-connection argument, but this seems not working Setting nova in debug mode, this is the last entry in the nova-manage log file: 2019-07-16 12:11:42.774 22013 DEBUG migrate.versioning.repository [req-8ca52407-dbe1-4a62-ad6c-16631c2a9a06 - - - - -] Config: OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), ('repository_id', 'nova'), ('version_table', 'migrate_version'), ('required_dbs', '[]')]))]) __init__ /usr/lib/python2.7/site-packages/migrate/versioning/repository.py:83 Any hints ? Thanks, Massimo -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.sgaravatto at gmail.com Tue Jul 16 10:43:01 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Tue, 16 Jul 2019 12:43:01 +0200 Subject: [nova] [ops] Problems running db sync for nova (Ocata --> Pike) Message-ID: Resending with the right tags in the subject line ... Hi We are trying to update our Ocata cloud to Pike (this is the first step: we will go through Ocata --> Pike --> Queens --> Rocky) but we have a problem with nova-manage db sync: [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db sync" nova ERROR: Could not access cell0. Has the nova_api database been created? Has the nova_cell0 database been created? Has "nova-manage api_db sync" been run? Has "nova-manage cell_v2 map_cell0" been run? Is [api_database]/connection set in nova.conf? Is the cell0 database connection URL correct? Error: "Database schema file with version 390 doesn't exist." We have these settings in nova.conf: [database] connection = mysql+pymysql://nova_prod:xyz at 192.168.60.10:6306/nova_prod [api_database] connection = mysql+pymysql:// nova_api_prod:xyz at 192.168.60.10:6306/nova_api_prod I can't see problems accessing the databases. i.e. these commands work: mysql -u nova_api_prod -pxyz -h 192.168.60.10 -P 6306 nova_api_prod mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod_cell0 If I try to rerun nova-manage cell_v2 map_cell0: [root at cld-ctrl-01 ~]# nova-manage cell_v2 map_cell0 Cell0 is already setup [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage api_db version" nova 45 [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db version" nova 362 [root at cld-ctrl-01 ~]# nova-manage cell_v2 list_cells +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ | Name | UUID | Transport URL | Database Connection | +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ | cell0 | 00000000-0000-0000-0000-000000000000 | none:/// | mysql+pymysql://nova_prod:****@ 192.168.60.10:6306/nova_prod_cell0 | | cell1 | 5e42faa0-710b-4967-bb42-fcf53602c96e | rabbit://openstack_prod:****@192.168.60.183:5672 | mysql+pymysql://nova_prod:****@192.168.60.10:6306/nova_prod | +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ Following some doc we also tried to specify the --database_connection / --database-connection argument, but this seems not working Setting nova in debug mode, this is the last entry in the nova-manage log file: 2019-07-16 12:11:42.774 22013 DEBUG migrate.versioning.repository [req-8ca52407-dbe1-4a62-ad6c-16631c2a9a06 - - - - -] Config: OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), ('repository_id', 'nova'), ('version_table', 'migrate_version'), ('required_dbs', '[]')]))]) __init__ /usr/lib/python2.7/site-packages/migrate/versioning/repository.py:83 Any hints ? Thanks, Massimo   OpenStack Discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Tue Jul 16 10:48:39 2019 From: eblock at nde.ag (Eugen Block) Date: Tue, 16 Jul 2019 10:48:39 +0000 Subject: [nova] [ops] Problems running db sync for nova (Ocata --> Pike) In-Reply-To: Message-ID: <20190716104839.Horde.Pe3wSo-woFy0u8rF9gnehyD@webmail.nde.ag> Hi, I think you need to run nova-manage db sync --local_cell in Ocata. I can't quite remember the reason, but you should try it. I had the same problems and this command worked out for me. Regards, Eugen Zitat von Massimo Sgaravatto : > Resending with the right tags in the subject line ... > > > Hi > > We are trying to update our Ocata cloud to Pike (this is the first step: > we will go through Ocata --> Pike --> Queens --> Rocky) but we have a > problem with nova-manage db sync: > > > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db sync" nova > ERROR: Could not access cell0. > Has the nova_api database been created? > Has the nova_cell0 database been created? > Has "nova-manage api_db sync" been run? > Has "nova-manage cell_v2 map_cell0" been run? > Is [api_database]/connection set in nova.conf? > Is the cell0 database connection URL correct? > Error: "Database schema file with version 390 doesn't exist." > > We have these settings in nova.conf: > > [database] > connection = mysql+pymysql://nova_prod:xyz at 192.168.60.10:6306/nova_prod > [api_database] > connection = mysql+pymysql:// > nova_api_prod:xyz at 192.168.60.10:6306/nova_api_prod > > I can't see problems accessing the databases. i.e. these commands work: > > mysql -u nova_api_prod -pxyz -h 192.168.60.10 -P 6306 nova_api_prod > > mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod > > mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod_cell0 > > If I try to rerun nova-manage cell_v2 map_cell0: > > [root at cld-ctrl-01 ~]# nova-manage cell_v2 map_cell0 > Cell0 is already setup > > > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage api_db version" nova > 45 > > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db version" nova > 362 > > [root at cld-ctrl-01 ~]# nova-manage cell_v2 list_cells > +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ > | Name | UUID | Transport > URL | Database Connection > | > +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ > | cell0 | 00000000-0000-0000-0000-000000000000 | > none:/// | mysql+pymysql://nova_prod:****@ > 192.168.60.10:6306/nova_prod_cell0 | > | cell1 | 5e42faa0-710b-4967-bb42-fcf53602c96e | > rabbit://openstack_prod:****@192.168.60.183:5672 | > mysql+pymysql://nova_prod:****@192.168.60.10:6306/nova_prod | > +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ > > Following some doc we also tried to specify the --database_connection / > --database-connection argument, but this seems not working > > > Setting nova in debug mode, this is the last entry in the nova-manage log > file: > > 2019-07-16 12:11:42.774 22013 DEBUG migrate.versioning.repository > [req-8ca52407-dbe1-4a62-ad6c-16631c2a9a06 - - - - -] Config: > OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), > ('repository_id', 'nova'), ('version_table', 'migrate_version'), > ('required_dbs', '[]')]))]) __init__ > /usr/lib/python2.7/site-packages/migrate/versioning/repository.py:83 > > Any hints ? > > Thanks, Massimo >  >  > OpenStack Discuss From ruslanas at lpic.lt Tue Jul 16 10:59:48 2019 From: ruslanas at lpic.lt (=?UTF-8?Q?Ruslanas_G=C5=BEibovskis?=) Date: Tue, 16 Jul 2019 12:59:48 +0200 Subject: ImportError: No module named django.core.wsgi - DevStack - Stable/Stein - Ubuntu 18.04 In-Reply-To: References: Message-ID: I had something similar, try reinstalling python from OSP repo, not common, as it rewrites some python lib file... at least it helped me. On Tue, 16 Jul 2019 at 12:05, Manula Thantriwatte < manulachathurika at gmail.com> wrote: > Hi All, > > I have successfully install DevStack in Ubuntu 18.04. But when I'm > accessing the dashboard I'm getting 500 error. What I'm getting > in horizon_error.log is, > > 2019-07-15 14:10:43.218296 mod_wsgi (pid=31763): Target WSGI script > '/opt/stack/horizon/openstack_dashboard/wsgi.py' cannot be loaded as Python > module. > 2019-07-15 14:10:43.218323 mod_wsgi (pid=31763): Exception occurred > processing WSGI script '/opt/stack/horizon/openstack_dashboard/wsgi.py'. > 2019-07-15 14:10:43.218349 Traceback (most recent call last): > 2019-07-15 14:10:43.218370 File > "/opt/stack/horizon/openstack_dashboard/wsgi.py", line 21, in > 2019-07-15 14:10:43.218401 from django.core.wsgi import > get_wsgi_application > 2019-07-15 14:10:43.218422 ImportError: No module named django.core.wsgi > > Python Version is : 3.6.8 > Django version is : 2.0.5 > > I tried with uninstalling Djanago and reinstalling it. But it didn't work > for me. > > Can someone help me on how to resole this issue ? > > Thanks ! > -- > Regards, > Manula Chathurika Thantriwatte > phone : (+94) 772492511 > email : manulachathurika at gmail.com > Linkedin : *http://lk.linkedin.com/in/manulachathurika > * > blog : http://manulachathurika.blogspot.com/ > > -- Ruslanas Gžibovskis +370 6030 7030 -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.sgaravatto at gmail.com Tue Jul 16 11:05:54 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Tue, 16 Jul 2019 13:05:54 +0200 Subject: [nova] [ops] Problems running db sync for nova (Ocata --> Pike) In-Reply-To: <20190716104839.Horde.Pe3wSo-woFy0u8rF9gnehyD@webmail.nde.ag> References: <20190716104839.Horde.Pe3wSo-woFy0u8rF9gnehyD@webmail.nde.ag> Message-ID: Mmm. Doesn't "--local_cell" mean that cell0 is not updated ? I was already using cell0 in Ocata ... Cheers, Massimo On Tue, Jul 16, 2019 at 12:52 PM Eugen Block wrote: > Hi, > > I think you need to run > > nova-manage db sync --local_cell > > in Ocata. I can't quite remember the reason, but you should try it. > I had the same problems and this command worked out for me. > > Regards, > Eugen > > > > Zitat von Massimo Sgaravatto : > > > Resending with the right tags in the subject line ... > > > > > > Hi > > > > We are trying to update our Ocata cloud to Pike (this is the first step: > > we will go through Ocata --> Pike --> Queens --> Rocky) but we have a > > problem with nova-manage db sync: > > > > > > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db sync" nova > > ERROR: Could not access cell0. > > Has the nova_api database been created? > > Has the nova_cell0 database been created? > > Has "nova-manage api_db sync" been run? > > Has "nova-manage cell_v2 map_cell0" been run? > > Is [api_database]/connection set in nova.conf? > > Is the cell0 database connection URL correct? > > Error: "Database schema file with version 390 doesn't exist." > > > > We have these settings in nova.conf: > > > > [database] > > connection = mysql+pymysql://nova_prod:xyz at 192.168.60.10:6306/nova_prod > > [api_database] > > connection = mysql+pymysql:// > > nova_api_prod:xyz at 192.168.60.10:6306/nova_api_prod > > > > I can't see problems accessing the databases. i.e. these commands work: > > > > mysql -u nova_api_prod -pxyz -h 192.168.60.10 -P 6306 nova_api_prod > > > > mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod > > > > mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod_cell0 > > > > If I try to rerun nova-manage cell_v2 map_cell0: > > > > [root at cld-ctrl-01 ~]# nova-manage cell_v2 map_cell0 > > Cell0 is already setup > > > > > > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage api_db version" nova > > 45 > > > > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db version" nova > > 362 > > > > [root at cld-ctrl-01 ~]# nova-manage cell_v2 list_cells > > > +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ > > | Name | UUID | > Transport > > URL | Database Connection > > | > > > +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ > > | cell0 | 00000000-0000-0000-0000-000000000000 | > > none:/// | mysql+pymysql://nova_prod:****@ > > 192.168.60.10:6306/nova_prod_cell0 | > > | cell1 | 5e42faa0-710b-4967-bb42-fcf53602c96e | > > rabbit://openstack_prod:****@192.168.60.183:5672 | > > mysql+pymysql://nova_prod:****@192.168.60.10:6306/nova_prod | > > > +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ > > > > Following some doc we also tried to specify the --database_connection / > > --database-connection argument, but this seems not working > > > > > > Setting nova in debug mode, this is the last entry in the nova-manage log > > file: > > > > 2019-07-16 12:11:42.774 22013 DEBUG migrate.versioning.repository > > [req-8ca52407-dbe1-4a62-ad6c-16631c2a9a06 - - - - -] Config: > > OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), > > ('repository_id', 'nova'), ('version_table', 'migrate_version'), > > ('required_dbs', '[]')]))]) __init__ > > /usr/lib/python2.7/site-packages/migrate/versioning/repository.py:83 > > > > Any hints ? > > > > Thanks, Massimo > >  > >  > > OpenStack Discuss > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Tue Jul 16 11:11:51 2019 From: eblock at nde.ag (Eugen Block) Date: Tue, 16 Jul 2019 11:11:51 +0000 Subject: [nova] [ops] Problems running db sync for nova (Ocata --> Pike) In-Reply-To: References: <20190716104839.Horde.Pe3wSo-woFy0u8rF9gnehyD@webmail.nde.ag> Message-ID: <20190716111151.Horde.2mXkgZIEbUsVwdTqrDfdbD4@webmail.nde.ag> AFAIK cell0 is the local cell, isn't it? I'm not sure, maybe someone else can shed some light on this. Zitat von Massimo Sgaravatto : > Mmm. > Doesn't "--local_cell" mean that cell0 is not updated ? > I was already using cell0 in Ocata ... > > Cheers, Massimo > > On Tue, Jul 16, 2019 at 12:52 PM Eugen Block wrote: > >> Hi, >> >> I think you need to run >> >> nova-manage db sync --local_cell >> >> in Ocata. I can't quite remember the reason, but you should try it. >> I had the same problems and this command worked out for me. >> >> Regards, >> Eugen >> >> >> >> Zitat von Massimo Sgaravatto : >> >> > Resending with the right tags in the subject line ... >> > >> > >> > Hi >> > >> > We are trying to update our Ocata cloud to Pike (this is the first step: >> > we will go through Ocata --> Pike --> Queens --> Rocky) but we have a >> > problem with nova-manage db sync: >> > >> > >> > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db sync" nova >> > ERROR: Could not access cell0. >> > Has the nova_api database been created? >> > Has the nova_cell0 database been created? >> > Has "nova-manage api_db sync" been run? >> > Has "nova-manage cell_v2 map_cell0" been run? >> > Is [api_database]/connection set in nova.conf? >> > Is the cell0 database connection URL correct? >> > Error: "Database schema file with version 390 doesn't exist." >> > >> > We have these settings in nova.conf: >> > >> > [database] >> > connection = mysql+pymysql://nova_prod:xyz at 192.168.60.10:6306/nova_prod >> > [api_database] >> > connection = mysql+pymysql:// >> > nova_api_prod:xyz at 192.168.60.10:6306/nova_api_prod >> > >> > I can't see problems accessing the databases. i.e. these commands work: >> > >> > mysql -u nova_api_prod -pxyz -h 192.168.60.10 -P 6306 nova_api_prod >> > >> > mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod >> > >> > mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod_cell0 >> > >> > If I try to rerun nova-manage cell_v2 map_cell0: >> > >> > [root at cld-ctrl-01 ~]# nova-manage cell_v2 map_cell0 >> > Cell0 is already setup >> > >> > >> > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage api_db version" nova >> > 45 >> > >> > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db version" nova >> > 362 >> > >> > [root at cld-ctrl-01 ~]# nova-manage cell_v2 list_cells >> > >> +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ >> > | Name | UUID | >> Transport >> > URL | Database Connection >> > | >> > >> +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ >> > | cell0 | 00000000-0000-0000-0000-000000000000 | >> > none:/// | mysql+pymysql://nova_prod:****@ >> > 192.168.60.10:6306/nova_prod_cell0 | >> > | cell1 | 5e42faa0-710b-4967-bb42-fcf53602c96e | >> > rabbit://openstack_prod:****@192.168.60.183:5672 | >> > mysql+pymysql://nova_prod:****@192.168.60.10:6306/nova_prod | >> > >> +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ >> > >> > Following some doc we also tried to specify the --database_connection / >> > --database-connection argument, but this seems not working >> > >> > >> > Setting nova in debug mode, this is the last entry in the nova-manage log >> > file: >> > >> > 2019-07-16 12:11:42.774 22013 DEBUG migrate.versioning.repository >> > [req-8ca52407-dbe1-4a62-ad6c-16631c2a9a06 - - - - -] Config: >> > OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), >> > ('repository_id', 'nova'), ('version_table', 'migrate_version'), >> > ('required_dbs', '[]')]))]) __init__ >> > /usr/lib/python2.7/site-packages/migrate/versioning/repository.py:83 >> > >> > Any hints ? >> > >> > Thanks, Massimo >> >  >> >  >> > OpenStack Discuss >> >> >> >> >> From massimo.sgaravatto at gmail.com Tue Jul 16 11:16:19 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Tue, 16 Jul 2019 13:16:19 +0200 Subject: [nova] [ops] Problems running db sync for nova (Ocata --> Pike) In-Reply-To: <20190716111151.Horde.2mXkgZIEbUsVwdTqrDfdbD4@webmail.nde.ag> References: <20190716104839.Horde.Pe3wSo-woFy0u8rF9gnehyD@webmail.nde.ag> <20190716111151.Horde.2mXkgZIEbUsVwdTqrDfdbD4@webmail.nde.ag> Message-ID: My understanding (e.g. from #comment 1 of this bug: https://bugs.launchpad.net/grenade/+bug/1761775) is that --local-cell updates only the database specified as connection in [database], i.e. not also the cell0 database On Tue, Jul 16, 2019 at 1:12 PM Eugen Block wrote: > AFAIK cell0 is the local cell, isn't it? I'm not sure, maybe someone > else can shed some light on this. > > > Zitat von Massimo Sgaravatto : > > > Mmm. > > Doesn't "--local_cell" mean that cell0 is not updated ? > > I was already using cell0 in Ocata ... > > > > Cheers, Massimo > > > > On Tue, Jul 16, 2019 at 12:52 PM Eugen Block wrote: > > > >> Hi, > >> > >> I think you need to run > >> > >> nova-manage db sync --local_cell > >> > >> in Ocata. I can't quite remember the reason, but you should try it. > >> I had the same problems and this command worked out for me. > >> > >> Regards, > >> Eugen > >> > >> > >> > >> Zitat von Massimo Sgaravatto : > >> > >> > Resending with the right tags in the subject line ... > >> > > >> > > >> > Hi > >> > > >> > We are trying to update our Ocata cloud to Pike (this is the first > step: > >> > we will go through Ocata --> Pike --> Queens --> Rocky) but we have a > >> > problem with nova-manage db sync: > >> > > >> > > >> > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db sync" nova > >> > ERROR: Could not access cell0. > >> > Has the nova_api database been created? > >> > Has the nova_cell0 database been created? > >> > Has "nova-manage api_db sync" been run? > >> > Has "nova-manage cell_v2 map_cell0" been run? > >> > Is [api_database]/connection set in nova.conf? > >> > Is the cell0 database connection URL correct? > >> > Error: "Database schema file with version 390 doesn't exist." > >> > > >> > We have these settings in nova.conf: > >> > > >> > [database] > >> > connection = mysql+pymysql:// > nova_prod:xyz at 192.168.60.10:6306/nova_prod > >> > [api_database] > >> > connection = mysql+pymysql:// > >> > nova_api_prod:xyz at 192.168.60.10:6306/nova_api_prod > >> > > >> > I can't see problems accessing the databases. i.e. these commands > work: > >> > > >> > mysql -u nova_api_prod -pxyz -h 192.168.60.10 -P 6306 nova_api_prod > >> > > >> > mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod > >> > > >> > mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod_cell0 > >> > > >> > If I try to rerun nova-manage cell_v2 map_cell0: > >> > > >> > [root at cld-ctrl-01 ~]# nova-manage cell_v2 map_cell0 > >> > Cell0 is already setup > >> > > >> > > >> > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage api_db version" > nova > >> > 45 > >> > > >> > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db version" nova > >> > 362 > >> > > >> > [root at cld-ctrl-01 ~]# nova-manage cell_v2 list_cells > >> > > >> > +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ > >> > | Name | UUID | > >> Transport > >> > URL | Database Connection > >> > | > >> > > >> > +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ > >> > | cell0 | 00000000-0000-0000-0000-000000000000 | > >> > none:/// | mysql+pymysql://nova_prod:****@ > >> > 192.168.60.10:6306/nova_prod_cell0 | > >> > | cell1 | 5e42faa0-710b-4967-bb42-fcf53602c96e | > >> > rabbit://openstack_prod:****@192.168.60.183:5672 | > >> > mysql+pymysql://nova_prod:****@192.168.60.10:6306/nova_prod | > >> > > >> > +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ > >> > > >> > Following some doc we also tried to specify the --database_connection > / > >> > --database-connection argument, but this seems not working > >> > > >> > > >> > Setting nova in debug mode, this is the last entry in the nova-manage > log > >> > file: > >> > > >> > 2019-07-16 12:11:42.774 22013 DEBUG migrate.versioning.repository > >> > [req-8ca52407-dbe1-4a62-ad6c-16631c2a9a06 - - - - -] Config: > >> > OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), > >> > ('repository_id', 'nova'), ('version_table', 'migrate_version'), > >> > ('required_dbs', '[]')]))]) __init__ > >> > /usr/lib/python2.7/site-packages/migrate/versioning/repository.py:83 > >> > > >> > Any hints ? > >> > > >> > Thanks, Massimo > >> >  > >> >  > >> > OpenStack Discuss > >> > >> > >> > >> > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From witold.bedyk at suse.com Tue Jul 16 11:27:19 2019 From: witold.bedyk at suse.com (Witek Bedyk) Date: Tue, 16 Jul 2019 13:27:19 +0200 Subject: [monasca] Virtual Midcycle Meeting scheduling In-Reply-To: <40121ffb-3306-ef6b-b2ba-9d75e2a1e130@suse.de> References: <40121ffb-3306-ef6b-b2ba-9d75e2a1e130@suse.de> Message-ID: <92004787-2fc1-650f-91d0-be1fe28c97bd@suse.de> Hello everyone, our Virtual Midcycle Meeting will take place: Wed, Jul 24, 2019 2:00 PM - 4:00 PM UTC Here the URL for joining the meeting: https://global.gotomeeting.com/join/402046285 More details are available in the etherpad below. Everyone is welcome to join. Cheers Witek On 7/9/19 1:45 PM, Witek Bedyk wrote: > Hello, > > as discussed in the last team meeting, we will hold a virtual Midcycle > Meeting. The goal is to sync on the progress of the development and > update the stories if needed. I plan with 2 hours meeting. Please select > the times which work best for you: > > https://doodle.com/poll/zszfxakcbfm6sdha > > Please fill in the topics you would like to discuss or update on in the > etherpad: > > https://etherpad.openstack.org/p/monasca-train-midcycle > > Thanks > Witek > > -- Witek Bedyk Cloud Developer SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nürnberg Tel: +49-911-74053-0; Fax: +49-911-7417755; https://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) From hjensas at redhat.com Tue Jul 16 11:28:59 2019 From: hjensas at redhat.com (Harald =?ISO-8859-1?Q?Jens=E5s?=) Date: Tue, 16 Jul 2019 13:28:59 +0200 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> Message-ID: <37937901d16a52f3ef5e6761a34812f4b87723cd.camel@redhat.com> On Mon, 2019-07-15 at 11:25 -0700, Dan Sneddon wrote: > This is my main question about this proposal. When TripleO was in its > infancy, there wasn't a mechanism to create Neutron ports separately > from the server, so we created a Nova Server resource that specified > which network the port was on (originally there was only one port > created, now we create additional ports in Neutron). This can be seen > in the puppet/-role.yaml file, for example: > > resources: > Controller: > type: OS::TripleO::ControllerServer > deletion_policy: {get_param: ServerDeletionPolicy} > metadata: > os-collect-config: > command: {get_param: ConfigCommand} > splay: {get_param: ConfigCollectSplay} > properties: > [...] > networks: > - if: > - ctlplane_fixed_ip_set > - network: ctlplane > subnet: {get_param: ControllerControlPlaneSubnet} > fixed_ip: > yaql: > expression: $.data.where(not isEmpty($)).first() > data: > - get_param: [ControllerIPs, 'ctlplane', > {get_param: NodeIndex}] > - network: ctlplane > subnet: {get_param: ControllerControlPlaneSubnet} > > This has the side-effect that the ports are created by Nova calling > Neutron rather than by Heat calling Neutron for port creation. We > have maintained this mechanism even in the latest versions of THT for > backwards compatibility. This would all be easier if we were creating > the Neutron ctlplane port and then assigning it to the server, but > that breaks backwards-compatibility. > This is indeed an issue that both nova-less and N=1 need to find a solution for. As soon as the nova server resources are removed from a stack the server and ctlplane port will be deleted. We loose track of which IP was assigned to which server at that point. I believe the plan in nova-less is to use the "protected" flag for Ironic nodes to ensure the baremetal node is not unprovisioned (destroyed). So the overcloud node will keep running. This however does'nt solve the problem with the ctlplane port being deleted. We need to ensure that the port is either not deleted, or that a new port is immediately created using the same IP address as before. If we don't we will very likely have duplicate IP issues on next scale out. > How would the creation of the ctlplane port be handled in this > proposal? If metalsmith is creating the ctlplane port, do we still > need a separate Server resource for every node? If so, I imagine it > would have a much smaller stack than what we currently create for > each server. If not, would metalsmith create a port on the ctlplane > as part of the provisioning steps, and then pass this port back? We > still need to be able to support fixed IPs for ctlplane ports, so we > need to be able to pass a specific IP to metalsmith. > The way nova-less works is that "openstack overcloud node provision" call's metalsmith to create a port and deploy the server. Once done the data for the servers are placed in a heat environment file defining the 'DeployedServerPortMap' parameter etc so that the already existing pre- deployed-server workflow[1] can be utilized. Using fixed IPs for ctlplane ports is possible with nova-less. But the interface to do so is changed, see[2]. [1] https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/deployed_server.html#network-configuration [2] https://specs.openstack.org/openstack/tripleo-specs/specs/stein/nova-less-deploy.html#examples From james.slagle at gmail.com Tue Jul 16 12:15:02 2019 From: james.slagle at gmail.com (James Slagle) Date: Tue, 16 Jul 2019 08:15:02 -0400 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> Message-ID: On Mon, Jul 15, 2019 at 2:25 PM Dan Sneddon wrote: > > > > On Mon, Jul 15, 2019 at 2:13 AM Harald Jensås wrote: >> >> On Sat, 2019-07-13 at 16:19 -0400, James Slagle wrote: >> > On Fri, Jul 12, 2019 at 3:59 PM Harald Jensås >> > wrote: >> > > I've said this before, but I think we should turn this nova-less >> > > around. Now with nova-less we create a bunch of servers, and write >> > > up >> > > the parameters file to use the deployed-server approach. >> > > Effectively we >> > > still neet to have the resource group in heat making a server >> > > resource >> > > for every server. Creating the fake server resource is fast, >> > > because >> > > Heat does'nt call Nova,Ironic to create any resources. But the >> > > stack is >> > > equally big, with a stack for every node. i.e not N=1. >> > > >> > > What you are doing here, is essentially to say we don't create a >> > > resource group that then creates N number of role stacks, one for >> > > each >> > > overcloud node. You are creating a single generic "server" >> > > definition >> > > per Role. So we drop the resource group and create >> > > OS::Triple::{{Role}}.Server 1-time (once). To me it's backwards to >> > > push >> > > a large struct with properties for N=many nodes into the creation >> > > of >> > > that stack. >> > >> > I'm not entirely following what you're saying is backwards. What I've >> > proposed is that we *don't* have any node specific data in the stack. >> > It sounds like you're saying the way we do it today is backwards. >> > >> >> What I mean to say is that I think the way we are integrating nova-less >> by first deploying the servers, to then provide the data to Heat to >> create the resource groups as we do today becomes backwards when your >> work on N=1 is introduced. >> >> >> > It's correct that what's been proposed with metalsmith currently >> > still >> > requires the full ResourceGroup with a member for each node. With the >> > template changes I'm proposing, that wouldn't be required, so we >> > could >> > actually do the Heat stack first, then metalsmith. >> > >> >> Yes, this is what I think we should do. Especially if your changes here >> removes the resource group entirely. It makes more sense to create the >> stack, and once that is created we can do deployment, scaling etc >> without updating the stack again. >> >> > > Currently the puppet/role-role.yaml creates all the network ports >> > > etc. >> > > As you only want to create it once, it instead could simply output >> > > the >> > > UUID of the networks+subnets. These are identical for all servers >> > > in >> > > the role. So we end up with a small heat stack. >> > > >> > > Once the stack is created we could use that generic "server" role >> > > data >> > > to feed into something (ansible?, python?, mistral?) that calls >> > > metalsmith to build the servers, then create ports for each server >> > > in >> > > neutron, one port for each network+subnet defined in the role. Then >> > > feed that output into the json (hieradata) that is pushed to each >> > > node >> > > and used during service configuration, all the things we need to >> > > configure network interfaces, /etc/hosts and so on. We need a way >> > > to >> > > keep track of which ports belong to wich node, but I guess >> > > something >> > > simple like using the node's ironic UUID in either the name, >> > > description or tag field of the neutron port will work. There is >> > > also >> > > the extra filed in Ironic which is json type, so we could place a >> > > map >> > > of network->port_uuid in there as well. >> > >> > It won't matter whether we do baremetal provisioning before or after >> > the Heat stack. Heat won't care, as it won't have any expectation to >> > create any servers or that they are already created. We can define >> > where we end up calling the metalsmith piece as it should be >> > independent of the Heat stack if we make these template changes. >> > >> >> This is true. But, in your previous mail in this thread you wrote: >> >> """ >> Other points: >> >> - Baremetal provisioning and port creation are presently handled by >> Heat. With the ongoing efforts to migrate baremetal provisioning out >> of Heat (nova-less deploy), I think these efforts are very >> complimentary. Eventually, we get to a point where Heat is not >> actually creating any other OpenStack API resources. For now, the >> patches only work when using pre-provisioned nodes. >> """ >> >> IMO "baremetal provision and port creation" fit together. (I read the >> above statement so as well.) Currently nova-less creates the ctlplane >> port and provision the baremetal node. If we want to do both baremetal >> provisioning and port creation togheter (I think this makes sense), we >> have to do it after the stack has created the networks. >> >> What I envision is to have one method that creates all the ports, >> ctlplane + composable networks in a unified way. Today these are >> created differently, the ctlplane port is part of the server resource >> (or metalsmith in nova-less case) and the other ports are created by >> heat. > > > This is my main question about this proposal. When TripleO was in its infancy, there wasn't a mechanism to create Neutron ports separately from the server, so we created a Nova Server resource that specified which network the port was on (originally there was only one port created, now we create additional ports in Neutron). This can be seen in the puppet/-role.yaml file, for example: > > resources: > Controller: > type: OS::TripleO::ControllerServer > deletion_policy: {get_param: ServerDeletionPolicy} > metadata: > os-collect-config: > command: {get_param: ConfigCommand} > splay: {get_param: ConfigCollectSplay} > properties: > [...] > networks: > - if: > - ctlplane_fixed_ip_set > - network: ctlplane > subnet: {get_param: ControllerControlPlaneSubnet} > fixed_ip: > yaql: > expression: $.data.where(not isEmpty($)).first() > data: > - get_param: [ControllerIPs, 'ctlplane', {get_param: NodeIndex}] > - network: ctlplane > subnet: {get_param: ControllerControlPlaneSubnet} > > This has the side-effect that the ports are created by Nova calling Neutron rather than by Heat calling Neutron for port creation. We have maintained this mechanism even in the latest versions of THT for backwards compatibility. This would all be easier if we were creating the Neutron ctlplane port and then assigning it to the server, but that breaks backwards-compatibility. > > How would the creation of the ctlplane port be handled in this proposal? If metalsmith is creating the ctlplane port, do we still need a separate Server resource for every node? If so, I imagine it would have a much smaller stack than what we currently create for each server. If not, would metalsmith create a port on the ctlplane as part of the provisioning steps, and then pass this port back? We still need to be able to support fixed IPs for ctlplane ports, so we need to be able to pass a specific IP to metalsmith. I think most of your questions pertain to defining the right interface for baremetal provisioning with metalsmith. We more or less have a clean slate there in terms of how we want that to look going forward. Given that it won't use Nova, my understanding is that the port(s) will be created via Neutron directly. We won't need separate server resources in the stack for every node once provisioning is not part of the stack. We will need to look at how we are creating the other network isolation ports per server however. It's something that we'll need to look at to see if we want to keep using Neutron just for IPAM. It seems a little wasteful to me, but perhaps it's not an issue even with thousands of ports. Initially, you'd be able to scale with just Ansible as long as the operator does mistakenly use overlapping IP's. We could also add ansible tasks that created the ports in Neutron (or verified they were already created) so that the actual IPAM usage is properly reflected in Neutron. -- -- James Slagle -- From james.slagle at gmail.com Tue Jul 16 12:21:32 2019 From: james.slagle at gmail.com (James Slagle) Date: Tue, 16 Jul 2019 08:21:32 -0400 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: <37937901d16a52f3ef5e6761a34812f4b87723cd.camel@redhat.com> References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> <37937901d16a52f3ef5e6761a34812f4b87723cd.camel@redhat.com> Message-ID: On Tue, Jul 16, 2019 at 7:29 AM Harald Jensås wrote: > As soon as the nova server resources are removed from a stack the > server and ctlplane port will be deleted. We loose track of which IP > was assigned to which server at that point. > > I believe the plan in nova-less is to use the "protected" flag for > Ironic nodes to ensure the baremetal node is not unprovisioned > (destroyed). So the overcloud node will keep running. This however > does'nt solve the problem with the ctlplane port being deleted. We > need to ensure that the port is either not deleted, or that a new port > is immediately created using the same IP address as before. If we don't > we will very likely have duplicate IP issues on next scale out. Heat provides a supported interface mechanism to override it's built-in resource types. We in fact already do this for both OS::Nova::Server and OS::Neutron::Port. When we manage resources of those types in our stack, we are actually using our custom plugins from tripleo-common. We can add additional logic there to handle this case and define whatever we want to have happen when the resources are deleted from the stack. This would address that issue for both N=1 and nova-less. -- -- James Slagle -- From james.slagle at gmail.com Tue Jul 16 12:25:16 2019 From: james.slagle at gmail.com (James Slagle) Date: Tue, 16 Jul 2019 08:25:16 -0400 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: <49e44a0f-84c9-195a-99d5-b843e4045454@redhat.com> References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> <6bdebb51-0b3c-2888-a691-720bd5ac039a@redhat.com> <49e44a0f-84c9-195a-99d5-b843e4045454@redhat.com> Message-ID: On Tue, Jul 16, 2019 at 3:23 AM Dmitry Tantsur wrote: > > On 7/16/19 12:26 AM, Steve Baker wrote: > > > > On 15/07/19 9:12 PM, Harald Jensås wrote: > >> On Sat, 2019-07-13 at 16:19 -0400, James Slagle wrote: > >>> On Fri, Jul 12, 2019 at 3:59 PM Harald Jensås > >>> wrote: > >>>> I've said this before, but I think we should turn this nova-less > >>>> around. Now with nova-less we create a bunch of servers, and write > >>>> up > >>>> the parameters file to use the deployed-server approach. > >>>> Effectively we > >>>> still neet to have the resource group in heat making a server > >>>> resource > >>>> for every server. Creating the fake server resource is fast, > >>>> because > >>>> Heat does'nt call Nova,Ironic to create any resources. But the > >>>> stack is > >>>> equally big, with a stack for every node. i.e not N=1. > >>>> > >>>> What you are doing here, is essentially to say we don't create a > >>>> resource group that then creates N number of role stacks, one for > >>>> each > >>>> overcloud node. You are creating a single generic "server" > >>>> definition > >>>> per Role. So we drop the resource group and create > >>>> OS::Triple::{{Role}}.Server 1-time (once). To me it's backwards to > >>>> push > >>>> a large struct with properties for N=many nodes into the creation > >>>> of > >>>> that stack. > >>> I'm not entirely following what you're saying is backwards. What I've > >>> proposed is that we *don't* have any node specific data in the stack. > >>> It sounds like you're saying the way we do it today is backwards. > >>> > >> What I mean to say is that I think the way we are integrating nova-less > >> by first deploying the servers, to then provide the data to Heat to > >> create the resource groups as we do today becomes backwards when your > >> work on N=1 is introduced. > >> > >> > >>> It's correct that what's been proposed with metalsmith currently > >>> still > >>> requires the full ResourceGroup with a member for each node. With the > >>> template changes I'm proposing, that wouldn't be required, so we > >>> could > >>> actually do the Heat stack first, then metalsmith. > >>> > >> Yes, this is what I think we should do. Especially if your changes here > >> removes the resource group entirely. It makes more sense to create the > >> stack, and once that is created we can do deployment, scaling etc > >> without updating the stack again. > > > > I think this is something we can move towards after James has finished this > > work. It would probably mean deprecating "openstack overcloud node provision" > > and providing some other way of running the baremetal provisioning in isolation > > after the heat stack operation, like an equivalent to "openstack overcloud > > deploy --config-download-only" > > > > > > I'm very much against on deprecating "openstack overcloud node provision", it's > one of the reasons of this whole effort. I'm equally -2 on making the bare metal > provisioning depending on heat in any way for the same reason. I think what's being proposed here is just that we'd change the ordering of the workflow in that we'd do the Heat stack first. That being said, I see the lack of dependency working both ways. Baremetal provisioning should not depend on Heat, and Heat should not depend on baremetal provisioning. You should be able to create the Heat stack without the servers actually existing (same as you can do today with pre-provisioned nodes). -- -- James Slagle -- From gmann at ghanshyammann.com Tue Jul 16 12:26:51 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 16 Jul 2019 21:26:51 +0900 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> Message-ID: <16bfabff222.12ae1a1f0212237.3798556913715077596@ghanshyammann.com> ---- On Mon, 15 Jul 2019 18:50:05 +0900 Thierry Carrez wrote ---- > Andreas Jaeger wrote: > > [...] > > I see the following options: > > > > 1) Retiring developer.openstack.org completely, this would mean we would > > host the api-guides and api-references on docs.openstack.org (perhaps > > with moving them into doc/source). If we go down this road, we need to > > discuss what this means (redirects) and what to do with the Api-Guide > > and the FirstApp guide. +1 on option 1. openstack api-guides (not the individual projects api-guides) make more sense under os-api-ref which is nothing but overall openstack APIs state and less maintenance effort as you mentioned. api-references content is on project side ./api-ref/source. Do you mean to move them to doc/source ? or cannot we host docs.openstack.org from the same existing ./api-ref/source location? -gmann > > > > 2) Fully revitialize the repo and have it owned by an official team or > > SIG (this means reverting parts of https://review.opendev.org/485249/) > > > > 3) Retire the document "Writing your first OpenStack Application", and > > unretire api-site and have it owned by some official team/SIG. > > > > Any other options? What shall we do? > > Thanks Andreas for raising this. > > As an extra data point, my long-term plan was to have SDKs and CLIs > properly listed in the Software pages under SDKs[1], including > third-party ones in their own subtab, all driven from the > osf/openstack-map repository[2]. > > With that in mind, I think it would make sense to look into retiring > developer.openstack.org, and move docs to docs.openstack.org. We could > also revive https://www.openstack.org/appdev/ and use it as the base > landing page to direct application-side people to the various pieces. > > [1] https://www.openstack.org/software/project-navigator/sdks > [2] https://opendev.org/osf/openstack-map/ > > -- > Thierry Carrez (ttx) > > From gmann at ghanshyammann.com Tue Jul 16 12:29:42 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 16 Jul 2019 21:29:42 +0900 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: <116050ecdd0b7c5ecbe728d914b67d3f0770a2ea.camel@redhat.com> References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <116050ecdd0b7c5ecbe728d914b67d3f0770a2ea.camel@redhat.com> Message-ID: <16bfac28ea9.11c8a05f6212408.2764941371109330116@ghanshyammann.com> ---- On Mon, 15 Jul 2019 23:28:33 +0900 Sean Mooney wrote ---- > On Mon, 2019-07-15 at 11:50 +0200, Thierry Carrez wrote: > > Andreas Jaeger wrote: > > > [...] > > > I see the following options: > > > > > > 1) Retiring developer.openstack.org completely, this would mean we would > > > host the api-guides and api-references on docs.openstack.org (perhaps > > > with moving them into doc/source). If we go down this road, we need to > > > discuss what this means (redirects) and what to do with the Api-Guide > > > and the FirstApp guide. > > > > > > 2) Fully revitialize the repo and have it owned by an official team or > > > SIG (this means reverting parts of https://review.opendev.org/485249/) > > > > > > 3) Retire the document "Writing your first OpenStack Application", and > > > unretire api-site and have it owned by some official team/SIG. > > > > > > Any other options? What shall we do? > > > > Thanks Andreas for raising this. > > > > As an extra data point, my long-term plan was to have SDKs and CLIs > > properly listed in the Software pages under SDKs[1], including > > third-party ones in their own subtab, all driven from the > > osf/openstack-map repository[2]. > > > > With that in mind, I think it would make sense to look into retiring > > developer.openstack.org, > i use https://developer.openstack.org/api-ref/compute/ almost daily so unless > we host the api ref somwhere else and put redirect in place i would hope we > can keep this inplace. Yeah, this link is used very frequently by many developers as well as by users. Redirect is something much needed for this site. -gmann > if we move it under docs like the config stuff > https://docs.openstack.org/nova/latest/configuration/config.html > or somewhere else that is fine but i fine it very useful to be able > to link the rendered api docs to people on irc that ask questions. > > i can obviosly point peple to github > https://github.com/openstack/nova/blob/master/api-ref/source/servers.inc > but unlike the configs the api ref is much less readable with out rendering > it with sphinx > > > and move docs to docs.openstack.org. We could > > also revive https://www.openstack.org/appdev/ and use it as the base > > landing page to direct application-side people to the various pieces. > > > > [1] https://www.openstack.org/software/project-navigator/sdks > > [2] https://opendev.org/osf/openstack-map/ > > > > > From aj at suse.com Tue Jul 16 12:35:43 2019 From: aj at suse.com (Andreas Jaeger) Date: Tue, 16 Jul 2019 14:35:43 +0200 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: <16bfabff222.12ae1a1f0212237.3798556913715077596@ghanshyammann.com> References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <16bfabff222.12ae1a1f0212237.3798556913715077596@ghanshyammann.com> Message-ID: On 16/07/2019 14.26, Ghanshyam Mann wrote: > ---- On Mon, 15 Jul 2019 18:50:05 +0900 Thierry Carrez wrote ---- > > Andreas Jaeger wrote: > > > [...] > > > I see the following options: > > > > > > 1) Retiring developer.openstack.org completely, this would mean we would > > > host the api-guides and api-references on docs.openstack.org (perhaps > > > with moving them into doc/source). If we go down this road, we need to > > > discuss what this means (redirects) and what to do with the Api-Guide > > > and the FirstApp guide. > > +1 on option 1. > > openstack api-guides (not the individual projects api-guides) make more sense under > os-api-ref which is nothing but overall openstack APIs state and less maintenance effort as you mentioned. I propose move it to openstack-manuals instead of os-api-ref to make publishing easier. os-api-ref is a tool used by others. > api-references content is on project side ./api-ref/source. Do you mean to move them to > doc/source ? or cannot we host docs.openstack.org from the same existing ./api-ref/source location? The idea some time ago was to move them to doc/source But short-term we can just change the publishing jobs to publish to docs.openstack.org/api-reference instead of developer.openstack.org/api-reference, Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From zhangbailin at inspur.com Tue Jul 16 12:39:41 2019 From: zhangbailin at inspur.com (=?utf-8?B?QnJpbiBaaGFuZyjlvKDnmb7mnpcp?=) Date: Tue, 16 Jul 2019 12:39:41 +0000 Subject: =?utf-8?B?IFtsaXN0cy5vcGVuc3RhY2sub3Jn5Luj5Y+RXVJlOiBbZG9jc11bdGNdW2lu?= =?utf-8?B?ZnJhXSB3aGF0IHRvIGRvIHdpdGggZGV2ZWxvcGVyLm9wZW5zdGFjay5vcmcg?= =?utf-8?Q?and_api-site=3F?= Message-ID: <0b8be13f4fe54fe99b2f71d56b40b022@inspur.com> >>> Yeah, this link is used very frequently by many developers as well as by users. Redirect is something much needed for this site. +1, yeah. https://developer.openstack.org/api-ref/compute/ is used at the highest frequency, I think redirect this is necessary. ---- On Mon, 15 Jul 2019 23:28:33 +0900 Sean Mooney wrote ---- > On Mon, 2019-07-15 at 11:50 +0200, Thierry Carrez wrote: > > Andreas Jaeger wrote: > > > [...] > > > I see the following options: > > > > > > 1) Retiring developer.openstack.org completely, this would mean we would > > > host the api-guides and api-references on docs.openstack.org (perhaps > > > with moving them into doc/source). If we go down this road, we need to > > > discuss what this means (redirects) and what to do with the Api-Guide > > > and the FirstApp guide. > > > > > > 2) Fully revitialize the repo and have it owned by an official team or > > > SIG (this means reverting parts of https://review.opendev.org/485249/) > > > > > > 3) Retire the document "Writing your first OpenStack Application", and > > > unretire api-site and have it owned by some official team/SIG. > > > > > > Any other options? What shall we do? > > > > Thanks Andreas for raising this. > > > > As an extra data point, my long-term plan was to have SDKs and CLIs > > properly listed in the Software pages under SDKs[1], including > > third-party ones in their own subtab, all driven from the > > osf/openstack-map repository[2]. > > > > With that in mind, I think it would make sense to look into retiring > > developer.openstack.org, > i use https://developer.openstack.org/api-ref/compute/ almost daily so unless > we host the api ref somwhere else and put redirect in place i would hope we > can keep this inplace. Yeah, this link is used very frequently by many developers as well as by users. Redirect is something much needed for this site. -gmann > if we move it under docs like the config stuff > https://docs.openstack.org/nova/latest/configuration/config.html > or somewhere else that is fine but i fine it very useful to be able > to link the rendered api docs to people on irc that ask questions. > > i can obviosly point peple to github > https://github.com/openstack/nova/blob/master/api-ref/source/servers.inc > but unlike the configs the api ref is much less readable with out rendering > it with sphinx > > > and move docs to docs.openstack.org. We could > > also revive https://www.openstack.org/appdev/ and use it as the base > > landing page to direct application-side people to the various pieces. > > > > [1] https://www.openstack.org/software/project-navigator/sdks > > [2] https://opendev.org/osf/openstack-map/ > > > > > From gmann at ghanshyammann.com Tue Jul 16 12:49:15 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 16 Jul 2019 21:49:15 +0900 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <16bfabff222.12ae1a1f0212237.3798556913715077596@ghanshyammann.com> Message-ID: <16bfad474a0.c753acc3213344.8952022068766030087@ghanshyammann.com> ---- On Tue, 16 Jul 2019 21:35:43 +0900 Andreas Jaeger wrote ---- > On 16/07/2019 14.26, Ghanshyam Mann wrote: > > ---- On Mon, 15 Jul 2019 18:50:05 +0900 Thierry Carrez wrote ---- > > > Andreas Jaeger wrote: > > > > [...] > > > > I see the following options: > > > > > > > > 1) Retiring developer.openstack.org completely, this would mean we would > > > > host the api-guides and api-references on docs.openstack.org (perhaps > > > > with moving them into doc/source). If we go down this road, we need to > > > > discuss what this means (redirects) and what to do with the Api-Guide > > > > and the FirstApp guide. > > > > +1 on option 1. > > > > openstack api-guides (not the individual projects api-guides) make more sense under > > os-api-ref which is nothing but overall openstack APIs state and less maintenance effort as you mentioned. > > I propose move it to openstack-manuals instead of os-api-ref to make > publishing easier. os-api-ref is a tool used by others. I am ok to move under openstack-manual also. Though we do not know the future home of openstack-manual as docs team is moving towards SIG. But where ever it will go, 'OpenStack api guide' can go with this. > > > api-references content is on project side ./api-ref/source. Do you mean to move them to > > doc/source ? or cannot we host docs.openstack.org from the same existing ./api-ref/source location? > > The idea some time ago was to move them to doc/source > > But short-term we can just change the publishing jobs to publish to > docs.openstack.org/api-reference instead of > developer.openstack.org/api-reference, Yeah, I remember there were no clear consesus of moving the api-ref/source to doc/source so that might take time. publishing from api-ref/source as short term options looks good to me. Thanks for initiating this discussion and caring about these sites. -gmann > > Andreas > -- > Andreas Jaeger aj at suse.com Twitter: jaegerandi > SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah > HRB 21284 (AG Nürnberg) > GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 > From aj at suse.com Tue Jul 16 12:54:39 2019 From: aj at suse.com (Andreas Jaeger) Date: Tue, 16 Jul 2019 14:54:39 +0200 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: <16bfad474a0.c753acc3213344.8952022068766030087@ghanshyammann.com> References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <16bfabff222.12ae1a1f0212237.3798556913715077596@ghanshyammann.com> <16bfad474a0.c753acc3213344.8952022068766030087@ghanshyammann.com> Message-ID: <9cfa698b-4638-88ba-e0db-799cf9543ae4@suse.com> On 16/07/2019 14.49, Ghanshyam Mann wrote: > [...] > I am ok to move under openstack-manual also. Though we do not know the > future home of openstack-manual as docs team is moving towards SIG. But where ever > it will go, 'OpenStack api guide' can go with this. I think a SIG can own repositories, so there's no technical blocker for that, Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From thierry at openstack.org Tue Jul 16 13:26:39 2019 From: thierry at openstack.org (Thierry Carrez) Date: Tue, 16 Jul 2019 15:26:39 +0200 Subject: [dev][release][qa] patrole stable/stein is created by mistake ? In-Reply-To: <16bf8e85983.b41e1d9d193754.93039526410042655@ghanshyammann.com> References: <16beb436c21.c9d98154147017.455401003873410397@ghanshyammann.com> <16beb463457.1193bd7a4147042.5135658897011450644@ghanshyammann.com> <20190713203828.GA29711@sm-workstation> <16beddf697c.fd5e3e64150304.4626321901014847129@ghanshyammann.com> <0a85c82c-f312-a8a9-34af-f5b812c8f2b3@openstack.org> <717CD206-8A3A-438F-A52B-22396CFA9BAD@doughellmann.com> <16bf8e85983.b41e1d9d193754.93039526410042655@ghanshyammann.com> Message-ID: Ghanshyam Mann wrote: > ---- On Tue, 16 Jul 2019 03:37:05 +0900 Doug Hellmann wrote ---- > > If the project isn’t creating stable branches each cycle, how is it cycle-based? What’s wrong with moving it to use the independent release model, which was created for cases like this? > > Tempest and all Tempest plugins are "cycle-with-intermediary" because we want a particular release tag per OpenStack release. That is what we decided in ML discussion while deciding the release model for tempest plugins [1]. With independent release mode, we face the issue of not having the compatible versions of all Tempest plugins which user need to run on against their production cloud. > > [1] http://lists.openstack.org/pipermail/openstack-dev/2018-June/131837.html Yes, it's a weird corner case that we regularly rediscuss when we (re)discover it does not fit well in our traditional cycle-with-intermediary box. I feel like having an extra stable-branch-type will help us remember what to do with those and avoid mistakes. -- Thierry Carrez (ttx) From dtantsur at redhat.com Tue Jul 16 13:31:21 2019 From: dtantsur at redhat.com (Dmitry Tantsur) Date: Tue, 16 Jul 2019 15:31:21 +0200 Subject: [TripleO] Scaling node counts with only Ansible (N=1) In-Reply-To: References: <23924034ea0981350b7e241aed5e99c5e769b291.camel@redhat.com> <6bdebb51-0b3c-2888-a691-720bd5ac039a@redhat.com> <49e44a0f-84c9-195a-99d5-b843e4045454@redhat.com> Message-ID: On 7/16/19 2:25 PM, James Slagle wrote: > On Tue, Jul 16, 2019 at 3:23 AM Dmitry Tantsur wrote: >> >> On 7/16/19 12:26 AM, Steve Baker wrote: >>> >>> On 15/07/19 9:12 PM, Harald Jensås wrote: >>>> On Sat, 2019-07-13 at 16:19 -0400, James Slagle wrote: >>>>> On Fri, Jul 12, 2019 at 3:59 PM Harald Jensås >>>>> wrote: >>>>>> I've said this before, but I think we should turn this nova-less >>>>>> around. Now with nova-less we create a bunch of servers, and write >>>>>> up >>>>>> the parameters file to use the deployed-server approach. >>>>>> Effectively we >>>>>> still neet to have the resource group in heat making a server >>>>>> resource >>>>>> for every server. Creating the fake server resource is fast, >>>>>> because >>>>>> Heat does'nt call Nova,Ironic to create any resources. But the >>>>>> stack is >>>>>> equally big, with a stack for every node. i.e not N=1. >>>>>> >>>>>> What you are doing here, is essentially to say we don't create a >>>>>> resource group that then creates N number of role stacks, one for >>>>>> each >>>>>> overcloud node. You are creating a single generic "server" >>>>>> definition >>>>>> per Role. So we drop the resource group and create >>>>>> OS::Triple::{{Role}}.Server 1-time (once). To me it's backwards to >>>>>> push >>>>>> a large struct with properties for N=many nodes into the creation >>>>>> of >>>>>> that stack. >>>>> I'm not entirely following what you're saying is backwards. What I've >>>>> proposed is that we *don't* have any node specific data in the stack. >>>>> It sounds like you're saying the way we do it today is backwards. >>>>> >>>> What I mean to say is that I think the way we are integrating nova-less >>>> by first deploying the servers, to then provide the data to Heat to >>>> create the resource groups as we do today becomes backwards when your >>>> work on N=1 is introduced. >>>> >>>> >>>>> It's correct that what's been proposed with metalsmith currently >>>>> still >>>>> requires the full ResourceGroup with a member for each node. With the >>>>> template changes I'm proposing, that wouldn't be required, so we >>>>> could >>>>> actually do the Heat stack first, then metalsmith. >>>>> >>>> Yes, this is what I think we should do. Especially if your changes here >>>> removes the resource group entirely. It makes more sense to create the >>>> stack, and once that is created we can do deployment, scaling etc >>>> without updating the stack again. >>> >>> I think this is something we can move towards after James has finished this >>> work. It would probably mean deprecating "openstack overcloud node provision" >>> and providing some other way of running the baremetal provisioning in isolation >>> after the heat stack operation, like an equivalent to "openstack overcloud >>> deploy --config-download-only" >>> >>> >> >> I'm very much against on deprecating "openstack overcloud node provision", it's >> one of the reasons of this whole effort. I'm equally -2 on making the bare metal >> provisioning depending on heat in any way for the same reason. > > I think what's being proposed here is just that we'd change the > ordering of the workflow in that we'd do the Heat stack first. > > That being said, I see the lack of dependency working both ways. > Baremetal provisioning should not depend on Heat, and Heat should not > depend on baremetal provisioning. You should be able to create the > Heat stack without the servers actually existing (same as you can do > today with pre-provisioned nodes). > Right, and we should be able to provision bare metals without pre-creating heat stack.. and then I don't understand why we want to change the current proposal. From thierry at openstack.org Tue Jul 16 13:33:39 2019 From: thierry at openstack.org (Thierry Carrez) Date: Tue, 16 Jul 2019 15:33:39 +0200 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: <9cfa698b-4638-88ba-e0db-799cf9543ae4@suse.com> References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <16bfabff222.12ae1a1f0212237.3798556913715077596@ghanshyammann.com> <16bfad474a0.c753acc3213344.8952022068766030087@ghanshyammann.com> <9cfa698b-4638-88ba-e0db-799cf9543ae4@suse.com> Message-ID: <82ff59f8-5ed7-fad0-f5bb-867a1da58fcb@openstack.org> Andreas Jaeger wrote: > On 16/07/2019 14.49, Ghanshyam Mann wrote: >> [...] >> I am ok to move under openstack-manual also. Though we do not know the >> future home of openstack-manual as docs team is moving towards SIG. But where ever >> it will go, 'OpenStack api guide' can go with this. > > I think a SIG can own repositories, so there's no technical blocker for > that, Yes a SIG (and any other workgroup) can own git repositories. Only deliverables (groups of repositories released together as part of the OpenStack release) need to be owned by a project team. -- Thierry Carrez (ttx) From massimo.sgaravatto at gmail.com Tue Jul 16 13:37:34 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Tue, 16 Jul 2019 15:37:34 +0200 Subject: [nova] [ops] Problems running db sync for nova (Ocata --> Pike) In-Reply-To: References: <20190716104839.Horde.Pe3wSo-woFy0u8rF9gnehyD@webmail.nde.ag> <20190716111151.Horde.2mXkgZIEbUsVwdTqrDfdbD4@webmail.nde.ag> Message-ID: We think that the problem was 390 in nova_prod_cell0.migrate_version. Instead it should be 362. No idea how this happened ... We have manually changed that value to 362 and the db_sync worked On Tue, Jul 16, 2019 at 1:16 PM Massimo Sgaravatto < massimo.sgaravatto at gmail.com> wrote: > My understanding (e.g. from #comment 1 of this bug: > https://bugs.launchpad.net/grenade/+bug/1761775) is that --local-cell > updates only the database specified as connection in [database], i.e. not > also the cell0 database > > On Tue, Jul 16, 2019 at 1:12 PM Eugen Block wrote: > >> AFAIK cell0 is the local cell, isn't it? I'm not sure, maybe someone >> else can shed some light on this. >> >> >> Zitat von Massimo Sgaravatto : >> >> > Mmm. >> > Doesn't "--local_cell" mean that cell0 is not updated ? >> > I was already using cell0 in Ocata ... >> > >> > Cheers, Massimo >> > >> > On Tue, Jul 16, 2019 at 12:52 PM Eugen Block wrote: >> > >> >> Hi, >> >> >> >> I think you need to run >> >> >> >> nova-manage db sync --local_cell >> >> >> >> in Ocata. I can't quite remember the reason, but you should try it. >> >> I had the same problems and this command worked out for me. >> >> >> >> Regards, >> >> Eugen >> >> >> >> >> >> >> >> Zitat von Massimo Sgaravatto : >> >> >> >> > Resending with the right tags in the subject line ... >> >> > >> >> > >> >> > Hi >> >> > >> >> > We are trying to update our Ocata cloud to Pike (this is the first >> step: >> >> > we will go through Ocata --> Pike --> Queens --> Rocky) but we have a >> >> > problem with nova-manage db sync: >> >> > >> >> > >> >> > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db sync" nova >> >> > ERROR: Could not access cell0. >> >> > Has the nova_api database been created? >> >> > Has the nova_cell0 database been created? >> >> > Has "nova-manage api_db sync" been run? >> >> > Has "nova-manage cell_v2 map_cell0" been run? >> >> > Is [api_database]/connection set in nova.conf? >> >> > Is the cell0 database connection URL correct? >> >> > Error: "Database schema file with version 390 doesn't exist." >> >> > >> >> > We have these settings in nova.conf: >> >> > >> >> > [database] >> >> > connection = mysql+pymysql:// >> nova_prod:xyz at 192.168.60.10:6306/nova_prod >> >> > [api_database] >> >> > connection = mysql+pymysql:// >> >> > nova_api_prod:xyz at 192.168.60.10:6306/nova_api_prod >> >> > >> >> > I can't see problems accessing the databases. i.e. these commands >> work: >> >> > >> >> > mysql -u nova_api_prod -pxyz -h 192.168.60.10 -P 6306 nova_api_prod >> >> > >> >> > mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod >> >> > >> >> > mysql -u nova_prod -pxyz -h 192.168.60.10 -P 6306 nova_prod_cell0 >> >> > >> >> > If I try to rerun nova-manage cell_v2 map_cell0: >> >> > >> >> > [root at cld-ctrl-01 ~]# nova-manage cell_v2 map_cell0 >> >> > Cell0 is already setup >> >> > >> >> > >> >> > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage api_db version" >> nova >> >> > 45 >> >> > >> >> > [root at cld-ctrl-01 ~]# su -s /bin/sh -c "nova-manage db version" nova >> >> > 362 >> >> > >> >> > [root at cld-ctrl-01 ~]# nova-manage cell_v2 list_cells >> >> > >> >> >> +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ >> >> > | Name | UUID | >> >> Transport >> >> > URL | Database Connection >> >> > | >> >> > >> >> >> +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ >> >> > | cell0 | 00000000-0000-0000-0000-000000000000 | >> >> > none:/// | mysql+pymysql://nova_prod:****@ >> >> > 192.168.60.10:6306/nova_prod_cell0 | >> >> > | cell1 | 5e42faa0-710b-4967-bb42-fcf53602c96e | >> >> > rabbit://openstack_prod:****@192.168.60.183:5672 | >> >> > mysql+pymysql://nova_prod:****@192.168.60.10:6306/nova_prod | >> >> > >> >> >> +-------+--------------------------------------+--------------------------------------------------+-------------------------------------------------------------------+ >> >> > >> >> > Following some doc we also tried to specify the >> --database_connection / >> >> > --database-connection argument, but this seems not working >> >> > >> >> > >> >> > Setting nova in debug mode, this is the last entry in the >> nova-manage log >> >> > file: >> >> > >> >> > 2019-07-16 12:11:42.774 22013 DEBUG migrate.versioning.repository >> >> > [req-8ca52407-dbe1-4a62-ad6c-16631c2a9a06 - - - - -] Config: >> >> > OrderedDict([('db_settings', OrderedDict([('__name__', >> 'db_settings'), >> >> > ('repository_id', 'nova'), ('version_table', 'migrate_version'), >> >> > ('required_dbs', '[]')]))]) __init__ >> >> > /usr/lib/python2.7/site-packages/migrate/versioning/repository.py:83 >> >> > >> >> > Any hints ? >> >> > >> >> > Thanks, Massimo >> >> >  >> >> >  >> >> > OpenStack Discuss >> >> >> >> >> >> >> >> >> >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Tue Jul 16 14:02:10 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 16 Jul 2019 23:02:10 +0900 Subject: [dev][release][qa] patrole stable/stein is created by mistake ? In-Reply-To: References: <16beb436c21.c9d98154147017.455401003873410397@ghanshyammann.com> <16beb463457.1193bd7a4147042.5135658897011450644@ghanshyammann.com> <20190713203828.GA29711@sm-workstation> <16beddf697c.fd5e3e64150304.4626321901014847129@ghanshyammann.com> <0a85c82c-f312-a8a9-34af-f5b812c8f2b3@openstack.org> <717CD206-8A3A-438F-A52B-22396CFA9BAD@doughellmann.com> <16bf8e85983.b41e1d9d193754.93039526410042655@ghanshyammann.com> Message-ID: <16bfb1734a1.11fb152da217117.4639583386960737975@ghanshyammann.com> ---- On Tue, 16 Jul 2019 22:26:39 +0900 Thierry Carrez wrote ---- > Ghanshyam Mann wrote: > > ---- On Tue, 16 Jul 2019 03:37:05 +0900 Doug Hellmann wrote ---- > > > If the project isn’t creating stable branches each cycle, how is it cycle-based? What’s wrong with moving it to use the independent release model, which was created for cases like this? > > > > Tempest and all Tempest plugins are "cycle-with-intermediary" because we want a particular release tag per OpenStack release. That is what we decided in ML discussion while deciding the release model for tempest plugins [1]. With independent release mode, we face the issue of not having the compatible versions of all Tempest plugins which user need to run on against their production cloud. > > > > [1] http://lists.openstack.org/pipermail/openstack-dev/2018-June/131837.html > > Yes, it's a weird corner case that we regularly rediscuss when we > (re)discover it does not fit well in our traditional > cycle-with-intermediary box. > > I feel like having an extra stable-branch-type will help us remember > what to do with those and avoid mistakes. Yeah, I agree with that. Once your patch of stable-branch-type is merged, I can add this type as none for tempest and its plugins. -gmann > > -- > Thierry Carrez (ttx) > > From sfinucan at redhat.com Tue Jul 16 14:16:35 2019 From: sfinucan at redhat.com (Stephen Finucane) Date: Tue, 16 Jul 2019 15:16:35 +0100 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <16bfabff222.12ae1a1f0212237.3798556913715077596@ghanshyammann.com> Message-ID: <4ec701899069e823412cc908a67fc6a70713ebb1.camel@redhat.com> On Tue, 2019-07-16 at 14:35 +0200, Andreas Jaeger wrote: > On 16/07/2019 14.26, Ghanshyam Mann wrote: > > ---- On Mon, 15 Jul 2019 18:50:05 +0900 Thierry Carrez wrote ---- > > > Andreas Jaeger wrote: > > > > [...] > > > > I see the following options: > > > > > > > > 1) Retiring developer.openstack.org completely, this would mean we would > > > > host the api-guides and api-references on docs.openstack.org (perhaps > > > > with moving them into doc/source). If we go down this road, we need to > > > > discuss what this means (redirects) and what to do with the Api-Guide > > > > and the FirstApp guide. > > > > +1 on option 1. > > > > openstack api-guides (not the individual projects api-guides) make more sense under > > os-api-ref which is nothing but overall openstack APIs state and less maintenance effort as you mentioned. > > I propose move it to openstack-manuals instead of os-api-ref to make > publishing easier. os-api-ref is a tool used by others. > > > api-references content is on project side ./api-ref/source. Do you mean to move them to > > doc/source ? or cannot we host docs.openstack.org from the same existing ./api-ref/source location? > > The idea some time ago was to move them to doc/source Note that we can't do this because the API reference guides (and release notes, since reno does that for us) shouldn't be versioned by release, unlike the rest of the docs. > But short-term we can just change the publishing jobs to publish to > docs.openstack.org/api-reference instead of > developer.openstack.org/api-reference, Stephen > Andreas From aj at suse.com Tue Jul 16 14:17:40 2019 From: aj at suse.com (Andreas Jaeger) Date: Tue, 16 Jul 2019 16:17:40 +0200 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> Message-ID: <38f296e3-0292-76da-93af-2db554d918e3@suse.com> >From the feedback so far, let me refine the proposal: * Move OpenStack API-Guide to openstack-manuals repo and publish to docs.o.o * Kill "Writing Your First OpenStack Application" document * Publish all in-tree api-references and api-guides on docs.openstack.org instead of developer.o.o * List SDKs and CLIs on the software page * Add redirects on developer.openstack.org for moved documents * Once the above is all done, kill api-site (or leave it up for some time in frozen state with redirects). I can take care of most steps (exception is software page - this is for Thierry, correct?), Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From sfinucan at redhat.com Tue Jul 16 14:34:38 2019 From: sfinucan at redhat.com (Stephen Finucane) Date: Tue, 16 Jul 2019 15:34:38 +0100 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: <38f296e3-0292-76da-93af-2db554d918e3@suse.com> References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <38f296e3-0292-76da-93af-2db554d918e3@suse.com> Message-ID: <501409a191d4fdb8067b87843c3b38f3cb1a259f.camel@redhat.com> On Tue, 2019-07-16 at 16:17 +0200, Andreas Jaeger wrote: > From the feedback so far, let me refine the proposal: > * Move OpenStack API-Guide to openstack-manuals repo and publish to docs.o.o > * Kill "Writing Your First OpenStack Application" document > * Publish all in-tree api-references and api-guides on > docs.openstack.org instead of developer.o.o > * List SDKs and CLIs on the software page > * Add redirects on developer.openstack.org for moved documents > * Once the above is all done, kill api-site (or leave it up for some > time in frozen state with redirects). This all make sense to me, though the crucial part is those redirects, of course. I don't know how much time I can spare but let me know if you need help with anything. Stephen > I can take care of most steps (exception is software page - this is for > Thierry, correct?), > > Andreas From smooney at redhat.com Tue Jul 16 14:39:15 2019 From: smooney at redhat.com (Sean Mooney) Date: Tue, 16 Jul 2019 15:39:15 +0100 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: <38f296e3-0292-76da-93af-2db554d918e3@suse.com> References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <38f296e3-0292-76da-93af-2db554d918e3@suse.com> Message-ID: On Tue, 2019-07-16 at 16:17 +0200, Andreas Jaeger wrote: > From the feedback so far, let me refine the proposal: > * Move OpenStack API-Guide to openstack-manuals repo and publish to docs.o.o > * Kill "Writing Your First OpenStack Application" document > * Publish all in-tree api-references and api-guides on > docs.openstack.org instead of developer.o.o > * List SDKs and CLIs on the software page > * Add redirects on developer.openstack.org for moved documents it looks like my reply earlirer got droped when i rebotted my laptop. just wanted to say i use the config referecnce https://docs.openstack.org/nova/latest/configuration/config.html and the api doc https://developer.openstack.org/api-ref/compute/ almost daily espcially when helping people on irc. as long as we continue to have a rendered copy of them hosted that is fine and idealy with redirect as you are suggesting. unlike some of the other docs the api-ref is much harder to consume in plain text form the git repo or via github https://github.com/openstack/nova/blob/master/api-ref/source/servers.inc or opendev https://opendev.org/openstack/nova/src/branch/master/api-ref/source/servers.inc although if it wasnt fro the lag the opendev version is much better then the way github tries to render it as markdown. it sound like the plan will contiue to preseve a hosted copy of the api refes so i guess that is my main request. > * Once the above is all done, kill api-site (or leave it up for some > time in frozen state with redirects). > > I can take care of most steps (exception is software page - this is for > Thierry, correct?), > > Andreas From gmann at ghanshyammann.com Tue Jul 16 14:45:35 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 16 Jul 2019 23:45:35 +0900 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: <38f296e3-0292-76da-93af-2db554d918e3@suse.com> References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <38f296e3-0292-76da-93af-2db554d918e3@suse.com> Message-ID: <16bfb3ef329.c7ad3046218824.2120105264400443075@ghanshyammann.com> ---- On Tue, 16 Jul 2019 23:17:40 +0900 Andreas Jaeger wrote ---- > From the feedback so far, let me refine the proposal: > * Move OpenStack API-Guide to openstack-manuals repo and publish to docs.o.o > * Kill "Writing Your First OpenStack Application" document > * Publish all in-tree api-references and api-guides on > docs.openstack.org instead of developer.o.o > * List SDKs and CLIs on the software page > * Add redirects on developer.openstack.org for moved documents > * Once the above is all done, kill api-site (or leave it up for some > time in frozen state with redirects). > > I can take care of most steps (exception is software page - this is for > Thierry, correct?), +1 on all. Thanks again. -gmann > > Andreas > -- > Andreas Jaeger aj at suse.com Twitter: jaegerandi > SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah > HRB 21284 (AG Nürnberg) > GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 > > From aj at suse.com Tue Jul 16 14:46:42 2019 From: aj at suse.com (Andreas Jaeger) Date: Tue, 16 Jul 2019 16:46:42 +0200 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <38f296e3-0292-76da-93af-2db554d918e3@suse.com> Message-ID: <632f06ff-73bf-1483-a13b-36ce0057a9d5@suse.com> On 7/16/19 4:39 PM, Sean Mooney wrote: > On Tue, 2019-07-16 at 16:17 +0200, Andreas Jaeger wrote: >> From the feedback so far, let me refine the proposal: >> * Move OpenStack API-Guide to openstack-manuals repo and publish to docs.o.o >> * Kill "Writing Your First OpenStack Application" document >> * Publish all in-tree api-references and api-guides on >> docs.openstack.org instead of developer.o.o >> * List SDKs and CLIs on the software page >> * Add redirects on developer.openstack.org for moved documents > it looks like my reply earlirer got droped when i rebotted my laptop. > just wanted to say i use the config referecnce > https://docs.openstack.org/nova/latest/configuration/config.html > and the api doc https://developer.openstack.org/api-ref/compute/ > almost daily espcially when helping people on irc. Yes, the steps above will ensure that we publish to https://docs.openstack.org/api-ref/compute/ and set up a redirect to the new location. So, your bookmarks will continue to work - and going forward, I recommend updating them ;) > as long as we continue to have a rendered copy of them hosted that is fine > and idealy with redirect as you are suggesting. unlike some of the other > docs the api-ref is much harder to consume in plain text form the git repo > or via github https://github.com/openstack/nova/blob/master/api-ref/source/servers.inc > or opendev https://opendev.org/openstack/nova/src/branch/master/api-ref/source/servers.inc > although if it wasnt fro the lag the opendev version is much better then the way > github tries to render it as markdown. > > it sound like the plan will contiue to preseve a hosted copy of the api refes so > i guess that is my main request. Correct, Andreas >> * Once the above is all done, kill api-site (or leave it up for some >> time in frozen state with redirects). >> >> I can take care of most steps (exception is software page - this is for >> Thierry, correct?), >> >> Andreas > > -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From dangtrinhnt at gmail.com Tue Jul 16 14:47:21 2019 From: dangtrinhnt at gmail.com (Trinh Nguyen) Date: Tue, 16 Jul 2019 23:47:21 +0900 Subject: [telemetry] Looking for new meeting time In-Reply-To: References: Message-ID: So I guess, It's better to keep the conversation on ML only for now. And then we can come back to chat/video whenever we can find a common timeline. What do you think? Bests, On Mon, Jul 15, 2019 at 1:05 PM Lingxian Kong wrote: > congratulations first :-) > > I'm on UTC+12, so meeting during 13:00-15:00 UTC is hard for me :-( > > Best regards, > Lingxian Kong > Catalyst Cloud > > > On Mon, Jul 15, 2019 at 1:23 PM Trinh Nguyen > wrote: > >> Hi team, >> >> I've changed job and could not afford doing the meeting at 02:00 UTC on >> Thursday anymore. My possible time slot is from 13:00-15:00 UTC >> Tuesday-Thursday. >> >> Please propose a better time frame for our team meeting. >> >> Sincerely, >> >> -- >> *Trinh Nguyen* >> *www.edlab.xyz * >> >> -- *Trinh Nguyen* *www.edlab.xyz * -------------- next part -------------- An HTML attachment was scrubbed... URL: From openstack at nemebean.com Tue Jul 16 16:24:55 2019 From: openstack at nemebean.com (Ben Nemec) Date: Tue, 16 Jul 2019 11:24:55 -0500 Subject: [oslo] PTG Planning Etherpad Message-ID: <99952df7-e9dd-6fcb-9607-44d1f4c262e2@nemebean.com> Hi, I've created a planning etherpad for Shanghai: https://etherpad.openstack.org/p/oslo-shanghai-topics We need to know by Aug. 11 if we want to request PTG space this time around so when you find out if you're going please add yourself to the list on the etherpad. There's also the usual space for topics if you know of anything we need to discuss already. Thanks. -Ben From rraja at redhat.com Mon Jul 15 11:19:50 2019 From: rraja at redhat.com (Ramana Venkatesh Raja) Date: Mon, 15 Jul 2019 16:49:50 +0530 Subject: [Manila] CephFS deferred deletion In-Reply-To: <20190712131540.3eqvltysfix6eivd@barron.net> References: <20190712131540.3eqvltysfix6eivd@barron.net> Message-ID: On Fri, Jul 12, 2019 at 6:45 PM Tom Barron wrote: > > On 12/07/19 13:03 +0000, Jose Castro Leon wrote: > >Dear all, > > > >Lately, one of our clients stored 300k files in a manila cephfs share. > >Then he deleted the share in Manila. This event make the driver > >unresponsive for several hours until all the data was removed in the > >cluster. > > > >We had a quick look at the code in manila [1] and the deletion is done > >first by calling the following api calls in the ceph bindings > >(delete_volume[1] and then purge_volume[2]). The first call moves the > >directory to a volumes_deleted directory. The second call does a > >deletion in depth of all the contents of that directory. > > > >The last operation is the one that trigger the issue. > > > >We had a similar issue in the past in Cinder. There, Arne proposed to > >do a deferred deletion of volumes. I think we could do the same in > >Manila for the cephfs driver. > > > >The idea is to continue to call to the delete_volume. And then inside a > >periodic task in the driver, asynchronously it will get the contents of > >that directory and trigger the purge command. > > > >I can propose the change and contribute with the code, but before going > >to deep I would like to know if there is a reason of having a singleton > >for the volume_client connection. If I compare with cinder code the > >connection is established and closed in each operation with the > >backend. > > > >If you are not the maintainer, could you please point me to he/she? > >I can post it in the mailing list if you prefer > > > >Cheers > >Jose Castro Leon > >CERN Cloud Infrastructure > > > >[1] > >https://github.com/openstack/manila/blob/master/manila/share/drivers/cephfs/driver.py#L260-L267 > > > > > >[2] > >https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L700-L734 > > > > > >[2] > >https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L736-L790 > > > > > >PS: The issue was triggered by one of our clients in kubernetes using > >the Manila CSI driver > > Hi Jose, > > Let's get this fixed since there's a lot of interest in Manila CSI > driver and I think we can expect more batched deletes with it than we > have had historically. The plan is to have manila's CephFS driver use the ceph-mgr's new volumes module, https://github.com/ceph/ceph/blob/master/src/pybind/mgr/volumes/module.py to create/delete manila groups/shares/snapshots, authorize/de-authorize access to the shares. Manila shares, essentially CephFS subdirectories with a specific data layout and quota, are referred to as FS subvolumes, and Ceph filesystems as FS volumes in the ceph-mgr volumes module. The ceph-mgr volumes modules is under active development. The latest Ceph CSI (v1.1.0) release is the first consumer of this module. The Ceph CSI issues CLI calls to the ceph-mgr to manage the lifecycle of the FS subvolumes, https://github.com/ceph/ceph-csi/pull/400 We're implementing the asynchronous purge of FS subvolumes in the ceph-mgr module. The PR is close to being merged, https://github.com/ceph/ceph/pull/28003/ https://github.com/ceph/ceph/pull/28003/commits/483a2141fe8c9a58bc25a544412cdf5b047ad772 http://tracker.ceph.com/issues/40036 Additional reviews will be great. Issuing the `ceph fs subvolume rm` command in the Ceph CSI driver (and later in the manila driver) will move the FS subvolume to a trash directory, whose contents will be asynchronously purged by a set of worker threads. > > I've copied Ramana Raja and Patrick Donnelly since they will be able > to answer your question about the singleton volume_client connection > more authoritatively than I can. Currently, in the mgr-volumes module we establish and close connection to a FS volume (a Ceph filesystem) for each FS subvolume (CephFS subdirectory within the filesystem) operation, https://github.com/ceph/ceph/pull/28082/commits/8d29816f0f3db6c7d287bbb7469db77c9de701d1#diff-cfd3b6f517caccc18f7f066395e8a4bdR174 Instead, we want to maintain a connection to a FS volume and perform operations on its subvolumes, until the FS volume is deleted. This would reduce the time taken to perform subvolume operations, important in CSI work loads (and in OpenStack workloads?). The code is in review, https://github.com/ceph/ceph/pull/28003/commits/5c41e949af9acabd612b0644de0603e374b4b42a Thanks, Ramana > > Thanks for volunteering to propose a review to deal with this issue! > > -- Tom Barron > From bansalnehal26 at gmail.com Mon Jul 15 11:35:22 2019 From: bansalnehal26 at gmail.com (Nehal Bansal) Date: Mon, 15 Jul 2019 17:05:22 +0530 Subject: [GNOCCHI] Regarding wrong ports in gnocchi-api and gnocchi-common packages Message-ID: Hi, I am installing ceilometer queens version. While installing gnocchi-api, gnocchi-metricd and python-gnocchiclient packages I encountered the error Registering service and endpoints for gnocchi with type metric at http://192.168.133.81:8041 Failed to discover available identity versions when contacting http://127.0.0.1:35357/v3/. Attempting to parse version from URL. Unable to establish connection to http://127.0.0.1:35357/v3/auth/tokens: HTTPConnectionPool(host='127.0.0.1', port=35357): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) dpkg: error processing package gnocchi-api (--configure): subprocess installed post-installation script returned error exit status 1 Errors were encountered while processing: gnocchi-api E: Sub-process /usr/bin/dpkg returned an error code (1) I checked the postinst scripts of the packages and found that port 35357 has been configured instead of 5000. Therefore, I fetched the deb packages separately and edited the postinst files of the packages, repackaged it and installed the packages with dpkg -i . Now the logs are (Reading database ... 154597 files and directories currently installed.) Preparing to unpack .../python3-gnocchi_4.2.5-0ubuntu1~cloud0_all.deb ... Unpacking python3-gnocchi (4.2.5-0ubuntu1~cloud0) over (4.2.5-0ubuntu1~cloud0) ... Setting up python3-gnocchi (4.2.5-0ubuntu1~cloud0) ... 2019-07-15 16:48:48,434 - INFO - executing "dpkg -i /var/cache/apt/archives/packages/fixed-gnocchi-common.deb" ... (Reading database ... 154597 files and directories currently installed.) Preparing to unpack .../fixed-gnocchi-common.deb ... Unpacking gnocchi-common (4.2.5-0ubuntu1~cloud0) over (4.2.5-0ubuntu1~cloud0) ... Setting up gnocchi-common (4.2.5-0ubuntu1~cloud0) ... 2019-07-15 16:48:49,324 - INFO - executing "dpkg -i /var/cache/apt/archives/packages/fixed-gnocchi-api.deb" ... Selecting previously unselected package gnocchi-api. (Reading database ... 154597 files and directories currently installed.) Preparing to unpack .../packages/fixed-gnocchi-api.deb ... Unpacking gnocchi-api (4.2.5-0ubuntu1~cloud0) ... Setting up gnocchi-api (4.2.5-0ubuntu1~cloud0) ... Registering service and endpoints for gnocchi with type metric at http://192.168.133.81:8041 Service already registered: skipping service endpoint creation. apache2_invoke: Enable site gnocchi-api.conf 2019-07-15 16:49:01,438 - INFO - executing "dpkg -i /var/cache/apt/archives/packages/gnocchi-metricd_4.2.5-0ubuntu1~cloud0_all.deb" ... Selecting previously unselected package gnocchi-metricd. (Reading database ... 154601 files and directories currently installed.) Preparing to unpack .../gnocchi-metricd_4.2.5-0ubuntu1~cloud0_all.deb ... Unpacking gnocchi-metricd (4.2.5-0ubuntu1~cloud0) ... Setting up gnocchi-metricd (4.2.5-0ubuntu1~cloud0) ... Processing triggers for systemd (229-4ubuntu21.22) ... Processing triggers for ureadahead (0.100.0-19.1) ... 2019-07-15 16:49:02,416 - INFO - executing "dpkg -i /var/cache/apt/archives/packages/python-gnocchiclient_7.0.1-0ubuntu1~cloud0_all.deb" ... Selecting previously unselected package python-gnocchiclient. (Reading database ... 154606 files and directories currently installed.) Preparing to unpack .../python-gnocchiclient_7.0.1-0ubuntu1~cloud0_all.deb ... Unpacking python-gnocchiclient (7.0.1-0ubuntu1~cloud0) ... Setting up python-gnocchiclient (7.0.1-0ubuntu1~cloud0) ... update-alternatives: using /usr/bin/python2-gnocchi to provide /usr/bin/gnocchi (gnocchi) in auto mode The service gnocchi-metricd gets created but there are no signs of gnocchi-api in /etc/init.d/. Kindly, tell me how to proceed. Thank you. Regards, Nehal -------------- next part -------------- An HTML attachment was scrubbed... URL: From merlin.blom at bertelsmann.de Mon Jul 15 12:38:35 2019 From: merlin.blom at bertelsmann.de (Blom, Merlin, NMU-OI) Date: Mon, 15 Jul 2019 12:38:35 +0000 Subject: [telemetry] Gnocchi: Aggregates Operation Syntax Message-ID: Hey, I would like to aggregate data from the gnocchi database by using the gnocchi aggregates function of the CLI/API The documentation does not cover the operations that are available nor the syntax that has to be used: https://gnocchi.xyz/gnocchiclient/shell.html?highlight=reaggregation#aggrega tes Searching for more information I found a GitHub Issue: https://github.com/gnocchixyz/gnocchi/issues/393 But I cannot use the syntax from that ether. My use case: I want to aggregate the vcpus hours per month, vram hours per month, . per server or project. - when an instance is stopped only storage is counted - the exact usage is used e.g. 2 vcpus between 1st and 7th day 4vcpus between 8th and last month no mean calculations Do you have detailed documentation about the gnocchi Aggregates Operation Syntax? Do you have complex examples for gnocchi aggregations? Especially when using the python bindings: conn_gnocchi.metric.aggregation(metrics="memory", query=[XXXXXXXX], resource_type='instance', groupby='original_resource_id') Can you give me advice regarding my use case? Do's and don'ts. Thank you for your help in advance! Merlin Blom -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5195 bytes Desc: not available URL: From zhenhua_huang at trendmicro.com Tue Jul 16 09:47:17 2019 From: zhenhua_huang at trendmicro.com (zhenhua_huang at trendmicro.com) Date: Tue, 16 Jul 2019 09:47:17 +0000 Subject: Linux bridge agent Agent out of sync with plugin! Message-ID: Hello, I am a newcomer to openstack. I have a problem. When I enter the command "openstack network agent list" on the controller node, the compute node is not found. Below are the output of the compute node's and controller node's linuxbridge-agent.log.look forward to your reply. [cid:image002.png at 01D53BFE.57DAB2A0][cid:image002.png at 01D53BFE.57DAB2A0] compute node [cid:image003.png at 01D53BFE.57DAB2A0][cid:image003.png at 01D53BFE.57DAB2A0] controller node 
TREND MICRO EMAIL NOTICE
The information contained in this email and any attachments is confidential 
and may be subject to copyright or other intellectual property protection. 
If you are not the intended recipient, you are not authorized to use or 
disclose this information, and we request that you notify us by reply mail or
telephone and delete the original message from your mail system.

For details about what personal information we collect and why, please see our Privacy Notice on our website at: [ https://www.trendmicro.com/privacy]
-------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 216868 bytes Desc: image002.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.png Type: image/png Size: 88687 bytes Desc: image003.png URL: From aj at suse.com Tue Jul 16 16:59:14 2019 From: aj at suse.com (Andreas Jaeger) Date: Tue, 16 Jul 2019 18:59:14 +0200 Subject: [docs][tc][infra] what to do with developer.openstack.org and api-site? In-Reply-To: <38f296e3-0292-76da-93af-2db554d918e3@suse.com> References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <38f296e3-0292-76da-93af-2db554d918e3@suse.com> Message-ID: <41d9ff37-e76b-b9a6-629a-a00f578a8d07@suse.com> On 16/07/2019 16.17, Andreas Jaeger wrote: > From the feedback so far, let me refine the proposal: > * Move OpenStack API-Guide to openstack-manuals repo and publish to docs.o.o > * Kill "Writing Your First OpenStack Application" document > * Publish all in-tree api-references and api-guides on > docs.openstack.org instead of developer.o.o > * List SDKs and CLIs on the software page > * Add redirects on developer.openstack.org for moved documents > * Once the above is all done, kill api-site (or leave it up for some > time in frozen state with redirects). > > I can take care of most steps (exception is software page - this is for > Thierry, correct?), I've started pushing changes with topic retire-api-site, https://review.opendev.org/#/q/topic:retire-api-site Will WIP for two days, Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF: Nils Brauckmann, Felix Imendörffer, Enrica Angelone, HRB 247165 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From aj at suse.com Tue Jul 16 17:42:30 2019 From: aj at suse.com (Andreas Jaeger) Date: Tue, 16 Jul 2019 19:42:30 +0200 Subject: [docs][tc][infra][uc][sdk] what to do with developer.openstack.org and api-site? In-Reply-To: <41d9ff37-e76b-b9a6-629a-a00f578a8d07@suse.com> References: <48048bf0-a79c-6abf-b88f-a1132afc0d6b@suse.com> <38f296e3-0292-76da-93af-2db554d918e3@suse.com> <41d9ff37-e76b-b9a6-629a-a00f578a8d07@suse.com> Message-ID: <5b9db20e-1d75-8eaa-060d-fdd1f71d4d48@suse.com> Let me add some more tags and cross-post to user-committee and reword. The api-site repo is orphaned, it is used to publish the content of developer.openstack.org. I propose now the following actions: * Move OpenStack API-Guide to openstack-manuals repo and publish to docs.o.o (currently published at https://developer.openstack.org/api-guide/quick-start/) * Add a redirect for this on developer.o.o * Kill the "Writing Your First OpenStack Application" document, since it's completely dead and outdated (last non-trivial change was 22nd February 2017). This is currently published to https://developer.openstack.org/firstapp-libcloud/ and https://developer.openstack.org/firstapp-shade/ * Publish all api-guide and api-ref documents to docs.openstack.org, add redirects for them on developer.openstack.org * List SDKs and CLIs on the software page on www.openstack.org (Thierry plans to do this) * Remove landing page from developer.o.o, add redirects. * After some time, retire developer.o.o and api-site repo completely. I've started pushing changes with topic retire-api-site, https://review.opendev.org/#/q/topic:retire-api-site Will WIP until Monday, 22nd of July. If any team wants to adopt api-site completely, or parts of it, please speak up, happy to help a hand-over - or wait a bit longer if further discussion is needed, Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF: Nils Brauckmann, Felix Imendörffer, Enrica Angelone, HRB 247165 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From jimmy at openstack.org Tue Jul 16 18:40:10 2019 From: jimmy at openstack.org (Jimmy McArthur) Date: Tue, 16 Jul 2019 13:40:10 -0500 Subject: Community Voting is now Open! Message-ID: <5D2E1A0A.8060406@openstack.org> Community voting for the Open Infrastructure Summit Shanghai sessions is open! You can VOTE HERE , but what does that mean? Now that the Call for Presentations has closed, all submissions are available for community vote and input. After community voting closes, the volunteer Programming Committee members will receive the results to review to help them determine the final selections for Summit schedule. While community votes are meant to help inform the decision, Programming Committee members are expected to exercise judgment in their area of expertise and help ensure diversity of sessions and speakers. View full details of the session selection process . In order to vote, you need an OSF community membership. If you do not have an account, please create one by going to openstack.org/join . If you need to reset your password, you can do that here . Hurry, voting closes Monday, July 22 at 11:59pmPacific Time (Tuesday, July 23 at 6:59 UTC). Continue to visit https://www.openstack.org/summit/shanghai-2019 for all Summit-related information. REGISTER Register for the Summit before prices increase on August 7th! VISA APPLICATION PROCESS Make sure to secure your Visa soon. More information about the Visa application process. TRAVEL SUPPORT PROGRAM August 8th is the last day to submit applications. Please submit your applications by 11:59pm Pacific Time (August 9th at 6:59am UTC). If you have any questions, please email summit at openstack.org . Cheers, Jimmy -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Tue Jul 16 20:13:44 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Tue, 16 Jul 2019 22:13:44 +0200 Subject: Linux bridge agent Agent out of sync with plugin! In-Reply-To: References: Message-ID: Hi, What exactly do You mean by saying “the compute node is not found”? Is neutron-linuxbridge-agent from compute node not visible in “neutron agent-list” result or it is visible but down? Or something else? Also, please send logs as plain text instead of screenshots - it will be much easier to read :) > On 16 Jul 2019, at 11:47, zhenhua_huang at trendmicro.com wrote: > > Hello, I am a newcomer to openstack. I have a problem. > When I enter the command "openstack network agent list" on the controller node, the compute node is not found. Below are the output of the compute node's and controller node’s linuxbridge-agent.log.look forward to your reply. > > > > compute node > > > controller node > > TREND MICRO EMAIL NOTICE > The information contained in this email and any attachments is confidential > and may be subject to copyright or other intellectual property protection. > If you are not the intended recipient, you are not authorized to use or > disclose this information, and we request that you notify us by reply mail or > telephone and delete the original message from your mail system. > > For details about what personal information we collect and why, please see our Privacy Notice on our website at: [ > https://www.trendmicro.com/privacy > ] > — Slawek Kaplonski Senior software engineer Red Hat From li.canwei2 at zte.com.cn Wed Jul 17 05:24:59 2019 From: li.canwei2 at zte.com.cn (li.canwei2 at zte.com.cn) Date: Wed, 17 Jul 2019 13:24:59 +0800 (CST) Subject: =?UTF-8?B?W1dhdGNoZXJdIHRlYW0gbWVldGluZyAgYXQgMDg6MDAgVVRDIHRvZGF5?= Message-ID: <201907171324592577754@zte.com.cn> Hi all, Watcher team will have a meeting at 08:00 UTC today in the #openstack-meeting-alt channel. The agenda is available on https://wiki.openstack.org/wiki/Watcher_Meeting_Agenda feel free to add any additional items. Thanks! Canwei Li -------------- next part -------------- An HTML attachment was scrubbed... URL: From manulachathurika at gmail.com Wed Jul 17 05:37:31 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Wed, 17 Jul 2019 11:07:31 +0530 Subject: ImportError: No module named django.core.wsgi - DevStack - Stable/Stein - Ubuntu 18.04 In-Reply-To: References: Message-ID: Hi Ruslanas, Thank you very much. I'll try this option as well and let you know. Thanks ! On Tue, Jul 16, 2019 at 4:30 PM Ruslanas Gžibovskis wrote: > I had something similar, try reinstalling python from OSP repo, not > common, as it rewrites some python lib file... at least it helped me. > > On Tue, 16 Jul 2019 at 12:05, Manula Thantriwatte < > manulachathurika at gmail.com> wrote: > >> Hi All, >> >> I have successfully install DevStack in Ubuntu 18.04. But when I'm >> accessing the dashboard I'm getting 500 error. What I'm getting >> in horizon_error.log is, >> >> 2019-07-15 14:10:43.218296 mod_wsgi (pid=31763): Target WSGI script >> '/opt/stack/horizon/openstack_dashboard/wsgi.py' cannot be loaded as Python >> module. >> 2019-07-15 14:10:43.218323 mod_wsgi (pid=31763): Exception occurred >> processing WSGI script '/opt/stack/horizon/openstack_dashboard/wsgi.py'. >> 2019-07-15 14:10:43.218349 Traceback (most recent call last): >> 2019-07-15 14:10:43.218370 File >> "/opt/stack/horizon/openstack_dashboard/wsgi.py", line 21, in >> 2019-07-15 14:10:43.218401 from django.core.wsgi import >> get_wsgi_application >> 2019-07-15 14:10:43.218422 ImportError: No module named django.core.wsgi >> >> Python Version is : 3.6.8 >> Django version is : 2.0.5 >> >> I tried with uninstalling Djanago and reinstalling it. But it didn't work >> for me. >> >> Can someone help me on how to resole this issue ? >> >> Thanks ! >> -- >> Regards, >> Manula Chathurika Thantriwatte >> phone : (+94) 772492511 >> email : manulachathurika at gmail.com >> Linkedin : *http://lk.linkedin.com/in/manulachathurika >> * >> blog : http://manulachathurika.blogspot.com/ >> >> > > -- > Ruslanas Gžibovskis > +370 6030 7030 > -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From renat.akhmerov at gmail.com Wed Jul 17 06:30:25 2019 From: renat.akhmerov at gmail.com (Renat Akhmerov) Date: Wed, 17 Jul 2019 13:30:25 +0700 Subject: [Mistral] Guaranteed notification delivery In-Reply-To: <13379051562307593@iva3-4a4d8f90d111.qloud-c.yandex.net> References: <13379051562307593@iva3-4a4d8f90d111.qloud-c.yandex.net> Message-ID: <8970b413-9ef1-46ef-ae6c-e5c58770b7e2@Spark> Hi Artem, Ok, we’ll look at it. Thanks Renat Akhmerov @Nokia On 5 Jul 2019, 19:21 +0700, Лапин Артем , wrote: > Good day, please look at https://blueprints.launchpad.net/mistral/+spec/guaranteed-notifies. I would like to know your opinion about this feature. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anlin.kong at gmail.com Wed Jul 17 06:35:08 2019 From: anlin.kong at gmail.com (Lingxian Kong) Date: Wed, 17 Jul 2019 18:35:08 +1200 Subject: [devstack] [trove] Enabling other datastores, plugin variable design less than tidy In-Reply-To: References: Message-ID: Hi Mark, I agree with all your suggestions, do you mind proposing a patch to fix that? Best regards, Lingxian Kong Catalyst Cloud On Tue, Jul 16, 2019 at 11:26 AM Mark Kirkwood < mark.kirkwood at catalyst.net.nz> wrote: > Hi, > > I'm doing some work with the Trove Postgres datastore. So the first > thing I did was attempt to get Devstack to set up a stack with Postgres > instead of Mysql. > > Reading the docs, it seemed that all I needed to do was this: > > $ cat local.conf > > [[local|localrc]] > ADMIN_PASSWORD=password > DATABASE_PASSWORD=$ADMIN_PASSWORD > RABBIT_PASSWORD=$ADMIN_PASSWORD > SERVICE_PASSWORD=$ADMIN_PASSWORD > TROVE_DATASTORE_TYPE=postgresql > TROVE_DATASTORE_VERSION=9.6 > TROVE_DATASTORE_PACKAGE=postgresql-9.6 > enable_plugin trove https://opendev.org/openstack/trove > > Wrong! After watching the plugin try to build a Mysql guest image of > version 9.6 (!), I realized that more reading of the plugin source was > required. So iteration 2 (or maybe it was 3...lol), of my local.conf is: > > [[local|localrc]] > ADMIN_PASSWORD=password > DATABASE_PASSWORD=$ADMIN_PASSWORD > RABBIT_PASSWORD=$ADMIN_PASSWORD > SERVICE_PASSWORD=$ADMIN_PASSWORD > TROVE_DATASTORE_TYPE=postgresql > TROVE_DATASTORE_VERSION=9.6 > TROVE_DATASTORE_PACKAGE=postgresql-9.6 > SERVICE_TYPE=$TROVE_DATASTORE_TYPE > DATASTORE_VERSION=$TROVE_DATASTORE_VERSION > VM=/opt/stack/images/ubuntu_postgresql/ubuntu_postgresql > enable_plugin trove https://opendev.org/openstack/trove > > This works. However, it seems like those last 3 variable substitutions > should not be required. i.e: > > - SERVICE_TYPE should not exist (we should use TROVE_DATASTORE_TYPE) > > - DATASTORE_VERSION should not exist (we should use > TROVE_DATASTORE_VERSION) > > - VM should be constructed out of DISTRO and TROVE_DATASTORE_TYPE > > Thoughts? > > regards > > Mark > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From km.giuseppesannino at gmail.com Wed Jul 17 07:41:28 2019 From: km.giuseppesannino at gmail.com (Giuseppe Sannino) Date: Wed, 17 Jul 2019 09:41:28 +0200 Subject: [kolla][nova][neutron] Access to VMs is slow when running on a remote compute host In-Reply-To: References: <01AA2BA1-3EEC-4142-BCDB-F9861707102E@redhat.com> <9f46833c423d5e87a19e93ad77f7999e44cc5268.camel@redhat.com> Message-ID: Hi Radoslaw, all, I applied the GSSAPIAuthentication setting along with the UseDNS one. No luck. One thing I want to share and that maybe goes in a "no network" direction. I disabled the execution of the motd script during the login phase via "chmod -x /etc/update-motd.d/*". I'm able to reduce the login time from ~12secs down to ~3sec. To me, it looks like the issue is with the access in RW to the Guest VM filesystem which takes quite a while. One more thing, in the tcpdump trace, during the SSH login, I can see that when the procedure gets stuck (that pledge network... log in ssh) is the Server (so the Guest OS) that takes time to reply with a "Server packet. But, I can see a TCP ACK sent back from the server. This makes me quite sure that from a networking point of view the things are handled with no delay. /G On Mon, 15 Jul 2019 at 19:27, Radosław Piliszek wrote: > For completeness - always also ensure that 'GSSAPIAuthentication' is set > to 'no' because in default config it might require DNS lookups too. > (Obviously you can run GSSAPIAuthentication and avoid DNS lookups by > configuring GSSAPI appropriately ;-) ). > > Kind regards, > Radek > > pon., 15 lip 2019 o 19:14 Giuseppe Sannino > napisał(a): > >> Hi Alex, >> yeah, it was the first suspect also based on the various research on >> internet. >> I currently have the "useDNS" set to no but still the issue persists. >> >> /G >> >> On Mon, 15 Jul 2019 at 18:46, Alex Schultz wrote: >> >>> >>> On Mon, Jul 15, 2019 at 10:40 AM Giuseppe Sannino < >>> km.giuseppesannino at gmail.com> wrote: >>> >>>> Hi Sean, >>>> the ssh to localhost is slow as well. >>>> "telnet localhost" is also slow. >>>> >>>> >>> Are you having dns issues? Historically if you have UseDNS set to true >>> and your dns servers are bad it can just be slow to connect as it tries to >>> do the reverse lookup. >>> >>> >>>> /Giuseppe >>>> >>>> On Mon, 15 Jul 2019 at 18:18, Sean Mooney wrote: >>>> >>>>> On Mon, 2019-07-15 at 16:29 +0200, Giuseppe Sannino wrote: >>>>> > Hi! >>>>> > first of all, thanks for the fast replies. I do appreciate that. >>>>> > >>>>> > I did some more test trying to figure out the issue. >>>>> > - Set UseDNS to "no" in sshd_config => Issue persists >>>>> > - Installed and configured Telnet => Telnet login is slow as well >>>>> > >>>>> > From the "top" or "auth.log"nothing specific popped up. I can sshd >>>>> taking >>>>> > some cpu for a short while but nothing more than that. >>>>> > >>>>> > Once logged in the VM is not too slow. CLI doesn't get stuck or >>>>> similar. >>>>> > One thing worthwhile to mention, it seems like the writing >>>>> throughput on >>>>> > the disk is a bit slow: 67MB/s wrt around 318MB/s of another VM >>>>> running on >>>>> > a "datacenter" Openstack installation. >>>>> unless you see iowait in the guest its likely not related to the disk >>>>> speed. >>>>> you might be able to improve the disk performace by changeing the >>>>> chache mode >>>>> but unless you are seeing io wait that is just an optimisation to try >>>>> later. >>>>> >>>>> when you are logged into the vm have you tried ssh again via localhost >>>>> to >>>>> determin if the long login time is related to the network or the vm. >>>>> >>>>> if its related to the network it will be fast over localhost >>>>> if its related to the vm, e.g. because of disk, cpu load, memory load >>>>> or ssh server configuration >>>>> then the local ssh will be slow. >>>>> >>>>> > >>>>> > The Cinder Volume docker is running on the Compute Host and Cinder >>>>> is using >>>>> > the filesystem as backend. >>>>> > >>>>> > BR >>>>> > /Giuseppe >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > On Fri, 12 Jul 2019 at 17:41, Slawek Kaplonski >>>>> wrote: >>>>> > >>>>> > > Hi, >>>>> > > >>>>> > > I suspect some problems with names resolving. Can You check if You >>>>> have >>>>> > > also such delay when doing e.g. “sudo” commands after You ssh to >>>>> the >>>>> > > instance? >>>>> > > >>>>> > > > On 12 Jul 2019, at 16:23, Brian Haley >>>>> wrote: >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > On 7/12/19 9:57 AM, Giuseppe Sannino wrote: >>>>> > > > > Hi community, >>>>> > > > > I need your help ,tips, advices. >>>>> > > > > *> Environment <* >>>>> > > > > I have deployed Openstack "Stein" using the latest >>>>> kolla-ansible on the >>>>> > > >>>>> > > following deployment topology: >>>>> > > > > 1) OS Controller running as VM on a "cloud" location >>>>> > > > > 2) OS Compute running on a baremetal server remotely (wrt OS >>>>> > > >>>>> > > Controller) location >>>>> > > > > 3) Network node running on the Compute host >>>>> > > > > As per the above info, Controller and compute run on two >>>>> different >>>>> > > >>>>> > > networks. >>>>> > > > > Kolla-Ansible is not really designed for such scenario but >>>>> after >>>>> > > >>>>> > > manipulating the globals.yml and the inventory files (basically I >>>>> had to >>>>> > > move node specific network settings from the globals to the >>>>> inventory >>>>> > > file), eventually the deployment works fine. >>>>> > > > > *> Problem <* >>>>> > > > > I have no specific issue working with this deployment except >>>>> the >>>>> > > >>>>> > > following: >>>>> > > > > "SSH connection to the VM is quite slow". >>>>> > > > > It takes around 20 seconds for me to log into the VM (Ubuntu, >>>>> CentOS, >>>>> > > >>>>> > > whatever). >>>>> > > > >>>>> > > > But once logged-in things are OK? For example, an scp stalls >>>>> the same >>>>> > > >>>>> > > way, but the transfer is fast? >>>>> > > > >>>>> > > > > *> Observations <* >>>>> > > > > * Except for the slowness during the SSH login, I don't have >>>>> any >>>>> > > > > further specific issue working with this envirorment >>>>> > > > > * With the Network on the Compute I can turn the OS >>>>> controller off >>>>> > > > > with no impact on the VM. Still the connection is slow >>>>> > > > > * I tried different type of images (Ubuntu, CentOS, Windows) >>>>> always >>>>> > > > > with the same result. >>>>> > > > > * SSH connection is slow even if I try to login into the VM >>>>> within the >>>>> > > > > IP Namespace >>>>> > > > > From the ssh -vvv, I can see that the authentication gets >>>>> stuck here: >>>>> > > > > debug1: Authentication succeeded (publickey). >>>>> > > > > Authenticated to ***** >>>>> > > > > debug1: channel 0: new [client-session] >>>>> > > > > debug3: ssh_session2_open: channel_new: 0 >>>>> > > > > debug2: channel 0: send open >>>>> > > > > debug3: send packet: type 90 >>>>> > > > > debug1: Requesting no-more-sessions at openssh.com >>>> > > >>>>> > > no-more-sessions at openssh.com> >>>>> > > > > debug3: send packet: type 80 >>>>> > > > > debug1: Entering interactive session. >>>>> > > > > debug1: pledge: network >>>>> > > > > > > > > > 10 to 15 seconds later >>>>> > > > >>>>> > > > What is sshd doing at this time? Have you tried enabling debug >>>>> or >>>>> > > >>>>> > > running tcpdump when a new connection is attempted? At first >>>>> glance I'd >>>>> > > say it's a DNS issue since it eventually succeeds, the logs would >>>>> help to >>>>> > > point in a direction. >>>>> > > > >>>>> > > > -Brian >>>>> > > > >>>>> > > > >>>>> > > > > debug3: receive packet: type 80 >>>>> > > > > debug1: client_input_global_request: rtype >>>>> hostkeys-00 at openssh.com >>>>> > > >>>>> > > want_reply 0 >>>>> > > > > debug3: receive packet: type 91 >>>>> > > > > debug2: callback start >>>>> > > > > debug2: fd 3 setting TCP_NODELAY >>>>> > > > > debug3: ssh_packet_set_tos: set IP_TOS 0x10 >>>>> > > > > debug2: client_session2_setup: id 0 >>>>> > > > > debug2: channel 0: request pty-req confirm 1 >>>>> > > > > Have you ever experienced such issue ? >>>>> > > > > Any suggestion? >>>>> > > > > Many thanks >>>>> > > > > /Giuseppe >>>>> > > >>>>> > > — >>>>> > > Slawek Kaplonski >>>>> > > Senior software engineer >>>>> > > Red Hat >>>>> > > >>>>> > > >>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alessio.Fioravanti at konicaminolta.it Wed Jul 17 08:05:11 2019 From: Alessio.Fioravanti at konicaminolta.it (Fioravanti, Alessio) Date: Wed, 17 Jul 2019 08:05:11 +0000 Subject: [kolla] [freezer] freezer scheduler auth_url error Message-ID: <1563350711224.53650@konicaminolta.it> Hi, this is my first post here. I'm experiencing a problem using freezer-scheduler component. When I create a job and schedule it from Horizon, freezer-scheduler always fails. The error shown in freezer-scheduler.log (from fluentd container) is: 2019-07-17 07:48:23.294 6 WARNING freezer.scheduler.scheduler_job [-] Job 1754ea4157b94acb916f8b9911886792 failed backup action, retrying in 60 seconds 2019-07-17 07:49:24.342 6 ERROR freezer.scheduler.scheduler_job [-] Freezer client error: Critical Error: auth_url required to authenticate. Make sure to export OS_AUTH_URL=http://keystone_url:5000/v3 The instance has been deployed with Kolla. Here some details: Environment kolla version: 8.0.1rc1 openstack release: stein base container: centos host os: ubuntu 18.04 environment: 1 controller on a dedicated host + 1 compute on a dedicated host Can you please give me your help? Thanks, Alessio -------------- next part -------------- An HTML attachment was scrubbed... URL: From amotoki at gmail.com Wed Jul 17 08:33:20 2019 From: amotoki at gmail.com (Akihiro Motoki) Date: Wed, 17 Jul 2019 17:33:20 +0900 Subject: ImportError: No module named django.core.wsgi - DevStack - Stable/Stein - Ubuntu 18.04 In-Reply-To: References: Message-ID: While I don't see any issue around devstack horizon with python3, I see some points to be checked. - Do you use USE_PYTHON3 in local.conf? - Is the right wsgi module for apache installed? - Can you import django.core.wsgi with right python3 interpreter manually? The following is an example in my environment. $ which python3 /usr/bin/python3 $ python3 manage.py shell Python 3.6.8 (default, Jan 14 2019, 11:02:34) [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> from django.core.wsgi import get_wsgi_application >>> now exiting InteractiveConsole... $ cat /etc/apache2/mods-enabled/wsgi.load LoadModule wsgi_module /usr/lib/apache2/modules/mod_wsgi.so $ dpkg -S /usr/lib/apache2/modules/mod_wsgi.so libapache2-mod-wsgi-py3: /usr/lib/apache2/modules/mod_wsgi.so On Tue, Jul 16, 2019 at 7:04 PM Manula Thantriwatte wrote: > > Hi All, > > I have successfully install DevStack in Ubuntu 18.04. But when I'm accessing the dashboard I'm getting 500 error. What I'm getting in horizon_error.log is, > > 2019-07-15 14:10:43.218296 mod_wsgi (pid=31763): Target WSGI script '/opt/stack/horizon/openstack_dashboard/wsgi.py' cannot be loaded as Python module. > 2019-07-15 14:10:43.218323 mod_wsgi (pid=31763): Exception occurred processing WSGI script '/opt/stack/horizon/openstack_dashboard/wsgi.py'. > 2019-07-15 14:10:43.218349 Traceback (most recent call last): > 2019-07-15 14:10:43.218370 File "/opt/stack/horizon/openstack_dashboard/wsgi.py", line 21, in > 2019-07-15 14:10:43.218401 from django.core.wsgi import get_wsgi_application > 2019-07-15 14:10:43.218422 ImportError: No module named django.core.wsgi > > Python Version is : 3.6.8 > Django version is : 2.0.5 > > I tried with uninstalling Djanago and reinstalling it. But it didn't work for me. > > Can someone help me on how to resole this issue ? > > Thanks ! > -- > Regards, > Manula Chathurika Thantriwatte > phone : (+94) 772492511 > email : manulachathurika at gmail.com > Linkedin : http://lk.linkedin.com/in/manulachathurika > blog : http://manulachathurika.blogspot.com/ > From mark.kirkwood at catalyst.net.nz Wed Jul 17 09:08:28 2019 From: mark.kirkwood at catalyst.net.nz (Mark Kirkwood) Date: Wed, 17 Jul 2019 21:08:28 +1200 Subject: [devstack] [trove] Enabling other datastores, plugin variable design less than tidy In-Reply-To: References: Message-ID: <7198c31d-91fe-2020-3478-335682473e28@catalyst.net.nz> Sure, will do. On 17/07/19 6:35 PM, Lingxian Kong wrote: > Hi Mark, > > I agree with all your suggestions, do you mind proposing a patch to > fix that? > > Best regards, > Lingxian Kong > Catalyst Cloud > > > On Tue, Jul 16, 2019 at 11:26 AM Mark Kirkwood > > > wrote: > > Hi, > > I'm doing some work with the Trove Postgres datastore. So the first > thing I did was attempt to get Devstack to set up a stack with > Postgres > instead of Mysql. > > Reading the docs, it seemed that all I needed to do was this: > > $ cat local.conf > > [[local|localrc]] > ADMIN_PASSWORD=password > DATABASE_PASSWORD=$ADMIN_PASSWORD > RABBIT_PASSWORD=$ADMIN_PASSWORD > SERVICE_PASSWORD=$ADMIN_PASSWORD > TROVE_DATASTORE_TYPE=postgresql > TROVE_DATASTORE_VERSION=9.6 > TROVE_DATASTORE_PACKAGE=postgresql-9.6 > enable_plugin trove https://opendev.org/openstack/trove > > Wrong! After watching the plugin try to build a Mysql guest image of > version 9.6 (!), I realized that more reading of the plugin source > was > required. So iteration 2 (or maybe it was 3...lol), of my > local.conf is: > > [[local|localrc]] > ADMIN_PASSWORD=password > DATABASE_PASSWORD=$ADMIN_PASSWORD > RABBIT_PASSWORD=$ADMIN_PASSWORD > SERVICE_PASSWORD=$ADMIN_PASSWORD > TROVE_DATASTORE_TYPE=postgresql > TROVE_DATASTORE_VERSION=9.6 > TROVE_DATASTORE_PACKAGE=postgresql-9.6 > SERVICE_TYPE=$TROVE_DATASTORE_TYPE > DATASTORE_VERSION=$TROVE_DATASTORE_VERSION > VM=/opt/stack/images/ubuntu_postgresql/ubuntu_postgresql > enable_plugin trove https://opendev.org/openstack/trove > > This works. However, it seems like those last 3 variable > substitutions > should not be required. i.e: > > - SERVICE_TYPE should not exist (we should use TROVE_DATASTORE_TYPE) > > - DATASTORE_VERSION should not exist (we should use > TROVE_DATASTORE_VERSION) > > - VM should be constructed out of DISTRO and TROVE_DATASTORE_TYPE > > Thoughts? > > regards > > Mark > > From jacob.anders.au at gmail.com Wed Jul 17 09:43:54 2019 From: jacob.anders.au at gmail.com (Jacob Anders) Date: Wed, 17 Jul 2019 19:43:54 +1000 Subject: [ironic] Moving to office hours as opposed to weekly meetings for the next month In-Reply-To: References: Message-ID: Hi Julia, Do we have more clarity regarding the second (APAC) session? I see the polls have been open for some time, but haven't seen a mention of a specific time. Thank you, Jacob On Wed, Jul 3, 2019 at 9:39 AM Julia Kreger wrote: > Greetings Everyone! > > This week, during the weekly meeting, we seemed to reach consensus that > we would try taking a break from meetings[0] and moving to orienting > around using the mailing list[1] and our etherpad "whiteboard" [2]. > With this, we're going to want to re-evaluate in about a month. > I suspect it would be a good time for us to have a "mid-cycle" style > set of topical calls. I've gone ahead and created a poll to try and > identify a couple days that might be ideal for contributors[3]. > > But in the mean time, we want to ensure that we have some times for > office hours. The suggestion was also made during this week's meeting > that we may want to make the office hours window a little larger to > enable more discussion. > > So when will we have office hours? > ---------------------------------- > > Ideally we'll start with two time windows. One to provide coverage > to US and Europe friendly time zones, and another for APAC contributors. > > * I think 2-4 PM UTC on Mondays would be ideal. This translates to > 7-9 AM US-Pacific or 10 AM to 12 PM US-Eastern. > * We need to determine a time window that would be ideal for APAC > contributors. I've created a poll to help facilitate discussion[4]. > > So what is Office Hours? > ------------------------ > > Office hours are a time window when we expect some contributors to be > on IRC and able to partake in higher bandwidth discussions. > These times are not absolute. They can change and evolve, > and that is the most important thing for us to keep in mind. > > -- > > If there are any questions, Please let me know! > Otherwise I'll send a summary email out on next Monday. > > -Julia > > [0]: > http://eavesdrop.openstack.org/meetings/ironic/2019/ironic.2019-07-01-15.00.log.html#l-123 > [1]: > http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007038.html > [2]: https://etherpad.openstack.org/p/IronicWhiteBoard > [3]: https://doodle.com/poll/652gzta6svsda343 > [4]: https://doodle.com/poll/2ta5vbskytpntmgv > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Wed Jul 17 10:19:31 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Wed, 17 Jul 2019 12:19:31 +0200 Subject: Linux bridge agent Agent out of sync with plugin! In-Reply-To: <650a138d5ceb43a399c9a967a5ab52cd@trendmicro.com> References: <650a138d5ceb43a399c9a967a5ab52cd@trendmicro.com> Message-ID: Hi, Please continue this discussion on ML that anyone can see it :) What agents do You exactly see in “neutron agent-list” command then? What You have set as “host” value in linuxbridge agent’s config on different nodes? Can You enable DEBUG log and send it also? > On 17 Jul 2019, at 04:05, zhenhua_huang at trendmicro.com wrote: > > Hello: > > Thank you for reading my letter. .I mean that neutron-linuxbridge-agent from compute node is not visible in “neutron agent-list” result. > The compute node's linuxbridge-agent.log. is as follows: > 2019-07-16 02:12:48.328 20720 INFO neutron.plugins.ml2.drivers.agent._common_agent [-] Stopping Linux bridge agent agent. > 2019-07-16 02:12:50.468 21622 INFO neutron.common.config [-] Logging enabled! > 2019-07-16 02:12:50.468 21622 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 11.0.8 > 2019-07-16 02:12:50.468 21622 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'ens192'} > 2019-07-16 02:12:50.468 21622 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {} > 2019-07-16 02:12:50.485 21622 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Agent initialized successfully, now running... > 2019-07-16 02:12:50.865 21622 INFO oslo_rootwrap.client [req-7cca3135-d8f4-4156-98dd-9d123c492a85 - - - - -] Spawned new rootwrap daemon process with pid=21636 > 2019-07-16 02:12:50.906 21622 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-7cca3135-d8f4-4156-98dd-9d123c492a85 - - - - -] RPC agent_id: lb525400810624 > 2019-07-16 02:12:50.912 21622 INFO neutron.agent.agent_extensions_manager [req-7cca3135-d8f4-4156-98dd-9d123c492a85 - - - - -] Loaded agent extensions: [] > 2019-07-16 02:12:51.143 21622 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-7cca3135-d8f4-4156-98dd-9d123c492a85 - - - - -] Linux bridge agent Agent RPC Daemon Started! > 2019-07-16 02:12:51.143 21622 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-7cca3135-d8f4-4156-98dd-9d123c492a85 - - - - -] Linux bridge agent Agent out of sync with plugin! > 2019-07-16 02:12:51.154 21622 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect [req-7cca3135-d8f4-4156-98dd-9d123c492a85 - - - - -] Clearing orphaned ARP spoofing entries for devices [] > > > The controller node's linuxbridge-agent.log. is as follows: > 2019-07-16 21:37:32.127 15465 INFO neutron.plugins.ml2.drivers.agent._common_agent [-] Stopping Linux bridge agent agent. > 2019-07-16 21:37:32.138 15465 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=15491 > 2019-07-16 21:37:33.738 15657 INFO neutron.common.config [-] Logging enabled! > 2019-07-16 21:37:33.739 15657 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 11.0.8 > 2019-07-16 21:37:33.740 15657 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'ens192'} > 2019-07-16 21:37:33.740 15657 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {} > 2019-07-16 21:37:33.754 15657 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Agent initialized successfully, now running... > 2019-07-16 21:37:34.063 15657 INFO oslo_rootwrap.client [req-7df1f224-2dcd-4193-bcf1-d6c0b66f5c90 - - - - -] Spawned new rootwrap daemon process with pid=15683 > 2019-07-16 21:37:34.093 15657 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-7df1f224-2dcd-4193-bcf1-d6c0b66f5c90 - - - - -] RPC agent_id: lb525400be6607 > 2019-07-16 21:37:34.100 15657 INFO neutron.agent.agent_extensions_manager [req-7df1f224-2dcd-4193-bcf1-d6c0b66f5c90 - - - - -] Loaded agent extensions: [] > 2019-07-16 21:37:34.341 15657 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-7df1f224-2dcd-4193-bcf1-d6c0b66f5c90 - - - - -] Linux bridge agent Agent RPC Daemon Started! > 2019-07-16 21:37:34.342 15657 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-7df1f224-2dcd-4193-bcf1-d6c0b66f5c90 - - - - -] Linux bridge agent Agent out of sync with plugin! > 2019-07-16 21:37:34.350 15657 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect [req-7df1f224-2dcd-4193-bcf1-d6c0b66f5c90 - - - - -] Clearing orphaned ARP spoofing entries for devices [] > 2019-07-16 21:58:49.471 15657 INFO neutron.plugins.ml2.drivers.agent._common_agent [-] Stopping Linux bridge agent agent. > 2019-07-16 21:58:51.033 16466 INFO neutron.common.config [-] Logging enabled! > 2019-07-16 21:58:51.033 16466 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 11.0.8 > 2019-07-16 21:58:51.034 16466 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'ens192'} > 2019-07-16 21:58:51.034 16466 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {} > 2019-07-16 21:58:51.046 16466 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Agent initialized successfully, now running... > 2019-07-16 21:58:51.356 16466 INFO oslo_rootwrap.client [req-705ff1b1-c733-411f-a355-498369153e65 - - - - -] Spawned new rootwrap daemon process with pid=16492 > 2019-07-16 21:58:51.415 16466 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-705ff1b1-c733-411f-a355-498369153e65 - - - - -] RPC agent_id: lb525400be6607 > 2019-07-16 21:58:51.421 16466 INFO neutron.agent.agent_extensions_manager [req-705ff1b1-c733-411f-a355-498369153e65 - - - - -] Loaded agent extensions: [] > 2019-07-16 21:58:51.729 16466 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-705ff1b1-c733-411f-a355-498369153e65 - - - - -] Linux bridge agent Agent RPC Daemon Started! > 2019-07-16 21:58:51.730 16466 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-705ff1b1-c733-411f-a355-498369153e65 - - - - -] Linux bridge agent Agent out of sync with plugin! > 2019-07-16 21:58:51.739 16466 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect [req-705ff1b1-c733-411f-a355-498369153e65 - - - - -] Clearing orphaned ARP spoofing entries for devices [] > 2019-07-16 21:59:38.037 16466 INFO neutron.plugins.ml2.drivers.agent._common_agent [-] Stopping Linux bridge agent agent. > 2019-07-16 21:59:38.045 16466 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=16492 > 2019-07-16 21:59:39.679 16592 INFO neutron.common.config [-] Logging enabled! > 2019-07-16 21:59:39.679 16592 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 11.0.8 > 2019-07-16 21:59:39.679 16592 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'ens192'} > 2019-07-16 21:59:39.680 16592 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {} > 2019-07-16 21:59:39.694 16592 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Agent initialized successfully, now running... > 2019-07-16 21:59:40.043 16592 INFO oslo_rootwrap.client [req-eb16336e-e18d-4b80-b75c-718645f045c2 - - - - -] Spawned new rootwrap daemon process with pid=16618 > 2019-07-16 21:59:40.085 16592 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-eb16336e-e18d-4b80-b75c-718645f045c2 - - - - -] RPC agent_id: lb525400be6607 > 2019-07-16 21:59:40.092 16592 INFO neutron.agent.agent_extensions_manager [req-eb16336e-e18d-4b80-b75c-718645f045c2 - - - - -] Loaded agent extensions: [] > 2019-07-16 21:59:40.243 16592 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-eb16336e-e18d-4b80-b75c-718645f045c2 - - - - -] Linux bridge agent Agent RPC Daemon Started! > 2019-07-16 21:59:40.243 16592 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-eb16336e-e18d-4b80-b75c-718645f045c2 - - - - -] Linux bridge agent Agent out of sync with plugin! > 2019-07-16 21:59:40.254 16592 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect [req-eb16336e-e18d-4b80-b75c-718645f045c2 - - - - -] Clearing orphaned ARP spoofing entries for devices [] > 2019-07-16 22:02:15.863 16592 INFO neutron.plugins.ml2.drivers.agent._common_agent [-] Stopping Linux bridge agent agent. > 2019-07-16 22:02:15.867 16592 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=16618 > 2019-07-16 22:02:17.347 16795 INFO neutron.common.config [-] Logging enabled! > 2019-07-16 22:02:17.348 16795 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 11.0.8 > 2019-07-16 22:02:17.348 16795 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'ens192'} > 2019-07-16 22:02:17.348 16795 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {} > 2019-07-16 22:02:17.361 16795 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Agent initialized successfully, now running... > 2019-07-16 22:02:17.677 16795 INFO oslo_rootwrap.client [req-baf4316e-3d5a-41ce-a3cb-aece7104a568 - - - - -] Spawned new rootwrap daemon process with pid=16820 > 2019-07-16 22:02:17.707 16795 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-baf4316e-3d5a-41ce-a3cb-aece7104a568 - - - - -] RPC agent_id: lb525400be6607 > 2019-07-16 22:02:17.720 16795 INFO neutron.agent.agent_extensions_manager [req-baf4316e-3d5a-41ce-a3cb-aece7104a568 - - - - -] Loaded agent extensions: [] > 2019-07-16 22:02:17.908 16795 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-baf4316e-3d5a-41ce-a3cb-aece7104a568 - - - - -] Linux bridge agent Agent RPC Daemon Started! > 2019-07-16 22:02:17.908 16795 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-baf4316e-3d5a-41ce-a3cb-aece7104a568 - - - - -] Linux bridge agent Agent out of sync with plugin! > 2019-07-16 22:02:17.917 16795 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect [req-baf4316e-3d5a-41ce-a3cb-aece7104a568 - - - - -] Clearing orphaned ARP spoofing entries for devices [] > look forward to your reply. > -----Original Message----- > From: Slawek Kaplonski > Sent: 2019年7月17日 4:14 > To: Zhenhua Huang (RD-CN) > Cc: openstack-discuss at lists.openstack.org > Subject: Re: Linux bridge agent Agent out of sync with plugin! > > Hi, > > What exactly do You mean by saying “the compute node is not found”? Is neutron-linuxbridge-agent from compute node not visible in “neutron agent-list” result or it is visible but down? > Or something else? > Also, please send logs as plain text instead of screenshots - it will be much easier to read :) > >> On 16 Jul 2019, at 11:47, zhenhua_huang at trendmicro.com wrote: >> >> Hello, I am a newcomer to openstack. I have a problem. >> When I enter the command "openstack network agent list" on the controller node, the compute node is not found. Below are the output of the compute node's and controller node’s linuxbridge-agent.log.look forward to your reply. >> >> >> >> compute node >> >> >> controller node >> >> TREND MICRO EMAIL NOTICE >> The information contained in this email and any attachments is >> confidential and may be subject to copyright or other intellectual property protection. >> If you are not the intended recipient, you are not authorized to use >> or disclose this information, and we request that you notify us by >> reply mail or telephone and delete the original message from your mail system. >> >> For details about what personal information we collect and why, please >> see our Privacy Notice on our website at: [ >> https://www.trendmicro.com/privacy >> ] >> > > — > Slawek Kaplonski > Senior software engineer > Red Hat > > > 
> TREND MICRO EMAIL NOTICE
> The information contained in this email and any attachments is confidential 
> and may be subject to copyright or other intellectual property protection. 
> If you are not the intended recipient, you are not authorized to use or 
> disclose this information, and we request that you notify us by reply mail or
> telephone and delete the original message from your mail system.
> 
> For details about what personal information we collect and why, please see our Privacy Notice on our website at: [ https://www.trendmicro.com/privacy] 
>
— Slawek Kaplonski Senior software engineer Red Hat From hberaud at redhat.com Wed Jul 17 12:00:31 2019 From: hberaud at redhat.com (Herve Beraud) Date: Wed, 17 Jul 2019 14:00:31 +0200 Subject: [oslo] PTG Planning Etherpad In-Reply-To: <99952df7-e9dd-6fcb-9607-44d1f4c262e2@nemebean.com> References: <99952df7-e9dd-6fcb-9607-44d1f4c262e2@nemebean.com> Message-ID: Hi Thanks Ben. I've added a discussion to the topic list. It's related to the pep 518 and the future of pbr. Cheers! Le mar. 16 juil. 2019 à 18:28, Ben Nemec a écrit : > Hi, > > I've created a planning etherpad for Shanghai: > https://etherpad.openstack.org/p/oslo-shanghai-topics > > We need to know by Aug. 11 if we want to request PTG space this time > around so when you find out if you're going please add yourself to the > list on the etherpad. There's also the usual space for topics if you > know of anything we need to discuss already. > > Thanks. > > -Ben > > -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.sgaravatto at gmail.com Wed Jul 17 12:14:11 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Wed, 17 Jul 2019 14:14:11 +0200 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter Message-ID: We have just finished the update of our cloud to Rocky but we are seeing a strange issue with images with property: hypervisor_type='QEMU' The scheduling of instances created with such images fails because of the ImagePropertiesFilter [*] Removing that property from the image, the scheduling works. We also tried changing hypervisor_type='QEMU' --> hypervisor_type='qemu', but this didn't help. Any suggestions? Thanks, Massimo [*] 2019-07-17 13:52:58.148 13312 INFO nova.filters [req-1863bef0-0326-46d1-a836-436227e91eef 6e3b136d578f4292a5c03b16f443ab3d d27fe2becea94a3e980fb9f66e2f291a - default default] Filtering removed all hosts f\ or the request with instance ID '63810b60-76e4-4e76-a1c3-e4d3932c002e'. Filter results: ['AggregateMultiTenancyIsolation: (start: 49, end: 37)', 'AggregateInstanceExtraSpecsFilter: (start: 37, end: 34)', \ 'RetryFilter: (start: 34, end: 34)', 'AvailabilityZoneFilter: (start: 34, end: 34)', 'ComputeFilter: (start: 34, end: 32)', 'ComputeCapabilitiesFilter: (start: 32, end: 32)', 'ImagePropertiesFilter: (star\ t: 32, end: 0)'] -------------- next part -------------- An HTML attachment was scrubbed... URL: From raubvogel at gmail.com Wed Jul 17 12:50:50 2019 From: raubvogel at gmail.com (Mauricio Tavares) Date: Wed, 17 Jul 2019 08:50:50 -0400 Subject: Instructions for spooling up an instance Message-ID: I understand you create a flavor and an image (which can be customized with the properties from https://docs.openstack.org/glance/latest/admin/useful-image-properties.html) which can then be fed to openstack server create thingie. But, where is the script (think something like kickstart to see what I have in mind; if I am off, I am off) that defines how the hard drive will be partitioned and which interfaces (if more than one) will be turned on and which will be left untouched. You know, housekeeping things that are done before you feed the "--user-data" stuff. Where is that defined? Can I customize it? From noonedeadpunk at ya.ru Wed Jul 17 12:59:57 2019 From: noonedeadpunk at ya.ru (Dmitriy Rabotyagov) Date: Wed, 17 Jul 2019 15:59:57 +0300 Subject: [openstack-ansible][neutron] Drop of the legacy neutron L3 HA tool Message-ID: <4384501563368397@vla1-c9224cc75a57.qloud-c.yandex.net> Hi everyone, We're about to drop our legacy L3 HA tooling [1], since neutron has built-in HA for a long time. So in case you still need it please consider either switching to the native L3 HA or creating custom roles to deploy it afterwards. If you have any questions or suggestions regarding this, you are always welcome. [1] https://review.opendev.org/671247/ -- Kind regards, Dmitriy Rabotyagov From gaetan.trellu at incloudus.com Wed Jul 17 13:09:32 2019 From: gaetan.trellu at incloudus.com (=?ISO-8859-1?Q?Ga=EBtan_Trellu?=) Date: Wed, 17 Jul 2019 09:09:32 -0400 Subject: [kolla] [freezer] freezer scheduler auth_url error In-Reply-To: <1563350711224.53650@konicaminolta.it> Message-ID: An HTML attachment was scrubbed... URL: From aschultz at redhat.com Wed Jul 17 13:09:18 2019 From: aschultz at redhat.com (Alex Schultz) Date: Wed, 17 Jul 2019 07:09:18 -0600 Subject: [puppet] retiring puppet-crane Message-ID: Hello, We're going through the process to retire puppet-crane. It wasn't ever used and it was decided not to proceed forward with using crane (due to lack of python3 support). Thanks, -Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From amy at demarco.com Wed Jul 17 13:19:09 2019 From: amy at demarco.com (Amy Marrich) Date: Wed, 17 Jul 2019 08:19:09 -0500 Subject: Fwd: [openstack-community] [openstack-ansible] do I need deploymenthost on same br-mgmt network In-Reply-To: References: Message-ID: Rakesh, Forwarding this to the discuss list for more attention. Keep in mind OSA does take some time to run so including at what point you're failing would be helpful. Thanks, Amy (spotz) ---------- Forwarded message --------- From: rakesh zingade Date: Wed, Jul 17, 2019 at 3:11 AM Subject: [openstack-community] [openstack-ansible] do I need deploymenthost on same br-mgmt network To: Hi, I am pretty new to openstack and openstack ansible. I am trying to install openstack in our lab. So lab server are on different network ( 10.200.105.0/24 [external] and 10.200.75.0/24 [internal-within lab servers]) I suppose to deploy openstack by accessing remote connection and I am on different network (10.200.104.0/24) in same private lan & can access target hosts on ssh. my openstack_user_config.yml as below: cidr_networks: container: 10.200.105.0/24 storage: 10.200.75.0/24 tunnel: 10.200.105.0/24 used_ips: - "10.200.105.200,10.200.105.201" - "10.200.105.1,10.200.105.50" global_overrides: internal_lb_vip_address: 10.200.105.200 external_lb_vip_address: 10.200.105.200 management_bridge: "br-mgmt" provider_networks: - network: container_bridge: br-mgmt container_interface: eth1 container_type: veth group_binds: - all_containers - hosts ip_from_q: container is_container_address: true type: raw - network: container_bridge: br-storage container_interface: eth2 container_mtu: "9000" container_type: veth group_binds: - glance_api - cinder_api - cinder_volume - nova_compute - swift_proxy ip_from_q: storage type: raw - network: container_bridge: br-vxlan container_interface: eth10 container_mtu: "9000" container_type: veth group_binds: - neutron_linuxbridge_agent ip_from_q: tunnel net_name: vxlan range: "1:1000" type: vxlan - network: container_bridge: br-vlan container_interface: eth11 container_type: veth group_binds: - neutron_linuxbridge_agent net_name: vlan range: "101:200,301:400" type: vlan - network: container_bridge: br-vlan container_interface: eth12 container_type: veth group_binds: - neutron_linuxbridge_agent host_bind_override: eth12 net_name: flat type: flat shared-infra_hosts: infra1: container_vars: container_tech: lxc ip: 10.200.105.200 My deployment gets hanged after deploying lxc containers? any clue why this is happening? is my network is incorrect? do I have to deploy from same network i.e. 10.200.105.0/24 or 10.200.75.0/24? please let me know Thanks, -- Rakesh P. Zingade _______________________________________________ Community mailing list Community at lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community -------------- next part -------------- An HTML attachment was scrubbed... URL: From satish.txt at gmail.com Wed Jul 17 14:02:41 2019 From: satish.txt at gmail.com (Satish Patel) Date: Wed, 17 Jul 2019 10:02:41 -0400 Subject: [openstack-community] [openstack-ansible] do I need deploymenthost on same br-mgmt network In-Reply-To: References: Message-ID: Yes if you have seperate deployment host in that case you need br-mgmt ( if you deploying ceph with openstack-ansible then you need br-storage also) If you using one of infra (controller) node for deployment in that case you don't need to do anything. On Wed, Jul 17, 2019 at 9:24 AM Amy Marrich wrote: > > > Rakesh, > > Forwarding this to the discuss list for more attention. Keep in mind OSA does take some time to run so including at what point you're failing would be helpful. > > Thanks, > > Amy (spotz) > > ---------- Forwarded message --------- > From: rakesh zingade > Date: Wed, Jul 17, 2019 at 3:11 AM > Subject: [openstack-community] [openstack-ansible] do I need deploymenthost on same br-mgmt network > To: > > > Hi, > > I am pretty new to openstack and openstack ansible. I am trying to install openstack in our lab. So lab server are on different network (10.200.105.0/24 [external] and 10.200.75.0/24 [internal-within lab servers]) > I suppose to deploy openstack by accessing remote connection and I am on different network (10.200.104.0/24) in same private lan & can access target hosts on ssh. > > my openstack_user_config.yml as below: > > cidr_networks: > container: 10.200.105.0/24 > storage: 10.200.75.0/24 > tunnel: 10.200.105.0/24 > > used_ips: > - "10.200.105.200,10.200.105.201" > - "10.200.105.1,10.200.105.50" > > global_overrides: > internal_lb_vip_address: 10.200.105.200 > external_lb_vip_address: 10.200.105.200 > management_bridge: "br-mgmt" > > provider_networks: > - network: > container_bridge: br-mgmt > container_interface: eth1 > container_type: veth > group_binds: > - all_containers > - hosts > ip_from_q: container > is_container_address: true > type: raw > - network: > container_bridge: br-storage > container_interface: eth2 > container_mtu: "9000" > container_type: veth > group_binds: > - glance_api > - cinder_api > - cinder_volume > - nova_compute > - swift_proxy > ip_from_q: storage > type: raw > - network: > container_bridge: br-vxlan > container_interface: eth10 > container_mtu: "9000" > container_type: veth > group_binds: > - neutron_linuxbridge_agent > ip_from_q: tunnel > net_name: vxlan > range: "1:1000" > type: vxlan > - network: > container_bridge: br-vlan > container_interface: eth11 > container_type: veth > group_binds: > - neutron_linuxbridge_agent > net_name: vlan > range: "101:200,301:400" > type: vlan > - network: > container_bridge: br-vlan > container_interface: eth12 > container_type: veth > group_binds: > - neutron_linuxbridge_agent > host_bind_override: eth12 > net_name: flat > type: flat > shared-infra_hosts: > infra1: > container_vars: > container_tech: lxc > ip: 10.200.105.200 > > My deployment gets hanged after deploying lxc containers? any clue why this is happening? is my network is incorrect? do I have to deploy from same network i.e. 10.200.105.0/24 or 10.200.75.0/24? > > please let me know > > Thanks, > -- > Rakesh P. Zingade > > _______________________________________________ > Community mailing list > Community at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/community From donny at fortnebula.com Wed Jul 17 14:05:36 2019 From: donny at fortnebula.com (Donny Davis) Date: Wed, 17 Jul 2019 10:05:36 -0400 Subject: [Nova] Instances can't be started after compute nodes unexpectedly shut down because of power outage In-Reply-To: References: Message-ID: Any word yet? On Fri, Jul 12, 2019 at 3:37 PM Donny Davis wrote: > How is the recovery coming along Gökhan? I am curious to hear. > > On Fri, Jul 12, 2019 at 3:46 AM Gökhan IŞIK > wrote: > >> Awesome, thanks! Donny, >> I followed below steps and rescue my instance. >> >> 1. >> >> Find instance id and compute host >> >> root at infra1-utility-container-50bcf920:~# openstack server show 1d2e8a39-97ee-4ce7-a612-1b50f90cc51e -c id -c OS-EXT-SRV-ATTR:hypervisor_hostname >> +-------------------------------------+--------------------------------------+ >> | Field | Value | >> +-------------------------------------+--------------------------------------+ >> | OS-EXT-SRV-ATTR:hypervisor_hostname | compute06 | >> | id | 1d2e8a39-97ee-4ce7-a612-1b50f90cc51e | >> +-------------------------------------+--------------------------------------+ >> >> >> 2. >> >> Find image and backing image file on compute host >> >> root at compute06:~# qemu-img info -U --backing-chain /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk >> image: /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk >> file format: qcow2 >> virtual size: 160G (171798691840 bytes) >> disk size: 32G >> cluster_size: 65536 >> backing file: /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323 >> Format specific information: >> compat: 1.1 >> lazy refcounts: false >> refcount bits: 16 >> corrupt: false >> image: /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323 >> file format: raw >> virtual size: 160G (171798691840 bytes) >> disk size: 18G >> >> >> >> 3. Copy image and backing image file >> >> >> root at compute06:~# cp /var/lib/nova/instances/1d2e8a39-97ee-4ce7-a612-1b50f90cc51e/disk master >> root at compute06:~# cp /var/lib/nova/instances/_base/a1960f539532979a591c5f837ad604eedd9c7323 new-master >> >> >> 4. >> >> Rebase the image file that was backed off the original file so that >> it uses the new file i.e new-master then commit those changes back to >> original file master back into the new base new-master >> >> root at compute06:~# qemu-img rebase -b new-master -U master >> >> root at compute06:~# qemu-img commit master >> >> root at compute06:~# qemu-img info new-master >> >> >> >> >> 5. >> >> Convert raw image to qcow2 >> >> root at compute06:~# qemu-img convert -f raw -O qcow2 new-master new-master.qcow2 >> >> >> 6. Time to upload glance and then launch instance from this image :) >> >> >> Thanks, >> Gökhan. >> >> Donny Davis , 12 Tem 2019 Cum, 00:56 tarihinde >> şunu yazdı: >> >>> Of course you can also always just pull the disk images from the vm >>> folders, merge them back with the base file, upload to glance and then >>> relaunch the instances. >>> >>> You can give this method a spin with the lowest risk to your instances >>> >>> >>> https://medium.com/@kumar_pravin/qemu-merge-snapshot-and-backing-file-into-standalone-disk-c8d3a2b17c0e >>> >>> >>> >>> >>> >>> On Thu, Jul 11, 2019 at 4:10 PM Donny Davis >>> wrote: >>> >>>> You surely want to leave locking turned on. >>>> >>>> You may want to ask qemu-devel about the locking of a image file and >>>> how it works. This isn't really an Openstack issue, seems to be a layer >>>> below. >>>> >>>> Depending on how mission critical your VM's are, you could probably >>>> work around it by just passing in --force-share into the command openstack >>>> is trying to run. >>>> >>>> I cannot recommend this path, the best way is to find out how you >>>> remove the lock. >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Jul 11, 2019 at 3:23 PM Gökhan IŞIK >>>> wrote: >>>> >>>>> In [1] it says "Image locking is added and enabled by default. >>>>> Multiple QEMU processes cannot write to the same image as long as the host >>>>> supports OFD or posix locking, unless options are specified otherwise." May >>>>> be need to do something on nova side. >>>>> >>>>> I run this command and get same error. Output is in >>>>> http://paste.openstack.org/show/754311/ >>>>> >>>>> İf I run qemu-img info instance-0000219b with -U , it doesn't give any >>>>> errors. >>>>> >>>>> [1] https://wiki.qemu.org/ChangeLog/2.10 >>>>> >>>>> Donny Davis , 11 Tem 2019 Per, 22:11 tarihinde >>>>> şunu yazdı: >>>>> >>>>>> Well that is interesting. If you look in your libvirt config >>>>>> directory (/etc/libvirt on Centos) you can get a little more info on what >>>>>> is being used for locking. >>>>>> >>>>>> Maybe strace can shed some light on it. Try something like >>>>>> >>>>>> strace -ttt -f qemu-img info >>>>>> /var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jul 11, 2019 at 2:39 PM Gökhan IŞIK >>>>>> wrote: >>>>>> >>>>>>> I run virsh list --all command and output is below: >>>>>>> >>>>>>> root at compute06:~# virsh list --all >>>>>>> Id Name State >>>>>>> ---------------------------------------------------- >>>>>>> - instance-000012f9 shut off >>>>>>> - instance-000013b6 shut off >>>>>>> - instance-000016fb shut off >>>>>>> - instance-0000190a shut off >>>>>>> - instance-00001a8a shut off >>>>>>> - instance-00001e05 shut off >>>>>>> - instance-0000202a shut off >>>>>>> - instance-00002135 shut off >>>>>>> - instance-00002141 shut off >>>>>>> - instance-000021b6 shut off >>>>>>> - instance-000021ec shut off >>>>>>> - instance-000023db shut off >>>>>>> - instance-00002ad7 shut off >>>>>>> >>>>>>> And also when I try start instances with virsh , output is below: >>>>>>> >>>>>>> root at compute06:~# virsh start instance-0000219b >>>>>>> error: Failed to start domain instance-000012f9 >>>>>>> error: internal error: process exited while connecting to monitor: >>>>>>> 2019-07-11T18:36:34.229534Z qemu-system-x86_64: -chardev >>>>>>> pty,id=charserial0,logfile=/dev/fdset/2,logappend=on: char device >>>>>>> redirected to /dev/pts/3 (label charserial0) >>>>>>> 2019-07-11T18:36:34.243395Z qemu-system-x86_64: -drive >>>>>>> file=/var/lib/nova/instances/659b5853-d094-4425-85a9-5bcacf88c84e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=ignore: >>>>>>> Failed to get "write" lock >>>>>>> Is another process using the image? >>>>>>> >>>>>>> Thanks, >>>>>>> Gökhan >>>>>>> >>>>>>> Donny Davis , 11 Tem 2019 Per, 21:06 >>>>>>> tarihinde şunu yazdı: >>>>>>> >>>>>>>> Can you ssh to the hypervisor and run virsh list to make sure your >>>>>>>> instances are in fact down? >>>>>>>> >>>>>>>> On Thu, Jul 11, 2019 at 3:02 AM Gökhan IŞIK < >>>>>>>> skylightcoder at gmail.com> wrote: >>>>>>>> >>>>>>>>> Can anyone help me please ? I can no't rescue my instances yet :( >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Gökhan >>>>>>>>> >>>>>>>>> Gökhan IŞIK , 9 Tem 2019 Sal, 15:46 >>>>>>>>> tarihinde şunu yazdı: >>>>>>>>> >>>>>>>>>> Hi folks, >>>>>>>>>> Because of power outage, Most of our compute nodes unexpectedly >>>>>>>>>> shut down and now I can not start our instances. Error message is "Failed >>>>>>>>>> to get "write" lock another process using the image?". Instances Power >>>>>>>>>> status is No State. Full error log is >>>>>>>>>> http://paste.openstack.org/show/754107/. My environment is >>>>>>>>>> OpenStack Pike on Ubuntu 16.04 LTS servers and Instances are on a nfs >>>>>>>>>> shared storage. Nova version is 16.1.6.dev2. qemu version is 2.10.1. >>>>>>>>>> libvirt version is 3.6.0. I saw a commit [1], but it doesn't solve this >>>>>>>>>> problem. >>>>>>>>>> There are important instances on my environment. How can I rescue >>>>>>>>>> my instances? What would you suggest ? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Gökhan >>>>>>>>>> >>>>>>>>>> [1] https://review.opendev.org/#/c/509774/ >>>>>>>>>> >>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From alfredo.deluca at gmail.com Wed Jul 17 14:22:29 2019 From: alfredo.deluca at gmail.com (Alfredo De Luca) Date: Wed, 17 Jul 2019 16:22:29 +0200 Subject: [kolla] [freezer] freezer scheduler auth_url error In-Reply-To: References: <1563350711224.53650@konicaminolta.it> Message-ID: Hi Gaëtan. What do you mean *"will have to run into the instance you want to backup"?* I understand scheduler and agent must run on the same node (like openstack controller) but not sure on the VM instances you just created. Can you clarify? Cheers On Wed, Jul 17, 2019 at 3:14 PM Gaëtan Trellu wrote: > Hi Alessio, > > freezer-scheduler is not required on Openstack plane (controler/compute), > you could remove the control group from [freezer-scheduler:children] group > into the Ansible inventory. > > Basically, freezer-scheduler and freezer-agent will have to run into the > instance you want to backup. Into the instance you will need to have the > user RC file with the Openstack credentials and of course Keystone URL. > > This is required to authenticate the user to Keystone and get Swift > multi-tenancy. > > Gaetan > > On Jul 17, 2019 4:05 AM, "Fioravanti, Alessio" < > Alessio.Fioravanti at konicaminolta.it> wrote: > > Hi, > > this is my first post here. > > I'm experiencing a problem using freezer-scheduler component. When I > create a job and schedule it from Horizon, freezer-scheduler always fails. > The error shown in freezer-scheduler.log (from fluentd container) is: > > > 2019-07-17 07:48:23.294 6 WARNING freezer.scheduler.scheduler_job [-] Job > 1754ea4157b94acb916f8b9911886792 failed backup action, retrying in 60 > seconds > 2019-07-17 07:49:24.342 6 ERROR freezer.scheduler.scheduler_job [-] > Freezer client error: Critical Error: auth_url required to authenticate. > Make sure to export OS_AUTH_URL=http://keystone_url:5000/v3 > > The instance has been deployed with Kolla. Here some details: > > > Environment > > > *kolla version:* 8.0.1rc1 > > *openstack release:* stein > > *base container*: centos > > *host os*: ubuntu 18.04 > > *environment*: 1 controller on a dedicated host + 1 compute on a > dedicated host > > > Can you please give me your help? > > > Thanks, > > Alessio > > > > > -- *Alfredo* -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Wed Jul 17 15:27:33 2019 From: hberaud at redhat.com (Herve Beraud) Date: Wed, 17 Jul 2019 17:27:33 +0200 Subject: [oslo][walkthrough] How openstack services use the oslo.messaging rabbitmq driver Message-ID: Hello, During my working period related to the issue with the oslo.messaging rabbitmq heartbeat [1], I'd to studied how service consume oslo.messaging drivers to understand of things works together and also the execution model behind the scene. So during this period I wrote a walkthrough [2] to observe how nova-api use and consume the oslo.messaging rabbitmq driver. As I sayed, my walkthrough is focused on the studying of the nova-api, it start from how service are runned (the puppet steps), through how nova-api is runned by using mod_wsgi, to going to at least to how to nova-api instanciate the oslo.messaging rabbitmq driver. I also explain different solutions to fix the heartbeat rabbitmq driver issue and more globaly why potentially we facing this problem with apache, and the role of the MPM prefork module inside the issue. I think that some parts of this article and especially my walkthrough can be useful for other persons and openstackers, so I want share it with you. I surely wrote some erroned things in my article, so please do not hesitate to fix it if you see one and to send me a pull request [3]. Else if see other interesting things to add to it do not hesitate too. Enjoy your reading and I hope you will find some useful info by reading my article. Thank you for your attention. [1] https://bugs.launchpad.net/nova/+bug/1825584 [2] https://herve.beraud.io/openstack/nova/oslo.messaging/rabbitmq/driver/2019/07/15/how-nova-consume-oslo-messaging.html [3] https://github.com/4383/4383.github.io -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosmaita.fossdev at gmail.com Wed Jul 17 15:37:02 2019 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Wed, 17 Jul 2019 11:37:02 -0400 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: Message-ID: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> On 7/17/19 8:14 AM, Massimo Sgaravatto wrote: > We have just finished the update of our cloud to Rocky but we are seeing > a strange issue with images with property: > hypervisor_type='QEMU' > > The scheduling of instances created with such images fails because of > the ImagePropertiesFilter [*] > > Removing that property from the image, the scheduling works. > We also tried changing hypervisor_type='QEMU' > --> hypervisor_type='qemu', but this didn't help. > > Any suggestions? Commit e792d50efadb36437e82381f4c84d738cee25dfd in Ocata changed the image metadata that the ImagePropertiesFilter pays attention to: diff --git a/nova/scheduler/filters/image_props_filter.py b/nova/scheduler/filters/image_props_filter.py index 06def5c769..521a6816a3 100644 --- a/nova/scheduler/filters/image_props_filter.py +++ b/nova/scheduler/filters/image_props_filter.py @@ -43,9 +43,9 @@ class ImagePropertiesFilter(filters.BaseHostFilter): def _instance_supported(self, host_state, image_props, hypervisor_version): - img_arch = image_props.get('architecture', None) - img_h_type = image_props.get('hypervisor_type', None) - img_vm_mode = image_props.get('vm_mode', None) + img_arch = image_props.get('hw_architecture') + img_h_type = image_props.get('img_hv_type') + img_vm_mode = image_props.get('hw_vm_mode') checked_img_props = ( arch.canonicalize(img_arch), hv_type.canonicalize(img_h_type), Looks like 'img_hv_type' is the metadata key you need to use. If that works, please put up a patch to the Glance "useful image properties" docs [0], we seem to be out of date on this issue. [0] https://opendev.org/openstack/glance/src/branch/master/doc/source/admin/useful-image-properties.rst cheers, brian > Thanks, Massimo > > > > [*] > 2019-07-17 13:52:58.148 13312 INFO nova.filters > [req-1863bef0-0326-46d1-a836-436227e91eef > 6e3b136d578f4292a5c03b16f443ab3d d27fe2becea94a3e980fb9f66e2f291a - > default default] Filtering removed all hosts f\ > or the request with instance ID '63810b60-76e4-4e76-a1c3-e4d3932c002e'. > Filter results: ['AggregateMultiTenancyIsolation: (start: 49, end: 37)', > 'AggregateInstanceExtraSpecsFilter: (start: 37, end: 34)', \ > 'RetryFilter: (start: 34, end: 34)', 'AvailabilityZoneFilter: (start: > 34, end: 34)', 'ComputeFilter: (start: 34, end: 32)', > 'ComputeCapabilitiesFilter: (start: 32, end: 32)', > 'ImagePropertiesFilter: (star\ > t: 32, end: 0)'] From gaetan.trellu at incloudus.com Wed Jul 17 15:55:09 2019 From: gaetan.trellu at incloudus.com (gaetan.trellu at incloudus.com) Date: Wed, 17 Jul 2019 11:55:09 -0400 Subject: [kolla] [freezer] freezer scheduler auth_url error In-Reply-To: References: <1563350711224.53650@konicaminolta.it> Message-ID: Alfredo, From Freezer documentation: https://docs.openstack.org/freezer/latest/user/freezer-agent.html#freezer-scheduler "The freezer-scheduler is one of the two freezer components which is run on the client nodes; the other one being the freezer-agent. It has a double role: it is used both to start the scheduler process, and as a cli-tool which allows the user to interact with the API." Gaëtan On 2019-07-17 10:22, Alfredo De Luca wrote: > Hi Gaëtan. > What do you mean "WILL HAVE TO RUN INTO THE INSTANCE YOU WANT TO > BACKUP"? > I understand scheduler and agent must run on the same node (like > openstack controller) but not sure on the VM instances you just > created. > > Can you clarify? Cheers > > On Wed, Jul 17, 2019 at 3:14 PM Gaëtan Trellu > wrote: > Hi Alessio, > > freezer-scheduler is not required on Openstack plane > (controler/compute), you could remove the control group from > [freezer-scheduler:children] group into the Ansible inventory. > > Basically, freezer-scheduler and freezer-agent will have to run into > the instance you want to backup. Into the instance you will need to > have the user RC file with the Openstack credentials and of course > Keystone URL. > > This is required to authenticate the user to Keystone and get Swift > multi-tenancy. > > Gaetan > > On Jul 17, 2019 4:05 AM, "Fioravanti, Alessio" > wrote: > > Hi, > > this is my first post here. > > I'm experiencing a problem using freezer-scheduler component. When I > create a job and schedule it from Horizon, freezer-scheduler always > fails. The error shown in freezer-scheduler.log (from fluentd > container) is: > > 2019-07-17 07:48:23.294 6 WARNING freezer.scheduler.scheduler_job [-] > Job 1754ea4157b94acb916f8b9911886792 failed backup action, retrying in > 60 seconds > 2019-07-17 07:49:24.342 6 ERROR freezer.scheduler.scheduler_job [-] > Freezer client error: Critical Error: auth_url required to > authenticate. Make sure to export > OS_AUTH_URL=http://keystone_url:5000/v3 > > The instance has been deployed with Kolla. Here some details: > > Environment > > KOLLA VERSION: 8.0.1rc1 > > OPENSTACK RELEASE: stein > > BASE CONTAINER: centos > > HOST OS: ubuntu 18.04 > > ENVIRONMENT: 1 controller on a dedicated host + 1 compute on a > dedicated host > > Can you please give me your help? > > Thanks, > > Alessio -- _ALFREDO_ From radoslaw.piliszek at gmail.com Wed Jul 17 16:10:32 2019 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Wed, 17 Jul 2019 18:10:32 +0200 Subject: [kolla] [freezer] freezer scheduler auth_url error In-Reply-To: References: <1563350711224.53650@konicaminolta.it> Message-ID: I think this might be a bug in kolla-ansible. Please report to śr., 17 lip 2019 o 18:05 napisał(a): > Alfredo, > > From Freezer documentation: > > https://docs.openstack.org/freezer/latest/user/freezer-agent.html#freezer-scheduler > > "The freezer-scheduler is one of the two freezer components which is run > on the client nodes; the other one being the freezer-agent. It has a > double role: it is used both to start the scheduler process, and as a > cli-tool which allows the user to interact with the API." > > Gaëtan > > On 2019-07-17 10:22, Alfredo De Luca wrote: > > > Hi Gaëtan. > > What do you mean "WILL HAVE TO RUN INTO THE INSTANCE YOU WANT TO > > BACKUP"? > > I understand scheduler and agent must run on the same node (like > > openstack controller) but not sure on the VM instances you just > > created. > > > > Can you clarify? Cheers > > > > On Wed, Jul 17, 2019 at 3:14 PM Gaëtan Trellu > > wrote: > > Hi Alessio, > > > > freezer-scheduler is not required on Openstack plane > > (controler/compute), you could remove the control group from > > [freezer-scheduler:children] group into the Ansible inventory. > > > > Basically, freezer-scheduler and freezer-agent will have to run into > > the instance you want to backup. Into the instance you will need to > > have the user RC file with the Openstack credentials and of course > > Keystone URL. > > > > This is required to authenticate the user to Keystone and get Swift > > multi-tenancy. > > > > Gaetan > > > > On Jul 17, 2019 4:05 AM, "Fioravanti, Alessio" > > wrote: > > > > Hi, > > > > this is my first post here. > > > > I'm experiencing a problem using freezer-scheduler component. When I > > create a job and schedule it from Horizon, freezer-scheduler always > > fails. The error shown in freezer-scheduler.log (from fluentd > > container) is: > > > > 2019-07-17 07:48:23.294 6 WARNING freezer.scheduler.scheduler_job [-] > > Job 1754ea4157b94acb916f8b9911886792 failed backup action, retrying in > > 60 seconds > > 2019-07-17 07:49:24.342 6 ERROR freezer.scheduler.scheduler_job [-] > > Freezer client error: Critical Error: auth_url required to > > authenticate. Make sure to export > > OS_AUTH_URL=http://keystone_url:5000/v3 > > > > The instance has been deployed with Kolla. Here some details: > > > > Environment > > > > KOLLA VERSION: 8.0.1rc1 > > > > OPENSTACK RELEASE: stein > > > > BASE CONTAINER: centos > > > > HOST OS: ubuntu 18.04 > > > > ENVIRONMENT: 1 controller on a dedicated host + 1 compute on a > > dedicated host > > > > Can you please give me your help? > > > > Thanks, > > > > Alessio > > -- > > _ALFREDO_ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Wed Jul 17 16:12:12 2019 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Wed, 17 Jul 2019 18:12:12 +0200 Subject: [kolla] [freezer] freezer scheduler auth_url error In-Reply-To: References: <1563350711224.53650@konicaminolta.it> Message-ID: Sorry for the last cut message, I love it when I accidentally change keyboard layout and have different shorcuts. :-) I think this might be a bug in kolla-ansible. Please report to https://bugs.launchpad.net/kolla-ansible śr., 17 lip 2019 o 18:10 Radosław Piliszek napisał(a): > I think this might be a bug in kolla-ansible. > Please report to > > śr., 17 lip 2019 o 18:05 napisał(a): > >> Alfredo, >> >> From Freezer documentation: >> >> https://docs.openstack.org/freezer/latest/user/freezer-agent.html#freezer-scheduler >> >> "The freezer-scheduler is one of the two freezer components which is run >> on the client nodes; the other one being the freezer-agent. It has a >> double role: it is used both to start the scheduler process, and as a >> cli-tool which allows the user to interact with the API." >> >> Gaëtan >> >> On 2019-07-17 10:22, Alfredo De Luca wrote: >> >> > Hi Gaëtan. >> > What do you mean "WILL HAVE TO RUN INTO THE INSTANCE YOU WANT TO >> > BACKUP"? >> > I understand scheduler and agent must run on the same node (like >> > openstack controller) but not sure on the VM instances you just >> > created. >> > >> > Can you clarify? Cheers >> > >> > On Wed, Jul 17, 2019 at 3:14 PM Gaëtan Trellu >> > wrote: >> > Hi Alessio, >> > >> > freezer-scheduler is not required on Openstack plane >> > (controler/compute), you could remove the control group from >> > [freezer-scheduler:children] group into the Ansible inventory. >> > >> > Basically, freezer-scheduler and freezer-agent will have to run into >> > the instance you want to backup. Into the instance you will need to >> > have the user RC file with the Openstack credentials and of course >> > Keystone URL. >> > >> > This is required to authenticate the user to Keystone and get Swift >> > multi-tenancy. >> > >> > Gaetan >> > >> > On Jul 17, 2019 4:05 AM, "Fioravanti, Alessio" >> > wrote: >> > >> > Hi, >> > >> > this is my first post here. >> > >> > I'm experiencing a problem using freezer-scheduler component. When I >> > create a job and schedule it from Horizon, freezer-scheduler always >> > fails. The error shown in freezer-scheduler.log (from fluentd >> > container) is: >> > >> > 2019-07-17 07:48:23.294 6 WARNING freezer.scheduler.scheduler_job [-] >> > Job 1754ea4157b94acb916f8b9911886792 failed backup action, retrying in >> > 60 seconds >> > 2019-07-17 07:49:24.342 6 ERROR freezer.scheduler.scheduler_job [-] >> > Freezer client error: Critical Error: auth_url required to >> > authenticate. Make sure to export >> > OS_AUTH_URL=http://keystone_url:5000/v3 >> > >> > The instance has been deployed with Kolla. Here some details: >> > >> > Environment >> > >> > KOLLA VERSION: 8.0.1rc1 >> > >> > OPENSTACK RELEASE: stein >> > >> > BASE CONTAINER: centos >> > >> > HOST OS: ubuntu 18.04 >> > >> > ENVIRONMENT: 1 controller on a dedicated host + 1 compute on a >> > dedicated host >> > >> > Can you please give me your help? >> > >> > Thanks, >> > >> > Alessio >> >> -- >> >> _ALFREDO_ >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnsomor at gmail.com Wed Jul 17 16:13:34 2019 From: johnsomor at gmail.com (Michael Johnson) Date: Wed, 17 Jul 2019 09:13:34 -0700 Subject: [octavia] Shanghai PTG planning etherpad Message-ID: Hello Octavia! I have created an etherpad to start tracking topics for the Shanghai (U) PTG: https://etherpad.openstack.org/p/octavia-shanghai-U-ptg Michael From elanchezian.settu at gigamon.com Wed Jul 17 06:13:51 2019 From: elanchezian.settu at gigamon.com (Elanchezian Settu) Date: Wed, 17 Jul 2019 06:13:51 +0000 Subject: [networking-ovs-dpdk Message-ID: Hi, I'm trying to setup a multi-node OpenStack environment with OVS-DPDK. Unfortunately I'm running into issue that compute node not getting listed in hypervisor list as below screenshot. Due to this I'm unable to create instances on compute node. Herewith I had attached the local.conf files of both controller and compute node. Can someone help me to resolve this issue. [cid:image001.jpg at 01D53C94.E3DEBF30] Thanks & Regards, Elan This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. NOTE that all incoming emails sent to Gigamon email accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional emails ("spam"). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 140947 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: compute-node-local.conf Type: application/octet-stream Size: 1878 bytes Desc: compute-node-local.conf URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: controller-local.conf Type: application/octet-stream Size: 1810 bytes Desc: controller-local.conf URL: From cboylan at sapwetik.org Wed Jul 17 19:48:53 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Wed, 17 Jul 2019 12:48:53 -0700 Subject: Instructions for spooling up an instance In-Reply-To: References: Message-ID: <6c253bdb-ce18-4ed9-ad98-e949cd108235@www.fastmail.com> On Wed, Jul 17, 2019, at 5:51 AM, Mauricio Tavares wrote: > I understand you create a flavor and an image (which can be > customized with the properties from > https://docs.openstack.org/glance/latest/admin/useful-image-properties.html) > which can then be fed to openstack server create thingie. But, where > is the script (think something like kickstart to see what I have in > mind; if I am off, I am off) that defines how the hard drive will be > partitioned and which interfaces (if more than one) will be turned on > and which will be left untouched. You know, housekeeping things that > are done before you feed the "--user-data" stuff. Where is that > defined? Can I customize it? > > The two major ways you manage this is via your image builds (this is pre boot) and via a post boot manager like cloud-init or glean. With your image build you can preinstall all of the software you need (and configure it too). You can also configure disk partitioning. The tool the OpenDev Infra team uses this for all of our test node images and some of our control plane images is called diskimage-builder [0]. This tool can partition images using both mbr and gpt tables. For post boot management we use glean [1] which is a very simple post boot manager that can add ssh authorized keys to the root user and configure network interface. Cloud-init [2] is a far more configurable tool but I'm not super familiar with it at this point (due to using glean). Long story short that means bake what you can into the image then rely on a tool like glean or cloud-init to handle post boot actions that differ from instance to instance (like network interfaces). [0] https://docs.openstack.org/diskimage-builder/latest/ [1] https://docs.openstack.org/infra/glean/ [2] https://cloudinit.readthedocs.io/en/latest/ Hope this helps, Clark From fungi at yuggoth.org Wed Jul 17 20:45:27 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 17 Jul 2019 20:45:27 +0000 Subject: Instructions for spooling up an instance In-Reply-To: <6c253bdb-ce18-4ed9-ad98-e949cd108235@www.fastmail.com> References: <6c253bdb-ce18-4ed9-ad98-e949cd108235@www.fastmail.com> Message-ID: <20190717204526.zfs7e72xdcc2epy3@yuggoth.org> On 2019-07-17 12:48:53 -0700 (-0700), Clark Boylan wrote: [...] > With your image build you can preinstall all of the software you > need (and configure it too). You can also configure disk > partitioning. The tool the OpenDev Infra team uses this for all of > our test node images and some of our control plane images is > called diskimage-builder [0]. This tool can partition images using > both mbr and gpt tables. [...] I gather there's another popular alternative, which is to boot a server instance with an ISO of an operating system installer image attached, use that to configure the supplied root disk and install software into it, then shut that down and create a snapshot of the resulting instance's rootfs to boot additional instances from. It's a more traditional analog to how you'd do server installation in classic non-cloudy environments, but some folks still seem to prefer it over creating images of fully-working servers and then uploading those. That said, I'd recommend sticking to using/modifying officially provided "cloud" images from the distros you want since they do some useful things like wiping system identifiers and host keys so that they're regenerated independently on each new instance booted from them (if you don't remember to do that on your custom images you can run afoul of support licensing contracts or create unnecessary security risks). -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From openstack at fried.cc Wed Jul 17 21:39:08 2019 From: openstack at fried.cc (Eric Fried) Date: Wed, 17 Jul 2019 16:39:08 -0500 Subject: Instructions for spooling up an instance In-Reply-To: <20190717204526.zfs7e72xdcc2epy3@yuggoth.org> References: <6c253bdb-ce18-4ed9-ad98-e949cd108235@www.fastmail.com> <20190717204526.zfs7e72xdcc2epy3@yuggoth.org> Message-ID: <55ab1cb2-b00a-dfdc-0a36-83daaa15aad4@fried.cc> In case it helps, here's what Nova has to say about config drives: https://docs.openstack.org/nova/latest/admin/config-drive.html From no-reply at openstack.org Thu Jul 18 00:46:18 2019 From: no-reply at openstack.org (no-reply at openstack.org) Date: Thu, 18 Jul 2019 00:46:18 -0000 Subject: kolla 8.0.0.0rc2 (stein) Message-ID: Hello everyone, A new release candidate for kolla for the end of the Stein cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/kolla/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Stein release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/stein release branch at: https://opendev.org/openstack/kolla/log/?h=stable/stein Release notes for kolla can be found at: https://docs.openstack.org/releasenotes/kolla/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/kolla/+bugs and tag it *stein-rc-potential* to bring it to the kolla release crew's attention. From no-reply at openstack.org Thu Jul 18 00:47:22 2019 From: no-reply at openstack.org (no-reply at openstack.org) Date: Thu, 18 Jul 2019 00:47:22 -0000 Subject: kolla-ansible 8.0.0.0rc2 (stein) Message-ID: Hello everyone, A new release candidate for kolla-ansible for the end of the Stein cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/kolla-ansible/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Stein release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/stein release branch at: https://opendev.org/openstack/kolla-ansible/log/?h=stable/stein Release notes for kolla-ansible can be found at: https://docs.openstack.org/releasenotes/kolla-ansible/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/kolla-ansible/+bugs and tag it *stein-rc-potential* to bring it to the kolla-ansible release crew's attention. From missile0407 at gmail.com Thu Jul 18 01:25:54 2019 From: missile0407 at gmail.com (Eddie Yen) Date: Thu, 18 Jul 2019 09:25:54 +0800 Subject: [Kolla][Keystone] No "Domains" found in identity area. Message-ID: Hi, I'm using kolla-ansible stable/rocky release to deploy Openstack Try to test with multi-domains requirements. I'm do nothing but add "horizon_keystone_multidomain: True" into globals.yml After deployment, it works because I have to enter the domain name at login screen. Logged in with default domain and admin account. But I cannot found any "Domains" section in identity block. Is there something I miss? -------------- next part -------------- An HTML attachment was scrubbed... URL: From missile0407 at gmail.com Thu Jul 18 01:45:07 2019 From: missile0407 at gmail.com (Eddie Yen) Date: Thu, 18 Jul 2019 09:45:07 +0800 Subject: [Kolla][Keystone] No "Domains" found in identity area. In-Reply-To: References: Message-ID: Sorry, it should be Horizon, not Keystone. Eddie Yen 於 2019年7月18日 週四 上午9:25寫道： > Hi, > > I'm using kolla-ansible stable/rocky release to deploy Openstack > Try to test with multi-domains requirements. I'm do nothing but add > "horizon_keystone_multidomain: True" into globals.yml > > After deployment, it works because I have to enter the domain name at > login screen. > Logged in with default domain and admin account. But I cannot found any > "Domains" section in identity block. > > Is there something I miss? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From satish.txt at gmail.com Thu Jul 18 02:28:14 2019 From: satish.txt at gmail.com (Satish Patel) Date: Wed, 17 Jul 2019 22:28:14 -0400 Subject: opensips with asterisk integration question Message-ID: I am trying to conigured opensips as registar and asterisk for media services using this doc: https://www.opensips.org/Documentation/Tutorials-OpenSIPSAsteriskIntegration so far everything went well and i am able to use VoiceMail, MeetMe etc. but when i am trying to dial any SIP phone getting following error Asterisk logs: [Jul 17 15:23:05] NOTICE[9933][C-00000001]: chan_sip.c:31847 build_peer: The 'username' field for sip peers has been deprecated in favor of the term 'defaultuser' [Jul 17 15:23:05] WARNING[9933][C-00000001]: sip/config_parser.c:817 sip_parse_nat_option: nat=yes is deprecated, use nat=force_rport,comedia instead [Jul 17 15:23:05] WARNING[9972][C-00000001]: sip/config_parser.c:817 sip_parse_nat_option: nat=yes is deprecated, use nat=force_rport,comedia instead [Jul 17 15:23:05] NOTICE[9933][C-00000001]: chan_sip.c:24288 handle_response_invite: Failed to authenticate on INVITE to '"1001" ;tag=as1986a977' voip*CLI> In mysql sipusers look like following: *************************** 3. row *************************** name: 1001 username: 1001 type: friend secret: NULL host: opensips.org callerid: NULL context: default mailbox: 1001 nat: yes qualify: no fromuser: 1001 authuser: NULL fromdomain: opensips.org insecure: NULL canreinvite: no disallow: NULL allow: NULL restrictcid: NULL defaultip: opensips.org ipaddr: opensips.org port: 5060 regseconds: NULL basis dialplan ; announcement demo exten => 2000,1,Ringing exten => 2000,2,Playback(welcome) exten => 2000,3,Hangup ; voicemail exten => 777,1,Ringing exten => 777,n,Wait(1) exten => 777,n,Answer exten => 777,n,Wait(1) exten => 777,n,VoiceMailMain(@default) exten => 777,n,Hangup exten => 1001,1,Dial(SIP/1001) exten => 1002,1,Dial(SIP/1002) From manulachathurika at gmail.com Thu Jul 18 04:58:06 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Thu, 18 Jul 2019 10:28:06 +0530 Subject: ImportError: No module named django.core.wsgi - DevStack - Stable/Stein - Ubuntu 18.04 In-Reply-To: References: Message-ID: Thanks all. What I have done is, completely remove Ubuntu 18.04 and install Ubuntu 16.04. It works without any problem. :) Thanks ! On Wed, Jul 17, 2019 at 2:03 PM Akihiro Motoki wrote: > While I don't see any issue around devstack horizon with python3, > I see some points to be checked. > > - Do you use USE_PYTHON3 in local.conf? > - Is the right wsgi module for apache installed? > - Can you import django.core.wsgi with right python3 interpreter manually? > > The following is an example in my environment. > > $ which python3 > /usr/bin/python3 > $ python3 manage.py shell > Python 3.6.8 (default, Jan 14 2019, 11:02:34) > [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux > Type "help", "copyright", "credits" or "license" for more information. > (InteractiveConsole) > >>> from django.core.wsgi import get_wsgi_application > >>> > now exiting InteractiveConsole... > > $ cat /etc/apache2/mods-enabled/wsgi.load > LoadModule wsgi_module /usr/lib/apache2/modules/mod_wsgi.so > > $ dpkg -S /usr/lib/apache2/modules/mod_wsgi.so > libapache2-mod-wsgi-py3: /usr/lib/apache2/modules/mod_wsgi.so > > On Tue, Jul 16, 2019 at 7:04 PM Manula Thantriwatte > wrote: > > > > Hi All, > > > > I have successfully install DevStack in Ubuntu 18.04. But when I'm > accessing the dashboard I'm getting 500 error. What I'm getting in > horizon_error.log is, > > > > 2019-07-15 14:10:43.218296 mod_wsgi (pid=31763): Target WSGI script > '/opt/stack/horizon/openstack_dashboard/wsgi.py' cannot be loaded as Python > module. > > 2019-07-15 14:10:43.218323 mod_wsgi (pid=31763): Exception occurred > processing WSGI script '/opt/stack/horizon/openstack_dashboard/wsgi.py'. > > 2019-07-15 14:10:43.218349 Traceback (most recent call last): > > 2019-07-15 14:10:43.218370 File > "/opt/stack/horizon/openstack_dashboard/wsgi.py", line 21, in > > 2019-07-15 14:10:43.218401 from django.core.wsgi import > get_wsgi_application > > 2019-07-15 14:10:43.218422 ImportError: No module named django.core.wsgi > > > > Python Version is : 3.6.8 > > Django version is : 2.0.5 > > > > I tried with uninstalling Djanago and reinstalling it. But it didn't > work for me. > > > > Can someone help me on how to resole this issue ? > > > > Thanks ! > > -- > > Regards, > > Manula Chathurika Thantriwatte > > phone : (+94) 772492511 > > email : manulachathurika at gmail.com > > Linkedin : http://lk.linkedin.com/in/manulachathurika > > blog : http://manulachathurika.blogspot.com/ > > > -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobias.rydberg at citynetwork.eu Thu Jul 18 06:01:51 2019 From: tobias.rydberg at citynetwork.eu (Tobias Rydberg) Date: Thu, 18 Jul 2019 08:01:51 +0200 Subject: [sigs][publiccloud][publiccloud-wg][publiccloud-sig][billing] Meeting today at 1400 UTC regarding billing initiative Message-ID: Hi all, This is a reminder for todays meeting for the Public Cloud SIG - 1400 UTC in #openstack-publiccloud. The topic of the day will be continued discussions regarding the billing initiative. More information about that at https://etherpad.openstack.org/p/publiccloud-sig-billing-implementation-proposal See you all later today! Cheers, Tobias -- Tobias Rydberg Senior Developer Twitter & IRC: tobberydberg www.citynetwork.eu | www.citycloud.com INNOVATION THROUGH OPEN IT INFRASTRUCTURE ISO 9001, 14001, 27001, 27015 & 27018 CERTIFIED From manulachathurika at gmail.com Thu Jul 18 07:27:31 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Thu, 18 Jul 2019 12:57:31 +0530 Subject: Unable to ssh openstack instance In-Reply-To: References: Message-ID: Hi All, I'm trying to ssh openstack image I have created using cirros image. But I'm getting following error. ssh: connect to host 172.24.4.147 port 22: No route to host But when I go to the instance console I'm able to login. For the security groups I have enable port 22. I have associated floating IP as well. How to resolve this? Thanks ! -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lennyb at mellanox.com Thu Jul 18 07:40:02 2019 From: lennyb at mellanox.com (Lenny Verkhovsky) Date: Thu, 18 Jul 2019 07:40:02 +0000 Subject: Unable to ssh openstack instance In-Reply-To: References: Message-ID: Hi, You need to create public ip, add it to network port and try to ssh to the image # openstack floating ip create public # openstack floating ip set --port Best Regards Lenny From: Manula Thantriwatte Sent: Thursday, July 18, 2019 10:28 AM To: openstack-discuss Subject: Unable to ssh openstack instance Hi All, I'm trying to ssh openstack image I have created using cirros image. But I'm getting following error. ssh: connect to host 172.24.4.147 port 22: No route to host But when I go to the instance console I'm able to login. For the security groups I have enable port 22. I have associated floating IP as well. How to resolve this? Thanks ! -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : http://lk.linkedin.com/in/manulachathurika blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From manulachathurika at gmail.com Thu Jul 18 08:47:25 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Thu, 18 Jul 2019 14:17:25 +0530 Subject: Unable to ssh openstack instance In-Reply-To: References: Message-ID: Hi Lenny, When I run the above command it asking password. OS password and OpenStack admin password didn't work. Also I haven't set any password when installing DevStack. Is there a default password for this ? Thanks ! On Thu, Jul 18, 2019 at 1:10 PM Lenny Verkhovsky wrote: > Hi, > > You need to create public ip, add it to network port and try to ssh to the > image > > > > # openstack floating ip create public > > # openstack floating ip set --port > > > > Best Regards > > Lenny > > > > *From:* Manula Thantriwatte > *Sent:* Thursday, July 18, 2019 10:28 AM > *To:* openstack-discuss > *Subject:* Unable to ssh openstack instance > > > > Hi All, > > > > I'm trying to ssh openstack image I have created using cirros image. But > I'm getting following error. > > > > ssh: connect to host 172.24.4.147 port 22: No route to host > > > > But when I go to the instance console I'm able to login. > > > > For the security groups I have enable port 22. > > > > I have associated floating IP as well. > > > > How to resolve this? > > > > Thanks ! > > > -- > > Regards, > > Manula Chathurika Thantriwatte > > phone : (+94) 772492511 > > email : manulachathurika at gmail.com > > Linkedin : *http://lk.linkedin.com/in/manulachathurika > * > > > blog : http://manulachathurika.blogspot.com/ > > -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From missile0407 at gmail.com Thu Jul 18 08:53:27 2019 From: missile0407 at gmail.com (Eddie Yen) Date: Thu, 18 Jul 2019 16:53:27 +0800 Subject: [kolla][nova][cinder] Got Gateway-Timeout error on VM evacuation if it has volume attached. Message-ID: Hi everyone, I met an issue when try to evacuate host. The platform is stable/rocky and using kolla-ansible to deploy. And all storage backends are connected to Ceph. Before I try to evacuate host, the source host had about 24 VMs running. When I shutdown the node and execute evacuation, there're few VMs failed. The error code is 504. Strange is those VMs are all attach its own volume. Then I check nova-compute log, a detailed error has pasted at below link; https://pastebin.com/uaE7YrP1 Does anyone have any experience with this? I googled but no enough information about this. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From manulachathurika at gmail.com Thu Jul 18 08:56:12 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Thu, 18 Jul 2019 14:26:12 +0530 Subject: Unable to ssh openstack instance In-Reply-To: References: Message-ID: Hi Lenny, Got it. Need to source openrc file in devstack folder. Thanks ! On Thu, Jul 18, 2019 at 2:17 PM Manula Thantriwatte < manulachathurika at gmail.com> wrote: > Hi Lenny, > > When I run the above command it asking password. OS password and OpenStack > admin password didn't work. Also I haven't set any password when installing > DevStack. > > Is there a default password for this ? > > Thanks ! > > On Thu, Jul 18, 2019 at 1:10 PM Lenny Verkhovsky > wrote: > >> Hi, >> >> You need to create public ip, add it to network port and try to ssh to >> the image >> >> >> >> # openstack floating ip create public >> >> # openstack floating ip set --port >> >> >> >> Best Regards >> >> Lenny >> >> >> >> *From:* Manula Thantriwatte >> *Sent:* Thursday, July 18, 2019 10:28 AM >> *To:* openstack-discuss >> *Subject:* Unable to ssh openstack instance >> >> >> >> Hi All, >> >> >> >> I'm trying to ssh openstack image I have created using cirros image. But >> I'm getting following error. >> >> >> >> ssh: connect to host 172.24.4.147 port 22: No route to host >> >> >> >> But when I go to the instance console I'm able to login. >> >> >> >> For the security groups I have enable port 22. >> >> >> >> I have associated floating IP as well. >> >> >> >> How to resolve this? >> >> >> >> Thanks ! >> >> >> -- >> >> Regards, >> >> Manula Chathurika Thantriwatte >> >> phone : (+94) 772492511 >> >> email : manulachathurika at gmail.com >> >> Linkedin : *http://lk.linkedin.com/in/manulachathurika >> * >> >> >> blog : http://manulachathurika.blogspot.com/ >> >> > > > -- > Regards, > Manula Chathurika Thantriwatte > phone : (+94) 772492511 > email : manulachathurika at gmail.com > Linkedin : *http://lk.linkedin.com/in/manulachathurika > * > blog : http://manulachathurika.blogspot.com/ > -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From manulachathurika at gmail.com Thu Jul 18 09:03:19 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Thu, 18 Jul 2019 14:33:19 +0530 Subject: Unable to ssh openstack instance In-Reply-To: References: Message-ID: Hi Lenny, When I trying to associate floating IP it asking port. What would be the default port for this ? Thanks ! On Thu, Jul 18, 2019 at 2:26 PM Manula Thantriwatte < manulachathurika at gmail.com> wrote: > Hi Lenny, > > Got it. Need to source openrc file in devstack folder. > > Thanks ! > > On Thu, Jul 18, 2019 at 2:17 PM Manula Thantriwatte < > manulachathurika at gmail.com> wrote: > >> Hi Lenny, >> >> When I run the above command it asking password. OS password and >> OpenStack admin password didn't work. Also I haven't set any password when >> installing DevStack. >> >> Is there a default password for this ? >> >> Thanks ! >> >> On Thu, Jul 18, 2019 at 1:10 PM Lenny Verkhovsky >> wrote: >> >>> Hi, >>> >>> You need to create public ip, add it to network port and try to ssh to >>> the image >>> >>> >>> >>> # openstack floating ip create public >>> >>> # openstack floating ip set --port >>> >>> >>> >>> Best Regards >>> >>> Lenny >>> >>> >>> >>> *From:* Manula Thantriwatte >>> *Sent:* Thursday, July 18, 2019 10:28 AM >>> *To:* openstack-discuss >>> *Subject:* Unable to ssh openstack instance >>> >>> >>> >>> Hi All, >>> >>> >>> >>> I'm trying to ssh openstack image I have created using cirros image. But >>> I'm getting following error. >>> >>> >>> >>> ssh: connect to host 172.24.4.147 port 22: No route to host >>> >>> >>> >>> But when I go to the instance console I'm able to login. >>> >>> >>> >>> For the security groups I have enable port 22. >>> >>> >>> >>> I have associated floating IP as well. >>> >>> >>> >>> How to resolve this? >>> >>> >>> >>> Thanks ! >>> >>> >>> -- >>> >>> Regards, >>> >>> Manula Chathurika Thantriwatte >>> >>> phone : (+94) 772492511 >>> >>> email : manulachathurika at gmail.com >>> >>> Linkedin : *http://lk.linkedin.com/in/manulachathurika >>> * >>> >>> >>> blog : http://manulachathurika.blogspot.com/ >>> >>> >> >> >> -- >> Regards, >> Manula Chathurika Thantriwatte >> phone : (+94) 772492511 >> email : manulachathurika at gmail.com >> Linkedin : *http://lk.linkedin.com/in/manulachathurika >> * >> blog : http://manulachathurika.blogspot.com/ >> > > > -- > Regards, > Manula Chathurika Thantriwatte > phone : (+94) 772492511 > email : manulachathurika at gmail.com > Linkedin : *http://lk.linkedin.com/in/manulachathurika > * > blog : http://manulachathurika.blogspot.com/ > -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lennyb at mellanox.com Thu Jul 18 09:15:31 2019 From: lennyb at mellanox.com (Lenny Verkhovsky) Date: Thu, 18 Jul 2019 09:15:31 +0000 Subject: Unable to ssh openstack instance In-Reply-To: References: Message-ID: You can check it with # openstack server list # openstack port list You need to see which port is allocated to VM From: Manula Thantriwatte Sent: Thursday, July 18, 2019 12:03 PM To: Lenny Verkhovsky Cc: openstack-discuss Subject: Re: Unable to ssh openstack instance Hi Lenny, When I trying to associate floating IP it asking port. What would be the default port for this ? Thanks ! On Thu, Jul 18, 2019 at 2:26 PM Manula Thantriwatte > wrote: Hi Lenny, Got it. Need to source openrc file in devstack folder. Thanks ! On Thu, Jul 18, 2019 at 2:17 PM Manula Thantriwatte > wrote: Hi Lenny, When I run the above command it asking password. OS password and OpenStack admin password didn't work. Also I haven't set any password when installing DevStack. Is there a default password for this ? Thanks ! On Thu, Jul 18, 2019 at 1:10 PM Lenny Verkhovsky > wrote: Hi, You need to create public ip, add it to network port and try to ssh to the image # openstack floating ip create public # openstack floating ip set --port Best Regards Lenny From: Manula Thantriwatte > Sent: Thursday, July 18, 2019 10:28 AM To: openstack-discuss > Subject: Unable to ssh openstack instance Hi All, I'm trying to ssh openstack image I have created using cirros image. But I'm getting following error. ssh: connect to host 172.24.4.147 port 22: No route to host But when I go to the instance console I'm able to login. For the security groups I have enable port 22. I have associated floating IP as well. How to resolve this? Thanks ! -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : http://lk.linkedin.com/in/manulachathurika blog : http://manulachathurika.blogspot.com/ -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : http://lk.linkedin.com/in/manulachathurika blog : http://manulachathurika.blogspot.com/ -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : http://lk.linkedin.com/in/manulachathurika blog : http://manulachathurika.blogspot.com/ -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : http://lk.linkedin.com/in/manulachathurika blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Thu Jul 18 09:27:21 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 18 Jul 2019 11:27:21 +0200 Subject: Unable to ssh openstack instance In-Reply-To: References: Message-ID: <93D5544A-6206-4FE7-ADB8-7F0CD5FC070F@redhat.com> Hi, > On 18 Jul 2019, at 11:15, Lenny Verkhovsky wrote: > > You can check it with > # openstack server list > # openstack port list > You need to see which port is allocated to VM You can check that with command like: nova interface-list > > > From: Manula Thantriwatte > Sent: Thursday, July 18, 2019 12:03 PM > To: Lenny Verkhovsky > Cc: openstack-discuss > Subject: Re: Unable to ssh openstack instance > > Hi Lenny, > > When I trying to associate floating IP it asking port. What would be the default port for this ? > > Thanks ! > > On Thu, Jul 18, 2019 at 2:26 PM Manula Thantriwatte wrote: > Hi Lenny, > > Got it. Need to source openrc file in devstack folder. > > Thanks ! > > On Thu, Jul 18, 2019 at 2:17 PM Manula Thantriwatte wrote: > Hi Lenny, > > When I run the above command it asking password. OS password and OpenStack admin password didn't work. Also I haven't set any password when installing DevStack. > > Is there a default password for this ? > > Thanks ! > > On Thu, Jul 18, 2019 at 1:10 PM Lenny Verkhovsky wrote: > Hi, > You need to create public ip, add it to network port and try to ssh to the image > > # openstack floating ip create public > # openstack floating ip set --port > > Best Regards > Lenny > > From: Manula Thantriwatte > Sent: Thursday, July 18, 2019 10:28 AM > To: openstack-discuss > Subject: Unable to ssh openstack instance > > Hi All, > > I'm trying to ssh openstack image I have created using cirros image. But I'm getting following error. > > ssh: connect to host 172.24.4.147 port 22: No route to host > > But when I go to the instance console I'm able to login. > > For the security groups I have enable port 22. > > I have associated floating IP as well. > > How to resolve this? > > Thanks ! > > -- > Regards, > Manula Chathurika Thantriwatte > phone : (+94) 772492511 > email : manulachathurika at gmail.com > Linkedin : http://lk.linkedin.com/in/manulachathurika > blog : http://manulachathurika.blogspot.com/ > > > -- > Regards, > Manula Chathurika Thantriwatte > phone : (+94) 772492511 > email : manulachathurika at gmail.com > Linkedin : http://lk.linkedin.com/in/manulachathurika > blog : http://manulachathurika.blogspot.com/ > > > -- > Regards, > Manula Chathurika Thantriwatte > phone : (+94) 772492511 > email : manulachathurika at gmail.com > Linkedin : http://lk.linkedin.com/in/manulachathurika > blog : http://manulachathurika.blogspot.com/ > > > -- > Regards, > Manula Chathurika Thantriwatte > phone : (+94) 772492511 > email : manulachathurika at gmail.com > Linkedin : http://lk.linkedin.com/in/manulachathurika > blog : http://manulachathurika.blogspot.com/ — Slawek Kaplonski Senior software engineer Red Hat From manulachathurika at gmail.com Thu Jul 18 09:48:06 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Thu, 18 Jul 2019 15:18:06 +0530 Subject: Unable to ssh openstack instance In-Reply-To: <93D5544A-6206-4FE7-ADB8-7F0CD5FC070F@redhat.com> References: <93D5544A-6206-4FE7-ADB8-7F0CD5FC070F@redhat.com> Message-ID: Thanks all. I'm able to resolve the issue. Thanks ! On Thu, Jul 18, 2019 at 2:57 PM Slawek Kaplonski wrote: > Hi, > > > On 18 Jul 2019, at 11:15, Lenny Verkhovsky wrote: > > > > You can check it with > > # openstack server list > > # openstack port list > > You need to see which port is allocated to VM > > You can check that with command like: > > nova interface-list > > > > > > > From: Manula Thantriwatte > > Sent: Thursday, July 18, 2019 12:03 PM > > To: Lenny Verkhovsky > > Cc: openstack-discuss > > Subject: Re: Unable to ssh openstack instance > > > > Hi Lenny, > > > > When I trying to associate floating IP it asking port. What would be the > default port for this ? > > > > Thanks ! > > > > On Thu, Jul 18, 2019 at 2:26 PM Manula Thantriwatte < > manulachathurika at gmail.com> wrote: > > Hi Lenny, > > > > Got it. Need to source openrc file in devstack folder. > > > > Thanks ! > > > > On Thu, Jul 18, 2019 at 2:17 PM Manula Thantriwatte < > manulachathurika at gmail.com> wrote: > > Hi Lenny, > > > > When I run the above command it asking password. OS password and > OpenStack admin password didn't work. Also I haven't set any password when > installing DevStack. > > > > Is there a default password for this ? > > > > Thanks ! > > > > On Thu, Jul 18, 2019 at 1:10 PM Lenny Verkhovsky > wrote: > > Hi, > > You need to create public ip, add it to network port and try to ssh to > the image > > > > # openstack floating ip create public > > # openstack floating ip set --port > > > > Best Regards > > Lenny > > > > From: Manula Thantriwatte > > Sent: Thursday, July 18, 2019 10:28 AM > > To: openstack-discuss > > Subject: Unable to ssh openstack instance > > > > Hi All, > > > > I'm trying to ssh openstack image I have created using cirros image. But > I'm getting following error. > > > > ssh: connect to host 172.24.4.147 port 22: No route to host > > > > But when I go to the instance console I'm able to login. > > > > For the security groups I have enable port 22. > > > > I have associated floating IP as well. > > > > How to resolve this? > > > > Thanks ! > > > > -- > > Regards, > > Manula Chathurika Thantriwatte > > phone : (+94) 772492511 > > email : manulachathurika at gmail.com > > Linkedin : http://lk.linkedin.com/in/manulachathurika > > blog : http://manulachathurika.blogspot.com/ > > > > > > -- > > Regards, > > Manula Chathurika Thantriwatte > > phone : (+94) 772492511 > > email : manulachathurika at gmail.com > > Linkedin : http://lk.linkedin.com/in/manulachathurika > > blog : http://manulachathurika.blogspot.com/ > > > > > > -- > > Regards, > > Manula Chathurika Thantriwatte > > phone : (+94) 772492511 > > email : manulachathurika at gmail.com > > Linkedin : http://lk.linkedin.com/in/manulachathurika > > blog : http://manulachathurika.blogspot.com/ > > > > > > -- > > Regards, > > Manula Chathurika Thantriwatte > > phone : (+94) 772492511 > > email : manulachathurika at gmail.com > > Linkedin : http://lk.linkedin.com/in/manulachathurika > > blog : http://manulachathurika.blogspot.com/ > > — > Slawek Kaplonski > Senior software engineer > Red Hat > > -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Thu Jul 18 12:30:56 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Thu, 18 Jul 2019 21:30:56 +0900 Subject: [nova] API updates week 19-29 Message-ID: <16c0510641f.114a3898c284151.5364448129993326670@ghanshyammann.com> Hello Everyone, Please find the Nova API updates of this week. API Related BP : ============ COMPLETED: 1. Support adding description while locking an instance: - https://blueprints.launchpad.net/nova/+spec/add-locked-reason Code Ready for Review: ------------------------------ 1. Add host and hypervisor_hostname flag to create server - Topic: https://review.opendev.org/#/q/topic:bp/add-host-and-hypervisor-hostname-flag-to-create-server+(status:open+OR+status:merged) - Weekly Progress: API patch is merged. Oonly left is OSC patch and doc patch which is already +A. 2. Specifying az when restore shelved server - Topic: https://review.opendev.org/#/q/topic:bp/support-specifying-az-when-restore-shelved-server+(status:open+OR+status:merged) - Weekly Progress: Brin updated the patch after review comments. Schema and tests are fixed now and ready for next round of review. 3. Nova API cleanup - Topic: https://review.opendev.org/#/q/topic:bp/api-consistency-cleanup+(status:open+OR+status:merged) - Weekly Progress: Updated patch with latest microverion. Ready for review. This is in runway round for next week. Specs are merged and code in-progress: ------------------------------ ------------------ 4. Nova API policy improvement - Topic: https://review.openstack.org/#/q/topic:bp/policy-default-refresh+(status:open+OR+status:merged) - Weekly Progress: No much progress on this. Last week, I started the testing coverage of the existing policies with little refactoring on John PoC. 5. Detach and attach boot volumes: - Topic: https://review.openstack.org/#/q/topic:bp/detach-boot-volume+(status:open+OR+status:merged) - Weekly Progress: No Progress. Patches are in merge conflict. Spec Ready for Review: ----------------------------- 1. Support for changing deleted_on_termination after boot -Spec: https://review.openstack.org/#/c/580336/ - Weekly Progress: No update this week. Pending on Lee Yarwood proposal after PTG discussion. 3. Support delete_on_termination in volume attach api -Spec: https://review.openstack.org/#/c/612949/ - Weekly Progress: No updates this week. matt recommend to merging this with 580336 which is pending on Lee Yarwood proposal. Previously approved Spec needs to be re-proposed for Train: --------------------------------------------------------------------------- 1. Servers Ips non-unique network names : - https://blueprints.launchpad.net/nova/+spec/servers-ips-non-unique-network-names - https://review.openstack.org/#/q/topic:bp/servers-ips-non-unique-network-names+(status:open+OR+status:merged) - I remember I planned this to re-propose but could not get time. If anyone would like to help on this please repropose. otherwise I will start this in U cycle. 2. Volume multiattach enhancements: - https://blueprints.launchpad.net/nova/+spec/volume-multiattach-enhancements - https://review.openstack.org/#/q/topic:bp/volume-multiattach-enhancements+(status:open+OR+status:merged) - This also need volutneer - http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007411.html Others: 1. Add API ref guideline for body text - 2 api-ref are left to fix. Bugs: ==== No progress report in this week. NOTE- There might be some bug which is not tagged as 'api' or 'api-ref', those are not in the above list. Tag such bugs so that we can keep our eyes. From sfinucan at redhat.com Thu Jul 18 12:58:34 2019 From: sfinucan at redhat.com (Stephen Finucane) Date: Thu, 18 Jul 2019 13:58:34 +0100 Subject: [doc8] future development and maintenance In-Reply-To: <273715c4f2d4933232fc6b26cb982609daa1a2c7.camel@redhat.com> References: <9D091EF4-60F2-490C-BF76-AAD7A68FB8A1@redhat.com> <273715c4f2d4933232fc6b26cb982609daa1a2c7.camel@redhat.com> Message-ID: On Thu, 2019-07-11 at 17:10 +0100, Stephen Finucane wrote: > On Thu, 2019-07-11 at 12:01 +0100, Sorin Sbarnea wrote: > > It seems that the doc8 project is lacking some love, regardless the > > fact that is used by >90 projects from opendev.org. > > > > https://review.opendev.org/#/q/project:x/doc8 > > > > Last merge and release was more than 2 years ago and no reviews were > > performed either. > > > > I think it would be in our interest to assure that doc8 maintenance > > continues and that we can keep it usable. > > > > I would like to propose extenting the list of cores from the current > > 4 ones that I already listed in CC with 3 more, so we can effectively > > make a change that gets merged and later released (anyone willing to > > help?) > > > > If current cores agree, I would be happy to help with maintenance. I > > estimate that the effort needed would likely be less than 1h/month in > > longer term. If there is a desire to move it to github/travis, I > > would not mind either. > > I'd be tentatively interested in helping out here, though it's not as > if I don't already have a lot on my plate :) > > I wonder if this might have better success outside of OpenStack, > perhaps in the sphinx-doc or sphinx-contrib GitHub repo? Following up on this, the PyCQA organisation on GitHub have kindly accepted our request to move doc8 there [1]. I'd like to start with the paperwork of moving this across shortly and work with infra to get PyPI and the likes configured. However, if anyone has any objections to this plan please making them known now. Cheers, Stephen [1] https://mail.python.org/pipermail/code-quality/2019-July/001112.html From pabelanger at redhat.com Thu Jul 18 13:16:49 2019 From: pabelanger at redhat.com (Paul Belanger) Date: Thu, 18 Jul 2019 09:16:49 -0400 Subject: opensips with asterisk integration question In-Reply-To: References: Message-ID: <20190718131649.GA2898@localhost.localdomain> On Wed, Jul 17, 2019 at 10:28:14PM -0400, Satish Patel wrote: > I am trying to conigured opensips as registar and asterisk for media > services using this doc: > https://www.opensips.org/Documentation/Tutorials-OpenSIPSAsteriskIntegration > Greeting! You are currently on the OpenStack mailing list, you likely want to use: http://lists.digium.com/mailman/listinfo/asterisk-users > so far everything went well and i am able to use VoiceMail, MeetMe > etc. but when i am trying to dial any SIP phone getting following > error > These aren't errors, but warnings and notice. It shouldn't affect anything, but asterisk is suggesting you make changes. > Asterisk logs: > > [Jul 17 15:23:05] NOTICE[9933][C-00000001]: chan_sip.c:31847 > build_peer: The 'username' field for sip peers has been deprecated in > favor of the term 'defaultuser' > [Jul 17 15:23:05] WARNING[9933][C-00000001]: sip/config_parser.c:817 > sip_parse_nat_option: nat=yes is deprecated, use > nat=force_rport,comedia instead > [Jul 17 15:23:05] WARNING[9972][C-00000001]: sip/config_parser.c:817 > sip_parse_nat_option: nat=yes is deprecated, use > nat=force_rport,comedia instead > [Jul 17 15:23:05] NOTICE[9933][C-00000001]: chan_sip.c:24288 > handle_response_invite: Failed to authenticate on INVITE to '"1001" > ;tag=as1986a977' > voip*CLI> > Looking at your config below, I don't see you setting a secret. It is possible your opensips server is expecting one. If not, I would set one to help avoid unathenticated calls. > In mysql sipusers look like following: > > *************************** 3. row *************************** > name: 1001 > username: 1001 > type: friend > secret: NULL > host: opensips.org > callerid: NULL > context: default > mailbox: 1001 > nat: yes > qualify: no > fromuser: 1001 > authuser: NULL > fromdomain: opensips.org > insecure: NULL > canreinvite: no > disallow: NULL > allow: NULL > restrictcid: NULL > defaultip: opensips.org > ipaddr: opensips.org > port: 5060 > regseconds: NULL > > basis dialplan > > ; announcement demo > exten => 2000,1,Ringing > exten => 2000,2,Playback(welcome) > exten => 2000,3,Hangup > > ; voicemail > exten => 777,1,Ringing > exten => 777,n,Wait(1) > exten => 777,n,Answer > exten => 777,n,Wait(1) > exten => 777,n,VoiceMailMain(@default) > exten => 777,n,Hangup > > exten => 1001,1,Dial(SIP/1001) > exten => 1002,1,Dial(SIP/1002) > From mnaser at vexxhost.com Thu Jul 18 14:09:44 2019 From: mnaser at vexxhost.com (Mohammed Naser) Date: Thu, 18 Jul 2019 10:09:44 -0400 Subject: [openstack-ansible][tripleo] creating collaborative acl Message-ID: Hi everyone, I briefly brought this up in the TripleO Ansible meeting however there seems to be more work happening together on OpenStack Ansible roles from the TripleO side (alongside cross-project gating, which is awesome!) Having said that, it seems like we're building more of an upstream relationship to TripleO but I think there is a lot of benefit of making it more of a collaboration between the teams. Therefore, I'd like to propose that we create a Gerrit group called "openstack-ansible-collab". This ACL will be seeded with tripleo-ansible-core and openstack-ansible-core and it will ideally be placed on two initial repositories: - openstack/ansible-config_template - openstack/ansible-role-python_venv_build - openstack/openstack-ansible-os_tempest Those are repositories that are actively being used by TripleO and they are even getting CI tested from TripleO. They make the most sense to start with. TripleO has been around for a while and they have a solid core team that I (personally) feel okay with having core to those repos for now (and more as we work on them together). In terms of governance, these projects will remain under OpenStack-Ansible governance. Please let me know what you think of this proposal? Thanks, Mohammed From emilien at redhat.com Thu Jul 18 14:22:39 2019 From: emilien at redhat.com (Emilien Macchi) Date: Thu, 18 Jul 2019 10:22:39 -0400 Subject: [openstack-ansible][tripleo] creating collaborative acl In-Reply-To: References: Message-ID: On Thu, Jul 18, 2019 at 10:17 AM Mohammed Naser wrote: > Hi everyone, > > I briefly brought this up in the TripleO Ansible meeting however > there seems to be more work happening together on OpenStack Ansible > roles from the TripleO side (alongside cross-project gating, which is > awesome!) > > Having said that, it seems like we're building more of an upstream > relationship to TripleO but I think there is a lot of benefit of > making it more of a collaboration between the teams. Therefore, I'd > like to propose that we create a Gerrit group called > "openstack-ansible-collab". > > This ACL will be seeded with tripleo-ansible-core and > openstack-ansible-core and it will ideally be placed on two initial > repositories: > > - openstack/ansible-config_template > - openstack/ansible-role-python_venv_build > - openstack/openstack-ansible-os_tempest > We need to be careful about tripleo-ansible-core, it's actually including all TripleO cores: https://review.opendev.org/#/admin/groups/448,members We usually trust ourselves enough to not merge things we're not comfortable with but if that doesn't work for you, we can remove that group inclusion and select a group of people to add in tripleo-ansible-core. > Those are repositories that are actively being used by TripleO and > they are even getting CI tested from TripleO. They make the most > sense to start with. TripleO has been around for a while and they > have a solid core team that I (personally) feel okay with having core > to those repos for now (and more as we work on them together). > > In terms of governance, these projects will remain under > OpenStack-Ansible governance. Please let me know what you think of > this proposal? > I don't think it's a problem, +1. Thanks for the great proposal! -- Emilien Macchi -------------- next part -------------- An HTML attachment was scrubbed... URL: From aj at suse.com Thu Jul 18 14:26:25 2019 From: aj at suse.com (Andreas Jaeger) Date: Thu, 18 Jul 2019 16:26:25 +0200 Subject: [openstack-ansible][tripleo] creating collaborative acl In-Reply-To: References: Message-ID: <5ff846d3-56a0-59f8-b514-a939394ed9d3@suse.com> On 18/07/2019 16.09, Mohammed Naser wrote: > Hi everyone, > > I briefly brought this up in the TripleO Ansible meeting however > there seems to be more work happening together on OpenStack Ansible > roles from the TripleO side (alongside cross-project gating, which is > awesome!) > > Having said that, it seems like we're building more of an upstream > relationship to TripleO but I think there is a lot of benefit of > making it more of a collaboration between the teams. Therefore, I'd > like to propose that we create a Gerrit group called > "openstack-ansible-collab". > > This ACL will be seeded with tripleo-ansible-core and > openstack-ansible-core and it will ideally be placed on two initial > repositories: > > - openstack/ansible-config_template > - openstack/ansible-role-python_venv_build > - openstack/openstack-ansible-os_tempest Why not add triploe-core as group to the existing ACLs? Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF: Nils Brauckmann, Felix Imendörffer, Enrica Angelone, HRB 247165 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From fungi at yuggoth.org Thu Jul 18 14:27:11 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 18 Jul 2019 14:27:11 +0000 Subject: [openstack-ansible][tripleo] creating collaborative acl In-Reply-To: References: Message-ID: <20190718142711.gry6ue2p4s7qyog6@yuggoth.org> On 2019-07-18 10:09:44 -0400 (-0400), Mohammed Naser wrote: [...] > two initial repositories: > > - openstack/ansible-config_template > - openstack/ansible-role-python_venv_build > - openstack/openstack-ansible-os_tempest > [...] I see that you, like any proper computer scientist, always count from zero. ;) > In terms of governance, these projects will remain under > OpenStack-Ansible governance. Please let me know what you think > of this proposal? This is awesome; I'm thrilled to see another good example of cross-project collaboration surfacing in the community! -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From aschultz at redhat.com Thu Jul 18 14:36:15 2019 From: aschultz at redhat.com (Alex Schultz) Date: Thu, 18 Jul 2019 08:36:15 -0600 Subject: [openstack-ansible][tripleo] creating collaborative acl In-Reply-To: <5ff846d3-56a0-59f8-b514-a939394ed9d3@suse.com> References: <5ff846d3-56a0-59f8-b514-a939394ed9d3@suse.com> Message-ID: On Thu, Jul 18, 2019 at 8:31 AM Andreas Jaeger wrote: > On 18/07/2019 16.09, Mohammed Naser wrote: > > Hi everyone, > > > > I briefly brought this up in the TripleO Ansible meeting however > > there seems to be more work happening together on OpenStack Ansible > > roles from the TripleO side (alongside cross-project gating, which is > > awesome!) > > > > Having said that, it seems like we're building more of an upstream > > relationship to TripleO but I think there is a lot of benefit of > > making it more of a collaboration between the teams. Therefore, I'd > > like to propose that we create a Gerrit group called > > "openstack-ansible-collab". > > > > This ACL will be seeded with tripleo-ansible-core and > > openstack-ansible-core and it will ideally be placed on two initial > > repositories: > > > > - openstack/ansible-config_template > > - openstack/ansible-role-python_venv_build > > - openstack/openstack-ansible-os_tempest > > Why not add triploe-core as group to the existing ACLs? > The goal isn't necessarily to add tripleo-core, but rather tripleo-ansible-core. It just so happens that tripleo-ansible-core contains tripleo-core at the moment. We might want to add folks to tripleo-ansible but not tripleo-core. I think tripleo-ansible-core is the correct group in this case. We'll probably want to evaluate how we're doing this now that it'll be reused. > > Andreas > -- > Andreas Jaeger aj at suse.com Twitter: jaegerandi > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg > GF: Nils Brauckmann, Felix Imendörffer, Enrica Angelone, > HRB 247165 (AG Nürnberg) > GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at stackhpc.com Thu Jul 18 15:02:10 2019 From: pierre at stackhpc.com (Pierre Riteau) Date: Thu, 18 Jul 2019 17:02:10 +0200 Subject: [blazar] IRC meeting today: upstream contributions Message-ID: Hello, We have our biweekly Blazar IRC meeting in around one hour on #openstack-meeting-alt: https://wiki.openstack.org/wiki/Meetings/Blazar#Agenda_for_18_Jul_2019_.28Americas.29 I would like to focus the discussion on the status of upstream contributions from our user community. Everyone is welcome to join. Cheers, Pierre From dmendiza at redhat.com Thu Jul 18 15:20:25 2019 From: dmendiza at redhat.com (=?UTF-8?Q?Douglas_Mendiz=c3=a1bal?=) Date: Thu, 18 Jul 2019 10:20:25 -0500 Subject: Barbican support for Window In-Reply-To: References: Message-ID: <8ad5bbdb-3f3e-ebc1-4c99-db0f90e6a579@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi Rayudu, In theory Barbican should be able to run in a Windows environment. However, we don't test the project on Windows, so things may be broken. If you want to try, you should be able to download the barbican source and pip install it. I'm not sure I understand your second question? Barbican provides Key Management, so you should be able to use that in your product. Regards, - - Douglas Mendizábal Barbican PTL On 6/11/19 12:23 PM, Garaga Rayudu wrote: > Hi Team, > > > > Is it supported for window OS. If Yes, please let me know more > details about installation. > > > > Also let me know should I integrate with our product freely to > support key management. Since it look like open source product. > > > > Thanks, > > Rayudu > -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEan2ddQosxMRNS/FejiZC4mXuYkoFAl0wjjkACgkQjiZC4mXu YkpQ2wgAkl8ofiD2uS2JnopwnsHSF7edguwtr76rsjNFFBW6wJDHEeYP4834T8bW D1d5znMSIGL8wHndIhhB8DzqqVkQ4EDadHP7O5FRhVg+r36yixJ78RrJsk1YjiFs m6lBL45J1gkCEQimsEaU00072R6MlLnPEU2THsS8CKkHYDNM1CrBZt/OilFCHvs6 ZqO+qN9XdBbx2ylqee5/PWvlDkx4WStDC3dDXm9Uh8MJwPvlDZUUEVhFkeMosPa7 zjWg5trevW0nRgNtmnccLpMRPrYloEdAucEKI+eQ8cJeRcBZKzbs37mqrZMONcAS lRb39kgLrai5KQT5THyw9Ig/t2FX9A== =WpB1 -----END PGP SIGNATURE----- From isanjayk5 at gmail.com Thu Jul 18 16:16:31 2019 From: isanjayk5 at gmail.com (Sanjay K) Date: Thu, 18 Jul 2019 21:46:31 +0530 Subject: [stein][ceilometer] file/directory structure changes in ceilometer/agent and compute/pollsters with respect to liberty ceilometer Message-ID: Hi All, Currently we are trying to upgrade our older customized openstack deployment by upgrading it to Openstack Stein release from Openstack liberty release. In the older version, we had customized ceilometer source to integrate graphite so that we can send some metrices information to the carbon db which will be consumed by some other UI tools or application. For that we modified code in ceilometer/opts.py in list_opts() with - *('graphite_config', ceilometer.graphite.dispatcher_opts),* and defined dispatcher_opts for graphite host and ports in a new graphite.py file. Also we did some changes in ceilometer/agent/manager.py to import graphite config values like - *cfg.CONF.import_group('graphite_config', 'ceilometer.graphite')* In compute/pollsters, we modified cpu.py, memory.py and net.py files to send the metric data to graphite adding new function for it. *cfg.CONF.import_group('graphite_config', 'ceilometer.service')availability_zone = cfg.CONF.graphite_config.availability_zonehostname = socket.gethostname()CARBON_SERVER = cfg.CONF.graphite_config.graphite_hostCARBON_PORT = cfg.CONF.graphite_config.graphite_port* *def send_to_graphite(self,instance,cpupercent): path = 'ceilometer.' + availability_zone + '.' + hostname + '.' + instance + '.cpu.utilization' timestamp = int(time.time()) message = '%s %s %d\n' % (path, cpupercent, timestamp) sock = socket.socket() sock.connect((CARBON_SERVER, CARBON_PORT)) sock.sendall(message) sock.close()* The same function we redefined for cpu, memory and net. We also used this function inside get_sample() function to send the data to graphite. As I was applying the same changes into stein ceilometer, I found there are lot of structural changes in files and source code level. I was trying to correlate the old change with respect to new source in stein. However the files/directory structure is changed inside ceilometer/agent and compute/pollsters. Can you please guide me how these agent and pollsters changes can be mapped into the newer source code so that I can apply my old code changes in the latest ceilometer source files? As being new to openstack system, I don't have complete knowledge of what happened to the code in between liberty and stein version though I have gone through release notes and few details about these openstack services. Any pointers to specs/code reviews with respect to these structural changes will be helpful to me. Thank you for your help and pointers on this. best regards, Sanjay -------------- next part -------------- An HTML attachment was scrubbed... URL: From kecarter at redhat.com Thu Jul 18 17:16:16 2019 From: kecarter at redhat.com (Kevin Carter) Date: Thu, 18 Jul 2019 12:16:16 -0500 Subject: [TripleO] Transformation squad meeting time Message-ID: Hello, I'm writing to get feedback on what the best meeting time would be for all parties interested in participating in the TripleO Transformation Squad efforts. During this weeks meeting, it was raised that the current weekly time, Thursday 13:00 UTC, is less than ideal for a bunch of folks, and we should consider changing it [0]. To ensure we're getting everyone involved in the process, I've created a doodle that will run for one week [1]. In this Doodle, I'd like folks to vote on whatever time works best for them. During the next meeting, 25 Thursday 13:00 UTC, the results of the survey will be announced and the day and time with the highest number of votes will be used for all future meetings. Thank you, and please go vote! [0] http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-07-18-13.01.log.html#l-168 [1] https://doodle.com/poll/nkkqnkarhbgp8cxr -- Kevin Carter IRC: kecarter -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at openstack.org Thu Jul 18 19:19:05 2019 From: chris at openstack.org (Chris Hoge) Date: Thu, 18 Jul 2019 12:19:05 -0700 Subject: [baremetal-sig][ironic][il8n] Updated whitepaper timeline Message-ID: <0B408565-5AB0-4437-A42E-B71E656FD8E2@openstack.org> Hi everyone, I’ve updated the bare metal whitepaper draft to include an editorial schedule (also included in this email) for the planned November publication date. Please take a look and offer feedback, particularly the internationalzation and translation team. With the publication planned to coincide with the Shanghai Summit, we would like to offer a translation in Chinese if possible. https://docs.google.com/document/d/1KBhJcmpCTm8hn0BX-jO5MAyBYf2RHM5Jh5ib7GTZogM/edit# July 22 - Solicit contributions from Bare Metal SIG. July 29 - Review existing document and outline. August 5 - Assign sections to Bare Metal SIG volunteers. September 2 - Review of existing work, graphic and design requests to OSF. October 1 - Content finalized for editorial review and final graphics production. October 1 - Content to Il8n team for translation October 21 - Content (text and graphics) finalized for final copy preparation. November 4 - Publication for OpenInfrastructure Summit Shanghai. Thanks! Chris From gagehugo at gmail.com Thu Jul 18 20:04:09 2019 From: gagehugo at gmail.com (Gage Hugo) Date: Thu, 18 Jul 2019 15:04:09 -0500 Subject: [Security SIG] Weekly Newsletter July 18th 2019 Message-ID: ## Week of: 18 July 2019 - Security SIG Meeting Info: http://eavesdrop.openstack.org/#Security_SIG_meeting - Weekly on Thursday at 1500 UTC in #openstack-meeting - Agenda: https://etherpad.openstack.org/p/security-agenda - https://security.openstack.org/ - https://wiki.openstack.org/wiki/Security-SIG ## Meeting Notes - Summary: http://eavesdrop.openstack.org/meetings/security/2019/security.2019-07-18-15.01.html - Shanghai PTG - The Security SIG will not have a designated room for the Shanghai PTG, however fungi will likely be in attendance and is happy to discuss how security is a good thing. - Syntribos - We will send out an email later this week and see about retiring the project. - Newsletter etherpad corruption - The etherpad that hosts the templates and past newsletters became corrupt sometime over the holiday. Big thanks to fungi for helping fix it! ## VMT Reports - A full list of publicly marked security issues can be found here: https://bugs.launchpad.net/ossa/ - No new public security bugs this week -------------- next part -------------- An HTML attachment was scrubbed... URL: From cheng1.li at intel.com Thu Jul 18 01:59:48 2019 From: cheng1.li at intel.com (Li, Cheng1) Date: Thu, 18 Jul 2019 01:59:48 +0000 Subject: [networking-ovs-dpdk In-Reply-To: References: Message-ID: Please check the nova-compute log on os-compute-01 to see if any useful info. It may locate at /var/log/nova/ Thanks, Cheng From: Elanchezian Settu [mailto:elanchezian.settu at gigamon.com] Sent: Wednesday, July 17, 2019 2:14 PM To: openstack-dev at lists.openstack.org Subject: [networking-ovs-dpdk Hi, I'm trying to setup a multi-node OpenStack environment with OVS-DPDK. Unfortunately I'm running into issue that compute node not getting listed in hypervisor list as below screenshot. Due to this I'm unable to create instances on compute node. Herewith I had attached the local.conf files of both controller and compute node. Can someone help me to resolve this issue. [cid:image001.jpg at 01D53D4F.2602DC10] Thanks & Regards, Elan This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. NOTE that all incoming emails sent to Gigamon email accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional emails ("spam"). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 140947 bytes Desc: image001.jpg URL: From elanchezian.settu at gigamon.com Thu Jul 18 05:11:19 2019 From: elanchezian.settu at gigamon.com (Elanchezian Settu) Date: Thu, 18 Jul 2019 05:11:19 +0000 Subject: [networking-ovs-dpdk In-Reply-To: References: Message-ID: Hi Cheng, Thanks for your response. I had installed via Devstack, hence couldn't see nova directory in /var/log. Is there any other location I may need to look? And also please specify what logs I need to check in that log file. Thanks & Regards, Elan From: Li, Cheng1 [mailto:cheng1.li at intel.com] Sent: Thursday, July 18, 2019 7:30 AM To: Elanchezian Settu; openstack-dev at lists.openstack.org Subject: RE: [networking-ovs-dpdk Please check the nova-compute log on os-compute-01 to see if any useful info. It may locate at /var/log/nova/ Thanks, Cheng From: Elanchezian Settu [mailto:elanchezian.settu at gigamon.com] Sent: Wednesday, July 17, 2019 2:14 PM To: openstack-dev at lists.openstack.org Subject: [networking-ovs-dpdk Hi, I'm trying to setup a multi-node OpenStack environment with OVS-DPDK. Unfortunately I'm running into issue that compute node not getting listed in hypervisor list as below screenshot. Due to this I'm unable to create instances on compute node. Herewith I had attached the local.conf files of both controller and compute node. Can someone help me to resolve this issue. [cid:image001.jpg at 01D53D55.51BF8C80] Thanks & Regards, Elan This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. NOTE that all incoming emails sent to Gigamon email accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional emails ("spam"). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 140947 bytes Desc: image001.jpg URL: From manulachathurika at gmail.com Thu Jul 18 07:25:09 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Thu, 18 Jul 2019 12:55:09 +0530 Subject: Unable to ssh openstack instance Message-ID: Hi All, I'm trying to ssh openstack image I have created using cirros image. But I'm getting following error. ssh: connect to host 172.24.4.147 port 22: No route to host But when I go to the instance console I'm able to login. For the security groups I have enable port 22. I have associated floating IP as well. How to resolve this? Thanks ! -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2019-07-18 12-53-25.png Type: image/png Size: 17832 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2019-07-18 12-54-51.png Type: image/png Size: 12560 bytes Desc: not available URL: From elanchezian.settu at gigamon.com Thu Jul 18 16:10:09 2019 From: elanchezian.settu at gigamon.com (Elanchezian Settu) Date: Thu, 18 Jul 2019 16:10:09 +0000 Subject: [networking-ovs-dpdk In-Reply-To: References: Message-ID: Hi Cheng, Looks like the nova logs are running in syslog file itself. Herewith attached the error log details captured with respect to nova process. And also noticed that n-api-meta.service is not running correctly. Could this be a issue. ubuntu at os-compute-01:~$ sudo systemctl status devstack at n-api-meta.service ● devstack at n-api-meta.service - Devstack devstack at n-api-meta.service Loaded: loaded (/etc/systemd/system/devstack at n-api-meta.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-07-16 10:51:21 UTC; 2 days ago Main PID: 28278 (uwsgi) Status: "uWSGI is ready" Tasks: 10 (limit: 4915) CGroup: /system.slice/system-devstack.slice/devstack at n-api-meta.service ├─28278 nova-api-metauWSGI master ├─28286 nova-api-metauWSGI worker 1 ├─28287 nova-api-metauWSGI worker 2 ├─28288 nova-api-metauWSGI worker 3 ├─28289 nova-api-metauWSGI worker 4 ├─28290 nova-api-metauWSGI worker 5 ├─28291 nova-api-metauWSGI worker 6 ├─28292 nova-api-metauWSGI worker 7 ├─28293 nova-api-metauWSGI worker 8 └─28294 nova-api-metauWSGI http 1 Jul 16 10:51:22 os-compute-01 devstack at n-api-meta.service[28278]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/c Jul 16 10:51:22 os-compute-01 devstack at n-api-meta.service[28278]: ERROR nova reraise(type(exception), exception, tb=exc_tb, cause=cause) Jul 16 10:51:22 os-compute-01 devstack at n-api-meta.service[28278]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine Jul 16 10:51:22 os-compute-01 devstack at n-api-meta.service[28278]: ERROR nova cursor, statement, parameters, context Jul 16 10:51:22 os-compute-01 devstack at n-api-meta.service[28278]: ERROR nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine Jul 16 10:51:22 os-compute-01 devstack at n-api-meta.service[28278]: ERROR nova cursor.execute(statement, parameters) Jul 16 10:51:22 os-compute-01 devstack at n-api-meta.service[28278]: ERROR nova DBNonExistentTable: (sqlite3.OperationalError) no such table: se Jul 16 10:51:22 os-compute-01 devstack at n-api-meta.service[28278]: ERROR nova Jul 16 10:51:22 os-compute-01 devstack at n-api-meta.service[28278]: unable to load app 0 (mountpoint='') (callable not found or import error) Jul 16 10:51:22 os-compute-01 devstack at n-api-meta.service[28278]: *** no app loaded. going in full dynamic mode *** Thanks & Regards, Elan From: Elanchezian Settu Sent: Thursday, July 18, 2019 10:41 AM To: Li, Cheng1; openstack-dev at lists.openstack.org Subject: RE: [networking-ovs-dpdk Hi Cheng, Thanks for your response. I had installed via Devstack, hence couldn’t see nova directory in /var/log. Is there any other location I may need to look? And also please specify what logs I need to check in that log file. Thanks & Regards, Elan From: Li, Cheng1 [mailto:cheng1.li at intel.com] Sent: Thursday, July 18, 2019 7:30 AM To: Elanchezian Settu; openstack-dev at lists.openstack.org Subject: RE: [networking-ovs-dpdk Please check the nova-compute log on os-compute-01 to see if any useful info. It may locate at /var/log/nova/ Thanks, Cheng From: Elanchezian Settu [mailto:elanchezian.settu at gigamon.com] Sent: Wednesday, July 17, 2019 2:14 PM To: openstack-dev at lists.openstack.org Subject: [networking-ovs-dpdk Hi, I’m trying to setup a multi-node OpenStack environment with OVS-DPDK. Unfortunately I’m running into issue that compute node not getting listed in hypervisor list as below screenshot. Due to this I’m unable to create instances on compute node. Herewith I had attached the local.conf files of both controller and compute node. Can someone help me to resolve this issue. [cid:image001.jpg at 01D53DB1.5B34CB70] Thanks & Regards, Elan This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. NOTE that all incoming emails sent to Gigamon email accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional emails (“spam”). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 140947 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ERROR.log Type: application/octet-stream Size: 67239 bytes Desc: ERROR.log URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: error-1.log Type: application/octet-stream Size: 26263 bytes Desc: error-1.log URL: From eandersson at blizzard.com Thu Jul 18 20:10:59 2019 From: eandersson at blizzard.com (Erik Olof Gunnar Andersson) Date: Thu, 18 Jul 2019 20:10:59 +0000 Subject: [networking-ovs-dpdk In-Reply-To: References: Message-ID: Maybe the compute isn't yet mapped to the cell. * nova-manage cell_v2 discover_hosts --verbose Best Regards, Erik Olof Gunnar Andersson From: Li, Cheng1 Sent: Wednesday, July 17, 2019 7:00 PM To: Elanchezian Settu ; openstack-dev at lists.openstack.org Subject: RE: [networking-ovs-dpdk Please check the nova-compute log on os-compute-01 to see if any useful info. It may locate at /var/log/nova/ Thanks, Cheng From: Elanchezian Settu [mailto:elanchezian.settu at gigamon.com] Sent: Wednesday, July 17, 2019 2:14 PM To: openstack-dev at lists.openstack.org Subject: [networking-ovs-dpdk Hi, I'm trying to setup a multi-node OpenStack environment with OVS-DPDK. Unfortunately I'm running into issue that compute node not getting listed in hypervisor list as below screenshot. Due to this I'm unable to create instances on compute node. Herewith I had attached the local.conf files of both controller and compute node. Can someone help me to resolve this issue. Thanks & Regards, Elan -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Thu Jul 18 20:50:21 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 18 Jul 2019 22:50:21 +0200 Subject: Unable to ssh openstack instance In-Reply-To: References: Message-ID: Hi, It seems that 172.24.4.147 is “fixed IP” of Your instance. What network type are You using there? Usually tenant network is some tunnel network type, like VxLAN or GRE. In such case this network is isolated and You will not have access to it directly from host. You can try to do SSH to such IP address from DHCP or router namespace. DHCP namespace for Your network will be created on node where neutron-dhcp-agent is running and its name will be something like “qdhcp-”. For router it’s similar, namespace will be where neutron-l3-agent which is hosting Your router is running and name of namespace will be like “qrouter-” If You connected FIP to the instance, and You have some provider network used as external net (vlan or flat) than such IPs should be probably reachable from Your host - but this depends on Your network configuration on Your host(s). > On 18 Jul 2019, at 09:25, Manula Thantriwatte wrote: > > Hi All, > > I'm trying to ssh openstack image I have created using cirros image. But I'm getting following error. > > ssh: connect to host 172.24.4.147 port 22: No route to host > > But when I go to the instance console I'm able to login. > > For the security groups I have enable port 22. > > I have associated floating IP as well. > > How to resolve this? > > Thanks ! > > -- > Regards, > Manula Chathurika Thantriwatte > phone : (+94) 772492511 > email : manulachathurika at gmail.com > Linkedin : http://lk.linkedin.com/in/manulachathurika > blog : http://manulachathurika.blogspot.com/ > — Slawek Kaplonski Senior software engineer Red Hat From alfredo.deluca at gmail.com Thu Jul 18 21:14:25 2019 From: alfredo.deluca at gmail.com (Alfredo De Luca) Date: Thu, 18 Jul 2019 23:14:25 +0200 Subject: [kolla] [freezer] freezer scheduler auth_url error In-Reply-To: References: <1563350711224.53650@konicaminolta.it> Message-ID: Hi Radosław. No problem for the typo...that happens often to me. We ll open a bug and will take it from there. Cheers On Wed, Jul 17, 2019 at 6:12 PM Radosław Piliszek < radoslaw.piliszek at gmail.com> wrote: > Sorry for the last cut message, I love it when I accidentally change > keyboard layout and have different shorcuts. :-) > > I think this might be a bug in kolla-ansible. > Please report to https://bugs.launchpad.net/kolla-ansible > > śr., 17 lip 2019 o 18:10 Radosław Piliszek > napisał(a): > >> I think this might be a bug in kolla-ansible. >> Please report to >> >> śr., 17 lip 2019 o 18:05 napisał(a): >> >>> Alfredo, >>> >>> From Freezer documentation: >>> >>> https://docs.openstack.org/freezer/latest/user/freezer-agent.html#freezer-scheduler >>> >>> "The freezer-scheduler is one of the two freezer components which is run >>> on the client nodes; the other one being the freezer-agent. It has a >>> double role: it is used both to start the scheduler process, and as a >>> cli-tool which allows the user to interact with the API." >>> >>> Gaëtan >>> >>> On 2019-07-17 10:22, Alfredo De Luca wrote: >>> >>> > Hi Gaëtan. >>> > What do you mean "WILL HAVE TO RUN INTO THE INSTANCE YOU WANT TO >>> > BACKUP"? >>> > I understand scheduler and agent must run on the same node (like >>> > openstack controller) but not sure on the VM instances you just >>> > created. >>> > >>> > Can you clarify? Cheers >>> > >>> > On Wed, Jul 17, 2019 at 3:14 PM Gaëtan Trellu >>> > wrote: >>> > Hi Alessio, >>> > >>> > freezer-scheduler is not required on Openstack plane >>> > (controler/compute), you could remove the control group from >>> > [freezer-scheduler:children] group into the Ansible inventory. >>> > >>> > Basically, freezer-scheduler and freezer-agent will have to run into >>> > the instance you want to backup. Into the instance you will need to >>> > have the user RC file with the Openstack credentials and of course >>> > Keystone URL. >>> > >>> > This is required to authenticate the user to Keystone and get Swift >>> > multi-tenancy. >>> > >>> > Gaetan >>> > >>> > On Jul 17, 2019 4:05 AM, "Fioravanti, Alessio" >>> > wrote: >>> > >>> > Hi, >>> > >>> > this is my first post here. >>> > >>> > I'm experiencing a problem using freezer-scheduler component. When I >>> > create a job and schedule it from Horizon, freezer-scheduler always >>> > fails. The error shown in freezer-scheduler.log (from fluentd >>> > container) is: >>> > >>> > 2019-07-17 07:48:23.294 6 WARNING freezer.scheduler.scheduler_job [-] >>> > Job 1754ea4157b94acb916f8b9911886792 failed backup action, retrying in >>> > 60 seconds >>> > 2019-07-17 07:49:24.342 6 ERROR freezer.scheduler.scheduler_job [-] >>> > Freezer client error: Critical Error: auth_url required to >>> > authenticate. Make sure to export >>> > OS_AUTH_URL=http://keystone_url:5000/v3 >>> > >>> > The instance has been deployed with Kolla. Here some details: >>> > >>> > Environment >>> > >>> > KOLLA VERSION: 8.0.1rc1 >>> > >>> > OPENSTACK RELEASE: stein >>> > >>> > BASE CONTAINER: centos >>> > >>> > HOST OS: ubuntu 18.04 >>> > >>> > ENVIRONMENT: 1 controller on a dedicated host + 1 compute on a >>> > dedicated host >>> > >>> > Can you please give me your help? >>> > >>> > Thanks, >>> > >>> > Alessio >>> >>> -- >>> >>> _ALFREDO_ >>> >>> -- *Alfredo* -------------- next part -------------- An HTML attachment was scrubbed... URL: From moreira.belmiro.email.lists at gmail.com Thu Jul 18 21:43:43 2019 From: moreira.belmiro.email.lists at gmail.com (Belmiro Moreira) Date: Thu, 18 Jul 2019 23:43:43 +0200 Subject: [all][ops][User-Committee] OpenStack User Survey Message-ID: Hi, The User Survey as been an instrumental source of information that helps improve OpenStack. It provides important feedback to all the different projects. Don't miss the opportunity to register or update your OpenStack deployment information. https://www.openstack.org/user-survey/survey-2019 The 2019 User Survey will close on Thursday, August 22 at 11:59pm UTC. Thank you for your collaboration, Belmiro on behalf of the User-Committee -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at openstack.org Thu Jul 18 22:46:47 2019 From: chris at openstack.org (Chris Hoge) Date: Thu, 18 Jul 2019 15:46:47 -0700 Subject: [k8s][magnum] Kubernetes In Tree Provider Scheduled for Removal in 1.17 Message-ID: <12BC41D3-C6FC-439E-8405-1846603FBC0F@openstack.org> After speaking with the SIG-Cloud-Provider leadership this week about the status of the in-tree OpenStack provider for Kubernetes, we’ve decided that barring a major blocker we will be removing the code during the 1.17 release cycle. This means that any code that depends on the in-tree provider will not work with the Kubernetes 1.17 release and after. The in-tree code has been deprecated for over a year now[1], and SIG-OpenStack has maintained an external provider[2] that has moved beyond the in-tree code in features and support. Old releases of Kubernetes will continue to ship with the OpenStack provider code and work with existing projects. While I would strongly prefer to just remove the code in the 1.16 release, the logistics just aren't feasible.[3] It's time for us to turn our support fully to the external provider and stop depending on the in-tree code. Thanks, Chris Hoge SIG-OpenStack and SIG-Cloud-Provider co-lead [1] https://github.com/kubernetes/kubernetes/pull/63524 [2] https://github.com/kubernetes/cloud-provider-openstack [3] https://github.com/kubernetes/kubernetes/pull/80027 From feilong at catalyst.net.nz Thu Jul 18 23:01:32 2019 From: feilong at catalyst.net.nz (Feilong Wang) Date: Fri, 19 Jul 2019 11:01:32 +1200 Subject: [k8s][magnum] Kubernetes In Tree Provider Scheduled for Removal in 1.17 In-Reply-To: <12BC41D3-C6FC-439E-8405-1846603FBC0F@openstack.org> References: <12BC41D3-C6FC-439E-8405-1846603FBC0F@openstack.org> Message-ID: <16cc9b7e-16d7-2d6b-e2ed-6b3842a28cf0@catalyst.net.nz> Hi Chris, Thanks for the sending this out. In Magnum, we have already switched to the external provider[1]. The only concern for me now, is the cinder support. Based on understanding, the cinder support in cloud provider openstack is based on CSI. Is it ready for production now? Are there any feature gap between the in-tree and CPO? Thanks. [1] https://github.com/openstack/magnum/commit/6c61a1a949615f6dc1df36f3098cd97466ac7238 On 19/07/19 10:46 AM, Chris Hoge wrote: > After speaking with the SIG-Cloud-Provider leadership this week about the > status of the in-tree OpenStack provider for Kubernetes, we’ve decided > that barring a major blocker we will be removing the code during the 1.17 > release cycle. This means that any code that depends on the in-tree > provider will not work with the Kubernetes 1.17 release and after. > > The in-tree code has been deprecated for over a year now[1], and > SIG-OpenStack has maintained an external provider[2] that has moved > beyond the in-tree code in features and support. > > Old releases of Kubernetes will continue to ship with the OpenStack > provider code and work with existing projects. While I would strongly > prefer to just remove the code in the 1.16 release, the logistics just > aren't feasible.[3] > > It's time for us to turn our support fully to the external provider and > stop depending on the in-tree code. > > Thanks, > Chris Hoge > SIG-OpenStack and SIG-Cloud-Provider co-lead > > [1] https://github.com/kubernetes/kubernetes/pull/63524 > [2] https://github.com/kubernetes/cloud-provider-openstack > [3] https://github.com/kubernetes/kubernetes/pull/80027 > > -- Cheers & Best regards, Feilong Wang (王飞龙) -------------------------------------------------------------------------- Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang at catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington -------------------------------------------------------------------------- From soulxu at gmail.com Fri Jul 19 01:06:07 2019 From: soulxu at gmail.com (Alex Xu) Date: Fri, 19 Jul 2019 09:06:07 +0800 Subject: [ops][nova][ceilometer] Quick show of hands: any use Intel (non-CMT) `perf` events? In-Reply-To: <20190704103148.GF19519@paraplu> References: <20190704103148.GF19519@paraplu> Message-ID: Sorry for reply late. + ceilometer The ceilometer is still using the cmt meter https://github.com/openstack/ceilometer/blob/master/ceilometer/compute/virt/libvirt/inspector.py#L224 should we deprecate them? And there are some other meter depend on perf feature https://github.com/openstack/ceilometer/blob/master/ceilometer/compute/virt/libvirt/inspector.py#L218 So sounds like we shouldn't remove the whole perf feature. Thanks Alex Kashyap Chamarthy 于2019年7月4日周四 下午6:37写道： > Heya folks, > > While removing some dead code I was wondering if anyone here uses > "non-CMT" (Cache Monitoring Technology) performance events events? I'm > referring to the events here[0], besides the first three, which are > CMT-related. > > Background > ---------- > > The Intel CMT events (there are three of them) were deprecated during > the Rocky release, in this[1] commit, and with this rationale: > > Upstream Linux kernel has deleted[*] the `perf` framework integration > with Intel CMT (Cache Monitoring Technology; or "CQM" in Linux kernel > parlance), because the feature was broken by design -- an > incompatibility between Linux's `perf` infrastructure and Intel CMT > hardware support. It was removed in upstream kernel version v4.14; but > bear in mind that downstream Linux distributions with lower kernel > versions than 4.14 have backported the said change. > > Nova supports monitoring of the above mentioned Intel CMT events > (namely: 'cmt', 'mbm_local', and 'mbm_total') via the configuration > attribute `[libvirt]/enabled_perf_events`. Given that the underlying > Linux kernel infrastructure for Intel CMT is removed, we should remove > support for it in Nova too. Otherwise enabling them in Nova, and > updating to a Linux kernel 4.14 (or above) will result in instances > failing to boot. > > To that end, deprecate support for the three Intel CMT events in > "Rocky" release, with the intention to remove support for it in > the upcoming "Stein" release. Note that we cannot deprecate / > remove `enabled_perf_events` config attribute altogether -- > since there are other[+] `perf` events besides Intel CMT. > Whether anyone is using those other events with Nova is a good > question to which we don't have an equally good answer for, if > at all. > > Now we're removing[2] support for CMT events altogether. > > Question > -------- > > What I'm wondering now is the answer to the last sentence in the above > quoted commit: "Whether anyone is using those other events with Nova is > a good question to which we don't have an equally good answer for, if at > all". > > If we know that "no one" (as if we can tell for sure) is using them, we > can get rid of more dead code. > > So, any operators using the non-CMT events from here[0]? > > [0] https://libvirt.org/formatdomain.html#elementsPerf > [1] https://opendev.org/openstack/nova/commit/fc4794acc6 —libvirt: > Deprecate support for monitoring Intel CMT `perf` events > [2] https://review.opendev.org/669129 — libvirt: Remove support for > Intel CMT `perf` event > > -- > /kashyap > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jyotishri403 at gmail.com Fri Jul 19 04:31:10 2019 From: jyotishri403 at gmail.com (Jyoti Dahiwele) Date: Fri, 19 Jul 2019 10:01:10 +0530 Subject: Linstor volume driver Message-ID: Dear Team, Does stein cinder support drbd linstor driver? -------------- next part -------------- An HTML attachment was scrubbed... URL: From manulachathurika at gmail.com Fri Jul 19 08:05:59 2019 From: manulachathurika at gmail.com (Manula Thantriwatte) Date: Fri, 19 Jul 2019 13:35:59 +0530 Subject: How to restart DevStack Message-ID: Hi All, When I shutdown my PC, how to restart devstack. Is it ok to run ./stack.sh again or do I need to do it from the beginning? Thanks ! -- Regards, Manula Chathurika Thantriwatte phone : (+94) 772492511 email : manulachathurika at gmail.com Linkedin : *http://lk.linkedin.com/in/manulachathurika * blog : http://manulachathurika.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gauvain.pocentek at objectif-libre.com Fri Jul 19 08:16:06 2019 From: gauvain.pocentek at objectif-libre.com (Gauvain Pocentek) Date: Fri, 19 Jul 2019 10:16:06 +0200 Subject: [cloudkitty] Stepping down from the core team Message-ID: <28f6be21.AL4AAEUnnZYAAAAAAAAAAAQR_QkAAAAAZtYAAAAAAAzbjABdMXxG@mailjet.com> Hi all, I am stepping down from the Cloudkitty Core team: I have not been able to contribute for a while, and I'm moving to a new position that doesn't involve Cloudkitty at all. Gauvain Pocentek +46 768 54 12 56 www.objectif-libre.com | @OLNordics | www.linkedin.com/company/objectif-libre From anlin.kong at gmail.com Fri Jul 19 09:15:48 2019 From: anlin.kong at gmail.com (Lingxian Kong) Date: Fri, 19 Jul 2019 21:15:48 +1200 Subject: How to restart DevStack In-Reply-To: References: Message-ID: As far as I know, it'd better run "./unstack.sh; ./clear.sh" before running "./stack.sh", because some devstack plugins are not idempotent. Best regards, Lingxian Kong Catalyst Cloud On Fri, Jul 19, 2019 at 8:10 PM Manula Thantriwatte < manulachathurika at gmail.com> wrote: > Hi All, > > When I shutdown my PC, how to restart devstack. Is it ok to run ./stack.sh > again or do I need to do it from the beginning? > > Thanks ! > > -- > Regards, > Manula Chathurika Thantriwatte > phone : (+94) 772492511 > email : manulachathurika at gmail.com > Linkedin : *http://lk.linkedin.com/in/manulachathurika > * > blog : http://manulachathurika.blogspot.com/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Fri Jul 19 09:20:33 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Fri, 19 Jul 2019 11:20:33 +0200 Subject: [queens][cinder] emc fc driver Message-ID: Hello, Does anyone use cinder with emc san via FC ? Does it work ? Regards Ignazio -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdent+os at anticdent.org Fri Jul 19 11:18:00 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Fri, 19 Jul 2019 12:18:00 +0100 (BST) Subject: [placement] update 19-28 Message-ID: HTML: https://anticdent.org/placement-update-19-28.html This is pupdate 19-28. Next week is the Train-2 milestone. # Most Important Based on the discussion on the [PTG attendance thread](http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007654.html) and the notes on the [related etherpad](https://etherpad.openstack.org/p/placement-shanghai-ptg) I'm going to tell the Foundation there will be approximately seven Placement team members at Shanghai but formal space or scheduling will not be required. Instead any necessary discussions will be arranged on premise. If you disagree with this, please speak up soon. The main pending feature is consumer types, see below. # What's Changed * A bug in the resource class cache used in the placement server was found and [fixed](https://review.opendev.org/671341). It will be interesting to see how this impacts performance. While it increases database reads by one (for most requests) it removes a process-wide lock, so things could improve in threaded servers. * os-resource-classes [0.5.0](https://pypi.org/project/os-resource-classes/) was released, adding FPGA and PGPU resource classes. It's been discussed in IRC that we may wish to make 1.x releases of both os-resource-classes and os-traits at some point to make it clear that they are "real". If we do this, I recommend we do it near a cycle boundary. * An [integrated-gate-placement](https://review.opendev.org/669309) zuul template has merged. A [placement change to use it](https://review.opendev.org/671257) is ready and waiting to merge. This avoids running some tests which are unrelated; for example, cinder-only tests. # Specs/Features Since spec freeze for most projects is next week and placement has merged all its specs, until U opens, I'm going to skip this section. # Stories/Bugs (Numbers in () are the change since the last pupdate.) There are 22 (-1) stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 (0) are [untagged](https://storyboard.openstack.org/#!/worklist/580). 2 (-1) are [bugs](https://storyboard.openstack.org/#!/worklist/574). 5 (0) are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 10 (-1) are [rfes](https://storyboard.openstack.org/#!/worklist/594). 4 (0) are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 17 (1). * Placement related nova [in progress bugs](https://goo.gl/vzGGDQ) on launchpad: 4 (0). # osc-placement osc-placement is currently behind by 11 microversions. * Add support for multiple member_of. There's been some useful discussion about how to achieve this, and a consensus has emerged on how to get the best results. # Main Themes ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. * This is currently the migration to add the necessary table and column. ## Cleanup Cleanup is an overarching theme related to improving documentation, performance and the maintainability of the code. The changes we are making this cycle are fairly complex to use and are fairly complex to write, so it is good that we're going to have plenty of time to clean and clarify all these things. As mentioned for a few weeks, one of the important cleanup tasks that is not yet in progress is updating the [gabbit](https://opendev.org/openstack/placement/src/branch/master/gate/gabbits/nested-perfload.yaml) that creates the nested topology that's used in nested performance testing. We've [asked the startlingx community](http://lists.starlingx.io/pipermail/starlingx-discuss/2019-July/005330.html) for input. Another cleanup that needs to start is satisfying the community wide goal of [PDF doc generation](https://storyboard.openstack.org/#!/story/2006110). I did some experimenting on this and while I was able to get a book created, the number of errors, warnings, and manual interventions required meant I gave up until there's time to do a more in-depth exploration and learn the tools. # Other Placement Miscellaneous changes can be found in [the usual place](https://review.opendev.org/#/q/project:openstack/placement+status:open). There are two [os-traits changes](https://review.opendev.org/#/q/project:openstack/os-traits+status:open) being discussed. And zero [os-resource-classes changes](https://review.opendev.org/#/q/project:openstack/os-resource-classes+status:open) (yay!). # Other Service Users New discoveries are added to the end. Merged stuff is removed. Anything that has had no activity in 4 weeks has been removed. * Nova: nova-manage: heal port allocations * nova-spec: Allow compute nodes to use DISK_GB from shared storage RP * Cyborg: Placement report * helm: add placement chart * Nova: Use OpenStack SDK for placement * Nova: Spec: Provider config YAML file * libvirt: report pmem namespaces resources by provider tree * Nova: Remove PlacementAPIConnectFailure handling from AggregateAPI * Nova: support move ops with qos ports * Nova: get_ksa_adapter: nix by-service-type confgrp hack * Blazar: Create placement client for each request * nova: Support filtering of hosts by forbidden aggregates * blazar: Send global_request_id for tracing calls * Nova: Update HostState.\*\_allocation_ratio earlier * Nova: WIP: Add a placement audit command * zun: [WIP] Use placement for unified resource management * neutron: Filter placement API endpoint by type too * OSA: Install placement osc plugin on utility machines * Kayobe: Build placement images by default * blazar: Fix placement operations in multi-region deployments # End If you've done any performance or scale testing with placement, I'd love to hear about your observations. Please let me know. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From donny at fortnebula.com Fri Jul 19 11:27:02 2019 From: donny at fortnebula.com (Donny Davis) Date: Fri, 19 Jul 2019 07:27:02 -0400 Subject: How to restart DevStack In-Reply-To: References: Message-ID: I would also imagine it's better to run those before you do the shutdown or restart. On Fri, Jul 19, 2019 at 5:18 AM Lingxian Kong wrote: > As far as I know, it'd better run "./unstack.sh; ./clear.sh" before > running "./stack.sh", because some devstack plugins are not idempotent. > > Best regards, > Lingxian Kong > Catalyst Cloud > > > On Fri, Jul 19, 2019 at 8:10 PM Manula Thantriwatte < > manulachathurika at gmail.com> wrote: > >> Hi All, >> >> When I shutdown my PC, how to restart devstack. Is it ok to run >> ./stack.sh again or do I need to do it from the beginning? >> >> Thanks ! >> >> -- >> Regards, >> Manula Chathurika Thantriwatte >> phone : (+94) 772492511 >> email : manulachathurika at gmail.com >> Linkedin : *http://lk.linkedin.com/in/manulachathurika >> * >> blog : http://manulachathurika.blogspot.com/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donny at fortnebula.com Fri Jul 19 11:30:13 2019 From: donny at fortnebula.com (Donny Davis) Date: Fri, 19 Jul 2019 07:30:13 -0400 Subject: Linstor volume driver In-Reply-To: References: Message-ID: https://docs.openstack.org/cinder/stein/reference/support-matrix.html No sure the state of it, I have never used it. But its listed in the support matrix. I also see the driver was updated just a few months ago (4), so I would give it a whirl https://opendev.org/openstack/cinder/src/branch/master/cinder/volume/drivers/linstordrv.py On Fri, Jul 19, 2019 at 12:35 AM Jyoti Dahiwele wrote: > Dear Team, > > Does stein cinder support drbd linstor driver? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moreira.belmiro.email.lists at gmail.com Fri Jul 19 12:54:41 2019 From: moreira.belmiro.email.lists at gmail.com (Belmiro Moreira) Date: Fri, 19 Jul 2019 14:54:41 +0200 Subject: [glance][ops] (OSSN-0075) Obfuscate deleted records in glance image table Message-ID: Hi, because OSSN-0075 (https://wiki.openstack.org/wiki/OSSN/OSSN-0075) we don't purge the glance "images" table. This means that we continue to have information about the deleted images in the DB. For several reasons I would like to not keep data of deleted images that can point to users activity. Currently we simply "obfuscate" all this data. See: https://review.opendev.org/#/c/671707/ I'm curious to know how this problem has been addressed by other deployments. Belmiro CERN -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.sgaravatto at gmail.com Fri Jul 19 13:26:24 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Fri, 19 Jul 2019 15:26:24 +0200 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> Message-ID: This seems indeed the problem (I will file an issue for the documentation). Thanks a lot ! If I remove/fix that property from an image, I am now able to start new instances using that image. The problem is with instances created BEFORE removing the property: I am not able to migrate them (using nova migrate), unless I remove the ImagePropertiesFilter from the scheduler filters. Moreover: if a user creates a snapshot of one of these instances, the snapshot got created with this wrong property Is there some clean way to remove/change the problematic property from the relevant instances ? Thanks, Massimo On Wed, Jul 17, 2019 at 5:41 PM Brian Rosmaita wrote: > On 7/17/19 8:14 AM, Massimo Sgaravatto wrote: > > We have just finished the update of our cloud to Rocky but we are seeing > > a strange issue with images with property: > > hypervisor_type='QEMU' > > > > The scheduling of instances created with such images fails because of > > the ImagePropertiesFilter [*] > > > > Removing that property from the image, the scheduling works. > > We also tried changing hypervisor_type='QEMU' > > --> hypervisor_type='qemu', but this didn't help. > > > > Any suggestions? > > Commit e792d50efadb36437e82381f4c84d738cee25dfd in Ocata changed the > image metadata that the ImagePropertiesFilter pays attention to: > > diff --git a/nova/scheduler/filters/image_props_filter.py > b/nova/scheduler/filters/image_props_filter.py > index 06def5c769..521a6816a3 100644 > --- a/nova/scheduler/filters/image_props_filter.py > +++ b/nova/scheduler/filters/image_props_filter.py > @@ -43,9 +43,9 @@ class ImagePropertiesFilter(filters.BaseHostFilter): > > def _instance_supported(self, host_state, image_props, > hypervisor_version): > - img_arch = image_props.get('architecture', None) > - img_h_type = image_props.get('hypervisor_type', None) > - img_vm_mode = image_props.get('vm_mode', None) > + img_arch = image_props.get('hw_architecture') > + img_h_type = image_props.get('img_hv_type') > + img_vm_mode = image_props.get('hw_vm_mode') > checked_img_props = ( > arch.canonicalize(img_arch), > hv_type.canonicalize(img_h_type), > > Looks like 'img_hv_type' is the metadata key you need to use. > > If that works, please put up a patch to the Glance "useful image > properties" docs [0], we seem to be out of date on this issue. > > [0] > > https://opendev.org/openstack/glance/src/branch/master/doc/source/admin/useful-image-properties.rst > > cheers, > brian > > > Thanks, Massimo > > > > > > > > [*] > > 2019-07-17 13:52:58.148 13312 INFO nova.filters > > [req-1863bef0-0326-46d1-a836-436227e91eef > > 6e3b136d578f4292a5c03b16f443ab3d d27fe2becea94a3e980fb9f66e2f291a - > > default default] Filtering removed all hosts f\ > > or the request with instance ID '63810b60-76e4-4e76-a1c3-e4d3932c002e'. > > Filter results: ['AggregateMultiTenancyIsolation: (start: 49, end: 37)', > > 'AggregateInstanceExtraSpecsFilter: (start: 37, end: 34)', \ > > 'RetryFilter: (start: 34, end: 34)', 'AvailabilityZoneFilter: (start: > > 34, end: 34)', 'ComputeFilter: (start: 34, end: 32)', > > 'ComputeCapabilitiesFilter: (start: 32, end: 32)', > > 'ImagePropertiesFilter: (star\ > > t: 32, end: 0)'] > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dtroyer at gmail.com Fri Jul 19 13:55:57 2019 From: dtroyer at gmail.com (Dean Troyer) Date: Fri, 19 Jul 2019 08:55:57 -0500 Subject: How to restart DevStack In-Reply-To: References: Message-ID: > On Fri, Jul 19, 2019 at 5:18 AM Lingxian Kong wrote: >> As far as I know, it'd better run "./unstack.sh; ./clear.sh" before running "./stack.sh", because some devstack plugins are not idempotent. unstack.sh and clean.sh are meant to be used to prepare for a new run of stack.sh without a reboot. unstack.sh stops all services to prepare for stack.sh to bring them up in a predictable order. clean.sh is more geared for major configuration changes like changing databases or message queues. It runs unstack.sh then goes to some effort to remove generated files and certain packages because we found services did not always restart cleanly early on, and changing databases needs package attention too. If you are not making any configuration changes, both unstack and clean should be unnecessary after a reboot. On Fri, Jul 19, 2019 at 6:29 AM Donny Davis wrote: > I would also imagine it's better to run those before you do the shutdown or restart. If you are not making major config changes between runs this should not be necessary. The idea of needing a clean shutdown is less important when all data involved is going to be deleted and re-created on the next stack.sh run anyway. dt -- Dean Troyer dtroyer at gmail.com From a.settle at outlook.com Fri Jul 19 14:05:01 2019 From: a.settle at outlook.com (Alexandra Settle) Date: Fri, 19 Jul 2019 14:05:01 +0000 Subject: [all] [ptls] [tc] [nova] [neutron] [tripleo] Volunteers that know TeX for PDF community goal In-Reply-To: References: <20190624155629.GA26343@sinanju.localdomain> <80e2e8550fd39cf9e224e24b4e6ad806acdc9e16.camel@redhat.com> Message-ID: Hi all, BTW, it looks better to prepare some place to share our knowledge, etherpad? Agreed. Thank you for your suggestion. I've setup an etherpad for each of the projects (so far) that have volunteered to get testing started. A central(ish) place for everyone to record what they're doing. You can find those etherpads here: https://wiki.openstack.org/wiki/Documentation#PDF_for_Project_Docs_-_Community_Goal Then you can keep track of everything that's an issue and work with your team to fix those. I have also created a "common issues" etherpad, where we can discuss any common issues that arise: https://etherpad.openstack.org/p/pdf-goal-train-common-problems This is anything that may be particularly unique to Sphinx, or particular to the docs project and would require input from Stephen Finucane or myself. Any questions? Please reach out. We're still looking for volunteers from each project team to undertake testing. Cheers, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From jungleboyj at gmail.com Fri Jul 19 14:27:03 2019 From: jungleboyj at gmail.com (Jay Bryant) Date: Fri, 19 Jul 2019 09:27:03 -0500 Subject: [queens][cinder] emc fc driver In-Reply-To: References: Message-ID: Ignazio, Dell/EMC has continued to be an active contributor to the Cinder project and they have a working 3rd Party CI.  So, the driver should work and if you are seeing issues please provide details to the mailing list or come join us in the #openstack-cinder channel on IRC and we can try to help you. Thanks! Jay Bryant (irc: jungleboyj) On 7/19/2019 4:20 AM, Ignazio Cassano wrote: > Hello, > Does anyone use cinder with emc san via FC ? > Does it work ? > > Regards > Ignazio From jungleboyj at gmail.com Fri Jul 19 14:31:19 2019 From: jungleboyj at gmail.com (Jay Bryant) Date: Fri, 19 Jul 2019 09:31:19 -0500 Subject: Linstor volume driver In-Reply-To: References: Message-ID: Jyoti, The Linstor driver was added in Stein, replacing LinBit's old DRBD driver.  It is covered by 3rd Party CI and should work properly. If you have any issues with the driver please let us know.  I have a contact at LinBit that can help. Thanks! Jay Bryant (irc: jungleboyj) On 7/18/2019 11:31 PM, Jyoti Dahiwele wrote: > Dear Team, > > Does stein cinder support drbd linstor driver? From pawel.konczalski at everyware.ch Fri Jul 19 14:48:44 2019 From: pawel.konczalski at everyware.ch (Pawel Konczalski) Date: Fri, 19 Jul 2019 16:48:44 +0200 Subject: Octavia: Could not retrieve certificate when create HTTPS listener using application credentials Message-ID: <62f4a5ba-4230-381b-7d9f-35f86a77227e@everyware.ch> Hi, i try to create a Octavia HTTPS listener by using application credentials but get this error: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09'] (HTTP 400) (Request-ID: req-088d6eb0-a285-4089-bc11-ff0c3097123e) # openstack secret list +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name  | Created                   | Status | Content types                             | Algorithm | Bit length | Secret type | Mode | Expiration | +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35 | cert2 | 2019-07-19T13:42:21+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes       |        256 | opaque      | cbc  | None       | | https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09 | cert1 | 2019-07-19T13:42:12+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes       |        256 | opaque      | cbc  | None       | +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ # openstack loadbalancer listener create foo-lb1 \ --name foo-lb1-https-listener \ --protocol-port 443 \ --protocol TERMINATED_HTTPS \ --insert-headers X-Forwarded-For=true,X-Forwarded-Proto=true \ --default-tls-container=https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09 \ --sni-container-refs https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09 https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35 -------------------------------- Starting new HTTPS connection (1): octavia.service.dev.example.com:443 https://octavia.service.dev.example.com:443 "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 779 RESP: [200] Connection: keep-alive Content-Length: 779 Content-Type: application/json Date: Fri, 19 Jul 2019 13:56:24 GMT Server: WSGIServer/0.1 Python/2.7.15rc1 x-openstack-request-id: req-50b5a3bb-21ec-4a46-8d5c-61035afd3423 RESP BODY: {"loadbalancers": [{"provider": "amphora", "description": "", "admin_state_up": true, "pools": [{"id": "169722d1-0a73-4283-bb42-aee5b662e2e2"}], "created_at": "2019-07-19T13:34:52", "provisioning_status": "ACTIVE", "updated_at": "2019-07-19T13:39:34", "vip_qos_policy_id": null, "vip_network_id": "2064c61c-64a1-466f-983a-af435ae1d51c", "listeners": [{"id": "169a91f9-ef5c-4d38-8449-e24b64cf082d"}], "tenant_id": "9646533a8d834978a868e81c9b9a39cf", "vip_port_id": "dcfc6e44-4092-4f2b-bd50-24e02abb078f", "flavor_id": "", "vip_address": "10.0.1.4", "vip_subnet_id": "787035dc-add4-4227-844a-1cf803625abc", "project_id": "9646533a8d834978a868e81c9b9a39cf", "id": "e2ed48ab-3261-422f-b9b5-a5aa63486ae7", "operating_status": "OFFLINE", "name": "foo-lb1"}], "loadbalancers_links": []} GET call to https://octavia.service.dev.example.com/v2.0/lbaas/loadbalancers used request id req-50b5a3bb-21ec-4a46-8d5c-61035afd3423 REQ: curl -g -i -X POST https://octavia.service.dev.example.com/v2.0/lbaas/listeners -H "Content-Type: application/json" -H "User-Agent: openstacksdk/0.19.0 keystoneauth1/3.11.1 python-requests/2.20.1 CPython/2.7.15+" -H "X-Auth-Token: {SHA256}6414e14f4e78940902b11c89567689e3cc0d3ea62227b87a1e19361685c83584" -d '{"listener": {"insert_headers": {"X-Forwarded-For": "true", "X-Forwarded-Proto": "true"}, "protocol": "TERMINATED_HTTPS", "name": "foo-lb1-https-listener", "default_tls_container_ref": "https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09", "sni_container_refs": ["https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09", "https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35"], "admin_state_up": true, "protocol_port": 443, "loadbalancer_id": "e2ed48ab-3261-422f-b9b5-a5aa63486ae7"}}' https://octavia.service.dev.example.com:443 "POST /v2.0/lbaas/listeners HTTP/1.1" 400 357 RESP: [400] Connection: keep-alive Content-Length: 357 Content-Type: application/json Date: Fri, 19 Jul 2019 13:56:27 GMT Server: WSGIServer/0.1 Python/2.7.15rc1 x-openstack-request-id: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca RESP BODY: {"debuginfo": null, "faultcode": "Client", "faultstring": "Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09']"} POST call to https://octavia.service.dev.example.com/v2.0/lbaas/listeners used request id req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca Request returned failure status: 400 Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) Traceback (most recent call last):   File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand     result = cmd.run(parsed_args)   File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run     return super(Command, self).run(parsed_args)   File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run     column_names, data = self.take_action(parsed_args)   File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/osc/v2/listener.py", line 168, in take_action     json=body)   File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper     request_id=e.request_id) OctaviaClientException: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) clean_up CreateListener: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) Traceback (most recent call last):   File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/shell.py", line 136, in run     ret_val = super(OpenStackShell, self).run(argv)   File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 279, in run     result = self.run_subcommand(remainder)   File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/shell.py", line 176, in run_subcommand     ret_value = super(OpenStackShell, self).run_subcommand(argv)   File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand     result = cmd.run(parsed_args)   File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run     return super(Command, self).run(parsed_args)   File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run     column_names, data = self.take_action(parsed_args)   File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/osc/v2/listener.py", line 168, in take_action     json=body)   File "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper     request_id=e.request_id) OctaviaClientException: Could not retrieve certificate: ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09'] (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) ------------------------------ This issue occurs only when application credentials are used. Creation of HTTP listener with applications credentials works fine, also creation of HTTPS listener when user are authenticated by user / password. Does somebody know which additional ACLs / permissions are required to fix this? BR Pawel -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5227 bytes Desc: not available URL: From senrique at redhat.com Fri Jul 19 15:09:27 2019 From: senrique at redhat.com (Sofia Enriquez) Date: Fri, 19 Jul 2019 12:09:27 -0300 Subject: Unable to ssh openstack instance In-Reply-To: References: Message-ID: Hi Manula, This guide really helps me when testing things with Devstack[1]. Hope it can help you too. Sofia [1] https://docs.openstack.org/networking-ovn/latest/contributor/testing.html#booting-vms On Thu, Jul 18, 2019 at 5:54 PM Slawek Kaplonski wrote: > Hi, > > It seems that 172.24.4.147 is “fixed IP” of Your instance. What network > type are You using there? Usually tenant network is some tunnel network > type, like VxLAN or GRE. In such case this network is isolated and You will > not have access to it directly from host. > You can try to do SSH to such IP address from DHCP or router namespace. > DHCP namespace for Your network will be created on node where > neutron-dhcp-agent is running and its name will be something like > “qdhcp-”. > For router it’s similar, namespace will be where neutron-l3-agent which is > hosting Your router is running and name of namespace will be like > “qrouter-” > > If You connected FIP to the instance, and You have some provider network > used as external net (vlan or flat) than such IPs should be probably > reachable from Your host - but this depends on Your network configuration > on Your host(s). > > > On 18 Jul 2019, at 09:25, Manula Thantriwatte < > manulachathurika at gmail.com> wrote: > > > > Hi All, > > > > I'm trying to ssh openstack image I have created using cirros image. But > I'm getting following error. > > > > ssh: connect to host 172.24.4.147 port 22: No route to host > > > > But when I go to the instance console I'm able to login. > > > > For the security groups I have enable port 22. > > > > I have associated floating IP as well. > > > > How to resolve this? > > > > Thanks ! > > > > -- > > Regards, > > Manula Chathurika Thantriwatte > > phone : (+94) 772492511 > > email : manulachathurika at gmail.com > > Linkedin : http://lk.linkedin.com/in/manulachathurika > > blog : http://manulachathurika.blogspot.com/ > > 12-54-51.png> > > — > Slawek Kaplonski > Senior software engineer > Red Hat > > > -- L. Sofía Enriquez she/her Associate Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From colleen at gazlene.net Fri Jul 19 15:16:44 2019 From: colleen at gazlene.net (Colleen Murphy) Date: Fri, 19 Jul 2019 08:16:44 -0700 Subject: [keystone] Returning core Guang Yee Message-ID: <183bb224-92fb-486e-af2f-7baf0589905c@www.fastmail.com> Hi everyone, I'm thrilled to announce that Guang Yee has agreed to rejoin the keystone core reviewer team. Guang is a former core who was instrumental in many of keystone's key features. We're lucky to have him back to lend his experience and knowledge to our current challenges. Welcome back, Guang! Colleen From tpb at dyncloud.net Fri Jul 19 15:18:10 2019 From: tpb at dyncloud.net (Tom Barron) Date: Fri, 19 Jul 2019 11:18:10 -0400 Subject: [Manila] CephFS deferred deletion In-Reply-To: References: <20190712131540.3eqvltysfix6eivd@barron.net> Message-ID: <20190719151810.u5idvqx4g5vzqooa@barron.net> On 15/07/19 16:49 +0530, Ramana Raja wrote: >On Fri, Jul 12, 2019 at 6:45 PM Tom Barron wrote: >> >> On 12/07/19 13:03 +0000, Jose Castro Leon wrote: >> >Dear all, >> > >> >Lately, one of our clients stored 300k files in a manila cephfs share. >> >Then he deleted the share in Manila. This event make the driver >> >unresponsive for several hours until all the data was removed in the >> >cluster. >> > >> >We had a quick look at the code in manila [1] and the deletion is done >> >first by calling the following api calls in the ceph bindings >> >(delete_volume[1] and then purge_volume[2]). The first call moves the >> >directory to a volumes_deleted directory. The second call does a >> >deletion in depth of all the contents of that directory. >> > >> >The last operation is the one that trigger the issue. >> > >> >We had a similar issue in the past in Cinder. There, Arne proposed to >> >do a deferred deletion of volumes. I think we could do the same in >> >Manila for the cephfs driver. >> > >> >The idea is to continue to call to the delete_volume. And then inside a >> >periodic task in the driver, asynchronously it will get the contents of >> >that directory and trigger the purge command. >> > >> >I can propose the change and contribute with the code, but before going >> >to deep I would like to know if there is a reason of having a singleton >> >for the volume_client connection. If I compare with cinder code the >> >connection is established and closed in each operation with the >> >backend. >> > >> >If you are not the maintainer, could you please point me to he/she? >> >I can post it in the mailing list if you prefer >> > >> >Cheers >> >Jose Castro Leon >> >CERN Cloud Infrastructure >> > >> >[1] >> >https://github.com/openstack/manila/blob/master/manila/share/drivers/cephfs/driver.py#L260-L267 >> > >> > >> >[2] >> >https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L700-L734 >> > >> > >> >[2] >> >https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L736-L790 >> > >> > >> >PS: The issue was triggered by one of our clients in kubernetes using >> >the Manila CSI driver >> >> Hi Jose, >> >> Let's get this fixed since there's a lot of interest in Manila CSI >> driver and I think we can expect more batched deletes with it than we >> have had historically. > >The plan is to have manila's CephFS driver use the ceph-mgr's new >volumes module, >https://github.com/ceph/ceph/blob/master/src/pybind/mgr/volumes/module.py >to create/delete manila groups/shares/snapshots, >authorize/de-authorize access to the shares. Manila shares, >essentially CephFS subdirectories with a specific data layout and >quota, are referred to as FS subvolumes, and Ceph filesystems as FS >volumes in the ceph-mgr volumes module. > >The ceph-mgr volumes modules is under active development. The latest >Ceph CSI (v1.1.0) release is the first consumer of this module. The >Ceph CSI issues CLI calls to the ceph-mgr to manage the lifecycle of >the FS subvolumes, >https://github.com/ceph/ceph-csi/pull/400 > >We're implementing the asynchronous purge of FS subvolumes in the >ceph-mgr module. The PR is close to being merged, >https://github.com/ceph/ceph/pull/28003/ >https://github.com/ceph/ceph/pull/28003/commits/483a2141fe8c9a58bc25a544412cdf5b047ad772 >http://tracker.ceph.com/issues/40036 >Additional reviews will be great. Issuing the `ceph fs subvolume rm` >command in the Ceph CSI driver (and later in the manila driver) will >move the FS subvolume to a trash directory, whose contents will be >asynchronously purged by a set of worker threads. > >> >> I've copied Ramana Raja and Patrick Donnelly since they will be able >> to answer your question about the singleton volume_client connection >> more authoritatively than I can. > >Currently, in the mgr-volumes module we establish and close connection >to a FS volume (a Ceph filesystem) for each FS subvolume (CephFS >subdirectory within the filesystem) operation, >https://github.com/ceph/ceph/pull/28082/commits/8d29816f0f3db6c7d287bbb7469db77c9de701d1#diff-cfd3b6f517caccc18f7f066395e8a4bdR174 > >Instead, we want to maintain a connection to a FS volume and perform >operations on its subvolumes, until the FS volume is deleted. This >would >reduce the time taken to perform subvolume operations, important in >CSI work loads (and in OpenStack workloads?). Th code is in review, >https://github.com/ceph/ceph/pull/28003/commits/5c41e949af9acabd612b0644de0603e374b4b42a > >Thanks, >Ramana > >> >> Thanks for volunteering to propose a review to deal with this issue! >> >> -- Tom Barron >> Jose, I think it will be better to have the async expunge managed by CephFS itself rather than by a periodic task in Manila. For one thing, non-Manila CephFS clients like Ceph-CSI have the same issue so they will benefit from the approach Ramana describes. Other storage backends that need to reclaim space for deleted shares/volumes in the background do this themselves rather than relying on the client (Manila in this case) to manage expunge or equivalents. Do you agree? Victoria Martinez de la Cruz will be working more generally to adapt the Manila CephFS driver to use 'ceph fs subvolume' rather than the current ceph_volume library calls, so perhaps you can propose modification of the share deletion code in that context. I understand from Ramana that the new interface that we need to CephFS for all this stuff will be in Nautilus, so we'll need to think through compatability issues for deployers with earlier versions of Ceph. Does needing to use Ceph Nautilus to get a proper solution for volume expunges pose any significant issues for CERN? Thanks, -- Tom Barron From johnsomor at gmail.com Fri Jul 19 15:29:02 2019 From: johnsomor at gmail.com (Michael Johnson) Date: Fri, 19 Jul 2019 08:29:02 -0700 Subject: Octavia: Could not retrieve certificate when create HTTPS listener using application credentials In-Reply-To: <62f4a5ba-4230-381b-7d9f-35f86a77227e@everyware.ch> References: <62f4a5ba-4230-381b-7d9f-35f86a77227e@everyware.ch> Message-ID: Hi Pawel, First question is what version of Octavia are you using? Older versions required you to set some ACL permissions on the secrets in Barbican. You can check this by reviewing the load balancing cookbook for the version of Octavia you are running [1]. There is a drop down in the upper right corner of the document that allows you to select a version of the document. Second question, can you expand on what you mean by "application credentials"? Is this by using a pre-created token instead of having the username/password in your environment? Third, can you check your octavia.conf settings[2]? Check the following options are either the default (commented out) or set to the same settings as the default. [certificates] cert_manager = barbican_cert_manager barbican_auth = barbican_acl_auth (Note, this is only valid in the newer versions of Octavia as noted above) Fourth (last one), can you provide the associated log output from the Octavia API process that is handling this request? Debug if you can. Michael [1] https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html#deploy-a-tls-terminated-https-load-balancer [2] https://docs.openstack.org/octavia/latest/configuration/configref.html#certificates On Fri, Jul 19, 2019 at 7:51 AM Pawel Konczalski wrote: > > Hi, > > i try to create a Octavia HTTPS listener by using application > credentials but get this error: > > Could not retrieve certificate: > ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', > 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', > 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09'] > (HTTP 400) (Request-ID: req-088d6eb0-a285-4089-bc11-ff0c3097123e) > > > # openstack secret list > +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ > | Secret href | Name | Created | Status | Content > types | Algorithm | Bit length | Secret type > | Mode | Expiration | > +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ > | > https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35 > | cert2 | 2019-07-19T13:42:21+00:00 | ACTIVE | {u'default': > u'application/octet-stream'} | aes | 256 | opaque | > cbc | None | > | > https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09 > | cert1 | 2019-07-19T13:42:12+00:00 | ACTIVE | {u'default': > u'application/octet-stream'} | aes | 256 | opaque | > cbc | None | > +--------------------------------------------------------------------------------------+-------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ > > > # openstack loadbalancer listener create foo-lb1 \ > --name foo-lb1-https-listener \ > --protocol-port 443 \ > --protocol TERMINATED_HTTPS \ > --insert-headers X-Forwarded-For=true,X-Forwarded-Proto=true \ > --default-tls-container=https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09 > \ > --sni-container-refs > https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09 > https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35 > > > -------------------------------- > > Starting new HTTPS connection (1): octavia.service.dev.example.com:443 > https://octavia.service.dev.example.com:443 "GET > /v2.0/lbaas/loadbalancers HTTP/1.1" 200 779 > RESP: [200] Connection: keep-alive Content-Length: 779 Content-Type: > application/json Date: Fri, 19 Jul 2019 13:56:24 GMT Server: > WSGIServer/0.1 Python/2.7.15rc1 x-openstack-request-id: > req-50b5a3bb-21ec-4a46-8d5c-61035afd3423 > RESP BODY: {"loadbalancers": [{"provider": "amphora", "description": "", > "admin_state_up": true, "pools": [{"id": > "169722d1-0a73-4283-bb42-aee5b662e2e2"}], "created_at": > "2019-07-19T13:34:52", "provisioning_status": "ACTIVE", "updated_at": > "2019-07-19T13:39:34", "vip_qos_policy_id": null, "vip_network_id": > "2064c61c-64a1-466f-983a-af435ae1d51c", "listeners": [{"id": > "169a91f9-ef5c-4d38-8449-e24b64cf082d"}], "tenant_id": > "9646533a8d834978a868e81c9b9a39cf", "vip_port_id": > "dcfc6e44-4092-4f2b-bd50-24e02abb078f", "flavor_id": "", "vip_address": > "10.0.1.4", "vip_subnet_id": "787035dc-add4-4227-844a-1cf803625abc", > "project_id": "9646533a8d834978a868e81c9b9a39cf", "id": > "e2ed48ab-3261-422f-b9b5-a5aa63486ae7", "operating_status": "OFFLINE", > "name": "foo-lb1"}], "loadbalancers_links": []} > GET call to > https://octavia.service.dev.example.com/v2.0/lbaas/loadbalancers used > request id req-50b5a3bb-21ec-4a46-8d5c-61035afd3423 > REQ: curl -g -i -X POST > https://octavia.service.dev.example.com/v2.0/lbaas/listeners -H > "Content-Type: application/json" -H "User-Agent: openstacksdk/0.19.0 > keystoneauth1/3.11.1 python-requests/2.20.1 CPython/2.7.15+" -H > "X-Auth-Token: > {SHA256}6414e14f4e78940902b11c89567689e3cc0d3ea62227b87a1e19361685c83584" > -d '{"listener": {"insert_headers": {"X-Forwarded-For": "true", > "X-Forwarded-Proto": "true"}, "protocol": "TERMINATED_HTTPS", "name": > "foo-lb1-https-listener", "default_tls_container_ref": > "https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09", > "sni_container_refs": > ["https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09", > "https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35"], > "admin_state_up": true, "protocol_port": 443, "loadbalancer_id": > "e2ed48ab-3261-422f-b9b5-a5aa63486ae7"}}' > https://octavia.service.dev.example.com:443 "POST /v2.0/lbaas/listeners > HTTP/1.1" 400 357 > RESP: [400] Connection: keep-alive Content-Length: 357 Content-Type: > application/json Date: Fri, 19 Jul 2019 13:56:27 GMT Server: > WSGIServer/0.1 Python/2.7.15rc1 x-openstack-request-id: > req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca > RESP BODY: {"debuginfo": null, "faultcode": "Client", "faultstring": > "Could not retrieve certificate: > ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', > 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', > 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09']"} > POST call to > https://octavia.service.dev.example.com/v2.0/lbaas/listeners used > request id req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca > Request returned failure status: 400 > Could not retrieve certificate: > ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', > 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', > 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09'] > (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) > Traceback (most recent call last): > File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in > run_subcommand > result = cmd.run(parsed_args) > File > "/home/foo/.local/lib/python2.7/site-packages/osc_lib/command/command.py", > line 41, in run > return super(Command, self).run(parsed_args) > File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, > in run > column_names, data = self.take_action(parsed_args) > File > "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/osc/v2/listener.py", > line 168, in take_action > json=body) > File > "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/api/v2/octavia.py", > line 38, in wrapper > request_id=e.request_id) > OctaviaClientException: Could not retrieve certificate: > ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', > 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', > 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09'] > (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) > clean_up CreateListener: Could not retrieve certificate: > ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', > 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', > 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09'] > (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) > Traceback (most recent call last): > File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/shell.py", > line 136, in run > ret_val = super(OpenStackShell, self).run(argv) > File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 279, in run > result = self.run_subcommand(remainder) > File "/home/foo/.local/lib/python2.7/site-packages/osc_lib/shell.py", > line 176, in run_subcommand > ret_value = super(OpenStackShell, self).run_subcommand(argv) > File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in > run_subcommand > result = cmd.run(parsed_args) > File > "/home/foo/.local/lib/python2.7/site-packages/osc_lib/command/command.py", > line 41, in run > return super(Command, self).run(parsed_args) > File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, > in run > column_names, data = self.take_action(parsed_args) > File > "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/osc/v2/listener.py", > line 168, in take_action > json=body) > File > "/home/foo/.local/lib/python2.7/site-packages/octaviaclient/api/v2/octavia.py", > line 38, in wrapper > request_id=e.request_id) > OctaviaClientException: Could not retrieve certificate: > ['https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09', > 'https://barbican.service.dev.example.com/v1/secrets/593cc231-92ee-4b0a-8c58-0080052a6b35', > 'https://barbican.service.dev.example.com/v1/secrets/cb28220c-1339-4fc0-83f7-9cd155e3dc09'] > (HTTP 400) (Request-ID: req-5eef99bf-45c9-43eb-b7c7-2dacaff980ca) > ------------------------------ > > > This issue occurs only when application credentials are used. Creation > of HTTP listener with applications credentials works fine, also creation > of HTTPS listener when user are authenticated by user / password. > > Does somebody know which additional ACLs / permissions are required to > fix this? > > BR > > Pawel From ignaziocassano at gmail.com Fri Jul 19 15:36:19 2019 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Fri, 19 Jul 2019 17:36:19 +0200 Subject: [queens][cinder] emc fc driver In-Reply-To: References: Message-ID: Hello, I am going to test in my environment and I will send some reports. Thanks Ignazio Il Ven 19 Lug 2019 16:31 Jay Bryant ha scritto: > Ignazio, > > Dell/EMC has continued to be an active contributor to the Cinder project > and they have a working 3rd Party CI. So, the driver should work and if > you are seeing issues please provide details to the mailing list or come > join us in the #openstack-cinder channel on IRC and we can try to help you. > > Thanks! > Jay Bryant > > (irc: jungleboyj) > > On 7/19/2019 4:20 AM, Ignazio Cassano wrote: > > Hello, > > Does anyone use cinder with emc san via FC ? > > Does it work ? > > > > Regards > > Ignazio > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Prabhjit.Singh22 at T-Mobile.com Fri Jul 19 03:30:35 2019 From: Prabhjit.Singh22 at T-Mobile.com (Singh, Prabhjit) Date: Fri, 19 Jul 2019 03:30:35 +0000 Subject: [Octavia]-Seeking performance numbers on Octavia Message-ID: Hi I have been trying to test Octavia with some traffic generators and my tests are inconclusive. Appreciate your inputs on the following * It would be really nice to have some performance numbers that you guys have been able to achieve for this to be termed as carrier grade. * Would also appreciate if you could share any inputs on performance tuning Octavia * Any recommended flavor sizes for spinning up Amphorae, the default size of 1 core, 2 Gb disk and 1 Gig RAM does not seem enough. * Also I noticed when the Amphorae are spun up, at one time only one master is talking to the backend servers and has one IP that its using, it has to run out of ports after 64000 TCP concurrent sessions, id there a way to add more IPs or is this the limitation * If I needed some help with Octavia and some guidance around performance tuning can someone from the community help Thanks & Regards Prabhjit Singh -------------- next part -------------- An HTML attachment was scrubbed... URL: From elanchezian.settu at gigamon.com Fri Jul 19 07:00:00 2019 From: elanchezian.settu at gigamon.com (Elanchezian Settu) Date: Fri, 19 Jul 2019 07:00:00 +0000 Subject: [networking-ovs-dpdk In-Reply-To: References: Message-ID: Hi Erik, Thanks for your response. Getting below error message when I execute the command in compute node. Could this be the reason? ubuntu at os-compute-01:~$ nova-manage cell_v2 discover_hosts --verbose /usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: . """) An error has occurred: Traceback (most recent call last): File "/opt/stack/nova/nova/cmd/manage.py", line 2371, in main ret = fn(*fn_args, **fn_kwargs) File "/opt/stack/nova/nova/cmd/manage.py", line 1473, in discover_hosts by_service) File "/opt/stack/nova/nova/objects/host_mapping.py", line 248, in discover_hosts cell_mappings = objects.CellMappingList.get_all(ctxt) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper result = fn(cls, context, *args, **kwargs) File "/opt/stack/nova/nova/objects/cell_mapping.py", line 256, in get_all db_mappings = cls._get_all_from_db(context) File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1011, in wrapper with self._transaction_scope(context): File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1061, in _transaction_scope context=context) as resource: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session bind=self.connection, mode=self.mode) File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 418, in _create_session self._start() File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 510, in _start engine_args, maker_args) File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 532, in _setup_for_connection "No sql_connection parameter is established") CantStartEngineError: No sql_connection parameter is established Thanks & Regards, Elan From: Erik Olof Gunnar Andersson [mailto:eandersson at blizzard.com] Sent: Friday, July 19, 2019 1:41 AM To: Li, Cheng1; Elanchezian Settu; openstack-dev at lists.openstack.org Subject: RE: [networking-ovs-dpdk Maybe the compute isn't yet mapped to the cell. ? nova-manage cell_v2 discover_hosts --verbose Best Regards, Erik Olof Gunnar Andersson From: Li, Cheng1 > Sent: Wednesday, July 17, 2019 7:00 PM To: Elanchezian Settu >; openstack-dev at lists.openstack.org Subject: RE: [networking-ovs-dpdk Please check the nova-compute log on os-compute-01 to see if any useful info. It may locate at /var/log/nova/ Thanks, Cheng From: Elanchezian Settu [mailto:elanchezian.settu at gigamon.com] Sent: Wednesday, July 17, 2019 2:14 PM To: openstack-dev at lists.openstack.org Subject: [networking-ovs-dpdk Hi, I'm trying to setup a multi-node OpenStack environment with OVS-DPDK. Unfortunately I'm running into issue that compute node not getting listed in hypervisor list as below screenshot. Due to this I'm unable to create instances on compute node. Herewith I had attached the local.conf files of both controller and compute node. Can someone help me to resolve this issue. Thanks & Regards, Elan This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. NOTE that all incoming emails sent to Gigamon email accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional emails ("spam"). -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Fri Jul 19 16:04:31 2019 From: smooney at redhat.com (Sean Mooney) Date: Fri, 19 Jul 2019 17:04:31 +0100 Subject: How to restart DevStack In-Reply-To: References: Message-ID: <23cfc4b894a4611e66c754b35b4f7bed8e3d94d5.camel@redhat.com> On Fri, 2019-07-19 at 08:55 -0500, Dean Troyer wrote: > > On Fri, Jul 19, 2019 at 5:18 AM Lingxian Kong wrote: > > > As far as I know, it'd better run "./unstack.sh; ./clear.sh" before running "./stack.sh", because some devstack > > > plugins are not idempotent. > > unstack.sh and clean.sh are meant to be used to prepare for a new run > of stack.sh without a reboot. > > unstack.sh stops all services to prepare for stack.sh to bring them up > in a predictable order. > > clean.sh is more geared for major configuration changes like changing > databases or message queues. It runs unstack.sh then goes to some > effort to remove generated files and certain packages because we found > services did not always restart cleanly early on, and > changing databases needs package attention too. If you are not making > any configuration changes, both unstack and clean should be > unnecessary after a reboot. > > On Fri, Jul 19, 2019 at 6:29 AM Donny Davis wrote: > > I would also imagine it's better to run those before you do the shutdown or restart. > > If you are not making major config changes between runs this should > not be necessary. The idea of needing a clean shutdown is less > important when all data involved is going to be deleted and re-created > on the next stack.sh run anyway. for what its worth most service restart probly after reboot now that we use systemd. cinder i belive breaks on reboothing a devstack host but nova, keystone, horizon, glance, placment and neutron all work fine if i remember correctly it really would not take much to get all the service to work i have not looked at why cinder didint restart properly but is proably something trival like a service it depens on is not enabled to start by default like tgtd or somehting > > dt > From kristi at nikolla.me Fri Jul 19 16:43:10 2019 From: kristi at nikolla.me (Kristi Nikolla) Date: Fri, 19 Jul 2019 12:43:10 -0400 Subject: [keystone] Returning core Guang Yee In-Reply-To: <183bb224-92fb-486e-af2f-7baf0589905c@www.fastmail.com> References: <183bb224-92fb-486e-af2f-7baf0589905c@www.fastmail.com> Message-ID: Welcome back Guang Yee! Glad to have you again as part of the core team! On Fri, Jul 19, 2019 at 11:22 AM Colleen Murphy wrote: > Hi everyone, > > I'm thrilled to announce that Guang Yee has agreed to rejoin the keystone > core reviewer team. Guang is a former core who was instrumental in many of > keystone's key features. We're lucky to have him back to lend his > experience and knowledge to our current challenges. Welcome back, Guang! > > Colleen > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raubvogel at gmail.com Fri Jul 19 17:44:41 2019 From: raubvogel at gmail.com (Mauricio Tavares) Date: Fri, 19 Jul 2019 13:44:41 -0400 Subject: Instructions for spooling up an instance In-Reply-To: <20190717204526.zfs7e72xdcc2epy3@yuggoth.org> References: <6c253bdb-ce18-4ed9-ad98-e949cd108235@www.fastmail.com> <20190717204526.zfs7e72xdcc2epy3@yuggoth.org> Message-ID: Thanks for all the info. Replying inline On Wed, Jul 17, 2019 at 4:47 PM Jeremy Stanley wrote: > > On 2019-07-17 12:48:53 -0700 (-0700), Clark Boylan wrote: > [...] > > With your image build you can preinstall all of the software you > > need (and configure it too). You can also configure disk > > partitioning. The tool the OpenDev Infra team uses this for all of > > our test node images and some of our control plane images is > > called diskimage-builder [0]. This tool can partition images using > > both mbr and gpt tables. > [...] Thank you for the info on diskimage-builder. From what I understood in https://docs.openstack.org/diskimage-builder/latest/user_guide/building_an_image.html, its output is a partitoned disk image configured whatever way you want, which can then be fed to "openstack server create." After that it is a matter of installing the OS as you would in https://docs.openstack.org/image-guide/ubuntu-image.html. Correct me if I am wrong but that means customization done before OS is installed required separate images. Stuff like adding network cards can be customized because even adding the card itself can be done at any time. I think I now understand that. > > I gather there's another popular alternative, which is to boot a > server instance with an ISO of an operating system installer image > attached, use that to configure the supplied root disk and install > software into it, then shut that down and create a snapshot of the > resulting instance's rootfs to boot additional instances from. It's > a more traditional analog to how you'd do server installation in > classic non-cloudy environments, but some folks still seem to prefer > it over creating images of fully-working servers and then uploading > those. That said, I'd recommend sticking to using/modifying > officially provided "cloud" images from the distros you want since > they do some useful things like wiping system identifiers and host > keys so that they're regenerated independently on each new instance > booted from them (if you don't remember to do that on your custom > images you can run afoul of support licensing contracts or create > unnecessary security risks). > -- > Jeremy Stanley I think the merit of having a secure barebones image and then using the usual suspects -- ansible, chef -- to install and configure packages cuts down on how many images you have to keep current on the hard drive. They can force server keys if needed. Having customized images with some software installed and configured will lead to faster deployment, however. From morgan.fainberg at gmail.com Fri Jul 19 18:13:19 2019 From: morgan.fainberg at gmail.com (Morgan Fainberg) Date: Fri, 19 Jul 2019 11:13:19 -0700 Subject: [keystone] Returning core Guang Yee In-Reply-To: <183bb224-92fb-486e-af2f-7baf0589905c@www.fastmail.com> References: <183bb224-92fb-486e-af2f-7baf0589905c@www.fastmail.com> Message-ID: For multiple reasons I am glad to see Guang back as Keystone core: * Guang has been an amazing contributor over the years and it's good to have him as core (again) * It means Guang is contributing (has time to) to Keystone again. It is a very welcome sign to see him back on Keystone. Cheers, --Morgan On Fri, Jul 19, 2019 at 8:20 AM Colleen Murphy wrote: > Hi everyone, > > I'm thrilled to announce that Guang Yee has agreed to rejoin the keystone > core reviewer team. Guang is a former core who was instrumental in many of > keystone's key features. We're lucky to have him back to lend his > experience and knowledge to our current challenges. Welcome back, Guang! > > Colleen > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From corey.bryant at canonical.com Fri Jul 19 18:16:10 2019 From: corey.bryant at canonical.com (Corey Bryant) Date: Fri, 19 Jul 2019 14:16:10 -0400 Subject: [goal][python3] Train unit tests weekly update (goal-8) Message-ID: This is the goal-8 weekly update for the "Update Python 3 test runtimes for Train" goal [1]. There are 8 weeks remaining for completion of Train community goals [2]. == What's the Goal? == To ensure (in the Train cycle) that all official OpenStack repositories with Python 3 unit tests are exclusively using the 'openstack-python3-train-jobs' Zuul template or one of its variants (e.g. 'openstack-python3-train-jobs-neutron') to run unit tests, and that tests are passing. This will ensure that all official projects are running py36 and py37 unit tests in Train. For complete details please see [1]. == Ongoing Work == Patches have been submitted for all applicable projects except for one, OpenStack Charms. The charms have a mid-cycle freeze this week. I hope to have those submitted next week. Open patches needing reviews: https://review.openstack.org/#/q/topic:python3-train+is:open Failing patches: https://review.openstack.org/#/q/topic:python3-train+status:open+(+label:Verified-1+OR+label:Verified-2+) Patch automation scripts needing review: https://review.opendev.org/#/c/666934 == Completed Work == Merged patches: https://review.openstack.org/#/q/topic:python3-train+is:merged == How can you help? == Please take a look at the failing patches and help fix any failing unit tests for your projects. Python 3.7 unit tests will be self-testing in Zuul. == Reference Material == [1] Goal description: https://governance.openstack.org/tc/goals /train/python3-updates.html [2] Train release schedule: https://releases.openstack.org/train/schedule.html (see R-5 for "Train Community Goals Completed") Storyboard: https://storyboard.openstack.org/#!/story/2005924 Porting to Python 3.7: https://docs.python.org/3/whatsnew/3.7.html#porting-to-python-3-7 Python Update Process: https://opendev.org/openstack/governance/src/branch/master/resolutions/20181024-python-update-process.rst Train runtimes: https://opendev.org/openstack/governance/src/branch/master/reference/runtimes/train.rst Thanks, Corey -------------- next part -------------- An HTML attachment was scrubbed... URL: From raubvogel at gmail.com Fri Jul 19 18:20:58 2019 From: raubvogel at gmail.com (Mauricio Tavares) Date: Fri, 19 Jul 2019 14:20:58 -0400 Subject: vm guest creation Message-ID: If I am creating a kvm guest, I take it is doing something like virt-install \ --name desktop1 \ --disk path=./moocow.qcow2 \ --ram 4098 --vcpus 2 \ --os-type linux \ [....] --arch x86_64 Where does it do that? And, how much control per guest do I have? I am looking at https://docs.openstack.org/mitaka/config-reference/compute/hypervisor-kvm.html. What I see is that I can only define across the border options in nova.conf. Also, are there any docs on configuring the libvirt besides https://wiki.openstack.org/wiki/LibvirtXMLConfigAPIs? From rodrigodsousa at gmail.com Fri Jul 19 19:26:23 2019 From: rodrigodsousa at gmail.com (Rodrigo Duarte) Date: Fri, 19 Jul 2019 16:26:23 -0300 Subject: [keystone] Returning core Guang Yee In-Reply-To: References: <183bb224-92fb-486e-af2f-7baf0589905c@www.fastmail.com> Message-ID: Welcome back, gyee! On Fri, Jul 19, 2019 at 3:13 PM Morgan Fainberg wrote: > For multiple reasons I am glad to see Guang back as Keystone core: > > * Guang has been an amazing contributor over the years and it's good to > have him as core (again) > * It means Guang is contributing (has time to) to Keystone again. > > It is a very welcome sign to see him back on Keystone. > > Cheers, > --Morgan > > On Fri, Jul 19, 2019 at 8:20 AM Colleen Murphy > wrote: > >> Hi everyone, >> >> I'm thrilled to announce that Guang Yee has agreed to rejoin the keystone >> core reviewer team. Guang is a former core who was instrumental in many of >> keystone's key features. We're lucky to have him back to lend his >> experience and knowledge to our current challenges. Welcome back, Guang! >> >> Colleen >> >> -- Rodrigo http://rodrigods.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From berndbausch at gmail.com Fri Jul 19 20:32:16 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Sat, 20 Jul 2019 05:32:16 +0900 Subject: How to restart DevStack In-Reply-To: <23cfc4b894a4611e66c754b35b4f7bed8e3d94d5.camel@redhat.com> References: <23cfc4b894a4611e66c754b35b4f7bed8e3d94d5.camel@redhat.com> Message-ID: In my experience, after a reboot Devstack does not enable the loopback devices that implement Cinder's physical volumes and Swift's filesystem. The fix is running the losetup commands found in the stack.sh log. Bernd > On Jul 20, 2019, at 1:04, Sean Mooney wrote: > > it really would not take much to get all the service to work i have not looked at why cinder didint restart properly but > is proably something trival like a service it depens on is not enabled to start by default like tgtd or somehting From emilien at redhat.com Fri Jul 19 21:01:58 2019 From: emilien at redhat.com (Emilien Macchi) Date: Fri, 19 Jul 2019 17:01:58 -0400 Subject: [tripleo] specs for removing Puppet from TripleO Message-ID: As part of the Simplification of TripleO effort [1], I pushed a specification proposal to remove Puppet from TripleO. This is a *huge* effort and to be clear I'm still unsure it is really doable; but I would like our team to explore this path and see if there is enough value out of it. https://review.opendev.org/#/c/671563 HTML output with latest patchset: http://logs.openstack.org/63/671563/4/check/openstack-tox-docs/c511ace/html/specs/train/remove-puppet.html Feel free to give feedback directly in the specs or here in this thread. If we agree to move forward, my hope is we can push for a first "tech-preview" interface in Train; but keep in mind this would be a multi-cycles effort. [1] https://docs.google.com/document/d/1vY9rsccgp7NHFXpLtCFTHQm14F15Tij7lhn5X_P14Ys -- Emilien Macchi -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnsomor at gmail.com Fri Jul 19 21:59:38 2019 From: johnsomor at gmail.com (Michael Johnson) Date: Fri, 19 Jul 2019 14:59:38 -0700 Subject: [Octavia]-Seeking performance numbers on Octavia In-Reply-To: References: Message-ID: Hi Prabhjit, As you have mentioned, it is very challenging to get accurate performance results in cloud environments. There are a large number(very large in fact) of factors that can impact the overall performance of OpenStack and Octavia. In our OpenDev testing environment, we only have software emulation virtual machines available (Qemu running with the TCG engine) which performs extremely poorly. This means that the testing environment does not reflect how the software is used in real world deployments. An example of this is simply booting a VM can take up to ten minutes on Qemu with TCG when it takes about twenty seconds on a real OpenStack deployment. With this resource limitation, we cannot effectively run performance benchmarking test jobs on the OpenDev environment. Because of this, we don't publish performance numbers as they will not reflect what you can achieve in your environment. Let me try to speak to your bullet points: 1. The Octavia team has never (to my knowledge) claimed the Amphora driver is "carrier grade". We do consider the Amphora driver to be "operator grade", which speaks to a cloud operator's perspective versus the previous offering that did not support high availability, have appropriate maintenance tooling, upgrade paths, performance, etc. To me, "carrier grade" has an additional level of requirements including performance, latency, scale, and availability SLAs. This is not what the Octavia Amphora driver is currently ready for. That said, third party provider drivers for Octavia may be able to provide a "carrier grade" level of load balancing for OpenStack. 2. As for performance tuning, much of this is either automatically handled by Octavia or are dependent on the application you are load balancing and your cloud deployment. For example we have many configuration settings to tune how many retries we attempt when interacting with other services. In performing and stable clouds, these can be tuned down, in others the defaults may be appropriate. If you would like faster failover, at the expense of slightly more network traffic, you can tune the health monitoring and keepalived_vrrp settings. We do not currently have a performance tuning guide for Octavia but would support someone authoring one. 3. We do not currently have a guide for this. I will say with the version of HAproxy currently being shipped with the distributions, going beyond the 1vCPU per amphora does not gain you much. With the release of HAProxy 2.0 this has changed and we expect to be adding support for vertically scaling the Amphora in future releases. Disk space is only necessary if you are storing the flow logs locally, which I would not recommend for a performance load balancer (See the notes in the log offloading guide: https://docs.openstack.org/octavia/latest/admin/log-offloading.html). Finally, the RAM usage is a factor of the number of concurrent connections and if you are enabling TLS on the load balancer. For typical load balancing loads, the default is typically fine. However, if you have high connection counts and/or TLS offloading, you may want to experiment with increasing the available RAM. 4. The source IP issue is a known issue (https://storyboard.openstack.org/#!/story/1629066). We have not prioritized addressing this as we have not had anyone come forward that they needed this in their deployment. If this is an issue impacting your use case, please comment on the story to that effect and provide a use case. This will help the team prioritize this work. Also, patches are welcome! If you are interested in working on this issue, I can help you with information about how this could be added. It should also be noted that it is a limitation of 64,000 connections per-backend server, not per load balancer. 5. The team uses the #openstack-lbaas IRC channel on freenode and is happy to answer questions, etc. To date, we have had limited resources (people and equipment) available to do performance evaluation and tuning. There are definitely kernel and HAProxy tuning settings we have evaluated and added to the Amphora driver, but I know there is more work that can be done. If you are interested in help us with this work, please let us know. Michael P.S. Here are just a few considerations that can/will impact the performance of an Octavia Amphora load balancer: Hardware used for the compute nodes Network Interface Cards (NICs) used in the compute nodes Number of network ports enabled on the compute hosts Network switch configurations (Jumbo frames, and so on) Cloud network topology (leaf‐spine, fat‐tree, and so on) The OpenStack Neutron networking configuration (ML2 and ML3 drivers) Tenant networking configuration (VXLAN, VLANS, GRE, and so on) Colocation of applications and Octavia amphorae Over subscription of the compute and networking resources Protocols being load balanced Configuration settings used when creating the load balancer (connection limits, and so on) Version of OpenStack services (nova, neutron, and so on) Version of OpenStack Octavia Flavor of the OpenStack Octavia load balancer OS and hypervisor versions used Deployed security mitigations (Spectre, Meltdown, and so on) Customer application performance Health of the customer application On Fri, Jul 19, 2019 at 8:52 AM Singh, Prabhjit wrote: > > Hi > > > > I have been trying to test Octavia with some traffic generators and my tests are inconclusive. Appreciate your inputs on the following > > > > It would be really nice to have some performance numbers that you guys have been able to achieve for this to be termed as carrier grade. > Would also appreciate if you could share any inputs on performance tuning Octavia > Any recommended flavor sizes for spinning up Amphorae, the default size of 1 core, 2 Gb disk and 1 Gig RAM does not seem enough. > Also I noticed when the Amphorae are spun up, at one time only one master is talking to the backend servers and has one IP that its using, it has to run out of ports after 64000 TCP concurrent sessions, id there a way to add more IPs or is this the limitation > If I needed some help with Octavia and some guidance around performance tuning can someone from the community help > > > > Thanks & Regards > > > > Prabhjit Singh > > > > > > From fungi at yuggoth.org Fri Jul 19 22:29:55 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 19 Jul 2019 22:29:55 +0000 Subject: [Octavia]-Seeking performance numbers on Octavia In-Reply-To: References: Message-ID: <20190719222955.5p2ybojwc5wlve7p@yuggoth.org> On 2019-07-19 14:59:38 -0700 (-0700), Michael Johnson wrote: [...] > In our OpenDev testing environment, we only have software > emulation virtual machines available (Qemu running with the TCG > engine) which performs extremely poorly. This means that the > testing environment does not reflect how the software is used in > real world deployments. An example of this is simply booting a VM > can take up to ten minutes on Qemu with TCG when it takes about > twenty seconds on a real OpenStack deployment. > > With this resource limitation, we cannot effectively run > performance benchmarking test jobs on the OpenDev environment. [...] And even if we did have ubiquitous support for nested virtual machine acceleration across our providers, it still wouldn't provide a useful baseline because at least: 1. The hardware and software/hypervisors in the different donated environments (and even within some of them) vary significantly. 2. Most of these environments are in public service providers with mixed populations, and so many runs will be scheduled onto hosts with "noisy neighbor" situations leading to anomalous resource contention. 3. At peak load times, our job nodes may act as noisy neighbors to each other (especially in environments where we have dedicated host aggregates), leading to slower performance. We've optimized for test throughput, to make maximal use of the donations provided to us. Without carving out and dedicating environments with predictable performance characteristics, benchmarking is really a non-starter. However, it's also not something which needs to be run continuously, so can be done in an ad hoc fashion by interested individuals and the results published independently for comparison (as is often the case for other similar sorts of projects). -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From colleen at gazlene.net Fri Jul 19 23:43:29 2019 From: colleen at gazlene.net (Colleen Murphy) Date: Fri, 19 Jul 2019 16:43:29 -0700 Subject: [keystone] Keystone Team Update - Week of 15 July 2019 Message-ID: <4fbb7a20-9247-4ccc-adee-9508ad721354@www.fastmail.com> # Keystone Team Update - Week of 15 July 2019 ## News ### Returning core Welcome back, Guang![1] [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007918.html ### Midcycle next week The virtual midcycle will take place next week on Monday and Tuesday[2]. We will (try to) use Gotomeeting for the call, the details for which are in the etherpad. [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007593.html ### Train specs merged All specs proposed for the Train cycle are now merged! ## Open Specs Train specs: https://bit.ly/2uZ2tRl Ongoing specs: https://bit.ly/2OyDLTh ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 8 changes this week. ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 40 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ## Bugs This week we opened 5 new bugs and closed 4. Bugs opened (5) Bug #1837061 (keystone:Wishlist) opened by Morgan Fainberg https://bugs.launchpad.net/keystone/+bug/1837061 Bug #1836568 (keystone:Undecided) opened by Attila Fazekas https://bugs.launchpad.net/keystone/+bug/1836568 Bug #1836650 (keystone:Undecided) opened by Rafael Weingartner https://bugs.launchpad.net/keystone/+bug/1836650 Bug #1836872 (keystone:Undecided) opened by Yang Youseok https://bugs.launchpad.net/keystone/+bug/1836872 Bug #1837010 (keystone:Undecided) opened by Yang Youseok https://bugs.launchpad.net/keystone/+bug/1837010 Bugs closed (3) Bug #1832267 (keystone:Undecided) https://bugs.launchpad.net/keystone/+bug/1832267 Bug #1836650 (keystone:Undecided) https://bugs.launchpad.net/keystone/+bug/1836650 Bug #1836872 (keystone:Undecided) https://bugs.launchpad.net/keystone/+bug/1836872 Bugs fixed (1) Bug #1813057 (keystone:Medium) fixed by Guang Yee https://bugs.launchpad.net/keystone/+bug/1813057 ## Milestone Outlook https://releases.openstack.org/train/schedule.html Next week is spec freeze, which we've already met. One month from now is feature proposal freeze, by which point all code for new features should be proposed and ready for review - no WIPs. This gives us four weeks to review and incorporate feedback before feature freeze. ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter From doka.ua at gmx.com Sat Jul 20 10:42:35 2019 From: doka.ua at gmx.com (Volodymyr Litovka) Date: Sat, 20 Jul 2019 13:42:35 +0300 Subject: local storage + remote replication Message-ID: Dear colleagues, this question isn't about Openstack, it's rather general question about cloud data storing. It is obvious, that distributed network storage is generally slower than local storage, while is much more reliable. Keeping in mind, that reliability of hardware components became better and better with time (e.g. MTBF of HP DL380G2 is almost 14 years), power systems are reliable enough (if built with reliability in mind), it is safe to say that the most possible source of problems is failure of storage itself, not motherboard or NIC or power supply. Why not to use kind of caching, quickly saving data on local storage, ACKing the operation and replicating changed blocks to remote storage in background during idle time? Having a local RAID1 (with checksums to allow distinguish between failed and correct copy of the data block) + remote replication will give a high speed and enough level of reliability. The only case of temporary slowing down is migration to another node (not pretty often event?) with gradual back replication (from remote storage to local, as the VM will request data) which will last some time from start at new node. Are there serious concerns against such approach? I will appreciate your opinion on this topic. Thank you! -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison From tenobreg at redhat.com Sat Jul 20 15:17:23 2019 From: tenobreg at redhat.com (Telles Nobrega) Date: Sat, 20 Jul 2019 12:17:23 -0300 Subject: Unable to ssh openstack instance In-Reply-To: References: Message-ID: Hi, you probably checked this already but it is always good to double check that you opened the ssh rules on the security group. On Fri, 19 Jul 2019 at 12:13 Sofia Enriquez wrote: > Hi Manula, > > This guide really helps me when testing things with Devstack[1]. > Hope it can help you too. > > Sofia > > [1] > https://docs.openstack.org/networking-ovn/latest/contributor/testing.html#booting-vms > > On Thu, Jul 18, 2019 at 5:54 PM Slawek Kaplonski > wrote: > >> Hi, >> >> It seems that 172.24.4.147 is “fixed IP” of Your instance. What network >> type are You using there? Usually tenant network is some tunnel network >> type, like VxLAN or GRE. In such case this network is isolated and You will >> not have access to it directly from host. >> You can try to do SSH to such IP address from DHCP or router namespace. >> DHCP namespace for Your network will be created on node where >> neutron-dhcp-agent is running and its name will be something like >> “qdhcp-”. >> For router it’s similar, namespace will be where neutron-l3-agent which >> is hosting Your router is running and name of namespace will be like >> “qrouter-” >> >> If You connected FIP to the instance, and You have some provider network >> used as external net (vlan or flat) than such IPs should be probably >> reachable from Your host - but this depends on Your network configuration >> on Your host(s). >> >> > On 18 Jul 2019, at 09:25, Manula Thantriwatte < >> manulachathurika at gmail.com> wrote: >> > >> > Hi All, >> > >> > I'm trying to ssh openstack image I have created using cirros image. >> But I'm getting following error. >> > >> > ssh: connect to host 172.24.4.147 port 22: No route to host >> > >> > But when I go to the instance console I'm able to login. >> > >> > For the security groups I have enable port 22. >> > >> > I have associated floating IP as well. >> > >> > How to resolve this? >> > >> > Thanks ! >> > >> > -- >> > Regards, >> > Manula Chathurika Thantriwatte >> > phone : (+94) 772492511 >> > email : manulachathurika at gmail.com >> > Linkedin : http://lk.linkedin.com/in/manulachathurika >> > blog : http://manulachathurika.blogspot.com/ >> > > 12-54-51.png> >> >> — >> Slawek Kaplonski >> Senior software engineer >> Red Hat >> >> >> > > -- > > L. Sofía Enriquez > > she/her > > Associate Software Engineer > > Red Hat PnT > > IRC: @enriquetaso > @RedHat Red Hat > Red Hat > > > > -- Telles Nobrega Senior Software Engineer Red Hat tenobreg at redhat.com @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbragstad at gmail.com Sat Jul 20 16:05:34 2019 From: lbragstad at gmail.com (Lance Bragstad) Date: Sat, 20 Jul 2019 11:05:34 -0500 Subject: [keystone] Returning core Guang Yee In-Reply-To: <183bb224-92fb-486e-af2f-7baf0589905c@www.fastmail.com> References: <183bb224-92fb-486e-af2f-7baf0589905c@www.fastmail.com> Message-ID: <6d326818-c854-d58f-8368-a8bb56370294@gmail.com> Welcome back, Guang! On 7/19/19 10:16 AM, Colleen Murphy wrote: > Hi everyone, > > I'm thrilled to announce that Guang Yee has agreed to rejoin the keystone core reviewer team. Guang is a former core who was instrumental in many of keystone's key features. We're lucky to have him back to lend his experience and knowledge to our current challenges. Welcome back, Guang! > > Colleen > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From mark at stackhpc.com Sat Jul 20 17:41:11 2019 From: mark at stackhpc.com (Mark Goddard) Date: Sat, 20 Jul 2019 18:41:11 +0100 Subject: [kolla][ironic] Paternity leave Message-ID: Hi, I will be on paternity leave for the next few weeks. Please speak to another core in #openstack-kolla for Kolla or Kayobe issues. I'll check in from time to time. Cheers, Mark PS boy, Frank, very cute. -------------- next part -------------- An HTML attachment was scrubbed... URL: From moreira.belmiro.email.lists at gmail.com Sat Jul 20 18:43:02 2019 From: moreira.belmiro.email.lists at gmail.com (Belmiro Moreira) Date: Sat, 20 Jul 2019 20:43:02 +0200 Subject: [kolla][ironic] Paternity leave In-Reply-To: References: Message-ID: Congratulations Mark. On Sat, 20 Jul 2019 at 19:47, Mark Goddard wrote: > Hi, > > I will be on paternity leave for the next few weeks. Please speak to > another core in #openstack-kolla for Kolla or Kayobe issues. I'll check in > from time to time. > > Cheers, > Mark > > PS boy, Frank, very cute. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristi at nikolla.me Sat Jul 20 19:38:20 2019 From: kristi at nikolla.me (Kristi Nikolla) Date: Sat, 20 Jul 2019 15:38:20 -0400 Subject: [keystone] Shanghai PTG planning In-Reply-To: <60277b1f-9453-4abd-a3c4-a9117885e56f@www.fastmail.com> References: <60277b1f-9453-4abd-a3c4-a9117885e56f@www.fastmail.com> Message-ID: That attendance list is looking pretty sad so far :( On Wed, Jul 10, 2019 at 6:23 PM Colleen Murphy wrote: > Hi team, > > The foundation has asked us to let them know whether we'll be needing PTG > space in Shanghai. The keystone team usually has good attendance at PTGs > and makes good use of the time, but I know this next one may be hard for > some people to attend so I don't want to automatically assume we'll use it > this time. If you wish to attend the next PTG and have a semi-reasonable > amount of confidence that you may be able to, please add your name to the > etherpad: > > https://etherpad.openstack.org/p/keystone-shanghai-ptg > > Please do this by Monday, August 5 so that I have time to respond to the > foundation by August 11. > > Feel free to also start brainstorming topics for the PTG (no deadline on > this). > > Colleen > > -- Kristi -------------- next part -------------- An HTML attachment was scrubbed... URL: From Arkady.Kanevsky at dell.com Sun Jul 21 00:57:20 2019 From: Arkady.Kanevsky at dell.com (Arkady.Kanevsky at dell.com) Date: Sun, 21 Jul 2019 00:57:20 +0000 Subject: [kolla][ironic] Paternity leave In-Reply-To: References: Message-ID: <6d36350b4b5f462f8c63a0b94a2db497@AUSX13MPS308.AMER.DELL.COM> Congrats Mark. From: Mark Goddard Sent: Saturday, July 20, 2019 12:41 PM To: openstack-discuss Subject: [kolla][ironic] Paternity leave [EXTERNAL EMAIL] Hi, I will be on paternity leave for the next few weeks. Please speak to another core in #openstack-kolla for Kolla or Kayobe issues. I'll check in from time to time. Cheers, Mark PS boy, Frank, very cute. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jyotishri403 at gmail.com Sun Jul 21 06:53:41 2019 From: jyotishri403 at gmail.com (Jyoti Dahiwele) Date: Sun, 21 Jul 2019 12:23:41 +0530 Subject: Cinder data path and control path Message-ID: Dear Team, Plz clear doubt about cinder control path and data path. If control path goes off and data path continue then still vm continues to use attached volumes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wangxiyuan1007 at gmail.com Mon Jul 22 02:48:28 2019 From: wangxiyuan1007 at gmail.com (Xiyuan Wang) Date: Mon, 22 Jul 2019 10:48:28 +0800 Subject: [keystone] Returning core Guang Yee In-Reply-To: <6d326818-c854-d58f-8368-a8bb56370294@gmail.com> References: <183bb224-92fb-486e-af2f-7baf0589905c@www.fastmail.com> <6d326818-c854-d58f-8368-a8bb56370294@gmail.com> Message-ID: Big welcome back. Guang! Lance Bragstad 于2019年7月21日周日 上午12:07写道： > Welcome back, Guang! > > On 7/19/19 10:16 AM, Colleen Murphy wrote: > > Hi everyone, > > I'm thrilled to announce that Guang Yee has agreed to rejoin the keystone core reviewer team. Guang is a former core who was instrumental in many of keystone's key features. We're lucky to have him back to lend his experience and knowledge to our current challenges. Welcome back, Guang! > > Colleen > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean-philippe at evrard.me Mon Jul 22 06:37:52 2019 From: jean-philippe at evrard.me (Jean-Philippe Evrard) Date: Mon, 22 Jul 2019 08:37:52 +0200 Subject: [tripleo][openstack-ansible] Integrating ansible-role-collect-logs in OSA In-Reply-To: References: <71d568ee47ce516a5a4cab1422290da2be1baff6.camel@evrard.me> Message-ID: Sorry for the late answer... On Wed, 2019-07-10 at 12:12 -0600, Wesley Hayutin wrote: > > These are of course just passed in as extra-config. I think each > project would want to define their own list of files and maintain it > in their own project. WDYT? Looks good. We can either clean up the defaults, or OSA can just override the defaults, and it would be good enough. I would say that this can still be improved later, after OSA has started using the role too. > It simple enough. But I am happy to see a different approach. Simple is good! > Any thoughts on additional work that I am not seeing? None :) > > Thanks for responding! I know our team is very excited about the > continued collaboration with other upstream projects, so thanks!! > Likewise. Let's reduce tech debt/maintain more code together! Regards, Jean-Philippe Evrard (evrardjp) From geguileo at redhat.com Mon Jul 22 09:29:22 2019 From: geguileo at redhat.com (Gorka Eguileor) Date: Mon, 22 Jul 2019 11:29:22 +0200 Subject: Cinder data path and control path In-Reply-To: References: Message-ID: <20190722092922.x66unicbxns7bcxx@localhost> On 21/07, Jyoti Dahiwele wrote: > Dear Team, > > Plz clear doubt about cinder control path and data path. If control path > goes off and data path continue then still vm continues to use attached > volumes. That is correct, VM's data path does not go through Cinder itself, so as long as control path and data path/s are different the control path going down will not affect the data path. From jean-philippe at evrard.me Mon Jul 22 12:07:35 2019 From: jean-philippe at evrard.me (Jean-Philippe Evrard) Date: Mon, 22 Jul 2019 14:07:35 +0200 Subject: [openstack-ansible][tripleo] creating collaborative acl In-Reply-To: References: Message-ID: On Thu, 2019-07-18 at 10:09 -0400, Mohammed Naser wrote: > (snipped) Therefore, I'd > like to propose that we create a Gerrit group called > "openstack-ansible-collab". Not sure we need to create yet another group (if we are using groups it's easy to just use existing groups), but it's fine for me if you prefer. > TripleO has been around for a while and they > have a solid core team that I (personally) feel okay with having core > to those repos for now (and more as we work on them together). Fine for me. In fact, it's not just "fine". It's great to have more cores! :) > In terms of governance, these projects will remain under > OpenStack-Ansible governance. Please let me know what you think of > this proposal? > OK. Regards, Jean-Philippe (evrardjp) From Prabhjit.Singh22 at T-Mobile.com Mon Jul 22 00:24:14 2019 From: Prabhjit.Singh22 at T-Mobile.com (Singh, Prabhjit) Date: Mon, 22 Jul 2019 00:24:14 +0000 Subject: [Octavia]-Seeking performance numbers on Octavia In-Reply-To: References: Message-ID: Hi Michael, Thanks for taking the time out to send me your inputs and valuable suggestions. I do remember meeting you at the Denver Summit and hearing to a couple of your sessions. If you wouldn't mind, I do have a few more questions and your answers would help me understand that should I continue to invest in having Octavia as one of our available LBs. 1. Based on your response and the amount of time you are investing in supporting Octavia, what are some of the use cases, like for e.g. if load balancing web traffic how many transactions/connections minimum can be expected. I do understand you mentioned that it's hard to performance test Octavia but some real time situations from your testing and how customers have adopted Octavia would help me level set some expectations. 2. We are thinking of Octavia as one of the offerings, that offers a self-serve type model. Do you know of any customers who have been able to use Octavia as one of their primary load balancers and any encouraging feedback you have gotten on Octavia. 3. You suggested increasing the Ram size, I could go about making a whole new Flavor. 4. I also noticed on the haproxy.conf the maxconns is set to 2000, should I increase this, does this affect the connection per server, which you said 64000 conns per server, so if I have 10 servers can I expect somewhere close to 640000 sessions? 5. Based on some of the limitations and the dev work in progress, I think the most important feature that would make Octavia a real solid offering would be the Active-Active and Autoscaling feature. I brought this up with you in our brief conversation at the summit, and you did mention that its not a top priority at this time and you are looking for some help. I have noticed a lot of documentation has been updated on this feature, do you think with the available document and progress I could spin up a distributor and manage sessions between Amphora or it's not complete yet. 6. We have a Triple O setup, do you think I can make the above tweaks with the Triple O setup. Thanks & Regards Prabhjit Singh Systems Design and Strategy - Magentabox | O: (973) 397-4819 | M: (973) 563-4445 -----Original Message----- From: Michael Johnson Sent: Friday, July 19, 2019 6:00 PM To: Singh, Prabhjit Cc: openstack-discuss at lists.openstack.org Subject: Re: [Octavia]-Seeking performance numbers on Octavia [External] Hi Prabhjit, As you have mentioned, it is very challenging to get accurate performance results in cloud environments. There are a large number(very large in fact) of factors that can impact the overall performance of OpenStack and Octavia. In our OpenDev testing environment, we only have software emulation virtual machines available (Qemu running with the TCG engine) which performs extremely poorly. This means that the testing environment does not reflect how the software is used in real world deployments. An example of this is simply booting a VM can take up to ten minutes on Qemu with TCG when it takes about twenty seconds on a real OpenStack deployment. With this resource limitation, we cannot effectively run performance benchmarking test jobs on the OpenDev environment. Because of this, we don't publish performance numbers as they will not reflect what you can achieve in your environment. Let me try to speak to your bullet points: 1. The Octavia team has never (to my knowledge) claimed the Amphora driver is "carrier grade". We do consider the Amphora driver to be "operator grade", which speaks to a cloud operator's perspective versus the previous offering that did not support high availability, have appropriate maintenance tooling, upgrade paths, performance, etc. To me, "carrier grade" has an additional level of requirements including performance, latency, scale, and availability SLAs. This is not what the Octavia Amphora driver is currently ready for. That said, third party provider drivers for Octavia may be able to provide a "carrier grade" level of load balancing for OpenStack. 2. As for performance tuning, much of this is either automatically handled by Octavia or are dependent on the application you are load balancing and your cloud deployment. For example we have many configuration settings to tune how many retries we attempt when interacting with other services. In performing and stable clouds, these can be tuned down, in others the defaults may be appropriate. If you would like faster failover, at the expense of slightly more network traffic, you can tune the health monitoring and keepalived_vrrp settings. We do not currently have a performance tuning guide for Octavia but would support someone authoring one. 3. We do not currently have a guide for this. I will say with the version of HAproxy currently being shipped with the distributions, going beyond the 1vCPU per amphora does not gain you much. With the release of HAProxy 2.0 this has changed and we expect to be adding support for vertically scaling the Amphora in future releases. Disk space is only necessary if you are storing the flow logs locally, which I would not recommend for a performance load balancer (See the notes in the log offloading guide: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.openstack.org%2Foctavia%2Flatest%2Fadmin%2Flog-offloading.html&data=02%7C01%7CPrabhjit.Singh22%40t-mobile.com%7C8c8032b0852446142af508d70c946ba5%7Cbe0f980bdd994b19bd7bbc71a09b026c%7C0%7C0%7C636991703917513373&sdata=rUZSLn1dWt6iHqldGsqDv6DIdUysx3H0ogJYXyQRw7o%3D&reserved=0). Finally, the RAM usage is a factor of the number of concurrent connections and if you are enabling TLS on the load balancer. For typical load balancing loads, the default is typically fine. However, if you have high connection counts and/or TLS offloading, you may want to experiment with increasing the available RAM. 4. The source IP issue is a known issue (https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstoryboard.openstack.org%2F%23!%2Fstory%2F1629066&data=02%7C01%7CPrabhjit.Singh22%40t-mobile.com%7C8c8032b0852446142af508d70c946ba5%7Cbe0f980bdd994b19bd7bbc71a09b026c%7C0%7C0%7C636991703917523364&sdata=cIoDC1Ncd5uCy2as8NErGrJ0EO0IO64dZEob0XkLmyc%3D&reserved=0). We have not prioritized addressing this as we have not had anyone come forward that they needed this in their deployment. If this is an issue impacting your use case, please comment on the story to that effect and provide a use case. This will help the team prioritize this work. Also, patches are welcome! If you are interested in working on this issue, I can help you with information about how this could be added. It should also be noted that it is a limitation of 64,000 connections per-backend server, not per load balancer. 5. The team uses the #openstack-lbaas IRC channel on freenode and is happy to answer questions, etc. To date, we have had limited resources (people and equipment) available to do performance evaluation and tuning. There are definitely kernel and HAProxy tuning settings we have evaluated and added to the Amphora driver, but I know there is more work that can be done. If you are interested in help us with this work, please let us know. Michael P.S. Here are just a few considerations that can/will impact the performance of an Octavia Amphora load balancer: Hardware used for the compute nodes Network Interface Cards (NICs) used in the compute nodes Number of network ports enabled on the compute hosts Network switch configurations (Jumbo frames, and so on) Cloud network topology (leaf‐spine, fat‐tree, and so on) The OpenStack Neutron networking configuration (ML2 and ML3 drivers) Tenant networking configuration (VXLAN, VLANS, GRE, and so on) Colocation of applications and Octavia amphorae Over subscription of the compute and networking resources Protocols being load balanced Configuration settings used when creating the load balancer (connection limits, and so on) Version of OpenStack services (nova, neutron, and so on) Version of OpenStack Octavia Flavor of the OpenStack Octavia load balancer OS and hypervisor versions used Deployed security mitigations (Spectre, Meltdown, and so on) Customer application performance Health of the customer application On Fri, Jul 19, 2019 at 8:52 AM Singh, Prabhjit wrote: > > Hi > > > > I have been trying to test Octavia with some traffic generators and my > tests are inconclusive. Appreciate your inputs on the following > > > > It would be really nice to have some performance numbers that you guys have been able to achieve for this to be termed as carrier grade. > Would also appreciate if you could share any inputs on performance > tuning Octavia Any recommended flavor sizes for spinning up Amphorae, the default size of 1 core, 2 Gb disk and 1 Gig RAM does not seem enough. > Also I noticed when the Amphorae are spun up, at one time only one > master is talking to the backend servers and has one IP that its > using, it has to run out of ports after 64000 TCP concurrent sessions, > id there a way to add more IPs or is this the limitation If I needed > some help with Octavia and some guidance around performance tuning can > someone from the community help > > > > Thanks & Regards > > > > Prabhjit Singh > > > > > > From vungoctan252 at gmail.com Mon Jul 22 03:33:49 2019 From: vungoctan252 at gmail.com (Vu Tan) Date: Mon, 22 Jul 2019 10:33:49 +0700 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: <35400f83-c29d-475a-8d36-d56b3cf16d30@email.android.com> References: <35400f83-c29d-475a-8d36-d56b3cf16d30@email.android.com> Message-ID: Hi Patil, May I know when the proper document for masakari is released ? I have configured conf file in controller and compute node, it seems running but it is not running as it should be, a lots of error in logs, here is a sample log: 2.7/site-packages/oslo_config/cfg.py:3024 2019-07-19 10:25:26.360 7745 DEBUG oslo_service.service [-] bindir = /usr/local/bin log_opt_values /usr/lib/ python2.7/site-packages/oslo_config/cfg.py:3024 2019-07-19 18:46:21.291 7770 ERROR masakari File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 65, in _is_daemo n 2019-07-19 18:46:21.291 7770 ERROR masakari is_daemon = os.getpgrp() != os.tcgetpgrp(sys.stdout.fileno()) 2019-07-19 18:46:21.291 7770 ERROR masakari OSError: [Errno 5] Input/output error 2019-07-19 18:46:21.291 7770 ERROR masakari 2019-07-19 18:46:21.300 7745 CRITICAL masakari [-] Unhandled error: OSError: [Errno 5] Input/output error 2019-07-19 18:46:21.300 7745 ERROR masakari Traceback (most recent call last): 2019-07-19 18:46:21.300 7745 ERROR masakari File "/usr/bin/masakari-api", line 10, in I dont know if it is missing package or wrong configuration On Thu, Jul 11, 2019 at 6:14 PM Gaëtan Trellu wrote: > You will have to enable the debit, debug = true and check the APi log. > > Did you try to use the openstack CLi ? > > Gaetan > > On Jul 11, 2019 12:32 AM, Vu Tan wrote: > > I know it's just a warning, just take a look at this image: > [image: image.png] > it's just hang there forever, and in the log show what I have shown to you > > On Wed, Jul 10, 2019 at 8:07 PM Gaëtan Trellu > wrote: > > This is just a warning, not an error. > > On Jul 10, 2019 3:12 AM, Vu Tan wrote: > > Hi Gaetan, > I follow you the guide you gave me, but the problem still persist, can you > please take a look at my configuration to see what is wrong or what is > missing in my config ? > the error: > 2019-07-10 14:08:46.876 17292 WARNING keystonemiddleware._common.config > [-] The option "__file__" in conf is not known to auth_token > 2019-07-10 14:08:46.876 17292 WARNING keystonemiddleware._common.config > [-] The option "here" in conf is not known to auth_token > 2019-07-10 14:08:46.882 17292 WARNING keystonemiddleware.auth_token [-] > AuthToken middleware is set with keystone_authtoken.service_ > > the config: > > [DEFAULT] > enabled_apis = masakari_api > log_dir = /var/log/kolla/masakari > state_path = /var/lib/masakari > os_user_domain_name = default > os_project_domain_name = default > os_privileged_user_tenant = service > os_privileged_user_auth_url = http://controller:5000/v3 > os_privileged_user_name = nova > os_privileged_user_password = P at ssword > masakari_api_listen = controller > masakari_api_listen_port = 15868 > debug = False > auth_strategy=keystone > > [wsgi] > # The paste configuration file path > api_paste_config = /etc/masakari/api-paste.ini > > [keystone_authtoken] > www_authenticate_uri = http://controller:5000 > auth_url = http://controller:5000 > auth_type = password > project_domain_id = default > project_domain_name = default > user_domain_name = default > user_domain_id = default > project_name = service > username = masakari > password = P at ssword > region_name = RegionOne > > [oslo_middleware] > enable_proxy_headers_parsing = True > > [database] > connection = mysql+pymysql://masakari:P at ssword@controller/masakari > > > > On Tue, Jul 9, 2019 at 10:25 PM Vu Tan wrote: > > Thank Patil Tushar, I hope it will be available soon > > On Tue, Jul 9, 2019 at 8:18 AM Patil, Tushar > wrote: > > Hi Vu and Gaetan, > > Gaetan, thank you for helping out Vu in setting up masakari-monitors > service. > > As a masakari team ,we have noticed there is a need to add proper > documentation to help the community run Masakari services in their > environment. We are working on adding proper documentation in this 'Train' > cycle. > > Will send an email on this mailing list once the patches are uploaded on > the gerrit so that you can give your feedback on the same. > > If you have any trouble in setting up Masakari, please let us know on this > mailing list or join the bi-weekly IRC Masakari meeting on the > #openstack-meeting IRC channel. The next meeting will be held on 16th July > 2019 @0400 UTC. > > Regards, > Tushar Patil > > ________________________________________ > From: Vu Tan > Sent: Monday, July 8, 2019 11:21:16 PM > To: Gaëtan Trellu > Cc: openstack-discuss at lists.openstack.org > Subject: Re: [masakari] how to install masakari on centos 7 > > Hi Gaetan, > Thanks for pinpoint this out, silly me that did not notice the simple > "error InterpreterNotFound: python3". Thanks a lot, I appreciate it > > On Mon, Jul 8, 2019 at 9:15 PM gaetan.trellu at incloudus.com>> wrote: > Vu Tan, > > About "auth_token" error, you need "os_privileged_user_*" options into > your masakari.conf for the API. > As mentioned previously please have a look here to have an example of > configuration working (for me at least): > > - masakari.conf: > > https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari.conf.j2 > - masakari-monitor.conf: > > https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari-monitors.conf.j2 > > About your tox issue make sure you have Python3 installed. > > Gaëtan > > On 2019-07-08 06:08, Vu Tan wrote: > > > Hi Gaetan, > > I try to generate config file by using this command tox -egenconfig on > > top level of masakari but the output is error, is this masakari still > > in beta version ? > > [root at compute1 masakari-monitors]# tox -egenconfig > > genconfig create: /root/masakari-monitors/.tox/genconfig > > ERROR: InterpreterNotFound: python3 > > _____________________________________________________________ summary > > ______________________________________________________________ > > ERROR: genconfig: InterpreterNotFound: python3 > > > > On Mon, Jul 8, 2019 at 3:24 PM Vu Tan vungoctan252 at gmail.com>> wrote: > > Hi, > > Thanks a lot for your reply, I install pacemaker/corosync, > > masakari-api, maskari-engine on controller node, and I run masakari-api > > with this command: masakari-api, but I dont know whether the process is > > running like that or is it just hang there, here is what it shows when > > I run the command, I leave it there for a while but it does not change > > anything : > > [root at controller masakari]# masakari-api > > 2019-07-08 15:21:09.946 30250 INFO masakari.api.openstack [-] Loaded > > extensions: ['extensions', 'notifications', 'os-hosts', 'segments', > > 'versions'] > > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > > [-] The option "__file__" in conf is not known to auth_token > > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > > [-] The option "here" in conf is not known to auth_token > > 2019-07-08 15:21:09.960 30250 WARNING keystonemiddleware.auth_token [-] > > AuthToken middleware is set with > > keystone_authtoken.service_token_roles_required set to False. This is > > backwards compatible but deprecated behaviour. Please set this to True. > > 2019-07-08 15:21:09.974 30250 INFO masakari.wsgi [-] masakari_api > > listening on 127.0.0.1:15868 > > 2019-07-08 15:21:09.975 30250 INFO oslo_service.service [-] Starting 4 > > workers > > 2019-07-08 15:21:09.984 30274 INFO masakari.masakari_api.wsgi.server > > [-] (30274) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.985 30275 INFO masakari.masakari_api.wsgi.server > > [-] (30275) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.992 30277 INFO masakari.masakari_api.wsgi.server > > [-] (30277) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.994 30276 INFO masakari.masakari_api.wsgi.server > > [-] (30276) wsgi starting up on http://127.0.0.1:15868 > > > > On Sun, Jul 7, 2019 at 7:37 PM Gaëtan Trellu > > > wrote: > > > > Hi Vu Tan, > > > > Masakari documentation doesn't really exist... I had to figured some > > stuff by myself to make it works into Kolla project. > > > > On controller nodes you need: > > > > - pacemaker > > - corosync > > - masakari-api (openstack/masakari repository) > > - masakari- engine (openstack/masakari repository) > > > > On compute nodes you need: > > > > - pacemaker-remote (integrated to pacemaker cluster as a resource) > > - masakari- hostmonitor (openstack/masakari-monitor repository) > > - masakari-instancemonitor (openstack/masakari-monitor repository) > > - masakari-processmonitor (openstack/masakari-monitor repository) > > > > For masakari-hostmonitor, the service needs to have access to systemctl > > command (make sure you are not using sysvinit). > > > > For masakari-monitor, the masakari-monitor.conf is a bit different, you > > will have to configure the [api] section properly. > > > > RabbitMQ needs to be configured (as transport_url) on masakari-api and > > masakari-engine too. > > > > Please check this review[1], you will have masakari.conf and > > masakari-monitor.conf configuration examples. > > > > [1] https://review.opendev.org/#/c/615715 > > > > Gaëtan > > > > On Jul 7, 2019 12:08 AM, Vu Tan vungoctan252 at gmail.com>> wrote: > > > > VU TAN > > > > > 10:30 AM (35 minutes ago) > > > > to openstack-discuss > > > > Sorry, I resend this email because I realized that I lacked of prefix > > on this email's subject > > > > Hi, > > > > I would like to use Masakari and I'm having trouble finding a step by > > step or other documentation to get started with. Which part should be > > installed on controller, which is should be on compute, and what is the > > prerequisite to install masakari, I have installed corosync and > > pacemaker on compute and controller nodes, , what else do I need to do > > ? step I have done so far: > > - installed corosync/pacemaker > > - install masakari on compute node on this github repo: > > https://github.com/openstack/masakari > > - add masakari in to mariadb > > here is my configuration file of masakari.conf, do you mind to take a > > look at it, if I have misconfigured anything? > > > > [DEFAULT] > > enabled_apis = masakari_api > > > > # Enable to specify listening IP other than default > > masakari_api_listen = controller > > # Enable to specify port other than default > > masakari_api_listen_port = 15868 > > debug = False > > auth_strategy=keystone > > > > [wsgi] > > # The paste configuration file path > > api_paste_config = /etc/masakari/api-paste.ini > > > > [keystone_authtoken] > > www_authenticate_uri = http://controller:5000 > > auth_url = http://controller:5000 > > auth_type = password > > project_domain_id = default > > user_domain_id = default > > project_name = service > > username = masakari > > password = P at ssword > > > > [database] > > connection = mysql+pymysql://masakari:P at ssword@controller/masakari > Disclaimer: This email and any attachments are sent in strictest > confidence for the sole use of the addressee and may contain legally > privileged, confidential, and proprietary data. If you are not the intended > recipient, please advise the sender by replying promptly to this email and > then delete and destroy this email and any attachments without any further > use, copying or forwarding. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnaser at vexxhost.com Mon Jul 22 15:43:35 2019 From: mnaser at vexxhost.com (Mohammed Naser) Date: Mon, 22 Jul 2019 11:43:35 -0400 Subject: [tc] weekly update Message-ID: Hi everyone, This is the weekly update for what happened inside the Openstack TC. You can get more information by checking for changes in the openstack/governance repository. # Retired project: - TripleO-UI: https://review.opendev.org/#/c/665264/ # General changes: - the 'U' release name poll was changed to China to include all regions, provinces, and cities: https://review.opendev.org/#/c/668071/ - An empty deliverables list was added to refstack (retired team): https://review.opendev.org/#/c/671141/ Thanks! Mohammed From fungi at yuggoth.org Mon Jul 22 16:49:17 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 22 Jul 2019 16:49:17 +0000 Subject: [tc][ptl][infra] Resolution: Mandatory Repository Retirement In-Reply-To: <20190714223412.wc34j47lvgd3kykm@yuggoth.org> References: <20190714223412.wc34j47lvgd3kykm@yuggoth.org> Message-ID: <20190722164916.2uidxk66jbzsfkme@yuggoth.org> On 2019-07-14 22:34:13 +0000 (+0000), Jeremy Stanley wrote: [...] > The text of the proposed resolution can be found here: > > https://review.opendev.org/670741 [...] In case anyone was confused by the original text, this proposal has additional clarification added in the latest revision. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From hongbin034 at gmail.com Mon Jul 22 17:56:46 2019 From: hongbin034 at gmail.com (Hongbin Lu) Date: Mon, 22 Jul 2019 13:56:46 -0400 Subject: [neutron] Bug Deputy July 15 - 21 Message-ID: Hi all, Below is the list of bugs last week: * https://bugs.launchpad.net/neutron/+bug/1837252 IFLA_BR_AGEING_TIME of 0 causes flooding across bridges (Incompleted) * https://bugs.launchpad.net/neutron/+bug/1837169 Stein: testing _make_floatingip fails in Debian Sid (Invalid) * https://bugs.launchpad.net/neutron/+bug/1837012 Neutron-dynamic-routing tempest tests can't be run all in one job (Medium) * https://bugs.launchpad.net/neutron/+bug/1836834 [RFE] introduce distributed locks to ipam * https://bugs.launchpad.net/neutron/+bug/1836642 Metadata responses are very slow sometimes (High) * https://bugs.launchpad.net/neutron/+bug/1836595 test_server_connectivity_cold_migration_revert failing (Medium) * https://bugs.launchpad.net/neutron/+bug/1836565 Functional test test_keepalived_state_change_notification may fail do to race condition (Medium) Best regards, Hongbin -------------- next part -------------- An HTML attachment was scrubbed... URL: From johfulto at redhat.com Mon Jul 22 18:05:30 2019 From: johfulto at redhat.com (John Fulton) Date: Mon, 22 Jul 2019 14:05:30 -0400 Subject: [tripleo] CI scenario001-standalone broken Message-ID: As per $subject [0]. scenario010 is affected too. This correlates with the promotion of the ceph-ansible 4rc11 RPM [1]. 001 jobs using rc9 do not have this problem. I'm looking into a fix, e.g. removing the storage7-ceph-nautilus-release tag from the rc11 package as TripleO CI uses the version of ceph-ansible which is tagged for release [2]. I'll keep this thread updated with the status and when it's fixed. John [0] http://logs.openstack.org/74/670574/4/check/tripleo-ci-centos-7-scenario001-standalone/2f5c18b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-07-22_15_37_40 [1] https://cbs.centos.org/koji/buildinfo?buildID=26343 [2] https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-7/master.yml#L113 From rfolco at redhat.com Mon Jul 22 19:07:55 2019 From: rfolco at redhat.com (Rafael Folco) Date: Mon, 22 Jul 2019 16:07:55 -0300 Subject: [tripleo] TripleO CI Summary: Sprint 33 Message-ID: Greetings, The TripleO CI team has just completed Sprint 33 / Unified Sprint 12 (Jun 27 thru Jul 17). The following is a summary of completed work during this sprint cycle: - Tested check jobs for periodic image/container builds on RHEL8 and resolved its dependencies on rhui repositories. - Enabled tripleo-repos to support RHEL8. - Addressed changes in kolla to support RHEL8. - Created a nodepool image on RDO for RHEL8 w/ elements and dependencies installed. - Improved container build script to consolidate kolla build logs and ease troubleshooting. - Manually pushed rhel containers to RDO registry and identified gaps in authentication workflow for the container build jobs using buildah tooling. - Modified scripts and post playbooks to upload RHEL8 overcloud images. - Created RHEL8 jobs to build a periodic pipeline in the RDO Software Factory and provide feedback for CentOS8 coverage. - Resumed scenario007 and featureset039 job updates upstream. - Promotion status: green on all branches at most of the sprint. The planned work for the next sprint [1] are: - Get RDO on RHEL8 build jobs working in periodic pipeline and bootstrap standalone and featureset001 jobs to consume RHEL8 images/containers. - Get featureset039 job to run novajoin tests and merge scenario007 job changes. - Review templates for branched and non-branched projects to have more control on what jobs are being run and reduce queue lengths. - Resume the design work for a staging environment to test changes in the promoter server for the multi-arch builds. The Ruck and Rover for this sprint are Marios Andreou (marios) and Rafael Folco (rfolco). Please direct questions or queries to them regarding CI status or issues in #tripleo, ideally to whomever has the ‘|ruck’ suffix on their nick. Ruck/rover notes are being tracked in etherpad [2]. Thanks, rfolco [1] https://tree.taiga.io/project/tripleo-ci-board/taskboard/unified-sprint-13 [2] https://etherpad.openstack.org/p/ruckroversprint13 -------------- next part -------------- An HTML attachment was scrubbed... URL: From alifshit at redhat.com Mon Jul 22 21:09:43 2019 From: alifshit at redhat.com (Artom Lifshitz) Date: Mon, 22 Jul 2019 17:09:43 -0400 Subject: [nova] FYI to out-of-tree drivers: adding a positional argument to finish_revert_migration() Message-ID: [1] will most likely merge in its current shape or something very close to it. It adds a `migration` positional argument to the finish_revert_migration() virt driver method. `migration` is the, err, migration that's being reverted. If you have out of tree virt drivers, please update them to match this new method signature. Thanks! [1] https://review.opendev.org/#/c/668631/ From mriedemos at gmail.com Mon Jul 22 21:10:24 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Mon, 22 Jul 2019 16:10:24 -0500 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> Message-ID: <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> On 7/19/2019 8:26 AM, Massimo Sgaravatto wrote: > Is there some clean way to remove/change the problematic property from > the relevant instances ? Not really externally. For the snapshot images you should be able to update those properties on the image using the glance API (and CLIs), but for existing instances the image metadata properties are stored in the instance_system_metadata table with an image_ prefix so you'd have to update those. I would have thought there would be a translation shim for the rename of that property in nova though... https://review.opendev.org/#/c/202675/ doesn't explain at all why that was done. Note that the old hypervisor_type image meta key should be translated to img_hv_type in the object code: https://github.com/openstack/nova/blob/d5c67a3d954ddb571645886a23a0f251ae7dd2bb/nova/objects/image_meta.py#L515 -- Thanks, Matt From mriedemos at gmail.com Mon Jul 22 21:12:44 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Mon, 22 Jul 2019 16:12:44 -0500 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> Message-ID: <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> On 7/22/2019 4:10 PM, Matt Riedemann wrote: > Note that the old hypervisor_type image meta key should be translated to > img_hv_type in the object code: > > https://github.com/openstack/nova/blob/d5c67a3d954ddb571645886a23a0f251ae7dd2bb/nova/objects/image_meta.py#L515 In other words, it smells like a nova bug to me that it's not handling that renamed image property key. -- Thanks, Matt From mriedemos at gmail.com Mon Jul 22 21:21:01 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Mon, 22 Jul 2019 16:21:01 -0500 Subject: [nova] [horizon] Horizon and nova-cli inconsistency in live migration parameters In-Reply-To: <98059178-00c5-8bc0-7bbf-bf5c55c53397@mail.com> References: <98059178-00c5-8bc0-7bbf-bf5c55c53397@mail.com> Message-ID: <08d8b7bb-cb22-a0c8-82b4-61a4739088f3@gmail.com> On 7/14/2019 7:35 AM, Mary Atakova wrote: > I try to live-migrate a VM, and I do it through nova cli and through > Horizon, and nova-api logs contain different data depending on the way I > do it. > > 1. nova cli > > I run the command: > > nova live-migration 17359460-d23c-4acc-a9b1-5cf117b54430 > > and it shows up in nova-api logs as: > > {"os-migrateLive": {"block_migration": "auto", "host": null}} > > 2. horizon > > I run live migration via Horizon with unchecked Disk Overcommit and > unchecked Block Migration. > > It shows up in nova-api logs as: > > {"os-migrateLive": {"disk_over_commit": false, "block_migration": false, > "host": null}} > > This difference can lead to different parameters provided for live > migration when block_migration: auto is translated into block_migration: > true. The difference is in the compute API microversion used. The nova CLI by default negotiates the highest microversion to use between what the client understands and what the server provides. The 'auto' value for the block_migration parameter was added in 2.25 and disk_over_commit was removed in 2.25. Horizon is likely not specifying a specific microversion so it defaults to 2.1 and that's why you're getting "disk_over_commit": false, "block_migration": false because you unchecked those check boxes. In other words, for Horizon to support compute API 2.25 there would be an "Auto" option for the Block Migration value. -- Thanks, Matt From johfulto at redhat.com Mon Jul 22 21:38:09 2019 From: johfulto at redhat.com (John Fulton) Date: Mon, 22 Jul 2019 17:38:09 -0400 Subject: [tripleo] CI scenario001-standalone broken In-Reply-To: References: Message-ID: On Mon, Jul 22, 2019 at 2:05 PM John Fulton wrote: > > As per $subject [0]. scenario010 is affected too. > > This correlates with the promotion of the ceph-ansible 4rc11 RPM [1]. > 001 jobs using rc9 do not have this problem. I'm looking into a fix, > e.g. removing the storage7-ceph-nautilus-release tag from the rc11 > package as TripleO CI uses the version of ceph-ansible which is tagged > for release [2]. > > I'll keep this thread updated with the status and when it's fixed. We're now tracking this in: https://github.com/ceph/ceph-ansible/issues/4259 John > > John > > [0] http://logs.openstack.org/74/670574/4/check/tripleo-ci-centos-7-scenario001-standalone/2f5c18b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-07-22_15_37_40 > [1] https://cbs.centos.org/koji/buildinfo?buildID=26343 > [2] https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-7/master.yml#L113 From johnsomor at gmail.com Mon Jul 22 21:47:59 2019 From: johnsomor at gmail.com (Michael Johnson) Date: Mon, 22 Jul 2019 14:47:59 -0700 Subject: [Octavia]-Seeking performance numbers on Octavia In-Reply-To: References: Message-ID: Hi Prabhjit, Comments in-line below. Michael On Sun, Jul 21, 2019 at 5:24 PM Singh, Prabhjit wrote: > > Hi Michael, > > Thanks for taking the time out to send me your inputs and valuable suggestions. I do remember meeting you at the Denver Summit and hearing to a couple of your sessions. > If you wouldn't mind, I do have a few more questions and your answers would help me understand that should I continue to invest in having Octavia as one of our available LBs. > > 1. Based on your response and the amount of time you are investing in supporting Octavia, what are some of the use cases, like for e.g. if load balancing web traffic how many transactions/connections minimum can be expected. I do understand > you mentioned that it's hard to performance test Octavia but some real time situations from your testing and how customers have adopted Octavia would help me level set some expectations. This is really cloud and application specific. I would recommend you fire up an Octavia install and use your preferred tool to measure it. Some good tools are tsung, weighttp, and iperf3. > 2. We are thinking of Octavia as one of the offerings, that offers a self-serve type model. Do you know of any customers who have been able to use Octavia as one of their primary load balancers and any encouraging feedback you have gotten on Octavia. There are examples of organizations using Octavia available if you google Octavia. > 3. You suggested increasing the Ram size, I could go about making a whole new Flavor. Yes, to increase the allocated RAM for a load balancer, you would create an additional nova flavor with the specifications you would like. You can then either set this as the default nova flavor for amphora (amp_flavor_id is the setting) or you can create an Octavia flavor that specifies the nova compute flavor to use (See https://docs.openstack.org/octavia/latest/admin/flavors.html for more information on Octavia flavors). > 4. I also noticed on the haproxy.conf the maxconns is set to 2000, should I increase this, does this affect the connection per server, which you said 64000 conns per server, so if I have 10 servers can I expect somewhere close to 640000 sessions? I think you are looking at the haproxy.conf file provided by your operating system package. Octavia does not use this file, it creates it's own HAProxy configuration files as needed under /var/lib/octavia inside the amphora. The default, if the user does not specify one at listener creation, is 1,000,000. > 5. Based on some of the limitations and the dev work in progress, I think the most important feature that would make Octavia a real solid offering would be the Active-Active and Autoscaling feature. I brought this up with you in our brief conversation at the summit, and you did mention that its not a top priority at this time and you are looking for some help. I have noticed a lot of documentation has been updated on this feature, do you think with the available document and progress I could spin up a distributor and manage sessions between Amphora or it's not complete yet. Active/Active is still on our roadmap, but unfortunately the people that were working on it had to stop for personal reasons. There may be some folks picking up this work again soon. At this point the Active/Active patches up for review are non-functional and still a work in progress. > 6. We have a Triple O setup, do you think I can make the above tweaks with the Triple O setup. I think you are able to make various adjustments to Octavia with Triple O, but I do not have specifics on that. > Thanks & Regards > > Prabhjit Singh > Systems Design and Strategy - Magentabox > | O: (973) 397-4819 | M: (973) 563-4445 > > > > -----Original Message----- > From: Michael Johnson > Sent: Friday, July 19, 2019 6:00 PM > To: Singh, Prabhjit > Cc: openstack-discuss at lists.openstack.org > Subject: Re: [Octavia]-Seeking performance numbers on Octavia > > [External] > > > Hi Prabhjit, > > As you have mentioned, it is very challenging to get accurate performance results in cloud environments. There are a large number(very large in fact) of factors that can impact the overall performance of OpenStack and Octavia. > > In our OpenDev testing environment, we only have software emulation virtual machines available (Qemu running with the TCG engine) which performs extremely poorly. This means that the testing environment does not reflect how the software is used in real world deployments. > An example of this is simply booting a VM can take up to ten minutes on Qemu with TCG when it takes about twenty seconds on a real OpenStack deployment. > With this resource limitation, we cannot effectively run performance benchmarking test jobs on the OpenDev environment. > > Because of this, we don't publish performance numbers as they will not reflect what you can achieve in your environment. > > Let me try to speak to your bullet points: > 1. The Octavia team has never (to my knowledge) claimed the Amphora driver is "carrier grade". We do consider the Amphora driver to be "operator grade", which speaks to a cloud operator's perspective versus the previous offering that did not support high availability, have appropriate maintenance tooling, upgrade paths, performance, etc. > To me, "carrier grade" has an additional level of requirements including performance, latency, scale, and availability SLAs. This is not what the Octavia Amphora driver is currently ready for. That said, third party provider drivers for Octavia may be able to provide a "carrier grade" level of load balancing for OpenStack. > 2. As for performance tuning, much of this is either automatically handled by Octavia or are dependent on the application you are load balancing and your cloud deployment. For example we have many configuration settings to tune how many retries we attempt when interacting with other services. In performing and stable clouds, these can be tuned down, in others the defaults may be appropriate. If you would like faster failover, at the expense of slightly more network traffic, you can tune the health monitoring and keepalived_vrrp settings. We do not currently have a performance tuning guide for Octavia but would support someone authoring one. > 3. We do not currently have a guide for this. I will say with the version of HAproxy currently being shipped with the distributions, going beyond the 1vCPU per amphora does not gain you much. With the release of HAProxy 2.0 this has changed and we expect to be adding support for vertically scaling the Amphora in future releases. Disk space is only necessary if you are storing the flow logs locally, which I would not recommend for a performance load balancer (See the notes in the log offloading guide: > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.openstack.org%2Foctavia%2Flatest%2Fadmin%2Flog-offloading.html&data=02%7C01%7CPrabhjit.Singh22%40t-mobile.com%7C8c8032b0852446142af508d70c946ba5%7Cbe0f980bdd994b19bd7bbc71a09b026c%7C0%7C0%7C636991703917513373&sdata=rUZSLn1dWt6iHqldGsqDv6DIdUysx3H0ogJYXyQRw7o%3D&reserved=0). > Finally, the RAM usage is a factor of the number of concurrent connections and if you are enabling TLS on the load balancer. For typical load balancing loads, the default is typically fine. However, if you have high connection counts and/or TLS offloading, you may want to experiment with increasing the available RAM. > 4. The source IP issue is a known issue > (https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstoryboard.openstack.org%2F%23!%2Fstory%2F1629066&data=02%7C01%7CPrabhjit.Singh22%40t-mobile.com%7C8c8032b0852446142af508d70c946ba5%7Cbe0f980bdd994b19bd7bbc71a09b026c%7C0%7C0%7C636991703917523364&sdata=cIoDC1Ncd5uCy2as8NErGrJ0EO0IO64dZEob0XkLmyc%3D&reserved=0). We have not prioritized addressing this as we have not had anyone come forward that they needed this in their deployment. If this is an issue impacting your use case, please comment on the story to that effect and provide a use case. This will help the team prioritize this work. > Also, patches are welcome! If you are interested in working on this issue, I can help you with information about how this could be added. > It should also be noted that it is a limitation of 64,000 connections per-backend server, not per load balancer. > 5. The team uses the #openstack-lbaas IRC channel on freenode and is happy to answer questions, etc. > > To date, we have had limited resources (people and equipment) available to do performance evaluation and tuning. There are definitely kernel and HAProxy tuning settings we have evaluated and added to the Amphora driver, but I know there is more work that can be done. If you are interested in help us with this work, please let us know. > > Michael > > P.S. Here are just a few considerations that can/will impact the performance of an Octavia Amphora load balancer: > > Hardware used for the compute nodes > Network Interface Cards (NICs) used in the compute nodes Number of network ports enabled on the compute hosts Network switch configurations (Jumbo frames, and so on) Cloud network topology (leaf‐spine, fat‐tree, and so on) The OpenStack Neutron networking configuration (ML2 and ML3 drivers) Tenant networking configuration (VXLAN, VLANS, GRE, and so on) Colocation of applications and Octavia amphorae Over subscription of the compute and networking resources Protocols being load balanced Configuration settings used when creating the load balancer (connection limits, and so on) Version of OpenStack services (nova, neutron, and so on) Version of OpenStack Octavia Flavor of the OpenStack Octavia load balancer OS and hypervisor versions used Deployed security mitigations (Spectre, Meltdown, and so on) Customer application performance Health of the customer application > > On Fri, Jul 19, 2019 at 8:52 AM Singh, Prabhjit wrote: > > > > Hi > > > > > > > > I have been trying to test Octavia with some traffic generators and my > > tests are inconclusive. Appreciate your inputs on the following > > > > > > > > It would be really nice to have some performance numbers that you guys have been able to achieve for this to be termed as carrier grade. > > Would also appreciate if you could share any inputs on performance > > tuning Octavia Any recommended flavor sizes for spinning up Amphorae, the default size of 1 core, 2 Gb disk and 1 Gig RAM does not seem enough. > > Also I noticed when the Amphorae are spun up, at one time only one > > master is talking to the backend servers and has one IP that its > > using, it has to run out of ports after 64000 TCP concurrent sessions, > > id there a way to add more IPs or is this the limitation If I needed > > some help with Octavia and some guidance around performance tuning can > > someone from the community help > > > > > > > > Thanks & Regards > > > > > > > > Prabhjit Singh > > > > > > > > > > > > From mriedemos at gmail.com Mon Jul 22 22:36:02 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Mon, 22 Jul 2019 17:36:02 -0500 Subject: [kolla][nova][cinder] Got Gateway-Timeout error on VM evacuation if it has volume attached. In-Reply-To: References: Message-ID: On 7/18/2019 3:53 AM, Eddie Yen wrote: > Before I try to evacuate host, the source host had about 24 VMs running. > When I shutdown the node and execute evacuation, there're few VMs > failed. The error code is 504. > Strange is those VMs are all attach its own volume. > > Then I check nova-compute log, a detailed error has pasted at below link; > https://pastebin.com/uaE7YrP1 > > Does anyone have any experience with this? I googled but no enough > information about this. Are there errors in the cinder-api logs during the evacuate of all VMs from the host? Are you doing the evacuate operation on all VMs on the host concurrently or in serial? I wonder if you're over-loading cinder and that's causing the timeout somehow. The timeout from cinder is when deleting volume attachment records, which would be terminating connections in the storage backend under the covers. Check the cinder-volume logs for errors as well. -- Thanks, Matt From missile0407 at gmail.com Mon Jul 22 23:39:13 2019 From: missile0407 at gmail.com (Eddie Yen) Date: Tue, 23 Jul 2019 07:39:13 +0800 Subject: [kolla][nova][cinder] Got Gateway-Timeout error on VM evacuation if it has volume attached. In-Reply-To: References: Message-ID: Hi Matt, thanks for your reply first. The log I paste is from nova-compute. And I also check cinder-api & cinder-volume logs according from timestamp. Strange is, no error messages found during that time. I remember I launch evacuation on the host. Perhaps it's over-loading but it's not happening on the cinder. Because the environment is 3 all-in-one installation model. That means control+compute per node, and 3 nodes become controller HA. When I shutdown one of the node, I found all requests from API is pretty slow (can feel that when using dashboard.) And back to normal again when the node is back. I'll try do the evacuation again but with just disable nova host or stop nova services, to test if that happen again or not. Matt Riedemann 於 2019年7月23日 週二 上午6:40寫道： > On 7/18/2019 3:53 AM, Eddie Yen wrote: > > Before I try to evacuate host, the source host had about 24 VMs running. > > When I shutdown the node and execute evacuation, there're few VMs > > failed. The error code is 504. > > Strange is those VMs are all attach its own volume. > > > > Then I check nova-compute log, a detailed error has pasted at below link; > > https://pastebin.com/uaE7YrP1 > > > > Does anyone have any experience with this? I googled but no enough > > information about this. > > Are there errors in the cinder-api logs during the evacuate of all VMs > from the host? Are you doing the evacuate operation on all VMs on the > host concurrently or in serial? I wonder if you're over-loading cinder > and that's causing the timeout somehow. The timeout from cinder is when > deleting volume attachment records, which would be terminating > connections in the storage backend under the covers. Check the > cinder-volume logs for errors as well. > > -- > > Thanks, > > Matt > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From berndbausch at gmail.com Tue Jul 23 00:09:18 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Tue, 23 Jul 2019 09:09:18 +0900 Subject: [opendev] git clone fails: "The TLS connection was non-properly terminated" Message-ID: There seems to be something wrong with the opendev server. Is this a glitch, a daily occurrence (backup?), or a deeper problem? This happens around midnight between July 22 and 23 UTC. $ git clone https://opendev.org/openstack/devstack.git -b stable/stein Cloning into 'devstack'... fatal: unable to access 'https://opendev.org/openstack/devstack.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated. Same result when I try to clone any other random repo. I can get Devstack from Github, but when setting up the cloud, stack.sh wants to clone everything from git.openstack.org, which fails again. I could replace all the hardcoded /git.openstack.org/ strings with the equivalent /github.com/ URL, but I am not sure if I can trust that all Github repos are up to date. Bernd. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Tue Jul 23 00:17:08 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 22 Jul 2019 17:17:08 -0700 Subject: =?UTF-8?Q?Re:_[opendev]_git_clone_fails:_"The_TLS_connection_was_non-pro?= =?UTF-8?Q?perly_terminated"?= In-Reply-To: References: Message-ID: <584e116c-05c9-4eac-8bf2-804a098912e1@www.fastmail.com> On Mon, Jul 22, 2019, at 5:09 PM, Bernd Bausch wrote: > There seems to be something wrong with the opendev server. Is this a > glitch, a daily occurrence (backup?), or a deeper problem? > > This happens around midnight between July 22 and 23 UTC. > > $ git clone https://opendev.org/openstack/devstack.git -b stable/stein > Cloning into 'devstack'... > fatal: unable to access 'https://opendev.org/openstack/devstack.git/': > gnutls_handshake() failed: The TLS connection was non-properly > terminated. > > Same result when I try to clone any other random repo. I can get > Devstack from Github, but when setting up the cloud, stack.sh wants to > clone everything from git.openstack.org, which fails again. I could > replace all the hardcoded *git.openstack.org* strings with the > equivalent *github.com* URL, but I am not sure if I can trust that all > Github repos are up to date. > > Bernd. I think this may have happened because because one of the backend servers was removed from the load balancer rotation. And while we thought we had removed that server from the haproxy config [0] that didn't happen because our ansible doesn't know how to gracefully restart haproxy in a container. At this point haproxy should be aware that the server is gone and it will prevent any new connections from going to it. All that to say I believe this should be a one off. We'll have to be more careful to ensure backends are rotated off gracefully when replacing them. If you see this persist that information would be very useful. Thank you. [0] https://review.opendev.org/#/c/672083/ From fungi at yuggoth.org Tue Jul 23 00:53:42 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 23 Jul 2019 00:53:42 +0000 Subject: [opendev] git clone fails: "The TLS connection was non-properly terminated" In-Reply-To: <584e116c-05c9-4eac-8bf2-804a098912e1@www.fastmail.com> References: <584e116c-05c9-4eac-8bf2-804a098912e1@www.fastmail.com> Message-ID: <20190723005342.zr6h5tyqdetmu3ss@yuggoth.org> On 2019-07-22 17:17:08 -0700 (-0700), Clark Boylan wrote: > On Mon, Jul 22, 2019, at 5:09 PM, Bernd Bausch wrote: > > There seems to be something wrong with the opendev server. Is this a > > glitch, a daily occurrence (backup?), or a deeper problem? > > > > This happens around midnight between July 22 and 23 UTC. > > > > $ git clone https://opendev.org/openstack/devstack.git -b stable/stein > > Cloning into 'devstack'... > > fatal: unable to access 'https://opendev.org/openstack/devstack.git/': > > gnutls_handshake() failed: The TLS connection was non-properly > > terminated. [...] > I think this may have happened because because one of the backend > servers was removed from the load balancer rotation. And while we > thought we had removed that server from the haproxy config [0] > that didn't happen because our ansible doesn't know how to > gracefully restart haproxy in a container. At this point haproxy > should be aware that the server is gone and it will prevent any > new connections from going to it. [...] Yep, sorry, we commented gitea01 out of the load balancer and I subsequently deleted its server instance at 2019-07-22 23:25 UTC, but I neglected to double-check that haproxy had actually stopped distributing connections to it. That one's on me, seems we still have some work to do making sure haproxy config changes are applied automatically there. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From berndbausch at gmail.com Tue Jul 23 01:02:20 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Tue, 23 Jul 2019 10:02:20 +0900 Subject: [opendev] git clone fails: "The TLS connection was non-properly terminated" In-Reply-To: References: Message-ID: <815106df-8b34-28e2-aea5-f4d2c35e5310@gmail.com> Please ignore; this seems to be a problem on my side. On 7/23/2019 9:09 AM, Bernd Bausch wrote: > > There seems to be something wrong with the opendev server. Is this a > glitch, a daily occurrence (backup?), or a deeper problem? > > This happens around midnight between July 22 and 23 UTC. > > $ git clone https://opendev.org/openstack/devstack.git -b stable/stein > Cloning into 'devstack'... > fatal: unable to access 'https://opendev.org/openstack/devstack.git/': > gnutls_handshake() failed: The TLS connection was non-properly terminated. > > Same result when I try to clone any other random repo. I can get > Devstack from Github, but when setting up the cloud, stack.sh wants to > clone everything from git.openstack.org, which fails again. I could > replace all the hardcoded /git.openstack.org/ strings with the > equivalent /github.com/ URL, but I am not sure if I can trust that all > Github repos are up to date. > > Bernd. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangtrinhnt at gmail.com Tue Jul 23 01:28:46 2019 From: dangtrinhnt at gmail.com (Trinh Nguyen) Date: Tue, 23 Jul 2019 10:28:46 +0900 Subject: [telemetry] Looking for new meeting time In-Reply-To: References: Message-ID: How about Saturday, old meeting time? On Mon, Jul 15, 2019 at 1:05 PM Lingxian Kong wrote: > congratulations first :-) > > I'm on UTC+12, so meeting during 13:00-15:00 UTC is hard for me :-( > > Best regards, > Lingxian Kong > Catalyst Cloud > > > On Mon, Jul 15, 2019 at 1:23 PM Trinh Nguyen > wrote: > >> Hi team, >> >> I've changed job and could not afford doing the meeting at 02:00 UTC on >> Thursday anymore. My possible time slot is from 13:00-15:00 UTC >> Tuesday-Thursday. >> >> Please propose a better time frame for our team meeting. >> >> Sincerely, >> >> -- >> *Trinh Nguyen* >> *www.edlab.xyz * >> >> -- *Trinh Nguyen* *www.edlab.xyz * -------------- next part -------------- An HTML attachment was scrubbed... URL: From marios at redhat.com Tue Jul 23 07:51:58 2019 From: marios at redhat.com (Marios Andreou) Date: Tue, 23 Jul 2019 10:51:58 +0300 Subject: [tripleo] CI scenario001-standalone broken In-Reply-To: References: Message-ID: thanks John we have that https://bugs.launchpad.net/tripleo/+bug/1837451 and also a temp workaround from ykarel to exclude the problematic rpm https://review.opendev.org/#/c/672197/1 On Tue, Jul 23, 2019 at 12:39 AM John Fulton wrote: > On Mon, Jul 22, 2019 at 2:05 PM John Fulton wrote: > > > > As per $subject [0]. scenario010 is affected too. > > > > This correlates with the promotion of the ceph-ansible 4rc11 RPM [1]. > > 001 jobs using rc9 do not have this problem. I'm looking into a fix, > > e.g. removing the storage7-ceph-nautilus-release tag from the rc11 > > package as TripleO CI uses the version of ceph-ansible which is tagged > > for release [2]. > > > > I'll keep this thread updated with the status and when it's fixed. > > We're now tracking this in: > https://github.com/ceph/ceph-ansible/issues/4259 > > John > > > > > John > > > > [0] > http://logs.openstack.org/74/670574/4/check/tripleo-ci-centos-7-scenario001-standalone/2f5c18b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-07-22_15_37_40 > > [1] https://cbs.centos.org/koji/buildinfo?buildID=26343 > > [2] > https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-7/master.yml#L113 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at stackhpc.com Tue Jul 23 10:29:28 2019 From: mark at stackhpc.com (Mark Goddard) Date: Tue, 23 Jul 2019 11:29:28 +0100 Subject: [kolla] Stein release now available Message-ID: Hi, I'm pleased to announce that the first official Stein cycle releases for Kolla and Kolla Ansible are now available - both 8.0.0. A lot of work has gone into this release, including improving the CI tests, making it the most well tested release yet. Thanks to everyone who contributed. Release notes: https://docs.openstack.org/releasenotes/kolla/stein.html https://docs.openstack.org/releasenotes/kolla-ansible/stein.html Join us on #openstack-kolla to help make the Train cycle release even better. Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean-philippe at evrard.me Tue Jul 23 10:55:09 2019 From: jean-philippe at evrard.me (Jean-Philippe Evrard) Date: Tue, 23 Jul 2019 12:55:09 +0200 Subject: [all][meta-sig][containers-sig] Creation of Containers SIG Message-ID: Hello everyone, I have discussed with some of you about the fact OpenStack is lacking a a group interested in sharing the practices for the building/management of OpenStack application containers. Every container related project is doing its own way (and it's probably fine), but very few experiences are shared (and it's sad). I started an etherpad [1] to see areas where container projects/people could collaborate. I already received significant feedback on how to improve some of my practices, so I think it's already a positive experience for me to have this group started. Anyway, long story short, I believe we should create a SIG for this, so I proposed the addition of the containers-sig into governance [2]. We have enough work to started this group nowadays (with documentation, bindep shared practices, and improvement of projects). I suppose technologies will evolve over time, so I expect a SIG is the best tool to do this job of *for example* gathering experience, as it's not really time bound, and it _is_ a quite special interest. If you're interested by learning what is going on in the container building world for OpenStack, please add [containers-sig] to your openstack-discuss topics watch list. Regards, Jean-Philippe Evrard (evrardjp) [1]: https://etherpad.openstack.org/p/containers-sig [2]: https://review.opendev.org/#/c/666001/ From andrecutm at gmail.com Tue Jul 23 11:12:54 2019 From: andrecutm at gmail.com (Emanuel Andrecut) Date: Tue, 23 Jul 2019 14:12:54 +0300 Subject: [dev][magnum] Magnum event notifications not having relevant content related to clusters In-Reply-To: References: Message-ID: Hello again, I've actually started a change here https://review.opendev.org/#/c/670312/ Any chance it will get accepted in the current release series? Thanks On Thu, 27 Jun 2019 at 03:57, Feilong wrote: > > Hi Emnauel, any patch is welcome. Pls go for it. > > > > Sent from my phone. Please excuse the brevity of the message and typos. > On Emnauel Andrecut , Jun 26, 2019 22:20 wrote: > > I'm writing about Magnum event notifications that do not contain any > relevant information about the related Cluster (missing cluster id, > other attributes..) > As I've seen the PyCADF library receives no targetId and so the id > (and mostly any other id) is randomly generated. > Do you plan to add more information about the cluster on create > success/pending events. Or would you accept a patch that does that? > From gfidente at redhat.com Tue Jul 23 11:14:40 2019 From: gfidente at redhat.com (Giulio Fidente) Date: Tue, 23 Jul 2019 13:14:40 +0200 Subject: [tripleo] CI scenario001-standalone broken In-Reply-To: References: Message-ID: <8743b415-9a89-7df2-409b-611d9dfd31a3@redhat.com> On 7/23/19 9:51 AM, Marios Andreou wrote: > thanks John we have that https://bugs.launchpad.net/tripleo/+bug/1837451 > and also a temp workaround from ykarel to exclude the problematic > rpm https://review.opendev.org/#/c/672197/1 we also have a fixed build of ceph-ansible (rc12) in centos cbs now, the plan is to test it via [1] and [2] and if passes promote for -release 1. https://review.opendev.org/#/c/501987/ 2. https://review.opendev.org/#/c/656935/ -- Giulio Fidente GPG KEY: 08D733BA From gfidente at redhat.com Tue Jul 23 11:45:00 2019 From: gfidente at redhat.com (Giulio Fidente) Date: Tue, 23 Jul 2019 13:45:00 +0200 Subject: [tripleo] Roadmap to simplify TripleO In-Reply-To: References: Message-ID: <1a1620bf-010c-fe37-8b35-7d8b506dbaac@redhat.com> On 7/11/19 7:52 PM, Emilien Macchi wrote: > Even though TripleO is well known for its maturity, it also has a > reputation of being complex when it comes to the number of tools that it > uses. > Somewhat related to the efforts that James is leading with "Scaling > TripleO" [1] [2], I would like to formalize our joint efforts to make > TripleO simpler in the future. Some work has been done over the last > releases already and yet we have seen net benefits; however we still > have challenges ahead of us. > > - With no UI anymore, do we really need an API? > - How can we reduce the number of languages in TripleO? ... and make > Python + Ansible the main ones. > - How can we reduce our dependencies? > > I created a document which explains the problems and propose some solutions: >   > https://docs.google.com/document/d/1vY9rsccgp7NHFXpLtCFTHQm14F15Tij7lhn5X_P14Ys > For those who can't or don't want Google Doc, I've put together the > notes into etherpad [3] and I'll take care of making sure it's updated > at last at the beginning until we sort things out. thanks Emilien, I added a couple of comments in the doc related to two topics which I think have been half-baked in past releases and could be reviewed with the removal of mistral, specifically: - TripleO invocation environment files should not be order dependent - Performing a stack update should not require all original environment files not sure if people familiar with workflows would be interested in helping with these two? -- Giulio Fidente GPG KEY: 08D733BA From juliaashleykreger at gmail.com Tue Jul 23 12:10:23 2019 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Tue, 23 Jul 2019 08:10:23 -0400 Subject: [ironic] Using ironic in a devstack environment to deploy OS on bare-metal host. In-Reply-To: <8a82c02e5ae64fc79e02e10629a65556@lenovo.com> References: <8a82c02e5ae64fc79e02e10629a65556@lenovo.com> Message-ID: Greetings, Typically this sort of error is seen when the IPA ramdisk is unable to find the drivers for the disk device. Are you trying to build your own IPA image for your hardware? Are you trying to use tinyipa? Coreos? If your using a hardware raid controller? have you pre-created a volume for the OS? Could it be that the hard disk failed? -Julia On Mon, Jul 15, 2019 at 2:56 AM Guannan GN2 Sun wrote: > > Hi team, > > > I met some problem when using ironic in a devstack environment to deploy OS on bare-metal host. > > By now I can successfully boot into the tiny os and set ip, also I can ping the ip from my devstack server. However, when the tiny os running some configuration, it will failed after about 5 minutes. > > I found some log information in "/var/log/ironic-python-agent.log" in the tiny os that may relate to the problem: > > > "DEBUG oslo_concurrency.processutils [-] u'iscsistart -f' failed. Not Retrying. execute /usr/local/lib/python2.7/site-packages/oslo_concurrency/processutils.py:457" > > "DEBUG root [-] No iscsi connection detected. Skipping iscsi. Error: [Errno 2] No such file or directory _check_for_iscsi /usr/local/lib/python2.7/site-packages/ironic_python_agent/hardware.py:107" > > "WARNING root [-] The root device was not detected in 27 seconds: DeviceNotFound: Error finding the disk or partition device to deploy the image onto: No suitable device was found for deployment - root device hints were not provided and all found block devices are smaller than 4294967296B." > > "WARNING root [-] Can't find field vendor for device usb0 in device class net: IOError: [Errno 2] No such file or directory: '/sys/class/net/usb0/device/vendor'" > > "WARNING root [-] Can't find field vendor for device tunl0 in device class net: IOError: [Errno 2] No such file or directory: '/sys/class/net/tunl0/device/vendor'" > > "DEBUG root [-] No Mellanox devices found evaluate_hardware_support /usr/local/lib/python2.7/site-packages/ironic_python_agent/hardware_managers/mlnx.py:84" > > > And when it deploy faild and I run command "openstack baremetal node show server-51", it will show following information on 'last_error' field: > > > "Asynchronous exception: Node failed to deploy. Exception: Connection to agent failed: Failed to connect to the agent running on node f8794658-bc69-40cb-9673-be75f78ced21 for invoking command iscsi.start_iscsi_target. Error: ('Connection aborted.', BadStatusLine("''",)) for node" > > > I'm trying to debug, but have not find the root cause, please give me your advice if someone have experience on it. Thank you! > > > Best Regards, > > Guannan From e0ne at e0ne.info Tue Jul 23 12:36:57 2019 From: e0ne at e0ne.info (Ivan Kolodyazhny) Date: Tue, 23 Jul 2019 15:36:57 +0300 Subject: [opendev] git clone fails: "The TLS connection was non-properly terminated" In-Reply-To: <815106df-8b34-28e2-aea5-f4d2c35e5310@gmail.com> References: <815106df-8b34-28e2-aea5-f4d2c35e5310@gmail.com> Message-ID: Hi all, Once it works well on my laptop, I'm still facing the same issue on the fresh ubuntu18.04-based VM. Regards, Ivan Kolodyazhny, http://blog.e0ne.info/ On Tue, Jul 23, 2019 at 4:02 AM Bernd Bausch wrote: > Please ignore; this seems to be a problem on my side. > On 7/23/2019 9:09 AM, Bernd Bausch wrote: > > There seems to be something wrong with the opendev server. Is this a > glitch, a daily occurrence (backup?), or a deeper problem? > > This happens around midnight between July 22 and 23 UTC. > > $ git clone https://opendev.org/openstack/devstack.git -b stable/stein > Cloning into 'devstack'... > fatal: unable to access 'https://opendev.org/openstack/devstack.git/': > gnutls_handshake() failed: The TLS connection was non-properly terminated. > > Same result when I try to clone any other random repo. I can get Devstack > from Github, but when setting up the cloud, stack.sh wants to clone > everything from git.openstack.org, which fails again. I could replace all > the hardcoded *git.openstack.org * strings with > the equivalent *github.com * URL, but I am not sure if > I can trust that all Github repos are up to date. > > Bernd. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From e0ne at e0ne.info Tue Jul 23 12:50:48 2019 From: e0ne at e0ne.info (Ivan Kolodyazhny) Date: Tue, 23 Jul 2019 15:50:48 +0300 Subject: [opendev] git clone fails: "The TLS connection was non-properly terminated" In-Reply-To: References: <815106df-8b34-28e2-aea5-f4d2c35e5310@gmail.com> Message-ID: Looks like some CIs are affected too: http://logs.openstack.org/88/577388/23/check/grenade-vitrage/13de30a/logs/grenade.sh.txt.gz#_2019-07-23_12_42_37_019 Regards, Ivan Kolodyazhny, http://blog.e0ne.info/ On Tue, Jul 23, 2019 at 3:36 PM Ivan Kolodyazhny wrote: > Hi all, > > Once it works well on my laptop, I'm still facing the same issue on the > fresh ubuntu18.04-based VM. > > Regards, > Ivan Kolodyazhny, > http://blog.e0ne.info/ > > > On Tue, Jul 23, 2019 at 4:02 AM Bernd Bausch > wrote: > >> Please ignore; this seems to be a problem on my side. >> On 7/23/2019 9:09 AM, Bernd Bausch wrote: >> >> There seems to be something wrong with the opendev server. Is this a >> glitch, a daily occurrence (backup?), or a deeper problem? >> >> This happens around midnight between July 22 and 23 UTC. >> >> $ git clone https://opendev.org/openstack/devstack.git -b stable/stein >> Cloning into 'devstack'... >> fatal: unable to access 'https://opendev.org/openstack/devstack.git/': >> gnutls_handshake() failed: The TLS connection was non-properly terminated. >> >> Same result when I try to clone any other random repo. I can get Devstack >> from Github, but when setting up the cloud, stack.sh wants to clone >> everything from git.openstack.org, which fails again. I could replace >> all the hardcoded *git.openstack.org * strings >> with the equivalent *github.com * URL, but I am not >> sure if I can trust that all Github repos are up to date. >> >> Bernd. >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tushar.Patil at nttdata.com Tue Jul 23 13:12:46 2019 From: Tushar.Patil at nttdata.com (Patil, Tushar) Date: Tue, 23 Jul 2019 13:12:46 +0000 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: References: <35400f83-c29d-475a-8d36-d56b3cf16d30@email.android.com>, Message-ID: Hi Vu Tan, I'm trying to install Masakari using source code to reproduce the issue. If I hit the same issue as yours, I will troubleshoot this issue and let you know the solution or will update you what steps I have followed to bring up Masakari services successfully. Regards, Tushar Patil ________________________________________ From: Vu Tan Sent: Monday, July 22, 2019 12:33 PM To: Gaëtan Trellu Cc: Patil, Tushar; openstack-discuss at lists.openstack.org Subject: Re: [masakari] how to install masakari on centos 7 Hi Patil, May I know when the proper document for masakari is released ? I have configured conf file in controller and compute node, it seems running but it is not running as it should be, a lots of error in logs, here is a sample log: 2.7/site-packages/oslo_config/cfg.py:3024 2019-07-19 10:25:26.360 7745 DEBUG oslo_service.service [-] bindir = /usr/local/bin log_opt_values /usr/lib/ python2.7/site-packages/oslo_config/cfg.py:3024 2019-07-19 18:46:21.291 7770 ERROR masakari File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 65, in _is_daemo n 2019-07-19 18:46:21.291 7770 ERROR masakari is_daemon = os.getpgrp() != os.tcgetpgrp(sys.stdout.fileno()) 2019-07-19 18:46:21.291 7770 ERROR masakari OSError: [Errno 5] Input/output error 2019-07-19 18:46:21.291 7770 ERROR masakari 2019-07-19 18:46:21.300 7745 CRITICAL masakari [-] Unhandled error: OSError: [Errno 5] Input/output error 2019-07-19 18:46:21.300 7745 ERROR masakari Traceback (most recent call last): 2019-07-19 18:46:21.300 7745 ERROR masakari File "/usr/bin/masakari-api", line 10, in I dont know if it is missing package or wrong configuration On Thu, Jul 11, 2019 at 6:14 PM Gaëtan Trellu > wrote: You will have to enable the debit, debug = true and check the APi log. Did you try to use the openstack CLi ? Gaetan On Jul 11, 2019 12:32 AM, Vu Tan > wrote: I know it's just a warning, just take a look at this image: [image.png] it's just hang there forever, and in the log show what I have shown to you On Wed, Jul 10, 2019 at 8:07 PM Gaëtan Trellu > wrote: This is just a warning, not an error. On Jul 10, 2019 3:12 AM, Vu Tan > wrote: Hi Gaetan, I follow you the guide you gave me, but the problem still persist, can you please take a look at my configuration to see what is wrong or what is missing in my config ? the error: 2019-07-10 14:08:46.876 17292 WARNING keystonemiddleware._common.config [-] The option "__file__" in conf is not known to auth_token 2019-07-10 14:08:46.876 17292 WARNING keystonemiddleware._common.config [-] The option "here" in conf is not known to auth_token 2019-07-10 14:08:46.882 17292 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_ the config: [DEFAULT] enabled_apis = masakari_api log_dir = /var/log/kolla/masakari state_path = /var/lib/masakari os_user_domain_name = default os_project_domain_name = default os_privileged_user_tenant = service os_privileged_user_auth_url = http://controller:5000/v3 os_privileged_user_name = nova os_privileged_user_password = P at ssword masakari_api_listen = controller masakari_api_listen_port = 15868 debug = False auth_strategy=keystone [wsgi] # The paste configuration file path api_paste_config = /etc/masakari/api-paste.ini [keystone_authtoken] www_authenticate_uri = http://controller:5000 auth_url = http://controller:5000 auth_type = password project_domain_id = default project_domain_name = default user_domain_name = default user_domain_id = default project_name = service username = masakari password = P at ssword region_name = RegionOne [oslo_middleware] enable_proxy_headers_parsing = True [database] connection = mysql+pymysql://masakari:P at ssword@controller/masakari On Tue, Jul 9, 2019 at 10:25 PM Vu Tan > wrote: Thank Patil Tushar, I hope it will be available soon On Tue, Jul 9, 2019 at 8:18 AM Patil, Tushar > wrote: Hi Vu and Gaetan, Gaetan, thank you for helping out Vu in setting up masakari-monitors service. As a masakari team ,we have noticed there is a need to add proper documentation to help the community run Masakari services in their environment. We are working on adding proper documentation in this 'Train' cycle. Will send an email on this mailing list once the patches are uploaded on the gerrit so that you can give your feedback on the same. If you have any trouble in setting up Masakari, please let us know on this mailing list or join the bi-weekly IRC Masakari meeting on the #openstack-meeting IRC channel. The next meeting will be held on 16th July 2019 @0400 UTC. Regards, Tushar Patil ________________________________________ From: Vu Tan > Sent: Monday, July 8, 2019 11:21:16 PM To: Gaëtan Trellu Cc: openstack-discuss at lists.openstack.org Subject: Re: [masakari] how to install masakari on centos 7 Hi Gaetan, Thanks for pinpoint this out, silly me that did not notice the simple "error InterpreterNotFound: python3". Thanks a lot, I appreciate it On Mon, Jul 8, 2019 at 9:15 PM >> wrote: Vu Tan, About "auth_token" error, you need "os_privileged_user_*" options into your masakari.conf for the API. As mentioned previously please have a look here to have an example of configuration working (for me at least): - masakari.conf: https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari.conf.j2 - masakari-monitor.conf: https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari-monitors.conf.j2 About your tox issue make sure you have Python3 installed. Gaëtan On 2019-07-08 06:08, Vu Tan wrote: > Hi Gaetan, > I try to generate config file by using this command tox -egenconfig on > top level of masakari but the output is error, is this masakari still > in beta version ? > [root at compute1 masakari-monitors]# tox -egenconfig > genconfig create: /root/masakari-monitors/.tox/genconfig > ERROR: InterpreterNotFound: python3 > _____________________________________________________________ summary > ______________________________________________________________ > ERROR: genconfig: InterpreterNotFound: python3 > > On Mon, Jul 8, 2019 at 3:24 PM Vu Tan >> wrote: > Hi, > Thanks a lot for your reply, I install pacemaker/corosync, > masakari-api, maskari-engine on controller node, and I run masakari-api > with this command: masakari-api, but I dont know whether the process is > running like that or is it just hang there, here is what it shows when > I run the command, I leave it there for a while but it does not change > anything : > [root at controller masakari]# masakari-api > 2019-07-08 15:21:09.946 30250 INFO masakari.api.openstack [-] Loaded > extensions: ['extensions', 'notifications', 'os-hosts', 'segments', > 'versions'] > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > [-] The option "__file__" in conf is not known to auth_token > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > [-] The option "here" in conf is not known to auth_token > 2019-07-08 15:21:09.960 30250 WARNING keystonemiddleware.auth_token [-] > AuthToken middleware is set with > keystone_authtoken.service_token_roles_required set to False. This is > backwards compatible but deprecated behaviour. Please set this to True. > 2019-07-08 15:21:09.974 30250 INFO masakari.wsgi [-] masakari_api > listening on 127.0.0.1:15868 > 2019-07-08 15:21:09.975 30250 INFO oslo_service.service [-] Starting 4 > workers > 2019-07-08 15:21:09.984 30274 INFO masakari.masakari_api.wsgi.server > [-] (30274) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.985 30275 INFO masakari.masakari_api.wsgi.server > [-] (30275) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.992 30277 INFO masakari.masakari_api.wsgi.server > [-] (30277) wsgi starting up on http://127.0.0.1:15868 > 2019-07-08 15:21:09.994 30276 INFO masakari.masakari_api.wsgi.server > [-] (30276) wsgi starting up on http://127.0.0.1:15868 > > On Sun, Jul 7, 2019 at 7:37 PM Gaëtan Trellu > >> wrote: > > Hi Vu Tan, > > Masakari documentation doesn't really exist... I had to figured some > stuff by myself to make it works into Kolla project. > > On controller nodes you need: > > - pacemaker > - corosync > - masakari-api (openstack/masakari repository) > - masakari- engine (openstack/masakari repository) > > On compute nodes you need: > > - pacemaker-remote (integrated to pacemaker cluster as a resource) > - masakari- hostmonitor (openstack/masakari-monitor repository) > - masakari-instancemonitor (openstack/masakari-monitor repository) > - masakari-processmonitor (openstack/masakari-monitor repository) > > For masakari-hostmonitor, the service needs to have access to systemctl > command (make sure you are not using sysvinit). > > For masakari-monitor, the masakari-monitor.conf is a bit different, you > will have to configure the [api] section properly. > > RabbitMQ needs to be configured (as transport_url) on masakari-api and > masakari-engine too. > > Please check this review[1], you will have masakari.conf and > masakari-monitor.conf configuration examples. > > [1] https://review.opendev.org/#/c/615715 > > Gaëtan > > On Jul 7, 2019 12:08 AM, Vu Tan >> wrote: > > VU TAN >> > > 10:30 AM (35 minutes ago) > > to openstack-discuss > > Sorry, I resend this email because I realized that I lacked of prefix > on this email's subject > > Hi, > > I would like to use Masakari and I'm having trouble finding a step by > step or other documentation to get started with. Which part should be > installed on controller, which is should be on compute, and what is the > prerequisite to install masakari, I have installed corosync and > pacemaker on compute and controller nodes, , what else do I need to do > ? step I have done so far: > - installed corosync/pacemaker > - install masakari on compute node on this github repo: > https://github.com/openstack/masakari > - add masakari in to mariadb > here is my configuration file of masakari.conf, do you mind to take a > look at it, if I have misconfigured anything? > > [DEFAULT] > enabled_apis = masakari_api > > # Enable to specify listening IP other than default > masakari_api_listen = controller > # Enable to specify port other than default > masakari_api_listen_port = 15868 > debug = False > auth_strategy=keystone > > [wsgi] > # The paste configuration file path > api_paste_config = /etc/masakari/api-paste.ini > > [keystone_authtoken] > www_authenticate_uri = http://controller:5000 > auth_url = http://controller:5000 > auth_type = password > project_domain_id = default > user_domain_id = default > project_name = service > username = masakari > password = P at ssword > > [database] > connection = mysql+pymysql://masakari:P at ssword@controller/masakari Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. From massimo.sgaravatto at gmail.com Tue Jul 23 13:12:38 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Tue, 23 Jul 2019 15:12:38 +0200 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> Message-ID: I was wrong: the alias work ! I.e.: img_hv_type=xyz is equivalent to: hypervisor_type=xyz The problem is when xyz is 'qemu'. This used to work with Ocata and now (Rocky) is not working anymore in "my" cloud. It works instead if I use kvm. This is weird since in https://docs.openstack.org/nova/rocky/admin/configuration/schedulers.html it is reported that: qemu is used for both QEMU and KVM hypervisor types Thanks, Massimo On Mon, Jul 22, 2019 at 11:16 PM Matt Riedemann wrote: > On 7/22/2019 4:10 PM, Matt Riedemann wrote: > > Note that the old hypervisor_type image meta key should be translated to > > img_hv_type in the object code: > > > > > https://github.com/openstack/nova/blob/d5c67a3d954ddb571645886a23a0f251ae7dd2bb/nova/objects/image_meta.py#L515 > > In other words, it smells like a nova bug to me that it's not handling > that renamed image property key. > > -- > > Thanks, > > Matt > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mriedemos at gmail.com Tue Jul 23 13:44:09 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Tue, 23 Jul 2019 08:44:09 -0500 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> Message-ID: <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> On 7/23/2019 8:12 AM, Massimo Sgaravatto wrote: > I was wrong: the alias work ! > I.e.: > > img_hv_type=xyz > > is equivalent to: > > hypervisor_type=xyz > > The problem is when xyz is 'qemu'. This used to work with Ocata and now > (Rocky) is not working anymore in "my" cloud. It works instead if I use kvm. > > This is weird since in > > https://docs.openstack.org/nova/rocky/admin/configuration/schedulers.html > > it is reported that: > > > qemu is used for both QEMU and KVM hypervisor types > > > Thanks, Massimo Hmm just to be clear, you used to use hypervisor_type=QEMU and that no longer works, and the problem isn't hypervisor_type vs img_hv_type (the key), but the value QEMU vs qemu. Which isn't working in Rocky? QEMU or qemu? What is the hypervisor_type on your nodes when listing them out of the API? (openstack hypervisor list --long should give you that output). Do you have a mix of QEMU vs qemu on some nodes? This sort of sounds like bug 1818092. -- Thanks, Matt From massimo.sgaravatto at gmail.com Tue Jul 23 13:57:48 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Tue, 23 Jul 2019 15:57:48 +0200 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> Message-ID: Sorry: I was not clear When running Ocata I had as property of some images: hypervisor_type='QEMU' and this worked Now in Rocjy: hypervisor_type='QEMU' --> doesn't work (i.e. all hypervisors are excluded by ImagePropertiesFilter) hypervisor_type='qemu' --> doesn't work (i.e. all hypervisors are excluded by ImagePropertiesFilter) hypervisor_type='kvm' --> works "openstack hypervisor list --long" reports "QEMU" as Hypervisor Type for all compute nodes Thanks again for your help Cheers, Massimo On Tue, Jul 23, 2019 at 3:44 PM Matt Riedemann wrote: > On 7/23/2019 8:12 AM, Massimo Sgaravatto wrote: > > I was wrong: the alias work ! > > I.e.: > > > > img_hv_type=xyz > > > > is equivalent to: > > > > hypervisor_type=xyz > > > > The problem is when xyz is 'qemu'. This used to work with Ocata and now > > (Rocky) is not working anymore in "my" cloud. It works instead if I use > kvm. > > > > This is weird since in > > > > > https://docs.openstack.org/nova/rocky/admin/configuration/schedulers.html > > > > it is reported that: > > > > > > qemu is used for both QEMU and KVM hypervisor types > > > > > > Thanks, Massimo > > Hmm just to be clear, you used to use hypervisor_type=QEMU and that no > longer works, and the problem isn't hypervisor_type vs img_hv_type (the > key), but the value QEMU vs qemu. Which isn't working in Rocky? QEMU or > qemu? > > What is the hypervisor_type on your nodes when listing them out of the > API? (openstack hypervisor list --long should give you that output). Do > you have a mix of QEMU vs qemu on some nodes? This sort of sounds like > bug 1818092. > > -- > > Thanks, > > Matt > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mriedemos at gmail.com Tue Jul 23 14:35:27 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Tue, 23 Jul 2019 09:35:27 -0500 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> Message-ID: On 7/23/2019 8:57 AM, Massimo Sgaravatto wrote: > > When running Ocata I had as property of some images: > > hypervisor_type='QEMU' > > and this worked > > Now in Rocjy: > > hypervisor_type='QEMU' --> doesn't work (i.e. all hypervisors are > excluded by  ImagePropertiesFilter) > hypervisor_type='qemu' --> doesn't work  (i.e. all hypervisors are > excluded by  ImagePropertiesFilter) > hypervisor_type='kvm' --> works > > "openstack hypervisor list --long"  reports "QEMU" as Hypervisor Type > for all compute nodes Apparently the filter doesn't use the ComputeNode.hypervisor_type field (which is what you see in the API/CLI output) to compare the img_hv_type property, it relies on some ComputeNode.supported_instances tuples which are reported differently by the driver. Can you enable debug in the scheduler so we could see this output when you get the NoValidHost? https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/filters/image_props_filter.py#L97 Did you upgrade libvirt/qemu as well when you upgraded these nodes to Rocky? I wonder if the supported instance hypervisor type reported by the virt driver is kvm now rather than qemu even though the hypervisor type reported in the API shows QEMU. FWIW this is the virt driver code that reports that supported_instances information for the compute node that's used by the scheduler filter: https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/driver.py#L5846 -- Thanks, Matt From massimo.sgaravatto at gmail.com Tue Jul 23 14:50:55 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Tue, 23 Jul 2019 16:50:55 +0200 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> Message-ID: This [*] is what appears in nova-scheduler after having enabled the debug. We performed a "yum update" so, yes, we also updated libvirt (now we are running v. 4.5.0) Thanks, Massimo [*] 2019-07-23 16:44:34.849 12561 DEBUG nova.scheduler.filters.image_props_filter [req-52638278-51b7-4768-836a-f70d8a8b016a ab573ba3ea014b778193b6922ffffe6d ee1865a76440481cbcff08544c7d580a - default default] Instance contains properties ImageMetaProps(hw_architecture=,hw_auto_disk_config=,hw_boot_menu=,hw_cdrom_bus=,hw_cpu_cores=,hw_cpu_max_cores=,hw_cpu_max_sockets=,hw_cpu_max_threads=,hw_cpu_policy=,hw_cpu_realtime_mask=,hw_cpu_sockets=,hw_cpu_thread_policy=,hw_cpu_threads=,hw_device_id=,hw_disk_bus=,hw_disk_type=,hw_firmware_type=,hw_floppy_bus=,hw_ipxe_boot=,hw_machine_type=,hw_mem_page_size=,hw_numa_cpus=,hw_numa_mem=,hw_numa_nodes=,hw_pointer_model=,hw_qemu_guest_agent=,hw_rescue_bus=,hw_rescue_device=,hw_rng_model=,hw_scsi_model=,hw_serial_port_count=,hw_video_model=,hw_video_ram=,hw_vif_model=,hw_vif_multiqueue_enabled=,hw_vm_mode=,hw_watchdog_action=,img_bdm_v2=,img_bittorrent=,img_block_device_mapping=,img_cache_in_nova=,img_compression_level=,img_config_drive=,img_hide_hypervisor_id=,img_hv_requested_version=,img_hv_type='qemu',img_linked_clone=,img_mappings=,img_owner_id=,img_root_device_name=,img_signature=,img_signature_certificate_uuid=,img_signature_hash_method=,img_signature_key_type=,img_use_agent=,img_version=,os_admin_user=,os_command_line=,os_distro=,os_require_quiesce=,os_secure_boot=,os_skip_agent_inject_files_at_boot=,os_skip_agent_inject_ssh=,os_type=,traits_required=) that are not provided by the compute node supported_instances [[u'i686', u'kvm', u'hvm'], [u'x86_64', u'kvm', u'hvm']] or hypervisor version 2012000 do not match _instance_supported /usr/lib/python2.7/site-packages/nova/scheduler/filters/image_props_filter.py:103 2019-07-23 16:44:34.852 12561 DEBUG nova.scheduler.filters.image_props_filter [req-52638278-51b7-4768-836a-f70d8a8b016a ab573ba3ea014b778193b6922ffffe6d ee1865a76440481cbcff08544c7d580a - default default] Instance contains properties ImageMetaProps(hw_architecture=,hw_auto_disk_config=,hw_boot_menu=,hw_cdrom_bus=,hw_cpu_cores=,hw_cpu_max_cores=,hw_cpu_max_sockets=,hw_cpu_max_threads=,hw_cpu_policy=,hw_cpu_realtime_mask=,hw_cpu_sockets=,hw_cpu_thread_policy=,hw_cpu_threads=,hw_device_id=,hw_disk_bus=,hw_disk_type=,hw_firmware_type=,hw_floppy_bus=,hw_ipxe_boot=,hw_machine_type=,hw_mem_page_size=,hw_numa_cpus=,hw_numa_mem=,hw_numa_nodes=,hw_pointer_model=,hw_qemu_guest_agent=,hw_rescue_bus=,hw_rescue_device=,hw_rng_model=,hw_scsi_model=,hw_serial_port_count=,hw_video_model=,hw_video_ram=,hw_vif_model=,hw_vif_multiqueue_enabled=,hw_vm_mode=,hw_watchdog_action=,img_bdm_v2=,img_bittorrent=,img_block_device_mapping=,img_cache_in_nova=,img_compression_level=,img_config_drive=,img_hide_hypervisor_id=,img_hv_requested_version=,img_hv_type='qemu',img_linked_clone=,img_mappings=,img_owner_id=,img_root_device_name=,img_signature=,img_signature_certificate_uuid=,img_signature_hash_method=,img_signature_key_type=,img_use_agent=,img_version=,os_admin_user=,os_command_line=,os_distro=,os_require_quiesce=,os_secure_boot=,os_skip_agent_inject_files_at_boot=,os_skip_agent_inject_ssh=,os_type=,traits_required=) that are not provided by the compute node supported_instances [[u'i686', u'kvm', u'hvm'], [u'x86_64', u'kvm', u'hvm']] or hypervisor version 2012000 do not match _instance_supported /usr/lib/python2.7/site-packages/nova/scheduler/filters/image_props_filter.py:103 On Tue, Jul 23, 2019 at 4:35 PM Matt Riedemann wrote: > On 7/23/2019 8:57 AM, Massimo Sgaravatto wrote: > > > > When running Ocata I had as property of some images: > > > > hypervisor_type='QEMU' > > > > and this worked > > > > Now in Rocjy: > > > > hypervisor_type='QEMU' --> doesn't work (i.e. all hypervisors are > > excluded by ImagePropertiesFilter) > > hypervisor_type='qemu' --> doesn't work (i.e. all hypervisors are > > excluded by ImagePropertiesFilter) > > hypervisor_type='kvm' --> works > > > > "openstack hypervisor list --long" reports "QEMU" as Hypervisor Type > > for all compute nodes > > Apparently the filter doesn't use the ComputeNode.hypervisor_type field > (which is what you see in the API/CLI output) to compare the img_hv_type > property, it relies on some ComputeNode.supported_instances tuples which > are reported differently by the driver. > > Can you enable debug in the scheduler so we could see this output when > you get the NoValidHost? > > > https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/filters/image_props_filter.py#L97 > > Did you upgrade libvirt/qemu as well when you upgraded these nodes to > Rocky? I wonder if the supported instance hypervisor type reported by > the virt driver is kvm now rather than qemu even though the hypervisor > type reported in the API shows QEMU. > > FWIW this is the virt driver code that reports that supported_instances > information for the compute node that's used by the scheduler filter: > > > https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/driver.py#L5846 > > -- > > Thanks, > > Matt > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pawel.konczalski at everyware.ch Tue Jul 23 14:57:03 2019 From: pawel.konczalski at everyware.ch (Pawel Konczalski) Date: Tue, 23 Jul 2019 16:57:03 +0200 Subject: DuplicateMessageError after restart of a control node Message-ID: <8c01078d-1c9a-3153-1856-cf9b3ef0f3d1@everyware.ch> Hello everybody, after restart of one of the control nodes i see this error message every few seconds int the logs: @timestamp               July 23rd 2019, 16:44:08.927 t  Hostname               ctlX t  Logger               openstack.nova t  Payload               Failed to process message ... skipping it.: DuplicateMessageError: Found duplicate message(f2178cffb5d249f3a1d2df1c0322f18c). Skipping it. 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 361, in _callback 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit self.callback(RabbitMessage(message)) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 251, in __call__ 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit     unique_id = self.msg_id_cache.check_duplicate_message(message) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py", line 122, in check_duplicate_message 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit     raise rpc_common.DuplicateMessageError(msg_id=msg_id) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit DuplicateMessageError: Found duplicate message(f2178cffb5d249f3a1d2df1c0322f18c). Skipping it. 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit t  Pid               82 t  Timestamp               2019-07-23 16:44:08.927 t  _id              AWwfSjEtsndUu9YeotwF t  _index              flog-2019.07.23 #  _score             - t  _type              fluentd t  log_level               ERROR t  programname               nova-scheduler t  python_module               oslo.messaging._drivers.impl_rabbit As soon as the restarted node is taken out of the cluster, the errors disappear. Any idea how to fix this? Thank you Pawel -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5227 bytes Desc: not available URL: From amy at demarco.com Tue Jul 23 15:12:20 2019 From: amy at demarco.com (Amy Marrich) Date: Tue, 23 Jul 2019 10:12:20 -0500 Subject: [Diversity] Grace Hopper Lead Co-Mentor(s) needed In-Reply-To: References: Message-ID: Just wanted to put this out there again as time has run out!. If you're interested in representing OpenStack at the largest Women in Tech conference please let me know by Friday! Thanks, Amy (spotz) On Mon, Jul 15, 2019 at 12:52 PM Amy Marrich wrote: > OpenStack has participated in Open Source Day at the Grace Hopper > Conference for the last 5 years and this year it is in Orlando on October > 3rd. Due to unforeseen circumstances we are one lead mentor short to > participate this year and are looking for someone willing to help lead the > session. The Anita Borg Institute will provide a ticket for the entire > conference, the downside is you or your company will be responsible for > your travel, lodging and food. This year CityNetwork is supplying the VM > infrastructure for us to use. > > The session submitted is as follows: > > Participants will sign up for the necessary accounts needed to contribute > to OpenStack and install Git and Gerrit. They will configure their systems > and learn how to file bugs and review code within the Open Dev > infrastructure. Due to time and resource requirements the groups will be > given a virtual machine running Devstack in which to create their own theme > for the Horizon Dashboard. The group will decide on which humanitarian > effort they wish to support but suggestions will be provided for those > groups that can not decide. The group will also work as a team to locate > the necessary documentation on the OpenStack site if they have not done so > in preparation. > > The selected Project Manager will lead these discussions as well as > making sure efforts are staying on track to meet the deadline of the > demonstrations. > > The Developers in the group will work to locate the necessary files and > directories on the VM in order to complete their task. They will be in > charge of creating/changing the necessary files to create a new theme > utilizing SCSS and restarting any services needed to implement. > > The Graphic Designer if present will create any necessary graphics for > the theme and will either scp them themselves to the virtual machine or > provide to the developer. If there is no designated graphic designer the PM > and devs will divide this duty up. > > After the themes are designed relevant files will be scped back down to > the participants machines and they will be able to commit their code for > review in a sandbox environment for other groups to review. > > If you are interested in participating please let me know by July 20th as > we need to send additional information to the OSD folks on July 21st. > > Any questions please reach out on the lists, IRC, or privately! > > Thanks, > > Amy (spotz) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mriedemos at gmail.com Tue Jul 23 15:14:40 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Tue, 23 Jul 2019 10:14:40 -0500 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> Message-ID: <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> On 7/23/2019 9:50 AM, Massimo Sgaravatto wrote: > > This [*] is what appears in nova-scheduler after having enabled the debug. > > We performed a "yum update" so, yes, we also updated libvirt (now we are > running v. 4.5.0) > > Thanks, Massimo > > [*] > > 2019-07-23 16:44:34.849 12561 DEBUG > nova.scheduler.filters.image_props_filter > [req-52638278-51b7-4768-836a-f70d8a8b016a > ab573ba3ea014b778193b6922ffffe6d ee1865a76440481cbcff08544c7d580a - > default default] Instance contains properties > ImageMetaProps(hw_architecture=,hw_auto_disk_config=,hw_boot_menu=,hw_cdrom_bus=,hw_cpu_cores=,hw_cpu_max_cores=,hw_cpu_max_sockets=,hw_cpu_max_threads=,hw_cpu_policy=,hw_cpu_realtime_mask=,hw_cpu_sockets=,hw_cpu_thread_policy=,hw_cpu_threads=,hw_device_id=,hw_disk_bus=,hw_disk_type=,hw_firmware_type=,hw_floppy_bus=,hw_ipxe_boot=,hw_machine_type=,hw_mem_page_size=,hw_numa_cpus=,hw_numa_mem=,hw_numa_nodes=,hw_pointer_model=,hw_qemu_guest_agent=,hw_rescue_bus=,hw_rescue_device=,hw_rng_model=,hw_scsi_model=,hw_serial_port_count=,hw_video_model=,hw_video_ram=,hw_vif_model=,hw_vif_multiqueue_enabled=,hw_vm_mode=,hw_watchdog_action=,img_bdm_v2=,img_bittorrent=,img_block_device_mapping=,img_cache_in_nova=,img_compression_level=,img_config_drive=,img_hide_hypervisor_id=,img_hv_requested_version=,img_hv_type='qemu',img_linked_clone=,img_mappings=,img_owner_id=,img_root_device_name=,img_signature=,img_signature_certificate_uuid=,img_signature_hash_method=,img_signature_key_type=,img_use_agent=,img_version=,os_admin_user=,os_command_line=,os_distro=,os_require_quiesce=,os_secure_boot=,os_skip_agent_inject_files_at_boot=,os_skip_agent_inject_ssh=,os_type=,traits_required=) > that are not provided by the compute node supported_instances [[u'i686', > u'kvm', u'hvm'], [u'x86_64', u'kvm', u'hvm']] or hypervisor version > 2012000 do not match _instance_supported > /usr/lib/python2.7/site-packages/nova/scheduler/filters/image_props_filter.py:103 > 2019-07-23 16:44:34.852 12561 DEBUG > nova.scheduler.filters.image_props_filter > [req-52638278-51b7-4768-836a-f70d8a8b016a > ab573ba3ea014b778193b6922ffffe6d ee1865a76440481cbcff08544c7d580a - > default default] Instance contains properties > ImageMetaProps(hw_architecture=,hw_auto_disk_config=,hw_boot_menu=,hw_cdrom_bus=,hw_cpu_cores=,hw_cpu_max_cores=,hw_cpu_max_sockets=,hw_cpu_max_threads=,hw_cpu_policy=,hw_cpu_realtime_mask=,hw_cpu_sockets=,hw_cpu_thread_policy=,hw_cpu_threads=,hw_device_id=,hw_disk_bus=,hw_disk_type=,hw_firmware_type=,hw_floppy_bus=,hw_ipxe_boot=,hw_machine_type=,hw_mem_page_size=,hw_numa_cpus=,hw_numa_mem=,hw_numa_nodes=,hw_pointer_model=,hw_qemu_guest_agent=,hw_rescue_bus=,hw_rescue_device=,hw_rng_model=,hw_scsi_model=,hw_serial_port_count=,hw_video_model=,hw_video_ram=,hw_vif_model=,hw_vif_multiqueue_enabled=,hw_vm_mode=,hw_watchdog_action=,img_bdm_v2=,img_bittorrent=,img_block_device_mapping=,img_cache_in_nova=,img_compression_level=,img_config_drive=,img_hide_hypervisor_id=,img_hv_requested_version=,img_hv_type='qemu',img_linked_clone=,img_mappings=,img_owner_id=,img_root_device_name=,img_signature=,img_signature_certificate_uuid=,img_signature_hash_method=,img_signature_key_type=,img_use_agent=,img_version=,os_admin_user=,os_command_line=,os_distro=,os_require_quiesce=,os_secure_boot=,os_skip_agent_inject_files_at_boot=,os_skip_agent_inject_ssh=,os_type=,traits_required=) > that are not provided by the compute node supported_instances [[u'i686', > u'kvm', u'hvm'], [u'x86_64', u'kvm', u'hvm']] or hypervisor version > 2012000 do not match _instance_supported > /usr/lib/python2.7/site-packages/nova/scheduler/filters/image_props_filter.py:103 Yeah at this point I'm not sure what's going on but the driver is reporting kvm now and your image is requesting qemu so that's why the hosts are getting filtered out. I'm not sure why the upgrade of libvirt/qemu would change what the driver is reporting now, but it's a bit lower level than I'd know about off hand. Maybe some of the Red Hat nova devs would know more about this or have seen it before. -- Thanks, Matt From johfulto at redhat.com Tue Jul 23 20:02:45 2019 From: johfulto at redhat.com (John Fulton) Date: Tue, 23 Jul 2019 16:02:45 -0400 Subject: [tripleo] CI scenario001-standalone broken In-Reply-To: <8743b415-9a89-7df2-409b-611d9dfd31a3@redhat.com> References: <8743b415-9a89-7df2-409b-611d9dfd31a3@redhat.com> Message-ID: On Tue, Jul 23, 2019 at 7:14 AM Giulio Fidente wrote: > > On 7/23/19 9:51 AM, Marios Andreou wrote: > > thanks John we have that https://bugs.launchpad.net/tripleo/+bug/1837451 > > and also a temp workaround from ykarel to exclude the problematic > > rpm https://review.opendev.org/#/c/672197/1 > > we also have a fixed build of ceph-ansible (rc12) in centos cbs now, the > plan is to test it via [1] and [2] and if passes promote for -release > > 1. https://review.opendev.org/#/c/501987/ > 2. https://review.opendev.org/#/c/656935/ Any blocking issues related to this issue should now be resolved. The oooq patch to disable ceph-ansible rc11 has merged [1] and a patch [2] previously blocked by this issue has passed tripleo-ci-centos-7-scenario001-standalone and tripleo-ci-centos-7-scenario010-standalone. We'll ensure rc12 works with [3][4] before promoting it to -release. After rc12 is promoted [1] may be removed. John [1] https://review.opendev.org/#/c/672197/ [2] https://review.opendev.org/#/c/670574/ [3] https://review.opendev.org/#/c/501987/ [4] https://review.opendev.org/#/c/656935/ > > -- > Giulio Fidente > GPG KEY: 08D733BA From colleen at gazlene.net Tue Jul 23 23:16:45 2019 From: colleen at gazlene.net (Colleen Murphy) Date: Tue, 23 Jul 2019 16:16:45 -0700 Subject: [keystone] Keystone Train Midcycle Report Message-ID: <3871ef79-1200-4119-8546-fa7b6e5cb87f@www.fastmail.com> This week the keystone team conducted a two-day virtual midcycle. Thanks to everyone who participated and contributed to making this a successful and productive meeting! This email will be a short summary of the key takeaways from the last two days. The full agenda and meeting notes can be found on this etherpad: https://etherpad.openstack.org/p/keystone-train-midcycle-topics # Caching Guide We've long been proposing to document best practices for configuring caching for keystone. We began the meeting with a discussion about what exactly should be contained in such a guide, starting with anecdotes from the team about our own operational experience with configuring memcached for keystone. We quickly realized the guide needed at least two parts, one for keystone itself and one for keystonemiddleware. We also made two technical observations, first that the default value for the memcache_socket_timeout option in oslo.cache could be reduced[1] and second that the memcache_socket_timeout option exposed in keystone is not actually used by keystone and needs to be removed[2]. We talked about security strategies for memcached: oslo.cache exposes encryption capabilities which is required for organizations that consider the data stored in memcached to be "at rest", but it also hurts performance. An alternative strategy that could be recommended in the guide is to set up TLS tunneling to secure in-flight data without impeding fetching performance. The initial caching guide update was proposed here[3]. This guide will be a living document and will continue to grow as we gather real-life data from operators. After this initial update, we'll start taking steps to ensure that keystone requires caching be set up in order to operate. [1] https://review.opendev.org/672063 [2] https://bugs.launchpad.net/keystone/+bug/1837407 [3] https://review.opendev.org/672120 # Community Goals We then had a short status update session on the various community goals we need to meet. The mutable config goal from a few cycles ago is not yet complete but has an implementation proposed[4] which needs more eyes and testing. The PDF generation goal implementation is underway[5] but there are some issues already being tracked by the community[6]. We will need to work to ensure the issues we're seeing are noted and addressed. The last of the Python3 Train job changes for keystone repos were merged, so I think we've completed our end of that goal. We also already have a patch proposed to add the ipv6-only job[7] which is just waiting on the base job to merge in tempest. [4] https://review.opendev.org/585417 [5] https://review.opendev.org/669982 [6] https://etherpad.openstack.org/p/pdf-goal-train-common-problems [7] https://review.opendev.org/671903 # Milestone 2 Train Retrospective We closed out the first day with a mid-cycle retrospective[8]. Some key areas for improvement from this were: * Do more video conferencing - The real-time face-to-face communication is really effective for getting to the heart of discussions and making decisions quickly - Since it is looking likely that most of us will not be at the next PTG in Shanghai, we can set up a pre- and post- PTG video meeting to plan the cycle and catch people up after the IRL PTG - Milestone-ly video meetings are working for helping us keep pace and sync up - We can keep a queue of potential topics and make sure we have a meeting when there are enough topics to discuss - We'll try using video conferencing for office hours - We'll try Jitsi again even though there were some technical issues last time since it is OSS and supports recording * Improve office hours - Office hours still don't feel useful and haven't happened regularly, but we still think they have potential - We'll use the weekly newsletter to commit to the topic the week before, which gives the team a better opportunity to prepare for it and can also attract interest from people outside the keystone team - Use video conferencing as mentioned above * Improve the weekly newsletter - We'll use the weekly newsletter to call out action items from the last meeting - In addition to providing statistics about the number of open reviews, we'll highlight certain reviews that need attention based on various criteria such as age, priority with regard to cycle commitments, community goals, and specific review requests. - We'll announce the following week's office hours topic in the newsletter [8] https://etherpad.openstack.org/p/keystone-train-midcycle-retrospective # System Scope and Default Roles The second day of the midcycle was used to go through our list of open system scope[9] and default roles[10] bugs and work through what scopes and roles each API should support. The notes in the etherpad largely capture every decision fairly well, but a few issues worth highlighting are: * Implied roles[11]: the regular roles API is currently system-scope only, even though it's possible to have domain-specific roles, because for the most part for a role to be useful the role's creator must also have the ability to configure policy, which domain admins do not. Implied roles still require access to policy configuration except in the case where you are only using implied roles for aliasing, in which case it may make sense for a domain admin to have the ability to create an implied role relationship. We decided to start with making the implied roles API system-scope only for now and to open a separate bug to discuss how to deal with domain-specific roles and implied roles. * Revocation list[12]: we're currently not enforcing policy on this API at all because it can only ever return a 410 Gone or a 403 Forbidden, never a success, so we will just deprecate the policy entirely rather than updating it to work with system scope or default roles. We reevaluated the priority on some of the bugs in the list. However, we noted that while some APIs are more heavily used than others, we can't change enforce_scope to True until all of them are completed - picking and choosing the most important ones and letting the others slide is not really an option. [9] https://bugs.launchpad.net/keystone/+bugs?field.tag=system-scope [10] https://bugs.launchpad.net/keystone/+bugs?field.tag=default-roles [11] https://bugs.launchpad.net/keystone/+bug/1805371 [12] https://bugs.launchpad.net/keystone/+bug/1818845 If I've left out any important details, don't hesitate to add them to this thread. Thanks again to everyone who was able to join the meeting! Thanks also to Julia Kreger for her advice on how to hold one of these virtual midcycles, it proved to be very useful and this first-timer really appreciated it :) Colleen From missile0407 at gmail.com Tue Jul 23 23:28:37 2019 From: missile0407 at gmail.com (Eddie Yen) Date: Wed, 24 Jul 2019 07:28:37 +0800 Subject: [Kolla][Keystone] No "Domains" found in identity area. In-Reply-To: References: Message-ID: Update: I tried the few steps from internet. Putting the policy.json with correct default domain both into Horizon and Keystone. And also modified few settings in local_settings. But I still can't see Identity > Domains in cloud_admin. If I type the URL manually () , it will throw unauthorized error. Reference: https://wiki.openstack.org/wiki/Horizon/DomainWorkFlow What am I miss? Eddie Yen 於 2019年7月18日 週四 上午9:45寫道： > Sorry, it should be Horizon, not Keystone. > > Eddie Yen 於 2019年7月18日 週四 上午9:25寫道： > >> Hi, >> >> I'm using kolla-ansible stable/rocky release to deploy Openstack >> Try to test with multi-domains requirements. I'm do nothing but add >> "horizon_keystone_multidomain: True" into globals.yml >> >> After deployment, it works because I have to enter the domain name at >> login screen. >> Logged in with default domain and admin account. But I cannot found any >> "Domains" section in identity block. >> >> Is there something I miss? >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianyrchoi at gmail.com Tue Jul 23 23:47:48 2019 From: ianyrchoi at gmail.com (Ian Y. Choi) Date: Wed, 24 Jul 2019 08:47:48 +0900 Subject: [I18n] project team status update regarding internationalization/translation as July 2019 Message-ID: <6058a029-9922-7c4a-9f95-a7348f3a7c44@gmail.com> Hello all, I would like to share current project team status for better visibility into I18n team members, copying to openstack-discuss and user-committee thread: 1. Translation and team effort status There are still lots of translation contribution as you see through Stackalytics: https://www.stackalytics.com/?metric=translations&release=train . The number of translated words is not larger than the previous cycle, but I estimate that it will be similar or a little bit less than the previous cycle since I18n usually contributes translations a lot especially during Soft & Hard StringFreeze periods. Note that I18n team currently sets higher priority on OpenStack user survey translation and project document translations. Such priorities information can be found on Zanata dashboard: https://translate.openstack.org/ . Other progress such as migrating Zanata into other translation platform seems slow - me and Frank are planning to have a virtual sprint. 2. Shanghai Summit & PTG preparation : Frank submitted his presentation including me and Hoarce (China team). Let's wait for final approval on presentation submission. : There should be PTG at Shanghai - please add your name and detail thinking to Etherpad mentioned in http://lists.openstack.org/pipermail/openstack-i18n/2019-July/003436.html .   I strongly think to more encourage participation since I expect more participation especially from Chinese translators & contributors. 3. Updating ATC & AUC : Frank previously submitted extra-ATC for Stein through https://review.opendev.org/#/c/633398/ , and this time I would like to submit the stat calculation from 2019-01-26 to 2019-07-25. : All contributors who had commits on openstack/i18n repository will be automatically regarded as Active Technical Contributors [1], and   I18n team would like to nominate all translators who contributed translations on OpenStack official repositories (https://governance.openstack.org/tc/reference/projects/ ) >= 300 words   as extra-ATCs assuming that their Zanata e-mail IDs are registered as Foundation members. : All translators who contributed >=300 word tranlations to the following projects will be nominated as Active User Contributors [2]   assuming that their Zanata e-mail IDs are registered as Foundation members: "contributor-guide, edge-computing, leveraging-containers-openstack, openstack-user-survey, operations-guide".   This is new nomination after the decision during Stein PTG with User Committee: Please see [3] and [4] if you would like to follow-up with more information. I will do this nomination for ATC & AUC as PTL just after July 25, although the due of extra ATC is Aug 26-30 [5] since I synced with Foundation that the price of Summit ticket will increase around Aug 7 : processing within the due will be fine, also I set period exactly for six months. 4. Team status : Although I regularly be online during I18n (+Docs) IRC team meetings, I couldn't find enough quorum, and I couldn't proceed more deep conversation with language teams (usually no reply from my e-mails unfortunately). : Current I18n team core members seem busy but when I communicate with them, all they are still very supportive to I18n team projects. I really appreciate their kind help. : Also, I find that some translators are very active comparing to some inactive language coordinators. In this case, would it be a good idea to nominate them as language coordinators? Please share your thoughts. : However, if such lack of team member conversations keep going, then I18n team might decide during upcoming PTG to make translation from an official team status into Special Interest Group [6].   This would mean that there will be no official PTL election and there will be no extra-ATC nomination, although I need to more figure out in details in future. With many thanks, /Ian [1] https://docs.openstack.org/i18n/latest/official_translator.html#atc-status-in-i18n-project [2]https://governance.openstack.org/uc/reference/charter.html#active-user-contributors-auc [3] http://lists.openstack.org/pipermail/openstack-i18n/2018-September/003316.html [4] http://eavesdrop.openstack.org/meetings/uc/2018/uc.2018-10-01-18.02.html [5] https://releases.openstack.org/train/schedule.html [6] https://governance.openstack.org/sigs/ From melwittt at gmail.com Wed Jul 24 00:11:13 2019 From: melwittt at gmail.com (melanie witt) Date: Tue, 23 Jul 2019 17:11:13 -0700 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> Message-ID: On 7/23/19 8:14 AM, Matt Riedemann wrote: > On 7/23/2019 9:50 AM, Massimo Sgaravatto wrote: >> >> This [*] is what appears in nova-scheduler after having enabled the >> debug. >> >> We performed a "yum update" so, yes, we also updated libvirt (now we >> are running v. 4.5.0) >> >> Thanks, Massimo >> >> [*] >> >> 2019-07-23 16:44:34.849 12561 DEBUG >> nova.scheduler.filters.image_props_filter >> [req-52638278-51b7-4768-836a-f70d8a8b016a >> ab573ba3ea014b778193b6922ffffe6d ee1865a76440481cbcff08544c7d580a - >> default default] Instance contains properties >> ImageMetaProps(hw_architecture=,hw_auto_disk_config=,hw_boot_menu=,hw_cdrom_bus=,hw_cpu_cores=,hw_cpu_max_cores=,hw_cpu_max_sockets=,hw_cpu_max_threads=,hw_cpu_policy=,hw_cpu_realtime_mask=,hw_cpu_sockets=,hw_cpu_thread_policy=,hw_cpu_threads=,hw_device_id=,hw_disk_bus=,hw_disk_type=,hw_firmware_type=,hw_floppy_bus=,hw_ipxe_boot=,hw_machine_type=,hw_mem_page_size=,hw_numa_cpus=,hw_numa_mem=,hw_numa_nodes=,hw_pointer_model=,hw_qemu_guest_agent=,hw_rescue_bus=,hw_rescue_device=,hw_rng_model=,hw_scsi_model=,hw_serial_port_count=,hw_video_model=,hw_video_ram=,hw_vif_model=,hw_vif_multiqueue_enabled=,hw_vm_mode=,hw_watchdog_action=,img_bdm_v2=,img_bittorrent=,img_block_device_mapping=,img_cache_in_nova=,img_compression_level=,img_config_drive=,img_hide_hypervisor_id=,img_hv_requested_version=,img_hv_type='qemu',img_linked_clone=,img_mappings=,img_owner_id=,img_root_device_name=,img_signature=,img_signature_certificate_uuid=,img_signature_hash_method=,img_signature_key_type=,img_use_agent=,img_version=,os_admin_user=,os_command_line=,os_distro=,os_require_quiesce=,os_secure_boot=,os_skip_agent_inject_files_at_boot=,os_skip_agent_inject_ssh=,os_type=,traits_required=) >> that are not provided by the compute node supported_instances >> [[u'i686', u'kvm', u'hvm'], [u'x86_64', u'kvm', u'hvm']] or hypervisor >> version 2012000 do not match _instance_supported >> /usr/lib/python2.7/site-packages/nova/scheduler/filters/image_props_filter.py:103 >> >> 2019-07-23 16:44:34.852 12561 DEBUG >> nova.scheduler.filters.image_props_filter >> [req-52638278-51b7-4768-836a-f70d8a8b016a >> ab573ba3ea014b778193b6922ffffe6d ee1865a76440481cbcff08544c7d580a - >> default default] Instance contains properties >> ImageMetaProps(hw_architecture=,hw_auto_disk_config=,hw_boot_menu=,hw_cdrom_bus=,hw_cpu_cores=,hw_cpu_max_cores=,hw_cpu_max_sockets=,hw_cpu_max_threads=,hw_cpu_policy=,hw_cpu_realtime_mask=,hw_cpu_sockets=,hw_cpu_thread_policy=,hw_cpu_threads=,hw_device_id=,hw_disk_bus=,hw_disk_type=,hw_firmware_type=,hw_floppy_bus=,hw_ipxe_boot=,hw_machine_type=,hw_mem_page_size=,hw_numa_cpus=,hw_numa_mem=,hw_numa_nodes=,hw_pointer_model=,hw_qemu_guest_agent=,hw_rescue_bus=,hw_rescue_device=,hw_rng_model=,hw_scsi_model=,hw_serial_port_count=,hw_video_model=,hw_video_ram=,hw_vif_model=,hw_vif_multiqueue_enabled=,hw_vm_mode=,hw_watchdog_action=,img_bdm_v2=,img_bittorrent=,img_block_device_mapping=,img_cache_in_nova=,img_compression_level=,img_config_drive=,img_hide_hypervisor_id=,img_hv_requested_version=,img_hv_type='qemu',img_linked_clone=,img_mappings=,img_owner_id=,img_root_device_name=,img_signature=,img_signature_certificate_uuid=,img_signature_hash_method=,img_signature_key_type=,img_use_agent=,img_version=,os_admin_user=,os_command_line=,os_distro=,os_require_quiesce=,os_secure_boot=,os_skip_agent_inject_files_at_boot=,os_skip_agent_inject_ssh=,os_type=,traits_required=) >> that are not provided by the compute node supported_instances >> [[u'i686', u'kvm', u'hvm'], [u'x86_64', u'kvm', u'hvm']] or hypervisor >> version 2012000 do not match _instance_supported >> /usr/lib/python2.7/site-packages/nova/scheduler/filters/image_props_filter.py:103 >> > > Yeah at this point I'm not sure what's going on but the driver is > reporting kvm now and your image is requesting qemu so that's why the > hosts are getting filtered out. I'm not sure why the upgrade of > libvirt/qemu would change what the driver is reporting now, but it's a > bit lower level than I'd know about off hand. Maybe some of the Red Hat > nova devs would know more about this or have seen it before. I'm not sure whether this is related, but this thread reminded me of a change that landed in Rocky where we started filtering hypervisor capabilities by the configured CONF.libvirt.virt_type: https://review.opendev.org/531347 I didn't see mention so far of how CONF.libvirt.virt_type has been configured in this deployment. Is it set to 'kvm' or 'qemu'? If it's set to 'kvm', that would cause 'qemu' capabilities to be filtered out, when they would not have been prior to Rocky. Apologies if this was an unrelated tangent. Cheers, -melanie From massimo.sgaravatto at gmail.com Wed Jul 24 06:37:03 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Wed, 24 Jul 2019 08:37:03 +0200 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> Message-ID: Melanie: I think this is indeed the problem ! But then, if I am not wrong, the note in: https://docs.openstack.org/nova/rocky/admin/configuration/schedulers.html Note qemu is used for both QEMU and KVM hypervisor types. should be removed. I can open a bug if you agree ... And maybe this is something worth to be mentioned in the release notes ? Thanks again for your help ! Cheers, Massimo On Wed, Jul 24, 2019 at 2:11 AM melanie witt wrote: > On 7/23/19 8:14 AM, Matt Riedemann wrote: > > On 7/23/2019 9:50 AM, Massimo Sgaravatto wrote: > >> > >> This [*] is what appears in nova-scheduler after having enabled the > >> debug. > >> > >> We performed a "yum update" so, yes, we also updated libvirt (now we > >> are running v. 4.5.0) > >> > >> Thanks, Massimo > >> > >> [*] > >> > >> 2019-07-23 16:44:34.849 12561 DEBUG > >> nova.scheduler.filters.image_props_filter > >> [req-52638278-51b7-4768-836a-f70d8a8b016a > >> ab573ba3ea014b778193b6922ffffe6d ee1865a76440481cbcff08544c7d580a - > >> default default] Instance contains properties > >> > ImageMetaProps(hw_architecture=,hw_auto_disk_config=,hw_boot_menu=,hw_cdrom_bus=,hw_cpu_cores=,hw_cpu_max_cores=,hw_cpu_max_sockets=,hw_cpu_max_threads=,hw_cpu_policy=,hw_cpu_realtime_mask=,hw_cpu_sockets=,hw_cpu_thread_policy=,hw_cpu_threads=,hw_device_id=,hw_disk_bus=,hw_disk_type=,hw_firmware_type=,hw_floppy_bus=,hw_ipxe_boot=,hw_machine_type=,hw_mem_page_size=,hw_numa_cpus=,hw_numa_mem=,hw_numa_nodes=,hw_pointer_model=,hw_qemu_guest_agent=,hw_rescue_bus=,hw_rescue_device=,hw_rng_model=,hw_scsi_model=,hw_serial_port_count=,hw_video_model=,hw_video_ram=,hw_vif_model=,hw_vif_multiqueue_enabled=,hw_vm_mode=,hw_watchdog_action=,img_bdm_v2=,img_bittorrent=,img_block_device_mapping=,img_cache_in_nova=,img_compression_level=,img_config_drive=,img_hide_hypervisor_id=,img_hv_requested_version=,img_hv_type='qemu',img_linked_clone=,img_mappings=,img_owner_id=,img_root_device_name=,img_signature=,img_signature_certificate_uuid=,img_signature_hash_method=,img_signature_key_type=,img_use_agent=,img_version=,os_admin_user=,os_command_line=,os_distro=,os_require_quiesce=,os_secure_boot=,os_skip_agent_inject_files_at_boot=,os_skip_agent_inject_ssh=,os_type=,traits_required=) > > >> that are not provided by the compute node supported_instances > >> [[u'i686', u'kvm', u'hvm'], [u'x86_64', u'kvm', u'hvm']] or hypervisor > >> version 2012000 do not match _instance_supported > >> > /usr/lib/python2.7/site-packages/nova/scheduler/filters/image_props_filter.py:103 > > >> > >> 2019-07-23 16:44:34.852 12561 DEBUG > >> nova.scheduler.filters.image_props_filter > >> [req-52638278-51b7-4768-836a-f70d8a8b016a > >> ab573ba3ea014b778193b6922ffffe6d ee1865a76440481cbcff08544c7d580a - > >> default default] Instance contains properties > >> > ImageMetaProps(hw_architecture=,hw_auto_disk_config=,hw_boot_menu=,hw_cdrom_bus=,hw_cpu_cores=,hw_cpu_max_cores=,hw_cpu_max_sockets=,hw_cpu_max_threads=,hw_cpu_policy=,hw_cpu_realtime_mask=,hw_cpu_sockets=,hw_cpu_thread_policy=,hw_cpu_threads=,hw_device_id=,hw_disk_bus=,hw_disk_type=,hw_firmware_type=,hw_floppy_bus=,hw_ipxe_boot=,hw_machine_type=,hw_mem_page_size=,hw_numa_cpus=,hw_numa_mem=,hw_numa_nodes=,hw_pointer_model=,hw_qemu_guest_agent=,hw_rescue_bus=,hw_rescue_device=,hw_rng_model=,hw_scsi_model=,hw_serial_port_count=,hw_video_model=,hw_video_ram=,hw_vif_model=,hw_vif_multiqueue_enabled=,hw_vm_mode=,hw_watchdog_action=,img_bdm_v2=,img_bittorrent=,img_block_device_mapping=,img_cache_in_nova=,img_compression_level=,img_config_drive=,img_hide_hypervisor_id=,img_hv_requested_version=,img_hv_type='qemu',img_linked_clone=,img_mappings=,img_owner_id=,img_root_device_name=,img_signature=,img_signature_certificate_uuid=,img_signature_hash_method=,img_signature_key_type=,img_use_agent=,img_version=,os_admin_user=,os_command_line=,os_distro=,os_require_quiesce=,os_secure_boot=,os_skip_agent_inject_files_at_boot=,os_skip_agent_inject_ssh=,os_type=,traits_required=) > > >> that are not provided by the compute node supported_instances > >> [[u'i686', u'kvm', u'hvm'], [u'x86_64', u'kvm', u'hvm']] or hypervisor > >> version 2012000 do not match _instance_supported > >> > /usr/lib/python2.7/site-packages/nova/scheduler/filters/image_props_filter.py:103 > > >> > > > > Yeah at this point I'm not sure what's going on but the driver is > > reporting kvm now and your image is requesting qemu so that's why the > > hosts are getting filtered out. I'm not sure why the upgrade of > > libvirt/qemu would change what the driver is reporting now, but it's a > > bit lower level than I'd know about off hand. Maybe some of the Red Hat > > nova devs would know more about this or have seen it before. > > I'm not sure whether this is related, but this thread reminded me of a > change that landed in Rocky where we started filtering hypervisor > capabilities by the configured CONF.libvirt.virt_type: > > https://review.opendev.org/531347 > > I didn't see mention so far of how CONF.libvirt.virt_type has been > configured in this deployment. Is it set to 'kvm' or 'qemu'? If it's set > to 'kvm', that would cause 'qemu' capabilities to be filtered out, when > they would not have been prior to Rocky. > > Apologies if this was an unrelated tangent. > > Cheers, > -melanie > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From renat.akhmerov at gmail.com Wed Jul 24 07:55:12 2019 From: renat.akhmerov at gmail.com (Renat Akhmerov) Date: Wed, 24 Jul 2019 14:55:12 +0700 Subject: [mistral] Office hour in ~5 min In-Reply-To: References: Message-ID: Hi, We’ll have an office hour in about 5 mins. Join us at #openstack-mistral if you have anything to discuss. Thanks Renat Akhmerov @Nokia -------------- next part -------------- An HTML attachment was scrubbed... URL: From berndbausch at gmail.com Wed Jul 24 07:57:54 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Wed, 24 Jul 2019 16:57:54 +0900 Subject: [devstack] Devstack fails with requirement pyparsing===2.4.1 Message-ID: <006b96bd-a7e2-84e1-9271-3eacac82b73e@gmail.com> Trying to install Devstack Stein on freshly installed Ubuntu 18.04. stack.sh fails with this: Downloading https://files.pythonhosted.org/packages/6f/7c/ebba93a2d3e09fd0220e1adfdb8230142f4e21d1527e5b06aa353dc27719/oslo.config-6.11.0-py2.py3-none-any.whl (131kB) Collecting pyparsing===2.4.1 (from -c https://releases.openstack.org/constraints/upper/master (line 492))   Could not find a version that satisfies the requirement pyparsing===2.4.1 (from -c https://releases.openstack.org/constraints/upper/master (line 492)) (from versions: 1.4.6, 1.4.7, 1.4.8, 1.4.11, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.7, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.1.7, 2.1.8, 2.1.9, 2.1.10, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0) No matching distribution found for *pyparsing===2.4.1* (from -c https://releases.openstack.org/constraints/upper/master (line 492)) (red highlight is mine) The most recent version of pyparsing is 2.4.0. I would file a bug, but I don't know against which project. Who owns https://releases.openstack.org/constraints/upper/master? Or is there a misunderstanding somewhere? My local.conf is below - it's based on a Rocky local.conf (which does work on Rocky) and might be incorrect. Bernd. [[local|localrc]] ########################################################################### ### Passwords ADMIN_PASSWORD=MYPASSWORD DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD ########################################################################### ### Logging LOG_COLOR=False LOGFILE=$DEST/logs/stack.sh.log ########################################################################### ### Networking IP_VERSION=4 HOST_IP=192.168.1.200 ########################################################################### ### Software sources # use https since the vlabs proxy seems to block the git:// scheme GIT_BASE=https://git.openstack.org # Ensure the latest software is installed PIP_UPGRADE=True ########################################################################### ### Glance DOWNLOAD_DEFAULT_IMAGES=False IMAGE_URLS="http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" # Glance's image import doesn't work when Glance runs as a UWSGI server # For this setting to take effect, devstack/lib/glance must be patched GLANCE_USE_UWSGI=False ########################################################################### ### Swift # by default, Devstack interactively asks for a hash value. Avoid this. SWIFT_HASH=123 enable_service swift ########################################################################### ### Heat enable_plugin heat https://git.openstack.org/openstack/heat stable/stein enable_service h-eng h-api h-api-cfn h-api-cw enable_plugin heat-dashboard https://git.openstack.org/openstack/heat-dashboard stable/stein enable_service heat-dashboard ########################################################################### ### Cinder: Enable backup enable_service c-bak ########################################################################### ### Telemetry: Use Gnocchi as sample repository ###            and configure medium archive policy CEILOMETER_BACKEND=gnocchi GNOCCHI_ARCHIVE_POLICY=medium enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer stable/stein enable_plugin aodh https://git.openstack.org/openstack/aodh stable/stein ########################################################################### ### Neutron: Enable internal DNS resolution in ml2_conf.ini Q_ML2_PLUGIN_EXT_DRIVERS=port_security,dns_domain_ports ########################################################################### ### Post config for Neutron (DNS) and Nova (allow KVM in Vmware machine) [[post-config|$NEUTRON_CONF]] [DEFAULT] dns_domain = devstack.org. [[post-config|/etc/nova/nova-cpu.conf]] [libvirt] # set machine type to work around bug # https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1691109 hw_machine_type = x86_64=pc-i440fx-xenial -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Wed Jul 24 08:09:40 2019 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 24 Jul 2019 10:09:40 +0200 Subject: [devstack] Devstack fails with requirement pyparsing===2.4.1 In-Reply-To: <006b96bd-a7e2-84e1-9271-3eacac82b73e@gmail.com> References: <006b96bd-a7e2-84e1-9271-3eacac82b73e@gmail.com> Message-ID: Hi, The same happened with me this morning. The interesting thing is (at least for me) that by the github page of pyparsing they have 2.4.1 release, at least they added the tag to the code: https://github.com/pyparsing/pyparsing/tags I checked some devstack runs in the gate, and it seems that the jobs (at that time at least) could fetch pyparsing===2.4.1: http://logs.openstack.org/53/671853/3/check/tempest-slow-py3/76fa978/controller/logs/devstacklog.txt.gz#_2019-07-23_18_18_58_513 Regards Lajos Bernd Bausch ezt írta (időpont: 2019. júl. 24., Sze, 10:02): > Trying to install Devstack Stein on freshly installed Ubuntu 18.04. > > stack.sh fails with this: > > Downloading > https://files.pythonhosted.org/packages/6f/7c/ebba93a2d3e09fd0220e1adfdb8230142f4e21d1527e5b06aa353dc27719/oslo.config-6.11.0-py2.py3-none-any.whl > (131kB) > Collecting pyparsing===2.4.1 (from -c > https://releases.openstack.org/constraints/upper/master (line 492)) > Could not find a version that satisfies the requirement > pyparsing===2.4.1 (from -c > https://releases.openstack.org/constraints/upper/master (line 492)) (from > versions: 1.4.6, 1.4.7, 1.4.8, 1.4.11, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, > 1.5.5, 1.5.6, 1.5.7, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, > 2.0.7, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.1.7, 2.1.8, > 2.1.9, 2.1.10, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0) > No matching distribution found for *pyparsing===2.4.1* (from -c > https://releases.openstack.org/constraints/upper/master (line 492)) > > (red highlight is mine) > > The most recent version of pyparsing is > 2.4.0. > > I would file a bug, but I don't know against which project. Who owns > https://releases.openstack.org/constraints/upper/master? Or is there a > misunderstanding somewhere? > > My local.conf is below - it's based on a Rocky local.conf (which does work > on Rocky) and might be incorrect. > > Bernd. > > [[local|localrc]] > > ########################################################################### > ### Passwords > > ADMIN_PASSWORD=MYPASSWORD > DATABASE_PASSWORD=$ADMIN_PASSWORD > RABBIT_PASSWORD=$ADMIN_PASSWORD > SERVICE_PASSWORD=$ADMIN_PASSWORD > > ########################################################################### > ### Logging > > LOG_COLOR=False > LOGFILE=$DEST/logs/stack.sh.log > > ########################################################################### > ### Networking > > IP_VERSION=4 > HOST_IP=192.168.1.200 > > ########################################################################### > ### Software sources > > # use https since the vlabs proxy seems to block the git:// scheme > GIT_BASE=https://git.openstack.org > # Ensure the latest software is installed > PIP_UPGRADE=True > > ########################################################################### > ### Glance > > DOWNLOAD_DEFAULT_IMAGES=False > IMAGE_URLS= > "http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" > > > # Glance's image import doesn't work when Glance runs as a UWSGI server > # For this setting to take effect, devstack/lib/glance must be patched > GLANCE_USE_UWSGI=False > > ########################################################################### > ### Swift > > # by default, Devstack interactively asks for a hash value. Avoid this. > SWIFT_HASH=123 > > enable_service swift > > ########################################################################### > ### Heat > > enable_plugin heat https://git.openstack.org/openstack/heat stable/stein > enable_service h-eng h-api h-api-cfn h-api-cw > > enable_plugin heat-dashboard > https://git.openstack.org/openstack/heat-dashboard stable/stein > enable_service heat-dashboard > > ########################################################################### > ### Cinder: Enable backup > > enable_service c-bak > > ########################################################################### > ### Telemetry: Use Gnocchi as sample repository > ### and configure medium archive policy > > CEILOMETER_BACKEND=gnocchi > GNOCCHI_ARCHIVE_POLICY=medium > enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer > stable/stein > enable_plugin aodh https://git.openstack.org/openstack/aodh stable/stein > > ########################################################################### > ### Neutron: Enable internal DNS resolution in ml2_conf.ini > > Q_ML2_PLUGIN_EXT_DRIVERS=port_security,dns_domain_ports > > ########################################################################### > ### Post config for Neutron (DNS) and Nova (allow KVM in Vmware machine) > > [[post-config|$NEUTRON_CONF]] > [DEFAULT] > dns_domain = devstack.org. > > [[post-config|/etc/nova/nova-cpu.conf]] > [libvirt] > # set machine type to work around bug > # https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1691109 > hw_machine_type = x86_64=pc-i440fx-xenial > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Wed Jul 24 08:21:54 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Wed, 24 Jul 2019 10:21:54 +0200 Subject: [devstack] Devstack fails with requirement pyparsing===2.4.1 In-Reply-To: References: <006b96bd-a7e2-84e1-9271-3eacac82b73e@gmail.com> Message-ID: <7671247F-0582-4ACC-94DF-C7BAF653085E@redhat.com> Hi, There is patch https://review.opendev.org/#/c/672395/ already to fix this :) > On 24 Jul 2019, at 10:09, Lajos Katona wrote: > > Hi, > > The same happened with me this morning. > The interesting thing is (at least for me) that by the github page of pyparsing they have 2.4.1 release, at least they added the tag to the code: > https://github.com/pyparsing/pyparsing/tags > > I checked some devstack runs in the gate, and it seems that the jobs (at that time at least) could fetch pyparsing===2.4.1: > http://logs.openstack.org/53/671853/3/check/tempest-slow-py3/76fa978/controller/logs/devstacklog.txt.gz#_2019-07-23_18_18_58_513 > > Regards > Lajos > > Bernd Bausch ezt írta (időpont: 2019. júl. 24., Sze, 10:02): > Trying to install Devstack Stein on freshly installed Ubuntu 18.04. > > stack.sh fails with this: > > Downloading https://files.pythonhosted.org/packages/6f/7c/ebba93a2d3e09fd0220e1adfdb8230142f4e21d1527e5b06aa353dc27719/oslo.config-6.11.0-py2.py3-none-any.whl (131kB) > Collecting pyparsing===2.4.1 (from -c https://releases.openstack.org/constraints/upper/master (line 492)) > Could not find a version that satisfies the requirement pyparsing===2.4.1 (from -c https://releases.openstack.org/constraints/upper/master (line 492)) (from versions: 1.4.6, 1.4.7, 1.4.8, 1.4.11, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.7, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.1.7, 2.1.8, 2.1.9, 2.1.10, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0) > No matching distribution found for pyparsing===2.4.1 (from -c https://releases.openstack.org/constraints/upper/master (line 492)) > > (red highlight is mine) > > The most recent version of pyparsing is 2.4.0. > > I would file a bug, but I don't know against which project. Who owns https://releases.openstack.org/constraints/upper/master? Or is there a misunderstanding somewhere? > > My local.conf is below - it's based on a Rocky local.conf (which does work on Rocky) and might be incorrect. > > Bernd. > > [[local|localrc]] > > ########################################################################### > ### Passwords > > ADMIN_PASSWORD=MYPASSWORD > DATABASE_PASSWORD=$ADMIN_PASSWORD > RABBIT_PASSWORD=$ADMIN_PASSWORD > SERVICE_PASSWORD=$ADMIN_PASSWORD > > ########################################################################### > ### Logging > > LOG_COLOR=False > LOGFILE=$DEST/logs/stack.sh.log > > ########################################################################### > ### Networking > > IP_VERSION=4 > HOST_IP=192.168.1.200 > > ########################################################################### > ### Software sources > > # use https since the vlabs proxy seems to block the git:// scheme > GIT_BASE=https://git.openstack.org > # Ensure the latest software is installed > PIP_UPGRADE=True > > ########################################################################### > ### Glance > > DOWNLOAD_DEFAULT_IMAGES=False > IMAGE_URLS="http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" > > # Glance's image import doesn't work when Glance runs as a UWSGI server > # For this setting to take effect, devstack/lib/glance must be patched > GLANCE_USE_UWSGI=False > > ########################################################################### > ### Swift > > # by default, Devstack interactively asks for a hash value. Avoid this. > SWIFT_HASH=123 > > enable_service swift > > ########################################################################### > ### Heat > > enable_plugin heat https://git.openstack.org/openstack/heat stable/stein > enable_service h-eng h-api h-api-cfn h-api-cw > > enable_plugin heat-dashboard https://git.openstack.org/openstack/heat-dashboard stable/stein > enable_service heat-dashboard > > ########################################################################### > ### Cinder: Enable backup > > enable_service c-bak > > ########################################################################### > ### Telemetry: Use Gnocchi as sample repository > ### and configure medium archive policy > > CEILOMETER_BACKEND=gnocchi > GNOCCHI_ARCHIVE_POLICY=medium > enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer stable/stein > enable_plugin aodh https://git.openstack.org/openstack/aodh stable/stein > > ########################################################################### > ### Neutron: Enable internal DNS resolution in ml2_conf.ini > > Q_ML2_PLUGIN_EXT_DRIVERS=port_security,dns_domain_ports > > ########################################################################### > ### Post config for Neutron (DNS) and Nova (allow KVM in Vmware machine) > > [[post-config|$NEUTRON_CONF]] > [DEFAULT] > dns_domain = devstack.org. > > [[post-config|/etc/nova/nova-cpu.conf]] > [libvirt] > # set machine type to work around bug > # https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1691109 > hw_machine_type = x86_64=pc-i440fx-xenial > > > — Slawek Kaplonski Senior software engineer Red Hat From marios at redhat.com Wed Jul 24 08:28:32 2019 From: marios at redhat.com (Marios Andreou) Date: Wed, 24 Jul 2019 11:28:32 +0300 Subject: [devstack] Devstack fails with requirement pyparsing===2.4.1 In-Reply-To: <7671247F-0582-4ACC-94DF-C7BAF653085E@redhat.com> References: <006b96bd-a7e2-84e1-9271-3eacac82b73e@gmail.com> <7671247F-0582-4ACC-94DF-C7BAF653085E@redhat.com> Message-ID: yeah fyi we have a TripleO bug at https://bugs.launchpad.net/tripleo/+bug/1837697 with pointers and apart from the fix Slawek points to that one also https://review.opendev.org/#/c/672388/2 (thanks to ykarel for pointing to it) for the failing job on the other fix On Wed, Jul 24, 2019 at 11:24 AM Slawek Kaplonski wrote: > Hi, > > There is patch https://review.opendev.org/#/c/672395/ already to fix this > :) > > > On 24 Jul 2019, at 10:09, Lajos Katona wrote: > > > > Hi, > > > > The same happened with me this morning. > > The interesting thing is (at least for me) that by the github page of > pyparsing they have 2.4.1 release, at least they added the tag to the code: > > https://github.com/pyparsing/pyparsing/tags > > > > I checked some devstack runs in the gate, and it seems that the jobs (at > that time at least) could fetch pyparsing===2.4.1: > > > http://logs.openstack.org/53/671853/3/check/tempest-slow-py3/76fa978/controller/logs/devstacklog.txt.gz#_2019-07-23_18_18_58_513 > > > > Regards > > Lajos > > > > Bernd Bausch ezt írta (időpont: 2019. júl. 24., > Sze, 10:02): > > Trying to install Devstack Stein on freshly installed Ubuntu 18.04. > > > > stack.sh fails with this: > > > > Downloading > https://files.pythonhosted.org/packages/6f/7c/ebba93a2d3e09fd0220e1adfdb8230142f4e21d1527e5b06aa353dc27719/oslo.config-6.11.0-py2.py3-none-any.whl > (131kB) > > Collecting pyparsing===2.4.1 (from -c > https://releases.openstack.org/constraints/upper/master (line 492)) > > Could not find a version that satisfies the requirement > pyparsing===2.4.1 (from -c > https://releases.openstack.org/constraints/upper/master (line 492)) (from > versions: 1.4.6, 1.4.7, 1.4.8, 1.4.11, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, > 1.5.5, 1.5.6, 1.5.7, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, > 2.0.7, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.1.7, 2.1.8, > 2.1.9, 2.1.10, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0) > > No matching distribution found for pyparsing===2.4.1 (from -c > https://releases.openstack.org/constraints/upper/master (line 492)) > > > > (red highlight is mine) > > > > The most recent version of pyparsing is 2.4.0. > > > > I would file a bug, but I don't know against which project. Who owns > https://releases.openstack.org/constraints/upper/master? Or is there a > misunderstanding somewhere? > > > > My local.conf is below - it's based on a Rocky local.conf (which does > work on Rocky) and might be incorrect. > > > > Bernd. > > > > [[local|localrc]] > > > > > ########################################################################### > > ### Passwords > > > > ADMIN_PASSWORD=MYPASSWORD > > DATABASE_PASSWORD=$ADMIN_PASSWORD > > RABBIT_PASSWORD=$ADMIN_PASSWORD > > SERVICE_PASSWORD=$ADMIN_PASSWORD > > > > > ########################################################################### > > ### Logging > > > > LOG_COLOR=False > > LOGFILE=$DEST/logs/stack.sh.log > > > > > ########################################################################### > > ### Networking > > > > IP_VERSION=4 > > HOST_IP=192.168.1.200 > > > > > ########################################################################### > > ### Software sources > > > > # use https since the vlabs proxy seems to block the git:// scheme > > GIT_BASE=https://git.openstack.org > > # Ensure the latest software is installed > > PIP_UPGRADE=True > > > > > ########################################################################### > > ### Glance > > > > DOWNLOAD_DEFAULT_IMAGES=False > > IMAGE_URLS=" > http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" > > > > # Glance's image import doesn't work when Glance runs as a UWSGI server > > # For this setting to take effect, devstack/lib/glance must be patched > > GLANCE_USE_UWSGI=False > > > > > ########################################################################### > > ### Swift > > > > # by default, Devstack interactively asks for a hash value. Avoid this. > > SWIFT_HASH=123 > > > > enable_service swift > > > > > ########################################################################### > > ### Heat > > > > enable_plugin heat https://git.openstack.org/openstack/heat stable/stein > > enable_service h-eng h-api h-api-cfn h-api-cw > > > > enable_plugin heat-dashboard > https://git.openstack.org/openstack/heat-dashboard stable/stein > > enable_service heat-dashboard > > > > > ########################################################################### > > ### Cinder: Enable backup > > > > enable_service c-bak > > > > > ########################################################################### > > ### Telemetry: Use Gnocchi as sample repository > > ### and configure medium archive policy > > > > CEILOMETER_BACKEND=gnocchi > > GNOCCHI_ARCHIVE_POLICY=medium > > enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer > stable/stein > > enable_plugin aodh https://git.openstack.org/openstack/aodh stable/stein > > > > > ########################################################################### > > ### Neutron: Enable internal DNS resolution in ml2_conf.ini > > > > Q_ML2_PLUGIN_EXT_DRIVERS=port_security,dns_domain_ports > > > > > ########################################################################### > > ### Post config for Neutron (DNS) and Nova (allow KVM in Vmware machine) > > > > [[post-config|$NEUTRON_CONF]] > > [DEFAULT] > > dns_domain = devstack.org. > > > > [[post-config|/etc/nova/nova-cpu.conf]] > > [libvirt] > > # set machine type to work around bug > > # https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1691109 > > hw_machine_type = x86_64=pc-i440fx-xenial > > > > > > > > — > Slawek Kaplonski > Senior software engineer > Red Hat > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ltoscano at redhat.com Wed Jul 24 08:27:51 2019 From: ltoscano at redhat.com (Luigi Toscano) Date: Wed, 24 Jul 2019 10:27:51 +0200 Subject: [devstack] Devstack fails with requirement pyparsing===2.4.1 In-Reply-To: References: <006b96bd-a7e2-84e1-9271-3eacac82b73e@gmail.com> Message-ID: <223447555.7ip8EmszUT@whitebase.usersys.redhat.com> On Wednesday, 24 July 2019 10:09:40 CEST Lajos Katona wrote: > Hi, > > The same happened with me this morning. > The interesting thing is (at least for me) that by the github page of > pyparsing they have 2.4.1 release, at least they added the tag to the code: > https://github.com/pyparsing/pyparsing/tags > > I checked some devstack runs in the gate, and it seems that the jobs (at > that time at least) could fetch pyparsing===2.4.1: > http://logs.openstack.org/53/671853/3/check/tempest-slow-py3/76fa978/control > ler/logs/devstacklog.txt.gz#_2019-07-23_18_18_58_513 > Yes, until it was removed: https://github.com/pyparsing/pyparsing/issues/105 -- Luigi From berndbausch at gmail.com Wed Jul 24 11:20:24 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Wed, 24 Jul 2019 20:20:24 +0900 Subject: [opendev] git clone fails: "The TLS connection was non-properly terminated" In-Reply-To: <20190723005342.zr6h5tyqdetmu3ss@yuggoth.org> References: <584e116c-05c9-4eac-8bf2-804a098912e1@www.fastmail.com> <20190723005342.zr6h5tyqdetmu3ss@yuggoth.org> Message-ID: Thanks much, gentlemen. I did hope this was a glitch on the server side, and indeed, I successfully cloned Devstack about 12 hours ago. Bernd On 7/23/2019 9:53 AM, Jeremy Stanley wrote: > On 2019-07-22 17:17:08 -0700 (-0700), Clark Boylan wrote: >> On Mon, Jul 22, 2019, at 5:09 PM, Bernd Bausch wrote: >>> There seems to be something wrong with the opendev server. Is this a >>> glitch, a daily occurrence (backup?), or a deeper problem? >>> >>> This happens around midnight between July 22 and 23 UTC. >>> >>> $ git clone https://opendev.org/openstack/devstack.git -b stable/stein >>> Cloning into 'devstack'... >>> fatal: unable to access 'https://opendev.org/openstack/devstack.git/': >>> gnutls_handshake() failed: The TLS connection was non-properly >>> terminated. > [...] >> I think this may have happened because because one of the backend >> servers was removed from the load balancer rotation. And while we >> thought we had removed that server from the haproxy config [0] >> that didn't happen because our ansible doesn't know how to >> gracefully restart haproxy in a container. At this point haproxy >> should be aware that the server is gone and it will prevent any >> new connections from going to it. > [...] > > Yep, sorry, we commented gitea01 out of the load balancer and I > subsequently deleted its server instance at 2019-07-22 23:25 UTC, > but I neglected to double-check that haproxy had actually stopped > distributing connections to it. That one's on me, seems we still > have some work to do making sure haproxy config changes are applied > automatically there. From mtreinish at kortar.org Wed Jul 24 13:11:36 2019 From: mtreinish at kortar.org (Matthew Treinish) Date: Wed, 24 Jul 2019 09:11:36 -0400 Subject: stestr Python 2 Support Message-ID: <20190724131136.GA11582@sinanju.localdomain> Hi Everyone, I just wanted to send a quick update about the state of python 2 support in stestr, since OpenStack is the largest user of the project. With the recent release of stestr 2.4.0 we've officially deprecated the python 2.7 support in stestr. It will emit a DeprecationWarning whenever it's run from the CLI with python 2.7 now. The plan (which is not set in stone) is that we will be pushing a 3.0.0 release that removes the python 2 support and compat code from stestr sometime in early 2020 (definitely after the Python 2.7 EoL on Jan. 1st). I don't believe this conflicts with the current Python version support plans in OpenStack [1] but just wanted to make sure people were aware so that there are no surprises when stestr stops working with Python 2 in 3.0.0. Thanks, -Matt Treinish [1] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From doug at doughellmann.com Wed Jul 24 13:31:57 2019 From: doug at doughellmann.com (Doug Hellmann) Date: Wed, 24 Jul 2019 09:31:57 -0400 Subject: [requirements] changes in requests library maintainership Message-ID: It looks like the Kenneth Reitz, the primary maintainer of the requests library, is stepping down [1]. There are several offers to take over right now, so it's not really clear to me what's going to happen. I think we'll want to keep an eye on both requests and requests3 until the new maintainer is designated. And of course if anyone is interested in helping out, I encourage you to get involved in the discussion on github. [1] https://github.com/not-kennethreitz/team/issues/21 -- Doug From mriedemos at gmail.com Wed Jul 24 14:34:01 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Wed, 24 Jul 2019 09:34:01 -0500 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> Message-ID: On 7/24/2019 1:37 AM, Massimo Sgaravatto wrote: > > Melanie: I think this is indeed the problem ! > > But then, if I am not wrong, the note in: > > https://docs.openstack.org/nova/rocky/admin/configuration/schedulers.html > > >  Note > > qemu is used for both QEMU and KVM hypervisor types. > > > should be removed. > I can open a bug if you agree ... > > And maybe this is something worth to be mentioned in the release notes ? > > Thanks again for your help ! > > Cheers, Massimo Massimo - just to confirm, your [libvirt]/virt_type is "kvm" rather than "qemu" correct? If so, then yeah what Melanie found is the problem and was regression in behavior for the ImagePropertiesFilter in Rocky and there should be a fix to the docs and likely a release note - though the release note is tricky since the regression was introduced in Rocky. -- Thanks, Matt From fungi at yuggoth.org Wed Jul 24 14:45:50 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 24 Jul 2019 14:45:50 +0000 Subject: [opendev] git clone fails: "The TLS connection was non-properly terminated" In-Reply-To: References: <584e116c-05c9-4eac-8bf2-804a098912e1@www.fastmail.com> <20190723005342.zr6h5tyqdetmu3ss@yuggoth.org> Message-ID: <20190724144550.h2k2jpkih2orapyi@yuggoth.org> On 2019-07-24 20:20:24 +0900 (+0900), Bernd Bausch wrote: > Thanks much, gentlemen. I did hope this was a glitch on the server > side, and indeed, I successfully cloned Devstack about 12 hours > ago. [...] Yes, sorry, we also subsequently discovered that haproxy was not configured with health checks for any of the backends, so it never removed the missing server from the relevant pools until I manually set the entries for it to maintenance mode. We have since merged fixes both to automatically apply haproxy configuration changes and to perform proper health checks of the Gitea servers, so this should hopefully not happen again. (We also updated how we restart Gitea itself so that each backend server is done sequentially and should provide ample alternatives for haproxy to roll through during updates now.) Thanks for letting us know! -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From mriedemos at gmail.com Wed Jul 24 14:47:49 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Wed, 24 Jul 2019 09:47:49 -0500 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> Message-ID: <13caab60-5afd-97ba-4b13-79ea4c9e15ff@gmail.com> On 7/24/2019 9:34 AM, Matt Riedemann wrote: > Massimo - just to confirm, your [libvirt]/virt_type is "kvm" rather than > "qemu" correct? If so, then yeah what Melanie found is the problem and > was regression in behavior for the ImagePropertiesFilter in Rocky and > there should be a fix to the docs and likely a release note - though the > release note is tricky since the regression was introduced in Rocky. Note that the glance description of the hypervisor_type property is also wrong: https://docs.openstack.org/glance/latest/admin/useful-image-properties.html "The hypervisor type. Note that qemu is used for both QEMU and KVM hypervisor types." Given https://review.opendev.org/#/c/531328/ was abandoned, if the API is still showing QEMU for the hypervisor type (which Massimo confirmed it is) even though the node is configured with virt_type=kvm and that's what the ImagePropertiesFilter is going to use, I think we'd be justified in reverting https://review.opendev.org/#/c/531347/ since it's totally confusing to operators if the API is showing the hypervisor_type as QEMU but the scheduler is filtering on "kvm". We can change the docs but that feels like papering over the issue to me. What would the docs say? Something like, "Since the 18.0.0 Rocky release, the hypervisor_type value for libvirt nodes should match the configured [libvirt]/virt_type value"? That won't fix any existing images with their properties embedded in an instance's system_metadata which could prevent you from being able to migrate those instances anywhere during the upgrade - you'd have to essentially do some database surgery in the instance_system_metadata table to fix the image_hypervisor_type value to match whatever the virt_type value is on that node. Alternatively we could try to put some targeted compat code in the ImagePropertiesFilter where if the hypervisor_type is QEMU/qemu but the node is reporting kvm, we let it slide and accept that host? -- Thanks, Matt From massimo.sgaravatto at gmail.com Wed Jul 24 14:51:39 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Wed, 24 Jul 2019 16:51:39 +0200 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> Message-ID: Correct. In my environment [libvirt]/virt_type is "kvm" (and this was the case also before updating to Rocky) Thanks, Massimo On Wed, Jul 24, 2019 at 4:34 PM Matt Riedemann wrote: > On 7/24/2019 1:37 AM, Massimo Sgaravatto wrote: > > > > Melanie: I think this is indeed the problem ! > > > > But then, if I am not wrong, the note in: > > > > > https://docs.openstack.org/nova/rocky/admin/configuration/schedulers.html > > > > > > Note > > > > qemu is used for both QEMU and KVM hypervisor types. > > > > > > should be removed. > > I can open a bug if you agree ... > > > > And maybe this is something worth to be mentioned in the release notes ? > > > > Thanks again for your help ! > > > > Cheers, Massimo > > Massimo - just to confirm, your [libvirt]/virt_type is "kvm" rather than > "qemu" correct? If so, then yeah what Melanie found is the problem and > was regression in behavior for the ImagePropertiesFilter in Rocky and > there should be a fix to the docs and likely a release note - though the > release note is tricky since the regression was introduced in Rocky. > > -- > > Thanks, > > Matt > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mriedemos at gmail.com Wed Jul 24 14:57:36 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Wed, 24 Jul 2019 09:57:36 -0500 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> Message-ID: <2d8418f6-67c4-41db-e5da-1c650e5a9793@gmail.com> On 7/24/2019 9:51 AM, Massimo Sgaravatto wrote: > Correct.   In my environment [libvirt]/virt_type is "kvm" (and this was > the case also before updating to Rocky) Please report a bug either way. I've also added comments in the bug associated with the patch that regressed the behavior [1]. [1] https://bugs.launchpad.net/nova/+bug/1195361/comments/22 -- Thanks, Matt From massimo.sgaravatto at gmail.com Wed Jul 24 15:19:15 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Wed, 24 Jul 2019 17:19:15 +0200 Subject: [ops] [nova] How to change/remove a property from an instance ? Message-ID: In my OpenStack Rocky cloud I have some instances that were created (before updating to Rocky) using images with the property hypervisor_type='qemu'. Since in my compute nodes [libvirt]/virt_type is "kvm", after having updated to Rocky I had to change these images removing that property of changing its value (qemu --> kvm): this is discussed in the thread: http://lists.openstack.org/pipermail/openstack-discuss/2019-July/thread.html#7842 Now I am not able to migrate anymore the instances that were created using those images (ImagePropertiesFilter filters out all compute nodes) before fixing the property. As suggested by Matt I tried to play with the instance_system_metadata table of the nova database. I first tried to change the value of that property (qemu --> kvm) [*] I also tried to remove the property [**] but the nova migrate keeps failing. In the request I keep seeing that Instance contains properties ImageMetaProps...img_hv_type='qemu' What else needs to be modified? Thanks, Massimo [*] mysql> select * from instance_system_metadata where instance_uuid='57dcdc2f-ef30-4512-beff-128c1621c5c6'; +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | created_at | updated_at | deleted_at | id | instance_uuid | key | value | deleted | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | 2017-05-15 12:32:57 | NULL | NULL | 9868217 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_hypervisor_type | kvm | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868220 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_disk_format | qcow2 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868223 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_container_format | bare | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868226 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_ram | 512 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868229 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_disk | 20 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868232 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_base_image_ref | c2d3e37c-b14f-4472-9bc2-d75f26ab900d | 0 | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ [**] mysql> delete from instance_system_metadata where id='9868217'; Query OK, 1 row affected (0.01 sec) mysql> select * from instance_system_metadata where instance_uuid='57dcdc2f-ef30-4512-beff-128c1621c5c6'; +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | created_at | updated_at | deleted_at | id | instance_uuid | key | value | deleted | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | 2017-05-15 12:32:57 | NULL | NULL | 9868220 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_disk_format | qcow2 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868223 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_container_format | bare | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868226 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_ram | 512 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868229 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_disk | 20 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868232 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_base_image_ref | c2d3e37c-b14f-4472-9bc2-d75f26ab900d | 0 | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ 5 rows in set (0.00 sec) -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Wed Jul 24 15:29:26 2019 From: smooney at redhat.com (Sean Mooney) Date: Wed, 24 Jul 2019 16:29:26 +0100 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: <13caab60-5afd-97ba-4b13-79ea4c9e15ff@gmail.com> References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> <13caab60-5afd-97ba-4b13-79ea4c9e15ff@gmail.com> Message-ID: <33da4e3ad316371da4c0d59e34a0ccb456a24b23.camel@redhat.com> On Wed, 2019-07-24 at 09:47 -0500, Matt Riedemann wrote: > On 7/24/2019 9:34 AM, Matt Riedemann wrote: > > Massimo - just to confirm, your [libvirt]/virt_type is "kvm" rather than > > "qemu" correct? If so, then yeah what Melanie found is the problem and > > was regression in behavior for the ImagePropertiesFilter in Rocky and > > there should be a fix to the docs and likely a release note - though the > > release note is tricky since the regression was introduced in Rocky. > > Note that the glance description of the hypervisor_type property is also > wrong: > > https://docs.openstack.org/glance/latest/admin/useful-image-properties.html > > "The hypervisor type. Note that qemu is used for both QEMU and KVM > hypervisor types." > > Given https://review.opendev.org/#/c/531328/ was abandoned, if the API > is still showing QEMU for the hypervisor type (which Massimo confirmed > it is) even though the node is configured with virt_type=kvm and that's > what the ImagePropertiesFilter is going to use, I think we'd be > justified in reverting https://review.opendev.org/#/c/531347/ since it's > totally confusing to operators if the API is showing the hypervisor_type > as QEMU but the scheduler is filtering on "kvm". the API and the fileters should both see the hyperviors type as QEMU regradles of if the virt type is qemu or kvm. kvm is just an accleration mechanium for QEMU it is not a sperate hypervior. we stopped progerssing https://review.opendev.org/#/c/531328/ as it was a breaking api change that leaked config info via the api. e.g. the virt_type. we should be consistent however an always report qemu both via the api and to the image proerties filetr for hypervisor_type. we could have a seperate imemamge metadata for forcing kvm or qemu but as documented in glance hypervior_type should be qemu when the tcg backend is user or the kvm backend. if we wanted to support kvm specificlay i would suggest supporting vm_mode=hvm > > We can change the docs but that feels like papering over the issue to > me. What would the docs say? Something like, "Since the 18.0.0 Rocky > release, the hypervisor_type value for libvirt nodes should match the > configured [libvirt]/virt_type value"? That won't fix any existing > images with their properties embedded in an instance's system_metadata > which could prevent you from being able to migrate those instances > anywhere during the upgrade - you'd have to essentially do some database > surgery in the instance_system_metadata table to fix the > image_hypervisor_type value to match whatever the virt_type value is on > that node. > > Alternatively we could try to put some targeted compat code in the > ImagePropertiesFilter where if the hypervisor_type is QEMU/qemu but the > node is reporting kvm, we let it slide and accept that host? > From smooney at redhat.com Wed Jul 24 15:30:34 2019 From: smooney at redhat.com (Sean Mooney) Date: Wed, 24 Jul 2019 16:30:34 +0100 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> Message-ID: <5c3abe5ab84378c400ddfc1df3ccb0fd62167a0d.camel@redhat.com> On Wed, 2019-07-24 at 16:51 +0200, Massimo Sgaravatto wrote: > Correct. In my environment [libvirt]/virt_type is "kvm" (and this was the > case also before updating to Rocky) the expected behavior in this case it the hypervior_type will be qemu if we are now reporting it as kvm that is a bug. > > > Thanks, Massimo > > > On Wed, Jul 24, 2019 at 4:34 PM Matt Riedemann wrote: > > > On 7/24/2019 1:37 AM, Massimo Sgaravatto wrote: > > > > > > Melanie: I think this is indeed the problem ! > > > > > > But then, if I am not wrong, the note in: > > > > > > > > > > https://docs.openstack.org/nova/rocky/admin/configuration/schedulers.html > > > > > > > > > Note > > > > > > qemu is used for both QEMU and KVM hypervisor types. > > > > > > > > > should be removed. > > > I can open a bug if you agree ... > > > > > > And maybe this is something worth to be mentioned in the release notes ? > > > > > > Thanks again for your help ! > > > > > > Cheers, Massimo > > > > Massimo - just to confirm, your [libvirt]/virt_type is "kvm" rather than > > "qemu" correct? If so, then yeah what Melanie found is the problem and > > was regression in behavior for the ImagePropertiesFilter in Rocky and > > there should be a fix to the docs and likely a release note - though the > > release note is tricky since the regression was introduced in Rocky. > > > > -- > > > > Thanks, > > > > Matt > > From mriedemos at gmail.com Wed Jul 24 15:45:02 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Wed, 24 Jul 2019 10:45:02 -0500 Subject: [ops] [nova] How to change/remove a property from an instance ? In-Reply-To: References: Message-ID: <8874a0da-d08a-87e1-5037-3b47ba8cac31@gmail.com> On 7/24/2019 10:19 AM, Massimo Sgaravatto wrote: > I first tried to change the value of that property (qemu --> kvm) [*] > I also tried to remove the property [**] but the nova migrate keeps > failing. In the request I keep seeing that > > Instance contains properties ImageMetaProps...img_hv_type='qemu' > > What else needs to be modified? The filter is working on the ImageMetaProps object hydrated from the json serialized string in the request_specs.spec column, which unfortunately makes fixing this a bigger pain in the ass since it's not a simple SQL update statement. If I were you I'd probably just revert https://review.opendev.org/#/c/531347/ so you can get on with your Rocky upgrade while we deal with fixing the issue long-term upstream, either by reverting that change upstream as well or adding a compat shim in the ImagePropertiesFilter. -- Thanks, Matt From massimo.sgaravatto at gmail.com Wed Jul 24 15:47:19 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Wed, 24 Jul 2019 17:47:19 +0200 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: <2d8418f6-67c4-41db-e5da-1c650e5a9793@gmail.com> References: <6a3a8f13-3bf3-c0e1-1a1e-67caea553786@gmail.com> <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> <2d8418f6-67c4-41db-e5da-1c650e5a9793@gmail.com> Message-ID: On Wed, Jul 24, 2019 at 4:57 PM Matt Riedemann wrote: > > Please report a bug either way. I've also added comments in the bug > associated with the patch that regressed the behavior [1]. > > [1] https://bugs.launchpad.net/nova/+bug/1195361/comments/22 > > > https://bugs.launchpad.net/nova/+bug/1837756 Thanks, Massimo -------------- next part -------------- An HTML attachment was scrubbed... URL: From openstack at nemebean.com Wed Jul 24 16:05:23 2019 From: openstack at nemebean.com (Ben Nemec) Date: Wed, 24 Jul 2019 11:05:23 -0500 Subject: [oslo] PTO next week Message-ID: <750f6a75-4c2e-2e5d-94ae-1bc3f29d6fef@nemebean.com> Just a heads up that I will be on PTO next week and incommunicado pretty much the whole time. Moises (moguimar) volunteered to run the Oslo meeting so that will still be happening. Herve (hberaud) and Doug (dhellmann) are our release liaisons so if you need a release of any Oslo libraries next week, they're the ones to talk to. Otherwise, things have been pretty quiet in Oslo-land (which would be Norway, I guess), so I expect all the things to break next week while I'm out. Good luck. ;-) Thanks. -Ben From mriedemos at gmail.com Wed Jul 24 16:30:23 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Wed, 24 Jul 2019 11:30:23 -0500 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: References: <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> <2d8418f6-67c4-41db-e5da-1c650e5a9793@gmail.com> Message-ID: <9390f785-9a57-37c0-51a3-4ffa02520a3f@gmail.com> On 7/24/2019 10:47 AM, Massimo Sgaravatto wrote: > > > On Wed, Jul 24, 2019 at 4:57 PM Matt Riedemann > wrote: > > > Please report a bug either way. I've also added comments in the bug > associated with the patch that regressed the behavior [1]. > > [1] https://bugs.launchpad.net/nova/+bug/1195361/comments/22 > > > > https://bugs.launchpad.net/nova/+bug/1837756 > > Thanks, Massimo Here is the revert: https://review.opendev.org/#/c/672559/ -- Thanks, Matt From pawel.konczalski at everyware.ch Wed Jul 24 16:35:52 2019 From: pawel.konczalski at everyware.ch (Pawel Konczalski) Date: Wed, 24 Jul 2019 18:35:52 +0200 Subject: DuplicateMessageError after restart of a control node In-Reply-To: <8c01078d-1c9a-3153-1856-cf9b3ef0f3d1@everyware.ch> References: <8c01078d-1c9a-3153-1856-cf9b3ef0f3d1@everyware.ch> Message-ID: <26eea5c6-fd23-a9e4-368a-c0635878c903@everyware.ch> Hello everybody, after some investigation in the RabbitMQ problems we found some duplicated messages and timeouts in logs. Restarting the whole RabbitMQ cluster (stop all rabbitmq containers and start one by one) solved the problems for now. The main cause for this issue seams to by the nova notifications configuration with was deployed by kolla-ansible. If searchlight is not installed the 'notifications/notification_format' should be 'unversioned'. Default is 'both' so nova will send a notification to the queue versioned_notifications with has no consumer. In our case the queue got huge amount of messages with made the rabbitmq cluster more and more unstable, see: https://bugzilla.redhat.com/show_bug.cgi?id=1592528 Following settings in nova.conf may solve this issue but we didn`t tested this yet: [notification] notification_format = unversioned BR Pawel -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5227 bytes Desc: not available URL: From laszlo.budai at gmail.com Wed Jul 24 17:04:00 2019 From: laszlo.budai at gmail.com (Budai Laszlo) Date: Wed, 24 Jul 2019 20:04:00 +0300 Subject: [Nova] migration issue Message-ID: <1d52d342-ed45-0718-e2ec-e5c5c90d7389@gmail.com> Dear all, we are testing the cold migratoin of instances that are using local ephemeral storage and it fails with the following error: 2019-07-24 13:47:13.115 58902 INFO nova.virt.libvirt.driver [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae 58-00d6-4317-b635-909fdf09ac49] Instance shutdown successfully after 3 seconds. 2019-07-24 13:47:13.126 58902 INFO nova.virt.libvirt.driver [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Instance destroyed successfully. 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58- 00d6-4317-b635-909fdf09ac49] Setting instance vm_state to ERROR: ProcessExecutionError: Unexpected error while running command. Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph 0 Exit code: 1 Stdout: u'' Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured syst em and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. * \n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No suc h file or directory\n' 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Traceback (most recent call last): 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 75 18, in _error_out_instance_on_exception 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] yield 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 42 75, in _resize_instance 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] timeout, retry_interval) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin e 8200, in migrate_disk_and_power_off 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] shared_storage) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220 , in __exit__ 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] self.force_reraise() 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196 , in force_reraise 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin e 8185, in migrate_disk_and_power_off 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in copy_image 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs .py", line 110, in copy_file 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs .py", line 196, in copy_file 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] on_execute=on_execute, on_completion=on_completion) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in exec ute 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] return processutils.execute(*cmd, **kwargs) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py" , line 424, in execute 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] cmd=sanitized_cmd) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] ProcessExecutionError: Unexpected error while running command. 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Exit code: 1 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stdout: u'' 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] 2019-07-24 13:47:14.380 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Swapping old allocation on 3da11b23-5ef8-4471-a3c9-5f743b0bbfd7 held by migration da66c416-2636-421b-99cb-470bd07dad4c for instance 2019-07-24 13:47:14.979 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Successfully reverted task state from resize_migrating on failure for instance. 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] Exception during message handling: ProcessExecutionError: Unexpected error while running command. Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 Exit code: 1 Stdout: u'' Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server function_name, call_dict, binary) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4240, in resize_instance 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self._revert_allocation(context, instance, migration) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4237, in resize_instance 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server instance_type, clean_shutdown) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4275, in _resize_instance 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server timeout, retry_interval) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8200, in migrate_disk_and_power_off 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server shared_storage) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8185, in migrate_disk_and_power_off 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server compression=compression) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in copy_image 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server compression=compression) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in copy_file 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server compression=compression) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in copy_file 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server on_execute=on_execute, on_completion=on_completion) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in execute 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command. 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Exit code: 1 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stdout: u'' 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server 2019-07-24 13:47:17.065 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] image ca89475a-cf15-46d1-ab2d-c2b96c403248 at (/var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07): checking 2019-07-24 13:47:17.066 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] Active base files: /var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07 2019-07-24 13:47:27.768 58902 INFO nova.compute.manager [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] VM Stopped (Lifecycle Event) 2019-07-24 13:47:27.918 58902 INFO nova.compute.manager [req-e53e928d-81fe-4185-9e62-abc2113e838c - - - - -] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. The disk has its root FS n an RBD device (a volume created by cinder) and it has its ephemeral disk on a local LVM volume created by nova. It seems that the /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 is not there when scp is trying to copy it to the other side. Any ideas how to fix this? The nova config on the nodes looks like this: # Ansible managed [DEFAULT] allow_resize_to_same_host = True # Compute compute_driver = libvirt.LibvirtDriver # Scheduler cpu_allocation_ratio = 2.0 # Logs / State debug = False # Hypervisor default_ephemeral_format = ext4 disk_allocation_ratio = 1.0 # Api's enabled_apis = osapi_compute,metadata executor_thread_pool_size = 64 fatal_deprecations = False # Configdrive force_config_drive = False host = compute8.mgmt.lab.mydomain.com image_cache_manager_interval = 0 instance_name_template = instance-%08x # Ceilometer notification configurations instance_usage_audit = True instance_usage_audit_period = hour instances_path = /var/lib/nova/instances ## Vif libvirt_vif_type = ethernet log_dir = /var/log/nova # Metadata metadata_workers = 16 # Network my_ip = 192.168.56.47 notify_on_state_change = vm_and_task_state osapi_compute_workers = 16 ram_allocation_ratio = 1.0 reserved_host_disk_mb = 0 reserved_host_memory_mb = 2048 resume_guests_state_on_host_boot = False rootwrap_config = /etc/nova/rootwrap.conf rpc_response_timeout = 60 service_down_time = 120 state_path = /var/lib/nova # Rpc all transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e 8cf8bd7 at 192.168.56.179:5671//nova # Disable stderr logging use_stderr = False vif_plugging_is_fatal = True vif_plugging_timeout = 30 [api] auth_strategy = keystone enable_instance_password = True use_forwarded_for = False vendordata_jsonfile_path = /etc/nova/vendor_data.json # Cache [cache] backend = oslo_cache.memcache_pool enabled = true memcache_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 # Cinder [cinder] cafile = /etc/ssl/certs/ca-certificates.crt catalog_info = volumev3:cinderv3:internalURL cross_az_attach = True os_region_name = A_Lab [conductor] workers = 16 [filter_scheduler] available_filters = nova.scheduler.filters.all_filters enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, AggregateRamFilter, ComputeFilter, AggregateCoreFilter, DiskFilter, AggregateDiskFilter, AggregateNumInstancesFilter, AggregateIoOpsFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, NUMATopologyFilter, SameHostFilter, DifferentHostFilter host_subset_size = 10 max_instances_per_host = 50 max_io_ops_per_host = 10 ram_weight_multiplier = 5.0 tracks_instance_changes = True weight_classes = nova.scheduler.weights.all_weighers # Glance [glance] api_servers = https://vip.mgmt.lab.mydomain.com:9292 cafile = /etc/ssl/certs/ca-certificates.crt [key_manager] fixed_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [keystone_authtoken] auth_type = password auth_uri = https://vip.mgmt.lab.mydomain.com:5000 auth_url = https://vip.mgmt.lab.mydomain.com:35357 cafile = /etc/ssl/certs/ca-certificates.crt insecure = False memcache_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # if your memcached server is shared, use these settings to avoid cache poisoning memcache_security_strategy = ENCRYPT memcached_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 password = xxxxxxxxxxxxxxxxxxxxxxx project_domain_id = default project_name = service region_name = A_Lab token_cache_time = 300 user_domain_id = default username = nova [libvirt] disk_cachemodes = network=writeback hw_disk_discard = unmap images_rbd_ceph_conf = /etc/ceph/ceph.conf images_rbd_pool = vms images_type = lvm images_volume_group = ssd_nova inject_key = False inject_partition = -2 inject_password = False live_migration_tunnelled = True live_migration_uri = "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa" rbd_secret_uuid = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # ceph rbd support rbd_user = cinder remove_unused_resized_minimum_age_seconds = 3600 use_virtio_for_bridges = True virt_type = kvm # Neutron [neutron] auth_type = password # Keystone client plugin authentication URL option auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 cafile = /etc/ssl/certs/ca-certificates.crt default_floating_pool = public insecure = False metadata_proxy_shared_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxx # Keystone client plugin password option password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx project_domain_name = Default project_name = service region_name = A_Lab service_metadata_proxy = True url = https://vip.mgmt.lab.mydomain.com:9696 user_domain_name = Default # Keystone client plugin username option username = neutron [oslo_concurrency] lock_path = /var/lock/nova # Notifications [oslo_messaging_notifications] driver = messagingv2 notification_topics = notifications transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.179:5671//nova [oslo_messaging_rabbit] rpc_conn_pool_size = 30 ssl = True # Placement [placement] auth_type = "password" auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 cafile = /etc/ssl/certs/ca-certificates.crt insecure = False os_interface = internal os_region_name = A_Lab password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx project_domain_name = Default project_name = service user_domain_name = Default username = placement [quota] cores = 20 injected_file_content_bytes = 10240 injected_file_path_length = 255 injected_files = 5 instances = 10 key_pairs = 100 max_age = 0 metadata_items = 128 ram = 32768 server_group_members = 10 server_groups = 10 [scheduler] discover_hosts_in_cells_interval = 60 host_manager = host_manager max_attempts = 5 periodic_task_interval = 60 scheduler_driver = filter_scheduler [spice] agent_enabled = True enabled = True # Console Url and binds html5proxy_base_url = https://dashboard.iaas.lab.mydomain.com:6080/spice_auto.html server_listen = 192.168.56.47 server_proxyclient_address = 192.168.56.47 [upgrade_levels] compute = auto [vnc] enabled = False [wsgi] api_paste_config = /etc/nova/api-paste.ini secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO Thank you in advance for any suggestions. Kind regards, Laszlo From Prabhjit.Singh22 at T-Mobile.com Tue Jul 23 13:45:16 2019 From: Prabhjit.Singh22 at T-Mobile.com (Singh, Prabhjit) Date: Tue, 23 Jul 2019 13:45:16 +0000 Subject: [Octavia]-Seeking performance numbers on Octavia In-Reply-To: References: Message-ID: Thanks so much for the valuable insights Michael! Appreciate it and keep up the good work, as I ramp up with more dev know how hopefully I would start making contributions and can maybe convince my team to start as well. Thanks & Regards Prabhjit Singh -----Original Message----- From: Michael Johnson Sent: Monday, July 22, 2019 5:48 PM To: Singh, Prabhjit Cc: openstack-discuss at lists.openstack.org Subject: Re: [Octavia]-Seeking performance numbers on Octavia [External] Hi Prabhjit, Comments in-line below. Michael On Sun, Jul 21, 2019 at 5:24 PM Singh, Prabhjit wrote: > > Hi Michael, > > Thanks for taking the time out to send me your inputs and valuable suggestions. I do remember meeting you at the Denver Summit and hearing to a couple of your sessions. > If you wouldn't mind, I do have a few more questions and your answers would help me understand that should I continue to invest in having Octavia as one of our available LBs. > > 1. Based on your response and the amount of time you are investing in > supporting Octavia, what are some of the use cases, like for e.g. if load balancing web traffic how many transactions/connections minimum can be expected. I do understand you mentioned that it's hard to performance test Octavia but some real time situations from your testing and how customers have adopted Octavia would help me level set some expectations. This is really cloud and application specific. I would recommend you fire up an Octavia install and use your preferred tool to measure it. Some good tools are tsung, weighttp, and iperf3. > 2. We are thinking of Octavia as one of the offerings, that offers a self-serve type model. Do you know of any customers who have been able to use Octavia as one of their primary load balancers and any encouraging feedback you have gotten on Octavia. There are examples of organizations using Octavia available if you google Octavia. > 3. You suggested increasing the Ram size, I could go about making a whole new Flavor. Yes, to increase the allocated RAM for a load balancer, you would create an additional nova flavor with the specifications you would like. You can then either set this as the default nova flavor for amphora (amp_flavor_id is the setting) or you can create an Octavia flavor that specifies the nova compute flavor to use (See https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.openstack.org%2Foctavia%2Flatest%2Fadmin%2Fflavors.html&data=02%7C01%7CPrabhjit.Singh22%40t-mobile.com%7Cfb41388d6020453d92c908d70eee4a72%7Cbe0f980bdd994b19bd7bbc71a09b026c%7C0%7C0%7C636994288931593870&sdata=FDlAK3%2FKh0DNo%2BMSJQ8kJ8lSnn01TJXASS6AHd1kRoA%3D&reserved=0 for more information on Octavia flavors). > 4. I also noticed on the haproxy.conf the maxconns is set to 2000, should I increase this, does this affect the connection per server, which you said 64000 conns per server, so if I have 10 servers can I expect somewhere close to 640000 sessions? I think you are looking at the haproxy.conf file provided by your operating system package. Octavia does not use this file, it creates it's own HAProxy configuration files as needed under /var/lib/octavia inside the amphora. The default, if the user does not specify one at listener creation, is 1,000,000. > 5. Based on some of the limitations and the dev work in progress, I think the most important feature that would make Octavia a real solid offering would be the Active-Active and Autoscaling feature. I brought this up with you in our brief conversation at the summit, and you did mention that its not a top priority at this time and you are looking for some help. I have noticed a lot of documentation has been updated on this feature, do you think with the available document and progress I could spin up a distributor and manage sessions between Amphora or it's not complete yet. Active/Active is still on our roadmap, but unfortunately the people that were working on it had to stop for personal reasons. There may be some folks picking up this work again soon. At this point the Active/Active patches up for review are non-functional and still a work in progress. > 6. We have a Triple O setup, do you think I can make the above tweaks with the Triple O setup. I think you are able to make various adjustments to Octavia with Triple O, but I do not have specifics on that. > Thanks & Regards > > Prabhjit Singh > Systems Design and Strategy - Magentabox > | O: (973) 397-4819 | M: (973) 563-4445 > > > > -----Original Message----- > From: Michael Johnson > Sent: Friday, July 19, 2019 6:00 PM > To: Singh, Prabhjit > Cc: openstack-discuss at lists.openstack.org > Subject: Re: [Octavia]-Seeking performance numbers on Octavia > > [External] > > > Hi Prabhjit, > > As you have mentioned, it is very challenging to get accurate performance results in cloud environments. There are a large number(very large in fact) of factors that can impact the overall performance of OpenStack and Octavia. > > In our OpenDev testing environment, we only have software emulation virtual machines available (Qemu running with the TCG engine) which performs extremely poorly. This means that the testing environment does not reflect how the software is used in real world deployments. > An example of this is simply booting a VM can take up to ten minutes on Qemu with TCG when it takes about twenty seconds on a real OpenStack deployment. > With this resource limitation, we cannot effectively run performance benchmarking test jobs on the OpenDev environment. > > Because of this, we don't publish performance numbers as they will not reflect what you can achieve in your environment. > > Let me try to speak to your bullet points: > 1. The Octavia team has never (to my knowledge) claimed the Amphora driver is "carrier grade". We do consider the Amphora driver to be "operator grade", which speaks to a cloud operator's perspective versus the previous offering that did not support high availability, have appropriate maintenance tooling, upgrade paths, performance, etc. > To me, "carrier grade" has an additional level of requirements including performance, latency, scale, and availability SLAs. This is not what the Octavia Amphora driver is currently ready for. That said, third party provider drivers for Octavia may be able to provide a "carrier grade" level of load balancing for OpenStack. > 2. As for performance tuning, much of this is either automatically handled by Octavia or are dependent on the application you are load balancing and your cloud deployment. For example we have many configuration settings to tune how many retries we attempt when interacting with other services. In performing and stable clouds, these can be tuned down, in others the defaults may be appropriate. If you would like faster failover, at the expense of slightly more network traffic, you can tune the health monitoring and keepalived_vrrp settings. We do not currently have a performance tuning guide for Octavia but would support someone authoring one. > 3. We do not currently have a guide for this. I will say with the version of HAproxy currently being shipped with the distributions, going beyond the 1vCPU per amphora does not gain you much. With the release of HAProxy 2.0 this has changed and we expect to be adding support for vertically scaling the Amphora in future releases. Disk space is only necessary if you are storing the flow logs locally, which I would not recommend for a performance load balancer (See the notes in the log offloading guide: > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.openstack.org%2Foctavia%2Flatest%2Fadmin%2Flog-offloading.html&data=02%7C01%7CPrabhjit.Singh22%40t-mobile.com%7Cfb41388d6020453d92c908d70eee4a72%7Cbe0f980bdd994b19bd7bbc71a09b026c%7C0%7C0%7C636994288931593870&sdata=qyX1BM6wR6v804WCYB2HY6IRmDfeQS1zi38FS34kB1U%3D&reserved=0). > Finally, the RAM usage is a factor of the number of concurrent connections and if you are enabling TLS on the load balancer. For typical load balancing loads, the default is typically fine. However, if you have high connection counts and/or TLS offloading, you may want to experiment with increasing the available RAM. > 4. The source IP issue is a known issue > (https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstoryboard.openstack.org%2F%23!%2Fstory%2F1629066&data=02%7C01%7CPrabhjit.Singh22%40t-mobile.com%7Cfb41388d6020453d92c908d70eee4a72%7Cbe0f980bdd994b19bd7bbc71a09b026c%7C0%7C0%7C636994288931593870&sdata=GkTPXRmOfjpMYXDYZ9t5xH1aEq0E%2BWDZRhK8ux%2FnrUQ%3D&reserved=0). We have not prioritized addressing this as we have not had anyone come forward that they needed this in their deployment. If this is an issue impacting your use case, please comment on the story to that effect and provide a use case. This will help the team prioritize this work. > Also, patches are welcome! If you are interested in working on this issue, I can help you with information about how this could be added. > It should also be noted that it is a limitation of 64,000 connections per-backend server, not per load balancer. > 5. The team uses the #openstack-lbaas IRC channel on freenode and is happy to answer questions, etc. > > To date, we have had limited resources (people and equipment) available to do performance evaluation and tuning. There are definitely kernel and HAProxy tuning settings we have evaluated and added to the Amphora driver, but I know there is more work that can be done. If you are interested in help us with this work, please let us know. > > Michael > > P.S. Here are just a few considerations that can/will impact the performance of an Octavia Amphora load balancer: > > Hardware used for the compute nodes > Network Interface Cards (NICs) used in the compute nodes Number of > network ports enabled on the compute hosts Network switch > configurations (Jumbo frames, and so on) Cloud network topology > (leaf‐spine, fat‐tree, and so on) The OpenStack Neutron networking > configuration (ML2 and ML3 drivers) Tenant networking configuration > (VXLAN, VLANS, GRE, and so on) Colocation of applications and Octavia > amphorae Over subscription of the compute and networking resources > Protocols being load balanced Configuration settings used when > creating the load balancer (connection limits, and so on) Version of > OpenStack services (nova, neutron, and so on) Version of OpenStack > Octavia Flavor of the OpenStack Octavia load balancer OS and > hypervisor versions used Deployed security mitigations (Spectre, > Meltdown, and so on) Customer application performance Health of the > customer application > > On Fri, Jul 19, 2019 at 8:52 AM Singh, Prabhjit wrote: > > > > Hi > > > > > > > > I have been trying to test Octavia with some traffic generators and > > my tests are inconclusive. Appreciate your inputs on the following > > > > > > > > It would be really nice to have some performance numbers that you guys have been able to achieve for this to be termed as carrier grade. > > Would also appreciate if you could share any inputs on performance > > tuning Octavia Any recommended flavor sizes for spinning up Amphorae, the default size of 1 core, 2 Gb disk and 1 Gig RAM does not seem enough. > > Also I noticed when the Amphorae are spun up, at one time only one > > master is talking to the backend servers and has one IP that its > > using, it has to run out of ports after 64000 TCP concurrent > > sessions, id there a way to add more IPs or is this the limitation > > If I needed some help with Octavia and some guidance around > > performance tuning can someone from the community help > > > > > > > > Thanks & Regards > > > > > > > > Prabhjit Singh > > > > > > > > > > > > From vungoctan252 at gmail.com Tue Jul 23 15:18:04 2019 From: vungoctan252 at gmail.com (Vu Tan) Date: Tue, 23 Jul 2019 22:18:04 +0700 Subject: [masakari] how to install masakari on centos 7 In-Reply-To: References: <35400f83-c29d-475a-8d36-d56b3cf16d30@email.android.com> Message-ID: Hi Patil, Thank you for your reply, please instruct me if you successfully install it. Thanks a lot On Tue, Jul 23, 2019 at 8:12 PM Patil, Tushar wrote: > Hi Vu Tan, > > I'm trying to install Masakari using source code to reproduce the issue. > If I hit the same issue as yours, I will troubleshoot this issue and let > you know the solution or will update you what steps I have followed to > bring up Masakari services successfully. > > Regards, > Tushar Patil > > ________________________________________ > From: Vu Tan > Sent: Monday, July 22, 2019 12:33 PM > To: Gaëtan Trellu > Cc: Patil, Tushar; openstack-discuss at lists.openstack.org > Subject: Re: [masakari] how to install masakari on centos 7 > > Hi Patil, > May I know when the proper document for masakari is released ? I have > configured conf file in controller and compute node, it seems running but > it is not running as it should be, a lots of error in logs, here is a > sample log: > > 2.7/site-packages/oslo_config/cfg.py:3024 > 2019-07-19 10:25:26.360 7745 DEBUG oslo_service.service [-] bindir > = /usr/local/bin log_opt_values /usr/lib/ > python2.7/site-packages/oslo_config/cfg.py:3024 > 2019-07-19 18:46:21.291 7770 ERROR masakari File > "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 65, in > _is_daemo > n > 2019-07-19 18:46:21.291 7770 ERROR masakari is_daemon = os.getpgrp() > != os.tcgetpgrp(sys.stdout.fileno()) > 2019-07-19 18:46:21.291 7770 ERROR masakari OSError: [Errno 5] > Input/output error > 2019-07-19 18:46:21.291 7770 ERROR masakari > 2019-07-19 18:46:21.300 7745 CRITICAL masakari [-] Unhandled error: > OSError: [Errno 5] Input/output error > 2019-07-19 18:46:21.300 7745 ERROR masakari Traceback (most recent call > last): > 2019-07-19 18:46:21.300 7745 ERROR masakari File > "/usr/bin/masakari-api", line 10, in > > I dont know if it is missing package or wrong configuration > > > On Thu, Jul 11, 2019 at 6:14 PM Gaëtan Trellu > wrote: > You will have to enable the debit, debug = true and check the APi log. > > Did you try to use the openstack CLi ? > > Gaetan > > On Jul 11, 2019 12:32 AM, Vu Tan vungoctan252 at gmail.com>> wrote: > I know it's just a warning, just take a look at this image: > [image.png] > it's just hang there forever, and in the log show what I have shown to you > > On Wed, Jul 10, 2019 at 8:07 PM Gaëtan Trellu > wrote: > This is just a warning, not an error. > > On Jul 10, 2019 3:12 AM, Vu Tan vungoctan252 at gmail.com>> wrote: > Hi Gaetan, > I follow you the guide you gave me, but the problem still persist, can you > please take a look at my configuration to see what is wrong or what is > missing in my config ? > the error: > 2019-07-10 14:08:46.876 17292 WARNING keystonemiddleware._common.config > [-] The option "__file__" in conf is not known to auth_token > 2019-07-10 14:08:46.876 17292 WARNING keystonemiddleware._common.config > [-] The option "here" in conf is not known to auth_token > 2019-07-10 14:08:46.882 17292 WARNING keystonemiddleware.auth_token [-] > AuthToken middleware is set with keystone_authtoken.service_ > > the config: > > [DEFAULT] > enabled_apis = masakari_api > log_dir = /var/log/kolla/masakari > state_path = /var/lib/masakari > os_user_domain_name = default > os_project_domain_name = default > os_privileged_user_tenant = service > os_privileged_user_auth_url = http://controller:5000/v3 > os_privileged_user_name = nova > os_privileged_user_password = P at ssword > masakari_api_listen = controller > masakari_api_listen_port = 15868 > debug = False > auth_strategy=keystone > > [wsgi] > # The paste configuration file path > api_paste_config = /etc/masakari/api-paste.ini > > [keystone_authtoken] > www_authenticate_uri = http://controller:5000 > auth_url = http://controller:5000 > auth_type = password > project_domain_id = default > project_domain_name = default > user_domain_name = default > user_domain_id = default > project_name = service > username = masakari > password = P at ssword > region_name = RegionOne > > [oslo_middleware] > enable_proxy_headers_parsing = True > > [database] > connection = mysql+pymysql://masakari:P at ssword@controller/masakari > > > > On Tue, Jul 9, 2019 at 10:25 PM Vu Tan vungoctan252 at gmail.com>> wrote: > Thank Patil Tushar, I hope it will be available soon > > On Tue, Jul 9, 2019 at 8:18 AM Patil, Tushar > wrote: > Hi Vu and Gaetan, > > Gaetan, thank you for helping out Vu in setting up masakari-monitors > service. > > As a masakari team ,we have noticed there is a need to add proper > documentation to help the community run Masakari services in their > environment. We are working on adding proper documentation in this 'Train' > cycle. > > Will send an email on this mailing list once the patches are uploaded on > the gerrit so that you can give your feedback on the same. > > If you have any trouble in setting up Masakari, please let us know on this > mailing list or join the bi-weekly IRC Masakari meeting on the > #openstack-meeting IRC channel. The next meeting will be held on 16th July > 2019 @0400 UTC. > > Regards, > Tushar Patil > > ________________________________________ > From: Vu Tan > > Sent: Monday, July 8, 2019 11:21:16 PM > To: Gaëtan Trellu > Cc: openstack-discuss at lists.openstack.org openstack-discuss at lists.openstack.org> > Subject: Re: [masakari] how to install masakari on centos 7 > > Hi Gaetan, > Thanks for pinpoint this out, silly me that did not notice the simple > "error InterpreterNotFound: python3". Thanks a lot, I appreciate it > > On Mon, Jul 8, 2019 at 9:15 PM gaetan.trellu at incloudus.com> gaetan.trellu at incloudus.com>>> wrote: > Vu Tan, > > About "auth_token" error, you need "os_privileged_user_*" options into > your masakari.conf for the API. > As mentioned previously please have a look here to have an example of > configuration working (for me at least): > > - masakari.conf: > > https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari.conf.j2 > - masakari-monitor.conf: > > https://review.opendev.org/#/c/615715/42/ansible/roles/masakari/templates/masakari-monitors.conf.j2 > > About your tox issue make sure you have Python3 installed. > > Gaëtan > > On 2019-07-08 06:08, Vu Tan wrote: > > > Hi Gaetan, > > I try to generate config file by using this command tox -egenconfig on > > top level of masakari but the output is error, is this masakari still > > in beta version ? > > [root at compute1 masakari-monitors]# tox -egenconfig > > genconfig create: /root/masakari-monitors/.tox/genconfig > > ERROR: InterpreterNotFound: python3 > > _____________________________________________________________ summary > > ______________________________________________________________ > > ERROR: genconfig: InterpreterNotFound: python3 > > > > On Mon, Jul 8, 2019 at 3:24 PM Vu Tan vungoctan252 at gmail.com> vungoctan252 at gmail.com>>> wrote: > > Hi, > > Thanks a lot for your reply, I install pacemaker/corosync, > > masakari-api, maskari-engine on controller node, and I run masakari-api > > with this command: masakari-api, but I dont know whether the process is > > running like that or is it just hang there, here is what it shows when > > I run the command, I leave it there for a while but it does not change > > anything : > > [root at controller masakari]# masakari-api > > 2019-07-08 15:21:09.946 30250 INFO masakari.api.openstack [-] Loaded > > extensions: ['extensions', 'notifications', 'os-hosts', 'segments', > > 'versions'] > > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > > [-] The option "__file__" in conf is not known to auth_token > > 2019-07-08 15:21:09.955 30250 WARNING keystonemiddleware._common.config > > [-] The option "here" in conf is not known to auth_token > > 2019-07-08 15:21:09.960 30250 WARNING keystonemiddleware.auth_token [-] > > AuthToken middleware is set with > > keystone_authtoken.service_token_roles_required set to False. This is > > backwards compatible but deprecated behaviour. Please set this to True. > > 2019-07-08 15:21:09.974 30250 INFO masakari.wsgi [-] masakari_api > > listening on 127.0.0.1:15868< > http://127.0.0.1:15868> > > 2019-07-08 15:21:09.975 30250 INFO oslo_service.service [-] Starting 4 > > workers > > 2019-07-08 15:21:09.984 30274 INFO masakari.masakari_api.wsgi.server > > [-] (30274) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.985 30275 INFO masakari.masakari_api.wsgi.server > > [-] (30275) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.992 30277 INFO masakari.masakari_api.wsgi.server > > [-] (30277) wsgi starting up on http://127.0.0.1:15868 > > 2019-07-08 15:21:09.994 30276 INFO masakari.masakari_api.wsgi.server > > [-] (30276) wsgi starting up on http://127.0.0.1:15868 > > > > On Sun, Jul 7, 2019 at 7:37 PM Gaëtan Trellu > > gaetan.trellu at incloudus.com>> wrote: > > > > Hi Vu Tan, > > > > Masakari documentation doesn't really exist... I had to figured some > > stuff by myself to make it works into Kolla project. > > > > On controller nodes you need: > > > > - pacemaker > > - corosync > > - masakari-api (openstack/masakari repository) > > - masakari- engine (openstack/masakari repository) > > > > On compute nodes you need: > > > > - pacemaker-remote (integrated to pacemaker cluster as a resource) > > - masakari- hostmonitor (openstack/masakari-monitor repository) > > - masakari-instancemonitor (openstack/masakari-monitor repository) > > - masakari-processmonitor (openstack/masakari-monitor repository) > > > > For masakari-hostmonitor, the service needs to have access to systemctl > > command (make sure you are not using sysvinit). > > > > For masakari-monitor, the masakari-monitor.conf is a bit different, you > > will have to configure the [api] section properly. > > > > RabbitMQ needs to be configured (as transport_url) on masakari-api and > > masakari-engine too. > > > > Please check this review[1], you will have masakari.conf and > > masakari-monitor.conf configuration examples. > > > > [1] https://review.opendev.org/#/c/615715 > > > > Gaëtan > > > > On Jul 7, 2019 12:08 AM, Vu Tan vungoctan252 at gmail.com> vungoctan252 at gmail.com>>> wrote: > > > > VU TAN VUNGOCTAN252 at GMAIL.COM>> > > > > 10:30 AM (35 minutes ago) > > > > to openstack-discuss > > > > Sorry, I resend this email because I realized that I lacked of prefix > > on this email's subject > > > > Hi, > > > > I would like to use Masakari and I'm having trouble finding a step by > > step or other documentation to get started with. Which part should be > > installed on controller, which is should be on compute, and what is the > > prerequisite to install masakari, I have installed corosync and > > pacemaker on compute and controller nodes, , what else do I need to do > > ? step I have done so far: > > - installed corosync/pacemaker > > - install masakari on compute node on this github repo: > > https://github.com/openstack/masakari > > - add masakari in to mariadb > > here is my configuration file of masakari.conf, do you mind to take a > > look at it, if I have misconfigured anything? > > > > [DEFAULT] > > enabled_apis = masakari_api > > > > # Enable to specify listening IP other than default > > masakari_api_listen = controller > > # Enable to specify port other than default > > masakari_api_listen_port = 15868 > > debug = False > > auth_strategy=keystone > > > > [wsgi] > > # The paste configuration file path > > api_paste_config = /etc/masakari/api-paste.ini > > > > [keystone_authtoken] > > www_authenticate_uri = http://controller:5000 > > auth_url = http://controller:5000 > > auth_type = password > > project_domain_id = default > > user_domain_id = default > > project_name = service > > username = masakari > > password = P at ssword > > > > [database] > > connection = mysql+pymysql://masakari:P at ssword@controller/masakari > Disclaimer: This email and any attachments are sent in strictest > confidence for the sole use of the addressee and may contain legally > privileged, confidential, and proprietary data. If you are not the intended > recipient, please advise the sender by replying promptly to this email and > then delete and destroy this email and any attachments without any further > use, copying or forwarding. > > > Disclaimer: This email and any attachments are sent in strictest > confidence for the sole use of the addressee and may contain legally > privileged, confidential, and proprietary data. If you are not the intended > recipient, please advise the sender by replying promptly to this email and > then delete and destroy this email and any attachments without any further > use, copying or forwarding. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Wed Jul 24 19:37:32 2019 From: eblock at nde.ag (Eugen Block) Date: Wed, 24 Jul 2019 19:37:32 +0000 Subject: [Nova] migration issue In-Reply-To: <1d52d342-ed45-0718-e2ec-e5c5c90d7389@gmail.com> Message-ID: <20190724193732.Horde.cbPZa4wb0ka-Rphu0o4kIuP@webmail.nde.ag> Hi, is this the first migration test or have there already been some successful migrations? If it's the first I would suggest to check if the nova user can access the other node via ssh without password. Is /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 present on the source node? Can you run the 'scp' command manually as nova user? Regards, Eugen Zitat von Budai Laszlo : > Dear all, > > we are testing the cold migratoin of instances that are using local > ephemeral storage and it fails with the following error: > > > 2019-07-24 13:47:13.115 58902 INFO nova.virt.libvirt.driver > [req-356fce31-5e98-425f-89d6-8a98664d31ad > 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - > default default] [instance: 5a6dae > 58-00d6-4317-b635-909fdf09ac49] Instance shutdown successfully after > 3 seconds. > 2019-07-24 13:47:13.126 58902 INFO nova.virt.libvirt.driver [-] > [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Instance destroyed > successfully. > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager > [req-356fce31-5e98-425f-89d6-8a98664d31ad > 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - > default default] [instance: 5a6dae58- > 00d6-4317-b635-909fdf09ac49] Setting instance vm_state to ERROR: > ProcessExecutionError: Unexpected error while running command. > Command: scp -r > /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 > 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph > > Exit code: 1 > Stdout: u'' > Stderr: > u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured > syst > em and your actions will be logged along *\n* with identifying > information. Disconnect immediately if you are not an *\n* > authorized user of this system. > * > \n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No > suc > h file or directory\n' > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] Traceback (most recent call > last): > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line > 75 > 18, in _error_out_instance_on_exception > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] yield > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line > 42 > 75, in _resize_instance > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] timeout, retry_interval) > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", > lin > e 8200, in migrate_disk_and_power_off > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] shared_storage) > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line > 220 > , in __exit__ > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] self.force_reraise() > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line > 196 > , in force_reraise > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] six.reraise(self.type_, > self.value, self.tb) > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", > lin > e 8185, in migrate_disk_and_power_off > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", > line > 226, in copy_image > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs > .py", line 110, in copy_file > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs > .py", line 196, in copy_file > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] on_execute=on_execute, > on_completion=on_completion) > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", > line 231, in exec > ute > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] return > processutils.execute(*cmd, **kwargs) > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py" > , line 424, in execute > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] cmd=sanitized_cmd) > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] ProcessExecutionError: > Unexpected error while running command. > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] Command: scp -r > /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 > 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] Exit code: 1 > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] Stdout: u'' > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] Stderr: > u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or > directory\n' > 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] > 2019-07-24 13:47:14.380 58902 INFO nova.compute.manager > [req-356fce31-5e98-425f-89d6-8a98664d31ad > 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - > default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] > Swapping old allocation on 3da11b23-5ef8-4471-a3c9-5f743b0bbfd7 held > by migration da66c416-2636-421b-99cb-470bd07dad4c for instance > 2019-07-24 13:47:14.979 58902 INFO nova.compute.manager > [req-356fce31-5e98-425f-89d6-8a98664d31ad > 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - > default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] > Successfully reverted task state from resize_migrating on failure > for instance. > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > [req-356fce31-5e98-425f-89d6-8a98664d31ad > 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - > default default] Exception during message handling: > ProcessExecutionError: Unexpected error while running command. > Command: scp -r > /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 > 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 > Exit code: 1 > Stdout: u'' > Stderr: > u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or > directory\n' > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > Traceback (most recent call last): > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in > _process_incoming > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > res = self.dispatcher.dispatch(message) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in > dispatch > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > return self._do_dispatch(endpoint, method, ctxt, args) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in > _do_dispatch > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > result = func(ctxt, **new_args) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in > wrapped > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > function_name, call_dict, binary) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in > __exit__ > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in > force_reraise > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > six.reraise(self.type_, self.value, self.tb) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in > wrapped > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > return f(self, context, *args, **kw) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in > decorated_function > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > "Error: %s", e, instance=instance) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in > __exit__ > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in > force_reraise > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > six.reraise(self.type_, self.value, self.tb) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in > decorated_function > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > return function(self, context, *args, **kwargs) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in > decorated_function > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > return function(self, context, *args, **kwargs) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in > decorated_function > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > kwargs['instance'], e, sys.exc_info()) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in > __exit__ > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in > force_reraise > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > six.reraise(self.type_, self.value, self.tb) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in > decorated_function > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > return function(self, context, *args, **kwargs) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4240, in > resize_instance > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > self._revert_allocation(context, instance, migration) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in > __exit__ > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in > force_reraise > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > six.reraise(self.type_, self.value, self.tb) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4237, in > resize_instance > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > instance_type, clean_shutdown) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4275, in > _resize_instance > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > timeout, retry_interval) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8200, in > migrate_disk_and_power_off > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > shared_storage) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in > __exit__ > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > self.force_reraise() > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in > force_reraise > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > six.reraise(self.type_, self.value, self.tb) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8185, in > migrate_disk_and_power_off > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > compression=compression) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in > copy_image > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > compression=compression) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in > copy_file > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > compression=compression) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in > copy_file > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > on_execute=on_execute, on_completion=on_completion) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", > line 231, in execute > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > return processutils.execute(*cmd, **kwargs) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File > "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in > execute > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > cmd=sanitized_cmd) > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > ProcessExecutionError: Unexpected error while running command. > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > Command: scp -r > /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 > 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Exit code: 1 > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stdout: u'' > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > Stderr: > u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or > directory\n' > 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server > 2019-07-24 13:47:17.065 58902 INFO nova.virt.libvirt.imagecache > [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] image > ca89475a-cf15-46d1-ab2d-c2b96c403248 at > (/var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07): > checking > 2019-07-24 13:47:17.066 58902 INFO nova.virt.libvirt.imagecache > [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] Active base > files: > /var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07 > 2019-07-24 13:47:27.768 58902 INFO nova.compute.manager [-] > [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] VM Stopped > (Lifecycle Event) > 2019-07-24 13:47:27.918 58902 INFO nova.compute.manager > [req-e53e928d-81fe-4185-9e62-abc2113e838c - - - - -] [instance: > 5a6dae58-00d6-4317-b635-909fdf09ac49] During > _sync_instance_power_state the DB power_state (1) does not match the > vm_power_state from the hypervisor (4). Updating power_state in the > DB to match the hypervisor. > > > The disk has its root FS n an RBD device (a volume created by > cinder) and it has its ephemeral disk on a local LVM volume created > by nova. > > It seems that the > /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 is not there when scp is trying to copy it to the other > side. > > Any ideas how to fix this? > > The nova config on the nodes looks like this: > > # Ansible managed > [DEFAULT] > allow_resize_to_same_host = True > # Compute > compute_driver = libvirt.LibvirtDriver > # Scheduler > cpu_allocation_ratio = 2.0 > # Logs / State > debug = False > # Hypervisor > default_ephemeral_format = ext4 > disk_allocation_ratio = 1.0 > # Api's > enabled_apis = osapi_compute,metadata > executor_thread_pool_size = 64 > fatal_deprecations = False > # Configdrive > force_config_drive = False > host = compute8.mgmt.lab.mydomain.com > image_cache_manager_interval = 0 > instance_name_template = instance-%08x > # Ceilometer notification configurations > instance_usage_audit = True > instance_usage_audit_period = hour > instances_path = /var/lib/nova/instances > ## Vif > libvirt_vif_type = ethernet > log_dir = /var/log/nova > # Metadata > metadata_workers = 16 > # Network > my_ip = 192.168.56.47 > notify_on_state_change = vm_and_task_state > osapi_compute_workers = 16 > ram_allocation_ratio = 1.0 > reserved_host_disk_mb = 0 > reserved_host_memory_mb = 2048 > resume_guests_state_on_host_boot = False > rootwrap_config = /etc/nova/rootwrap.conf > rpc_response_timeout = 60 > service_down_time = 120 > state_path = /var/lib/nova > # Rpc all > transport_url = > rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e > 8cf8bd7 at 192.168.56.179:5671//nova > # Disable stderr logging > use_stderr = False > vif_plugging_is_fatal = True > vif_plugging_timeout = 30 > > [api] > auth_strategy = keystone > enable_instance_password = True > use_forwarded_for = False > vendordata_jsonfile_path = /etc/nova/vendor_data.json > > # Cache > [cache] > backend = oslo_cache.memcache_pool > enabled = true > memcache_servers = > 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 > > # Cinder > [cinder] > cafile = /etc/ssl/certs/ca-certificates.crt > catalog_info = volumev3:cinderv3:internalURL > cross_az_attach = True > os_region_name = A_Lab > > [conductor] > workers = 16 > > [filter_scheduler] > available_filters = nova.scheduler.filters.all_filters > enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, > AggregateRamFilter, ComputeFilter, AggregateCoreFilter, DiskFilter, > AggregateDiskFilter, AggregateNumInstancesFilter, > AggregateIoOpsFilter, ComputeCapabilitiesFilter, > ImagePropertiesFilter, ServerGroupAntiAffinityFilter, > ServerGroupAffinityFilter, NUMATopologyFilter, SameHostFilter, > DifferentHostFilter > host_subset_size = 10 > max_instances_per_host = 50 > max_io_ops_per_host = 10 > ram_weight_multiplier = 5.0 > tracks_instance_changes = True > weight_classes = nova.scheduler.weights.all_weighers > > # Glance > [glance] > api_servers = https://vip.mgmt.lab.mydomain.com:9292 > cafile = /etc/ssl/certs/ca-certificates.crt > > [key_manager] > fixed_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > [keystone_authtoken] > auth_type = password > auth_uri = https://vip.mgmt.lab.mydomain.com:5000 > auth_url = https://vip.mgmt.lab.mydomain.com:35357 > cafile = /etc/ssl/certs/ca-certificates.crt > insecure = False > memcache_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > # if your memcached server is shared, use these settings to avoid > cache poisoning > memcache_security_strategy = ENCRYPT > memcached_servers = > 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 > password = xxxxxxxxxxxxxxxxxxxxxxx > > project_domain_id = default > project_name = service > region_name = A_Lab > token_cache_time = 300 > user_domain_id = default > username = nova > > [libvirt] > disk_cachemodes = network=writeback > hw_disk_discard = unmap > images_rbd_ceph_conf = /etc/ceph/ceph.conf > images_rbd_pool = vms > images_type = lvm > images_volume_group = ssd_nova > inject_key = False > inject_partition = -2 > inject_password = False > live_migration_tunnelled = True > live_migration_uri = > "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa" > rbd_secret_uuid = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > # ceph rbd support > rbd_user = cinder > remove_unused_resized_minimum_age_seconds = 3600 > use_virtio_for_bridges = True > virt_type = kvm > > # Neutron > [neutron] > auth_type = password > # Keystone client plugin authentication URL option > auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 > cafile = /etc/ssl/certs/ca-certificates.crt > default_floating_pool = public > insecure = False > metadata_proxy_shared_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxx > # Keystone client plugin password option > password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > project_domain_name = Default > project_name = service > region_name = A_Lab > service_metadata_proxy = True > url = https://vip.mgmt.lab.mydomain.com:9696 > user_domain_name = Default > # Keystone client plugin username option > username = neutron > > [oslo_concurrency] > lock_path = /var/lock/nova > # Notifications > [oslo_messaging_notifications] > driver = messagingv2 > notification_topics = notifications > transport_url = > rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.179:5671//nova > > [oslo_messaging_rabbit] > rpc_conn_pool_size = 30 > ssl = True > > # Placement > [placement] > auth_type = "password" > auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 > cafile = /etc/ssl/certs/ca-certificates.crt > insecure = False > os_interface = internal > os_region_name = A_Lab > password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > project_domain_name = Default > project_name = service > user_domain_name = Default > username = placement > > [quota] > cores = 20 > injected_file_content_bytes = 10240 > injected_file_path_length = 255 > injected_files = 5 > instances = 10 > key_pairs = 100 > max_age = 0 > metadata_items = 128 > ram = 32768 > server_group_members = 10 > server_groups = 10 > > [scheduler] > discover_hosts_in_cells_interval = 60 > host_manager = host_manager > max_attempts = 5 > periodic_task_interval = 60 > scheduler_driver = filter_scheduler > > [spice] > agent_enabled = True > enabled = True > # Console Url and binds > html5proxy_base_url = > https://dashboard.iaas.lab.mydomain.com:6080/spice_auto.html > server_listen = 192.168.56.47 > server_proxyclient_address = 192.168.56.47 > > [upgrade_levels] > compute = auto > > [vnc] > enabled = False > > [wsgi] > api_paste_config = /etc/nova/api-paste.ini > secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO > > > Thank you in advance for any suggestions. > > Kind regards, > Laszlo From kennelson11 at gmail.com Wed Jul 24 20:31:56 2019 From: kennelson11 at gmail.com (Kendall Nelson) Date: Wed, 24 Jul 2019 13:31:56 -0700 Subject: [release] (Belated) Release countdown for week R-12, July 22-26 Message-ID: Hello Everyone, Development Focus ----------------- The Train-2 milestone is July 25! Train-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/train/schedule.html for details. General Information ------------------- Libraries need to be released at least once per milestone period. This week, the release team will propose releases for any library that has not been otherwise released since milestone 1. See list at: http://paste.openstack.org/show/754817/. PTL's and release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well. A +1 would be appreciated, but if we do not hear anything at all by the end of the week, we will assume things are OK to proceed. Remember that non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. This week is also the deadline to freeze the contents of the final release. All new 'Train' deliverables need to have a deliverable file ( https://opendev.org/openstack/releases/src/branch/master/deliverables/train) and need to have done a release by milestone-2. The following new deliverables have not had a release yet (or not been already discussed in the etherpad[1]), and will not be included in Train unless a release is requested for them in the coming week: python-adjutantclient, cyborg-tempest-plugin, neutron-interconnection, openstack-helm, openstack-helm-addons, openstack-help-docs, openstack-helm-images, openstack-helm-infra, and compute-hyperv. The full list of currently deliverable-less deliverables can be found here[2]. Upcoming Deadlines & Dates -------------------------- Train-2 Milestone: July 25 (R-12 week) Thanks! -Kendall Nelson(diablo_rojo) + the Release Management Team [1]https://etherpad.openstack.org/p/train-membership-freeze [2] http://paste.openstack.org/show/754816/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From openstack at fried.cc Wed Jul 24 22:10:27 2019 From: openstack at fried.cc (Eric Fried) Date: Wed, 24 Jul 2019 17:10:27 -0500 Subject: [nova] Train Spec Freeze Message-ID: Hi Nova contributors and maintainers. Spec freeze for Nova is Thursday, July 25th. This means that any specs not approved by EOD [1] become drastically less likely to land in Train. For your convenience, here is a search that's intended to catch open Train specs while filtering out things like updates to existing specs: https://review.opendev.org/#/q/project:openstack/nova-specs+status:open+path:%255Especs/train/approved/.*+AND+added:%253E103 At the time of this writing, there are 15 on that list. Of those, IMO two are "close": Add Unified Limits Spec - https://review.opendev.org/602201 - Two +1s, needs review from "Nova API subteam" Spec for the Nova part of Image Encryption - https://review.opendev.org/608696 - +3, needs a second core If you own a spec you think should be able to land [2], please: - Check it to make sure any comments have been addressed (even if there's no -1). - Explicitly ask for reviews, here on the mailing list and/or in IRC in #openstack-nova. Reviewers, please make an effort to ensure that all the specs on that list have at least one vote. (Currently there are six with no vote.) I'll send an update Friday or Monday with a final tally. Thanks, efried [1] i.e. by the time cores have signed off for the day, wherever they are [2] meaning it has already been through some reviews and seems to have support. If you just proposed your spec this week, it's not reasonable to expect it to land. From rony.khan at brilliant.com.bd Thu Jul 25 05:48:03 2019 From: rony.khan at brilliant.com.bd (Md. Farhad Hasan Khan) Date: Thu, 25 Jul 2019 11:48:03 +0600 Subject: Error: python2-uritemplate conflicts with python2-uri-templates-0.6-5.el7.noarch Message-ID: <060001d542ac$8653e1b0$92fba510$@brilliant.com.bd> Hi, When I run # yum update and found below conflict: --> Processing Conflict: python2-uritemplate-3.0.0-1.el7.noarch conflicts python2-uri-templates --> Finished Dependency Resolution Error: python2-uritemplate conflicts with python2-uri-templates-0.6-5.el7.noarch You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles -nodigest If I remove python2-uri-templates-0.6-5.el7.noarch than below package ask to remove: Removing: python2-uri-templates noarch 0.6-5.el7 @centos-openstack-rocky 20 k Removing for dependencies: openstack-cinder noarch 1:13.0.5-1.el7 @centos-openstack-rocky 200 k python-cinder noarch 1:13.0.5-1.el7 @centos-openstack-rocky 19 M python2-google-api-client noarch 1.4.2-4.el7 @centos-openstack-rocky 305 k please let help me how can I update. Thanks/Rony -------------- next part -------------- An HTML attachment was scrubbed... URL: From berndbausch at gmail.com Thu Jul 25 06:18:27 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Thu, 25 Jul 2019 15:18:27 +0900 Subject: [keystone] Need clarification about Stein policies Message-ID: <701cdad6-e0b4-4705-ee11-4b81db307612@gmail.com> The Keystone policy.json file I created with oslo-policy-generator contains lines I don't understand. For example /list_users/. The comment says: # DEPRECATED "identity:list_users":"rule:admin_required" has been # deprecated since S in favor of "identity:list_users":"(role:reader # and system_scope:all) or (role:reader and # domain_id:%(target.domain_id)s)". I do understand the expression starting with (role:reader .... , but contrarily to the comment, the policy is "identity:list_users": "rule:identity:list_users" This looks like a circular definition, and in any case, nowhere do I seerule:identity:list_users defined. Can someone in the know explain how this policy is processed? Thanks much, Bernd -------------- next part -------------- An HTML attachment was scrubbed... URL: From laszlo.budai at gmail.com Thu Jul 25 06:55:13 2019 From: laszlo.budai at gmail.com (Budai Laszlo) Date: Thu, 25 Jul 2019 09:55:13 +0300 Subject: [Nova] migration issue In-Reply-To: <20190724193732.Horde.cbPZa4wb0ka-Rphu0o4kIuP@webmail.nde.ag> References: <20190724193732.Horde.cbPZa4wb0ka-Rphu0o4kIuP@webmail.nde.ag> Message-ID: Hi Eugen, Thank you for your suggestions. I have tested, and the nova user is able to scp from one compute host to the other. I have also tested the migration without the local LVM ephemeral disc, and it is working in that case. The issue seems to appear when I'm trying to use local ephemeral disk managed by LVM (images_type=lvm). In this case my ephemeral disk is created on a local LVM volume that is part of the ssd_nova VG (images_volume_group = ssd_nova). I could not see the source file that the nova is trying to copy, but this may have two reasons: 1. The file is not created 2. it's life is very short, and I cannot observe it. The directory /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize doesn't exists while my instance is running. It appears for a short period of time (fraction of a second) while the migrate is trying to do its job, and then disappears again. Kind regards, Laszlo On 7/24/19 10:37 PM, Eugen Block wrote: > Hi, > > is this the first migration test or have there already been some successful migrations? If it's the first I would suggest to check if the nova user can access the other node via ssh without password. Is /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 present on the source node? Can you run the 'scp' command manually as nova user? > > Regards, > Eugen > > > Zitat von Budai Laszlo : > >> Dear all, >> >> we are testing the cold migratoin of instances that are using local ephemeral storage and it fails with the following error: >> >> >> 2019-07-24 13:47:13.115 58902 INFO nova.virt.libvirt.driver [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae >> 58-00d6-4317-b635-909fdf09ac49] Instance shutdown successfully after 3 seconds. >> 2019-07-24 13:47:13.126 58902 INFO nova.virt.libvirt.driver [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Instance destroyed successfully. >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58- >> 00d6-4317-b635-909fdf09ac49] Setting instance vm_state to ERROR: ProcessExecutionError: Unexpected error while running command. >> Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph >> >> Exit code: 1 >> Stdout: u'' >> Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured syst >> em and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            * >> \n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No suc >> h file or directory\n' >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Traceback (most recent call last): >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 75 >> 18, in _error_out_instance_on_exception >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     yield >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 42 >> 75, in _resize_instance >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     timeout, retry_interval) >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin >> e 8200, in migrate_disk_and_power_off >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     shared_storage) >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220 >> , in __exit__ >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     self.force_reraise() >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196 >> , in force_reraise >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     six.reraise(self.type_, self.value, self.tb) >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin >> e 8185, in migrate_disk_and_power_off >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression) >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line >>  226, in copy_image >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression) >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs >> .py", line 110, in copy_file >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression) >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs >> .py", line 196, in copy_file >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     on_execute=on_execute, on_completion=on_completion) >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in exec >> ute >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     return processutils.execute(*cmd, **kwargs) >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py" >> , line 424, in execute >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     cmd=sanitized_cmd) >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] ProcessExecutionError: Unexpected error while running command. >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Exit code: 1 >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stdout: u'' >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' >> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >> 2019-07-24 13:47:14.380 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Swapping old allocation on 3da11b23-5ef8-4471-a3c9-5f743b0bbfd7 held by migration da66c416-2636-421b-99cb-470bd07dad4c for instance >> 2019-07-24 13:47:14.979 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Successfully reverted task state from resize_migrating on failure for instance. >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] Exception during message handling: ProcessExecutionError: Unexpected error while running command. >> Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >> Exit code: 1 >> Stdout: u'' >> Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Traceback (most recent call last): >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise() >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in decorated_function >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise() >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in decorated_function >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in decorated_function >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info()) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise() >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4240, in resize_instance >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self._revert_allocation(context, instance, migration) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise() >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4237, in resize_instance >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     instance_type, clean_shutdown) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4275, in _resize_instance >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     timeout, retry_interval) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8200, in migrate_disk_and_power_off >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     shared_storage) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise() >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8185, in migrate_disk_and_power_off >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     compression=compression) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in copy_image >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     compression=compression) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in copy_file >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     compression=compression) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in copy_file >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     on_execute=on_execute, on_completion=on_completion) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in execute >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd) >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command. >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Exit code: 1 >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stdout: u'' >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' >> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >> 2019-07-24 13:47:17.065 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] image ca89475a-cf15-46d1-ab2d-c2b96c403248 at (/var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07): checking >> 2019-07-24 13:47:17.066 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] Active base files: /var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07 >> 2019-07-24 13:47:27.768 58902 INFO nova.compute.manager [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] VM Stopped (Lifecycle Event) >> 2019-07-24 13:47:27.918 58902 INFO nova.compute.manager [req-e53e928d-81fe-4185-9e62-abc2113e838c - - - - -] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. >> >> >> The disk has its root FS n an RBD device (a volume created by cinder) and it has its ephemeral disk on a local LVM volume created by nova. >> >> It seems that the /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 is not there when scp is trying to copy it to the other side. >> >> Any ideas how to fix this? >> >> The nova config on the nodes looks like this: >> >> # Ansible managed >> [DEFAULT] >> allow_resize_to_same_host = True >> # Compute >> compute_driver = libvirt.LibvirtDriver >> # Scheduler >> cpu_allocation_ratio = 2.0 >> # Logs / State >> debug = False >> # Hypervisor >> default_ephemeral_format = ext4 >> disk_allocation_ratio = 1.0 >> # Api's >> enabled_apis = osapi_compute,metadata >> executor_thread_pool_size = 64 >> fatal_deprecations = False >> # Configdrive >> force_config_drive = False >> host = compute8.mgmt.lab.mydomain.com >> image_cache_manager_interval = 0 >> instance_name_template = instance-%08x >> # Ceilometer notification configurations >> instance_usage_audit = True >> instance_usage_audit_period = hour >> instances_path = /var/lib/nova/instances >> ## Vif >> libvirt_vif_type = ethernet >> log_dir = /var/log/nova >> # Metadata >> metadata_workers = 16 >> # Network >> my_ip = 192.168.56.47 >> notify_on_state_change = vm_and_task_state >> osapi_compute_workers = 16 >> ram_allocation_ratio = 1.0 >> reserved_host_disk_mb = 0 >> reserved_host_memory_mb = 2048 >> resume_guests_state_on_host_boot = False >> rootwrap_config = /etc/nova/rootwrap.conf >> rpc_response_timeout = 60 >> service_down_time = 120 >> state_path = /var/lib/nova >> # Rpc all >> transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e >> 8cf8bd7 at 192.168.56.179:5671//nova >> # Disable stderr logging >> use_stderr = False >> vif_plugging_is_fatal = True >> vif_plugging_timeout = 30 >> >> [api] >> auth_strategy = keystone >> enable_instance_password = True >> use_forwarded_for = False >> vendordata_jsonfile_path = /etc/nova/vendor_data.json >> >> # Cache >> [cache] >> backend = oslo_cache.memcache_pool >> enabled = true >> memcache_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 >> >> # Cinder >> [cinder] >> cafile = /etc/ssl/certs/ca-certificates.crt >> catalog_info = volumev3:cinderv3:internalURL >> cross_az_attach = True >> os_region_name = A_Lab >> >> [conductor] >> workers = 16 >> >> [filter_scheduler] >> available_filters = nova.scheduler.filters.all_filters >> enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, AggregateRamFilter, ComputeFilter, AggregateCoreFilter, DiskFilter, AggregateDiskFilter, AggregateNumInstancesFilter, AggregateIoOpsFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, NUMATopologyFilter, SameHostFilter, DifferentHostFilter >> host_subset_size = 10 >> max_instances_per_host = 50 >> max_io_ops_per_host = 10 >> ram_weight_multiplier = 5.0 >> tracks_instance_changes = True >> weight_classes = nova.scheduler.weights.all_weighers >> >> # Glance >> [glance] >> api_servers = https://vip.mgmt.lab.mydomain.com:9292 >> cafile = /etc/ssl/certs/ca-certificates.crt >> >> [key_manager] >> fixed_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> >> [keystone_authtoken] >> auth_type = password >> auth_uri = https://vip.mgmt.lab.mydomain.com:5000 >> auth_url = https://vip.mgmt.lab.mydomain.com:35357 >> cafile = /etc/ssl/certs/ca-certificates.crt >> insecure = False >> memcache_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> # if your memcached server is shared, use these settings to avoid cache poisoning >> memcache_security_strategy = ENCRYPT >> memcached_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 >> password = xxxxxxxxxxxxxxxxxxxxxxx >> >> project_domain_id = default >> project_name = service >> region_name = A_Lab >> token_cache_time = 300 >> user_domain_id = default >> username = nova >> >> [libvirt] >> disk_cachemodes = network=writeback >> hw_disk_discard = unmap >> images_rbd_ceph_conf = /etc/ceph/ceph.conf >> images_rbd_pool = vms >> images_type = lvm >> images_volume_group = ssd_nova >> inject_key = False >> inject_partition = -2 >> inject_password = False >> live_migration_tunnelled = True >> live_migration_uri = "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa" >> rbd_secret_uuid = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> # ceph rbd support >> rbd_user = cinder >> remove_unused_resized_minimum_age_seconds = 3600 >> use_virtio_for_bridges = True >> virt_type = kvm >> >> # Neutron >> [neutron] >> auth_type = password >> # Keystone client plugin authentication URL option >> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 >> cafile = /etc/ssl/certs/ca-certificates.crt >> default_floating_pool = public >> insecure = False >> metadata_proxy_shared_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxx >> # Keystone client plugin password option >> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> project_domain_name = Default >> project_name = service >> region_name = A_Lab >> service_metadata_proxy = True >> url = https://vip.mgmt.lab.mydomain.com:9696 >> user_domain_name = Default >> # Keystone client plugin username option >> username = neutron >> >> [oslo_concurrency] >> lock_path = /var/lock/nova >> # Notifications >> [oslo_messaging_notifications] >> driver = messagingv2 >> notification_topics = notifications >> transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.179:5671//nova >> >> [oslo_messaging_rabbit] >> rpc_conn_pool_size = 30 >> ssl = True >> >> # Placement >> [placement] >> auth_type = "password" >> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 >> cafile = /etc/ssl/certs/ca-certificates.crt >> insecure = False >> os_interface = internal >> os_region_name = A_Lab >> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> project_domain_name = Default >> project_name = service >> user_domain_name = Default >> username = placement >> >> [quota] >> cores = 20 >> injected_file_content_bytes = 10240 >> injected_file_path_length = 255 >> injected_files = 5 >> instances = 10 >> key_pairs = 100 >> max_age = 0 >> metadata_items = 128 >> ram = 32768 >> server_group_members = 10 >> server_groups = 10 >> >> [scheduler] >> discover_hosts_in_cells_interval = 60 >> host_manager = host_manager >> max_attempts = 5 >> periodic_task_interval = 60 >> scheduler_driver = filter_scheduler >> >> [spice] >> agent_enabled = True >> enabled = True >> # Console Url and binds >> html5proxy_base_url = https://dashboard.iaas.lab.mydomain.com:6080/spice_auto.html >> server_listen = 192.168.56.47 >> server_proxyclient_address = 192.168.56.47 >> >> [upgrade_levels] >> compute = auto >> >> [vnc] >> enabled = False >> >> [wsgi] >> api_paste_config = /etc/nova/api-paste.ini >> secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO >> >> >> Thank you in advance for any suggestions. >> >> Kind regards, >> Laszlo > > > > > From katonalala at gmail.com Thu Jul 25 07:33:22 2019 From: katonalala at gmail.com (Lajos Katona) Date: Thu, 25 Jul 2019 09:33:22 +0200 Subject: DuplicateMessageError after restart of a control node In-Reply-To: <26eea5c6-fd23-a9e4-368a-c0635878c903@everyware.ch> References: <8c01078d-1c9a-3153-1856-cf9b3ef0f3d1@everyware.ch> <26eea5c6-fd23-a9e4-368a-c0635878c903@everyware.ch> Message-ID: Hi, I am not sure about the actual default value for notification_format (I can recall that there was some debate recently in nova community), but the solution should be to select unversioned, as most consumers of nova notifications use the legacy unversioned notifications, so if the config is both, the new versioned notifications can cause trouble on message bus as nobody fetch them. Lajos Pawel Konczalski ezt írta (időpont: 2019. júl. 24., Sze, 18:42): > Hello everybody, > > after some investigation in the RabbitMQ problems we found some > duplicated messages and timeouts in logs. Restarting the whole RabbitMQ > cluster (stop all rabbitmq containers and start one by one) solved the > problems for now. > > The main cause for this issue seams to by the nova notifications > configuration with was deployed by kolla-ansible. If searchlight is not > installed the 'notifications/notification_format' should be > 'unversioned'. Default is 'both' so nova will send a notification to the > queue versioned_notifications with has no consumer. In our case the > queue got huge amount of messages with made the rabbitmq cluster more > and more unstable, see: > https://bugzilla.redhat.com/show_bug.cgi?id=1592528 > > Following settings in nova.conf may solve this issue but we didn`t > tested this yet: > [notification] > notification_format = unversioned > > BR > > Pawel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Thu Jul 25 07:54:36 2019 From: eblock at nde.ag (Eugen Block) Date: Thu, 25 Jul 2019 07:54:36 +0000 Subject: [Nova] migration issue In-Reply-To: References: <20190724193732.Horde.cbPZa4wb0ka-Rphu0o4kIuP@webmail.nde.ag> Message-ID: <20190725075436.Horde.2ZCfqrsgjQ-ugCgV6d4WnQ5@webmail.nde.ag> I'm not sure if I get this right, according to your config output you already use ceph as storage backend (which already provides ephemeral disk if configured to do so) but you also want to configure lvm? Can you explain what your goal is here? Zitat von Budai Laszlo : > Hi Eugen, > > Thank you for your suggestions. I have tested, and the nova user is > able to scp from one compute host to the other. I have also tested > the migration without the local LVM ephemeral disc, and it is > working in that case. The issue seems to appear when I'm trying to > use local ephemeral disk managed by LVM (images_type=lvm). > In this case my ephemeral disk is created on a local LVM volume that > is part of the ssd_nova VG (images_volume_group = ssd_nova). > > I could not see the source file that the nova is trying to copy, but > this may have two reasons: > 1. The file is not created > 2. it's life is very short, and I cannot observe it. > > The directory > /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize > doesn't exists while my instance is running. It appears for a short > period of time (fraction of a second) while the migrate is trying to > do its job, and then disappears again. > > Kind regards, > Laszlo > > On 7/24/19 10:37 PM, Eugen Block wrote: >> Hi, >> >> is this the first migration test or have there already been some >> successful migrations? If it's the first I would suggest to check >> if the nova user can access the other node via ssh without >> password. Is >> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 present on the source node? Can you run the 'scp' command manually as nova >> user? >> >> Regards, >> Eugen >> >> >> Zitat von Budai Laszlo : >> >>> Dear all, >>> >>> we are testing the cold migratoin of instances that are using >>> local ephemeral storage and it fails with the following error: >>> >>> >>> 2019-07-24 13:47:13.115 58902 INFO nova.virt.libvirt.driver >>> [req-356fce31-5e98-425f-89d6-8a98664d31ad >>> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 >>> - default default] [instance: 5a6dae >>> 58-00d6-4317-b635-909fdf09ac49] Instance shutdown successfully >>> after 3 seconds. >>> 2019-07-24 13:47:13.126 58902 INFO nova.virt.libvirt.driver [-] >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Instance >>> destroyed successfully. >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [req-356fce31-5e98-425f-89d6-8a98664d31ad >>> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 >>> - default default] [instance: 5a6dae58- >>> 00d6-4317-b635-909fdf09ac49] Setting instance vm_state to ERROR: >>> ProcessExecutionError: Unexpected error while running command. >>> Command: scp -r >>> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph >>> >>> Exit code: 1 >>> Stdout: u'' >>> Stderr: >>> u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured >>> syst >>> em and your actions will be logged along   *\n* with identifying >>> information. Disconnect immediately if you are not an     *\n* >>> authorized user of this >>> system.                                            * >>> \n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No >>> suc >>> h file or directory\n' >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Traceback (most >>> recent call last): >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line >>> 75 >>> 18, in _error_out_instance_on_exception >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     yield >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line >>> 42 >>> 75, in _resize_instance >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     timeout, >>> retry_interval) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", >>> lin >>> e 8200, in migrate_disk_and_power_off >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     shared_storage) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line >>> 220 >>> , in __exit__ >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     >>> self.force_reraise() >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line >>> 196 >>> , in force_reraise >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", >>> lin >>> e 8185, in migrate_disk_and_power_off >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     >>> compression=compression) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", >>> line >>>  226, in copy_image >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     >>> compression=compression) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs >>> .py", line 110, in copy_file >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     >>> compression=compression) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs >>> .py", line 196, in copy_file >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     >>> on_execute=on_execute, on_completion=on_completion) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", >>> line 231, in exec >>> ute >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     return >>> processutils.execute(*cmd, **kwargs) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py" >>> , line 424, in execute >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     >>> cmd=sanitized_cmd) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>> ProcessExecutionError: Unexpected error while running command. >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Command: scp -r >>> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Exit code: 1 >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stdout: u'' >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stderr: >>> u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or >>> directory\n' >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>> 2019-07-24 13:47:14.380 58902 INFO nova.compute.manager >>> [req-356fce31-5e98-425f-89d6-8a98664d31ad >>> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 >>> - default default] [instance: >>> 5a6dae58-00d6-4317-b635-909fdf09ac49] Swapping old allocation on >>> 3da11b23-5ef8-4471-a3c9-5f743b0bbfd7 held by migration >>> da66c416-2636-421b-99cb-470bd07dad4c for instance >>> 2019-07-24 13:47:14.979 58902 INFO nova.compute.manager >>> [req-356fce31-5e98-425f-89d6-8a98664d31ad >>> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 >>> - default default] [instance: >>> 5a6dae58-00d6-4317-b635-909fdf09ac49] Successfully reverted task >>> state from resize_migrating on failure for instance. >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> [req-356fce31-5e98-425f-89d6-8a98664d31ad >>> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 >>> - default default] Exception during message handling: >>> ProcessExecutionError: Unexpected error while running command. >>> Command: scp -r >>> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> Exit code: 1 >>> Stdout: u'' >>> Stderr: >>> u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or >>> directory\n' >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> Traceback (most recent call last): >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in >>> _process_incoming >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> res = self.dispatcher.dispatch(message) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in >>> dispatch >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> return self._do_dispatch(endpoint, method, ctxt, args) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in >>> _do_dispatch >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> result = func(ctxt, **new_args) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in >>> wrapped >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> function_name, call_dict, binary) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in >>> __exit__ >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> self.force_reraise() >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in >>> force_reraise >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in >>> wrapped >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> return f(self, context, *args, **kw) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in >>> decorated_function >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> "Error: %s", e, instance=instance) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in >>> __exit__ >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> self.force_reraise() >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in >>> force_reraise >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in >>> decorated_function >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> return function(self, context, *args, **kwargs) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in >>> decorated_function >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> return function(self, context, *args, **kwargs) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in >>> decorated_function >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> kwargs['instance'], e, sys.exc_info()) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in >>> __exit__ >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> self.force_reraise() >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in >>> force_reraise >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in >>> decorated_function >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> return function(self, context, *args, **kwargs) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4240, in >>> resize_instance >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> self._revert_allocation(context, instance, migration) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in >>> __exit__ >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> self.force_reraise() >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in >>> force_reraise >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4237, in >>> resize_instance >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> instance_type, clean_shutdown) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4275, in >>> _resize_instance >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> timeout, retry_interval) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8200, in >>> migrate_disk_and_power_off >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> shared_storage) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in >>> __exit__ >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> self.force_reraise() >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in >>> force_reraise >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8185, in >>> migrate_disk_and_power_off >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> compression=compression) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in >>> copy_image >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> compression=compression) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in >>> copy_file >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> compression=compression) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in >>> copy_file >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> on_execute=on_execute, on_completion=on_completion) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", >>> line 231, in execute >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> return processutils.execute(*cmd, **kwargs) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in >>> execute >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     >>> cmd=sanitized_cmd) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> ProcessExecutionError: Unexpected error while running command. >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> Command: scp -r >>> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Exit code: 1 >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stdout: u'' >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> Stderr: >>> u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or >>> directory\n' >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> 2019-07-24 13:47:17.065 58902 INFO nova.virt.libvirt.imagecache >>> [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] image >>> ca89475a-cf15-46d1-ab2d-c2b96c403248 at >>> (/var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07): >>> checking >>> 2019-07-24 13:47:17.066 58902 INFO nova.virt.libvirt.imagecache >>> [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] Active base >>> files: >>> /var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07 >>> 2019-07-24 13:47:27.768 58902 INFO nova.compute.manager [-] >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] VM Stopped >>> (Lifecycle Event) >>> 2019-07-24 13:47:27.918 58902 INFO nova.compute.manager >>> [req-e53e928d-81fe-4185-9e62-abc2113e838c - - - - -] [instance: >>> 5a6dae58-00d6-4317-b635-909fdf09ac49] During >>> _sync_instance_power_state the DB power_state (1) does not match >>> the vm_power_state from the hypervisor (4). Updating power_state >>> in the DB to match the hypervisor. >>> >>> >>> The disk has its root FS n an RBD device (a volume created by >>> cinder) and it has its ephemeral disk on a local LVM volume >>> created by nova. >>> >>> It seems that the >>> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 is not there when scp is trying to copy it to the other >>> side. >>> >>> Any ideas how to fix this? >>> >>> The nova config on the nodes looks like this: >>> >>> # Ansible managed >>> [DEFAULT] >>> allow_resize_to_same_host = True >>> # Compute >>> compute_driver = libvirt.LibvirtDriver >>> # Scheduler >>> cpu_allocation_ratio = 2.0 >>> # Logs / State >>> debug = False >>> # Hypervisor >>> default_ephemeral_format = ext4 >>> disk_allocation_ratio = 1.0 >>> # Api's >>> enabled_apis = osapi_compute,metadata >>> executor_thread_pool_size = 64 >>> fatal_deprecations = False >>> # Configdrive >>> force_config_drive = False >>> host = compute8.mgmt.lab.mydomain.com >>> image_cache_manager_interval = 0 >>> instance_name_template = instance-%08x >>> # Ceilometer notification configurations >>> instance_usage_audit = True >>> instance_usage_audit_period = hour >>> instances_path = /var/lib/nova/instances >>> ## Vif >>> libvirt_vif_type = ethernet >>> log_dir = /var/log/nova >>> # Metadata >>> metadata_workers = 16 >>> # Network >>> my_ip = 192.168.56.47 >>> notify_on_state_change = vm_and_task_state >>> osapi_compute_workers = 16 >>> ram_allocation_ratio = 1.0 >>> reserved_host_disk_mb = 0 >>> reserved_host_memory_mb = 2048 >>> resume_guests_state_on_host_boot = False >>> rootwrap_config = /etc/nova/rootwrap.conf >>> rpc_response_timeout = 60 >>> service_down_time = 120 >>> state_path = /var/lib/nova >>> # Rpc all >>> transport_url = >>> rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e >>> 8cf8bd7 at 192.168.56.179:5671//nova >>> # Disable stderr logging >>> use_stderr = False >>> vif_plugging_is_fatal = True >>> vif_plugging_timeout = 30 >>> >>> [api] >>> auth_strategy = keystone >>> enable_instance_password = True >>> use_forwarded_for = False >>> vendordata_jsonfile_path = /etc/nova/vendor_data.json >>> >>> # Cache >>> [cache] >>> backend = oslo_cache.memcache_pool >>> enabled = true >>> memcache_servers = >>> 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 >>> >>> # Cinder >>> [cinder] >>> cafile = /etc/ssl/certs/ca-certificates.crt >>> catalog_info = volumev3:cinderv3:internalURL >>> cross_az_attach = True >>> os_region_name = A_Lab >>> >>> [conductor] >>> workers = 16 >>> >>> [filter_scheduler] >>> available_filters = nova.scheduler.filters.all_filters >>> enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, >>> AggregateRamFilter, ComputeFilter, AggregateCoreFilter, >>> DiskFilter, AggregateDiskFilter, AggregateNumInstancesFilter, >>> AggregateIoOpsFilter, ComputeCapabilitiesFilter, >>> ImagePropertiesFilter, ServerGroupAntiAffinityFilter, >>> ServerGroupAffinityFilter, NUMATopologyFilter, SameHostFilter, >>> DifferentHostFilter >>> host_subset_size = 10 >>> max_instances_per_host = 50 >>> max_io_ops_per_host = 10 >>> ram_weight_multiplier = 5.0 >>> tracks_instance_changes = True >>> weight_classes = nova.scheduler.weights.all_weighers >>> >>> # Glance >>> [glance] >>> api_servers = https://vip.mgmt.lab.mydomain.com:9292 >>> cafile = /etc/ssl/certs/ca-certificates.crt >>> >>> [key_manager] >>> fixed_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> >>> [keystone_authtoken] >>> auth_type = password >>> auth_uri = https://vip.mgmt.lab.mydomain.com:5000 >>> auth_url = https://vip.mgmt.lab.mydomain.com:35357 >>> cafile = /etc/ssl/certs/ca-certificates.crt >>> insecure = False >>> memcache_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> # if your memcached server is shared, use these settings to avoid >>> cache poisoning >>> memcache_security_strategy = ENCRYPT >>> memcached_servers = >>> 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 >>> password = xxxxxxxxxxxxxxxxxxxxxxx >>> >>> project_domain_id = default >>> project_name = service >>> region_name = A_Lab >>> token_cache_time = 300 >>> user_domain_id = default >>> username = nova >>> >>> [libvirt] >>> disk_cachemodes = network=writeback >>> hw_disk_discard = unmap >>> images_rbd_ceph_conf = /etc/ceph/ceph.conf >>> images_rbd_pool = vms >>> images_type = lvm >>> images_volume_group = ssd_nova >>> inject_key = False >>> inject_partition = -2 >>> inject_password = False >>> live_migration_tunnelled = True >>> live_migration_uri = >>> "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa" >>> rbd_secret_uuid = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> # ceph rbd support >>> rbd_user = cinder >>> remove_unused_resized_minimum_age_seconds = 3600 >>> use_virtio_for_bridges = True >>> virt_type = kvm >>> >>> # Neutron >>> [neutron] >>> auth_type = password >>> # Keystone client plugin authentication URL option >>> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 >>> cafile = /etc/ssl/certs/ca-certificates.crt >>> default_floating_pool = public >>> insecure = False >>> metadata_proxy_shared_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> # Keystone client plugin password option >>> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> project_domain_name = Default >>> project_name = service >>> region_name = A_Lab >>> service_metadata_proxy = True >>> url = https://vip.mgmt.lab.mydomain.com:9696 >>> user_domain_name = Default >>> # Keystone client plugin username option >>> username = neutron >>> >>> [oslo_concurrency] >>> lock_path = /var/lock/nova >>> # Notifications >>> [oslo_messaging_notifications] >>> driver = messagingv2 >>> notification_topics = notifications >>> transport_url = >>> rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.179:5671//nova >>> >>> [oslo_messaging_rabbit] >>> rpc_conn_pool_size = 30 >>> ssl = True >>> >>> # Placement >>> [placement] >>> auth_type = "password" >>> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 >>> cafile = /etc/ssl/certs/ca-certificates.crt >>> insecure = False >>> os_interface = internal >>> os_region_name = A_Lab >>> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> project_domain_name = Default >>> project_name = service >>> user_domain_name = Default >>> username = placement >>> >>> [quota] >>> cores = 20 >>> injected_file_content_bytes = 10240 >>> injected_file_path_length = 255 >>> injected_files = 5 >>> instances = 10 >>> key_pairs = 100 >>> max_age = 0 >>> metadata_items = 128 >>> ram = 32768 >>> server_group_members = 10 >>> server_groups = 10 >>> >>> [scheduler] >>> discover_hosts_in_cells_interval = 60 >>> host_manager = host_manager >>> max_attempts = 5 >>> periodic_task_interval = 60 >>> scheduler_driver = filter_scheduler >>> >>> [spice] >>> agent_enabled = True >>> enabled = True >>> # Console Url and binds >>> html5proxy_base_url = >>> https://dashboard.iaas.lab.mydomain.com:6080/spice_auto.html >>> server_listen = 192.168.56.47 >>> server_proxyclient_address = 192.168.56.47 >>> >>> [upgrade_levels] >>> compute = auto >>> >>> [vnc] >>> enabled = False >>> >>> [wsgi] >>> api_paste_config = /etc/nova/api-paste.ini >>> secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO >>> >>> >>> Thank you in advance for any suggestions. >>> >>> Kind regards, >>> Laszlo >> >> >> >> >> From geguileo at redhat.com Thu Jul 25 08:14:07 2019 From: geguileo at redhat.com (Gorka Eguileor) Date: Thu, 25 Jul 2019 10:14:07 +0200 Subject: [kolla][nova][cinder] Got Gateway-Timeout error on VM evacuation if it has volume attached. In-Reply-To: References: Message-ID: <20190725081407.lkapi4skj2hcllzi@localhost> On 23/07, Eddie Yen wrote: > Hi Matt, thanks for your reply first. > > The log I paste is from nova-compute. > And I also check cinder-api & cinder-volume logs according from timestamp. > Strange is, no error messages found during that time. Hi, It could make sense that you see no errors in Cinder. The error from your pastebin is not coming from Cinder, it is coming from your HAProxy (or whatever load balancer you have in front of the Cinder-API nodes). Attachment delete is a synchronous operation, so all the different connection timeouts may affect the operation: Nova to HAProxy, HAProxy to Cinder-API, Cinder-API to Cinder-Volume via RabbitMQ, Cinder-Volume to Storage backend. I would recommend you looking at the specific attachment_delete request that failed in Cinder logs and see how long it took to complete, and then check how long it took for the 504 error to happen. With that info you can get an idea of how much higher your timeout must be. It could also happen that the Cinder-API raises a timeout error when calling the Cinder-Volume. In this case you should check the cinder-volume service to see how long it took it to complete, as the operation continues. Internally the Cinder-API to Cinder-Volume timeout is usually around 60 seconds (rpc_response_timeout). You need to ensure that your HAProxy and Cinder RPC timeouts are in sync and are enough for the operation to complete on the worst case scenario. Cheers, Gorka. > I remember I launch evacuation on the host. > > Perhaps it's over-loading but it's not happening on the cinder. Because the > environment is 3 all-in-one installation model. > That means control+compute per node, and 3 nodes become controller HA. > When I shutdown one of the node, I found all requests from API is pretty > slow (can feel that when using dashboard.) > And back to normal again when the node is back. > > I'll try do the evacuation again but with just disable nova host or stop > nova services, to test if that happen again or not. > > Matt Riedemann 於 2019年7月23日 週二 上午6:40寫道： > > > On 7/18/2019 3:53 AM, Eddie Yen wrote: > > > Before I try to evacuate host, the source host had about 24 VMs running. > > > When I shutdown the node and execute evacuation, there're few VMs > > > failed. The error code is 504. > > > Strange is those VMs are all attach its own volume. > > > > > > Then I check nova-compute log, a detailed error has pasted at below link; > > > https://pastebin.com/uaE7YrP1 > > > > > > Does anyone have any experience with this? I googled but no enough > > > information about this. > > > > Are there errors in the cinder-api logs during the evacuate of all VMs > > from the host? Are you doing the evacuate operation on all VMs on the > > host concurrently or in serial? I wonder if you're over-loading cinder > > and that's causing the timeout somehow. The timeout from cinder is when > > deleting volume attachment records, which would be terminating > > connections in the storage backend under the covers. Check the > > cinder-volume logs for errors as well. > > > > -- > > > > Thanks, > > > > Matt > > > > From masha.atakova at mail.com Thu Jul 25 08:41:09 2019 From: masha.atakova at mail.com (Mary Atakova) Date: Thu, 25 Jul 2019 11:41:09 +0300 Subject: [nova] [horizon] Horizon and nova-cli inconsistency in live migration parameters In-Reply-To: <08d8b7bb-cb22-a0c8-82b4-61a4739088f3@gmail.com> References: <98059178-00c5-8bc0-7bbf-bf5c55c53397@mail.com> <08d8b7bb-cb22-a0c8-82b4-61a4739088f3@gmail.com> Message-ID: Matt, Thank you so much for the clarification! I'm glad it's not a bug, and now it will be easier to understand the difference between the behaviours and adjust accordingly. Best regards, Mary On 23/07/2019 00:21, Matt Riedemann wrote: > On 7/14/2019 7:35 AM, Mary Atakova wrote: >> I try to live-migrate a VM, and I do it through nova cli and through >> Horizon, and nova-api logs contain different data depending on the >> way I do it. >> >> 1. nova cli >> >> I run the command: >> >> nova live-migration 17359460-d23c-4acc-a9b1-5cf117b54430 >> >> and it shows up in nova-api logs as: >> >> {"os-migrateLive": {"block_migration": "auto", "host": null}} >> >> 2. horizon >> >> I run live migration via Horizon with unchecked Disk Overcommit and >> unchecked Block Migration. >> >> It shows up in nova-api logs as: >> >> {"os-migrateLive": {"disk_over_commit": false, "block_migration": >> false, "host": null}} >> >> This difference can lead to different parameters provided for live >> migration when block_migration: auto is translated into >> block_migration: true. > > The difference is in the compute API microversion used. The nova CLI > by default negotiates the highest microversion to use between what the > client understands and what the server provides. The 'auto' value for > the block_migration parameter was added in 2.25 and disk_over_commit > was removed in 2.25. Horizon is likely not specifying a specific > microversion so it defaults to 2.1 and that's why you're getting > "disk_over_commit": false, "block_migration": false because you > unchecked those check boxes. > > In other words, for Horizon to support compute API 2.25 there would be > an "Auto" option for the Block Migration value. > From ghadge.dhairya23 at gmail.com Thu Jul 25 09:36:58 2019 From: ghadge.dhairya23 at gmail.com (Dhairyasheel Ghadge) Date: Thu, 25 Jul 2019 15:06:58 +0530 Subject: Heat resource types - aodh and ceilometer Message-ID: Hello all, I have installed heat engine along with heat dashboard for orchestration in openstack cloud. I am creating a template to autoscale a web server pool and spin a new VM when load on other one increases. For the autoscaling purpose, I require to have an alarming system which will monitor the cp meter of the instance, which is provided by ceilometer and aodh services of openstack. I cannot find resource types of aodh and ceilometer on the heat dashboard. These same resource types are present in the heat openstack resources directory. Are there any steps to be carried out to make it available on the dashboard? Thanks & Regards Dhairyasheel From laszlo.budai at gmail.com Thu Jul 25 09:39:13 2019 From: laszlo.budai at gmail.com (Budai Laszlo) Date: Thu, 25 Jul 2019 12:39:13 +0300 Subject: [Nova] migration issue In-Reply-To: <20190725075436.Horde.2ZCfqrsgjQ-ugCgV6d4WnQ5@webmail.nde.ag> References: <20190724193732.Horde.cbPZa4wb0ka-Rphu0o4kIuP@webmail.nde.ag> <20190725075436.Horde.2ZCfqrsgjQ-ugCgV6d4WnQ5@webmail.nde.ag> Message-ID: Sure. The goal is to be able to build instances that will have access to a locally provisioned storage (we have SSDs added to the compute nodes) for the application data. Indeed in the config we have these 2 entries: `images_rbd_ceph_conf = /etc/ceph/ceph.conf`, and `images_rbd_pool = vms`, but in these paricular hosts we are not using RBD (the hosts without SSDs will use it) hence the `images_type = lvm` setting. The instance creation works as we want. When we boot the image we are creating a volume for the root fs (that is created by cinder, and it is RBD) and we will have a locally provisioned ephemeral disk (vdb) which is created by nova in the ssd_nova VG (images_volume_group = ssd_nova). So, now we are testing whether we can migrate or not these instances. And that's when we ran into the mentioned trouble... :( I hope it is more clear now. Kind regards, Laszlo On 7/25/19 10:54 AM, Eugen Block wrote: > I'm not sure if I get this right, according to your config output you already use ceph as storage backend (which already provides ephemeral disk if configured to do so) but you also want to configure lvm? Can you explain what your goal is here? > > > Zitat von Budai Laszlo : > >> Hi Eugen, >> >> Thank you for your suggestions. I have tested, and the nova user is able to scp from one compute host to the other. I have also tested the migration without the local LVM ephemeral disc, and it is working in that case. The issue seems to appear when I'm trying to use local ephemeral disk managed by LVM (images_type=lvm). >> In this case my ephemeral disk is created on a local LVM volume that is part of the ssd_nova VG (images_volume_group = ssd_nova). >> >> I could not see the source file that the nova is trying to copy, but this may have two reasons: >> 1. The file is not created >> 2. it's life is very short, and I cannot observe it. >> >> The directory /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize doesn't exists while my instance is running. It appears for a short period of time (fraction of a second) while the migrate is trying to do its job, and then disappears again. >> >> Kind regards, >> Laszlo >> >> On 7/24/19 10:37 PM, Eugen Block wrote: >>> Hi, >>> >>> is this the first migration test or have there already been some successful migrations? If it's the first I would suggest to check if the nova user can access the other node via ssh without password. Is /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 present on the source node? Can you run the 'scp' command manually as nova user? >>> >>> Regards, >>> Eugen >>> >>> >>> Zitat von Budai Laszlo : >>> >>>> Dear all, >>>> >>>> we are testing the cold migratoin of instances that are using local ephemeral storage and it fails with the following error: >>>> >>>> >>>> 2019-07-24 13:47:13.115 58902 INFO nova.virt.libvirt.driver [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae >>>> 58-00d6-4317-b635-909fdf09ac49] Instance shutdown successfully after 3 seconds. >>>> 2019-07-24 13:47:13.126 58902 INFO nova.virt.libvirt.driver [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Instance destroyed successfully. >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58- >>>> 00d6-4317-b635-909fdf09ac49] Setting instance vm_state to ERROR: ProcessExecutionError: Unexpected error while running command. >>>> Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph >>>> >>>> Exit code: 1 >>>> Stdout: u'' >>>> Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured syst >>>> em and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            * >>>> \n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No suc >>>> h file or directory\n' >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Traceback (most recent call last): >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 75 >>>> 18, in _error_out_instance_on_exception >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     yield >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 42 >>>> 75, in _resize_instance >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     timeout, retry_interval) >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin >>>> e 8200, in migrate_disk_and_power_off >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     shared_storage) >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220 >>>> , in __exit__ >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     self.force_reraise() >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196 >>>> , in force_reraise >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     six.reraise(self.type_, self.value, self.tb) >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin >>>> e 8185, in migrate_disk_and_power_off >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression) >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line >>>>  226, in copy_image >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression) >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs >>>> .py", line 110, in copy_file >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression) >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs >>>> .py", line 196, in copy_file >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     on_execute=on_execute, on_completion=on_completion) >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in exec >>>> ute >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     return processutils.execute(*cmd, **kwargs) >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py" >>>> , line 424, in execute >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     cmd=sanitized_cmd) >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] ProcessExecutionError: Unexpected error while running command. >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Exit code: 1 >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stdout: u'' >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' >>>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>>> 2019-07-24 13:47:14.380 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Swapping old allocation on 3da11b23-5ef8-4471-a3c9-5f743b0bbfd7 held by migration da66c416-2636-421b-99cb-470bd07dad4c for instance >>>> 2019-07-24 13:47:14.979 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Successfully reverted task state from resize_migrating on failure for instance. >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] Exception during message handling: ProcessExecutionError: Unexpected error while running command. >>>> Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>>> Exit code: 1 >>>> Stdout: u'' >>>> Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Traceback (most recent call last): >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise() >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in decorated_function >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise() >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in decorated_function >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in decorated_function >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info()) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise() >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4240, in resize_instance >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self._revert_allocation(context, instance, migration) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise() >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4237, in resize_instance >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     instance_type, clean_shutdown) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4275, in _resize_instance >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     timeout, retry_interval) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8200, in migrate_disk_and_power_off >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     shared_storage) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise() >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8185, in migrate_disk_and_power_off >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     compression=compression) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in copy_image >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     compression=compression) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in copy_file >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     compression=compression) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in copy_file >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     on_execute=on_execute, on_completion=on_completion) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in execute >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd) >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command. >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Exit code: 1 >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stdout: u'' >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' >>>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>>> 2019-07-24 13:47:17.065 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] image ca89475a-cf15-46d1-ab2d-c2b96c403248 at (/var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07): checking >>>> 2019-07-24 13:47:17.066 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] Active base files: /var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07 >>>> 2019-07-24 13:47:27.768 58902 INFO nova.compute.manager [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] VM Stopped (Lifecycle Event) >>>> 2019-07-24 13:47:27.918 58902 INFO nova.compute.manager [req-e53e928d-81fe-4185-9e62-abc2113e838c - - - - -] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. >>>> >>>> >>>> The disk has its root FS n an RBD device (a volume created by cinder) and it has its ephemeral disk on a local LVM volume created by nova. >>>> >>>> It seems that the /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 is not there when scp is trying to copy it to the other side. >>>> >>>> Any ideas how to fix this? >>>> >>>> The nova config on the nodes looks like this: >>>> >>>> # Ansible managed >>>> [DEFAULT] >>>> allow_resize_to_same_host = True >>>> # Compute >>>> compute_driver = libvirt.LibvirtDriver >>>> # Scheduler >>>> cpu_allocation_ratio = 2.0 >>>> # Logs / State >>>> debug = False >>>> # Hypervisor >>>> default_ephemeral_format = ext4 >>>> disk_allocation_ratio = 1.0 >>>> # Api's >>>> enabled_apis = osapi_compute,metadata >>>> executor_thread_pool_size = 64 >>>> fatal_deprecations = False >>>> # Configdrive >>>> force_config_drive = False >>>> host = compute8.mgmt.lab.mydomain.com >>>> image_cache_manager_interval = 0 >>>> instance_name_template = instance-%08x >>>> # Ceilometer notification configurations >>>> instance_usage_audit = True >>>> instance_usage_audit_period = hour >>>> instances_path = /var/lib/nova/instances >>>> ## Vif >>>> libvirt_vif_type = ethernet >>>> log_dir = /var/log/nova >>>> # Metadata >>>> metadata_workers = 16 >>>> # Network >>>> my_ip = 192.168.56.47 >>>> notify_on_state_change = vm_and_task_state >>>> osapi_compute_workers = 16 >>>> ram_allocation_ratio = 1.0 >>>> reserved_host_disk_mb = 0 >>>> reserved_host_memory_mb = 2048 >>>> resume_guests_state_on_host_boot = False >>>> rootwrap_config = /etc/nova/rootwrap.conf >>>> rpc_response_timeout = 60 >>>> service_down_time = 120 >>>> state_path = /var/lib/nova >>>> # Rpc all >>>> transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e >>>> 8cf8bd7 at 192.168.56.179:5671//nova >>>> # Disable stderr logging >>>> use_stderr = False >>>> vif_plugging_is_fatal = True >>>> vif_plugging_timeout = 30 >>>> >>>> [api] >>>> auth_strategy = keystone >>>> enable_instance_password = True >>>> use_forwarded_for = False >>>> vendordata_jsonfile_path = /etc/nova/vendor_data.json >>>> >>>> # Cache >>>> [cache] >>>> backend = oslo_cache.memcache_pool >>>> enabled = true >>>> memcache_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 >>>> >>>> # Cinder >>>> [cinder] >>>> cafile = /etc/ssl/certs/ca-certificates.crt >>>> catalog_info = volumev3:cinderv3:internalURL >>>> cross_az_attach = True >>>> os_region_name = A_Lab >>>> >>>> [conductor] >>>> workers = 16 >>>> >>>> [filter_scheduler] >>>> available_filters = nova.scheduler.filters.all_filters >>>> enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, AggregateRamFilter, ComputeFilter, AggregateCoreFilter, DiskFilter, AggregateDiskFilter, AggregateNumInstancesFilter, AggregateIoOpsFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, NUMATopologyFilter, SameHostFilter, DifferentHostFilter >>>> host_subset_size = 10 >>>> max_instances_per_host = 50 >>>> max_io_ops_per_host = 10 >>>> ram_weight_multiplier = 5.0 >>>> tracks_instance_changes = True >>>> weight_classes = nova.scheduler.weights.all_weighers >>>> >>>> # Glance >>>> [glance] >>>> api_servers = https://vip.mgmt.lab.mydomain.com:9292 >>>> cafile = /etc/ssl/certs/ca-certificates.crt >>>> >>>> [key_manager] >>>> fixed_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> >>>> [keystone_authtoken] >>>> auth_type = password >>>> auth_uri = https://vip.mgmt.lab.mydomain.com:5000 >>>> auth_url = https://vip.mgmt.lab.mydomain.com:35357 >>>> cafile = /etc/ssl/certs/ca-certificates.crt >>>> insecure = False >>>> memcache_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> # if your memcached server is shared, use these settings to avoid cache poisoning >>>> memcache_security_strategy = ENCRYPT >>>> memcached_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 >>>> password = xxxxxxxxxxxxxxxxxxxxxxx >>>> >>>> project_domain_id = default >>>> project_name = service >>>> region_name = A_Lab >>>> token_cache_time = 300 >>>> user_domain_id = default >>>> username = nova >>>> >>>> [libvirt] >>>> disk_cachemodes = network=writeback >>>> hw_disk_discard = unmap >>>> images_rbd_ceph_conf = /etc/ceph/ceph.conf >>>> images_rbd_pool = vms >>>> images_type = lvm >>>> images_volume_group = ssd_nova >>>> inject_key = False >>>> inject_partition = -2 >>>> inject_password = False >>>> live_migration_tunnelled = True >>>> live_migration_uri = "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa" >>>> rbd_secret_uuid = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> # ceph rbd support >>>> rbd_user = cinder >>>> remove_unused_resized_minimum_age_seconds = 3600 >>>> use_virtio_for_bridges = True >>>> virt_type = kvm >>>> >>>> # Neutron >>>> [neutron] >>>> auth_type = password >>>> # Keystone client plugin authentication URL option >>>> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 >>>> cafile = /etc/ssl/certs/ca-certificates.crt >>>> default_floating_pool = public >>>> insecure = False >>>> metadata_proxy_shared_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> # Keystone client plugin password option >>>> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> project_domain_name = Default >>>> project_name = service >>>> region_name = A_Lab >>>> service_metadata_proxy = True >>>> url = https://vip.mgmt.lab.mydomain.com:9696 >>>> user_domain_name = Default >>>> # Keystone client plugin username option >>>> username = neutron >>>> >>>> [oslo_concurrency] >>>> lock_path = /var/lock/nova >>>> # Notifications >>>> [oslo_messaging_notifications] >>>> driver = messagingv2 >>>> notification_topics = notifications >>>> transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.179:5671//nova >>>> >>>> [oslo_messaging_rabbit] >>>> rpc_conn_pool_size = 30 >>>> ssl = True >>>> >>>> # Placement >>>> [placement] >>>> auth_type = "password" >>>> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 >>>> cafile = /etc/ssl/certs/ca-certificates.crt >>>> insecure = False >>>> os_interface = internal >>>> os_region_name = A_Lab >>>> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> project_domain_name = Default >>>> project_name = service >>>> user_domain_name = Default >>>> username = placement >>>> >>>> [quota] >>>> cores = 20 >>>> injected_file_content_bytes = 10240 >>>> injected_file_path_length = 255 >>>> injected_files = 5 >>>> instances = 10 >>>> key_pairs = 100 >>>> max_age = 0 >>>> metadata_items = 128 >>>> ram = 32768 >>>> server_group_members = 10 >>>> server_groups = 10 >>>> >>>> [scheduler] >>>> discover_hosts_in_cells_interval = 60 >>>> host_manager = host_manager >>>> max_attempts = 5 >>>> periodic_task_interval = 60 >>>> scheduler_driver = filter_scheduler >>>> >>>> [spice] >>>> agent_enabled = True >>>> enabled = True >>>> # Console Url and binds >>>> html5proxy_base_url = https://dashboard.iaas.lab.mydomain.com:6080/spice_auto.html >>>> server_listen = 192.168.56.47 >>>> server_proxyclient_address = 192.168.56.47 >>>> >>>> [upgrade_levels] >>>> compute = auto >>>> >>>> [vnc] >>>> enabled = False >>>> >>>> [wsgi] >>>> api_paste_config = /etc/nova/api-paste.ini >>>> secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO >>>> >>>> >>>> Thank you in advance for any suggestions. >>>> >>>> Kind regards, >>>> Laszlo >>> >>> >>> >>> >>> > > > > > From mriedemos at gmail.com Thu Jul 25 11:53:54 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Thu, 25 Jul 2019 06:53:54 -0500 Subject: [Nova] migration issue In-Reply-To: References: <20190724193732.Horde.cbPZa4wb0ka-Rphu0o4kIuP@webmail.nde.ag> <20190725075436.Horde.2ZCfqrsgjQ-ugCgV6d4WnQ5@webmail.nde.ag> Message-ID: On 7/25/2019 4:39 AM, Budai Laszlo wrote: > So, now we are testing whether we can migrate or not these instances. And that's when we ran into the mentioned trouble...:( Migration with local lvm-backed instances doesn't work, it's a long-standing known issue. https://bugs.launchpad.net/nova/+bug/1831657 https://review.opendev.org/#/c/337334/ -- Thanks, Matt From mriedemos at gmail.com Thu Jul 25 12:02:24 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Thu, 25 Jul 2019 07:02:24 -0500 Subject: [kolla][nova][cinder] Got Gateway-Timeout error on VM evacuation if it has volume attached. In-Reply-To: <20190725081407.lkapi4skj2hcllzi@localhost> References: <20190725081407.lkapi4skj2hcllzi@localhost> Message-ID: <1cdc4a94-1dc2-420d-5ae8-663d903d2dd2@gmail.com> On 7/25/2019 3:14 AM, Gorka Eguileor wrote: > Attachment delete is a synchronous operation, so all the different > connection timeouts may affect the operation: Nova to HAProxy, HAProxy > to Cinder-API, Cinder-API to Cinder-Volume via RabbitMQ, Cinder-Volume > to Storage backend. > > I would recommend you looking at the specific attachment_delete request > that failed in Cinder logs and see how long it took to complete, and > then check how long it took for the 504 error to happen. With that info > you can get an idea of how much higher your timeout must be. > > It could also happen that the Cinder-API raises a timeout error when > calling the Cinder-Volume. In this case you should check the > cinder-volume service to see how long it took it to complete, as the > operation continues. > > Internally the Cinder-API to Cinder-Volume timeout is usually around 60 > seconds (rpc_response_timeout). Yeah this is a known intermittent issue in our CI jobs as well, for example: http://status.openstack.org/elastic-recheck/#1763712 As I mentioned in the bug report for that issue: https://bugs.launchpad.net/cinder/+bug/1763712 It might be worth using the long_rpc_timeout approach for this assuming the http response doesn't timeout. Nova uses long_rpc_timeout for known long RPC calls: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.long_rpc_timeout Cinder should probably do the same for initialize connection style RPC calls. I've seen other gate failures where cinder-backup to cinder-volume rpc calls to initialize a connection have timed out as well, e.g.: https://bugs.launchpad.net/cinder/+bug/1739482 -- Thanks, Matt From mriedemos at gmail.com Thu Jul 25 12:47:48 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Thu, 25 Jul 2019 07:47:48 -0500 Subject: [nova] Train Spec Freeze In-Reply-To: References: Message-ID: <592d162a-5a90-d6bd-a97a-37e641ca9d7f@gmail.com> On 7/24/2019 5:10 PM, Eric Fried wrote: > If you own a spec you think should be able to land [2], please: > - Check it to make sure any comments have been addressed (even if > there's no -1). > - Explicitly ask for reviews, here on the mailing list and/or in IRC in > #openstack-nova. These two are pretty trivial changes IMO: 1. https://review.opendev.org/#/c/667894/ Add user_id to the migrations 2. https://review.opendev.org/#/c/612949/ Support delete_on_termination in volume attach api The author of each is in China and it's already getting late there. I've contacted Brin directly (WeChat FTW) about the second one. If he can confirm my comments I'll update the spec today to clean it up. The same could be done for the first spec. -- Thanks, Matt From laszlo.budai at gmail.com Thu Jul 25 12:50:42 2019 From: laszlo.budai at gmail.com (Budai Laszlo) Date: Thu, 25 Jul 2019 15:50:42 +0300 Subject: [Nova] migration issue In-Reply-To: References: <20190724193732.Horde.cbPZa4wb0ka-Rphu0o4kIuP@webmail.nde.ag> <20190725075436.Horde.2ZCfqrsgjQ-ugCgV6d4WnQ5@webmail.nde.ag> Message-ID: <246a1c7b-b0f2-4ffd-fe9f-1d248f2f5e22@gmail.com> Hi Matt, Thank you for the info. This is really important. I got one error at live migration which was saying that Live migration with local volumes is not working, but the cold migration it was trying to scp, so I was assuming that it is supposed to work ... Can you please clarify one more thing: how is the resize involved in this cold migration. Apparently it is trying to resize before it would copy the files. Is there any document that describes the steps of the migration? Kind regards, Laszlo On 7/25/19 2:53 PM, Matt Riedemann wrote: > On 7/25/2019 4:39 AM, Budai Laszlo wrote: >> So, now we are testing whether we can migrate or not these instances. And that's when we ran into the mentioned trouble...:( > > Migration with local lvm-backed instances doesn't work, it's a long-standing known issue. > > https://bugs.launchpad.net/nova/+bug/1831657 > > https://review.opendev.org/#/c/337334/ > From mriedemos at gmail.com Thu Jul 25 13:35:10 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Thu, 25 Jul 2019 08:35:10 -0500 Subject: [Nova] migration issue In-Reply-To: <246a1c7b-b0f2-4ffd-fe9f-1d248f2f5e22@gmail.com> References: <20190724193732.Horde.cbPZa4wb0ka-Rphu0o4kIuP@webmail.nde.ag> <20190725075436.Horde.2ZCfqrsgjQ-ugCgV6d4WnQ5@webmail.nde.ag> <246a1c7b-b0f2-4ffd-fe9f-1d248f2f5e22@gmail.com> Message-ID: On 7/25/2019 7:50 AM, Budai Laszlo wrote: > Can you please clarify one more thing: how is the resize involved in this cold migration. Apparently it is trying to resize before it would copy the files. Is there any document that describes the steps of the migration? Resize and cold migration are the same flow under the covers, the only difference being a resize has a new flavor while a cold migration does not (and for the libvirt driver you can't cold migrate to the same host like a resize). I don't think we have a reference doc on the low-level resize / cold migration flow like we have for live migration [1] (we should add one though, I could probably sketch it out pretty quickly if someone can throw it into the docs). I did find this pretty old blog [2] that might help - note it's from 2014 though so parts of it might be inaccurate by now. [1] https://docs.openstack.org/nova/latest/reference/live-migration.html [2] http://bodenr.blogspot.com/2014/03/openstack-nova-vm-migration-live-and.html -- Thanks, Matt From zheng.ma at intel.com Thu Jul 25 15:44:54 2019 From: zheng.ma at intel.com (Ma, Zheng) Date: Thu, 25 Jul 2019 15:44:54 +0000 Subject: [dev][nova][glance][cinder] introducing 'compressed' image container_format Message-ID: Hello Nova developers, Cinder is working on a feature that will allow it to compress volumes uploaded to Glance [0,1]. We are proposing a new container_format for Glance named 'compressed' [2]. The idea is that a consumer of a 'compressed' image would read the file to determine the kind of compression and take appropriate action (either successfully decompress to find a payload that matches the image's disk_format, or fail with 'unsupported container_format'). Our preliminary testing with a raw image with container_format == compressed, disk_format == raw is that Nova boots a VM from the image but the VM appears to be unusable. Nova seems taking it as a real RAW image as disk_format indicates. So we want to open a discussion about handling this. Thanks! [0] cinder spec patch https://review.opendev.org/#/c/652275/ [1] cinder code implementation https://review.opendev.org/#/c/668825 [2] glance spec patch https://review.opendev.org/#/c/670454/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From raubvogel at gmail.com Thu Jul 25 16:04:10 2019 From: raubvogel at gmail.com (Mauricio Tavares) Date: Thu, 25 Jul 2019 12:04:10 -0400 Subject: Forcing restart of a worker node with running guest Message-ID: So one of my worker nodes is, to put it mildly, rather unhappy: Message from syslogd at openstack-w1 at Jul 25 15:57:49 ... kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [vnc_worker:9141] Message from syslogd at openstack-w1 at Jul 25 15:57:57 ... kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [migration/10:60] Message from syslogd at openstack-w1 at Jul 25 15:58:05 ... kernel:NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [migration/8:49] I found out when it was taking 30 min to delete a guest. So, what I can do in a forceful way? 1. How to kill the guest? Can I kill it through virsh or openstack compute service will get sad? 2. What would happen if I stop the compute service? 3. What would happen if I get really annoyed and tell worker node to reboot? -------------- next part -------------- An HTML attachment was scrubbed... URL: From openstack at nemebean.com Thu Jul 25 16:46:53 2019 From: openstack at nemebean.com (Ben Nemec) Date: Thu, 25 Jul 2019 11:46:53 -0500 Subject: [keystone] Need clarification about Stein policies In-Reply-To: <701cdad6-e0b4-4705-ee11-4b81db307612@gmail.com> References: <701cdad6-e0b4-4705-ee11-4b81db307612@gmail.com> Message-ID: <18c88a59-c9ec-5874-71bf-551ae8400de2@nemebean.com> On 7/25/19 1:18 AM, Bernd Bausch wrote: > The Keystone policy.json file I created with oslo-policy-generator > contains lines I don't understand. For example /list_users/. The comment > says: > > # DEPRECATED "identity:list_users":"rule:admin_required" has been > # deprecated since S in favor of "identity:list_users":"(role:reader > # and system_scope:all) or (role:reader and > # domain_id:%(target.domain_id)s)". > > I do understand the expression starting with (role:reader .... , but > contrarily to the comment, the policy is > > "identity:list_users": "rule:identity:list_users" > > This looks like a circular definition, and in any case, nowhere do I > seerule:identity:list_users defined. > > Can someone in the know explain how this policy is processed? You're right, this is a circular definition and a bug in the policy generator. This behavior was intended to address [0], but when the deprecated rule name matches the current rule name it creates this nonsense policy. Since the bug doesn't apply in this case, we can just drop the unnecessary alias. Lance pushed a fix in [1] that should make this work sanely again. Thanks for bringing this to our attention. 0: https://bugs.launchpad.net/oslo.policy/+bug/1742569 1: https://review.opendev.org/#/c/672781/ > > Thanks much, > > Bernd > From openstack at fried.cc Thu Jul 25 17:25:37 2019 From: openstack at fried.cc (Eric Fried) Date: Thu, 25 Jul 2019 12:25:37 -0500 Subject: [Nova] migration issue In-Reply-To: References: <20190724193732.Horde.cbPZa4wb0ka-Rphu0o4kIuP@webmail.nde.ag> <20190725075436.Horde.2ZCfqrsgjQ-ugCgV6d4WnQ5@webmail.nde.ag> <246a1c7b-b0f2-4ffd-fe9f-1d248f2f5e22@gmail.com> Message-ID: <33aba3a9-6d47-12fb-76ab-416b5c13d412@fried.cc> > I don't think we have a reference doc on the low-level resize / cold > migration flow like we have for live migration [1] (we should add one > though, I could probably sketch it out pretty quickly if someone can > throw it into the docs). You know the drill: - napkin sketch - seqdiag - two years elapse - shiny nickel efried . From mriedemos at gmail.com Thu Jul 25 19:40:54 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Thu, 25 Jul 2019 14:40:54 -0500 Subject: Forcing restart of a worker node with running guest In-Reply-To: References: Message-ID: <0798fb66-2028-3a98-38b4-a5c5526b140f@gmail.com> On 7/25/2019 11:04 AM, Mauricio Tavares wrote: > I found out when it was taking 30 min to delete a guest. So, what I can > do in a forceful way? > > 1. How to kill the guest? Can I kill it through virsh or openstack > compute service will get sad? I would try to avoid this if possible, but you might need to kill the guest in the hypervisor if doing it through nova won't get the job done. What happens in nova-compute is undefined, but you'd probably see some errors as expected if you're doing anything with that server at the hypervisor layer, like trying to get the guest power state. What nova is tracking and what is in the hypervisor are different things, and if you delete the guest out of band from nova, you'll need to delete the server to sync the nova database. If the delete is stuck in the compute API, thinking it's already deleting (I think we have an old bug for that and force delete, and I hit something similar today), you could try resetting the server status to ERROR [1] and then try deleting it in the API again. > 2. What would happen if I stop the compute service? This won't really do anything to the guest in the hypervisor unless [2] tries to change the guest state on restart. In my experience that option has not been very reliable / predictable. > 3. What would happen if I get really annoyed and tell worker node to reboot? Pretty much the same as #2 from a nova perspective I think. Depending on how libvirt and/or the guest domain is configured, the libvirt-guest service might try to resume the guest. [1] openstack server set --state error [2] https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.resume_guests_state_on_host_boot -- Thanks, Matt From raubvogel at gmail.com Thu Jul 25 20:24:55 2019 From: raubvogel at gmail.com (Mauricio Tavares) Date: Thu, 25 Jul 2019 16:24:55 -0400 Subject: Forcing restart of a worker node with running guest In-Reply-To: <0798fb66-2028-3a98-38b4-a5c5526b140f@gmail.com> References: <0798fb66-2028-3a98-38b4-a5c5526b140f@gmail.com> Message-ID: On Thu, Jul 25, 2019 at 3:44 PM Matt Riedemann wrote: > > On 7/25/2019 11:04 AM, Mauricio Tavares wrote: > > I found out when it was taking 30 min to delete a guest. So, what I can > > do in a forceful way? > > > > 1. How to kill the guest? Can I kill it through virsh or openstack > > compute service will get sad? > > I would try to avoid this if possible, but you might need to kill the > guest in the hypervisor if doing it through nova won't get the job done. > What happens in nova-compute is undefined, but you'd probably see some > errors as expected if you're doing anything with that server at the > hypervisor layer, like trying to get the guest power state. > > What nova is tracking and what is in the hypervisor are different > things, and if you delete the guest out of band from nova, you'll need > to delete the server to sync the nova database. If the delete is stuck > in the compute API, thinking it's already deleting (I think we have an > old bug for that and force delete, and I hit something similar today), > you could try resetting the server status to ERROR [1] and then try > deleting it in the API again. > > > 2. What would happen if I stop the compute service? > > This won't really do anything to the guest in the hypervisor unless [2] > tries to change the guest state on restart. In my experience that option > has not been very reliable / predictable. > > > 3. What would happen if I get really annoyed and tell worker node to reboot? > > Pretty much the same as #2 from a nova perspective I think. Depending on > how libvirt and/or the guest domain is configured, the libvirt-guest > service might try to resume the guest. > Does that mean it is using the standard libvirt config files? > [1] openstack server set --state error > [2] > https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.resume_guests_state_on_host_boot > Thanks for the info. It turned out the issue is hardware related, so shutting the worker node down is way past the realm of possibility into the realm of it will happen today. > -- > > Thanks, > > Matt > From chris.friesen at windriver.com Thu Jul 25 22:00:12 2019 From: chris.friesen at windriver.com (Chris Friesen) Date: Thu, 25 Jul 2019 16:00:12 -0600 Subject: [placement] update 19-27 In-Reply-To: References: Message-ID: <60bbb1c5-6afa-2292-b100-d84f09506be8@windriver.com> On 7/12/2019 4:00 AM, Chris Dent wrote: > As mentioned last week, one of the important cleanup tasks that is not > yet in progress is updating the > [gabbit](https://opendev.org/openstack/placement/src/branch/master/gate/gabbits/nested-perfload.yaml) > > that creates the nested topology that's used in nested performance > testing. The topology there is simple, unrealistic, and doesn't > sufficiently exercise the several features that may be used during a > query that desires a nested response. This needs to be someone who > is more closely related to real world use of nested than me. efried? > gibi? With regards to the topology for nested performance testing, I think a more-complicated but reasonably typical nested topology would be a compute node with something like 2/4 numa nodes, with a mix of memory page sizes (4K/2MB/1GB), with HyperThreading, and with a few NUMA-aware PCI devices (SRIOV, FPGA, vGPU, etc.). Chris From zhangbailin at inspur.com Thu Jul 25 22:52:36 2019 From: zhangbailin at inspur.com (=?gb2312?B?QnJpbiBaaGFuZyjVxbDZwdYp?=) Date: Thu, 25 Jul 2019 22:52:36 +0000 Subject: =?gb2312?B?UmU6IFtsaXN0cy5vcGVuc3RhY2sub3JntPq3ol1SZTogW05vdmFdIG1pZ3Jh?= =?gb2312?Q?tion_issue?= In-Reply-To: References: <155399f078e03a0a661ab87c6c4c48e1@sslemail.net>, Message-ID: <0l90d37ejbaxlyg1xsf0ebra.1564094697424@email.jadenine.com> >I don't think we have a reference doc on the low-level resize / cold migration flow like we have for live migration [1] (we should add one though, I could probably sketch it out pretty quickly if someone can throw it into the docs). I did find this pretty old blog [2] that might help - note it's from 2014 though so parts of it might be inaccurate by now. I think we need a new live-migration doc, because it has changed a lot. ________________________________ 发件人：Matt Riedemann 时间：2019年07月25日 下午 9:45 收件人：openstack-discuss at lists.openstack.org 主题：[lists.openstack.org代发]Re: [Nova] migration issue On 7/25/2019 7:50 AM, Budai Laszlo wrote: > Can you please clarify one more thing: how is the resize involved in this cold migration. Apparently it is trying to resize before it would copy the files. Is there any document that describes the steps of the migration? Resize and cold migration are the same flow under the covers, the only difference being a resize has a new flavor while a cold migration does not (and for the libvirt driver you can't cold migrate to the same host like a resize). I don't think we have a reference doc on the low-level resize / cold migration flow like we have for live migration [1] (we should add one though, I could probably sketch it out pretty quickly if someone can throw it into the docs). I did find this pretty old blog [2] that might help - note it's from 2014 though so parts of it might be inaccurate by now. [1] https://docs.openstack.org/nova/latest/reference/live-migration.html [2] http://bodenr.blogspot.com/2014/03/openstack-nova-vm-migration-live-and.html -- Thanks, Matt From gouthampravi at gmail.com Fri Jul 26 03:41:46 2019 From: gouthampravi at gmail.com (Goutham Pacha Ravi) Date: Thu, 25 Jul 2019 20:41:46 -0700 Subject: [manila] Enabling CephFS snapshot support by default Message-ID: Hi Zorillas and interested parties, (Copying a couple of folks explicitly as they may not be part of openstack-discuss) Manila's CephFS driver has a configuration option "cephfs_enable_snapshots" that administrators can toggle in manila.conf to allow project users to take snapshots of their shares. [1][2] This option has defaulted to False, since the time the CephFS driver was introduced (Mitaka release of OpenStack Manila) IIUC, CephFS snapshots have been "stable" since the Mimic release of CephFS, which debuted in June 2018. [3] Since then, ceph deployers/administrators don't have to toggle anything on the backend to enable snapshots. So, can we consider changing the default value of the config opt in manila to "True" in the Train release? This will allow the driver to report the capability "snapshot_support" as enabled, and makes life easier for OpenStack deployers/administrators. I would also love to deprecate this option and remove it in a future release, since it isn't really necessary, read on: Users cannot arbitrarily take snapshots in manila, there are a couple of things that can be used to prevent them: Quotas and Share Types. Unless users create their shares using a share type that explicitly requests "snapshot_support", users can't snapshot their shares. So, we're covered by the service's API in case someone is afraid this has an impact on someone in the wild using newer versions of Manila with older versions of Ceph, where snapshots were unstable. Thoughts? [1] https://opendev.org/openstack/manila/src/commit/721bb70a81b80bf065705e8790bd117eae806e7d/manila/share/drivers/cephfs/driver.py#L66 [2] https://docs.openstack.org/manila/queens/admin/cephfs_driver.html#enabling-snapshot-support-in-ceph-backend [3] http://docs.ceph.com/docs/mimic/dev/cephfs-snapshots/ From missile0407 at gmail.com Fri Jul 26 04:54:18 2019 From: missile0407 at gmail.com (Eddie Yen) Date: Fri, 26 Jul 2019 12:54:18 +0800 Subject: [kolla][nova][cinder] Got Gateway-Timeout error on VM evacuation if it has volume attached. In-Reply-To: <1cdc4a94-1dc2-420d-5ae8-663d903d2dd2@gmail.com> References: <20190725081407.lkapi4skj2hcllzi@localhost> <1cdc4a94-1dc2-420d-5ae8-663d903d2dd2@gmail.com> Message-ID: Thanks for the advice. Actually, I tested evacuation again 2 days ago. And this time the evacuation is successful. All VMs included volume attached were evacuated with no error. The Horizon response still slow when shutdown one node. But it became more faster than before. But I think we still need to gain a longer timeout. And I think I should gain rpc_response_timeout rather than long_rpc_timeout in nova. Please correct me if wrong. Many thanks, Eddie. Matt Riedemann 於 2019年7月25日 週四 下午8:11寫道： > On 7/25/2019 3:14 AM, Gorka Eguileor wrote: > > Attachment delete is a synchronous operation, so all the different > > connection timeouts may affect the operation: Nova to HAProxy, HAProxy > > to Cinder-API, Cinder-API to Cinder-Volume via RabbitMQ, Cinder-Volume > > to Storage backend. > > > > I would recommend you looking at the specific attachment_delete request > > that failed in Cinder logs and see how long it took to complete, and > > then check how long it took for the 504 error to happen. With that info > > you can get an idea of how much higher your timeout must be. > > > > It could also happen that the Cinder-API raises a timeout error when > > calling the Cinder-Volume. In this case you should check the > > cinder-volume service to see how long it took it to complete, as the > > operation continues. > > > > Internally the Cinder-API to Cinder-Volume timeout is usually around 60 > > seconds (rpc_response_timeout). > > Yeah this is a known intermittent issue in our CI jobs as well, for > example: > > http://status.openstack.org/elastic-recheck/#1763712 > > As I mentioned in the bug report for that issue: > > https://bugs.launchpad.net/cinder/+bug/1763712 > > It might be worth using the long_rpc_timeout approach for this assuming > the http response doesn't timeout. Nova uses long_rpc_timeout for known > long RPC calls: > > > https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.long_rpc_timeout > > Cinder should probably do the same for initialize connection style RPC > calls. I've seen other gate failures where cinder-backup to > cinder-volume rpc calls to initialize a connection have timed out as > well, e.g.: > > https://bugs.launchpad.net/cinder/+bug/1739482 > > -- > > Thanks, > > Matt > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Fri Jul 26 09:54:11 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 26 Jul 2019 18:54:11 +0900 Subject: [goals][IPv6-Only Deployments and Testing] Week R-12 Update Message-ID: <16c2db3c410.f4ba6f0a491755.8935052370642166197@ghanshyammann.com> Hello Everyone, I am starting this status email for 'IPv6-Only Deployments and Testing ' community goal for Train. Storyboard: ========= - https://storyboard.openstack.org/#!/story/2005477 Current status: ============ 1. Base job 'devstack-tempest-ipv6' has been prepared on Tempest (patch yet to merge)[1] 2. 'tempest-ipv6-only' is defined in Tempest (patch yet to merge)[1] 3. 'tempest-ipv6-only' has been proposed to run on 5 services project date side [2]. 4. I have proposed few more projects jobs like congress etc and will keep doing for other projects also. 5. Neutron stadium projects jobs. I talked to slaweq on IRC [3]on what all neutron stadium falls under the IPv6 address communication criteria so that we can add ipv6-only job for those project only. He is going to prepare the list of those project and accordingly we will add jobs Everything related to this goal can be found under this topic: Topic: https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(status:open+OR+status:merged) How to define and run new IPv6 Job on project side: ======================================= - I prepared a wiki page to describe this section - https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing - I am adding this wiki info in goal doc also[4]. Review suggestion: ============== - Main goal of these jobs will be whether your service is able to listen on IPv6 and can communicate to any other services either OpenStack or DB or rabbitmq etc on IPv6 or not. So check your proposed job with that point of view. If anything missing, comment on patch. - One example was - I missed to configure novnc address to IPv6- https://review.opendev.org/#/c/672493/ - base script as part of 'devstack-tempest-ipv6' will do basic checks for endpoints on IPv6 and some devstack var setting. But if your project needs more specific varification then it can be added in project side job as post-run playbooks as described in wiki page[5]. [1] https://review.opendev.org/#/c/671231/ [2] https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(status:open+OR+status:merged) [3] http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2019-07-26.log.html#t2019-07-26T08:56:20 [4] https://review.opendev.org/#/c/671898/ [5] https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing -gmann From amotoki at gmail.com Fri Jul 26 11:29:51 2019 From: amotoki at gmail.com (Akihiro Motoki) Date: Fri, 26 Jul 2019 20:29:51 +0900 Subject: [all] [tc] General questions on PDF community goal Message-ID: Hi, I have a couple of general questions on PDF community goal. What is the criteria of completing the PDF community goal? Is it okay if we publish a PDF deliverable anyway? During working on it, I hit the following questions. We already reached Train-2 milestone, so I think it is the time to clarify the detail criteria of completing the community goal. - Where should the generated PDF file to be located and What is the recommended PDF file name? /.pdf? index.pdf? Any path and any file is okay? - Do we create a link to the PDF file from index.html (per-project top page)? If needed, perhaps this would be a thing covered by openstackdocstheme. Otherwise, how can normal consumers know PDF documents? - Which repositories need to provide PDF version of documents? My understanding (amotoki) is that repositories with 'publish-openstack-docs-pti' should publish PDF doc. right? - Do we need PDF version of release notes? - Do we need PDF version of API reference? I see no coordination efforts recently and am afraid that individual projects cannot decide whether patches to their repositories are okay to merge. Thanks, Akihiro Motoki (irc: amotoki) From mriedemos at gmail.com Fri Jul 26 13:33:08 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Fri, 26 Jul 2019 08:33:08 -0500 Subject: [kolla][nova][cinder] Got Gateway-Timeout error on VM evacuation if it has volume attached. In-Reply-To: References: <20190725081407.lkapi4skj2hcllzi@localhost> <1cdc4a94-1dc2-420d-5ae8-663d903d2dd2@gmail.com> Message-ID: On 7/25/2019 11:54 PM, Eddie Yen wrote: > And I think I should gain rpc_response_timeout rather than > long_rpc_timeout in nova. Since Cinder doesn't have the long_rpc_timeout option like Nova you're only option is to bump up the rpc_response_timeout in Cinder but that will be used by all RPC calls in Cinder, not just the initialize/terminate connection calls for attachments. Maybe that's not a problem, but long_rpc_timeout in Nova allows us to pick which RPC calls to use that on rather than everywhere. The changes to Cinder shouldn't be that hard if they follow the Nova patch [1]. [1] https://review.opendev.org/#/c/566696/ -- Thanks, Matt From mnaser at vexxhost.com Fri Jul 26 13:37:40 2019 From: mnaser at vexxhost.com (Mohammed Naser) Date: Fri, 26 Jul 2019 09:37:40 -0400 Subject: [openstack-ansible] weekly update Message-ID: Hi everyone, I'm going to try doing these more often to keep everyone updated on the current state of openstack-ansible (mostly around what is discussed during our office hours / meetings). - Dmitriy has been working on removing unused code in playbooks, mostly around logs as we move to 100% systemd-journald. - We're discussing the proposal to remove os-log-dir-setup.yml, as well as rsyslog_client and rsyslog_server roles as they aren’t used anymore with the move to systemd-journald - We're also discussing removing all the complex bind mounts, which would clean it and simplify the code base. - Our continuous integration has been stable but needs upgrade jobs and increased coverage. Our coverage would go up significantly by running aio_lxc for all supported systems. - Jonathan's work on binding to management IPs is making progress (and seems to be at the Galera stage). - CentOS CI has been very slow and nobody knows why. It seems to be something we've regressed at some point. - A unicast flood issue affecting deployments using linuxbridge that happened inside os-vif needs to be patched. It's still pending os-vif to patch. - Namrata mentioned an issue during the upgrade from Rocky to Stein (when upgrading WSREP SST method from xtrabackup-V2 to mariabackup) that will have to be fixed is Stein. That's something that's in progress now. Thanks everyone for reading! Regards, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser at vexxhost.com W. http://vexxhost.com From missile0407 at gmail.com Fri Jul 26 14:59:49 2019 From: missile0407 at gmail.com (Eddie Yen) Date: Fri, 26 Jul 2019 22:59:49 +0800 Subject: [kolla][nova][cinder] Got Gateway-Timeout error on VM evacuation if it has volume attached. In-Reply-To: References: <20190725081407.lkapi4skj2hcllzi@localhost> <1cdc4a94-1dc2-420d-5ae8-663d903d2dd2@gmail.com> Message-ID: Roger that, thanks for explanation. I think there's another reason to me that get this issue. The environment is stayed without any internet nor local NTP server, until the last test. Before the test, the nova and cinder services became unstable because they keeping up and down. And I found that the clock are out of sync between nodes. We let one of the node can connect outside and let NTP client pointed to that one on other nodes. Then problem solved. Of course the test is successful. I'm not sure but that's a one of reason right? But I think I still need to try optimize the timeout value since the API response is slow when shutting down a node. Wonder know why it become slow when a node down. I'll try to gain up rpc_response_timeout in Cinder and do more testing. Matt Riedemann 於 2019年7月26日 週五 下午9:42寫道： > On 7/25/2019 11:54 PM, Eddie Yen wrote: > > And I think I should gain rpc_response_timeout rather than > > long_rpc_timeout in nova. > > Since Cinder doesn't have the long_rpc_timeout option like Nova you're > only option is to bump up the rpc_response_timeout in Cinder but that > will be used by all RPC calls in Cinder, not just the > initialize/terminate connection calls for attachments. Maybe that's not > a problem, but long_rpc_timeout in Nova allows us to pick which RPC > calls to use that on rather than everywhere. The changes to Cinder > shouldn't be that hard if they follow the Nova patch [1]. > > [1] https://review.opendev.org/#/c/566696/ > > -- > > Thanks, > > Matt > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Fri Jul 26 15:01:46 2019 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 26 Jul 2019 17:01:46 +0200 Subject: [openstack-ansible] weekly update In-Reply-To: References: Message-ID: Hi Mohammed, regarding CentOS CI slowness - you are not alone, kola is experiencing it too. Some time ago I asked around and discovered that various projects experience noticeable CI slowness using CentOS compared to Ubuntu. The hit is more noticeable on image building than deploying and I guess since OSA does "building/installing" while deploying, it experiences our worse hit. We deploy in containers so I even checked the host:Ubuntu, container:CentOS scenario but it was noticeably slower anyway. Hence it seems related to some CentOS libraries or common stuff (like Python maybe?), not kernel. We even ran into broken CI situation by being hit by horizon changing translation-related internals and our once hacky way of compiling translations which resulted in n^2 compilations instead of n - Ubuntu was a bit slower than usual but CentOS timed out, possibly not even halfway through horizon image building process... Kind regards, Radek pt., 26 lip 2019 o 15:45 Mohammed Naser napisał(a): > Hi everyone, > > I'm going to try doing these more often to keep everyone updated on > the current state of openstack-ansible (mostly around what is > discussed during our office hours / meetings). > > - Dmitriy has been working on removing unused code in playbooks, > mostly around logs as we move to 100% systemd-journald. > - We're discussing the proposal to remove os-log-dir-setup.yml, as > well as rsyslog_client and rsyslog_server roles as they aren’t used > anymore with the move to systemd-journald > - We're also discussing removing all the complex bind mounts, which > would clean it and simplify the code base. > - Our continuous integration has been stable but needs upgrade jobs > and increased coverage. Our coverage would go up significantly by > running aio_lxc for all supported systems. > - Jonathan's work on binding to management IPs is making progress (and > seems to be at the Galera stage). > - CentOS CI has been very slow and nobody knows why. It seems to be > something we've regressed at some point. > - A unicast flood issue affecting deployments using linuxbridge that > happened inside os-vif needs to be patched. It's still pending os-vif > to patch. > - Namrata mentioned an issue during the upgrade from Rocky to Stein > (when upgrading WSREP SST method from xtrabackup-V2 to mariabackup) > that will have to be fixed is Stein. That's something that's in > progress now. > > Thanks everyone for reading! > > Regards, > Mohammed > > > > -- > Mohammed Naser — vexxhost > ----------------------------------------------------- > D. 514-316-8872 > D. 800-910-1726 ext. 200 > E. mnaser at vexxhost.com > W. http://vexxhost.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhangbailin at inspur.com Thu Jul 25 15:21:27 2019 From: zhangbailin at inspur.com (=?utf-8?B?QnJpbiBaaGFuZyjlvKDnmb7mnpcp?=) Date: Thu, 25 Jul 2019 15:21:27 +0000 Subject: =?utf-8?B?UmU6IFtsaXN0cy5vcGVuc3RhY2sub3Jn5Luj5Y+RXVJlOiBbTm92YV0gbWln?= =?utf-8?Q?ration_issue?= In-Reply-To: <20190725075436.Horde.2ZCfqrsgjQ-ugCgV6d4WnQ5@webmail.nde.ag> References: , <20190725075436.Horde.2ZCfqrsgjQ-ugCgV6d4WnQ5@webmail.nde.ag> Message-ID: Is it possible to configure the environment, what did the migrated server do before migration? And the migration log is given? Or submit a bug on the launchpad, explaining the version of the problem, the recurring steps, and so on. Above, in order to convenient and clearer analysis of problems. ________________________________ 发件人：Eugen Block 时间：2019年07月25日 下午 4:06 收件人：openstack-discuss at lists.openstack.org 主题：[lists.openstack.org代发]Re: [Nova] migration issue I'm not sure if I get this right, according to your config output you already use ceph as storage backend (which already provides ephemeral disk if configured to do so) but you also want to configure lvm? Can you explain what your goal is here? Zitat von Budai Laszlo : > Hi Eugen, > > Thank you for your suggestions. I have tested, and the nova user is > able to scp from one compute host to the other. I have also tested > the migration without the local LVM ephemeral disc, and it is > working in that case. The issue seems to appear when I'm trying to > use local ephemeral disk managed by LVM (images_type=lvm). > In this case my ephemeral disk is created on a local LVM volume that > is part of the ssd_nova VG (images_volume_group = ssd_nova). > > I could not see the source file that the nova is trying to copy, but > this may have two reasons: > 1. The file is not created > 2. it's life is very short, and I cannot observe it. > > The directory > /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize > doesn't exists while my instance is running. It appears for a short > period of time (fraction of a second) while the migrate is trying to > do its job, and then disappears again. > > Kind regards, > Laszlo > > On 7/24/19 10:37 PM, Eugen Block wrote: >> Hi, >> >> is this the first migration test or have there already been some >> successful migrations? If it's the first I would suggest to check >> if the nova user can access the other node via ssh without >> password. Is >> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 present on the source node? Can you run the 'scp' command manually as nova >> user? >> >> Regards, >> Eugen >> >> >> Zitat von Budai Laszlo : >> >>> Dear all, >>> >>> we are testing the cold migratoin of instances that are using >>> local ephemeral storage and it fails with the following error: >>> >>> >>> 2019-07-24 13:47:13.115 58902 INFO nova.virt.libvirt.driver >>> [req-356fce31-5e98-425f-89d6-8a98664d31ad >>> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 >>> - default default] [instance: 5a6dae >>> 58-00d6-4317-b635-909fdf09ac49] Instance shutdown successfully >>> after 3 seconds. >>> 2019-07-24 13:47:13.126 58902 INFO nova.virt.libvirt.driver [-] >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Instance >>> destroyed successfully. >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [req-356fce31-5e98-425f-89d6-8a98664d31ad >>> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 >>> - default default] [instance: 5a6dae58- >>> 00d6-4317-b635-909fdf09ac49] Setting instance vm_state to ERROR: >>> ProcessExecutionError: Unexpected error while running command. >>> Command: scp -r >>> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph >>> >>> Exit code: 1 >>> Stdout: u'' >>> Stderr: >>> u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured >>> syst >>> em and your actions will be logged along *\n* with identifying >>> information. Disconnect immediately if you are not an *\n* >>> authorized user of this >>> system. * >>> \n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No >>> suc >>> h file or directory\n' >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Traceback (most >>> recent call last): >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line >>> 75 >>> 18, in _error_out_instance_on_exception >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] yield >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line >>> 42 >>> 75, in _resize_instance >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] timeout, >>> retry_interval) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", >>> lin >>> e 8200, in migrate_disk_and_power_off >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] shared_storage) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line >>> 220 >>> , in __exit__ >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>> self.force_reraise() >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line >>> 196 >>> , in force_reraise >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", >>> lin >>> e 8185, in migrate_disk_and_power_off >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>> compression=compression) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", >>> line >>> 226, in copy_image >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>> compression=compression) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs >>> .py", line 110, in copy_file >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>> compression=compression) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs >>> .py", line 196, in copy_file >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>> on_execute=on_execute, on_completion=on_completion) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", >>> line 231, in exec >>> ute >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] return >>> processutils.execute(*cmd, **kwargs) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py" >>> , line 424, in execute >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>> cmd=sanitized_cmd) >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>> ProcessExecutionError: Unexpected error while running command. >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Command: scp -r >>> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Exit code: 1 >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stdout: u'' >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stderr: >>> u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or >>> directory\n' >>> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] >>> 2019-07-24 13:47:14.380 58902 INFO nova.compute.manager >>> [req-356fce31-5e98-425f-89d6-8a98664d31ad >>> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 >>> - default default] [instance: >>> 5a6dae58-00d6-4317-b635-909fdf09ac49] Swapping old allocation on >>> 3da11b23-5ef8-4471-a3c9-5f743b0bbfd7 held by migration >>> da66c416-2636-421b-99cb-470bd07dad4c for instance >>> 2019-07-24 13:47:14.979 58902 INFO nova.compute.manager >>> [req-356fce31-5e98-425f-89d6-8a98664d31ad >>> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 >>> - default default] [instance: >>> 5a6dae58-00d6-4317-b635-909fdf09ac49] Successfully reverted task >>> state from resize_migrating on failure for instance. >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> [req-356fce31-5e98-425f-89d6-8a98664d31ad >>> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 >>> - default default] Exception during message handling: >>> ProcessExecutionError: Unexpected error while running command. >>> Command: scp -r >>> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> Exit code: 1 >>> Stdout: u'' >>> Stderr: >>> u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or >>> directory\n' >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> Traceback (most recent call last): >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in >>> _process_incoming >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> res = self.dispatcher.dispatch(message) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in >>> dispatch >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> return self._do_dispatch(endpoint, method, ctxt, args) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in >>> _do_dispatch >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> result = func(ctxt, **new_args) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in >>> wrapped >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> function_name, call_dict, binary) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in >>> __exit__ >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> self.force_reraise() >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in >>> force_reraise >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in >>> wrapped >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> return f(self, context, *args, **kw) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in >>> decorated_function >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> "Error: %s", e, instance=instance) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in >>> __exit__ >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> self.force_reraise() >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in >>> force_reraise >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in >>> decorated_function >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> return function(self, context, *args, **kwargs) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in >>> decorated_function >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> return function(self, context, *args, **kwargs) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in >>> decorated_function >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> kwargs['instance'], e, sys.exc_info()) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in >>> __exit__ >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> self.force_reraise() >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in >>> force_reraise >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in >>> decorated_function >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> return function(self, context, *args, **kwargs) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4240, in >>> resize_instance >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> self._revert_allocation(context, instance, migration) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in >>> __exit__ >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> self.force_reraise() >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in >>> force_reraise >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4237, in >>> resize_instance >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> instance_type, clean_shutdown) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4275, in >>> _resize_instance >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> timeout, retry_interval) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8200, in >>> migrate_disk_and_power_off >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> shared_storage) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in >>> __exit__ >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> self.force_reraise() >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in >>> force_reraise >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> six.reraise(self.type_, self.value, self.tb) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8185, in >>> migrate_disk_and_power_off >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> compression=compression) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in >>> copy_image >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> compression=compression) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in >>> copy_file >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> compression=compression) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in >>> copy_file >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> on_execute=on_execute, on_completion=on_completion) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", >>> line 231, in execute >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> return processutils.execute(*cmd, **kwargs) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> File >>> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in >>> execute >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> cmd=sanitized_cmd) >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> ProcessExecutionError: Unexpected error while running command. >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> Command: scp -r >>> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Exit code: 1 >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stdout: u'' >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> Stderr: >>> u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or >>> directory\n' >>> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server >>> 2019-07-24 13:47:17.065 58902 INFO nova.virt.libvirt.imagecache >>> [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] image >>> ca89475a-cf15-46d1-ab2d-c2b96c403248 at >>> (/var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07): >>> checking >>> 2019-07-24 13:47:17.066 58902 INFO nova.virt.libvirt.imagecache >>> [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] Active base >>> files: >>> /var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07 >>> 2019-07-24 13:47:27.768 58902 INFO nova.compute.manager [-] >>> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] VM Stopped >>> (Lifecycle Event) >>> 2019-07-24 13:47:27.918 58902 INFO nova.compute.manager >>> [req-e53e928d-81fe-4185-9e62-abc2113e838c - - - - -] [instance: >>> 5a6dae58-00d6-4317-b635-909fdf09ac49] During >>> _sync_instance_power_state the DB power_state (1) does not match >>> the vm_power_state from the hypervisor (4). Updating power_state >>> in the DB to match the hypervisor. >>> >>> >>> The disk has its root FS n an RBD device (a volume created by >>> cinder) and it has its ephemeral disk on a local LVM volume >>> created by nova. >>> >>> It seems that the >>> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 is not there when scp is trying to copy it to the other >>> side. >>> >>> Any ideas how to fix this? >>> >>> The nova config on the nodes looks like this: >>> >>> # Ansible managed >>> [DEFAULT] >>> allow_resize_to_same_host = True >>> # Compute >>> compute_driver = libvirt.LibvirtDriver >>> # Scheduler >>> cpu_allocation_ratio = 2.0 >>> # Logs / State >>> debug = False >>> # Hypervisor >>> default_ephemeral_format = ext4 >>> disk_allocation_ratio = 1.0 >>> # Api's >>> enabled_apis = osapi_compute,metadata >>> executor_thread_pool_size = 64 >>> fatal_deprecations = False >>> # Configdrive >>> force_config_drive = False >>> host = compute8.mgmt.lab.mydomain.com >>> image_cache_manager_interval = 0 >>> instance_name_template = instance-%08x >>> # Ceilometer notification configurations >>> instance_usage_audit = True >>> instance_usage_audit_period = hour >>> instances_path = /var/lib/nova/instances >>> ## Vif >>> libvirt_vif_type = ethernet >>> log_dir = /var/log/nova >>> # Metadata >>> metadata_workers = 16 >>> # Network >>> my_ip = 192.168.56.47 >>> notify_on_state_change = vm_and_task_state >>> osapi_compute_workers = 16 >>> ram_allocation_ratio = 1.0 >>> reserved_host_disk_mb = 0 >>> reserved_host_memory_mb = 2048 >>> resume_guests_state_on_host_boot = False >>> rootwrap_config = /etc/nova/rootwrap.conf >>> rpc_response_timeout = 60 >>> service_down_time = 120 >>> state_path = /var/lib/nova >>> # Rpc all >>> transport_url = >>> rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e >>> 8cf8bd7 at 192.168.56.179:5671//nova >>> # Disable stderr logging >>> use_stderr = False >>> vif_plugging_is_fatal = True >>> vif_plugging_timeout = 30 >>> >>> [api] >>> auth_strategy = keystone >>> enable_instance_password = True >>> use_forwarded_for = False >>> vendordata_jsonfile_path = /etc/nova/vendor_data.json >>> >>> # Cache >>> [cache] >>> backend = oslo_cache.memcache_pool >>> enabled = true >>> memcache_servers = >>> 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 >>> >>> # Cinder >>> [cinder] >>> cafile = /etc/ssl/certs/ca-certificates.crt >>> catalog_info = volumev3:cinderv3:internalURL >>> cross_az_attach = True >>> os_region_name = A_Lab >>> >>> [conductor] >>> workers = 16 >>> >>> [filter_scheduler] >>> available_filters = nova.scheduler.filters.all_filters >>> enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, >>> AggregateRamFilter, ComputeFilter, AggregateCoreFilter, >>> DiskFilter, AggregateDiskFilter, AggregateNumInstancesFilter, >>> AggregateIoOpsFilter, ComputeCapabilitiesFilter, >>> ImagePropertiesFilter, ServerGroupAntiAffinityFilter, >>> ServerGroupAffinityFilter, NUMATopologyFilter, SameHostFilter, >>> DifferentHostFilter >>> host_subset_size = 10 >>> max_instances_per_host = 50 >>> max_io_ops_per_host = 10 >>> ram_weight_multiplier = 5.0 >>> tracks_instance_changes = True >>> weight_classes = nova.scheduler.weights.all_weighers >>> >>> # Glance >>> [glance] >>> api_servers = https://vip.mgmt.lab.mydomain.com:9292 >>> cafile = /etc/ssl/certs/ca-certificates.crt >>> >>> [key_manager] >>> fixed_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> >>> [keystone_authtoken] >>> auth_type = password >>> auth_uri = https://vip.mgmt.lab.mydomain.com:5000 >>> auth_url = https://vip.mgmt.lab.mydomain.com:35357 >>> cafile = /etc/ssl/certs/ca-certificates.crt >>> insecure = False >>> memcache_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> # if your memcached server is shared, use these settings to avoid >>> cache poisoning >>> memcache_security_strategy = ENCRYPT >>> memcached_servers = >>> 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 >>> password = xxxxxxxxxxxxxxxxxxxxxxx >>> >>> project_domain_id = default >>> project_name = service >>> region_name = A_Lab >>> token_cache_time = 300 >>> user_domain_id = default >>> username = nova >>> >>> [libvirt] >>> disk_cachemodes = network=writeback >>> hw_disk_discard = unmap >>> images_rbd_ceph_conf = /etc/ceph/ceph.conf >>> images_rbd_pool = vms >>> images_type = lvm >>> images_volume_group = ssd_nova >>> inject_key = False >>> inject_partition = -2 >>> inject_password = False >>> live_migration_tunnelled = True >>> live_migration_uri = >>> "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa" >>> rbd_secret_uuid = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> # ceph rbd support >>> rbd_user = cinder >>> remove_unused_resized_minimum_age_seconds = 3600 >>> use_virtio_for_bridges = True >>> virt_type = kvm >>> >>> # Neutron >>> [neutron] >>> auth_type = password >>> # Keystone client plugin authentication URL option >>> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 >>> cafile = /etc/ssl/certs/ca-certificates.crt >>> default_floating_pool = public >>> insecure = False >>> metadata_proxy_shared_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> # Keystone client plugin password option >>> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> project_domain_name = Default >>> project_name = service >>> region_name = A_Lab >>> service_metadata_proxy = True >>> url = https://vip.mgmt.lab.mydomain.com:9696 >>> user_domain_name = Default >>> # Keystone client plugin username option >>> username = neutron >>> >>> [oslo_concurrency] >>> lock_path = /var/lock/nova >>> # Notifications >>> [oslo_messaging_notifications] >>> driver = messagingv2 >>> notification_topics = notifications >>> transport_url = >>> rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.179:5671//nova >>> >>> [oslo_messaging_rabbit] >>> rpc_conn_pool_size = 30 >>> ssl = True >>> >>> # Placement >>> [placement] >>> auth_type = "password" >>> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 >>> cafile = /etc/ssl/certs/ca-certificates.crt >>> insecure = False >>> os_interface = internal >>> os_region_name = A_Lab >>> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> project_domain_name = Default >>> project_name = service >>> user_domain_name = Default >>> username = placement >>> >>> [quota] >>> cores = 20 >>> injected_file_content_bytes = 10240 >>> injected_file_path_length = 255 >>> injected_files = 5 >>> instances = 10 >>> key_pairs = 100 >>> max_age = 0 >>> metadata_items = 128 >>> ram = 32768 >>> server_group_members = 10 >>> server_groups = 10 >>> >>> [scheduler] >>> discover_hosts_in_cells_interval = 60 >>> host_manager = host_manager >>> max_attempts = 5 >>> periodic_task_interval = 60 >>> scheduler_driver = filter_scheduler >>> >>> [spice] >>> agent_enabled = True >>> enabled = True >>> # Console Url and binds >>> html5proxy_base_url = >>> https://dashboard.iaas.lab.mydomain.com:6080/spice_auto.html >>> server_listen = 192.168.56.47 >>> server_proxyclient_address = 192.168.56.47 >>> >>> [upgrade_levels] >>> compute = auto >>> >>> [vnc] >>> enabled = False >>> >>> [wsgi] >>> api_paste_config = /etc/nova/api-paste.ini >>> secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO >>> >>> >>> Thank you in advance for any suggestions. >>> >>> Kind regards, >>> Laszlo >> >> >> >> >> From zheng.ma at intel.com Thu Jul 25 15:26:07 2019 From: zheng.ma at intel.com (Ma, Zheng) Date: Thu, 25 Jul 2019 15:26:07 +0000 Subject: [dev][nova][glance][cinder] introducing 'compressed' image container_format Message-ID: Hello Nova developers, Cinder is working on a feature that will allow it to compress volumes uploaded to Glance [0,1]. We are proposing a new container_format for Glance named 'compressed' [2]. The idea is that a consumer of a 'compressed' image would read the file to determine the kind of compression and take appropriate action (either successfully decompress to find a payload that matches the image's disk_format, or fail with 'unsupported container_format'). Our preliminary testing with a raw image with container_format == compressed, disk_format == raw is that Nova boots a VM from the image but the VM appears to be unusable. Nova seems taking it as a real RAW image as disk_format indicates. So we want to open a discussion about handling this. Thanks! [0] cinder spec patch https://review.opendev.org/#/c/652275/ [1] cinder code implementation https://review.opendev.org/#/c/668825 [2] glance spec patch https://review.opendev.org/#/c/670454/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ajitharobert01 at gmail.com Fri Jul 26 07:14:47 2019 From: ajitharobert01 at gmail.com (Ajitha Robert) Date: Fri, 26 Jul 2019 12:44:47 +0530 Subject: [cinder][ceph][RBD mirroring] Error while creating a mirrored volume Message-ID: Hi all Following picture shows my setup [image: image.png] I m facing following error when i issue a command to enable rbd mirroring on a particular image either manually or through cinder. *"rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x7fb364025310 finish: resending after timeout"* in rbd-mirroring.log and getting *"failed to get omap key: client_0921b3ce-716a-4462-b3cf-86b5102bfd9"* error in osd.log. . Should I enable anything else apart from timeout parametes, if the distance between the two ceph clusters is greater than 300km? when I create a bootable volume of 100 GB with a glance image.Image get downloaded and from cinder, volume log throws with "volume is busy deleting volume that has snapshot" . Image was enabled with exclusive lock, journaling, layering, object-map, fast-diff and deep-flatten Cinder volume is in error state but the rbd image is created in primary but not in secondary. pls check the logs 1)Log for manual rbd image creation http://paste.openstack.org/show/754766/ 2)Log for 16gb volume created from cinder, status in cinder volume is available http://paste.openstack.org/show/754767/ 3)Log for 100gb volume created from cinder, status in cinder volume is error http://paste.openstack.org/show/754769/ -- *Regards,Ajitha R* -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 51778 bytes Desc: not available URL: From cdent+os at anticdent.org Fri Jul 26 15:43:17 2019 From: cdent+os at anticdent.org (Chris Dent) Date: Fri, 26 Jul 2019 16:43:17 +0100 (BST) Subject: [placement] update 19-29 Message-ID: HTML: https://anticdent.org/placement-update-19-29.html Welcome to a rushed pupdate 19-29. My morning was consumed by other things. A reminder: The Placement project holds office hours every Wednesday at 1500 UTC in the `#openstack-placement` IRC channel. If you have a topic that needs some synchronous discussion, then is an ideal time. Just start talking! # Most Important The two main things on the Placement radar are implementing Consumer Types and cleanups, performance analysis, and documentation related to nested resource providers. # What's Changed * The [api-ref](https://docs.openstack.org/api-ref/placement/) has moved to `docs.openstack.org` from `developer.openstack.org`. Redirects are in place. * Both traits and resource classes are [now cached](https://review.opendev.org/672298) per request, allowing for some name to id and id to name optimizations. * A new zuul template is [being used](https://review.opendev.org/671257) in placement that means fewer irrelevant tempest tests are run on placement changes. # Stories/Bugs (Numbers in () are the change since the last pupdate.) There are 21 (-1) stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 (0) are [untagged](https://storyboard.openstack.org/#!/worklist/580). 2 (0) are [bugs](https://storyboard.openstack.org/#!/worklist/574). 5 (0) are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 10 (0) are [rfes](https://storyboard.openstack.org/#!/worklist/594). 4 (0) are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 17 (0). * Placement related nova [in progress bugs](https://goo.gl/vzGGDQ) on launchpad: 4 (0). # osc-placement osc-placement is currently behind by 12 microversions. * Add support for multiple member_of. There's been some useful discussion about how to achieve this, and a consensus has emerged on how to get the best results. # Main Themes ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. * A WIP, as microversion 1.37, has started. ## Cleanup Cleanup is an overarching theme related to improving documentation, performance and the maintainability of the code. The changes we are making this cycle are fairly complex to use and are fairly complex to write, so it is good that we're going to have plenty of time to clean and clarify all these things. # Other Placement Miscellaneous changes can be found in [the usual place](https://review.opendev.org/#/q/project:openstack/placement+status:open). There are two [os-traits changes](https://review.opendev.org/#/q/project:openstack/os-traits+status:open) being discussed. And zero [os-resource-classes changes](https://review.opendev.org/#/q/project:openstack/os-resource-classes+status:open) (yay!). # Other Service Users New discoveries are added to the end. Merged stuff is removed. Anything that has had no activity in 4 weeks has been removed. * Nova: nova-manage: heal port allocations * Cyborg: Placement report * helm: add placement chart * Nova: Use OpenStack SDK for placement * libvirt: report pmem namespaces resources by provider tree * Nova: Remove PlacementAPIConnectFailure handling from AggregateAPI * Nova: support move ops with qos ports * Nova: get_ksa_adapter: nix by-service-type confgrp hack * Blazar: Create placement client for each request * nova: Support filtering of hosts by forbidden aggregates * blazar: Send global_request_id for tracing calls * Nova: Update HostState.\*\_allocation_ratio earlier * Nova: WIP: Add a placement audit command * zun: [WIP] Use placement for unified resource management * Kayobe: Build placement images by default * blazar: Fix placement operations in multi-region deployments * Watcher: Getting data from placement when updating datamodel * Nova: libvirt: Start reporting PCPU inventory to placement * chef: placement_api: create valid apache config before installing package * tempest: Add placement API methods for testing routed provider nets * openstack-helm: Build placement in OSH-images # End If we get the chance, it will be interesting to start working with placement with 1 million providers. Just to see. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent From laszlo.budai at gmail.com Fri Jul 26 15:47:29 2019 From: laszlo.budai at gmail.com (Budai Laszlo) Date: Fri, 26 Jul 2019 18:47:29 +0300 Subject: [nova] instence running, but status is Error Message-ID: <66f4bfcc-023d-a12b-830c-6a88953d88b2@gmail.com> Dear all, I am testing different migration scenarios in our openstack, and I ended up with an instance that is running (I can access it by ssh), but its Status is Error. and its task_state is resize_migrating. I suppose that I got here due to the fact that the `allow_resize_to_same_host` is set to True, and the scheduler has tried to use the same compute node in the migration as described here: https://bugs.launchpad.net/nova/+bug/1811235. Is there any possibility to be able to manage again that instance using openstack (except to delete it)? I have tried the shut off (from horizon) and it gives me an error: fault: code: 400 created: '2019-07-26T14:46:05Z' message: Unavailable console type novnc. Thank you, Laszlo From nsitlani03 at gmail.com Fri Jul 26 16:24:25 2019 From: nsitlani03 at gmail.com (Namrata Sitlani) Date: Fri, 26 Jul 2019 21:54:25 +0530 Subject: [nova] instence running, but status is Error In-Reply-To: <66f4bfcc-023d-a12b-830c-6a88953d88b2@gmail.com> References: <66f4bfcc-023d-a12b-830c-6a88953d88b2@gmail.com> Message-ID: Hi Laszlo, I am not sure about the scenario, but have you tried hard rebooting the instance? Thanks Namrata On Fri, Jul 26, 2019 at 9:20 PM Budai Laszlo wrote: > Dear all, > > I am testing different migration scenarios in our openstack, and I ended > up with an instance that is running (I can access it by ssh), but its > Status is Error. and its task_state is resize_migrating. I suppose that I > got here due to the fact that the `allow_resize_to_same_host` is set to > True, and the scheduler has tried to use the same compute node in the > migration as described here: https://bugs.launchpad.net/nova/+bug/1811235. > > > Is there any possibility to be able to manage again that instance using > openstack (except to delete it)? > I have tried the shut off (from horizon) and it gives me an error: > > fault: > code: 400 > created: '2019-07-26T14:46:05Z' > message: Unavailable console type novnc. > > > > Thank you, > Laszlo > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Fri Jul 26 16:36:35 2019 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 26 Jul 2019 18:36:35 +0200 Subject: [kolla][nova][cinder] Got Gateway-Timeout error on VM evacuation if it has volume attached. In-Reply-To: References: <20190725081407.lkapi4skj2hcllzi@localhost> <1cdc4a94-1dc2-420d-5ae8-663d903d2dd2@gmail.com> Message-ID: If time gets out of sync too much, you get auth errors due to tokens being from the future / from a too distant past. Kind regards, Radek pt., 26 lip 2019 o 17:12 Eddie Yen napisał(a): > Roger that, thanks for explanation. > > I think there's another reason to me that get this issue. > The environment is stayed without any internet nor local NTP server, until > the last test. > Before the test, the nova and cinder services became unstable because they > keeping up and down. And I found that the clock are out of sync between > nodes. > We let one of the node can connect outside and let NTP client pointed to > that one on other nodes. Then problem solved. > Of course the test is successful. > > I'm not sure but that's a one of reason right? > > But I think I still need to try optimize the timeout value since the API > response is slow when shutting down a node. > Wonder know why it become slow when a node down. > > I'll try to gain up rpc_response_timeout in Cinder and do more testing. > > Matt Riedemann 於 2019年7月26日 週五 下午9:42寫道： > >> On 7/25/2019 11:54 PM, Eddie Yen wrote: >> > And I think I should gain rpc_response_timeout rather than >> > long_rpc_timeout in nova. >> >> Since Cinder doesn't have the long_rpc_timeout option like Nova you're >> only option is to bump up the rpc_response_timeout in Cinder but that >> will be used by all RPC calls in Cinder, not just the >> initialize/terminate connection calls for attachments. Maybe that's not >> a problem, but long_rpc_timeout in Nova allows us to pick which RPC >> calls to use that on rather than everywhere. The changes to Cinder >> shouldn't be that hard if they follow the Nova patch [1]. >> >> [1] https://review.opendev.org/#/c/566696/ >> >> -- >> >> Thanks, >> >> Matt >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From laszlo.budai at gmail.com Fri Jul 26 16:37:18 2019 From: laszlo.budai at gmail.com (Budai Laszlo) Date: Fri, 26 Jul 2019 19:37:18 +0300 Subject: [nova] instence running, but status is Error In-Reply-To: References: <66f4bfcc-023d-a12b-830c-6a88953d88b2@gmail.com> Message-ID: Hi Namrata, I wasn't able to reboot, but the `nova reset-state --active ` command has helped to get the instance back to active state. Kind regards, Laszlo On 7/26/19 7:24 PM, Namrata Sitlani wrote: > Hi Laszlo, > > I am not sure about the scenario,  but have you tried hard rebooting the instance? > > Thanks > Namrata > > On Fri, Jul 26, 2019 at 9:20 PM Budai Laszlo > wrote: > > Dear all, > > I am testing different migration scenarios in our openstack, and I ended up with an instance that is running (I can access it by ssh), but its Status is Error. and its task_state is resize_migrating. I suppose that I got here due to the fact that the `allow_resize_to_same_host` is set to True, and the scheduler has tried to use the same compute node in the migration as described here: https://bugs.launchpad.net/nova/+bug/1811235. > > Is there any possibility to be able to manage again that instance using openstack (except to delete it)? > I have tried the shut off (from horizon) and it gives me an error: > > fault: >   code: 400 >   created: '2019-07-26T14:46:05Z' >   message: Unavailable console type novnc. > > > > Thank you, > Laszlo > From nsitlani03 at gmail.com Fri Jul 26 16:44:19 2019 From: nsitlani03 at gmail.com (Namrata Sitlani) Date: Fri, 26 Jul 2019 22:14:19 +0530 Subject: [nova] instence running, but status is Error In-Reply-To: References: <66f4bfcc-023d-a12b-830c-6a88953d88b2@gmail.com> Message-ID: Hi Laszlo, Sounds good. Thanks, Namrata On Fri, 26 Jul 2019 at 10:12 PM, Budai Laszlo wrote: > Hi Namrata, > > I wasn't able to reboot, but the `nova reset-state --active ` > command has helped to get the instance back to active state. > > Kind regards, > Laszlo > > On 7/26/19 7:24 PM, Namrata Sitlani wrote: > > Hi Laszlo, > > > > I am not sure about the scenario, but have you tried hard rebooting the > instance? > > > > Thanks > > Namrata > > > > On Fri, Jul 26, 2019 at 9:20 PM Budai Laszlo > wrote: > > > > Dear all, > > > > I am testing different migration scenarios in our openstack, and I > ended up with an instance that is running (I can access it by ssh), but its > Status is Error. and its task_state is resize_migrating. I suppose that I > got here due to the fact that the `allow_resize_to_same_host` is set to > True, and the scheduler has tried to use the same compute node in the > migration as described here: https://bugs.launchpad.net/nova/+bug/1811235. > > > > Is there any possibility to be able to manage again that instance > using openstack (except to delete it)? > > I have tried the shut off (from horizon) and it gives me an error: > > > > fault: > > code: 400 > > created: '2019-07-26T14:46:05Z' > > message: Unavailable console type novnc. > > > > > > > > Thank you, > > Laszlo > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From openstack at fried.cc Fri Jul 26 18:07:53 2019 From: openstack at fried.cc (Eric Fried) Date: Fri, 26 Jul 2019 13:07:53 -0500 Subject: [nova][oslo.log] global_request_id help Message-ID: <75394358-4c3a-07cf-eff7-3613939b037c@fried.cc> While doing something unrelated, I noticed this code [0] that shouldn't need to exist if global_request_id is working. I threw out some debug patches [1] to dump headers. Results: [2] (nova), [3] (placement). We're tracking global_request_id ``req-a51f...``. Nova is sending the global_request_id properly in the X-Openstack-Request-Id header, and Placement is receiving it correctly, because it's showing up in the expected spot in the Placement log line (after the opening bracket, per the logging_context_format_string in placement.conf [4]). Placement is sending back the request_id (``req-b3ce...``) in the x-openstack-request-id header in the response. (I thought this was a bug, but it turns out this is correct per the design [5].) So the problem is that the Nova log is printing ``None`` for the global_request_id. Based on logging_context_format_string in nova-cpu.conf [6], I expect it in the first field after the opening bracket, just like for Placement. If the global_request_id were showing up in the Nova log, we wouldn't need to correlate the Nova and Placement sides of this conversation via the Placement request_id, and we could get rid of [0] and its usages. I don't understand how LOG.xxx() is supposed to glean this value. Can someone help me find the disconnect? Thanks, efried [0] https://opendev.org/openstack/nova/src/commit/d358df32f7a2a1125962e27f2425359c768f79c1/nova/scheduler/client/report.py#L175-L177 [1] https://review.opendev.org/#/q/topic:where-are-you-global_request_id [2] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/logs/screen-n-cpu.txt.gz#_Jul_26_14_57_18_460328 [3] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/logs/screen-placement-api.txt.gz#_Jul_26_14_57_18_458647 [4] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/logs/etc/placement/placement.conf.txt.gz [5] https://specs.openstack.org/openstack/oslo-specs/specs/pike/global-req-id.html#oslo-middleware-request-id [6] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/logs/etc/nova/nova-cpu.conf.txt.gz From openstack at fried.cc Fri Jul 26 18:33:55 2019 From: openstack at fried.cc (Eric Fried) Date: Fri, 26 Jul 2019 13:33:55 -0500 Subject: [nova][sfe] Train Spec Freeze Exception Process Message-ID: <89cafacc-4131-7903-beae-b29bdb80e7cb@fried.cc> Hello Nova. Spec freeze happened, which means we won't be approving any more specs for Train. Unless we do. >From this point on, you will be required to run the following gauntlet if you want your spec to land in Train. - Put your name and a link to your spec on the agenda for the next Nova meeting [1][2]. - Send a note to openstack-discuss tagged with [nova][sfe] (like this one is) including the spec for which you are requesting an exception, and noting the date of the meeting. - Attend that meeting and state your case. Assuming a quorum of cores is present, we will attempt to render a decision, or at least indicate what further actions are necessary. Thanks, efried [1] https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting [2] If you can't attend a meeting, or "next Thursday" is just too long to wait, say so in your email and we'll endeavor to discuss it entirely on the mailing list. From corey.bryant at canonical.com Fri Jul 26 19:15:07 2019 From: corey.bryant at canonical.com (Corey Bryant) Date: Fri, 26 Jul 2019 15:15:07 -0400 Subject: [goal][python3] Train unit tests weekly update (goal-7) Message-ID: This is the goal-7 weekly update for the "Update Python 3 test runtimes for Train" goal [1]. There are 7 weeks remaining for completion of Train community goals [2]. == What's the Goal? == To ensure (in the Train cycle) that all official OpenStack repositories with Python 3 unit tests are exclusively using the 'openstack-python3-train-jobs' Zuul template or one of its variants (e.g. 'openstack-python3-train-jobs-neutron') to run unit tests, and that tests are passing. This will ensure that all official projects are running py36 and py37 unit tests in Train. For complete details please see [1]. == Ongoing Work == All patches have been submitted for all applicable projects. Open patches needing reviews: https://review.openstack.org/#/q/topic:python3-train+is:open Failing patches: https://review.openstack.org/#/q/topic:python3-train+status:open+(+label:Verified-1+OR+label:Verified-2+) Patch automation scripts needing review: https://review.opendev.org/#/c/666934 == Completed Work == Merged patches: https://review.openstack.org/#/q/topic:python3-train+is:merged == How can you help? == Please take a look at the failing patches and help fix any failing unit tests for your projects. Python 3.7 unit tests will be self-testing in Zuul. == Reference Material == [1] Goal description: https://governance.openstack.org/tc/goals /train/python3-updates.html [2] Train release schedule: https://releases.openstack.org/train/schedule.html (see R-5 for "Train Community Goals Completed") Storyboard: https://storyboard.openstack.org/#!/story/2005924 Porting to Python 3.7: https://docs.python.org/3/whatsnew/3.7.html#porting-to-python-3-7 Python Update Process: https://opendev.org/openstack/governance/src/branch/master/resolutions/20181024-python-update-process.rst Train runtimes: https://opendev.org/openstack/governance/src/branch/master/reference/runtimes/train.rst Thanks, Corey -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrian at fleio.com Fri Jul 26 19:33:47 2019 From: adrian at fleio.com (Adrian Andreias) Date: Fri, 26 Jul 2019 22:33:47 +0300 Subject: [nova-lxd] no one uses nova-lxd? Message-ID: Hey, We were planing to migrate some thousand containers from OpenVZ 6 to Nova-LXD this fall and I know at least one company with the same plans. I read the message about current team retiring from the project. Unfortunately we don't have the manpower to invest heavily in the project development. We would however be able to allocate a few hours per month, at least for bug fixing. So I'm curios if there are organizations using or planning to use Nova-LXD in production and they have the know-how and time to contribute. It would be a pity if the the project dies. Cheers! - Adrian Andreias https://fleio.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From raubvogel at gmail.com Fri Jul 26 19:54:56 2019 From: raubvogel at gmail.com (Mauricio Tavares) Date: Fri, 26 Jul 2019 15:54:56 -0400 Subject: Forcing restart of a worker node with running guest In-Reply-To: References: <0798fb66-2028-3a98-38b4-a5c5526b140f@gmail.com> Message-ID: On Thu, Jul 25, 2019 at 4:24 PM Mauricio Tavares wrote: > > On Thu, Jul 25, 2019 at 3:44 PM Matt Riedemann wrote: > > > > On 7/25/2019 11:04 AM, Mauricio Tavares wrote: > > > I found out when it was taking 30 min to delete a guest. So, what I can > > > do in a forceful way? > > > > > > 1. How to kill the guest? Can I kill it through virsh or openstack > > > compute service will get sad? > > > > I would try to avoid this if possible, but you might need to kill the > > guest in the hypervisor if doing it through nova won't get the job done. > > What happens in nova-compute is undefined, but you'd probably see some > > errors as expected if you're doing anything with that server at the > > hypervisor layer, like trying to get the guest power state. > > > > What nova is tracking and what is in the hypervisor are different > > things, and if you delete the guest out of band from nova, you'll need > > to delete the server to sync the nova database. If the delete is stuck > > in the compute API, thinking it's already deleting (I think we have an > > old bug for that and force delete, and I hit something similar today), > > you could try resetting the server status to ERROR [1] and then try > > deleting it in the API again. > > > > > 2. What would happen if I stop the compute service? > > > > This won't really do anything to the guest in the hypervisor unless [2] > > tries to change the guest state on restart. In my experience that option > > has not been very reliable / predictable. > > > > > 3. What would happen if I get really annoyed and tell worker node to reboot? > > > > Pretty much the same as #2 from a nova perspective I think. Depending on > > how libvirt and/or the guest domain is configured, the libvirt-guest > > service might try to resume the guest. > > > Does that mean it is using the standard libvirt config files? > > > [1] openstack server set --state error > > [2] > > https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.resume_guests_state_on_host_boot > > > Thanks for the info. It turned out the issue is hardware > related, so shutting the worker node down is way past the realm of > possibility into the realm of it will happen today. > Update: after I dealt with the hardware, I now was able to tell the instance to go to silicon heaven: [raub at openstack-hn ~(keystone_admin)]$ openstack server list +--------------------------------------+----------+--------+--------------------------------------------+--------+-------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+----------+--------+--------------------------------------------+--------+-------------+ | 1f76ca35-9d7f-4403-ae72-bcbfa1cc9b99 | desktop1 | ERROR | physnet1=10.20.20.66, 192.168.20.66 | centos | netro.small | +--------------------------------------+----------+--------+--------------------------------------------+--------+-------------+ [raub at openstack-hn ~(keystone_admin)]$ openstack server delete desktop1 [raub at openstack-hn ~(keystone_admin)]$ openstack server list [raub at openstack-hn ~(keystone_admin)]$ Thank you for providing all the different options to account for the possible increasing degrees of things going bad! I will save this message for next time... > > -- > > > > Thanks, > > > > Matt > > From openstack at fried.cc Fri Jul 26 20:06:20 2019 From: openstack at fried.cc (Eric Fried) Date: Fri, 26 Jul 2019 15:06:20 -0500 Subject: [nova] Train Cycle Themes - An Interactive Experience Message-ID: <888d482f-29eb-d798-9ef6-7decbbafece9@fried.cc> Hi all. Now that we're past spec freeze, Matt suggested I write a report on how we're doing against our cycle themes. That sounded really tedious and boring and project-managementy, so I thought I would spread the joy. Bear with me, this is going to be a little different. I've created a null review in the nova-specs repository [1]. I'd like us to use this as a mechanism to collaboratively gather thoughts via review comments [2]. I've made a start on some bits. Have fun! efried [1] https://review.opendev.org/#/c/673095/ [2] I could have used an etherpad. But this will be more... permanent. From aschultz at redhat.com Fri Jul 26 21:00:23 2019 From: aschultz at redhat.com (Alex Schultz) Date: Fri, 26 Jul 2019 15:00:23 -0600 Subject: [tripleo] Proposing Kevin Carter (cloudnull) for tripleo-ansible core Message-ID: Hey folks, I'd like to propose Kevin as a core for the tripleo-ansible repo (e.g. tripleo-ansible-core). He has made excellent progress centralizing our ansible roles and improving the testing around them. Please reply with your approval/objections. If there are no objections, we'll add him to tripleo-ansible-core next Friday Aug 2, 2019. Thanks, -Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From emilien at redhat.com Fri Jul 26 21:16:14 2019 From: emilien at redhat.com (Emilien Macchi) Date: Fri, 26 Jul 2019 17:16:14 -0400 Subject: [tripleo] Proposing Kevin Carter (cloudnull) for tripleo-ansible core In-Reply-To: References: Message-ID: +2 On Fri, Jul 26, 2019 at 5:08 PM Alex Schultz wrote: > Hey folks, > > I'd like to propose Kevin as a core for the tripleo-ansible repo (e.g. > tripleo-ansible-core). He has made excellent progress centralizing our > ansible roles and improving the testing around them. > > Please reply with your approval/objections. If there are no objections, > we'll add him to tripleo-ansible-core next Friday Aug 2, 2019. > > Thanks, > -Alex > -- Emilien Macchi -------------- next part -------------- An HTML attachment was scrubbed... URL: From doug at doughellmann.com Fri Jul 26 21:26:43 2019 From: doug at doughellmann.com (Doug Hellmann) Date: Fri, 26 Jul 2019 17:26:43 -0400 Subject: [all] [tc] General questions on PDF community goal In-Reply-To: References: Message-ID: Akihiro Motoki writes: > Hi, > > I have a couple of general questions on PDF community goal. > > What is the criteria of completing the PDF community goal? > Is it okay if we publish a PDF deliverable anyway? > > During working on it, I hit the following questions. > We already reached Train-2 milestone, so I think it is the time to clarify > the detail criteria of completing the community goal. > > - Where should the generated PDF file to be located and What is the > recommended PDF file name? > /.pdf? index.pdf? Any path and any > file is okay? The job will run the sphinx instructions to build PDFs, and then copy them from build/latex (or build/pdf, I can't remember) into the build/html directory so they are published as part of the project's series-specific documentation set. Project teams should not add anything to their HTML documentation build to move, rename, etc. the PDF files. That will all be done by the job changes Stephen has been developing. Project teams should ensure there is exactly 1 PDF file being built and that it has a meaningful name. That could be ${repository_name}.pdf as you suggest, but it could be something else, for now. > - Do we create a link to the PDF file from index.html (per-project top page)? > If needed, perhaps this would be a thing covered by openstackdocstheme. > Otherwise, how can normal consumers know PDF documents? Not yet. We should be able to do that automatically through the theme by looking at the latex build parameters. If we do it in the theme, rather than having projects add link content to their builds, we can ensure that all projects have the link in a consistent location in their docs pages with 1 patch. If you want to work on that, it would be good, but it isn't part of the goal. > - Which repositories need to provide PDF version of documents? > My understanding (amotoki) is that repositories with > 'publish-openstack-docs-pti' should publish PDF doc. right? Yes, all repositories that follow the documentation PTI. > - Do we need PDF version of release notes? > - Do we need PDF version of API reference? The goal is focused on publishing the content of the doc/source directory in each repository. There is no need to deal with release notes or the API reference. We may work on those later, but not for this goal. > I see no coordination efforts recently and am afraid that individual > projects cannot decide whether patches to their repositories are okay > to merge. The goal champions have been on vacation. Have a bit of patience, please. :-) In the mean time, if there are questions about specific patches, please raise those here on the mailing list. The most important thing to accomplish is to ensure that one PDF builds *at all* from the content in doc/source in each repo. The goal was purposefully scoped to this one task to allow teams to focus on getting successful PDF builds from their content, because we weren't sure what issues we might encounter. We can come back around later and improve the experience of consuming the PDFs but there is no point in making a bunch of decisions about how to do that until we know we have the files to publish. -- Doug From johfulto at redhat.com Fri Jul 26 22:00:45 2019 From: johfulto at redhat.com (John Fulton) Date: Fri, 26 Jul 2019 18:00:45 -0400 Subject: [tripleo] Proposing Kevin Carter (cloudnull) for tripleo-ansible core In-Reply-To: References: Message-ID: +2 On Fri, Jul 26, 2019, 5:18 PM Emilien Macchi wrote: > +2 > > On Fri, Jul 26, 2019 at 5:08 PM Alex Schultz wrote: > >> Hey folks, >> >> I'd like to propose Kevin as a core for the tripleo-ansible repo (e.g. >> tripleo-ansible-core). He has made excellent progress centralizing our >> ansible roles and improving the testing around them. >> >> Please reply with your approval/objections. If there are no objections, >> we'll add him to tripleo-ansible-core next Friday Aug 2, 2019. >> >> Thanks, >> -Alex >> > > > -- > Emilien Macchi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sshnaidm at redhat.com Fri Jul 26 22:13:23 2019 From: sshnaidm at redhat.com (Sagi Shnaidman) Date: Sat, 27 Jul 2019 01:13:23 +0300 Subject: [tripleo] Proposing Kevin Carter (cloudnull) for tripleo-ansible core In-Reply-To: References: Message-ID: +2 of course On Sat, Jul 27, 2019 at 12:03 AM Alex Schultz wrote: > Hey folks, > > I'd like to propose Kevin as a core for the tripleo-ansible repo (e.g. > tripleo-ansible-core). He has made excellent progress centralizing our > ansible roles and improving the testing around them. > > Please reply with your approval/objections. If there are no objections, > we'll add him to tripleo-ansible-core next Friday Aug 2, 2019. > > Thanks, > -Alex > -- Best regards Sagi Shnaidman -------------- next part -------------- An HTML attachment was scrubbed... URL: From colleen at gazlene.net Fri Jul 26 22:25:18 2019 From: colleen at gazlene.net (Colleen Murphy) Date: Fri, 26 Jul 2019 15:25:18 -0700 Subject: [keystone] Keystone Team Update - Week of 22 July 2019 Message-ID: <0a6c27d4-3b6e-4831-970d-3419ec1cc522@www.fastmail.com> # Keystone Team Update - Week of 22 July 2019 ## News ### Midcycle held this week We held our first midcycle since the PTGs became a thing, possibly also the team's first virtual midcycle (I'm not sure). It went quite well and it was great to sync up with everyone. Recap posted on the mailing list[1]. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007987.html ## Action Items As discussed during the midcycle retrospective, we'll start calling out action items from the last meeting (or last midcycle, or last PTG, or perhaps any other commitment worth mentioning in a weekly update). * Remove the keystone socket timeout option (IN PROGRESS: https://bugs.launchpad.net/keystone/+bug/1837407) * Seek feedback from operators on caching guide (guide draft IN PROGRESS: https://review.opendev.org/672120) ## Office Hours When there are topics to cover, the keystone team holds office hours on Tuesdays at 17:00 UTC. The topic for next week's office hour will be: bug triage! The location for next week's office hour will be: https://meet.jit.si/keystone-office-hours (fallback to #openstack-keystone if needed) Add topics you would like to see covered during office hours to the etherpad: https://etherpad.openstack.org/p/keystone-office-hours-topics ## Open Specs Ongoing specs: https://bit.ly/2OyDLTh ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 16 changes this week, which included progress made on system scope/default roles[2] and the last of the changes needed for the python3-train goal[3][4]. [2] https://review.opendev.org/670926 [3] https://review.opendev.org/667755 [4] https://review.opendev.org/667756 ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 35 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ### Priority Reviews * Train Roadmap Stories System scope/default roles (https://trello.com/c/ERo50T7r , https://trello.com/c/RlYyb4DU) - https://review.opendev.org/645968 (Implement domain reader support for grants) - https://review.opendev.org/667730 (Implement domain admin support for grants) Application credential access rules (https://trello.com/c/XyBGhKrE) - https://review.opendev.org/668238 (Expose access rules as its own API) - https://review.opendev.org/663440 (Add user_id, external_id to access rules table) Caching Guide (https://trello.com/c/UCFt3mfF) - https://review.opendev.org/672120 (Update the caching guide) Predictable IDs (https://trello.com/c/MVuu6DbU) - https://review.opendev.org/651655 (Predictable IDs for Roles) Federation Improvements (https://trello.com/c/GEaIpbpm) - https://review.opendev.org/637305 (Add new attribute to the federation protocol API) Oslo.limit (https://trello.com/c/KGGkNijR) - https://review.opendev.org/667242 (Add usage example) - https://review.opendev.org/666444 (Flush out basic enforcer and model relationship) - https://review.opendev.org/666085 (Add ksa connection logic) YAML Catalog (https://trello.com/c/Qv14G0xp) - https://review.opendev.org/483514 (Add yaml-loaded filesystem catalog backend) * Needs Discussion - https://review.opendev.org/669959 (discourage using X.509 with external auth) - https://review.opendev.org/655166 (Allows to use application credentials through group membership) * Oldest - https://review.opendev.org/448755 (Add federated support for creating a user) ## Bugs This week we opened 3 new bugs and closed 7 (yay!) Bugs opened (3) Bug #1837407 (keystone:Low) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1837407 Bug #1837513 (keystone:Undecided) opened by ahmed elbendary https://bugs.launchpad.net/keystone/+bug/1837513 Bug #1837741 (oslo.policy:High) opened by Ben Nemec https://bugs.launchpad.net/oslo.policy/+bug/1837741 Bugs closed (1) Bug #1837513 (keystone:Undecided) https://bugs.launchpad.net/keystone/+bug/1837513 Bugs fixed (6) Bug #1750615 (keystone:High) fixed by Guang Yee https://bugs.launchpad.net/keystone/+bug/1750615 Bug #1782922 (keystone:Medium) fixed by Guang Yee https://bugs.launchpad.net/keystone/+bug/1782922 Bug #1818725 (keystone:Medium) fixed by Guang Yee https://bugs.launchpad.net/keystone/+bug/1818725 Bug #1828565 (keystone:Low) fixed by Jose Castro Leon https://bugs.launchpad.net/keystone/+bug/1828565 Bug #1818845 (keystone:Wishlist) fixed by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1818845 Bug #1829802 (keystonemiddleware:Undecided) fixed by norman shen https://bugs.launchpad.net/keystonemiddleware/+bug/1829802 ## Milestone Outlook https://releases.openstack.org/train/schedule.html This week marks the milestone 2 spec freeze. Feature proposal freeze is in three (3!) weeks. This means code implementing approved specs needs to be in a reviewable, non-WIP state. ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter From cboylan at sapwetik.org Fri Jul 26 23:53:28 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Fri, 26 Jul 2019 16:53:28 -0700 Subject: [qa][openstackclient] Debugging devstack slowness Message-ID: <56e637a9-8ef6-4783-98b0-325797b664b9@www.fastmail.com> Today I have been digging into devstack runtime costs to help Donny Davis understand why tempest jobs sometimes timeout on the FortNebula cloud. One thing I discovered was that the keystone user, group, project, role, and domain setup [0] can take many minutes [1][2] (in the examples here almost 5). I've rewritten create_keystone_accounts to be a python tool [3] and get the runtime for that subset of setup from ~100s to ~9s [4]. I imagine that if we applied this to the other create_X_accounts functions we would see similar results. I think this is so much faster because we avoid repeated costs in openstack client including: python process startup, pkg_resource disk scanning to find entrypoints, and needing to convert names to IDs via the API every time osc is run. Given my change shows this can be so much quicker is there any interest in modifying devstack to be faster here? And if so what do we think an appropriate approach would be? [0] https://opendev.org/openstack/devstack/src/commit/6aeaceb0c4ef078d028fb6605cac2a37444097d8/stack.sh#L1146-L1161 [1] http://logs.openstack.org/05/672805/4/check/tempest-full/14f3211/job-output.txt.gz#_2019-07-26_12_31_04_488228 [2] http://logs.openstack.org/05/672805/4/check/tempest-full/14f3211/job-output.txt.gz#_2019-07-26_12_35_53_445059 [3] https://review.opendev.org/#/c/673108/ [4] http://logs.openstack.org/08/673108/6/check/devstack-xenial/a4107d0/job-output.txt.gz#_2019-07-26_23_18_37_211013 Note the jobs compared above all ran on rax-dfw. Clark From rony.khan at brilliant.com.bd Sat Jul 27 11:14:28 2019 From: rony.khan at brilliant.com.bd (Md. Farhad Hasan Khan) Date: Sat, 27 Jul 2019 17:14:28 +0600 Subject: Error: python2-uritemplate conflicts with python2-uri-templates-0.6-5.el7.noarch In-Reply-To: <060001d542ac$8653e1b0$92fba510$@brilliant.com.bd> References: <060001d542ac$8653e1b0$92fba510$@brilliant.com.bd> Message-ID: <02b701d5446c$75b4d640$611e82c0$@brilliant.com.bd> Hi, Please help me how can I fix this. Thanks/Rony From: Md. Farhad Hasan Khan [mailto:rony.khan at brilliant.com.bd] Sent: Thursday, July 25, 2019 11:48 AM To: openstack-discuss at lists.openstack.org Subject: Error: python2-uritemplate conflicts with python2-uri-templates-0.6-5.el7.noarch Hi, When I run # yum update and found below conflict: --> Processing Conflict: python2-uritemplate-3.0.0-1.el7.noarch conflicts python2-uri-templates --> Finished Dependency Resolution Error: python2-uritemplate conflicts with python2-uri-templates-0.6-5.el7.noarch You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles -nodigest If I remove python2-uri-templates-0.6-5.el7.noarch than below package ask to remove: Removing: python2-uri-templates noarch 0.6-5.el7 @centos-openstack-rocky 20 k Removing for dependencies: openstack-cinder noarch 1:13.0.5-1.el7 @centos-openstack-rocky 200 k python-cinder noarch 1:13.0.5-1.el7 @centos-openstack-rocky 19 M python2-google-api-client noarch 1.4.2-4.el7 @centos-openstack-rocky 305 k please let help me how can I update. Thanks/Rony -------------- next part -------------- An HTML attachment was scrubbed... URL: From mthode at mthode.org Sat Jul 27 22:29:01 2019 From: mthode at mthode.org (Matthew Thode) Date: Sat, 27 Jul 2019 17:29:01 -0500 Subject: [requirements] docutils-0.15.1 pypi missing wheel Message-ID: <20190727222901.2yudcjmgfeakytxz@mthode.org> Hi, In trying to update openstack past the broken 0.15 release to the 0.15.1 bugfix release I ran into a problem when trying to install it in python3 (we are currently testing on ppython 3.6). It seems like the missing wheel is the issue. When do you plan on publishing it? The following demonstrates the issue. The logs are not kept forever but it's a standard error. ERROR: No matching distribution found for docutils===0.15.1 http://logs.openstack.org/73/672873/3/check/requirements-tox-py36-check-uc/880ba52/job-output.txt.gz#_2019-07-27_18_06_03_192644 It seems like other devs/users have reported the problem as well but there hasn't been any activity to address it. See the following messages: https://sourceforge.net/p/docutils/mailman/message/36726543/ https://sourceforge.net/p/docutils/mailman/message/36726888/ Please let me know if you need anything else from me. -- Matthew Thode -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From fungi at yuggoth.org Sat Jul 27 22:50:12 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Sat, 27 Jul 2019 22:50:12 +0000 Subject: [requirements] docutils-0.15.1 pypi missing wheel In-Reply-To: <20190727222901.2yudcjmgfeakytxz@mthode.org> References: <20190727222901.2yudcjmgfeakytxz@mthode.org> Message-ID: <20190727225011.ttxb2am3igjmkl4b@yuggoth.org> [not copying docutils ML because I'm not subscribed] On 2019-07-27 17:29:01 -0500 (-0500), Matthew Thode wrote: > In trying to update openstack past the broken 0.15 release to the 0.15.1 > bugfix release I ran into a problem when trying to install it in python3 > (we are currently testing on ppython 3.6). It seems like the missing > wheel is the issue. When do you plan on publishing it? > > ERROR: No matching distribution found for docutils===0.15.1 [...] So, normally this should work. Interestingly, `pip install docutils` in a Python3 venv gets me docutils 0.15.1 via building the docutils-0.15.1-post1.tar.gz sdist, and `pip install docutils===0.15.1.post1` also works fine this way but `pip freeze` reports it as "docutils==0.15.1" and this is how we build our constraints lists. The docutils maintainers have also seen fit to upload a pure Python2-only wheel (docutils-0.15.1-py2-none-any.whl) which of course isn't much use with Python 3.x, but I think your beef is actually with pip which doesn't consider `pip install docutils===0.15.1` to match 0.15.1.post1 even though `pip freeze` reports the result as "docutils===0.15.1". -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From mthode at mthode.org Sat Jul 27 23:22:20 2019 From: mthode at mthode.org (Matthew Thode) Date: Sat, 27 Jul 2019 18:22:20 -0500 Subject: [requirements] docutils-0.15.1 pypi missing wheel In-Reply-To: <20190727225011.ttxb2am3igjmkl4b@yuggoth.org> References: <20190727222901.2yudcjmgfeakytxz@mthode.org> <20190727225011.ttxb2am3igjmkl4b@yuggoth.org> Message-ID: <20190727232220.w7mn76jdk25hjrer@mthode.org> On 19-07-27 22:50:12, Jeremy Stanley wrote: > [not copying docutils ML because I'm not subscribed] > > On 2019-07-27 17:29:01 -0500 (-0500), Matthew Thode wrote: > > In trying to update openstack past the broken 0.15 release to the 0.15.1 > > bugfix release I ran into a problem when trying to install it in python3 > > (we are currently testing on ppython 3.6). It seems like the missing > > wheel is the issue. When do you plan on publishing it? > > > > ERROR: No matching distribution found for docutils===0.15.1 > [...] > > So, normally this should work. Interestingly, `pip install docutils` > in a Python3 venv gets me docutils 0.15.1 via building the > docutils-0.15.1-post1.tar.gz sdist, and `pip install > docutils===0.15.1.post1` also works fine this way but `pip freeze` > reports it as "docutils==0.15.1" and this is how we build our > constraints lists. The docutils maintainers have also seen fit to > upload a pure Python2-only wheel (docutils-0.15.1-py2-none-any.whl) > which of course isn't much use with Python 3.x, but I think your > beef is actually with pip which doesn't consider `pip install > docutils===0.15.1` to match 0.15.1.post1 even though `pip freeze` > reports the result as "docutils===0.15.1". Ya, I suspect the core of the issue is actually a pip issue, guess it's about time to go pull some teeth I think. -- Matthew Thode -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From berndbausch at gmail.com Sun Jul 28 04:06:33 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Sun, 28 Jul 2019 13:06:33 +0900 Subject: [osc] Very slow openstack commands in Stein Devstack running on VM Message-ID: <9488d7de-83d4-c603-70c2-829002776f6a@gmail.com> I run Stein Devstack in a virtual machine. Occasionally, openstack client commands take minutes to complete, which didn't happen at earlier releases, and which is rather annoying. The delay seems to take place at an early stage: Even when I forget to set the OS_* environment, it takes minutes until I am told that the request requires authentication. Interrupting the command, the stack trace confirms this (as a side remark, it also takes a long while until my ^C is processed): Traceback (most recent call last):   File "/usr/local/bin/openstack", line 7, in     from openstackclient.shell import main   File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 22, in     from osc_lib.api import auth   File "/usr/local/lib/python2.7/dist-packages/osc_lib/api/auth.py", line 19, in     from keystoneauth1.loading import base   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/__init__.py", line 13, in     from keystoneauth1.loading import adapter   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/adapter.py", line 13, in     from keystoneauth1 import adapter   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 17, in     from keystoneauth1 import session   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 25, in     import requests   File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 95, in     from urllib3.contrib import pyopenssl   File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 48, in     from cryptography.hazmat.backends.openssl import backend as openssl_backend   File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in     from cryptography.hazmat.backends.openssl.backend import backend   File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 2419, in     backend = Backend()   File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 119, in __init__     self.activate_osrandom_engine()   File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 163, in activate_osrandom_engine     with self._get_osurandom_engine() as e:   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__     return self.gen.next()   File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 146, in _get_osurandom_engine     res = self._lib.ENGINE_init(e) KeyboardInterrupt My wild speculation is that the delay has something to do with obtaining a random number, which can be a problem on virtual machines due to their lack of entropy. My knowledge in this area is paper-thin, though. And I wonder what happens to all the Devstacks used in CI? Thus my questions: *Is the delay really caused by my running the cloud on a VM? What can I do to improve this?* Thanks, Bernd -------------- next part -------------- An HTML attachment was scrubbed... URL: From berndbausch at gmail.com Sun Jul 28 04:37:44 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Sun, 28 Jul 2019 13:37:44 +0900 Subject: [osc] Very slow openstack commands in Stein Devstack running on VM In-Reply-To: <9488d7de-83d4-c603-70c2-829002776f6a@gmail.com> References: <9488d7de-83d4-c603-70c2-829002776f6a@gmail.com> Message-ID: <1570c0c0-95a1-7bde-3a32-c758cebaf578@gmail.com> After a little bit of self-education, I found that /proc/sys/kernel/random/entropy_avail was below 100, which is probably a good explanation for my problem. A good value would have four digits, it seems. My Devstack host runs Ubuntu 18, and I am now trying the rngd-tools5 package to improve the entropy situation. Since this is a test system, I am not so concerned about random number quality. In short, the problem seems to be solved, but if anybody has additional comments, I would be interested to hear them. Bernd On 7/28/2019 1:06 PM, Bernd Bausch wrote: > > I run Stein Devstack in a virtual machine. Occasionally, openstack > client commands take minutes to complete, which didn't happen at > earlier releases, and which is rather annoying. The delay seems to > take place at an early stage: Even when I forget to set the OS_* > environment, it takes minutes until I am told that the request > requires authentication. > > Interrupting the command, the stack trace confirms this (as a side > remark, it also takes a long while until my ^C is processed): > > Traceback (most recent call last): >   File "/usr/local/bin/openstack", line 7, in >     from openstackclient.shell import main >   File > "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", > line 22, in >     from osc_lib.api import auth >   File "/usr/local/lib/python2.7/dist-packages/osc_lib/api/auth.py", > line 19, in >     from keystoneauth1.loading import base >   File > "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/__init__.py", > line 13, in >     from keystoneauth1.loading import adapter >   File > "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/adapter.py", > line 13, in >     from keystoneauth1 import adapter >   File > "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", > line 17, in >     from keystoneauth1 import session >   File > "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", > line 25, in >     import requests >   File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", > line 95, in >     from urllib3.contrib import pyopenssl >   File > "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", > line 48, in >     from cryptography.hazmat.backends.openssl import backend as > openssl_backend >   File > "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", > line 7, in >     from cryptography.hazmat.backends.openssl.backend import backend >   File > "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", > line 2419, in >     backend = Backend() >   File > "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", > line 119, in __init__ >     self.activate_osrandom_engine() >   File > "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", > line 163, in activate_osrandom_engine >     with self._get_osurandom_engine() as e: >   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ >     return self.gen.next() >   File > "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", > line 146, in _get_osurandom_engine >     res = self._lib.ENGINE_init(e) > KeyboardInterrupt > > My wild speculation is that the delay has something to do with > obtaining a random number, which can be a problem on virtual machines > due to their lack of entropy. My knowledge in this area is paper-thin, > though. And I wonder what happens to all the Devstacks used in CI? > > Thus my questions: *Is the delay really caused by my running the cloud > on a VM? What can I do to improve this?* > > Thanks, > > Bernd > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raubvogel at gmail.com Sun Jul 28 11:31:24 2019 From: raubvogel at gmail.com (Mauricio Tavares) Date: Sun, 28 Jul 2019 07:31:24 -0400 Subject: [osc] Very slow openstack commands in Stein Devstack running on VM In-Reply-To: <1570c0c0-95a1-7bde-3a32-c758cebaf578@gmail.com> References: <9488d7de-83d4-c603-70c2-829002776f6a@gmail.com> <1570c0c0-95a1-7bde-3a32-c758cebaf578@gmail.com> Message-ID: On Sun, Jul 28, 2019 at 12:41 AM Bernd Bausch wrote: > > After a little bit of self-education, I found that /proc/sys/kernel/random/entropy_avail was below 100, which is probably a good explanation for my problem. A good value would have four digits, it seems. > > My Devstack host runs Ubuntu 18, and I am now trying the rngd-tools5 package to improve the entropy situation. Since this is a test system, I am not so concerned about random number quality. > > In short, the problem seems to be solved, but if anybody has additional comments, I would be interested to hear them. > I am running packstack (which IMHO for all practical purposes is the same) centos 7 stein in a ESXi vm guest (hope to move it to kvm as it is getting to my nerves). It is certainly not as fast as running on barebones but it is not minutes slow. > Bernd > > On 7/28/2019 1:06 PM, Bernd Bausch wrote: > > I run Stein Devstack in a virtual machine. Occasionally, openstack client commands take minutes to complete, which didn't happen at earlier releases, and which is rather annoying. The delay seems to take place at an early stage: Even when I forget to set the OS_* environment, it takes minutes until I am told that the request requires authentication. > > Interrupting the command, the stack trace confirms this (as a side remark, it also takes a long while until my ^C is processed): > > Traceback (most recent call last): > File "/usr/local/bin/openstack", line 7, in > from openstackclient.shell import main > File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 22, in > from osc_lib.api import auth > File "/usr/local/lib/python2.7/dist-packages/osc_lib/api/auth.py", line 19, in > from keystoneauth1.loading import base > File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/__init__.py", line 13, in > from keystoneauth1.loading import adapter > File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/adapter.py", line 13, in > from keystoneauth1 import adapter > File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 17, in > from keystoneauth1 import session > File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 25, in > import requests > File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 95, in > from urllib3.contrib import pyopenssl > File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 48, in > from cryptography.hazmat.backends.openssl import backend as openssl_backend > File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in > from cryptography.hazmat.backends.openssl.backend import backend > File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 2419, in > backend = Backend() > File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 119, in __init__ > self.activate_osrandom_engine() > File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 163, in activate_osrandom_engine > with self._get_osurandom_engine() as e: > File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ > return self.gen.next() > File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 146, in _get_osurandom_engine > res = self._lib.ENGINE_init(e) > KeyboardInterrupt > > My wild speculation is that the delay has something to do with obtaining a random number, which can be a problem on virtual machines due to their lack of entropy. My knowledge in this area is paper-thin, though. And I wonder what happens to all the Devstacks used in CI? > > Thus my questions: Is the delay really caused by my running the cloud on a VM? What can I do to improve this? > > Thanks, > > Bernd From cboylan at sapwetik.org Sun Jul 28 15:37:24 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Sun, 28 Jul 2019 08:37:24 -0700 Subject: =?UTF-8?Q?Re:_[osc]_Very_slow_openstack_commands_in_Stein_Devstack_runni?= =?UTF-8?Q?ng_on_VM?= In-Reply-To: <9488d7de-83d4-c603-70c2-829002776f6a@gmail.com> References: <9488d7de-83d4-c603-70c2-829002776f6a@gmail.com> Message-ID: On Sat, Jul 27, 2019, at 9:07 PM, Bernd Bausch wrote: snip > My wild speculation is that the delay has something to do with > obtaining a random number, which can be a problem on virtual machines > due to their lack of entropy. My knowledge in this area is paper-thin, > though. And I wonder what happens to all the Devstacks used in CI? All of the test VMs run haveged. We preinstall that on our images: https://opendev.org/openstack/project-config/src/commit/e02cdfa1f489838aa2bb4b449467c0ac4d83645f/nodepool/elements/infra-package-needs/package-installs.yaml#L29 > > Thus my questions: *Is the delay really caused by my running the cloud > on a VM? What can I do to improve this?* > > Thanks, > > Bernd > From rbrady at redhat.com Sun Jul 28 22:26:05 2019 From: rbrady at redhat.com (Ryan Brady) Date: Sun, 28 Jul 2019 18:26:05 -0400 Subject: [tripleo] Roadmap to simplify TripleO In-Reply-To: <1a1620bf-010c-fe37-8b35-7d8b506dbaac@redhat.com> References: <1a1620bf-010c-fe37-8b35-7d8b506dbaac@redhat.com> Message-ID: On Tue, Jul 23, 2019 at 7:49 AM Giulio Fidente wrote: > On 7/11/19 7:52 PM, Emilien Macchi wrote: > > Even though TripleO is well known for its maturity, it also has a > > reputation of being complex when it comes to the number of tools that it > > uses. > > Somewhat related to the efforts that James is leading with "Scaling > > TripleO" [1] [2], I would like to formalize our joint efforts to make > > TripleO simpler in the future. Some work has been done over the last > > releases already and yet we have seen net benefits; however we still > > have challenges ahead of us. > > > > - With no UI anymore, do we really need an API? > > - How can we reduce the number of languages in TripleO? ... and make > > Python + Ansible the main ones. > > - How can we reduce our dependencies? > > > > I created a document which explains the problems and propose some > solutions: > > > > > https://docs.google.com/document/d/1vY9rsccgp7NHFXpLtCFTHQm14F15Tij7lhn5X_P14Ys > > For those who can't or don't want Google Doc, I've put together the > > notes into etherpad [3] and I'll take care of making sure it's updated > > at last at the beginning until we sort things out. > thanks Emilien, > > I added a couple of comments in the doc related to two topics which I > think have been half-baked in past releases and could be reviewed with > the removal of mistral, specifically: > > - TripleO invocation environment files should not be order dependent > - Performing a stack update should not require all original environment > files > > not sure if people familiar with workflows would be interested in > helping with these two? > I'm happy to help. Maybe we can sync up in IRC this week to discuss the environment ordering. That was a feature we added last year after some discussion. > -- > Giulio Fidente > GPG KEY: 08D733BA > > -- Ryan Brady Senior Software Engineer Red Hat 100 East Davie St. Raleigh, NC 27601 rbrady at redhat.com T: 9198908925 IM: rbrady -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjeanner at redhat.com Mon Jul 29 05:42:18 2019 From: cjeanner at redhat.com (=?UTF-8?Q?C=c3=a9dric_Jeanneret?=) Date: Mon, 29 Jul 2019 07:42:18 +0200 Subject: [tripleo] Proposing Kevin Carter (cloudnull) for tripleo-ansible core In-Reply-To: References: Message-ID: +42 of course :) On 7/26/19 11:00 PM, Alex Schultz wrote: > Hey folks, > > I'd like to propose Kevin as a core for the tripleo-ansible repo (e.g. > tripleo-ansible-core). He has made excellent progress centralizing our > ansible roles and improving the testing around them. > > Please reply with your approval/objections. If there are no objections, > we'll add him to tripleo-ansible-core next Friday Aug 2, 2019. > > Thanks, > -Alex -- Cédric Jeanneret (He/Him/His) Software Engineer - OpenStack Platform Red Hat EMEA https://www.redhat.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From marios at redhat.com Mon Jul 29 06:29:31 2019 From: marios at redhat.com (Marios Andreou) Date: Mon, 29 Jul 2019 09:29:31 +0300 Subject: [tripleo] Proposing Kevin Carter (cloudnull) for tripleo-ansible core In-Reply-To: References: Message-ID: +1 one thought - your subject says tripleo-ansible core which of-course makes sense - however bear in mind a lot of what we do in tripleo-heat-templates, tripleo-ci and then all the ansible-role-* ... is ansible. IMO we could say something like tripleo-ansible and the ansible-role-* used by tripleo, and basically anywhere else in tripleo that is ansible. We already rely on our reviewers' good judgement when applying their +2. Just a thought. marios On Sat, Jul 27, 2019 at 12:02 AM Alex Schultz wrote: > Hey folks, > > I'd like to propose Kevin as a core for the tripleo-ansible repo (e.g. > tripleo-ansible-core). He has made excellent progress centralizing our > ansible roles and improving the testing around them. > > Please reply with your approval/objections. If there are no objections, > we'll add him to tripleo-ansible-core next Friday Aug 2, 2019. > > Thanks, > -Alex > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ccamacho at redhat.com Mon Jul 29 08:19:33 2019 From: ccamacho at redhat.com (Carlos Camacho Gonzalez) Date: Mon, 29 Jul 2019 10:19:33 +0200 Subject: [tripleo] Proposing Kevin Carter (cloudnull) for tripleo-ansible core In-Reply-To: References: Message-ID: +1 to Marios comments. Thank you Kevin for all the effort around improving our Ansible support. Cheers, Carlos. On Mon, Jul 29, 2019 at 8:29 AM Marios Andreou wrote: > > +1 > > one thought - your subject says tripleo-ansible core which of-course makes sense - however bear in mind a lot of what we do in tripleo-heat-templates, tripleo-ci and then all the ansible-role-* ... is ansible. > > IMO we could say something like tripleo-ansible and the ansible-role-* used by tripleo, and basically anywhere else in tripleo that is ansible. We already rely on our reviewers' good judgement when applying their +2. Just a thought. > > marios > > On Sat, Jul 27, 2019 at 12:02 AM Alex Schultz wrote: >> >> Hey folks, >> >> I'd like to propose Kevin as a core for the tripleo-ansible repo (e.g. tripleo-ansible-core). He has made excellent progress centralizing our ansible roles and improving the testing around them. >> >> Please reply with your approval/objections. If there are no objections, we'll add him to tripleo-ansible-core next Friday Aug 2, 2019. >> >> Thanks, >> -Alex From skaplons at redhat.com Mon Jul 29 09:05:38 2019 From: skaplons at redhat.com (Slawek Kaplonski) Date: Mon, 29 Jul 2019 11:05:38 +0200 Subject: [goals][IPv6-Only Deployments and Testing] Week R-12 Update In-Reply-To: <16c2db3c410.f4ba6f0a491755.8935052370642166197@ghanshyammann.com> References: <16c2db3c410.f4ba6f0a491755.8935052370642166197@ghanshyammann.com> Message-ID: Hi, Neutron stadium projects which has some additional communication and may use IPv6 for it: - https://opendev.org/openstack/neutron-dynamic-routing - it has agent which communicates with neutron-server via RPC, - https://opendev.org/openstack/networking-bagpipe - it uses some RPC communication with agent, but according to https://opendev.org/openstack/networking-bagpipe/src/branch/master/doc/source/user/design.rst it is just an extension to ovs/linuxbridge agent so in fact this communication is already tested in jobs defined in neutron repo; But there is also communication between e.g. bagpipe backend and other BGP neighbours, - https://docs.openstack.org/networking-bgpvpn/latest/user/overview.html - it is also using RPC communication but it’s with OVS agent, so same as what “standard” neutron is doing; - https://opendev.org/openstack/networking-midonet - this one is communicating with midonet cluster and uses API for that, so it may be done by using IPv6 too. But I don’t know too much about this project so the best guy to ask will be probably Yamamoto :) - https://opendev.org/openstack/networking-odl - similar to midonet plugin, so it is communicating with ODL using REST API, but probably the best guy to ask about it will be Lajos Katona as he is now maintainer of this project, - https://opendev.org/openstack/networking-ovn - this one is communicating with ovsdb server to connect to SB and NB databases, and this can be done using IPv6 too AFAICT, Other stadium projects are providing some extensions/drivers/plugins which integrates with neutron-server and/or existing neutron agents and are only extending existing communication channels used by default neutron resources so I don’t think we need to add any special jobs for them. > On 26 Jul 2019, at 11:54, Ghanshyam Mann wrote: > > Hello Everyone, > > I am starting this status email for 'IPv6-Only Deployments and Testing ' community goal for Train. > > Storyboard: > ========= > - https://storyboard.openstack.org/#!/story/2005477 > > Current status: > ============ > 1. Base job 'devstack-tempest-ipv6' has been prepared on Tempest (patch yet to merge)[1] > 2. 'tempest-ipv6-only' is defined in Tempest (patch yet to merge)[1] > 3. 'tempest-ipv6-only' has been proposed to run on 5 services project date side [2]. > 4. I have proposed few more projects jobs like congress etc and will keep doing for other projects also. > 5. Neutron stadium projects jobs. I talked to slaweq on IRC [3]on what all neutron stadium falls under the IPv6 address > communication criteria so that we can add ipv6-only job for those project only. He is going to prepare the list of those > project and accordingly we will add jobs > > Everything related to this goal can be found under this topic: > Topic: https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(status:open+OR+status:merged) > > How to define and run new IPv6 Job on project side: > ======================================= > - I prepared a wiki page to describe this section - https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing > - I am adding this wiki info in goal doc also[4]. > > Review suggestion: > ============== > - Main goal of these jobs will be whether your service is able to listen on IPv6 and can communicate to any > other services either OpenStack or DB or rabbitmq etc on IPv6 or not. So check your proposed job with > that point of view. If anything missing, comment on patch. > - One example was - I missed to configure novnc address to IPv6- https://review.opendev.org/#/c/672493/ > - base script as part of 'devstack-tempest-ipv6' will do basic checks for endpoints on IPv6 and some devstack var > setting. But if your project needs more specific varification then it can be added in project side job as post-run > playbooks as described in wiki page[5]. > > [1] https://review.opendev.org/#/c/671231/ > [2] https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(status:open+OR+status:merged) > [3] http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2019-07-26.log.html#t2019-07-26T08:56:20 > [4] https://review.opendev.org/#/c/671898/ > [5] https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing > > -gmann > > — Slawek Kaplonski Senior software engineer Red Hat From gmann at ghanshyammann.com Mon Jul 29 10:50:03 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Mon, 29 Jul 2019 19:50:03 +0900 Subject: [goals][IPv6-Only Deployments and Testing] Week R-12 Update In-Reply-To: References: <16c2db3c410.f4ba6f0a491755.8935052370642166197@ghanshyammann.com> Message-ID: <16c3d59fce4.e39e2a84538206.4186945823064957638@ghanshyammann.com> ---- On Mon, 29 Jul 2019 18:05:38 +0900 Slawek Kaplonski wrote ---- > Hi, > > Neutron stadium projects which has some additional communication and may use IPv6 for it: > > - https://opendev.org/openstack/neutron-dynamic-routing - it has agent which communicates with neutron-server via RPC, > - https://opendev.org/openstack/networking-bagpipe - it uses some RPC communication with agent, but according to https://opendev.org/openstack/networking-bagpipe/src/branch/master/doc/source/user/design.rst it is just an extension to ovs/linuxbridge agent so in fact this communication is already tested in jobs defined in neutron repo; But there is also communication between e.g. bagpipe backend and other BGP neighbours, > - https://docs.openstack.org/networking-bgpvpn/latest/user/overview.html - it is also using RPC communication but it’s with OVS agent, so same as what “standard” neutron is doing; > - https://opendev.org/openstack/networking-midonet - this one is communicating with midonet cluster and uses API for that, so it may be done by using IPv6 too. But I don’t know too much about this project so the best guy to ask will be probably Yamamoto :) > - https://opendev.org/openstack/networking-odl - similar to midonet plugin, so it is communicating with ODL using REST API, but probably the best guy to ask about it will be Lajos Katona as he is now maintainer of this project, > - https://opendev.org/openstack/networking-ovn - this one is communicating with ovsdb server to connect to SB and NB databases, and this can be done using IPv6 too AFAICT, > > Other stadium projects are providing some extensions/drivers/plugins which integrates with neutron-server and/or existing neutron agents and are only extending existing communication channels used by default neutron resources so I don’t think we need to add any special jobs for them. Thanks slaweq for detailed information. This is helpful for this goal. I will prepare the jobs for listed project only. -gmann > > > > On 26 Jul 2019, at 11:54, Ghanshyam Mann wrote: > > > > Hello Everyone, > > > > I am starting this status email for 'IPv6-Only Deployments and Testing ' community goal for Train. > > > > Storyboard: > > ========= > > - https://storyboard.openstack.org/#!/story/2005477 > > > > Current status: > > ============ > > 1. Base job 'devstack-tempest-ipv6' has been prepared on Tempest (patch yet to merge)[1] > > 2. 'tempest-ipv6-only' is defined in Tempest (patch yet to merge)[1] > > 3. 'tempest-ipv6-only' has been proposed to run on 5 services project date side [2]. > > 4. I have proposed few more projects jobs like congress etc and will keep doing for other projects also. > > 5. Neutron stadium projects jobs. I talked to slaweq on IRC [3]on what all neutron stadium falls under the IPv6 address > > communication criteria so that we can add ipv6-only job for those project only. He is going to prepare the list of those > > project and accordingly we will add jobs > > > > Everything related to this goal can be found under this topic: > > Topic: https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(status:open+OR+status:merged) > > > > How to define and run new IPv6 Job on project side: > > ======================================= > > - I prepared a wiki page to describe this section - https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing > > - I am adding this wiki info in goal doc also[4]. > > > > Review suggestion: > > ============== > > - Main goal of these jobs will be whether your service is able to listen on IPv6 and can communicate to any > > other services either OpenStack or DB or rabbitmq etc on IPv6 or not. So check your proposed job with > > that point of view. If anything missing, comment on patch. > > - One example was - I missed to configure novnc address to IPv6- https://review.opendev.org/#/c/672493/ > > - base script as part of 'devstack-tempest-ipv6' will do basic checks for endpoints on IPv6 and some devstack var > > setting. But if your project needs more specific varification then it can be added in project side job as post-run > > playbooks as described in wiki page[5]. > > > > [1] https://review.opendev.org/#/c/671231/ > > [2] https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(status:open+OR+status:merged) > > [3] http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2019-07-26.log.html#t2019-07-26T08:56:20 > > [4] https://review.opendev.org/#/c/671898/ > > [5] https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing > > > > -gmann > > > > > > — > Slawek Kaplonski > Senior software engineer > Red Hat > > > From smooney at redhat.com Mon Jul 29 10:59:06 2019 From: smooney at redhat.com (Sean Mooney) Date: Mon, 29 Jul 2019 11:59:06 +0100 Subject: [osc] Very slow openstack commands in Stein Devstack running on VM In-Reply-To: <1570c0c0-95a1-7bde-3a32-c758cebaf578@gmail.com> References: <9488d7de-83d4-c603-70c2-829002776f6a@gmail.com> <1570c0c0-95a1-7bde-3a32-c758cebaf578@gmail.com> Message-ID: <4f064929b55bd2313cf647d879140fa93b673644.camel@redhat.com> On Sun, 2019-07-28 at 13:37 +0900, Bernd Bausch wrote: > After a little bit of self-education, I found that > /proc/sys/kernel/random/entropy_avail was below 100, which is probably a > good explanation for my problem. A good value would have four digits, it > seems. > > My Devstack host runs Ubuntu 18, and I am now trying the rngd-tools5 > package to improve the entropy situation. Since this is a test system, I > am not so concerned about random number quality. > > In short, the problem seems to be solved, but if anybody has additional > comments, I would be interested to hear them. am i have not encountered this on my own ububtu 18.04 devstack vms but i tend to enable the RNG device whcich effectivly presnets the host /dev/urandom to the guests so perhaps that helps. if you are virtulaising with kvm/libvit i would suggest adding the RNG device and seeing if that has a similar effect. we had also condidered change openstack default at one point to make it so that RNG devices would be added by default in the futre rather im not sure if we ever did anything in that regard but if it can have that much of an impact on things like osc it proably would be a performace imporvment for many applicaitons that use openssl and need entropy. > > Bernd > > On 7/28/2019 1:06 PM, Bernd Bausch wrote: > > > > I run Stein Devstack in a virtual machine. Occasionally, openstack > > client commands take minutes to complete, which didn't happen at > > earlier releases, and which is rather annoying. The delay seems to > > take place at an early stage: Even when I forget to set the OS_* > > environment, it takes minutes until I am told that the request > > requires authentication. > > > > Interrupting the command, the stack trace confirms this (as a side > > remark, it also takes a long while until my ^C is processed): > > > > Traceback (most recent call last): > > File "/usr/local/bin/openstack", line 7, in > > from openstackclient.shell import main > > File > > "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", > > line 22, in > > from osc_lib.api import auth > > File "/usr/local/lib/python2.7/dist-packages/osc_lib/api/auth.py", > > line 19, in > > from keystoneauth1.loading import base > > File > > "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/__init__.py", > > line 13, in > > from keystoneauth1.loading import adapter > > File > > "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/adapter.py", > > line 13, in > > from keystoneauth1 import adapter > > File > > "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", > > line 17, in > > from keystoneauth1 import session > > File > > "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", > > line 25, in > > import requests > > File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", > > line 95, in > > from urllib3.contrib import pyopenssl > > File > > "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", > > line 48, in > > from cryptography.hazmat.backends.openssl import backend as > > openssl_backend > > File > > "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", > > line 7, in > > from cryptography.hazmat.backends.openssl.backend import backend > > File > > "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", > > line 2419, in > > backend = Backend() > > File > > "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", > > line 119, in __init__ > > self.activate_osrandom_engine() > > File > > "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", > > line 163, in activate_osrandom_engine > > with self._get_osurandom_engine() as e: > > File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ > > return self.gen.next() > > File > > "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", > > line 146, in _get_osurandom_engine > > res = self._lib.ENGINE_init(e) > > KeyboardInterrupt > > > > My wild speculation is that the delay has something to do with > > obtaining a random number, which can be a problem on virtual machines > > due to their lack of entropy. My knowledge in this area is paper-thin, > > though. And I wonder what happens to all the Devstacks used in CI? > > > > Thus my questions: *Is the delay really caused by my running the cloud > > on a VM? What can I do to improve this?* > > > > Thanks, > > > > Bernd > > From davidhqwang at gmail.com Mon Jul 29 11:11:50 2019 From: davidhqwang at gmail.com (David Wang) Date: Mon, 29 Jul 2019 19:11:50 +0800 Subject: [nova][sfe] Use PCPU and VCPU in One Instance Message-ID: Hi, We are proposing a Nova spec for the feature of creating an instance of mixed dedicated CPUs and shared CPUs and experience communities review. The spec link is https://review.opendev.org/#/c/668656/ and spec name is "Use PCPU and VCPU in One Instance". I recognized that Nova Sepc for Train release is frozen, I'd like to request a spec freeze exception process and let the spec to be merged in train release. I may have trouble in attending next week's team meeting, on Aug 1, so, please discuss in the email thread. Thanks Huaqiang -------------- next part -------------- An HTML attachment was scrubbed... URL: From juliaashleykreger at gmail.com Mon Jul 29 11:56:20 2019 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Mon, 29 Jul 2019 07:56:20 -0400 Subject: [ironic] Moving to office hours as opposed to weekly meetings for the next month In-Reply-To: References: Message-ID: Hi Jacob, Sorry for the delay. My hope was that APAC contributors would coalesce around a time, but it really seems that has not happened, and I am starting to think that the office hours experiment has not really helped as there has not been a regular reminder each week. :( Happy to discuss more, but perhaps a establishing a dedicated APAC sync-up meeting is what is required? Thoughts? -Julia On Wed, Jul 17, 2019 at 5:44 AM Jacob Anders wrote: > > Hi Julia, > > Do we have more clarity regarding the second (APAC) session? I see the polls have been open for some time, but haven't seen a mention of a specific time. > > Thank you, > Jacob > > On Wed, Jul 3, 2019 at 9:39 AM Julia Kreger wrote: >> >> Greetings Everyone! >> >> This week, during the weekly meeting, we seemed to reach consensus that >> we would try taking a break from meetings[0] and moving to orienting >> around using the mailing list[1] and our etherpad "whiteboard" [2]. >> With this, we're going to want to re-evaluate in about a month. >> I suspect it would be a good time for us to have a "mid-cycle" style >> set of topical calls. I've gone ahead and created a poll to try and >> identify a couple days that might be ideal for contributors[3]. >> >> But in the mean time, we want to ensure that we have some times for >> office hours. The suggestion was also made during this week's meeting >> that we may want to make the office hours window a little larger to >> enable more discussion. >> >> So when will we have office hours? >> ---------------------------------- >> >> Ideally we'll start with two time windows. One to provide coverage >> to US and Europe friendly time zones, and another for APAC contributors. >> >> * I think 2-4 PM UTC on Mondays would be ideal. This translates to >> 7-9 AM US-Pacific or 10 AM to 12 PM US-Eastern. >> * We need to determine a time window that would be ideal for APAC >> contributors. I've created a poll to help facilitate discussion[4]. >> >> So what is Office Hours? >> ------------------------ >> >> Office hours are a time window when we expect some contributors to be >> on IRC and able to partake in higher bandwidth discussions. >> These times are not absolute. They can change and evolve, >> and that is the most important thing for us to keep in mind. >> >> -- >> >> If there are any questions, Please let me know! >> Otherwise I'll send a summary email out on next Monday. >> >> -Julia >> >> [0]: http://eavesdrop.openstack.org/meetings/ironic/2019/ironic.2019-07-01-15.00.log.html#l-123 >> [1]: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007038.html >> [2]: https://etherpad.openstack.org/p/IronicWhiteBoard >> [3]: https://doodle.com/poll/652gzta6svsda343 >> [4]: https://doodle.com/poll/2ta5vbskytpntmgv >> From bodenvmw at gmail.com Mon Jul 29 12:27:49 2019 From: bodenvmw at gmail.com (Boden Russell) Date: Mon, 29 Jul 2019 06:27:49 -0600 Subject: [neutron] Bug Deputy July 22 - July 29 Message-ID: Below is a list of neutron bugs that came in last week. To the best of my knowledge, the only bug still under discussion is #2 below (and the RFE in #3). Thanks to everyone on the team for their bug diligence. It looks like Brian H. is on deck for bug duty this week. 1) https://bugs.launchpad.net/neutron/+bug/1838068 "QoSTest:test_qos_basic_and_update" failing in DVR node scenario 2) https://bugs.launchpad.net/neutron/+bug/1837916 Trunk subports sometimes not becoming ACTIVE and trunk:subport 3) https://bugs.launchpad.net/neutron/+bug/1837847 [RFE] neutron-vpnaas OpenVPN driver 4) https://bugs.launchpad.net/neutron/+bug/1837635 HA router state change from "standby" to "master" should be delayed 5) https://bugs.launchpad.net/neutron/+bug/1837553 Neutron api-ref should mention that list of e.g. IDs is supported in GET requests 6) https://bugs.launchpad.net/neutron/+bug/1837552 neutron-tempest-with-uwsgi job finish with timeout very often 7) https://bugs.launchpad.net/neutron/+bug/1837529 Cannot use push-notification with custom objects From gfidente at redhat.com Mon Jul 29 12:29:47 2019 From: gfidente at redhat.com (Giulio Fidente) Date: Mon, 29 Jul 2019 14:29:47 +0200 Subject: [tripleo] Proposing Kevin Carter (cloudnull) for tripleo-ansible core In-Reply-To: References: Message-ID: <486e840c-c07b-861d-ea6d-5dd475dd0f6c@redhat.com> On 7/26/19 11:00 PM, Alex Schultz wrote: > Hey folks, > > I'd like to propose Kevin as a core for the tripleo-ansible repo (e.g. > tripleo-ansible-core). He has made excellent progress centralizing our > ansible roles and improving the testing around them. > > Please reply with your approval/objections. If there are no objections, > we'll add him to tripleo-ansible-core next Friday Aug 2, 2019. thanks Kevin for igniting a long waited transformation +1 -- Giulio Fidente GPG KEY: 08D733BA From li.canwei2 at zte.com.cn Mon Jul 29 12:33:34 2019 From: li.canwei2 at zte.com.cn (li.canwei2 at zte.com.cn) Date: Mon, 29 Jul 2019 20:33:34 +0800 (CST) Subject: =?UTF-8?B?W1dhdGNoZXJdIE5vIElSQyBtZWV0aW5nIHRoaXMgd2Vlaw==?= Message-ID: <201907292033340740631@zte.com.cn> Hi team, I'll be on vacation until to August 5, so I can't host meeting this week. Thanks, licanwei -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnaser at vexxhost.com Mon Jul 29 12:41:13 2019 From: mnaser at vexxhost.com (Mohammed Naser) Date: Mon, 29 Jul 2019 08:41:13 -0400 Subject: [tc] weekly update Message-ID: Hi everyone, This is the weekly update for what happened inside the Openstack TC last week. You can check for more information in the openstack/governance repository. We had one change only which was the retirement of puppet-crane (from TripleO). Thanks, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser at vexxhost.com W. http://vexxhost.com From juliaashleykreger at gmail.com Mon Jul 29 13:03:43 2019 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Mon, 29 Jul 2019 09:03:43 -0400 Subject: [ironic][edge] L3/DHCP-Less deployments In-Reply-To: References: Message-ID: Circling back to this entire discussion. Ilya Etingof is starting to propose a revision[0] to the specification that was originally posted in ironic-specs. Given the amount of interest, I think it would be good for this to have higher visibility so interested parties can be involved. While I think it would still be good to have a multiple party discussion to try and help clarify wants/desires and hopefully figure out next steps, I suspect most of that discussion will take place on the specification revision. -Julia [0]: https://review.opendev.org/#/c/672780/ On Wed, Jul 10, 2019 at 2:39 AM chandra sekhar wrote: > > Hi Julia, > I am based in Finland . Anytime from 8:00 to 19:00 (EEST) is good for me. > > Regards > Chandra R. > > > On Tue, 9 Jul 2019, 18:33 Julia Kreger, wrote: >> >> Greetings everyone! >> >> Over the last few weeks, I've had numerous discussions with >> contributors who have expressed interest in the DHCP-less deployment >> method specification document[0]. >> >> It seems many parties are interested! I think the path forward at this >> time is to get everyone talking about where and how they can help push >> this feature forward. >> >> If those interested could propose time windows when they are >> available, I'll try to find a mutual time window where we can try to >> get everyone on IRC or on a call and try to map out the next steps >> together. >> >> Sound good? >> >> -Julia >> >> [0]: https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/L3-based-deployment.html >> From dangtrinhnt at gmail.com Mon Jul 29 13:04:06 2019 From: dangtrinhnt at gmail.com (Trinh Nguyen) Date: Mon, 29 Jul 2019 22:04:06 +0900 Subject: [searchlight] Meeting today cancelled Message-ID: Hi team, I cannot hold the meeting today so please email or ping me on #openstack-searchlight channel. Bests, -- *Trinh Nguyen* *www.edlab.xyz * -------------- next part -------------- An HTML attachment was scrubbed... URL: From aschultz at redhat.com Mon Jul 29 13:40:11 2019 From: aschultz at redhat.com (Alex Schultz) Date: Mon, 29 Jul 2019 07:40:11 -0600 Subject: [qa][openstackclient] Debugging devstack slowness In-Reply-To: <56e637a9-8ef6-4783-98b0-325797b664b9@www.fastmail.com> References: <56e637a9-8ef6-4783-98b0-325797b664b9@www.fastmail.com> Message-ID: On Fri, Jul 26, 2019 at 5:57 PM Clark Boylan wrote: > Today I have been digging into devstack runtime costs to help Donny Davis > understand why tempest jobs sometimes timeout on the FortNebula cloud. One > thing I discovered was that the keystone user, group, project, role, and > domain setup [0] can take many minutes [1][2] (in the examples here almost > 5). > > I've rewritten create_keystone_accounts to be a python tool [3] and get > the runtime for that subset of setup from ~100s to ~9s [4]. I imagine that > if we applied this to the other create_X_accounts functions we would see > similar results. > > I think this is so much faster because we avoid repeated costs in > openstack client including: python process startup, pkg_resource disk > scanning to find entrypoints, and needing to convert names to IDs via the > API every time osc is run. Given my change shows this can be so much > quicker is there any interest in modifying devstack to be faster here? And > if so what do we think an appropriate approach would be? > > In tripleo, we've also run into the same thing for other actions. While you can do bulk openstack client actions[0], it's not the best thing if you need to create a resource and fetch an ID for a subsequent action. We ported our post-installation items to python[1] and noticed a dramatic improvement as well. It might be beneficial to maybe add some caching into openstackclient so that the startup cost isn't so large every time? [0] https://review.opendev.org/#/c/521146/ [1] https://review.opendev.org/#/c/614540/ > [0] > https://opendev.org/openstack/devstack/src/commit/6aeaceb0c4ef078d028fb6605cac2a37444097d8/stack.sh#L1146-L1161 > [1] > http://logs.openstack.org/05/672805/4/check/tempest-full/14f3211/job-output.txt.gz#_2019-07-26_12_31_04_488228 > [2] > http://logs.openstack.org/05/672805/4/check/tempest-full/14f3211/job-output.txt.gz#_2019-07-26_12_35_53_445059 > [3] https://review.opendev.org/#/c/673108/ > [4] > http://logs.openstack.org/08/673108/6/check/devstack-xenial/a4107d0/job-output.txt.gz#_2019-07-26_23_18_37_211013 > > Note the jobs compared above all ran on rax-dfw. > > Clark > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephine.seifert at secustack.com Mon Jul 29 13:48:10 2019 From: josephine.seifert at secustack.com (Josephine Seifert) Date: Mon, 29 Jul 2019 15:48:10 +0200 Subject: [nova][sfe] Nova part of Image Encryption Message-ID: <7a07b1a6-a076-d2dc-96fa-4d87ab370dd2@secustack.com> Hi, we would like to have a spec freeze exception for the spec for the Nova Part of Image Encryption [1] as this is part of a cross-project contribution with specs also in cinder (already merged) and glance. We will attend the next meeting on August 1st. Greetings Josephine (Luzi) [1] https://review.opendev.org/#/c/608696 From gr at ham.ie Mon Jul 29 14:11:20 2019 From: gr at ham.ie (Graham Hayes) Date: Mon, 29 Jul 2019 15:11:20 +0100 Subject: [requirements] changes in requests library maintainership In-Reply-To: References: Message-ID: <50d81290-2f8c-fc25-a88a-17589e4a2248@ham.ie> On 24/07/2019 14:31, Doug Hellmann wrote: > > It looks like the Kenneth Reitz, the primary maintainer of the requests > library, is stepping down [1]. There are several offers to take over > right now, so it's not really clear to me what's going to happen. I > think we'll want to keep an eye on both requests and requests3 until the > new maintainer is designated. And of course if anyone is interested in > helping out, I encourage you to get involved in the discussion on > github. > > [1] https://github.com/not-kennethreitz/team/issues/21 > Looks like it ended up in the PSF [2] - which seems like the best outcome. 2 - https://github.com/psf/requests -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From doug at doughellmann.com Mon Jul 29 14:18:42 2019 From: doug at doughellmann.com (Doug Hellmann) Date: Mon, 29 Jul 2019 10:18:42 -0400 Subject: [tc][all] seeking candidates to run in the next TC election Message-ID: Nominations for the next Technical Committee term open on 27 August, just over 4 weeks from now. In the past, we have had people wait until the last minute to consider running, and then in some cases they had issues getting approval from their managers to serve in the role. This time, since I know there will be at least one open seat when I step down, I thought it would be a good idea to start that recruiting process a little earlier. Our community places a high value on democratically electing representative leadership — selecting members of the community to help steer the project. That system only works if we have people willing and able to serve in the elected roles like the Technical Committee. There is no fall-back option. We need community members to step up and run. So, if you are interested in serving on the TC, now is the time ask the necessary questions within your company. If you need advice about how to approach your manager, any current TC member, including me, would be happy to help you. Doug From ykarel at redhat.com Mon Jul 29 14:29:44 2019 From: ykarel at redhat.com (Yatin Karel) Date: Mon, 29 Jul 2019 19:59:44 +0530 Subject: Error: python2-uritemplate conflicts with python2-uri-templates-0.6-5.el7.noarch In-Reply-To: <02b701d5446c$75b4d640$611e82c0$@brilliant.com.bd> References: <060001d542ac$8653e1b0$92fba510$@brilliant.com.bd> <02b701d5446c$75b4d640$611e82c0$@brilliant.com.bd> Message-ID: Hi Rony, See inline reply. On Sat, Jul 27, 2019 at 4:48 PM Md. Farhad Hasan Khan wrote: > > Hi, > > Please help me how can I fix this. > > > > Thanks/Rony > > > > > > From: Md. Farhad Hasan Khan [mailto:rony.khan at brilliant.com.bd] > Sent: Thursday, July 25, 2019 11:48 AM > To: openstack-discuss at lists.openstack.org > Subject: Error: python2-uritemplate conflicts with python2-uri-templates-0.6-5.el7.noarch > > > > Hi, > > When I run # yum update and found below conflict: > > > > --> Processing Conflict: python2-uritemplate-3.0.0-1.el7.noarch conflicts python2-uri-templates > > --> Finished Dependency Resolution > > Error: python2-uritemplate conflicts with python2-uri-templates-0.6-5.el7.noarch > > You could try using --skip-broken to work around the problem > > You could try running: rpm -Va --nofiles –nodigest > > > > If I remove python2-uri-templates-0.6-5.el7.noarch than below package ask to remove: > > > > Removing: > > python2-uri-templates noarch 0.6-5.el7 @centos-openstack-rocky 20 k > > Removing for dependencies: > > openstack-cinder noarch 1:13.0.5-1.el7 @centos-openstack-rocky 200 k > > python-cinder noarch 1:13.0.5-1.el7 @centos-openstack-rocky 19 M > > python2-google-api-client noarch 1.4.2-4.el7 @centos-openstack-rocky 305 k > > > > > > please let help me how can I update. > > >From logs(python2-uritemplate-3.0.0-1.el7.noarch) it appears you have enabled "epel" repository, you can confirm it with yum repolist command. Mixing epel with OpenStack repos can have such issues. For now to move ahead i would suggest you disable epel repository before running "yum update". This should avoid pulling of package python2-uritemplate and thus conflict with python2-uri-templates) > > Thanks/Rony > > Thanks and Regards Yatin Karel From cboylan at sapwetik.org Mon Jul 29 15:51:44 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 29 Jul 2019 08:51:44 -0700 Subject: [qa][openstackclient] Debugging devstack slowness In-Reply-To: References: <56e637a9-8ef6-4783-98b0-325797b664b9@www.fastmail.com> Message-ID: On Mon, Jul 29, 2019, at 6:41 AM, Alex Schultz wrote: > > > On Fri, Jul 26, 2019 at 5:57 PM Clark Boylan wrote: > > Today I have been digging into devstack runtime costs to help Donny Davis understand why tempest jobs sometimes timeout on the FortNebula cloud. One thing I discovered was that the keystone user, group, project, role, and domain setup [0] can take many minutes [1][2] (in the examples here almost 5). > > > > I've rewritten create_keystone_accounts to be a python tool [3] and get the runtime for that subset of setup from ~100s to ~9s [4]. I imagine that if we applied this to the other create_X_accounts functions we would see similar results. > > > > I think this is so much faster because we avoid repeated costs in openstack client including: python process startup, pkg_resource disk scanning to find entrypoints, and needing to convert names to IDs via the API every time osc is run. Given my change shows this can be so much quicker is there any interest in modifying devstack to be faster here? And if so what do we think an appropriate approach would be? > > > > In tripleo, we've also run into the same thing for other actions. While > you can do bulk openstack client actions[0], it's not the best thing if > you need to create a resource and fetch an ID for a subsequent action. > We ported our post-installation items to python[1] and noticed a > dramatic improvement as well. It might be beneficial to maybe add some > caching into openstackclient so that the startup cost isn't so large > every time? Reading more of what devstack does I've realized that there is quite a bit of logic tied around devstack's use of OSC. In particular if you select one option you get this endpoint and if you select another option you get that endpoint, or if this service and that service are enabled then they need this common role, etc. I think the best way to tackle this would be to have devstack write a manifest file, then have a tool (maybe in osc or sdk?) that can read a manifest and execute the api updates in order, storing intermediate results so that they can be referred to without doing further API lookups. Sounds like such a thing would be useful outside of devstack as well. I brought this up briefly with Monty and he said he would explore it a bit on the SDK side of things. Does this seem like a reasonable approach? Anyone else have better ideas? The big key here seems to be reusing authentication tokens and remembering resource ID data so that we can avoid unnecessary (and costly) lookups every time we want to modify a resource or associate resources. > > [0] https://review.opendev.org/#/c/521146/ > [1] https://review.opendev.org/#/c/614540/ > > [0] https://opendev.org/openstack/devstack/src/commit/6aeaceb0c4ef078d028fb6605cac2a37444097d8/stack.sh#L1146-L1161 > > [1] http://logs.openstack.org/05/672805/4/check/tempest-full/14f3211/job-output.txt.gz#_2019-07-26_12_31_04_488228 > > [2] http://logs.openstack.org/05/672805/4/check/tempest-full/14f3211/job-output.txt.gz#_2019-07-26_12_35_53_445059 > > [3] https://review.opendev.org/#/c/673108/ > > [4] http://logs.openstack.org/08/673108/6/check/devstack-xenial/a4107d0/job-output.txt.gz#_2019-07-26_23_18_37_211013 > > > > Note the jobs compared above all ran on rax-dfw. > > > > Clark > > From massimo.sgaravatto at gmail.com Mon Jul 29 15:52:57 2019 From: massimo.sgaravatto at gmail.com (Massimo Sgaravatto) Date: Mon, 29 Jul 2019 17:52:57 +0200 Subject: [ops][glance][nova] scheduling problem because of ImagePropertiesFilter In-Reply-To: <9390f785-9a57-37c0-51a3-4ffa02520a3f@gmail.com> References: <1373daf4-d073-a432-f1ee-36c50480f5bb@gmail.com> <9536c249-b87c-9890-b899-75b1790ad67f@gmail.com> <14807dc9-9530-16ed-822e-d8e11b0875ad@gmail.com> <0fe1409e-d96e-2bc5-c6b3-0baeeef0959a@gmail.com> <2d8418f6-67c4-41db-e5da-1c650e5a9793@gmail.com> <9390f785-9a57-37c0-51a3-4ffa02520a3f@gmail.com> Message-ID: > > Here is the revert: https://review.opendev.org/#/c/672559/ > > We tried to apply this patch on some compute nodes of our cloud and it works (both scheduling of new instances and migration of previously created VMs) Thanks a lot for your help ! -------------- next part -------------- An HTML attachment was scrubbed... URL: From amotoki at gmail.com Mon Jul 29 16:25:14 2019 From: amotoki at gmail.com (Akihiro Motoki) Date: Tue, 30 Jul 2019 01:25:14 +0900 Subject: [OpenStack-I18n] [I18n] project team status update regarding internationalization/translation as July 2019 In-Reply-To: <6058a029-9922-7c4a-9f95-a7348f3a7c44@gmail.com> References: <6058a029-9922-7c4a-9f95-a7348f3a7c44@gmail.com> Message-ID: Hi Ian, Thanks for sharing the current status. > 3. Updating ATC & AUC > : Frank previously submitted extra-ATC for Stein through > https://review.opendev.org/#/c/633398/ , and this time I would like to > submit the stat calculation from 2019-01-26 to 2019-07-25. > : All contributors who had commits on openstack/i18n repository will be > automatically regarded as Active Technical Contributors [1], and > I18n team would like to nominate all translators who contributed > translations on OpenStack official repositories > (https://governance.openstack.org/tc/reference/projects/ ) >= 300 words > as extra-ATCs assuming that their Zanata e-mail IDs are registered as > Foundation members. > : All translators who contributed >=300 word tranlations to the > following projects will be nominated as Active User Contributors [2] > assuming that their Zanata e-mail IDs are registered as Foundation > members: "contributor-guide, edge-computing, > leveraging-containers-openstack, openstack-user-survey, operations-guide". Does this mean if someone translates >=300 words but translates <300 words in ATC targeted projects and <300 words in AUC targeted projects they are not recognized as either of ATC and AUC? The i18n doc [1] just mentions translators who translate >=300 words will be regarded as ATC. From the i18n guide perspective, the above statement in Ian's mail looks undocumented. It sounds better to be consistent (though I am not sure we have such translators.) [1] https://docs.openstack.org/i18n/latest/official_translator.html#atc-status-in-i18n-project Find my question inline below. Thanks, Akihiro On Wed, Jul 24, 2019 at 8:48 AM Ian Y. Choi wrote: > > Hello all, > > I would like to share current project team status for better visibility > into I18n team members, copying to openstack-discuss and user-committee > thread: > > 1. Translation and team effort status > There are still lots of translation contribution as you see through > Stackalytics: > https://www.stackalytics.com/?metric=translations&release=train . > > The number of translated words is not larger than the previous cycle, > but I estimate that it will be similar or a little bit less than the > previous cycle since I18n usually contributes translations a lot especially > during Soft & Hard StringFreeze periods. > Note that I18n team currently sets higher priority on OpenStack user > survey translation and project document translations. > Such priorities information can be found on Zanata dashboard: > https://translate.openstack.org/ . > Other progress such as migrating Zanata into other translation platform > seems slow - me and Frank are planning to have a virtual sprint. > > 2. Shanghai Summit & PTG preparation > : Frank submitted his presentation including me and Hoarce (China team). > Let's wait for final approval on presentation submission. > : There should be PTG at Shanghai - please add your name and detail > thinking to Etherpad mentioned in > http://lists.openstack.org/pipermail/openstack-i18n/2019-July/003436.html . > I strongly think to more encourage participation since I expect more > participation especially from Chinese translators & contributors. > > 3. Updating ATC & AUC > : Frank previously submitted extra-ATC for Stein through > https://review.opendev.org/#/c/633398/ , and this time I would like to > submit the stat calculation from 2019-01-26 to 2019-07-25. > : All contributors who had commits on openstack/i18n repository will be > automatically regarded as Active Technical Contributors [1], and > I18n team would like to nominate all translators who contributed > translations on OpenStack official repositories > (https://governance.openstack.org/tc/reference/projects/ ) >= 300 words > as extra-ATCs assuming that their Zanata e-mail IDs are registered as > Foundation members. > : All translators who contributed >=300 word tranlations to the > following projects will be nominated as Active User Contributors [2] > assuming that their Zanata e-mail IDs are registered as Foundation > members: "contributor-guide, edge-computing, > leveraging-containers-openstack, openstack-user-survey, operations-guide". > > This is new nomination after the decision during Stein PTG with User > Committee: Please see [3] and [4] if you would like to follow-up with > more information. > > I will do this nomination for ATC & AUC as PTL just after July 25, > although the due of extra ATC is Aug 26-30 [5] since I synced with > Foundation that the price of Summit ticket will increase around Aug 7 > : processing within the due will be fine, also I set period exactly for > six months. > > 4. Team status > : Although I regularly be online during I18n (+Docs) IRC team meetings, > I couldn't find enough quorum, and I couldn't proceed more deep > conversation with language teams (usually no reply from my e-mails > unfortunately). > : Current I18n team core members seem busy but when I communicate with > them, all they are still very supportive to I18n team projects. I really > appreciate their kind help. > : Also, I find that some translators are very active comparing to some > inactive language coordinators. In this case, would it be a good idea to > nominate them as language coordinators? Please share your thoughts. > : However, if such lack of team member conversations keep going, then > I18n team might decide during upcoming PTG to make translation from an > official team status into Special Interest Group [6]. > This would mean that there will be no official PTL election and there > will be no extra-ATC nomination, although I need to more figure out in > details in future. > > > With many thanks, > > /Ian > > [1] > https://docs.openstack.org/i18n/latest/official_translator.html#atc-status-in-i18n-project > [2]https://governance.openstack.org/uc/reference/charter.html#active-user-contributors-auc > [3] > http://lists.openstack.org/pipermail/openstack-i18n/2018-September/003316.html > [4] http://eavesdrop.openstack.org/meetings/uc/2018/uc.2018-10-01-18.02.html > [5] https://releases.openstack.org/train/schedule.html > [6] https://governance.openstack.org/sigs/ > > _______________________________________________ > OpenStack-I18n mailing list > OpenStack-I18n at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-i18n From mriedemos at gmail.com Mon Jul 29 17:06:37 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Mon, 29 Jul 2019 12:06:37 -0500 Subject: [forum][sdk][nova] Closing compute API feature gaps in the openstack CLI - recap In-Reply-To: <799f4669-5c92-5cd5-f8ee-4e9a8baae35a@gmail.com> References: <799f4669-5c92-5cd5-f8ee-4e9a8baae35a@gmail.com> Message-ID: <12b4bead-98c5-53b4-2134-a9a633787e10@gmail.com> On 5/2/2019 11:11 AM, Matt Riedemann wrote: > **High Priority** > > 1. Make the boot-from-volume experience better by: > > a) Support type=image for the --block-device-mapping option. > > b) Add a --boot-from-volume option which will translate to a root > --block-device-mapping using the provided --image value (nova will > create the root volume under the covers). I'm now tracking this item in storyboard and have a WIP patch for (a). https://storyboard.openstack.org/#!/story/2006302 -- Thanks, Matt From miguel at mlavalle.com Mon Jul 29 17:21:50 2019 From: miguel at mlavalle.com (Miguel Lavalle) Date: Mon, 29 Jul 2019 12:21:50 -0500 Subject: [openstack-dev] [neutron] [rally] [osprofiler] Message-ID: Hi, The Neutron team has been playing lately with Rally scenarios and osprofiler. One example is http://logs.openstack.org/80/671780/1/check/neutron-rally-task/6cc66e4/results/report.html.gz#/NeutronNetworks.create_and_bind_ports/output. As can be seen in this example, we get a lot of trace points from SQLAlchemy with the corresponding SQL statement that was sent the DB. Is there a way we can add information to these trace points to help us relate them to the Neutron code that issued the statement? Cheers Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: From thuanlk at viettel.com.vn Mon Jul 29 15:38:31 2019 From: thuanlk at viettel.com.vn (thuanlk at viettel.com.vn) Date: Mon, 29 Jul 2019 22:38:31 +0700 (ICT) Subject: [neutron] OpenvSwitch firewall sctp getting dropped Message-ID: <000001d54623$a91ae750$fb50b5f0$@viettel.com.vn> I have installed Openstack Queens on CentOs 7 with OvS and I recently used the native openvswitch firewall to implement SecusiryGroup. The native OvS firewall seems to work just fine with TCP/UDP traffic but it does not forward any SCTP traffic going to the VMs no matter how I change the security groups, But it run if i disable port security completely or use iptables_hybrid firewall driver. What do I have to do to allow SCTP packets to reach the VMs? -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Mon Jul 29 18:26:56 2019 From: smooney at redhat.com (Sean Mooney) Date: Mon, 29 Jul 2019 19:26:56 +0100 Subject: [neutron] OpenvSwitch firewall sctp getting dropped In-Reply-To: <000001d54623$a91ae750$fb50b5f0$@viettel.com.vn> References: <000001d54623$a91ae750$fb50b5f0$@viettel.com.vn> Message-ID: On Mon, 2019-07-29 at 22:38 +0700, thuanlk at viettel.com.vn wrote: > I have installed Openstack Queens on CentOs 7 with OvS and I recently used > the native openvswitch firewall to implement SecusiryGroup. The native OvS > firewall seems to work just fine with TCP/UDP traffic but it does not > forward any SCTP traffic going to the VMs no matter how I change the > security groups, But it run if i disable port security completely or use > iptables_hybrid firewall driver. What do I have to do to allow SCTP packets > to reach the VMs? the security groups api is a whitelist model so all traffic is droped by default. if you want to allow sctp you would ihave to create an new security group rule with ip_protocol set to the protocol number for sctp. e.g. openstack security group rule create --protocol sctp ... im not sure if neutron support --dst-port for sctp but you can still filter on --remote-ip or --remote-group and can specify the rule as an --ingress or --egress rule as normal. https://docs.openstack.org/python-openstackclient/stein/cli/command-objects/security-group-rule.html based on this commit https://github.com/openstack/neutron/commit/f711ad78c5c0af44318c6234957590c91592b984 it looks like neutron now validates the prot ranges for sctp impligying it support setting them so i gues its just a gap in the documentation. > From corvus at inaugust.com Mon Jul 29 20:52:20 2019 From: corvus at inaugust.com (James E. Blair) Date: Mon, 29 Jul 2019 13:52:20 -0700 Subject: [release][infra] Supporting rget in our release process Message-ID: <8736io7ed7.fsf@meyer.lemoncheese.net> Hi, A colleague at Red Hat is working on an effort to record signatures of release artifacts. Essentially it's a way to help users verify release artifacts (or determine if they have been changed) independent of PGP signatures. You can read about it here: https://github.com/merklecounty/rget#rget It sounds like an interesting and useful effort, and I think we can support it at little cost. If we wanted to do so, I think we would need to do the following things: 1) Generate SHA256SUMS of our release artifacts. These could even include the GPG signature files. 2) Run "rget submit" on the resulting files after publication. That's it. Both of those would be changes to the release publication jobs, and wouldn't require any other changes to our processes. As mentioned in the README this is very early stages and the author, Brandon Philips, welcomes both further testing and feedback on the process in general. Thoughts? -Jim From jlibosva at redhat.com Mon Jul 29 20:31:56 2019 From: jlibosva at redhat.com (Jakub Libosvar) Date: Mon, 29 Jul 2019 22:31:56 +0200 Subject: [neutron] OpenvSwitch firewall sctp getting dropped In-Reply-To: <000001d54623$a91ae750$fb50b5f0$@viettel.com.vn> References: <000001d54623$a91ae750$fb50b5f0$@viettel.com.vn> Message-ID: On 29/07/2019 17:38, thuanlk at viettel.com.vn wrote: > I have installed Openstack Queens on CentOs 7 with OvS and I recently used > the native openvswitch firewall to implement SecusiryGroup. The native OvS > firewall seems to work just fine with TCP/UDP traffic but it does not > forward any SCTP traffic going to the VMs no matter how I change the > security groups, But it run if i disable port security completely or use > iptables_hybrid firewall driver. What do I have to do to allow SCTP packets > to reach the VMs? > > You need to load kernel module for netfilter that supports sctp. Depending on the kernel you're using, it could be either compiled in or compiled as a module. You can try to modprobe ip_conntrack_proto_sctp to see if it fixes the issue for you. Kuba From cboylan at sapwetik.org Mon Jul 29 21:52:25 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 29 Jul 2019 14:52:25 -0700 Subject: [release][infra] Supporting rget in our release process In-Reply-To: <8736io7ed7.fsf@meyer.lemoncheese.net> References: <8736io7ed7.fsf@meyer.lemoncheese.net> Message-ID: <74d85430-9197-465a-bfcf-a6651b057d2d@www.fastmail.com> On Mon, Jul 29, 2019, at 1:52 PM, James E. Blair wrote: > Hi, > > A colleague at Red Hat is working on an effort to record signatures of > release artifacts. Essentially it's a way to help users verify release > artifacts (or determine if they have been changed) independent of PGP > signatures. You can read about it here: > https://github.com/merklecounty/rget#rget > > It sounds like an interesting and useful effort, and I think we can > support it at little cost. If we wanted to do so, I think we would need > to do the following things: > > 1) Generate SHA256SUMS of our release artifacts. These could even > include the GPG signature files. We'll also need to publish the sha256sums file. We are already publishing the other files so this should be easy. > > 2) Run "rget submit" on the resulting files after publication. > > That's it. There is also a step 0) install rget. Unfortunately their docs don't mention how to verify the installation of rget (self bootstrapping) though I think you would download rget, hash it before running it, then run it to get the hashes of rget and then compare? Though a modified rget could just tell you it was the unmodified version. Maybe that is why they don't bother telling you what to do there. We may also want to set up some sort of periodic audit of our "certs" in the certificate transparency logs. Just to ensure there are no unexpected changes. > > Both of those would be changes to the release publication jobs, and > wouldn't require any other changes to our processes. > > As mentioned in the README this is very early stages and the author, > Brandon Philips, welcomes both further testing and feedback on the > process in general. > > Thoughts? Overall seems like a good way for people (including ourselves) to sanity check that our releases are not changing unexpectedly. > > -Jim > > From fungi at yuggoth.org Mon Jul 29 22:03:03 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 29 Jul 2019 22:03:03 +0000 Subject: [OpenStack-Infra] [release][infra] Supporting rget in our release process In-Reply-To: <8736io7ed7.fsf@meyer.lemoncheese.net> References: <8736io7ed7.fsf@meyer.lemoncheese.net> Message-ID: <20190729220303.ukvhvhzppezy7ufo@yuggoth.org> On 2019-07-29 13:52:20 -0700 (-0700), James E. Blair wrote: > A colleague at Red Hat is working on an effort to record signatures of > release artifacts. Essentially it's a way to help users verify release > artifacts (or determine if they have been changed) independent of PGP > signatures. You can read about it here: > https://github.com/merklecounty/rget#rget [...] > As mentioned in the README this is very early stages and the author, > Brandon Philips, welcomes both further testing and feedback on the > process in general. > > Thoughts? I really like the way it leverages RFC 6962 Certificate Transparency logs for checksum distribution; the decision not to fall into the blockchain-all-things trap lends a lot of additional credibility to the idea. I agree this would be fairly trivial to integrate into our release publication jobs, and even to backfill with our existing archives. It would grant consumers of our release artifacts the ability to validate them against an open third-party registry, separately from checking the cryptographic release signatures we currently provide alongside them... it could even be used to detect tampering with the signatures themselves in the event of a signing key compromise. This seems like a great idea for URLs of artifacts we host, and I'm happy to hack on the implementation in OpenDev's CI system, likely via a new role in Zuul's standard library of job components. For artifacts we upload to third-party services like PyPI and Docker Hub on the other hand, assuming I've digested (pun intended) the relevant literature correctly, it might make more sense for the maintainers of those services to do something similar as they tend to perform a fair amount of URL indirection and so trying to keep up historical data for those URLs ourselves could be tricky. On the other hand if those third-party services were to integrate rget updating as part of their infrastructure it would be a lot more seamless (especially if they similarly integrated CT checks into the corresponding client-side tooling). Another challenge I see is that, due to the fact that most of what we host is source code, and most consumers of our source code are obtaining it via Git rather than release artifacts, rget wouldn't really do much for them as far as I can see... though once Git completes its planned transition to SHA2-256 in the coming years, I could see a call for some solution to publish known object hashes to a CT log in a similar fashion. I suppose it could be done now by re-checksumming all content over a Git object and submitting a certificate for that, but it seems a bit heavy-weight and I'll admit to not having thought through it completely so there are likely hidden gotchas with that idea. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From kecarter at redhat.com Mon Jul 29 22:51:32 2019 From: kecarter at redhat.com (Kevin Carter) Date: Mon, 29 Jul 2019 17:51:32 -0500 Subject: [TripleO] Transformation squad meeting time In-Reply-To: References: Message-ID: Hello all, Based on the doodle[1] and the results of the tie-breaker vote held at our last meeting[0], the TripleO Transformation squad will now meet every Wednesday at 14:00 UTC. Please update your calendars accordingly. Thank you to all the folks who participated in this process, and I look forward to chatting with everyone at the new meeting time this Wednesday. [0] http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-07-25-13.00.log.html#l-112 [1] https://doodle.com/poll/nkkqnkarhbgp8cxr -- Kevin Carter IRC: cloudnull On Thu, Jul 18, 2019 at 12:16 PM Kevin Carter wrote: > Hello, > > I'm writing to get feedback on what the best meeting time would be for all > parties interested in participating in the TripleO Transformation Squad > efforts. During this weeks meeting, it was raised that the current weekly > time, Thursday 13:00 UTC, is less than ideal for a bunch of folks, and we > should consider changing it [0]. To ensure we're getting everyone involved > in the process, I've created a doodle that will run for one week [1]. In > this Doodle, I'd like folks to vote on whatever time works best for them. > During the next meeting, 25 Thursday 13:00 UTC, the results of the survey > will be announced and the day and time with the highest number of votes > will be used for all future meetings. > > Thank you, and please go vote! > > [0] > http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-07-18-13.01.log.html#l-168 > [1] https://doodle.com/poll/nkkqnkarhbgp8cxr > > -- > Kevin Carter > IRC: kecarter > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianyrchoi at gmail.com Mon Jul 29 23:55:45 2019 From: ianyrchoi at gmail.com (Ian Y. Choi) Date: Tue, 30 Jul 2019 08:55:45 +0900 Subject: [OpenStack-I18n] [I18n] project team status update regarding internationalization/translation as July 2019 In-Reply-To: References: <6058a029-9922-7c4a-9f95-a7348f3a7c44@gmail.com> Message-ID: Hello Motiki-san, Thank you for your comments. For undocumented stuff, I have just proposed a patch: https://review.opendev.org/#/c/673375/ . And I am now collecting the statistics (it takes so long / is difficult due to unstable network connection from my side). Let's see where we have such translators or not.. With many thanks, /Ian Akihiro Motoki wrote on 7/30/2019 1:25 AM: > Hi Ian, > > Thanks for sharing the current status. > >> 3. Updating ATC & AUC >> : Frank previously submitted extra-ATC for Stein through >> https://review.opendev.org/#/c/633398/ , and this time I would like to >> submit the stat calculation from 2019-01-26 to 2019-07-25. >> : All contributors who had commits on openstack/i18n repository will be >> automatically regarded as Active Technical Contributors [1], and >> I18n team would like to nominate all translators who contributed >> translations on OpenStack official repositories >> (https://governance.openstack.org/tc/reference/projects/ ) >= 300 words >> as extra-ATCs assuming that their Zanata e-mail IDs are registered as >> Foundation members. >> : All translators who contributed >=300 word tranlations to the >> following projects will be nominated as Active User Contributors [2] >> assuming that their Zanata e-mail IDs are registered as Foundation >> members: "contributor-guide, edge-computing, >> leveraging-containers-openstack, openstack-user-survey, operations-guide". > Does this mean if someone translates >=300 words but translates <300 > words in ATC targeted projects and <300 words in AUC targeted projects > they are not recognized as either of ATC and AUC? The i18n doc [1] > just mentions translators who translate >=300 words will be regarded > as ATC. From the i18n guide perspective, the above statement in Ian's > mail looks undocumented. It sounds better to be consistent (though I > am not sure we have such translators.) > > [1] https://docs.openstack.org/i18n/latest/official_translator.html#atc-status-in-i18n-project > Find my question inline below. > > Thanks, > Akihiro > > On Wed, Jul 24, 2019 at 8:48 AM Ian Y. Choi wrote: >> Hello all, >> >> I would like to share current project team status for better visibility >> into I18n team members, copying to openstack-discuss and user-committee >> thread: >> >> 1. Translation and team effort status >> There are still lots of translation contribution as you see through >> Stackalytics: >> https://www.stackalytics.com/?metric=translations&release=train . >> >> The number of translated words is not larger than the previous cycle, >> but I estimate that it will be similar or a little bit less than the >> previous cycle since I18n usually contributes translations a lot especially >> during Soft & Hard StringFreeze periods. >> Note that I18n team currently sets higher priority on OpenStack user >> survey translation and project document translations. >> Such priorities information can be found on Zanata dashboard: >> https://translate.openstack.org/ . >> Other progress such as migrating Zanata into other translation platform >> seems slow - me and Frank are planning to have a virtual sprint. >> >> 2. Shanghai Summit & PTG preparation >> : Frank submitted his presentation including me and Hoarce (China team). >> Let's wait for final approval on presentation submission. >> : There should be PTG at Shanghai - please add your name and detail >> thinking to Etherpad mentioned in >> http://lists.openstack.org/pipermail/openstack-i18n/2019-July/003436.html . >> I strongly think to more encourage participation since I expect more >> participation especially from Chinese translators & contributors. >> >> 3. Updating ATC & AUC >> : Frank previously submitted extra-ATC for Stein through >> https://review.opendev.org/#/c/633398/ , and this time I would like to >> submit the stat calculation from 2019-01-26 to 2019-07-25. >> : All contributors who had commits on openstack/i18n repository will be >> automatically regarded as Active Technical Contributors [1], and >> I18n team would like to nominate all translators who contributed >> translations on OpenStack official repositories >> (https://governance.openstack.org/tc/reference/projects/ ) >= 300 words >> as extra-ATCs assuming that their Zanata e-mail IDs are registered as >> Foundation members. >> : All translators who contributed >=300 word tranlations to the >> following projects will be nominated as Active User Contributors [2] >> assuming that their Zanata e-mail IDs are registered as Foundation >> members: "contributor-guide, edge-computing, >> leveraging-containers-openstack, openstack-user-survey, operations-guide". >> >> This is new nomination after the decision during Stein PTG with User >> Committee: Please see [3] and [4] if you would like to follow-up with >> more information. >> >> I will do this nomination for ATC & AUC as PTL just after July 25, >> although the due of extra ATC is Aug 26-30 [5] since I synced with >> Foundation that the price of Summit ticket will increase around Aug 7 >> : processing within the due will be fine, also I set period exactly for >> six months. >> >> 4. Team status >> : Although I regularly be online during I18n (+Docs) IRC team meetings, >> I couldn't find enough quorum, and I couldn't proceed more deep >> conversation with language teams (usually no reply from my e-mails >> unfortunately). >> : Current I18n team core members seem busy but when I communicate with >> them, all they are still very supportive to I18n team projects. I really >> appreciate their kind help. >> : Also, I find that some translators are very active comparing to some >> inactive language coordinators. In this case, would it be a good idea to >> nominate them as language coordinators? Please share your thoughts. >> : However, if such lack of team member conversations keep going, then >> I18n team might decide during upcoming PTG to make translation from an >> official team status into Special Interest Group [6]. >> This would mean that there will be no official PTL election and there >> will be no extra-ATC nomination, although I need to more figure out in >> details in future. >> >> >> With many thanks, >> >> /Ian >> >> [1] >> https://docs.openstack.org/i18n/latest/official_translator.html#atc-status-in-i18n-project >> [2]https://governance.openstack.org/uc/reference/charter.html#active-user-contributors-auc >> [3] >> http://lists.openstack.org/pipermail/openstack-i18n/2018-September/003316.html >> [4] http://eavesdrop.openstack.org/meetings/uc/2018/uc.2018-10-01-18.02.html >> [5] https://releases.openstack.org/train/schedule.html >> [6] https://governance.openstack.org/sigs/ >> >> _______________________________________________ >> OpenStack-I18n mailing list >> OpenStack-I18n at lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-i18n From gmann at ghanshyammann.com Tue Jul 30 02:31:44 2019 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 30 Jul 2019 11:31:44 +0900 Subject: [qa]Proposing Attila Fazekas (afazekas) to Devstack Core list Message-ID: <16c40b81fd9.b5888c7e559372.8430955883467629240@ghanshyammann.com> Hello Everyone, I would like to propose to add Attila Fazekas (afazekas) in devstack core list. He has been contributing and reviewing code in devstack since a long time and has deep knowledge of code base. Please reply with your approval or objections. If no objection then till 5th Aug, I will add him in the list. -gmann From khacthuan.hut at gmail.com Tue Jul 30 04:32:02 2019 From: khacthuan.hut at gmail.com (KhacThuan Bk) Date: Tue, 30 Jul 2019 11:32:02 +0700 Subject: [neutron] OpenvSwitch firewall sctp getting dropped In-Reply-To: References: <000001d54623$a91ae750$fb50b5f0$@viettel.com.vn> Message-ID: modprobe ip_conntrack_proto_sctp --> Module not found My kernel is 3.10.0-957.el7.x86_64 Vào Th 3, 30 thg 7, 2019 lúc 04:13 Jakub Libosvar đã viết: > On 29/07/2019 17:38, thuanlk at viettel.com.vn wrote: > > I have installed Openstack Queens on CentOs 7 with OvS and I recently > used > > the native openvswitch firewall to implement SecusiryGroup. The native > OvS > > firewall seems to work just fine with TCP/UDP traffic but it does not > > forward any SCTP traffic going to the VMs no matter how I change the > > security groups, But it run if i disable port security completely or use > > iptables_hybrid firewall driver. What do I have to do to allow SCTP > packets > > to reach the VMs? > > > > > You need to load kernel module for netfilter that supports sctp. > Depending on the kernel you're using, it could be either compiled in or > compiled as a module. You can try to > > modprobe ip_conntrack_proto_sctp > > to see if it fixes the issue for you. > > Kuba > > > -- *Lăng Khắc Thuận* *Phone*: 01649729889 *Email: khacthuan.hut at gmail.com * *Skype: khacthuan_bk* *Student at Applied Mathematics and Informatics* *Center for training of excellent students* *Hanoi University of Science and Technology. * -------------- next part -------------- An HTML attachment was scrubbed... URL: From rony.khan at brilliant.com.bd Tue Jul 30 05:03:42 2019 From: rony.khan at brilliant.com.bd (Md. Farhad Hasan Khan) Date: Tue, 30 Jul 2019 11:03:42 +0600 Subject: Error: python2-uritemplate conflicts with python2-uri-templates-0.6-5.el7.noarch In-Reply-To: References: <060001d542ac$8653e1b0$92fba510$@brilliant.com.bd> <02b701d5446c$75b4d640$611e82c0$@brilliant.com.bd> Message-ID: <064c01d54694$289adec0$79d09c40$@brilliant.com.bd> HI Yatin Karel, Thanks. For help. I shall check what you suggest and will update you. Thanks & B'Rgds, Rony -----Original Message----- From: Yatin Karel [mailto:ykarel at redhat.com] Sent: Monday, July 29, 2019 8:30 PM To: rony.khan at brilliant.com.bd Cc: OpenStack Discuss Subject: Re: Error: python2-uritemplate conflicts with python2-uri-templates-0.6-5.el7.noarch Hi Rony, See inline reply. On Sat, Jul 27, 2019 at 4:48 PM Md. Farhad Hasan Khan wrote: > > Hi, > > Please help me how can I fix this. > > > > Thanks/Rony > > > > > > From: Md. Farhad Hasan Khan [mailto:rony.khan at brilliant.com.bd] > Sent: Thursday, July 25, 2019 11:48 AM > To: openstack-discuss at lists.openstack.org > Subject: Error: python2-uritemplate conflicts with > python2-uri-templates-0.6-5.el7.noarch > > > > Hi, > > When I run # yum update and found below conflict: > > > > --> Processing Conflict: python2-uritemplate-3.0.0-1.el7.noarch > --> conflicts python2-uri-templates > > --> Finished Dependency Resolution > > Error: python2-uritemplate conflicts with > python2-uri-templates-0.6-5.el7.noarch > > You could try using --skip-broken to work around the problem > > You could try running: rpm -Va --nofiles –nodigest > > > > If I remove python2-uri-templates-0.6-5.el7.noarch than below package ask to remove: > > > > Removing: > > python2-uri-templates noarch 0.6-5.el7 @centos-openstack-rocky 20 k > > Removing for dependencies: > > openstack-cinder noarch 1:13.0.5-1.el7 @centos-openstack-rocky 200 k > > python-cinder noarch 1:13.0.5-1.el7 @centos-openstack-rocky 19 M > > python2-google-api-client noarch 1.4.2-4.el7 @centos-openstack-rocky 305 k > > > > > > please let help me how can I update. > > >From logs(python2-uritemplate-3.0.0-1.el7.noarch) it appears you have enabled "epel" repository, you can confirm it with yum repolist command. Mixing epel with OpenStack repos can have such issues. For now to move ahead i would suggest you disable epel repository before running "yum update". This should avoid pulling of package python2-uritemplate and thus conflict with python2-uri-templates) > > Thanks/Rony > > Thanks and Regards Yatin Karel From thierry at openstack.org Tue Jul 30 08:28:59 2019 From: thierry at openstack.org (Thierry Carrez) Date: Tue, 30 Jul 2019 10:28:59 +0200 Subject: [OpenStack-Infra] [release][infra] Supporting rget in our release process In-Reply-To: <20190729220303.ukvhvhzppezy7ufo@yuggoth.org> References: <8736io7ed7.fsf@meyer.lemoncheese.net> <20190729220303.ukvhvhzppezy7ufo@yuggoth.org> Message-ID: <949b37fd-29df-0bc0-b1c7-ba548cf69cef@openstack.org> Jeremy Stanley wrote: > [...] > For artifacts we upload to third-party services like PyPI and Docker > Hub on the other hand, assuming I've digested (pun intended) the > relevant literature correctly, it might make more sense for the > maintainers of those services to do something similar as they tend > to perform a fair amount of URL indirection and so trying to keep up > historical data for those URLs ourselves could be tricky. On the > other hand if those third-party services were to integrate rget > updating as part of their infrastructure it would be a lot more > seamless (especially if they similarly integrated CT checks into the > corresponding client-side tooling). > > Another challenge I see is that, due to the fact that most of what > we host is source code, and most consumers of our source code are > obtaining it via Git rather than release artifacts, rget wouldn't > really do much for them as far as I can see... though once Git > completes its planned transition to SHA2-256 in the coming years, I > could see a call for some solution to publish known object hashes to > a CT log in a similar fashion. I suppose it could be done now by > re-checksumming all content over a Git object and submitting a > certificate for that, but it seems a bit heavy-weight and I'll admit > to not having thought through it completely so there are likely > hidden gotchas with that idea. I agree with Jeremy, it seems to cover a limited amount of use cases (people who download tarball source releases from releases.o.o). But covering only a few use cases is not a reason not to do it: we should support it for the same reason we provide signatures for released artifacts right now. Furthermore it is an initiative I'm fine being early adopters of this idea, if only so that one day we may find it covering other ways to retrieve our deliverables (pypi, git). -- Thierry Carrez (ttx) From ildiko.vancsa at gmail.com Tue Jul 30 12:04:02 2019 From: ildiko.vancsa at gmail.com (Ildiko Vancsa) Date: Tue, 30 Jul 2019 14:04:02 +0200 Subject: [edge][all] Edge Hacking Days Message-ID: <7966B87C-E600-4681-83FB-FD947250012A@gmail.com> Hi, I’m reaching out with an attempt to organize hacking days to work on edge related tasks. The idea is to get together remotely on IRC/Zoom or any other platform that supports remote communication and work on items like building and testing our reference architectures or work on some project specific items like in Keystone or Ironic. Here are Doodle polls for the next three months: August: https://doodle.com/poll/ucfc9w7iewe6gdp4 September: https://doodle.com/poll/3cyqxzr9vd82pwtr October: https://doodle.com/poll/6nzziuihs65hwt7b Please mark any day when you have some availability to dedicate to hack even if it’s not a full day. Please let me know if you have any questions. As a reminder you can find the edge computing group’s resources and information about latest activities here: https://wiki.openstack.org/wiki/Edge_Computing_Group Thanks and Best Regards, Ildikó (IRC: ildikov) From smooney at redhat.com Tue Jul 30 12:21:45 2019 From: smooney at redhat.com (Sean Mooney) Date: Tue, 30 Jul 2019 13:21:45 +0100 Subject: Deprecate Aggregate[Core|Ram|Disk]Filters Message-ID: Hi all, back in january 2018 we annouced that the behavior of the Aggregate[Core|Ram|Disk]Filters http://lists.openstack.org/pipermail/openstack-dev/2018-January/126283.html form ocata on due to the use of placement you can no longer manage overcomit ratios using aggregate metadata as placement has no access to this data. as a result these filters if you use with placement then they only work if the aggregates enforce lower limits then placement default. if you also limit the responces from placemnet this also does not work as we may not recive any host that satisfy the lower limit in the aggregate metadata even if such host exist in the aggreate. we have a prominent warning https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#scheduling-considerations advising operator of the impact of this change as well as this wont fix bug from last November https://bugs.launchpad.net/nova/+bug/1804125 11 months ago we deprecated the non aggreate [Core|Ram|Disk] Filters https://review.opendev.org/#/c/596502/ for several reasons noted in the commit and we furthermore removed the caching scheduler in stien. a few days ago we finally deleted the non aggreagate filters with https://github.com/openstack/nova/commit/78645e61c63bf042453d1f822ae8b3f1ee6a311b but the Aggregate[Core|Ram|Disk]Filters are still present. we had intended to deprecate the Aggregate[Core|Ram|Disk]Filters in ocata when we sent the initial email about the behavior change however we never actully followed up and formally deprecated it in code. similarly when we formally deprecated the [Core|Ram|Disk] Filters we could have deprecated the Aggregate[Core|Ram|Disk]Filters too but we did not. As a result we cannot remove the Aggregate[Core|Ram|Disk]Filters in Train but i have proposed : https://review.opendev.org/673496 To formally deprecate them with the intent of removing them in the U cycle. In the 19.0.0 Stein release initial allocation ratios support was completed which allow managing of the allocation ratios via the placement api in addition to the existing capability to manage allocation ratios via the nova config. https://github.com/openstack/nova-specs/blob/master/specs/stein/implemented/initial-allocation-ratios.rst As of the 18.0.0 rocky release nova automatically mirrors ``host-aggregates`` to placement aggregates. https://github.com/openstack/nova-specs/blob/master/specs/rocky/implemented/placement-mirror-host-aggregates.rst There is ongoing work in Train to enhance osc-placement to add a 'resource provider inventory update' command to make setting overcommit allocation ratios on a per-aggregate basis easier https://review.opendev.org/#/c/640898/ . These 3 features combined will allow operators to manage Aggreate Resource class allocation ratios in a placement native way. finally in the case the above was not sufficent motivation the AggregateCoreFilter it does not support handeling vCPU and pCPU resouce classes seperatly and will not work with the new cpu resouce tracking spec correctly. https://github.com/openstack/nova-specs/blob/master/specs/train/approved/cpu-resources.rst the AggregateRamFilter also does not understand the difference bewteen hugepage memory and standard memroy which means that even if we were to keep the filters they do not work correctly in all cases and will continue to becomre more broken over time, so i think the time has come to finally do this. regards sean From mordred at inaugust.com Tue Jul 30 14:03:03 2019 From: mordred at inaugust.com (Monty Taylor) Date: Tue, 30 Jul 2019 10:03:03 -0400 Subject: [release][infra] Supporting rget in our release process In-Reply-To: <74d85430-9197-465a-bfcf-a6651b057d2d@www.fastmail.com> References: <8736io7ed7.fsf@meyer.lemoncheese.net> <74d85430-9197-465a-bfcf-a6651b057d2d@www.fastmail.com> Message-ID: <2880293e-dab3-6730-396c-3fc383d80d44@inaugust.com> On 7/29/19 5:52 PM, Clark Boylan wrote: > On Mon, Jul 29, 2019, at 1:52 PM, James E. Blair wrote: >> Hi, >> >> A colleague at Red Hat is working on an effort to record signatures of >> release artifacts. Essentially it's a way to help users verify release >> artifacts (or determine if they have been changed) independent of PGP >> signatures. You can read about it here: >> https://github.com/merklecounty/rget#rget >> >> It sounds like an interesting and useful effort, and I think we can >> support it at little cost. If we wanted to do so, I think we would need >> to do the following things: >> >> 1) Generate SHA256SUMS of our release artifacts. These could even >> include the GPG signature files. > > We'll also need to publish the sha256sums file. We are already publishing the other files so this should be easy. > >> >> 2) Run "rget submit" on the resulting files after publication. >> >> That's it. > > There is also a step 0) install rget. Unfortunately their docs don't mention how to verify the installation of rget (self bootstrapping) though I think you would download rget, hash it before running it, then run it to get the hashes of rget and then compare? Though a modified rget could just tell you it was the unmodified version. Maybe that is why they don't bother telling you what to do there. Actually - it turns out this is just doing a single submit to a URL, so it will likely be much easier to just use curl or an ansible URI call to do the submission step. (it's great that rget has this for folks, but in our case I think we don't need to use rget just to do the submission) > We may also want to set up some sort of periodic audit of our "certs" in the certificate transparency logs. Just to ensure there are no unexpected changes. ++ >> >> Both of those would be changes to the release publication jobs, and >> wouldn't require any other changes to our processes. >> >> As mentioned in the README this is very early stages and the author, >> Brandon Philips, welcomes both further testing and feedback on the >> process in general. >> >> Thoughts? > > Overall seems like a good way for people (including ourselves) to sanity check that our releases are not changing unexpectedly. > >> >> -Jim >> >> > > From aj at suse.com Tue Jul 30 14:42:16 2019 From: aj at suse.com (Andreas Jaeger) Date: Tue, 30 Jul 2019 16:42:16 +0200 Subject: Retiring tc-as-a-service Message-ID: <75896d01-f3bd-3f1f-0b6a-b86b47cfaaeb@suse.com> This repo was imported and never used, I'll start the retirement process for it now. Ofer, one of the initial developer on the project confirmed that the repo is dead, Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From lehrer at gmail.com Tue Jul 30 15:01:20 2019 From: lehrer at gmail.com (Mark Lehrer) Date: Tue, 30 Jul 2019 09:01:20 -0600 Subject: [cinder] lvm_mirrors option seems to be broken Message-ID: I have 4 fast nvme targets that I want to combine using RAID 10 and use with cinder-lvm. Unfortunately, lvm+mdadm is breaking all of my writes down to 4K -- without even increasing the queue depth to match. SSDs hate this workload. I tried bypassing mdadm by seting up LVM RAID10 by hand and got roughly a 10x increase in performance, so I definitely want to use this if possible. However, it looks like maybe the lvm_mirrors code hasn't been kept up to date. I ran into two main problems trying to use it: 1) The algorithm to calculate free space is broken and always returns 0. This means that the scheduler will never even try to use this pool 2) Thin provisioning is getting in the way... with lvm_mirrors enabled the code is apparently trying to create a new LV on top of the thin provisioned LV... and it is using an lvcreate option that appears to be deprecated (--mirrorlog mirrored) I expect I can fix #1, but what is the best way to handle #2? I was thinking that the easiest fix would be to use the mirror options when the thin LV is created, though I haven't tried it by hand yet to see if it works. The advantage here is that this is a one-time setting and the code path to create individual volumes wouldn't need to check lvm_mirrors at all. Thanks, Mark From mihalis68 at gmail.com Tue Jul 30 15:09:38 2019 From: mihalis68 at gmail.com (Chris Morgan) Date: Tue, 30 Jul 2019 11:09:38 -0400 Subject: [ops] NYC ops meetup sept 3,4 registration site live! Message-ID: Registration: https://go.bloomberg.com/attend/invite/open-stack-operators-quarterly-summit/ planning etherpad : https://etherpad.openstack.org/p/NYC19-OPS-MEETUP Chris -- Chris Morgan -------------- next part -------------- An HTML attachment was scrubbed... URL: From lehrer at gmail.com Tue Jul 30 16:08:28 2019 From: lehrer at gmail.com (Mark Lehrer) Date: Tue, 30 Jul 2019 10:08:28 -0600 Subject: [cinder] lvm_mirrors option seems to be broken In-Reply-To: References: Message-ID: OK things aren't quite as broken as I thought at first. It looks like lvm_type needs to be set to default (instead of thin), otherwise the lvm_mirrors option causes havoc. Maybe a warning in the logs about incompatible options would save the next person some time. Thanks! Mark On Tue, Jul 30, 2019 at 9:01 AM Mark Lehrer wrote: > > I have 4 fast nvme targets that I want to combine using RAID 10 and > use with cinder-lvm. > > Unfortunately, lvm+mdadm is breaking all of my writes down to 4K -- > without even increasing the queue depth to match. SSDs hate this > workload. > > I tried bypassing mdadm by seting up LVM RAID10 by hand and got > roughly a 10x increase in performance, so I definitely want to use > this if possible. > > However, it looks like maybe the lvm_mirrors code hasn't been kept up > to date. I ran into two main problems trying to use it: > > 1) The algorithm to calculate free space is broken and always returns > 0. This means that the scheduler will never even try to use this pool > > 2) Thin provisioning is getting in the way... with lvm_mirrors enabled > the code is apparently trying to create a new LV on top of the thin > provisioned LV... and it is using an lvcreate option that appears to > be deprecated (--mirrorlog mirrored) > > I expect I can fix #1, but what is the best way to handle #2? I was > thinking that the easiest fix would be to use the mirror options when > the thin LV is created, though I haven't tried it by hand yet to see > if it works. The advantage here is that this is a one-time setting > and the code path to create individual volumes wouldn't need to check > lvm_mirrors at all. > > Thanks, > Mark From lehrer at gmail.com Tue Jul 30 16:45:20 2019 From: lehrer at gmail.com (Mark Lehrer) Date: Tue, 30 Jul 2019 10:45:20 -0600 Subject: [cinder] lvm_mirrors option seems to be broken In-Reply-To: References: Message-ID: Final suggestion: on my system, lvm seems to work best with the "--type raid10" option. Perhaps a cinder.conf option for lvm_type=raid10 would be useful? raid1 too? I am seeing this warning from lvcreate quite a lot, so something needs to change fairly soon: WARNING: Log type "mirrored" is DEPRECATED and will be removed in the future. Use RAID1 LV or disk log instead. Thanks, Mark On Tue, Jul 30, 2019 at 10:08 AM Mark Lehrer wrote: > > OK things aren't quite as broken as I thought at first. > > It looks like lvm_type needs to be set to default (instead of thin), > otherwise the lvm_mirrors option causes havoc. Maybe a warning in the > logs about incompatible options would save the next person some time. > > Thanks! > Mark > > > On Tue, Jul 30, 2019 at 9:01 AM Mark Lehrer wrote: > > > > I have 4 fast nvme targets that I want to combine using RAID 10 and > > use with cinder-lvm. > > > > Unfortunately, lvm+mdadm is breaking all of my writes down to 4K -- > > without even increasing the queue depth to match. SSDs hate this > > workload. > > > > I tried bypassing mdadm by seting up LVM RAID10 by hand and got > > roughly a 10x increase in performance, so I definitely want to use > > this if possible. > > > > However, it looks like maybe the lvm_mirrors code hasn't been kept up > > to date. I ran into two main problems trying to use it: > > > > 1) The algorithm to calculate free space is broken and always returns > > 0. This means that the scheduler will never even try to use this pool > > > > 2) Thin provisioning is getting in the way... with lvm_mirrors enabled > > the code is apparently trying to create a new LV on top of the thin > > provisioned LV... and it is using an lvcreate option that appears to > > be deprecated (--mirrorlog mirrored) > > > > I expect I can fix #1, but what is the best way to handle #2? I was > > thinking that the easiest fix would be to use the mirror options when > > the thin LV is created, though I haven't tried it by hand yet to see > > if it works. The advantage here is that this is a one-time setting > > and the code path to create individual volumes wouldn't need to check > > lvm_mirrors at all. > > > > Thanks, > > Mark From sean.mcginnis at gmx.com Tue Jul 30 19:31:35 2019 From: sean.mcginnis at gmx.com (Sean McGinnis) Date: Tue, 30 Jul 2019 14:31:35 -0500 Subject: [cinder] lvm_mirrors option seems to be broken In-Reply-To: References: Message-ID: <20190730193135.GA3136@sm-workstation> On Tue, Jul 30, 2019 at 10:45:20AM -0600, Mark Lehrer wrote: > Final suggestion: on my system, lvm seems to work best with the > "--type raid10" option. > > Perhaps a cinder.conf option for lvm_type=raid10 would be useful? > raid1 too? I am seeing this warning from lvcreate quite a lot, so > something needs to change fairly soon: > > WARNING: Log type "mirrored" is DEPRECATED and will be removed in the > future. Use RAID1 LV or disk log instead. > > Thanks, > Mark > Looks like we have a couple of bugs. Mirroring does not appear to be supported with thin provisioned volumes, but our code does not currently check if the two are used in combination. At a minimum we should recognize that and give a nice informative warning message in the log. Since we now default to trying to create a thin pool unless explicitly asked for thick provisioning, this could be more of an issue. We should also put a warning in the help text for the lvm_mirrors config option. The `--mirrorlog mirrored` option is deprecated. Based on your experiments, we may want to just change that to `--type raid1`, or maybe better `--type raid10`. I'm not sure all the permutations of possible configurations here, so it might take someone a little time to validate the correct args for each situation. But definitely looks like this is an area that needs some attention. Thanks for digging in to this and reporting your results. Sean From lehrer at gmail.com Tue Jul 30 20:04:32 2019 From: lehrer at gmail.com (Mark Lehrer) Date: Tue, 30 Jul 2019 14:04:32 -0600 Subject: [cinder] lvm_mirrors option seems to be broken In-Reply-To: <20190730193135.GA3136@sm-workstation> References: <20190730193135.GA3136@sm-workstation> Message-ID: Any time! Thanks for the quick response here and on freenode. BLZb^H^H^H^H Mark On Tue, Jul 30, 2019 at 1:31 PM Sean McGinnis wrote: > > On Tue, Jul 30, 2019 at 10:45:20AM -0600, Mark Lehrer wrote: > > Final suggestion: on my system, lvm seems to work best with the > > "--type raid10" option. > > > > Perhaps a cinder.conf option for lvm_type=raid10 would be useful? > > raid1 too? I am seeing this warning from lvcreate quite a lot, so > > something needs to change fairly soon: > > > > WARNING: Log type "mirrored" is DEPRECATED and will be removed in the > > future. Use RAID1 LV or disk log instead. > > > > Thanks, > > Mark > > > > Looks like we have a couple of bugs. > > Mirroring does not appear to be supported with thin provisioned volumes, but > our code does not currently check if the two are used in combination. At a > minimum we should recognize that and give a nice informative warning message in > the log. Since we now default to trying to create a thin pool unless explicitly > asked for thick provisioning, this could be more of an issue. We should also > put a warning in the help text for the lvm_mirrors config option. > > The `--mirrorlog mirrored` option is deprecated. Based on your experiments, we > may want to just change that to `--type raid1`, or maybe better `--type > raid10`. > > I'm not sure all the permutations of possible configurations here, so it might > take someone a little time to validate the correct args for each situation. But > definitely looks like this is an area that needs some attention. > > Thanks for digging in to this and reporting your results. > > Sean From colleen at gazlene.net Tue Jul 30 20:28:57 2019 From: colleen at gazlene.net (Colleen Murphy) Date: Tue, 30 Jul 2019 13:28:57 -0700 Subject: [ops][keystone] Does anyone use external auth? Message-ID: <5131b946-e261-433b-a751-9ed3704c9c92@www.fastmail.com> Currently, one of the default auth methods for keystone is 'external', meaning keystone offloads authentication to an HTTPD auth module like mod_ssl or mod_auth_kerb and gets the user's identity from the REMOTE_USER variable passed in by the web server: https://docs.openstack.org/keystone/latest/admin/external-authentication.html The 'external' auth method existed before federation. The biggest problem with external auth now is that it is effectively single-domain, there's no way to parse anything besides a user identifier from the REMOTE_USER variable, and keystone is barreling full steam ahead to a multidomain world. The 'external' auth method conflicts with the 'mapped' auth method as mentioned in the "Caution" notice in the above document for the same reason. Moreover, we should be able to achieve the same behavior with just federation, e.g. you can create a federated IdP representing your SSL CA, and continue to use mod_ssl with a mapping to properly parse all the attributes coming in from the auth module. We'd like to start discouraging, deprecating, and removing external auth in keystone. So our question to operators is: are you currently using external auth? If so, which HTTPD auth modules are you using? And is it a use case that we can't support with federated auth? Colleen (cmurphy) From Kevin.Fox at pnnl.gov Tue Jul 30 20:45:29 2019 From: Kevin.Fox at pnnl.gov (Fox, Kevin M) Date: Tue, 30 Jul 2019 20:45:29 +0000 Subject: [ops][keystone] Does anyone use external auth? In-Reply-To: <5131b946-e261-433b-a751-9ed3704c9c92@www.fastmail.com> References: <5131b946-e261-433b-a751-9ed3704c9c92@www.fastmail.com> Message-ID: <1A3C52DFCD06494D8528644858247BF01C3E3C5C@EX10MBOX03.pnnl.gov> https://www.youtube.com/watch?v=7BSnhRZ8nhs mentions they use mod_auth_oidc. Not sure that is still true. But may want to reach out to them. Thanks, Kevin ________________________________________ From: Colleen Murphy [colleen at gazlene.net] Sent: Tuesday, July 30, 2019 1:28 PM To: openstack-discuss at lists.openstack.org Subject: [ops][keystone] Does anyone use external auth? Currently, one of the default auth methods for keystone is 'external', meaning keystone offloads authentication to an HTTPD auth module like mod_ssl or mod_auth_kerb and gets the user's identity from the REMOTE_USER variable passed in by the web server: https://docs.openstack.org/keystone/latest/admin/external-authentication.html The 'external' auth method existed before federation. The biggest problem with external auth now is that it is effectively single-domain, there's no way to parse anything besides a user identifier from the REMOTE_USER variable, and keystone is barreling full steam ahead to a multidomain world. The 'external' auth method conflicts with the 'mapped' auth method as mentioned in the "Caution" notice in the above document for the same reason. Moreover, we should be able to achieve the same behavior with just federation, e.g. you can create a federated IdP representing your SSL CA, and continue to use mod_ssl with a mapping to properly parse all the attributes coming in from the auth module. We'd like to start discouraging, deprecating, and removing external auth in keystone. So our question to operators is: are you currently using external auth? If so, which HTTPD auth modules are you using? And is it a use case that we can't support with federated auth? Colleen (cmurphy) From colleen at gazlene.net Tue Jul 30 21:00:09 2019 From: colleen at gazlene.net (Colleen Murphy) Date: Tue, 30 Jul 2019 14:00:09 -0700 Subject: [ops][keystone] Does anyone use external auth? In-Reply-To: <1A3C52DFCD06494D8528644858247BF01C3E3C5C@EX10MBOX03.pnnl.gov> References: <5131b946-e261-433b-a751-9ed3704c9c92@www.fastmail.com> <1A3C52DFCD06494D8528644858247BF01C3E3C5C@EX10MBOX03.pnnl.gov> Message-ID: On Tue, Jul 30, 2019, at 13:45, Fox, Kevin M wrote: > https://www.youtube.com/watch?v=7BSnhRZ8nhs mentions they use > mod_auth_oidc. Not sure that is still true. But may want to reach out > to them. The video shows that they're using federated authentication, which provides them with the ability to create a rich attribute mapping and connect their OpenIDC IdP to keystone while still using the auth module in front of keystone. The 'external' auth method only provides a subset of that functionality, which is simply the ability to accept the auth module's parameters as a valid authentication. Colleen > > Thanks, > Kevin > ________________________________________ > From: Colleen Murphy [colleen at gazlene.net] > Sent: Tuesday, July 30, 2019 1:28 PM > To: openstack-discuss at lists.openstack.org > Subject: [ops][keystone] Does anyone use external auth? > > Currently, one of the default auth methods for keystone is 'external', > meaning keystone offloads authentication to an HTTPD auth module like > mod_ssl or mod_auth_kerb and gets the user's identity from the > REMOTE_USER variable passed in by the web server: > > https://docs.openstack.org/keystone/latest/admin/external-authentication.html > > The 'external' auth method existed before federation. The biggest > problem with external auth now is that it is effectively single-domain, > there's no way to parse anything besides a user identifier from the > REMOTE_USER variable, and keystone is barreling full steam ahead to a > multidomain world. The 'external' auth method conflicts with the > 'mapped' auth method as mentioned in the "Caution" notice in the above > document for the same reason. Moreover, we should be able to achieve > the same behavior with just federation, e.g. you can create a federated > IdP representing your SSL CA, and continue to use mod_ssl with a > mapping to properly parse all the attributes coming in from the auth > module. > > We'd like to start discouraging, deprecating, and removing external > auth in keystone. So our question to operators is: are you currently > using external auth? If so, which HTTPD auth modules are you using? And > is it a use case that we can't support with federated auth? > > Colleen (cmurphy) > > From chris at openstack.org Tue Jul 30 23:51:40 2019 From: chris at openstack.org (Chris Hoge) Date: Tue, 30 Jul 2019 16:51:40 -0700 Subject: [loci][openstack-helm][tc] Proposal to fold Loci into OpenStack-Helm Message-ID: <0FBDA438-3B13-42A1-BD96-F8B30CB27197@openstack.org> In the interest of consolidating workload, I'm proposing that we roll the Loci project under the governance of OpenStack Helm. Loci is a fairly low-bandwidth project, with the primary consumer of it being OpenStack Helm. By rolling Loci into OSH, we will be able focus shared efforts and increase review velocity for the Loci project. The architecture of OpenStack Helm allows it to be container build-system agnostic, and similarly the architecture of Loci allows it to be deployment tooling agnostic. Neither of these goals will change, as there are users of both projects that aren't using the other. However, as they are the largest consumers for each other's work, this is a natural pairing and both project stand to benefit from a tighter governance coupling. We already share a good proportion of the development team and frequently plan PTG sessions to allow for collaboration. If the community agrees with this approach, we would like to finalize the governance change before the next development cycle to allow for a graceful transition. Thanks, Chris Hoge Loci PTL From berndbausch at gmail.com Wed Jul 31 00:17:39 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Wed, 31 Jul 2019 09:17:39 +0900 Subject: [rdo] [packstack] failing to setup Stein with Openvswitch/VXLAN networking Message-ID: Trying to set up a Stein cloud with Packstack. I want the Openvswitch mech driver and VXLAN type driver. A few weeks ago, the following invocation was successful: |sudo packstack --debug --allinone --default-password pw \ --os-neutron-ovs-bridge-interfaces=br-ex:eth0 \ --os-neutron-ml2-tenant-network-types=vxlan \ --os-neutron-ml2-mechanism-drivers=openvswitch \ --os-neutron-ml2-type-drivers=vxlan,flat \ --os-neutron-l2-agent=openvswitch \ --provision-demo-floatrange=10.1.1.0/24\ --provision-demo-allocation-pools '["start=10.1.1.10,end=10.1.1.50"]'\ --os-heat-install=y --os-heat-cfn-install=y||| Now, it fails during network setup. My network connection to the Packstack server is severed, and it turns out that its only network interface /eth0 /has no IP address and is down. No bridge exists. In the /network.pp.finished /file, I find various /ovs-vsctl /commands including /add-br/, and a command /ifdown eth0 /which fails with exit code 1 (no error message from the /ifdown /command is logged). *Can somebody recommend the options required to successfully deploy a Stein cloud based on the Openvswitch and VXLAN drivers?* Thanks much, Bernd || || -------------- next part -------------- An HTML attachment was scrubbed... URL: From berndbausch at gmail.com Wed Jul 31 01:22:14 2019 From: berndbausch at gmail.com (Bernd Bausch) Date: Wed, 31 Jul 2019 10:22:14 +0900 Subject: [telemetry][ceilometer][gnocchi] How to configure aggregate for cpu_util or calculate from metrics In-Reply-To: <48533933-1443-6ad3-9cf1-940ac4d52d6f@dantalion.nl> References: <14ff728c-f19e-e869-90b1-4ff37f7170af@suse.com> <20AC2324-24B6-40D1-A0A4-0382BCE430A7@cern.ch> <48533933-1443-6ad3-9cf1-940ac4d52d6f@dantalion.nl> Message-ID: The message at the end of this email is some three months old. I have the same problem. The question is: *How to use the new rate metrics in Gnocchi. *I am using a Stein Devstack for my tests.* * For example, I need the CPU rate, formerly named /cpu_util/. I created a new archive policy that uses /rate:mean/ aggregation and has a 1 minute granularity: $ gnocchi archive-policy show ceilometer-medium-rate +---------------------+------------------------------------------------------------------+ | Field               | Value | +---------------------+------------------------------------------------------------------+ | aggregation_methods | rate:mean, mean                                                  | | back_window         | 0 | | definition          | - points: 10080, granularity: 0:01:00, timespan: 7 days, 0:00:00 | | name                | ceilometer-medium-rate | +---------------------+------------------------------------------------------------------+ I added the new policy to the publishers in /pipeline.yaml/: $ tail -n5 /etc/ceilometer/pipeline.yaml sinks:     - name: meter_sink       publishers:           - gnocchi://?archive_policy=medium&filter_project=gnocchi_swift *- gnocchi://?archive_policy=ceilometer-medium-rate&filter_project=gnocchi_swift* After restarting all of Ceilometer, my hope was that the CPU rate would magically appear in the metric list. But no: All metrics are linked to archive policy /medium/, and looking at the details of an instance, I don't detect anything rate-related: $ gnocchi resource show ae3659d6-8998-44ae-a494-5248adbebe11 +-----------------------+---------------------------------------------------------------------+ | Field                 | Value | +-----------------------+---------------------------------------------------------------------+ ... | metrics               | compute.instance.booting.time: 76fac1f5-962e-4ff2-8790-1f497c99c17d | |                       | cpu: af930d9a-a218-4230-b729-fee7e3796944                           | |                       | disk.ephemeral.size: 0e838da3-f78f-46bf-aefb-aeddf5ff3a80           | |                       | disk.root.size: 5b971bbf-e0de-4e23-ba50-a4a9bf7dfe6e                | |                       | memory.resident: 09efd98d-c848-4379-ad89-f46ec526c183               | |                       | memory.swap.in: 1bb4bb3c-e40a-4810-997a-295b2fe2d5eb                | |                       | memory.swap.out: 4d012697-1d89-4794-af29-61c01c925bb4               | |                       | memory.usage: 93eab625-0def-4780-9310-eceff46aab7b                  | |                       | memory: ea8f2152-09bd-4aac-bea5-fa8d4e72bbb1                        | |                       | vcpus: e1c5acaf-1b10-4d34-98b5-3ad16de57a98                         | | original_resource_id  | ae3659d6-8998-44ae-a494-5248adbebe11 | ... | type                  | instance | | user_id               | a9c935f52e5540fc9befae7f91b4b3ae | +-----------------------+---------------------------------------------------------------------+ Obviously, I am missing something. Where is the missing link? What do I have to do to get CPU usage rates? Do I have to create metrics? Do//I have to ask Ceilometer to create metrics? How? Right now, no instructions seem to exist at all. If that is correct, I would be happy to write documentation once I understand how it works. Thanks a lot. Bernd On 5/10/2019 3:49 PM, info at dantalion.nl wrote: > Hello, > > I am working on Watcher and we are currently changing how metrics are > retrieved from different datasources such as Monasca or Gnocchi. Because > of this major overhaul I would like to validate that everything is > working correctly. > > Almost all of the optimization strategies in Watcher require the cpu > utilization of an instance as metric but with newer versions of > Ceilometer this has become unavailable. > > On IRC I received the information that Gnocchi could be used to > configure an aggregate and this aggregate would then report cpu > utilization, however, I have been unable to find documentation on how to > achieve this. > > I was also notified that cpu_util is something that could be computed > from other metrics. When reading > https://docs.openstack.org/ceilometer/rocky/admin/telemetry-measurements.html#openstack-compute > the documentation seems to agree on this as it states that cpu_util is > measured by using a 'rate of change' transformer. But I have not been > able to find how this can be computed. > > I was hoping someone could spare the time to provide documentation or > information on how this currently is best achieved. > > Kind Regards, > Corne Lukken (Dantali0n) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From renat.akhmerov at gmail.com Wed Jul 31 07:27:37 2019 From: renat.akhmerov at gmail.com (Renat Akhmerov) Date: Wed, 31 Jul 2019 14:27:37 +0700 Subject: [mistral] Office hour in ~30 min In-Reply-To: <94b5cc1d-5740-428f-b4d9-83516d317b82@Spark> References: <94b5cc1d-5740-428f-b4d9-83516d317b82@Spark> Message-ID: <5a439af2-8300-4ac8-be50-b1336682e3dc@Spark> Hi, We’ll have an office hour at 8.00 UTC today (in ~30 min) so you’re welcome to join and chat with us. Thanks Renat Akhmerov @Nokia -------------- next part -------------- An HTML attachment was scrubbed... URL: From stig.openstack at telfer.org Wed Jul 31 07:47:22 2019 From: stig.openstack at telfer.org (Stig Telfer) Date: Wed, 31 Jul 2019 08:47:22 +0100 Subject: [scientific-sig][cloudkitty][monasca] SIG IRC meeting today - accounting using Monasca and CloudKitty Message-ID: Hi All - We have an IRC meeting today at 1100 UTC in channel #openstack-meeting. Everyone is welcome. Today’s agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_July_31st_2019 Today we are looking to gather data on accounting, chargeback and billing, and we have a report on recent investigations using Monasca and CloudKitty together for this purpose. Cheers, Stig -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry at openstack.org Wed Jul 31 08:18:42 2019 From: thierry at openstack.org (Thierry Carrez) Date: Wed, 31 Jul 2019 10:18:42 +0200 Subject: [loci][openstack-helm][tc] Proposal to fold Loci into OpenStack-Helm In-Reply-To: <0FBDA438-3B13-42A1-BD96-F8B30CB27197@openstack.org> References: <0FBDA438-3B13-42A1-BD96-F8B30CB27197@openstack.org> Message-ID: <1989b7a6-e22f-d0bc-3674-0dcf31710152@openstack.org> Chris Hoge wrote: > In the interest of consolidating workload, I'm proposing that we roll the > Loci project under the governance of OpenStack Helm. Loci is a fairly > low-bandwidth project, with the primary consumer of it being OpenStack > Helm. By rolling Loci into OSH, we will be able focus shared efforts and > increase review velocity for the Loci project. > > The architecture of OpenStack Helm allows it to be container build-system > agnostic, and similarly the architecture of Loci allows it to be > deployment tooling agnostic. Neither of these goals will change, as there > are users of both projects that aren't using the other. However, as they > are the largest consumers for each other's work, this is a natural > pairing and both project stand to benefit from a tighter governance > coupling. We already share a good proportion of the development team and > frequently plan PTG sessions to allow for collaboration. That sounds like a good idea, as long as we preserve the agnosticism on both sides. It might require an OSH team mission update to reflect that. -- Thierry Carrez (ttx) From coolsvap at gmail.com Wed Jul 31 08:32:02 2019 From: coolsvap at gmail.com (=?UTF-8?B?yoLKjcmSz4HGnsSvxYIg0p7GsMi0xLfJksqByonJqA==?=) Date: Wed, 31 Jul 2019 14:02:02 +0530 Subject: [all] Pycharm Licences Message-ID: I have renewed the Pycharm licenses for community contributors until July 29, 2020. For everyone who is using it will be updated automatically. Please do not request again for renewal. Best Regards, Swapnil From feilong at catalyst.net.nz Wed Jul 31 09:03:39 2019 From: feilong at catalyst.net.nz (feilong) Date: Wed, 31 Jul 2019 21:03:39 +1200 Subject: [openstack-dev][magnum] Project updates Message-ID: Hi all, Now we're very close to the milestone-2 of Train release. I'd like to take this opportunity to update the project status of Magnum. So far, we have done some great work in this cycle which make Magnum to achieve to a higher level. The first feature I'd like to highlight is the k8s version rolling upgrade. With this feature, user can upgrade their k8s version with zero downtime for their application. Another one is auto scaling, now Magnum can support k8s cluster auto scaling by leveraging the Cluster Autoscaler (there is a OpenStack Magnum driver contributed by CERN team). Besides, Magnum now can support auto repair, aka auto healing, by using a combination of Draino, Cluster Autoscaler and Node Problem Detector. Except above work we have done, there are still a lot of cool features coming: 1. The first one I'd like to mention is the boot volume support. With this feature, user can boot k8s cluster node from volume and user can even set the volume type which is super useful if the cloud provider can support high performance storage, e.g. NVME. And a new useful label named "etcd_volume_type" will be introduced by the way so that user can set the volume type for etcd volume to get a great performance for etcd cluster. 2. Private cluster. With this feature, user can set the network, subnet and floating IP when creating cluster with a public cluster template, which is very handy when user want to use public templates but still want to create the cluster in their existing network. 3. Another super fancy feature is introducing a new auto healing controller except the Draino+CA+NPD combination. Recently, we (Catalyst Cloud) contributed our Magnum auto healer[1] to Cloud-Provider-Openstack repo and we're trying to introduce it to Magnum as another auto healing controller, it would be also very useful for end user to get a zero downtime k8s cluster. As you can see, there are quite a lot of exciting features are being introduced into Magnum, however all of them are done by such a small team. What a cool team! As you know, Kubernetes is still evolving very fast, so we're very keen to get more help from the community. We're working hard to add some new core members to our existing team, but we do need more. In short, if you're interested in provisioning Kubernetes on OpenStack, please try Magnum and feel free to pop up on #openstack-containers for any question. Cheers. -- Cheers & Best regards, Feilong Wang (王飞龙) ------------------------------------------------------ Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang at catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington ------------------------------------------------------ From grant at civo.com Wed Jul 31 09:14:18 2019 From: grant at civo.com (Grant Morley) Date: Wed, 31 Jul 2019 10:14:18 +0100 Subject: Slow instance launch times due to RabbitMQ Message-ID: <0723e410-6029-fcf7-2bd0-8c38e4586cdd@civo.com> Hi all, We are randomly seeing slow instance launch / deletion times and it appears to be because of RabbitMQ. We are seeing a lot of these messages in the logs for Nova and Neutron: ERROR oslo.messaging._drivers.impl_rabbit [-] [f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on 10.6.2.212:5671 is unreachable: Too many heartbeats missed. Trying again in 1 seconds. Client port: 37098: ConnectionForced: Too many heartbeats missed The RabbitMQ cluster isn't under high load and I am not seeing any packets drop over the network when I do some tracing. We are only running 15 compute nodes currently and have >1000 instances so it isn't a large deployment. Are there any good configuration tweaks for RabbitMQ running on OpenStack Queens? Many Thanks, -- Grant Morley Cloud Lead, Civo Ltd www.civo.com | Signup for an account! -------------- next part -------------- An HTML attachment was scrubbed... URL: From donny at fortnebula.com Wed Jul 31 10:37:56 2019 From: donny at fortnebula.com (Donny Davis) Date: Wed, 31 Jul 2019 06:37:56 -0400 Subject: Slow instance launch times due to RabbitMQ In-Reply-To: <0723e410-6029-fcf7-2bd0-8c38e4586cdd@civo.com> References: <0723e410-6029-fcf7-2bd0-8c38e4586cdd@civo.com> Message-ID: What distro/deployer are you using? Donny Davis c: 805 814 6800 On Wed, Jul 31, 2019, 5:17 AM Grant Morley wrote: > Hi all, > > We are randomly seeing slow instance launch / deletion times and it > appears to be because of RabbitMQ. We are seeing a lot of these messages in > the logs for Nova and Neutron: > > ERROR oslo.messaging._drivers.impl_rabbit [-] > [f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on 10.6.2.212:5671 is > unreachable: Too many heartbeats missed. Trying again in 1 seconds. Client > port: 37098: ConnectionForced: Too many heartbeats missed > > The RabbitMQ cluster isn't under high load and I am not seeing any packets > drop over the network when I do some tracing. > > We are only running 15 compute nodes currently and have >1000 instances so > it isn't a large deployment. > > Are there any good configuration tweaks for RabbitMQ running on OpenStack > Queens? > > Many Thanks, > -- > > Grant Morley > Cloud Lead, Civo Ltd > www.civo.com | Signup for an account! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurentfdumont at gmail.com Wed Jul 31 10:49:15 2019 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Wed, 31 Jul 2019 06:49:15 -0400 Subject: Slow instance launch times due to RabbitMQ In-Reply-To: <0723e410-6029-fcf7-2bd0-8c38e4586cdd@civo.com> References: <0723e410-6029-fcf7-2bd0-8c38e4586cdd@civo.com> Message-ID: Could you forward the output of the following commands on a controller node? : rabbitmqctl cluster_status rabbitmqctl list_queues You won't necessarily see a high load on a Rabbit cluster that is in a bad state. On Wed, Jul 31, 2019 at 5:19 AM Grant Morley wrote: > Hi all, > > We are randomly seeing slow instance launch / deletion times and it > appears to be because of RabbitMQ. We are seeing a lot of these messages in > the logs for Nova and Neutron: > > ERROR oslo.messaging._drivers.impl_rabbit [-] > [f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on 10.6.2.212:5671 is > unreachable: Too many heartbeats missed. Trying again in 1 seconds. Client > port: 37098: ConnectionForced: Too many heartbeats missed > > The RabbitMQ cluster isn't under high load and I am not seeing any packets > drop over the network when I do some tracing. > > We are only running 15 compute nodes currently and have >1000 instances so > it isn't a large deployment. > > Are there any good configuration tweaks for RabbitMQ running on OpenStack > Queens? > > Many Thanks, > -- > > Grant Morley > Cloud Lead, Civo Ltd > www.civo.com | Signup for an account! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pawel.konczalski at everyware.ch Wed Jul 31 11:17:37 2019 From: pawel.konczalski at everyware.ch (Pawel Konczalski) Date: Wed, 31 Jul 2019 13:17:37 +0200 Subject: Magnum / Kubernetes: No such image: gcr.io/google_containers/pause:3.0 In-Reply-To: <8c93a364-030c-d0d2-447b-3e737641d24a@everyware.ch> References: <4FFA2395-960B-4DA7-8481-F2AD93EAB500@stackhpc.com> <8c93a364-030c-d0d2-447b-3e737641d24a@everyware.ch> Message-ID: <8983e724-ac4e-6130-f17e-2bbfb09a7e2b@everyware.ch> Hi together, i just try to (re)create a Kubernetes Magnum cluster on a OpenStack installation where those already was working but get recently flowing error in Docker logs: Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0 # openstack coe cluster template create kubernetes-cluster-template \   --image "Fedora AtomicHost 29" \   --external-network public \   --dns-nameserver 8.8.8.8 \   --master-flavor m1.kubernetes \   --flavor m1.kubernetes \   --coe kubernetes \   --volume-driver cinder \   --network-driver flannel \   --docker-volume-size 25 \   --public \   --labels kube_tag=v1.13.4,cloud_provider_tag=1.13.1 # openstack coe cluster create kubernetes-cluster10 \   --cluster-template kubernetes-cluster-template \   --master-count 1 \   --node-count 2 \   --keypair mykey [root at pko-k8s-cluster10-jpojymhgkhdc-master-0 ~]# journalctl -u docker.service -- Logs begin at Wed 2019-07-31 10:02:52 UTC, end at Wed 2019-07-31 10:59:56 UTC. -- Jul 31 10:03:10 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Starting Docker Application Container Engine... Jul 31 10:03:10 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:10.966720103Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.242863167Z" level=info msg="Graph migration to content-addressability took 0.00 seconds" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.243328235Z" level=warning msg="Your kernel does not support cgroup rt period" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.243346111Z" level=warning msg="Your kernel does not support cgroup rt runtime" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.243358575Z" level=warning msg="Your kernel does not support cgroup blkio weight" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.243369525Z" level=warning msg="Your kernel does not support cgroup blkio weight_device" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.243880444Z" level=info msg="Loading containers: start." Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.340931401Z" level=info msg="Firewalld running: false" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.423128319Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.497740931Z" level=info msg="Loading containers: done." Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.813539429Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.822435917Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.823644437Z" level=info msg="Daemon has completed initialization" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.825054441Z" level=info msg="Docker daemon" commit="1185cfd/1.13.1" graphdriver=overlay2 version=1.13.1 Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.903581154Z" level=info msg="API listen on /var/run/docker.sock" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Started Docker Application Container Engine. Jul 31 10:03:13 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:13.886226816Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:03:13 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:13.887560748Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:03:13 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:13.995141066Z" level=error msg="Handler for GET /v1.26/containers/etcd/json returned error: No such container: etcd" Jul 31 10:03:13 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:13.995439344Z" level=error msg="Handler for GET /v1.26/containers/etcd/json returned error: No such container: etcd" Jul 31 10:03:52 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Stopping Docker Application Container Engine... Jul 31 10:03:52 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:52.843904648Z" level=info msg="Processing signal 'terminated'" Jul 31 10:03:52 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Stopped Docker Application Container Engine. Jul 31 10:07:32 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Starting Docker Application Container Engine... Jul 31 10:07:32 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:32.665652883Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found" Jul 31 10:07:32 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:32.841139565Z" level=info msg="devmapper: Creating filesystem xfs on device docker-253:0-8716235-base" Jul 31 10:07:32 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:32.985841976Z" level=info msg="devmapper: Successfully created filesystem xfs on device docker-253:0-8716235-base" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.045750870Z" level=info msg="Graph migration to content-addressability took 0.00 seconds" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.046156219Z" level=warning msg="Your kernel does not support cgroup rt period" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.046176360Z" level=warning msg="Your kernel does not support cgroup rt runtime" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.046189025Z" level=warning msg="Your kernel does not support cgroup blkio weight" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.046199775Z" level=warning msg="Your kernel does not support cgroup blkio weight_device" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.046703801Z" level=info msg="Loading containers: start." Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.053599598Z" level=info msg="Firewalld running: false" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.212814678Z" level=info msg="Loading containers: done." Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.244296246Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.244734418Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.244925171Z" level=info msg="Daemon has completed initialization" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.244951281Z" level=info msg="Docker daemon" commit="1185cfd/1.13.1" graphdriver=devicemapper version=1.13.1 Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Started Docker Application Container Engine. Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.263178384Z" level=info msg="API listen on /var/run/docker.sock" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.663856788Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.664293204Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.679268641Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.679675879Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.744362224Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.744862040Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.761317130Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.761846808Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.779548407Z" level=error msg="Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.779851911Z" level=error msg="Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.794963363Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.795546041Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:12:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:12:33.784829232Z" level=error msg="Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0" Jul 31 10:12:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:12:33.785054762Z" level=error msg="Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0" Jul 31 10:17:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:17:33.788586476Z" level=error msg="Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0" Did somebody know if Google registry has changed recently? BR Pawel -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5227 bytes Desc: not available URL: From mnasiadka at gmail.com Wed Jul 31 11:34:22 2019 From: mnasiadka at gmail.com (=?UTF-8?Q?Micha=C5=82_Nasiadka?=) Date: Wed, 31 Jul 2019 13:34:22 +0200 Subject: [kolla] cancelling meeting today Message-ID: Hello, Since there are no hot topics and the PTL/core team are unavailable for the meeting - it will be cancelled. See you next week on #openstack-kolla -- Michał Nasiadka mnasiadka at gmail.com From amoralej at redhat.com Wed Jul 31 13:13:17 2019 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Wed, 31 Jul 2019 15:13:17 +0200 Subject: [rdo] [packstack] failing to setup Stein with Openvswitch/VXLAN networking In-Reply-To: References: Message-ID: Hi, So, IIUC, your server has a single NIC, eth0, right? Could you provide the configuration file for eth0 before running packstack?, i guess you are using ifcfg files? Best regards, Alfredo On Wed, Jul 31, 2019 at 2:20 AM Bernd Bausch wrote: > Trying to set up a Stein cloud with Packstack. I want the Openvswitch mech > driver and VXLAN type driver. A few weeks ago, the following invocation was > successful: > > sudo packstack --debug --allinone --default-password pw \ > --os-neutron-ovs-bridge-interfaces=br-ex:eth0 \ > --os-neutron-ml2-tenant-network-types=vxlan \ > --os-neutron-ml2-mechanism-drivers=openvswitch \ > --os-neutron-ml2-type-drivers=vxlan,flat \ > --os-neutron-l2-agent=openvswitch \ > --provision-demo-floatrange=10.1.1.0/24 \ > --provision-demo-allocation-pools '["start=10.1.1.10,end=10.1.1.50"]' \ > --os-heat-install=y --os-heat-cfn-install=y > > Now, it fails during network setup. My network connection to the Packstack > server is severed, and it turns out that its only network interface *eth0 > *has no IP address and is down. No bridge exists. > > In the *network.pp.finished *file, I find various *ovs-vsctl *commands > including *add-br*, and a command *ifdown eth0 *which fails with exit > code 1 (no error message from the *ifdown *command is logged). > > *Can somebody recommend the options required to successfully deploy a > Stein cloud based on the Openvswitch and VXLAN drivers?* > > Thanks much, > > Bernd > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From grant at civo.com Wed Jul 31 13:35:11 2019 From: grant at civo.com (Grant Morley) Date: Wed, 31 Jul 2019 14:35:11 +0100 Subject: Slow instance launch times due to RabbitMQ In-Reply-To: References: <0723e410-6029-fcf7-2bd0-8c38e4586cdd@civo.com> Message-ID: <1c6078ab-bc1d-7a84-cc87-1b470235ccc0@civo.com> Hi guys, We are using Ubuntu 16 and OpenStack ansible to do our setup. rabbitmqctl list_queues Listing queues (Doesn't appear to be any queues ) rabbitmqctl cluster_status Cluster status of node 'rabbit at management-1-rabbit-mq-container-b4d7791f' [{nodes,[{disc,['rabbit at management-1-rabbit-mq-container-b4d7791f', 'rabbit at management-2-rabbit-mq-container-b455e77d', 'rabbit at management-3-rabbit-mq-container-1d6ae377']}]},  {running_nodes,['rabbit at management-3-rabbit-mq-container-1d6ae377', 'rabbit at management-2-rabbit-mq-container-b455e77d', 'rabbit at management-1-rabbit-mq-container-b4d7791f']},  {cluster_name,<<"openstack">>},  {partitions,[]},  {alarms,[{'rabbit at management-3-rabbit-mq-container-1d6ae377',[]},           {'rabbit at management-2-rabbit-mq-container-b455e77d',[]}, {'rabbit at management-1-rabbit-mq-container-b4d7791f',[]}]}] Regards, On 31/07/2019 11:49, Laurent Dumont wrote: > Could you forward the output of the following commands on a controller > node? : > > rabbitmqctl cluster_status > rabbitmqctl list_queues > > You won't necessarily see a high load on a Rabbit cluster that is in a > bad state. > > On Wed, Jul 31, 2019 at 5:19 AM Grant Morley > wrote: > > Hi all, > > We are randomly seeing slow instance launch / deletion times and > it appears to be because of RabbitMQ. We are seeing a lot of these > messages in the logs for Nova and Neutron: > > ERROR oslo.messaging._drivers.impl_rabbit [-] > [f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on > 10.6.2.212:5671 is unreachable: Too many > heartbeats missed. Trying again in 1 seconds. Client port: 37098: > ConnectionForced: Too many heartbeats missed > > The RabbitMQ cluster isn't under high load and I am not seeing any > packets drop over the network when I do some tracing. > > We are only running 15 compute nodes currently and have >1000 > instances so it isn't a large deployment. > > Are there any good configuration tweaks for RabbitMQ running on > OpenStack Queens? > > Many Thanks, > > -- > > Grant Morley > Cloud Lead, Civo Ltd > www.civo.com | Signup for an account! > > -- Grant Morley Cloud Lead, Civo Ltd www.civo.com | Signup for an account! -------------- next part -------------- An HTML attachment was scrubbed... URL: From thuanlk at viettel.com.vn Tue Jul 30 04:21:32 2019 From: thuanlk at viettel.com.vn (thuanlk at viettel.com.vn) Date: Tue, 30 Jul 2019 11:21:32 +0700 (ICT) Subject: [neutron] OpenvSwitch firewall sctp getting dropped In-Reply-To: References: <000001d54623$a91ae750$fb50b5f0$@viettel.com.vn> Message-ID: <029d01d5468e$445948a0$cd0bd9e0$@viettel.com.vn> I have tried config SCTP but nothing change! openstack security group rule create --ingress --remote-ip 0.0.0.0/0 --protocol 132 --dst-port 2000:10000 --description "SCTP" sctp openstack security group rule create --egress --remote-ip 0.0.0.0/0 --protocol 132 --dst-port 2000:10000 --description "SCTP" sctp Displaying 2 items Direction Ether Type IP Protocol Port Range Remote IP Prefix Remote Security Group Actions Egress IPv4 132 2000 - 10000 0.0.0.0/0 - Ingress IPv4 132 2000 - 10000 0.0.0.0/0 - Thanks and best regards ! --------------------------------------- Lăng Khắc Thuận OCS Cloud | OCS (VTTEK) +(84)- 966463589 -----Original Message----- From: smooney at redhat.com [mailto:smooney at redhat.com] Sent: Tuesday, July 30, 2019 1:27 AM To: thuanlk at viettel.com.vn; openstack-discuss at lists.openstack.org Subject: Re: [neutron] OpenvSwitch firewall sctp getting dropped On Mon, 2019-07-29 at 22:38 +0700, thuanlk at viettel.com.vn wrote: > I have installed Openstack Queens on CentOs 7 with OvS and I recently > used the native openvswitch firewall to implement SecusiryGroup. The > native OvS firewall seems to work just fine with TCP/UDP traffic but > it does not forward any SCTP traffic going to the VMs no matter how I > change the security groups, But it run if i disable port security > completely or use iptables_hybrid firewall driver. What do I have to > do to allow SCTP packets to reach the VMs? the security groups api is a whitelist model so all traffic is droped by default. if you want to allow sctp you would ihave to create an new security group rule with ip_protocol set to the protocol number for sctp. e.g. openstack security group rule create --protocol sctp ... im not sure if neutron support --dst-port for sctp but you can still filter on --remote-ip or --remote-group and can specify the rule as an --ingress or --egress rule as normal. https://docs.openstack.org/python-openstackclient/stein/cli/command-objects/security-group-rule.html based on this commit https://github.com/openstack/neutron/commit/f711ad78c5c0af44318c6234957590c91592b984 it looks like neutron now validates the prot ranges for sctp impligying it support setting them so i gues its just a gap in the documentation. > From ondrej.duchon at ultimum.io Wed Jul 31 13:58:15 2019 From: ondrej.duchon at ultimum.io (=?UTF-8?B?T25kxZllaiBEdWNob8WI?=) Date: Wed, 31 Jul 2019 15:58:15 +0200 Subject: [dev][keystone][keystonemiddleware] Keystone CADF notifications for rabbitmq Message-ID: Hello, I am trying to make to work CADF notifying for nova_api and other modules into rabbitmq. I tried to use keystonemiddleware ( https://docs.openstack.org/keystonemiddleware/queens/audit.html), but no notifications appeared. The only thing that is sending CADF notifications into rabbitmq is keystone. I assume that is because keystone has its own notifier. (keystone/notifications.py) Do you have some idea about this problem? Which approach is correct in this case? Thank you for your help, Ondrej -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurentfdumont at gmail.com Wed Jul 31 14:19:54 2019 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Wed, 31 Jul 2019 10:19:54 -0400 Subject: Slow instance launch times due to RabbitMQ In-Reply-To: <1c6078ab-bc1d-7a84-cc87-1b470235ccc0@civo.com> References: <0723e410-6029-fcf7-2bd0-8c38e4586cdd@civo.com> <1c6078ab-bc1d-7a84-cc87-1b470235ccc0@civo.com> Message-ID: That is a bit strange, list_queues should return stuff. Couple of ideas : - Are the Rabbit connection failure logs on the compute pointing to a specific controller? - Are there any logs within Rabbit on the controller that would point to a transient issue? - cluster_status is a snapshot of the cluster at the time you ran the command. If the alarms have cleared, you won't see anything. - If you have the RabbitMQ management plugin activated, I would recommend a quick look to see the historical metrics and overall status. On Wed, Jul 31, 2019 at 9:35 AM Grant Morley wrote: > Hi guys, > > We are using Ubuntu 16 and OpenStack ansible to do our setup. > > rabbitmqctl list_queues > Listing queues > > (Doesn't appear to be any queues ) > > rabbitmqctl cluster_status > > Cluster status of node 'rabbit at management-1-rabbit-mq-container-b4d7791f' > [{nodes,[{disc,['rabbit at management-1-rabbit-mq-container-b4d7791f', > 'rabbit at management-2-rabbit-mq-container-b455e77d', > 'rabbit at management-3-rabbit-mq-container-1d6ae377']}]}, > {running_nodes,['rabbit at management-3-rabbit-mq-container-1d6ae377', > 'rabbit at management-2-rabbit-mq-container-b455e77d', > 'rabbit at management-1-rabbit-mq-container-b4d7791f']}, > {cluster_name,<<"openstack">>}, > {partitions,[]}, > {alarms,[{'rabbit at management-3-rabbit-mq-container-1d6ae377',[]}, > {'rabbit at management-2-rabbit-mq-container-b455e77d',[]}, > {'rabbit at management-1-rabbit-mq-container-b4d7791f',[]}]}] > > Regards, > On 31/07/2019 11:49, Laurent Dumont wrote: > > Could you forward the output of the following commands on a controller > node? : > > rabbitmqctl cluster_status > rabbitmqctl list_queues > > You won't necessarily see a high load on a Rabbit cluster that is in a bad > state. > > On Wed, Jul 31, 2019 at 5:19 AM Grant Morley wrote: > >> Hi all, >> >> We are randomly seeing slow instance launch / deletion times and it >> appears to be because of RabbitMQ. We are seeing a lot of these messages in >> the logs for Nova and Neutron: >> >> ERROR oslo.messaging._drivers.impl_rabbit [-] >> [f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on 10.6.2.212:5671 is >> unreachable: Too many heartbeats missed. Trying again in 1 seconds. Client >> port: 37098: ConnectionForced: Too many heartbeats missed >> >> The RabbitMQ cluster isn't under high load and I am not seeing any >> packets drop over the network when I do some tracing. >> >> We are only running 15 compute nodes currently and have >1000 instances >> so it isn't a large deployment. >> >> Are there any good configuration tweaks for RabbitMQ running on OpenStack >> Queens? >> >> Many Thanks, >> -- >> >> Grant Morley >> Cloud Lead, Civo Ltd >> www.civo.com | Signup for an account! >> > -- > > Grant Morley > Cloud Lead, Civo Ltd > www.civo.com | Signup for an account! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafaelweingartner at gmail.com Wed Jul 31 14:32:30 2019 From: rafaelweingartner at gmail.com (=?UTF-8?Q?Rafael_Weing=C3=A4rtner?=) Date: Wed, 31 Jul 2019 11:32:30 -0300 Subject: [dev][keystone][keystonemiddleware] Keystone CADF notifications for rabbitmq In-Reply-To: References: Message-ID: Could you share your oslo_messaging section in keystone.conf? BTW: are you suing Kolla-ansible to deploy your OpenStack environment? On Wed, Jul 31, 2019 at 11:04 AM Ondřej Duchoň wrote: > Hello, > > I am trying to make to work CADF notifying for nova_api and other modules > into rabbitmq. I tried to use keystonemiddleware ( > https://docs.openstack.org/keystonemiddleware/queens/audit.html), but no > notifications appeared. The only thing that is sending CADF notifications > into rabbitmq is keystone. I assume that is because keystone has its own > notifier. (keystone/notifications.py) > > Do you have some idea about this problem? Which approach is correct in > this case? > > Thank you for your help, > Ondrej > -- Rafael Weingärtner -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianyrchoi at gmail.com Wed Jul 31 14:39:26 2019 From: ianyrchoi at gmail.com (Ian Y. Choi) Date: Wed, 31 Jul 2019 23:39:26 +0900 Subject: [User-committee] [all][uc] Train UC Election Season In-Reply-To: References: Message-ID: <38f2f319-f744-38b3-1947-37d703866779@gmail.com> + openstack-discuss mailing list for broader announce of Train UC Election :) - Ed & Ian Ed Leafe wrote on 7/31/2019 11:20 PM: > Hello All! > > The User Committee (UC) election is coming! > > The OpenStack User Committee [1] is one of the governing bodies of the OpenStack project to serve the user community of OpenStack, representing them to the broader community. > > If you are not aware of UC election, please see the details first: > > Election details: > https://governance.openstack.org/uc/reference/uc-election-aug2019.html > > Please read the stipulations and timelines for candidates and electorate contained in this governance documentation. Note that the nomination period begins soon! (August 5). > > There will be further announcements posted to the mailing list as action is required from the electorate or candidates. This email is for information purposes only. > > If you have any questions which you feel affect others please reply to this email thread. > > If you have any questions that you which to discuss in private please email any of the election officials [2] so that we may address your concerns. > > Thank you, > > - Ed & Ian > > [1] https://governance.openstack.org/uc/reference/charter.html > [2] https://governance.openstack.org/uc/reference/uc-election-aug2019.html#officials > > > _______________________________________________ > User-committee mailing list > User-committee at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee From thierry at openstack.org Wed Jul 31 14:55:52 2019 From: thierry at openstack.org (Thierry Carrez) Date: Wed, 31 Jul 2019 16:55:52 +0200 Subject: [tripleo][charms][helm][kolla][ansible][puppet][chef] Deployment tools capabilities v0.1.0 In-Reply-To: References: Message-ID: <1ae16474-33a6-396d-a5c6-b45f662f18b5@openstack.org> Quick reminder as we are still waiting for capabilities for: - openstack-helm - openstack-charms - chef-openstack - LOCI - RPM-packaging We'd like to have all OpenStack community tools covered before publishing that data. Let me know if you have questions or need help. Thanks in advance, Thierry Carrez wrote: > Hi, deployment tools teams, > > As mentioned here last month[1], we are working to improve the > information present on the deployment tools pages on the OpenStack > website, and we need your help! > > After the Forum session on this topic in Denver[2], a workgroup worked > on producing a set of base capabilities that can be asserted by the > various deployment tools we have. You can find version 0.1.0 of those > capabilities here: > > https://opendev.org/osf/openstack-map/src/branch/master/deployment_tools_capabilities.yaml > > > As an example, I pushed a change that makes every deployment tool assert > the capability to deploy keystone ("components:keystone" tag) at: > > https://review.opendev.org/#/c/669648/ > > Now it's your turn. Please have a look at the list of the capabilities > above, and propose a change to add those that are relevant to your > deployment tool in the following file: > > https://opendev.org/osf/openstack-map/src/branch/master/deployment_tools.yaml > > > Capabilities are all of the form "category:tag" (components:keystone, > starts-from:os-installed, technology:puppet...). Once all deployment > projects have completed that task, we'll add the capabilities to the > rendered page on the website and allow for basic searching for tools > with matching capability. > > Now, capabilities go only so far in describing your deployment tool. I > also encourage you to improve in the same file the "desc" field: that > one is directly displayed on the site.Uuse it to describe in more > details how your deployment tool actually works and what makes it > unique, beyond basic capabilities tags. > > Please feel free to use this thread (or personal email) if you have > questions on this. And thanks in advance for your help! > > [1] > http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006964.html > > [2] https://etherpad.openstack.org/p/DEN-deployment-tools-capabilities > -- Thierry Carrez (ttx) From mnaser at vexxhost.com Wed Jul 31 15:48:08 2019 From: mnaser at vexxhost.com (Mohammed Naser) Date: Wed, 31 Jul 2019 11:48:08 -0400 Subject: [openstack-ansible] office hours update Message-ID: Hey everyone, Here is the update of what happened during the OpenStack Ansible office hours this week. We thought there was an issue with the logic that determines nova_cinder_rbd_inuse running on localhost and setting the fact only on localhost (and not the host that’s it’s running it on). So the variable ‘nova_cinder_rbd_inuse’ was always set to false on these hosts. The issue was a missing rbd configuration in nova.conf and that issue has been fixed. Thanks for tuning in and thanks for Tiffanie for taking care of the basic meeting logistics! Regards, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser at vexxhost.com W. http://vexxhost.com From g.santomaggio at gmail.com Wed Jul 31 15:48:57 2019 From: g.santomaggio at gmail.com (Gabriele Santomaggio) Date: Wed, 31 Jul 2019 15:48:57 +0000 Subject: Slow instance launch times due to RabbitMQ In-Reply-To: References: <0723e410-6029-fcf7-2bd0-8c38e4586cdd@civo.com> <1c6078ab-bc1d-7a84-cc87-1b470235ccc0@civo.com>, Message-ID: Hi, Are you using ssl connections ? Can be this issue ? https://bugs.launchpad.net/ubuntu/+source/oslo.messaging/+bug/1800957 ________________________________ From: Laurent Dumont Sent: Wednesday, July 31, 2019 4:20 PM To: Grant Morley Cc: openstack-operators at lists.openstack.org Subject: Re: Slow instance launch times due to RabbitMQ That is a bit strange, list_queues should return stuff. Couple of ideas : * Are the Rabbit connection failure logs on the compute pointing to a specific controller? * Are there any logs within Rabbit on the controller that would point to a transient issue? * cluster_status is a snapshot of the cluster at the time you ran the command. If the alarms have cleared, you won't see anything. * If you have the RabbitMQ management plugin activated, I would recommend a quick look to see the historical metrics and overall status. On Wed, Jul 31, 2019 at 9:35 AM Grant Morley > wrote: Hi guys, We are using Ubuntu 16 and OpenStack ansible to do our setup. rabbitmqctl list_queues Listing queues (Doesn't appear to be any queues ) rabbitmqctl cluster_status Cluster status of node 'rabbit at management-1-rabbit-mq-container-b4d7791f' [{nodes,[{disc,['rabbit at management-1-rabbit-mq-container-b4d7791f', 'rabbit at management-2-rabbit-mq-container-b455e77d', 'rabbit at management-3-rabbit-mq-container-1d6ae377']}]}, {running_nodes,['rabbit at management-3-rabbit-mq-container-1d6ae377', 'rabbit at management-2-rabbit-mq-container-b455e77d', 'rabbit at management-1-rabbit-mq-container-b4d7791f']}, {cluster_name,<<"openstack">>}, {partitions,[]}, {alarms,[{'rabbit at management-3-rabbit-mq-container-1d6ae377',[]}, {'rabbit at management-2-rabbit-mq-container-b455e77d',[]}, {'rabbit at management-1-rabbit-mq-container-b4d7791f',[]}]}] Regards, On 31/07/2019 11:49, Laurent Dumont wrote: Could you forward the output of the following commands on a controller node? : rabbitmqctl cluster_status rabbitmqctl list_queues You won't necessarily see a high load on a Rabbit cluster that is in a bad state. On Wed, Jul 31, 2019 at 5:19 AM Grant Morley > wrote: Hi all, We are randomly seeing slow instance launch / deletion times and it appears to be because of RabbitMQ. We are seeing a lot of these messages in the logs for Nova and Neutron: ERROR oslo.messaging._drivers.impl_rabbit [-] [f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on 10.6.2.212:5671 is unreachable: Too many heartbeats missed. Trying again in 1 seconds. Client port: 37098: ConnectionForced: Too many heartbeats missed The RabbitMQ cluster isn't under high load and I am not seeing any packets drop over the network when I do some tracing. We are only running 15 compute nodes currently and have >1000 instances so it isn't a large deployment. Are there any good configuration tweaks for RabbitMQ running on OpenStack Queens? Many Thanks, -- [https://www.civo.com/images/email-logo.jpg] Grant Morley Cloud Lead, Civo Ltd www.civo.com | Signup for an account! -- [https://www.civo.com/images/email-logo.jpg] Grant Morley Cloud Lead, Civo Ltd www.civo.com | Signup for an account! -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Wed Jul 31 17:10:49 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 31 Jul 2019 17:10:49 +0000 Subject: [openstack-dev][magnum] Project updates In-Reply-To: References: Message-ID: <20190731171049.gayatjbtjvgxya25@yuggoth.org> On 2019-07-31 21:03:39 +1200 (+1200), feilong wrote: [...] > So far, we have done some great work in this cycle which make > Magnum to achieve to a higher level. [...] This is all great stuff, thanks for the update! > Kubernetes is still evolving very fast [...] On that note, the Stein release announcement[0] mentioned that Magnum was a "Certified Kubernetes Installer." I don't see it listed on the Kubernetes Conformance site[1] now, but it was apparently still on there as recently as early July[2]. It seemed like this was a big deal at one point, but wasn't kept up. Is there any interest from the Magnum maintainers in adding support for recent versions of Kubernetes and reacquiring that certification in time for the Train release? [0] https://www.openstack.org/software/stein/ [1] https://www.cncf.io/certification/software-conformance/ [2] https://web.archive.org/web/20190705004545/https://www.cncf.io/certification/software-conformance/ -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From kennelson11 at gmail.com Wed Jul 31 18:21:03 2019 From: kennelson11 at gmail.com (Kendall Nelson) Date: Wed, 31 Jul 2019 11:21:03 -0700 Subject: =?UTF-8?Q?=E2=80=8B=5Brelease=5D_=28a_bit_belated=29_Release_countdown_for_w?= =?UTF-8?Q?eek_R=2D11=2C_July_29_=2D_August_2?= Message-ID: Hello Everyone! Development Focus ----------------- We are now past the Train-2 milestone, and entering the last development phase of the cycle. Teams should be focused on implementing planned work for the cycle. Now is a good time to review those plans and reprioritize anything if needed based on the what progress has been made and what looks realistic to complete in the next few weeks. General Information ------------------- The following cycle-with-intermediary deliverables have not done any intermediary release yet during this cycle. The cycle-with-rc release model is more suited for deliverables that plan to be released only once per cycle. As a result, we will be proposing to change the release model for the following deliverables: blazar-dashboard cloudkitty-dashboard ec2-api freezer-web-ui freezer heat-agents heat-dashboard ironic-ui karbor-dashboard karbor kuryr-kubernetes magnum-ui manila-ui masakari-dashboard monasca-agent monasca-api monasca-ceilometer monasca-events-api monasca-kibana-plugin monasca-log-api monasca-notification monasca-persister monasca-thresh monasca-transform monasca-ui murano-agent networking-baremetal networking-generic-switch networking-hyperv neutron-fwaas-dashboard neutron-vpnaas-dashboard requirements sahara-extra senlin-dashboard solum-dashboard tacker-horizon tricircle vitrage-dashboard vitrage watcher-dashboard PTLs and release liaisons for each of those deliverables can either +1 the release model change when we get them pushed, or propose an intermediary release for that deliverable. In absence of answer by the start of R-10 week we'll consider that the switch to cycle-with-rc is preferable. Upcoming Deadlines & Dates -------------------------- Non-client library freeze: September 05 (R-6 week) Client library freeze: September 12 (R-5 week) Train-3 milestone: September 12 (R-5 week) -Kendall (diablo_rojo) + the Release Management Team -------------- next part -------------- An HTML attachment was scrubbed... URL: From feilong at catalyst.net.nz Wed Jul 31 18:45:30 2019 From: feilong at catalyst.net.nz (feilong) Date: Thu, 1 Aug 2019 06:45:30 +1200 Subject: [openstack-dev][magnum] Project updates In-Reply-To: <20190731171049.gayatjbtjvgxya25@yuggoth.org> References: <20190731171049.gayatjbtjvgxya25@yuggoth.org> Message-ID: <45d4effd-ae55-cfe4-60dd-3635d7558eb5@catalyst.net.nz> On 1/08/19 5:10 AM, Jeremy Stanley wrote: > On 2019-07-31 21:03:39 +1200 (+1200), feilong wrote: > [...] >> So far, we have done some great work in this cycle which make >> Magnum to achieve to a higher level. > [...] > > This is all great stuff, thanks for the update! > >> Kubernetes is still evolving very fast > [...] > > On that note, the Stein release announcement[0] mentioned that > Magnum was a "Certified Kubernetes Installer." I don't see it listed > on the Kubernetes Conformance site[1] now, but it was apparently > still on there as recently as early July[2]. It seemed like this was > a big deal at one point, but wasn't kept up. Is there any interest > from the Magnum maintainers in adding support for recent versions of > Kubernetes and reacquiring that certification in time for the Train > release? TBH, I don't know why it's removed from the list and I didn't get any notice about that. But now I'm working with Chris to get it get. Thanks for reminding. > [0] https://www.openstack.org/software/stein/ > [1] https://www.cncf.io/certification/software-conformance/ > [2] https://web.archive.org/web/20190705004545/https://www.cncf.io/certification/software-conformance/ -- Cheers & Best regards, Feilong Wang (王飞龙) ------------------------------------------------------ Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang at catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington ------------------------------------------------------ From tim at swiftstack.com Wed Jul 31 18:57:23 2019 From: tim at swiftstack.com (tim at swiftstack.com) Date: Wed, 31 Jul 2019 11:57:23 -0700 Subject: [swift] Shanghai PTG attendance and etherpad Message-ID: <235fd45c8fe538a735147eab64e477ad167f1810.camel@swiftstack.com> 你好! Most teams sent out emails two or three weeks ago with etherpads to start estimating headcount and coming up with discussion topics. I was a little slow, but here's Swift's: https://etherpad.openstack.org/p/swift-ptg-shanghai We're thinking of doing something a little different from our previous PTGs and hackathons and want to offer a kind of extended onboarding session. If you're new to Swift, this is a great opportunity to learn more about - how Swift works, - how to write applications against Swift, - how to deploy and operate your own Swift cluster, or - how to do upstream development for Swift. Interested? We'd love to hear from you and meet you! If there are any other aspects of Swift you'd like to learn about or dive deeper on, feel free to add them! I'll try to develop some materials based on people's level of interest so it won't all be off-the-cuff :-) Of course, there will also be more-traditional design discussions going on. (I don't think I could stop this if I tried -- it seems *any time* there are three or more Swift contributors around, there are going to be design discussions happening!) If you know you want to talk about some upcoming features or other work, please update the Development Topics at the bottom. 在上海见到您们! Tim From mriedemos at gmail.com Wed Jul 31 21:25:15 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Wed, 31 Jul 2019 16:25:15 -0500 Subject: [all] Pycharm Licences In-Reply-To: References: Message-ID: <5e2439bf-0950-a149-125b-906725eb5dd1@gmail.com> On 7/31/2019 3:32 AM, ʂʍɒρƞįł Ҟưȴķɒʁʉɨ wrote: > I have renewed the Pycharm licenses for community contributors until > July 29, 2020. For everyone who is using it will be updated > automatically. Please do not request again for renewal. > > Best Regards, > Swapnil > Thanks for doing this Swapnil. -- Thanks, Matt From openstack at fried.cc Wed Jul 31 22:25:35 2019 From: openstack at fried.cc (Eric Fried) Date: Wed, 31 Jul 2019 17:25:35 -0500 Subject: [dev][nova][glance][cinder] introducing 'compressed' image container_format In-Reply-To: References: Message-ID: > Our preliminary testing with a raw image with container_format == > compressed, disk_format == raw is that Nova boots a VM from the image > but the VM appears to be unusable. Nova seems taking it as a real RAW > image as disk_format indicates. It looks like [3] has been introduced to fail the instance creation, which seems like a legit first step. > So we want to open a discussion about handling this. Is there a sense of what it would take for nova to support compressed images? We're past spec freeze, but if the impact is small... efried [3] https://review.opendev.org/#/c/673407/