April 2019 - openstack-discuss - lists.openstack.org

[tripleo] Canceling IRC meeting on April 30, 2019
by Alex Schultz 16 Apr '19

16 Apr '19

Due to the summit and PTG, most folks won't be around so we'll cancel the meeting for that week. We will resume the weekly IRC meeting on May 7, 2019. Thanks, -Alex

1 0

[ironic] Cancelling meetings April 22nd through May 6th, resuming May 13th
by Julia Kreger 16 Apr '19

16 Apr '19

Greeting fellow practitioners of irony! I meant to raise this question during our weekly meeting yesterday, but simply forgot. Some discussion on irc[0] today has reached a consensus that we should just cancel the next few meetings. * This next week, April 22nd, out meeting conflicts with holidays for a number of contributors. * The week after, April 29th, will be during the Open Infrastructure Summit. Historically we cancel the meeting on the week following such a gathering to give contributors time to re-group. We will be resuming meetings on May 13th. As always you can find ironic contributors in #openstack-ironic. Review priorities should be sourced from the whiteboard[1] using the "This Week's Priorities" section.. Thanks everyone! -Julia [0]: http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/%23openstack-iro… [1]: https://etherpad.openstack.org/p/IronicWhiteBoard

1 0

Re: [stackalytics] Reported numbers seem inaccurate
by Jeremy Stanley 16 Apr '19

16 Apr '19

On 2019-04-11 18:21:13 UTC (+0000), Mark Goddard wrote: > I've heard a couple of people say they don't feel the numbers > reported by Stackalytics accurately reflect reality. I've been > trying to gather a few stats for my Kolla update session in > Denver, and am finding the same. [...] I'm not going to attempt to debate the value of the Stackalytics site, but given the maintainers seem to have set their sights beyond OpenStack lately they're probably not following discussions on the openstack-discuss mailing list (their lack of replies here after nearly a week would indicate this as well). According to the readme[*] for the project, they seem to prefer bug reports via Launchpad so that may be a more effective place to follow up on this concern if there is a desire for continued use of Stackalytics by the OpenStack community. That said, someone already seems to have reported[**] the problem back in January and the bug hasn't been triaged yet, but maybe additional mee-toos will up their urgency? [*] https://opendev.org/openstack/stackalytics/src/branch/master/README.rst#pro… [**] https://launchpad.net/bugs/1811028 -- Jeremy Stanley

1 0

[nova][scheduler] scheduler spawns to the same compute node only
by Nicolas Ghirlanda 16 Apr '19

16 Apr '19

Hi all, we have a kolla-ansible deployed "Queens" Release of openstack with 8 compute nodes and an external Percona XtraDB Cluster (with read-write split with haproxy). New VMs are just currently always scheduled to the same compute node, even though a manual live-migration is working fine to other compute nodes. We're not sure, what the issue is, but perhaps someone may spot it from our config: # nova.conf scheduler config default_availability_zone = az1 ... [filter_scheduler] available_filters = nova.scheduler.filters.all_filters enabled_filters = RetryFilter, AvailabilityZoneFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, AggregateInstanceExtraSpecsFilter, AggregateMultiTenancyIsolation, DifferentHostFilter, RamFilter, SameHostFilter, NUMATopologyFilter Database is an external Percona XtraDB Cluster (Version 5.7.24) with haproxy for read-write-splitting (currently only one write node). We do see mysql errors in the nova-scheduler.log on the write DB node when an instance is created. 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status: OperationalError: (pymysql.err.OperationalError) (1213, u'WSREP detected deadlock/conflict and aborted the transaction. Try restarting the transaction') (Background on this error at: http://sqlalche.me/e/e3q8) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/objects/service.py", line 397, in save 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/db/api.py", line 183, in service_update 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self.force_reraise() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 227, in wrapped 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self.gen.next() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1043, in _transaction_scope 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db yield resource 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self.gen.next() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 653, in _session 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self.session.rollback() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self.force_reraise() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 650, in _session 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self._end_session_transaction(self.session) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 678, in _end_session_transaction 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db session.commit() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 943, in commit 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self.transaction.commit() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 471, in commit 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db t[1].commit() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1643, in commit 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self._do_commit() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1674, in _do_commit 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self.connection._commit_impl() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 726, in _commit_impl 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self._handle_dbapi_exception(e, None, None, None, None) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 724, in _commit_impl 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self.engine.dialect.do_commit(self.connection) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 1765, in do_commit 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db dbapi_connection.commit() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 422, in commit 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db self._read_ok_packet() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 396, in _read_ok_packet 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db pkt = self._read_packet() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db packet.check_error() 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db err.raise_mysql_exception(self._data) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db raise errorclass(errno, errval) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db OperationalError: (pymysql.err.OperationalError) (1213, u'WSREP detected deadlock/conflict and aborted the transaction. Try restarting the transaction') (Background on this error at: http://sqlalche.me/e/e3q8) 2019-04-15 16:52:10.016 24 ERROR nova.servicegroup.drivers.db 2019-04-15 16:52:20.020 24 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status. The deadlock message is quite strange, as we have haproxy configured so all write requests are handled by one node. There are NO errors in the mysqld.log WHILE creating an instance, but we see from time to time aborted connections from nova. 2019-04-15T14:22:36.232108Z 30616972 [Note] Aborted connection 30616972 to db: 'nova' user: 'nova' host: '10.x.y.z' (Got an error reading communication packets) As I said, all instances are allocated to the same compute node. nova-compute.log doesn't show an error while creating the instance. Beside that, we also see messages from nova.scheduler.host_manager on all other nodes like (but those messages are _not_ triggered, when an instance is spawned.!) 2019-04-15 16:28:47.771 22 INFO nova.scheduler.host_manager [req-f92e340e-a88a-44a0-8cad-588390c25bc2 - - - - -] The instance sync for host 'xxx' did not match. Re-created its InstanceList. Don't know if that may be relevant, but somehow our (currently single) AZ is listed several times. # openstack availability zone list +------------+-------------+ | Zone Name | Zone Status | +------------+-------------+ | internal | available | | az1 | available | | az1 | available | | az1 | available | | az1 | available | +------------+-------------+ May that be related somehow? Thanks for any consideration and support! kind regards Nicolas --

4 7

[taas] stein
by Takashi Yamamoto 16 Apr '19

16 Apr '19

hi, i'll branch and release it sooner or later. probably in this week, or maybe the next one. wrt dashboard, i'm not sure if the repo is in a ready-to-release state. have anyone tested it recently?

1 1

[ceilometer] Missing some notifications
by Florian Engelmann 16 Apr '19

16 Apr '19

Hi, I am not sure if this problem is related to ceilometer or rabbitmq but it looks like our ceilometer notification agent is missing some notifications. Creating, eg. 10 cinder volumes in a row results in some of those volumes are not stored in gnocchi. "openstack metric resource show <ID>" results in a 404 error. Attaching the volume and detaching it again sometimes helps to "create" it in gnocchi. We do use gnocchi for billing. So missing volumes is very bad. Any recommendation on how to debug this problem? Thank you very much, flo

1 0

[scientific-sig] Meeting tonight: Ironic and Data Center Inventory, Denver Summit
by Stig Telfer 16 Apr '19

16 Apr '19

Hi All - We have a Scientific SIG IRC meeting today at 2100 UTC in channel #openstack-meeting. Everyone is welcome. Today's agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_April_16th_2019 <https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_April_16th_2019> We’d like to continue the discussion started last week (in the other timezone) on surveying data center inventory management systems used in research computing, and how those might be better integrated with Ironic. We also have preparations for the Denver summit to discuss. Cheers, Stig

1 0

[tc][all] Dropping py35 testing
by Ghanshyam Mann 15 Apr '19

15 Apr '19

Hello Everyone, All the integration testing has been moved to Bionic[1] now and as discussed in the TC meeting[2], we are good to drop the py35 testing. Train: Good to drop. I am going to push the patches to remove the py35 jobs from the master (which is Train now). Stable/Stein: Ok to drop if the project wants. Stable/Stein gate is also moved to Bionic so we are good to drop py35 from stable/stein too. But backporting all the patches will be too many patches for projects to merge on stable/stein so we discussed not to propose all of them to stable/stein. If the project really wants to drop it from stable/stein then, they can backport. NOTE: Till stable/rocky, we should keep testing py35. If your project is branchless like Tempest and gate the same jobs for stable/rocky or older then, you should keep testing the py35 until stable/rocky EOL. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004647.ht… [2] http://eavesdrop.openstack.org/meetings/tc_python3/2019/tc_python3.2019-03-… http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003610.ht… -gmann

7 12

[telemetry] Vote for team meeting datetime
by Trinh Nguyen 15 Apr '19

15 Apr '19

Hi team, There're many things we need to discuss for the Train cycle so I propose we have a bi-weekly meeting and the first one would be on May 2 (next two weeks will be the vacation of most of the people). Please cast your vote for an appropriate time: https://doodle.com/poll/8akawam8yk38q2hh Yours, -- *Trinh Nguyen* *www.edlab.xyz <https://www.edlab.xyz>*

2 6

[kuryr] [kuryr-kubernetes] Unable to use kubernetes-related things after rebooting host
by Jaewook Oh 15 Apr '19

15 Apr '19

Hello kuryr-kubernetes team, currently, I deployed openstack with devstack, and I added "kuryr-kubernetes" to use kubernetes vim. At first it worked fine and no error was occurred, so I made some c-vnfs with kuryr-kubernetes. But after rebooting the host, the systems that worked well before are not working normally at all. First, I deployed an example one ('redis') with kubectl command, (Not using Tacker, but only using kubectl cli) it showed below error, Events: │obeTime":null,"lastTransitionTime":"2019-04-15T02:28:31Z"}],"hostIP":"192.168.1.20","startTime":"2019-04-15T02:28:31Z" Type Reason Age From Message │,"containerStatuses":[{"name":"web-server","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":f ---- ------ ---- ---- ------- │alse,"restartCount":0,"image":"kyleoh95/iistrc-k8s-demo2:0.4","imageID":""}],"qosClass":"Guaranteed"}}]} Normal Scheduled 2m25s default-scheduler Successfully assigned default/redis-master-6fbbc44567-ms8n│Apr 15 11:36:52 master tacker-server[1881]: from (pid=1881) request /usr/local/lib/python2.7/dist-packages/kubernetes k to master │/client/rest.py:219 Warning MissingClusterDNS 2m25s kubelet, master pod: "redis-master-6fbbc44567-ms8nk_default(8a6f9079-5f27-│Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.633 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne 11e9-9b42-a4bf01550f1a)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" pol│tes_driver [-] status: Pending from (pid=1881) create_wait /opt/stack/tacker/tacker/vnfm/infra_drivers/kubernetes/kube icy. Falling back to "Default" policy. │rnetes_driver.py:137 Warning FailedCreatePodSandBox 19s kubelet, master Failed create pod sandbox: rpc error: code = Unknown desc │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne = failed to set up sandbox container "310e18b859e118ad6abcaa60d29a9e332da516ce57b8cdf47456ea4882ca66f5" network for pod│tes_driver [-] VNF initializing status: ['default', 'svc-vdu1-b9e851'] Pending from (pid=1881) create_wait /opt/stack/ "redis-master-6fbbc44567-ms8nk": NetworkPlugin cni failed to set up pod "redis-master-6fbbc44567-ms8nk_default" networ│tacker/tacker/vnfm/infra_drivers/kubernetes/kubernetes_driver.py:141 k: Got invalid status code from CNI daemon.; Traceback (most recent call last): │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 WARNING tacker.vnfm.infra_drivers.kubernetes.kuber File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/cni/api.py", line 80, in run │netes_driver [-] VNF Creation failed: Resource creation is not completed within 500 seconds as creation of stack defau vif = self._add(params) Are there any reason for this error? And how can I resolve this problem? Best Regards, Jaewook. ================================================ *Jaewook Oh* (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618 E-mail : jwoh95(a)dcn.ssu.ac.kr ================================================

5 6