- openstack-discuss - lists.openstack.org

Re: RabbitMQ annoying disconnections [kolla-ansible, rabbitmq]
by bkslash 23 Sep '21

23 Sep '21

Hi, I’ve tried to set heartbeat_in_pthread = true, but nothing has changed after kolla-ansible reconfigure - still got disconnections and heartbeat_in_pthread option was present inside configs on all nodes. But in the meantime I tried to alternate rabbit configuration and I realized that Kolla ignored /etc/kolla/config/rabbitmq.conf file (of course at first I checked permissions, etc.) So I tried to put heartbeat = 600 inside /etc/kolla/config/rabbitmq/rabbitmq.conf - this overwrote rabbitmq.conf totally (I mean Kolla didn’t just add heartbeat = 600 to rabbitmq.conf on each controller node, but put ONLY this line in all rabbitmq.conf files ! - rest of the file was cleared). So finally I tried to create separate config for each controller node. So I copied default content of /etc/kolla/rabbitmq/rabbitmq.conf on each controller (because of course these files differs from each other - they have different IPs inside) to /etc/kolla/config/rabbitmq/controller1(2,3)/rabbitmq.conf And finally heartbeat = 600 started to work - no more disconnections. I left Oslo heartbeats and timeouts intact (with the values I’ve set previously) and I don’t have disconnections from services side too. So there are two questions: 1. Why heartbeat_in_pthread didn’t change situation? And if it’s deprecated it was kind of risky to use it long term (after upgrade to wallaby it would stop working anyway?) 2. Why Kolla-ansible ignored rabbitmq.conf file if it is located in /etc/kolla/config, and overwrites ALL content of rabbitmq.conf file when it is located in /etc/kolla/config/rabbitmq/ ? Best regards Adam Tomas > Wiadomość napisana przez Herve Beraud <hberaud(a)redhat.com <mailto:hberaud@redhat.com>> w dniu 21.09.2021, o godz. 08:12: > > > > Le sam. 18 sept. 2021 à 16:12, bkslash <bkslash(a)poczta.onet.pl <mailto:bkslash@poczta.onet.pl>> a écrit : > Hi Ben, > Thank you for the answer. I’m using open stack Victoria deployed with Kolla-ansible. > So I assume I should add (in /etc/kolla/config/global.conf) > > [oslo_messaging_rabbit] > heartbeat_in_pthread = True > > > But for Victoria (nova sample configuration file) it seems to be deprecated? > # DEPRECATED: Run the health check heartbeat thread through a native python > # thread by default. If this option is equal to False then the health check > # heartbeat will inherit the execution model from the parent process. For > # example if the parent process has monkey patched the stdlib by using > # eventlet/greenlet then the heartbeat will be run through a green thread > # (boolean value) > # This option is deprecated for removal. > # Its value may be silently ignored in the future. > #heartbeat_in_pthread = true > So will it work? > > > Yes it will work. > > Few months ago I proposed to remove this deprecation and to keep the option available [1], unfortunately my removal isn't yet merged. > Only the default value is changed and the option will be turned on by default. > > [1] https://review.opendev.org/c/openstack/oslo.messaging/+/800621 <https://review.opendev.org/c/openstack/oslo.messaging/+/800621> > > Best regards > Adam Tomas > > >> Wiadomość napisana przez Ben Nemec <openstack(a)nemebean.com <mailto:openstack@nemebean.com>> w dniu 17.09.2021, o godz. 18:57: >> >> This sounds like you're hitting the heartbeat/eventlet bug. Basically eventlet won't schedule the heartbeat thread unless something is regularly waking up the api threads. Depending on what release you're running, you may be able to set heartbeat_in_pthread[0] to fix this. >> >> 0: https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messagi… <https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messagi…> >> >> On 9/17/21 7:00 AM, Adam Tomas wrote: >>> Hi, >>> after some struggling I have almost „clear” logs (clear=error free :) ) Almost…. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description: >>> https://bugzilla.redhat.com/show_bug.cgi?id=1711794 <https://bugzilla.redhat.com/show_bug.cgi?id=1711794> <https://bugzilla.redhat.com/show_bug.cgi?id=1711794 <https://bugzilla.redhat.com/show_bug.cgi?id=1711794>> >>> in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.” >>> but… It is reported :( And produces tons of logs. >>> I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :( >>> [oslo_messaging_rabbit] >>> heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4 >>> How to avoid this disconnections?? Or what to do to avoid writing this „errors”to logs? >>> Best regards >>> Adam Tomas >>> /var/log/kolla/rabbitmq/rabbitXXX.log >>> 2021-09-17 06:43:08.771 [error] <0.16056.678> closing AMQP connection <0.16056.678> (x.x.x.x00:44882 -> x.x.x.x00:5672 - mod_wsgi:959:aaa12af5-bb0b-456e-a129-a466eccc24f0): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:08.773 [info] <0.7412.679> Closing all channels from connection 'x.x.x.x00:44882 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:09.282 [info] <0.4574.679> accepting AMQP connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.283 [info] <0.28410.678> accepting AMQP connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.285 [info] <0.28410.678> Connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394 >>> 2021-09-17 06:43:09.285 [info] <0.4574.679> Connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93 >>> 2021-09-17 06:43:09.286 [info] <0.4574.679> connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672 - magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.286 [info] <0.28410.678> connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672 - magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.504 [error] <0.14465.678> closing AMQP connection <0.14465.678> (x.x.x.x00:44914 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:09.507 [info] <0.14985.679> Closing all channels from connection 'x.x.x.x00:44914 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:09.521 [info] <0.11574.678> accepting AMQP connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.523 [info] <0.11574.678> Connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) has a client-provided name: mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3 >>> 2021-09-17 06:43:09.525 [info] <0.11574.678> connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.654 [info] <0.31575.679> accepting AMQP connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.656 [info] <0.31575.679> Connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7 >>> 2021-09-17 06:43:09.656 [info] <0.31180.678> accepting AMQP connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.658 [info] <0.31575.679> connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672 - magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.658 [info] <0.31180.678> Connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b >>> 2021-09-17 06:43:09.660 [info] <0.2757.679> accepting AMQP connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.660 [info] <0.31180.678> connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672 - magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.662 [info] <0.2757.679> Connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315 >>> 2021-09-17 06:43:09.663 [info] <0.2757.679> connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672 - magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.814 [error] <0.22797.678> closing AMQP connection <0.22797.678> (x.x.x.x05:57676 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:09.817 [info] <0.7903.679> Closing all channels from connection 'x.x.x.x05:57676 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:10.319 [error] <0.18268.678> closing AMQP connection <0.18268.678> (x.x.x.x00:44924 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:10.322 [info] <0.28032.678> Closing all channels from connection 'x.x.x.x00:44924 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:10.330 [info] <0.19596.678> accepting AMQP connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:10.333 [info] <0.19596.678> Connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) has a client-provided name: nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09 >>> 2021-09-17 06:43:10.334 [info] <0.19596.678> connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:10.831 [info] <0.9838.678> accepting AMQP connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:10.834 [info] <0.9838.678> Connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) has a client-provided name: neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e >>> 2021-09-17 06:43:10.835 [info] <0.9838.678> connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:11.738 [error] <0.15275.678> closing AMQP connection <0.15275.678> (x.x.x.x01:53414 -> x.x.x.x00:5672 - mod_wsgi:968:b60f723c-882e-4780-a162-55c03ebe6179): >>> missed heartbeats from client, timeout: 60s >>> /var/log/kolla/glance/glance-api.log >>> 2021-09-17 13:17:23.212 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:17:29.208 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:17:42.190 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:17:42] "GET / HTTP/1.1" 300 1446 0.001949 >>> 2021-09-17 13:17:46.351 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:12.336 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:12] "GET / HTTP/1.1" 300 1446 0.003754 >>> 2021-09-17 13:18:13.703 956 INFO eventlet.wsgi.server [req-1a00db34-349b-4ed3-8625-edfadc939be4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:13] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.139229 >>> 2021-09-17 13:18:33.065 955 INFO eventlet.wsgi.server [req-a57046eb-549e-4837-95c7-d0994c05449e d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.096302 >>> 2021-09-17 13:18:33.077 955 INFO eventlet.wsgi.server [req-3a8af281-6732-4afd-b3ec-e502367bce51 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.003982 >>> 2021-09-17 13:18:36.747 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:36.759 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:42.460 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:42] "GET / HTTP/1.1" 300 1446 0.001383 >>> 2021-09-17 13:18:52.659 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:52.670 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:12.598 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:19:12] "GET / HTTP/1.1" 300 1446 0.001590 >>> 2021-09-17 13:19:16.367 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:27.127 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:27.184 955 INFO eventlet.wsgi.server [req-8f68efc9-f46a-450d-8138-928e4f2585b4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:19:27] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.032917 >>> /var/log/kolla/keystone/keystone.log >>> 2021-09-17 13:13:54.670 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.148 960 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.150 962 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.345 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:56.044 961 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:18.402 966 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:31.372 963 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:36.580 967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:38.771 965 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:16:12.226 964 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> /var/log/kolla/nova/nova-compute.log >>> 2021-09-17 13:10:07.710 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56226. >>> 2021-09-17 13:11:11.473 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:11:34.046 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:11:34.721 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:07.699 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:13:08.734 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56434. >>> 2021-09-17 13:14:11.501 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:34.070 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:34.745 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:16:08.724 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:16:09.756 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56638. >>> 2021-09-17 13:17:34.098 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:09.747 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:19:10.781 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56834. >>> 2021-09-17 13:20:34.124 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:20:34.777 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:20:51.789 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:22:10.772 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:22:11.806 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57038. >>> 2021-09-17 13:23:34.149 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:23:34.803 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:23:51.818 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:25:11.795 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:25:12.842 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57240. >>> 2021-09-17 13:26:34.176 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:26:34.830 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:28:12.812 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:28:13.848 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57446. > > > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ <https://github.com/4383/> > https://twitter.com/4383hberaud <https://twitter.com/4383hberaud> >

4 4

[all][tc] Technical Committee next weekly meeting on Sept 23rd at 1500 UTC
by Ghanshyam Mann 22 Sep '21

22 Sep '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 23rd at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 22th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[stable] remove formal 'unmaintained' phase from EM to EOL transition
by Előd Illés 22 Sep '21

22 Sep '21

Hi stable maintainers! On the Xena Virtual PTG there was a discussion about 'Stable Extended Maintenance to End of Life transition' [1] and at some point we touched that the listed 'Unmaintained' phase [2] does not add value in the process (rather some confusion) so that was agreed that it should be removed (line 397). I've seen that the removal was not yet proposed so I created a patch [3] for it now. So this is just a warning for the change. Thanks, Előd [1] Line 342-397 https://etherpad.opendev.org/p/tc-xena-ptg [2] https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… [3] https://review.opendev.org/c/openstack/project-team-guide/+/810499

1 0

[heat] Proposing Brendan Shephard for heat core
by Rico Lin 22 Sep '21

22 Sep '21

Hi all I would like to suggest to add Brendan Shephard(bshephar(a)redhat.com) to the heat core reviewer group. He has been working on heat actively. provide good quality reviews and commits. I think it will be great to have him as the heat core reviewer. *Rico Lin* OIF Individual Board of directors, OpenStack TC, Multi-arch SIG chair, Heat PTL, Senior Software Engineer@EasyStack

3 2

[cinder] Bug deputy report for week of 2021-22-09
by Sofia Enriquez 22 Sep '21

22 Sep '21

This is a bug report from 2021-08-09 to 2021-15-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- High - https://bugs.launchpad.net/cinder/+bug/1944118 " cinder-manage db version fails with: AttributeError: 'Engine' object has no attribute 'get_main_option'". Assigned to Stephen Finucane. Medium - https://bugs.launchpad.net/os-brick/+bug/1944474 "iSCSI connections are not reinitiated after host reboot". Assigned to Sophie Huang. - https://bugs.launchpad.net/cinder/+bug/1926412 "When the backup cannot find an available service, update the volume status to none". Assigned to ZhaoYixin - https://bugs.launchpad.net/cinder/+bug/1920729 "PowerStore driver reports no iSCSI targets found". Assigned to Ivan Pchelintsev. Low - https://bugs.launchpad.net/cinder/+bug/1924798 "NetApp ONTAP cannot initialize the driver with SVM scoped mode". Assigned to Felipe Rodrigues. - https://bugs.launchpad.net/cinder/+bug/1924643 "NetApp ONTAP cannot resize a volume LUN with a sub-clone". Assigned to Felipe Rodrigues. Invalid - https://bugs.launchpad.net/cinder/+bug/1922507 "cinder-volume-driver test fails "ModuleNotFoundError: No module named 'fixtures'" Thanks -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[monasca][cloudkitty] [kolla-ansible][victoria] metrics metadata
by Tomasz Rutkowski 22 Sep '21

22 Sep '21

Hi, perhaps I'm missing something - monasca returns metrics metadata when asked for measurements (/metrics/measurements) however cloudkitty asks only for aggregated data (/metrics/statistics). Is there some way to push cloudkitty to work with monasca also by the field-id mappings (metadata values)? with best regards -- Tomasz Rutkowski

1 0

[Outreachy] Call for mentors, project submission by September 26
by Mahati Chamarthy 21 Sep '21

21 Sep '21

Hello! *TL;DR OpenStack is participating in Outreachy for the upcoming December round!* *Please submit projects as soon as possible, the deadline is September 26th, 4 pm UTC:* *https://www.outreachy.org/communities/cfp/openstack/ <https://www.outreachy.org/communities/cfp/openstack/>* Outreachy's goal is to support people from groups underrepresented in the technology industry. Interns will work remotely with mentors from our community. We are seeking mentors to propose projects that Outreachy interns can work on during their internship. If you or your colleagues are interested in mentoring in the next round, please submit your project proposal by *September 26th,* *2021 *over here https://www.outreachy.org/communities/cfp/openstack/ Mentors should read the mentor FAQ https://www.outreachy.org/mentor/mentor-faq and find more details about the Outreachy program and timeline in https://www.outreachy.org/communities/cfp/ . If you want help crafting your project proposal, please contact me Mahati Chamarthy <mahati.chamarthy(a)gmail.com> or Samuel Queiroz < samueldmq(a)gmail.com> and we will be glad to assist you. Thank you, Samuel Queiroz

1 1

[requirements][release][ironic] Exception for ironic deliverables - Xena
by Iury Gregory 21 Sep '21

21 Sep '21

Hello everyone, The ironic CI last week was in a very bad shape and we couldn't merge some patches that we wanted to have in the Xena release, we knew last week was the RC1 and we should have proposed the release for our derivables. Today we were able to merge a workaround to get our CI green again, we will probably have all the required patches merged by EOD or tomorrow morning. I would like to ask for an exception so we can release our deliverables tomorrow. Thanks! -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

3 2

Constant HA Virtual Router Flapping
by Justin Moravec 21 Sep '21

21 Sep '21

Any ideas welcome, Kolla-ansible Ussuri deployment. Have some projects that are using virtual routers that are in HA setup that are constantly flapping (flipping between one neutron node to another) causing disconnects. This only happens in SOME projects, other projects with HA routers are working as intended. This only affects HA router deployments, non HA routers work as intended no matter the neutron node they are on. Have verified that each virtual router can reach the external network gateway while it is on each of the neutron nodes. Thanks, Justin

1 0

[neutron] Bug deputy report (Sep 13 - 19)
by Akihiro Motoki 21 Sep '21

21 Sep '21

Hi, Here is my bug deputy report last week. Sorry for late. [Summary] * There are several RC2 candidate bugs. I added a DVR one to Xena RC2 milestone. * dynamic-routing and vpnaas bugs need to be investigated. I would be nice if someone can look into the dynamic-routing one. Regarding vpnaas ones, unfortunately I had no time to check them last week but I will visit them this week. [RC candidates] * https://bugs.launchpad.net/neutron/+bug/1943714 DB session commit error in resource_registry.set_resources_dirty In Progress, High, Fix already approved * https://bugs.launchpad.net/neutron/+bug/1943930 [DHCP] new line symbol in opt_name of extra_dhcp_opt causes dnsmasq to fail In Progress, High, Fix already approved * https://bugs.launchpad.net/neutron/+bug/1943846 Create floating agent gw port for DVR fails In Progress, High, Fix already approved NOTE: It is caused by the change in the notification payload during Xena, so I targeted to Xena-RC2. [QoS] * https://bugs.launchpad.net/neutron/+bug/1943724 Change a QoS policy on a bound port does not handle minimum bandwidth direction change In Progress, Medium NOTE: A fix is proposed but it is in the feature patch series of the min pps rule. Hopefully someone familiar with QoS can check the dependency of the fix. [dynamic-routing] * https://bugs.launchpad.net/neutron/+bug/1943725 Automatic cleanup of BGP speakers is too aggressive New, Unassigned [VPNaaS] * https://bugs.launchpad.net/neutron/+bug/1943449 VPNaaS reconfiguration creates duplicate IPtables rules causes the VPN connection to remain DOWN New, Unassigned * https://bugs.launchpad.net/neutron/+bug/1943716 vpn services / vpn connections are stuck in PENDING CREATE New, Unassigned NOTE: RabbitMQ topic mismatch is reported here. It is interesting. I will check later this week. Thanks, Akihiro Motoki (irc: amotoki)

1 0

[release] Release countdown for week R-2, Sep 20 - Sep 24
by Herve Beraud 20 Sep '21

20 Sep '21

Development Focus ----------------- At this point we should have release candidates (RC1 or recent intermediary release) for all the Xena deliverables. Teams should be working on any release-critical bugs that would require another RC or intermediary release before the final release. Actions ------- Early in the week, the release team will be proposing stable/xena branch creation for all deliverables that have not branched yet, using the latest available Xena release as the branch point. If your team is ready to go for creating that branch, please let us know by leaving a +1 on these patches. If you would like to wait for another release before branching, you can -1 the patch and update it later in the week with the new release you would like to use. By the end of the week the release team will merge those patches though, unless an exception is granted. Once stable/xena branches are created, if a release-critical bug is detected, you will need to fix the issue in the master branch first, then backport the fix to the stable/xena branch before releasing out of the stable/xena branch. After all of the cycle-with-rc projects have branched we will branch devstack, grenade, and the requirements repos. This will effectively open them up for Yoga development, though the focus should still be on finishing up Xena until the final release. For projects with translations, watch for any translation patches coming through and merge them quickly. A new release should be produced so that translations are included in the final Xena release. Finally, now is a good time to finalize release notes. In particular, consider adding any relevant "prelude" content. Release notes are targeted for the downstream consumers of your project, so it would be great to include any useful information for those that are going to pick up and use or deploy the Xena version of your project. Upcoming Deadlines & Dates -------------------------- Final RC deadline: September 30 (R-1 week) Final Xena release: October 6 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0

Changing Default Network Quotas in Openstack
by Karera Tony 20 Sep '21

20 Sep '21

Hello Team, I have a working Wallaby Openstack deployed using kolla-ansible. However, I have not been able to find a way of changing the Network quotas as I need to increase the defaults. Can someone kindly advise on how to go about it ? Regards Tony Karera

3 2

OpenInfra Live - September 23 at 9am CT
by Ashlee Ferguson 20 Sep '21

20 Sep '21

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenInfra Community. Join the volunteer organizers of OpenInfra Days to hear about the happenings at their latest local events, what they're looking forward to in the future, and learn what's happening with OpenInfra all over the world. Episode: OpenInfra Days 2021 Recap Date and time: September 23, 2021 at 9am CT (1400 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=-s1eJa6u_2A <https://www.youtube.com/watch?v=-s1eJa6u_2A> LinkedIn: https://www.linkedin.com/feed/update/urn:li:ugcPost:6844374141978734592/ <https://www.linkedin.com/feed/update/urn:li:ugcPost:6844374141978734592/> Facebook: https://www.facebook.com/104139126308032/posts/4341771172544785/ <https://www.facebook.com/104139126308032/posts/4341771172544785/> WeChat: recording will be posted on OpenStack WeChat after the live stream Speakers: Akihiro Hasegawa, Cloud Operator Days Tokyo Organizer Rico Lin, OpenInfra Days Asia Organizer Sartika Lestari, OpenInfra Days Indonesia Organizer Have an idea for a future episode? Share it now at ideas.openinfra.live <https://openinfrafoundation.formstack.com/forms/openinfralive>. Register now for OpenInfra Live: Keynotes, a special edition of OpenInfra Live on November 17-18th starting at 1500 UTC: https://openinfralivekeynotes.eventbrite.com/ <https://openinfralivekeynotes.eventbrite.com/> Thanks, Ashlee

1 0

[openstack-cli] Missing volume backend commands
by Laurent Dumont 20 Sep '21

20 Sep '21

Hey everyone, Trying to use the volume backend command to get some information regarding storage pools. https://docs.openstack.org/python-openstackclient/latest/cli/command-object… Not sure if it's something on my side, but I cannot see to access a subsection of the volume commands from the CLI client. openstack volume backend pool list openstack: 'volume backend pool list' is not an openstack command. See 'openstack --help'. Did you mean one of these? volume attachment complete volume attachment create volume attachment delete volume attachment list volume attachment set volume attachment show volume backup create volume backup delete volume backup list /usr/local/lib/python3.9/site-packages/openstack/ drwxr-xr-x 41 laurentdumont admin 1.3K 2 Jun 17:03 . drwxr-xr-x 187 laurentdumont admin 5.8K 20 Sep 09:47 .. -rw-r--r-- 1 laurentdumont admin 2.4K 2 Jun 17:03 __init__.py -rw-r--r-- 1 laurentdumont admin 1.2K 2 Jun 17:03 __main__.py drwxr-xr-x 15 laurentdumont admin 480B 2 Jun 17:03 __pycache__ -rw-r--r-- 1 laurentdumont admin 1.4K 2 Jun 17:03 _hacking.py -rw-r--r-- 1 laurentdumont admin 4.6K 2 Jun 17:03 _log.py -rw-r--r-- 1 laurentdumont admin 5.6K 2 Jun 17:03 _services_mixin.py drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 accelerator drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 baremetal drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 baremetal_introspection drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 block_storage drwxr-xr-x 28 laurentdumont admin 896B 2 Jun 17:03 cloud drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 clustering drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 compute drwxr-xr-x 14 laurentdumont admin 448B 2 Jun 17:03 config -rw-r--r-- 1 laurentdumont admin 21K 2 Jun 17:03 connection.py drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 database drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 dns -rw-r--r-- 1 laurentdumont admin 8.6K 2 Jun 17:03 exceptions.py drwxr-xr-x 5 laurentdumont admin 160B 2 Jun 17:03 fixture -rw-r--r-- 1 laurentdumont admin 1.6K 2 Jun 17:03 format.py drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 identity drwxr-xr-x 11 laurentdumont admin 352B 2 Jun 17:03 image drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 instance_ha drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 key_manager drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 load_balancer drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 message drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 network drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 object_store drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 orchestration drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 placement -rw-r--r-- 1 laurentdumont admin 29K 2 Jun 17:03 proxy.py -rw-r--r-- 1 laurentdumont admin 0B 2 Jun 17:03 py.typed -rw-r--r-- 1 laurentdumont admin 88K 2 Jun 17:03 resource.py -rw-r--r-- 1 laurentdumont admin 13K 2 Jun 17:03 service_description.py drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 shared_file_system drwxr-xr-x 9 laurentdumont admin 288B 2 Jun 17:03 tests -rw-r--r-- 1 laurentdumont admin 12K 2 Jun 17:03 utils.py -rw-r--r-- 1 laurentdumont admin 641B 2 Jun 17:03 version.py drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 workflow usr/local/…/site-packages/openstack % python3 Python 3.9.2 (default, Mar 26 2021, 23:27:12) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> I cannot see the volume folder (but it's on the opendev) The command was added here https://opendev.org/openstack/python-openstackclient/commit/9647d43bd5e77c8… Removing + reinstall the client doesn't seem to work either. Any clues?

2 2

Install controller computehci on 3 nodes with tripleO
by wodel youchi 20 Sep '21

20 Sep '21

Hi, Is it technically possible to install openstack tripleO on 3 nodes, such that all 3 nodes play the roles of controller compute and ceph storage (hci) at the same time, with replication and HA and all? Regards.

3 4

[Swift] Rebalancing EC question
by Reid Guyett 20 Sep '21

20 Sep '21

Hello, We were working on expanding one of our clusters (Ussuri on Ubuntu 18.04) and are wondering about the rebalance behavior of swift-ring-builder. When we run it in debug mode on a 15/4 EC ring, we see this message about "Unable to finish rebalance plan after 2 attempts" and are seeing 100% partitions reassigned. DEBUG: Placed 10899/2 onto dev r1z3-10.40.48.72/d10 DEBUG: Placed 2183/3 onto dev r1z5-10.40.48.76/d11 DEBUG: Placed 1607/1 onto dev r1z3-10.40.48.70/d28 DEBUG: Assigned 32768 parts DEBUG: Gather start is 10278 (Last start was 25464) DEBUG: Unable to finish rebalance plan after 2 attempts Reassigned 32768 (100.00%) partitions. Balance is now 63.21. Dispersion is now 0.00 ------------------------------------------------------------------------------- NOTE: Balance of 63.21 indicates you should push this ring, wait at least 1 hours, and rebalance/repush. ------------------------------------------------------------------------------- Moving 100% seems scary, what does that mean in this situation? Is this message because 1 fragment from every partition is moved and that is the most that it can do per rebalance because they are technically the same partition? When we compare the swift-ring-builder output (partitions per device) between rebalances we can see some partitions move each time until we no longer see the push/wait/rebalance message again. So it's not really moving 100% partitions. Reid

2 2

FreeNAS intergration witn Openstack
by open infra 20 Sep '21

20 Sep '21

Hi, Is it possible to use a freenas storage instead of ceph cluster? Regards, Danishka

3 2

[TripleO] Installation of Undercloud machine on Centos 8 fails in prefetching the Images
by Anirudh Gupta 20 Sep '21

20 Sep '21

Hi Team, I was trying to install Undercloud on Centos 8 using the link below, but facing error in executing "openstack undercloud install" command. The error is in prefetch image Steps followed: 1. Created a non root stack user - sudo useradd stack - sudo passwd stack # specify a password - echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack - sudo chmod 0440 /etc/sudoers.d/stack - su - stack 2. Install the python-tripleo-repos - sudo dnf install -y https://trunk.rdoproject.org/centos8/component/tripleo/current/python3-trip… - In order to install victoria release, executed the command - *sudo -E tripleo-repos -b victoria current ceph* 3. As per the document, "*The remaining repositories configuration steps below should not be done for stable releases!"* - *So, I didn't executed the command* - "sudo -E tripleo-repos current-tripleo-dev ceph" - *Is my understanding correct? This step has to be skipped in case we are going with a stable victoria release?* 4. Modified undercloud.conf file and executed the command - openstack undercloud install *ISSUE:* 2021-09-20 11:16:18,724 p=59556 u=root n=ansible | 2021-09-20 11:16:18.723422 | 525400a8-fc39-4771-208c-000000000e71 | FATAL | Pre-fetch all the containers | undercloud | item= quay.io/tripleovictoria/openstack-swift-account:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg":" *Failed to pull image quay.io/tripleovictoria/openstack-swift-account:current-tripleo <http://quay.io/tripleovictoria/openstack-swift-account:current-tripleo>", "prefetch_image": "quay.io/tripleovictoria/openstack-swift-account:current-tripleo <http://quay.io/tripleovictoria/openstack-swift-account:current-tripleo>"}* 5. In order to eliminate the problem, I manually installed the image - - sudo podman pull *quay.io/tripleovictoria/openstack-swift-account:current-tripleo <http://quay.io/tripleovictoria/openstack-swift-account:current-tripleo>* After re-executing, then it again failed for the keystone image, which was already installed on the system. I verified in *sudo podman images* command *I tried running this 5-6 times and issue generates randomly for any openstack image ( sometimes for rabbitmq sometimes for swift, keystone) which is already installed in machine* Can someone please help or suggest what step I am missing Note: This I have tried using Centos 8 Stream and Centos 8.4(using --no-stream), but same issue is observed in both Regards Anirudh Gupta

1 0

[keystone] Ed25519 support
by Eduardo Santos 20 Sep '21

20 Sep '21

Hi folks, I'm trying to create an instance with an assigned keypair that uses the Ed25519 algorithm, but I'm unable to login into the instance without providing a password (I generated the public key without a passphrase). RSA keypairs work fine. Is Ed25519 not supported? I didn't find a support matrix in the documentation. Thanks.

2 1

RabbitMQ annoying disconnections
by Adam Tomas 20 Sep '21

20 Sep '21

Hi, after some struggling I have almost „clear” logs (clear=error free :) ) Almost…. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description: https://bugzilla.redhat.com/show_bug.cgi?id=1711794 <https://bugzilla.redhat.com/show_bug.cgi?id=1711794> in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.” but… It is reported :( And produces tons of logs. I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :( [oslo_messaging_rabbit] heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4 How to avoid this disconnections?? Or what to do to avoid writing this „errors”to logs? Best regards Adam Tomas /var/log/kolla/rabbitmq/rabbitXXX.log 2021-09-17 06:43:08.771 [error] <0.16056.678> closing AMQP connection <0.16056.678> (x.x.x.x00:44882 -> x.x.x.x00:5672 - mod_wsgi:959:aaa12af5-bb0b-456e-a129-a466eccc24f0): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:08.773 [info] <0.7412.679> Closing all channels from connection 'x.x.x.x00:44882 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:09.282 [info] <0.4574.679> accepting AMQP connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) 2021-09-17 06:43:09.283 [info] <0.28410.678> accepting AMQP connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) 2021-09-17 06:43:09.285 [info] <0.28410.678> Connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394 2021-09-17 06:43:09.285 [info] <0.4574.679> Connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93 2021-09-17 06:43:09.286 [info] <0.4574.679> connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672 - magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.286 [info] <0.28410.678> connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672 - magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.504 [error] <0.14465.678> closing AMQP connection <0.14465.678> (x.x.x.x00:44914 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:09.507 [info] <0.14985.679> Closing all channels from connection 'x.x.x.x00:44914 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:09.521 [info] <0.11574.678> accepting AMQP connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) 2021-09-17 06:43:09.523 [info] <0.11574.678> Connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) has a client-provided name: mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3 2021-09-17 06:43:09.525 [info] <0.11574.678> connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.654 [info] <0.31575.679> accepting AMQP connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) 2021-09-17 06:43:09.656 [info] <0.31575.679> Connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7 2021-09-17 06:43:09.656 [info] <0.31180.678> accepting AMQP connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) 2021-09-17 06:43:09.658 [info] <0.31575.679> connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672 - magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.658 [info] <0.31180.678> Connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b 2021-09-17 06:43:09.660 [info] <0.2757.679> accepting AMQP connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) 2021-09-17 06:43:09.660 [info] <0.31180.678> connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672 - magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.662 [info] <0.2757.679> Connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315 2021-09-17 06:43:09.663 [info] <0.2757.679> connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672 - magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.814 [error] <0.22797.678> closing AMQP connection <0.22797.678> (x.x.x.x05:57676 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:09.817 [info] <0.7903.679> Closing all channels from connection 'x.x.x.x05:57676 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:10.319 [error] <0.18268.678> closing AMQP connection <0.18268.678> (x.x.x.x00:44924 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:10.322 [info] <0.28032.678> Closing all channels from connection 'x.x.x.x00:44924 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:10.330 [info] <0.19596.678> accepting AMQP connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) 2021-09-17 06:43:10.333 [info] <0.19596.678> Connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) has a client-provided name: nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09 2021-09-17 06:43:10.334 [info] <0.19596.678> connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:10.831 [info] <0.9838.678> accepting AMQP connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) 2021-09-17 06:43:10.834 [info] <0.9838.678> Connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) has a client-provided name: neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e 2021-09-17 06:43:10.835 [info] <0.9838.678> connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:11.738 [error] <0.15275.678> closing AMQP connection <0.15275.678> (x.x.x.x01:53414 -> x.x.x.x00:5672 - mod_wsgi:968:b60f723c-882e-4780-a162-55c03ebe6179): missed heartbeats from client, timeout: 60s /var/log/kolla/glance/glance-api.log 2021-09-17 13:17:23.212 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:17:29.208 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:17:42.190 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:17:42] "GET / HTTP/1.1" 300 1446 0.001949 2021-09-17 13:17:46.351 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:12.336 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:12] "GET / HTTP/1.1" 300 1446 0.003754 2021-09-17 13:18:13.703 956 INFO eventlet.wsgi.server [req-1a00db34-349b-4ed3-8625-edfadc939be4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:13] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.139229 2021-09-17 13:18:33.065 955 INFO eventlet.wsgi.server [req-a57046eb-549e-4837-95c7-d0994c05449e d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.096302 2021-09-17 13:18:33.077 955 INFO eventlet.wsgi.server [req-3a8af281-6732-4afd-b3ec-e502367bce51 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.003982 2021-09-17 13:18:36.747 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:36.759 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:42.460 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:42] "GET / HTTP/1.1" 300 1446 0.001383 2021-09-17 13:18:52.659 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:52.670 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:12.598 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:19:12] "GET / HTTP/1.1" 300 1446 0.001590 2021-09-17 13:19:16.367 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:27.127 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:27.184 955 INFO eventlet.wsgi.server [req-8f68efc9-f46a-450d-8138-928e4f2585b4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:19:27] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.032917 /var/log/kolla/keystone/keystone.log 2021-09-17 13:13:54.670 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.148 960 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.150 962 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.345 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:56.044 961 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:18.402 966 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:31.372 963 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:36.580 967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:38.771 965 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:16:12.226 964 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer /var/log/kolla/nova/nova-compute.log 2021-09-17 13:10:07.710 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56226. 2021-09-17 13:11:11.473 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:11:34.046 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:11:34.721 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:07.699 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:13:08.734 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56434. 2021-09-17 13:14:11.501 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:34.070 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:34.745 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:16:08.724 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:16:09.756 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56638. 2021-09-17 13:17:34.098 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:09.747 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:19:10.781 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56834. 2021-09-17 13:20:34.124 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:20:34.777 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:20:51.789 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:22:10.772 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:22:11.806 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57038. 2021-09-17 13:23:34.149 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:23:34.803 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:23:51.818 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:25:11.795 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:25:12.842 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57240. 2021-09-17 13:26:34.176 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:26:34.830 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:28:12.812 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:28:13.848 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57446.

4 5

[ops][cinder] is anyone using RateLimitingMiddleware ?
by Brian Rosmaita 20 Sep '21

20 Sep '21

Hello operators, Takashi Kajinami noticed that cinder's RateLimitingMiddleware class lives among the legacy v2 classes and should be moved to live with the v3 classes, but then it occurred to him that maybe it should simply be deprecated and removed [0]. There are a few reasons to believe that nobody is actually using this functionality, for example, it's not included in the default api-paste.ini file, this middleware no longer exists in Nova, it gives you inexact behavior with multiple API nodes, etc. We discussed this topic at today's cinder meeting and decided to ask whether anyone is using the RateLimitingMiddleware with cinder before deprecating it for removal. If it's going to be deprecated, we'd like to do it in Xena, so please reply to this email or leave a comment on the deprecation patch [1] (or preferably both) before 23:59 UTC on Tuesday 14 September 2021. Thank you! [0] https://bugs.launchpad.net/cinder/+bug/1942696 [1] https://review.opendev.org/c/openstack/cinder/+/807469

2 1

Edge for scientific research session on September 13 Edge WG meeting
by Ildiko Vancsa 18 Sep '21

18 Sep '21

Hi, I’m very excited to invite you to a session with the Chameleon project to discuss how edge computing is powering education, scientific research and more! As always in the area of edge there are challenges and gaps that we need to solve together. In that sense the session will start with a presentation that we will then turn into a discussion to turn towards discussing solutions. Please also note that the Chameleon project is using OpenStack to build edge solutions. As they identified some gaps they would like to work with the community to address the challenges that they’ve been hitting. Please join us to get their feedback and find the best way to improve OpenStack to deliver a platform that can be used on the edge! The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. For the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings Please find the details of the session below. Thanks and Best Regards, Ildikó Title: Chameleon: a cloud to edge infrastructure for science Abstract: The NSF-funded Chameleon project (www.chameleoncloud.org) provides an OpenStack-based bare metal reconfigurable cloud that supports systems research, education, and emergent application projects in science. The facility has been in existence for 6+ years, and has served ~6,000 users working on a variety of projects ranging from networking, operating systems design, and security, to innovative applications in astronomy, physics, and life sciences. Recent research trends in IoT and edge computing, as evidenced by increased numbers of project applications in those areas, made it clear that Chameleon needed to expand into the edge space. Remaining within the OpenStack framework allowed us to leverage end-user features that we developed for Chameleon, such as access via federated identity and Jupyter integration. Accordingly, we developed infrastructure called CHI@Edge that adapts Zun to provision containers on edge devices and significantly expands its capabilities to work in the edge space. The talk will describe our use case and the emergent product definition, the OpenStack adaptations we made to the current preview implementation, the directions we plan to take in our further work, as well as the ecosystem of projects and NSF infrastructure deployments leveraging this work and collaborating with us.

3 9

OVS_AFXDP for production
by Satish Patel 18 Sep '21

18 Sep '21

Folks, I’m running some high performance workload on SRIOV and planning to move from SRIOV to DPDK but recently Hearing a lots about AFDXP data path and just wanted to check with expert, is this ready for production? How good it is to compare with DPDK? Sent from my iPhone

1 0

[Interop] New meeting time
by Arkady Kanevsky 17 Sep '21

17 Sep '21

The interop WG changed its weekly meeting time to Thursday 16:00pm UTC. The Google meet and etherpad stay the same. See https://wiki.openstack.org/wiki/Governance/InteropWG#Meetings Thanks, Arkady -- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456

1 0

[all][tc][goals] Project Specific PTL and Contributor Documentation: Week R-20 Update
by Ghanshyam Mann 17 Sep '21

17 Sep '21

Hello Everyone, As you know, we did not select any community-wide goal for Xena cycle and in PTG, TC decided to spend more time on the previous cycle pending community-wide goal work[1] One of the previous cycle (Ussuri) goals is 'Project Specific PTL and Contributor Documentation' which still not completed by many projects. - https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-c… I am starting the work to finish this goal which is pretty straight forwards and sending the updates in this ML thread. Please review the patches and merge them or push the patches for your project's repo if not yet done. Gerrit Topic: https://review.opendev.org/q/topic:%22project-ptl-and-contrib-docs%22+(stat… Tracking: https://storyboard.openstack.org/#!/story/2007236 Progress Summary: =============== * Projects completed: 32 * Projects for which patches are up and required to merge those: 10 * Projects for which patches are not yet up: 12 [1] https://etherpad.opendev.org/p/tc-xena-tracker (Item#2) -gmann

2 2

[all][tc] What's happening in Technical Committee: summary 17th Sept, 21: Reading: 5 min
by Ghanshyam Mann 17 Sep '21

17 Sep '21

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC this week IRC meeting held on Sept 16th Thursday. * Most of the meeting discussions are summarized in the below sections (Completed or in-progress activities section). To know more details, you can check the complete logs @ https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-16-15.00.log.html * We will have next week's meeting on Sept 23rd, Thursday 15:00 UTC on IRC, feel free the topic in agenda [1] by Sept 22nd. 2. What we completed this week: ========================= * PTLs for all the six leaderless projects are now assigned and merged[2] * Added openstack-loadbalancer charm and interfaces[3] * TC chair and vice-chair for Yoga cycle are selected. I will continue as chair[3] and yoctozepto as vice-chair[4]. 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ TC is using the etherpad[6] for Xena cycle working item. We will be checking and updating the status biweekly in the same etherpad. Open Reviews ----------------- * Three open reviews for ongoing activities[7]. TC tags analysis ------------------- * As discussed in the last PTG, TC is working on an analysis of the usefulness of the Tags framework[8] or what all tags can be cleaned up. * It seems no response yet on email from yoctozepto[9], if you are an operator, please respond to the email and based on the feedback we will continue the discussion in PTG. Collecting Pain points from projects/SIGs/pop-up ----------------------------------------------------------- * No updates on this, we will continue discussing it in PTG[10], join us if you are interested to help/discuss this topic. Project updates ------------------- * Add the cinder-netapp charm to Openstack charms[11] * Retiring js-openstack-lib [12] * Retire puppet-freezer[13] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [14]. * Current status: "Secure RBAC" is selected for Yoga cycle[15]. PTG planning ---------------- * We are collecting the PTG topics in etherpad[16], feel free to add any topic you would like to discuss. * We discussed the live stream of one of the TC PTG sessions like we did last time. Once we will have more topics in etherpad then we can select the appropriate one. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[17], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[18]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [19] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [20] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://etherpad.opendev.org/p/yoga-leaderless [3] https://review.opendev.org/c/openstack/governance/+/807837 [4] https://review.opendev.org/c/openstack/governance/+/807163 [5] https://review.opendev.org/c/openstack/governance/+/807178 [6] https://etherpad.opendev.org/p/tc-xena-tracker [7] https://review.opendev.org/q/project:openstack/governance+status:open [8] https://governance.openstack.org/tc/reference/tags/index.html [9] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/02480… [10] https://etherpad.opendev.org/p/pain-point-elimination [11] https://review.opendev.org/c/openstack/governance/+/809011 [12] https://review.opendev.org/c/openstack/governance/+/798540 [13] https://review.opendev.org/c/openstack/governance/+/807163 [14] https://etherpad.opendev.org/p/y-series-goals [15] https://review.opendev.org/c/openstack/governance/+/803783 [16] https://etherpad.opendev.org/p/tc-yoga-ptg [17] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [18] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [19] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [20] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann

1 0

[release][PTL] broken gates and stuck releases
by Herve Beraud 17 Sep '21

17 Sep '21

Hello PTLs, For your information our gates are currently broken so we are not able to merge patches and so to release deliverables. indeed openstack/release rely on openstack/governance to validate releases. The governance repo relies on pybot2 which is no longer maintained and became not installable a few hours ago. That broke the release gate and led us to a catch-22 situation where we are not able to release the governance repo to fix our repo. Our current solution is to pull openstack/governance from git source instead of from pypi. By doing that we will pull the fixed version from opendev git and we will unblock our gates. In this way, we will be able to restart to merge patches and release deliverables. We currently wait for some feedback from the infra team to help us to force merging the needed fixes by using their gerrit super admin rights, hence, bypassing the catch-22 situation. This situation highlighted the cross dependency between these two repos (governance and releases) and I don't think that this is something healthy. I propose to keep pulling the governance repo from git source even after we will fix the catch-22 situation. Any opinions? Thanks for your understanding, we will inform you when we will have updates. https://opendev.org/openstack/governance/commit/9128e502253db9aac346ce25726… https://review.opendev.org/c/openstack/releases/+/809626 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

3 3

Delete Volume on Instance Delete
by Derek O keeffe 17 Sep '21

17 Sep '21

Hi all, I was able to set "Create new volume" to no by default on the launch instance page by adding the following to /etc/openstack-dashboard/local_settings.py LAUNCH_INSTANCE_DEFAULTS ={ 'create_volume': False,} Is there anyway I can set "Delete Volume on Instance Delete" to yes by default should someone select to create a new volume when launching an instance. I tried the following but no luck LAUNCH_INSTANCE_DEFAULTS ={ 'create_volume': False, 'delete_on_termination': True} Thanks for any help/info in advance. Regards,Derek

2 2

How to resume/abort a migration in error state
by Yingji Sun 17 Sep '21

17 Sep '21

Buddies, I am in openstack train and I have some failure live-migration cases that leave some migration states to error. These failure cases may block the following migration request in “queued” state. Is it possible that I force to complete the failure cases ? live_migrate_force_complete only works for a “running”migration and live_migrate_abort works only for “running/queued/preparing”migration. Any suggestion would be appreciated. Yingji.

1 0

[neutron] Drivers meeting - Friday 17.09.2021 - cancelled
by Lajos Katona 16 Sep '21

16 Sep '21

Hi, Due to lack of topics for Today's Drivers meeting, let's cancel the meeting. Lajos (lajoskatona)

1 0

Autoscaling based on requests
by open infra 16 Sep '21

16 Sep '21

Hi, I have a requirement to allocate a server(VM) for each user request and make this automatic. Is it possible with Openstack autoscale? Scaling should not be done based on either cpu or memory usage as only one user is supposed to access a single VM. Regards, Danishka

1 0

[blazar] Third-party resource type plugins
by Jason Anderson 16 Sep '21

16 Sep '21

Hi all, I’ve just submitted a spec for making resources in Blazar follow a more explicit plugin architecture: https://review.opendev.org/c/openstack/blazar-specs/+/809485 The intent is to allow out-of-tree resource types, to allow deployers/operators to define their own abstractions. This could support some pretty wide-ranging implementations, like reserving bandwidth allocations on switches or perhaps access to other controlled/limited resources. I hope that this change will also enable Blazar to be adopted by a wider set of users and use cases. Please, have a look—it is currently proposed under Xena because there is not yet a Yoga spec section in blazar-specs. Cheers, /Jason

1 0

[cinder] festival of XS reviews 17 September 2021
by Brian Rosmaita 16 Sep '21

16 Sep '21

Hello Cinder community members, This is a reminder that the most recent edition of the Cinder Festival of XS Reviews will be held at the end of this week (tomorrow!) on Friday 17 September. Since we'd like to keep the master branch clean in case any release-critical bugs are discovered that will need to be backported to stable/xena, I propose that we focus on drivers reviews tomorrow, which will not impact the main cinder code. who: Everyone! what: The Cinder Festival of XS Reviews when: Friday 17 September 2021 from 1400-1600 UTC where: https://meetpad.opendev.org/cinder-festival-of-reviews This recurring meeting can be placed on your calendar by using this handy ICS file: http://eavesdrop.openstack.org/calendars/cinder-festival-of-reviews.ics See you there! brian

2 1

[PTG] Last call to sign up your team!
by Ashlee Ferguson 16 Sep '21

16 Sep '21

Hi everyone, The next virtual PTG (Project Teams Gathering)[1] is taking place October 18-22, 2021! This is your final chance to signup your team. To do so you must complete BOTH the survey[2] AND reserve time in the ethercalc[3] by end of day Friday, September 17. Then *register* so you don’t miss important updates: https://openinfra-ptg.eventbrite.com/ <https://openinfra-ptg.eventbrite.com/> Teams that have already signed up for PTG include: - Airship - Barbican - Blazar - Cinder - CloudKitty - Cyborg - Designate - Diversity and Inclusion WG - First Contact SIG - Glance - Heat - Horizon - Interop/Refstack - Ironic - Kata-Containers - Keystone - Kolla - Kuryr - Magma - Magnum - Manila - Masakari - Multi-arch SIG - Neutron - Nova - Octavia - OpenDev - OpenInfra Edge Computing Group - Openstack-Ansible - OpenStack Ansible Modules - OpenStack Charms - OpenStack-Helm - OpenStack Telemetry - QA - Scientific SIG - SDK/CLI - Security SIG - Skyline - StarlingX - Swift - Tacker - Technical Committee - TripleO - Venus We look forward to seeing you at PTG! If you have any questions, please email ptg(a)openstack.org <mailto:ptg@openstack.org>. Thanks! Ashlee [1] PTG Registration: https://www.openstack.org/ptg/ <https://www.openstack.org/ptg/> [2] Team Survey: https://openinfrafoundation.formstack.com/forms/oct2021_vptg_survey <https://openinfrafoundation.formstack.com/forms/oct2021_vptg_survey> [3] Ethercalc Signup: https://ethercalc.openstack.org/8tum5yl1bx43 <https://ethercalc.openstack.org/8tum5yl1bx43>

1 0

[glance] No meeting on 23rd September
by Abhishek Kekane 16 Sep '21

16 Sep '21

Hi All, We decided to cancel our next (September 23rd) weekly meeting. According to schedule we will meet directly on September 30th. In case of any queries, reach us on #openstack-glance IRC channel. Thank you, Abhishek

1 0

[oslo] next meeting canceled
by Herve Beraud 16 Sep '21

16 Sep '21

Hey folks, Our next meeting (September 20) will be canceled. I'll be away from the keyboard all the entire day, hence, I couldn't drive our meeting. According to our schedule our next meeting will be on October the 4th. Thanks for your understanding. See you soon, -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0

[oslo][docs] Request to release openstackdocstheme
by Camp, MaryX 16 Sep '21

16 Sep '21

Hi to the Oslo team, Can you please release the openstackdocstheme so it can be used by the StarlingX community? This will correct the double headings issue we see now on every doc page. We've made similar historical release requests for 2.2.4 [1], 2.2.5 [2], and 2.2.7 [3] The StarlingX docs site utilizes a theme variant within the openstackdocstheme repo, but consumes packaged releases. Several recent changes merged to address Sphinx 4.x changes, and the StarlingX community would appreciate a new openstackdocstheme release to make use of them. [1] 2.2.4 https://review.opendev.org/c/openstack/releases/+/737105 [2] 2.2.5 https://review.opendev.org/c/openstack/releases/+/741550 [3] 2.2.7 https://review.opendev.org/c/openstack/releases/+/765192 Thanks in advance! Mary Camp StarlingX Documentation Kelly Services Technical Writer | mailto:maryx.camp@intel.com

2 1

[neutron][networking-midonet] Propose to EOL stable/pike, stable/queens and stable/rocky branches of networking-midonet
by Lajos Katona 16 Sep '21

16 Sep '21

Hi, The older than rocky branches of networking-midonet are broken, as it seems that the last merge on stable/rocky and stable/queens happened a year ago, and even the master branch of networking-midonet is not maintained. Considering the above I propose to make stable/rocky, stable/queens, stable/pike branches End Of Life. I will wait until the end of next week for anyone who would like to maybe step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in networking-midonet. Lajos Katona (lajoskatona)

1 0

[nova]The code about BP Allow migrating PMEM's dat
by Juntingqiu Qiujunting (邱军婷) 16 Sep '21

16 Sep '21

Hi all: Is there anything in the code logic that needs to be modified? If there is no problem with the implementation structure of the code, I start adding unit tests. https://review.opendev.org/c/openstack/nova/+/802225 Thank you Fossen. --------------------------------- Fossen Qiu | 邱军婷 CBRD | 云计算与大数据研发部 T: 18249256272 E: qiujunting(a)inspur.com<mailto:qiujunting@inspur.com> 浪潮云海

2 1

Changing the default Network Quotas
by Karera Tony 16 Sep '21

16 Sep '21

Hello Team, Can anybody advise on how to change the Network Quotas for Openstack Wallaby deployed using Kolla-ansbile. Regards Tony Karera

2 2

Neutron
by Zeeshan Haider 16 Sep '21

16 Sep '21

**Scenario**: We are trying to develop our Virtual Network using `Open Virtual Network` or `OVN`, We need our Virtual Network Switches running firstly on the root Virtual Router, and there are some other Virtual Switches as well. I was Working on the `openvswitch` and `OVN` but a lot of things are missing for me, and there are some unlinked dots between the deployment itself and there are some problems. My Question has 3 further Parts. 1. Is it possible at all to use `Neutron` as our Network Manager without OpenStack cloud installation? 2. Is it a good approach? 3. if not then are there any cookbooks available for `OVN`, I am not talking about man-pages. Any help would be highly appreciated. P.S. I am thinking about this because OVN got merged with Neutron, and it could be one of the possibilities. **Note**: Please Don't downvote the question because I don't have that much experience with the network could be a stupid question, will remove it if you provide me the info in the comment section. I have found out that installation is possible I am quoting the answer from `https://bugs.launchpad.net/neutron` by `https://launchpad.net/~amotoki` below. "The way to install neutron is simple: to install neutron with pip (from PyPI package or the git repo), prepare a config file for neutron and start the neutron server (in case of OVN mechanism driver). Distro packages and/or various tools like Ansible playbook are just a wrapper for this. The minimum required config to enable noauth (ie without keystone) is to set auth_strategy to noauth in the config file. I don't know the detail instruction step by step to satisfy your need." Now I want to find out detailed instructions for this purpose. is this possible to get the instructions here

2 1

kubernetes cluster stuck in CREATE_IN_PROGRESS but stack list shows CREATE_COMPLETE
by Karera Tony 15 Sep '21

15 Sep '21

Dear Team, I have configured Magnum and I am able to kubernetes clusters and when I log in to the nodes everything is OK. But Minus the first cluster I created which completed well as per the openstack coe cluster list command and the output on Horizon, the rest of the clusters get stack in CREATE_IN_PROGRESS . Please check the below commands output and advise. (kolla-openstack) stack@deployment:~$ openstack stack list +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | e9bfad7c-c82e-4fc6-9f05-793b31631761 | k8s-cluster2-zsgfkrobjlw6 | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-15T08:01:50Z | None | | 9dde1665-8e50-4482-a986-810eb2a9cbaa | k8s-cluster1-eogcos7ee3ub | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-10T07:27:41Z | None | +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ (kolla-openstack) stack@deployment:~$ openstack coe cluster list +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status | +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | 277e0047-4ae1-4006-869e-a824cc602bcc | k8s-cluster1 | Newkey | 2 | 1 | CREATE_COMPLETE | HEALTHY | | 4336f9d9-9bea-477f-bc13-c16ff0f7e141 | k8s-cluster2 | Newkey | 2 | 1 | CREATE_IN_PROGRESS | None | +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ Regards Tony Karera

3 3

[all][tc] Technical Committee next weekly meeting on Sept 16th at 1500 UTC
by Ghanshyam Mann 15 Sep '21

15 Sep '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 16th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 15th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[horizon] Support for Django 3.x
by Thomas Goirand 15 Sep '21

15 Sep '21

Hi, It's been a long time that Django 3.x is around, and therefore, it'd be nice if Horizon had some kind of support for it. Indeed, sooner or later, Django 3.x will be uploaded to Debian unstable. That day, I hope Horizon doesn't just break. So, what's the status? Is anyone is at least working on it? Cheers, Thomas Goirand (zigo)

2 4

[TripleO][RDO] Plan for TripleO packages in RDO Xena release
by Alfredo Moralejo Alonso 15 Sep '21

15 Sep '21

Hi, With OpenStack Xena GA approaching and TripleO as an independent project, we need to clarify how we will handle it in RDO Xena Release. According to the spec [1]: "TripleO would no longer be able to deploy all versions of OpenStack. One idea that was brough forth in the discussions around this topic thus far, is that we can attempt to address this by designating a range of git tags as compatible with a particular OpenStack stable branch." In practical terms: - When will this list be provided? Can I assume shortly after Xena release (around mid october)? - How will this "list of tags compatible with Xena" be provided by the TripleO project? - How may we describe the maintenance status of TripleO in RDO Xena release? We'd like to include a statement in the release notes. Best regards, Alfredo https://specs.openstack.org/openstack/tripleo-specs/specs/xena/tripleo-inde…

1 0

Steps for Controller Node Removal in Victoria Version Openstack using ansible
by Adivya Singh 15 Sep '21

15 Sep '21

Hello Team, Is there any way , We can have a link or guide to decommission a controller node We have 3 controller nodes and 4 compute nodes, and we are planning to Scale down the Infrastructure by 1 controller and 2 compute nodes Regards Adivya Singh

1 1

LVM migration from one SAN storage to another New SAN Box
by KK CHN 15 Sep '21

15 Sep '21

List, I want to migrate our old SAN storage backend which contains LVMs used for the cloud storage, to a New SAN Storage box. The requirement was to migrate all storage of the old SAN box, which was mounted to the controller node as /dev/sdb and /dev/sdc around total 6TB storage. 80 % full. Then AMC of this box getting over. So what all are the steps to perform the migration of the old storage to new SAN storage . Any hints most welcome Kris

3 2

[cinder] Bug deputy report for week of 2021-15-09
by Sofia Enriquez 15 Sep '21

15 Sep '21

This is a bug report from 2021-08-09 to 2021-15-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/cinder/+bug/1943493 "backups.num_dependent_backups doesn't need to be stored in the db as an integer". Unassigned - https://bugs.launchpad.net/cinder/+bug/1934025 "Create a separate [quota] section for quota parameters". Unassigned - https://bugs.launchpad.net/cinder/+bug/1930290 "PowerMax - QoS should not be set on the parent storage group". Unassigned - https://bugs.launchpad.net/os-brick/+bug/1943615 "nvmeof connector gets device failing on ubuntu focal kernel". Unassigned - https://bugs.launchpad.net/os-brick/+bug/1888675 "fail to extend in-use fibre channel volume due to multipath-tools version" . Unassigned Low - https://bugs.launchpad.net/cinder/+bug/1943689 "[Doc] The interface of list snapshots and details supports volume filtering, but this is not documented". Unassigned - https://bugs.launchpad.net/cinder/+bug/1943682 "[Storwize] update rccg name to metadata for clone-group-volumes". Unassigned Invalid - https://bugs.launchpad.net/cinder/+bug/1927330 " Erroneous formula for calculation of virtual free capacity ". Unassigned -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

Need some clarification about tenant network in TripleO
by wodel youchi 15 Sep '21

15 Sep '21

Hi, It is a newbie question, but I didn't find a real answer, and I have trouble with my deployment, so some help will be appreciated. In the deployment of TripleO in network isolation, a Tenant network is created, that network (in the examples I have seen) have a VLAN-ID and a subnet. I did the same in my deployment, I created that network. My questions : - Is the Tenant network, mentioned in the deployment process, is it the first tenant network created for the admin tenant? and for new tenant I have to create a new network tenant, despite I am not seeing how I would do that? - Or is it a pipe (a conduit) used to pass the traffic of all the tenants that will be created after? Regards.

2 2

Cannot create a swift container on Horizon dashboard , mandatory "Storage Policy" dropdown field is empty
by Michel Niyoyita 15 Sep '21

15 Sep '21

Hello team , I am replacing swift by Ceph Radosgateway , and I am successful by creating containers through openstack and ceph CLI side . but once trying to create through the horizon dashboard I get errors: *Error: *Unable to fetch the policy details. , Unable to get the Swift service info and Unable to get the Swift container listing. Kindly help to solve the issue I deployed Openstack wallaby using kolla-ansible running on ubuntu 20.04 and ceph pacific using ansible running on ubuntu 20.04 Michel

1 0