Open Stack

Hello,
looking at destination kvm host I got the following in instance log under
/var/log/libvirt/qemu:
2021-03-15 11:48:31.996+0000: starting up libvirt version: 4.5.0, package:
36.el7_9.3 (CentOS BuildSystem <http://bugs.centos.org>,
2020-11-16-16:25:20, x86-01.bsys.centos.org), qemu version:
2.12.0qemu-kvm-ev-2.12.0-44.1.el7_8.1, kernel: 3.10.0-1160.15.2.el7.x86_64,
hostname: podto2-kvmae
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
QEMU_AUDIO_DRV=none \
/usr/libexec/qemu-kvm \
-name guest=instance-00002a52,debug-threads=on \
-S \
-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-73-instance-00002a52/master-key.aes
\
-machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \
-cpu
Broadwell-IBRS,vme=on,f16c=on,rdrand=on,hypervisor=on,arat=on,xsaveopt=on,abm=on
\
-m 4096 \
-realtime mlock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-uuid c6ea7ed2-e7ce-4df6-a767-6bb95ae8fdc6 \
-smbios 'type=1,manufacturer=RDO,product=OpenStack
Compute,version=17.0.11-1.el7,serial=3dec30fe-a31f-4ea6-971f-6f993589ef04,uuid=c6ea7ed2-e7ce-4df6-a767-6bb95ae8fdc6,family=Virtual
Machine' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=139,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \
-drive
file=/var/lib/nova/instances/c6ea7ed2-e7ce-4df6-a767-6bb95ae8fdc6/disk.config,format=raw,if=none,id=drive-ide0-0-0,readonly=on,cache=none
\
-device
ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,write-cache=on \
-drive
file=/var/lib/nova/mnt/7eb4b0178ee3ec9ad7cbbc20c62b1912/volume-d5c812c5-2c27-4e82-a38d-83fc79ab848e,format=raw,if=none,id=drive-virtio-disk0,serial=d5c812c5-2c27-4e82-a38d-83fc79ab848e,cache=none,aio=native
\
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on
\
-netdev tap,fd=141,id=hostnet0,vhost=on,vhostfd=142 \
-device
virtio-net-pci,host_mtu=9000,netdev=hostnet0,id=net0,mac=fa:16:3e:3f:e5:32,bus=pci.0,addr=0x3
\
-add-fd set=3,fd=144 \
-chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=143,server,nowait \
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
\
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:55 \
-k en-us \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-incoming defer \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-03-15 11:48:31.996+0000: Domain id=73 is tainted: high-privileges
2021-03-15T11:48:32.163025Z qemu-kvm: -chardev
pty,id=charserial0,logfile=/dev/fdset/3,logappend=on: char device
redirected to /dev/pts/57 (label charserial0)
2021-03-15T11:48:32.167206Z qemu-kvm: -drive
file=/var/lib/nova/mnt/7eb4b0178ee3ec9ad7cbbc20c62b1912/volume-d5c812c5-2c27-4e82-a38d-83fc79ab848e,format=raw,if=none,id=drive-virtio-disk0,serial=d5c812c5-2c27-4e82-a38d-83fc79ab848e,cache=none,aio=native:
'serial' is deprecated, please use the corresponding option of '-device'
instead
2021-03-15T11:48:37.779611Z qemu-kvm: Failed to load
virtio_pci/modern_queue_state:desc
2021-03-15T11:48:37.780020Z qemu-kvm: Failed to load
virtio_pci/modern_state:vqs
2021-03-15T11:48:37.780042Z qemu-kvm: Failed to load
virtio/extra_state:extra_state
2021-03-15T11:48:37.780062Z qemu-kvm: Failed to load virtio-balloon:virtio
2021-03-15T11:48:37.780082Z qemu-kvm: error while loading state for
instance 0x0 of device '0000:00:06.0/virtio-balloon'
2021-03-15T11:48:37.781465Z qemu-kvm: load of migration failed:
Input/output error
2021-03-15 11:48:38.231+0000: shutting down, reason=crashed
"instance-00002a52.log" 102L, 7122C

But hard resetting the vm it starts
Ignazio


Il giorno lun 15 mar 2021 alle ore 17:36 Ignazio Cassano <
ignaziocassano at gmail.com> ha scritto:

> Hello all,
> we are facing a problem while migrating some virtual machines on centos 7
> queens.
> If we create a new virtual machine it migrates.
> Old virtual machines got the following:
> 37eb1d0c4c - default default] [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] Increasing downtime to 50 ms after 0
> sec elapsed time
> 2021-03-15 17:26:44.428 82382 INFO nova.virt.libvirt.driver
> [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
> 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] Migration running for 0 secs, memory
> 100% remaining; (bytes processed=0, remaining=0, total=0)
> 2021-03-15 17:26:57.922 82382 INFO nova.compute.manager
> [req-83fd3d62-292e-4990-8f7a-47f404b557cc - - - - -] [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] VM Paused (Lifecycle Event)
> 2021-03-15 17:26:58.617 82382 INFO nova.virt.libvirt.driver
> [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
> 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] Migration operation has completed
> 2021-03-15 17:26:58.618 82382 INFO nova.compute.manager
> [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
> 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] _post_live_migration() is started..
> 2021-03-15 17:26:58.658 82382 ERROR nova.virt.libvirt.driver
> [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
> 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] Live Migration failure: operation
> failed: domain is not running: libvirtError: operation failed: domain is
> not running
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager
> [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
> 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] Post live migration at destination
> podto2-kvmae failed: InstanceNotFound_Remote: Instance
> 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
> InstanceNotFound: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not
> be found.
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last):
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6505, in
> _post_live_migration
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     instance, block_migration, dest)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 783, in
> post_live_migration_at_destination
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     instance=instance,
> block_migration=block_migration)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174,
> in call
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     retry=self.retry)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131,
> in _send
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     timeout=timeout, retry=retry)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 559, in send
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     retry=retry)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 550, in _send
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     raise result
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound_Remote: Instance
> 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last):
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166,
> in _process_incoming
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     res =
> self.dispatcher.dispatch(message)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line
> 220, in dispatch
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     return
> self._do_dispatch(endpoint, method, ctxt, args)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line
> 190, in _do_dispatch
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     result = func(ctxt, **new_args)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in
> wrapped
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     function_name, call_dict, binary)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     self.force_reraise()
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     six.reraise(self.type_,
> self.value, self.tb)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in
> wrapped
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     return f(self, context, *args,
> **kw)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1000, in
> decorated_function
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     return function(self, context,
> *args, **kwargs)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in
> decorated_function
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     return function(self, context,
> *args, **kwargs)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6658, in
> post_live_migration_at_destination
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     'destination host.',
> instance=instance)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     self.force_reraise()
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     six.reraise(self.type_,
> self.value, self.tb)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6653, in
> post_live_migration_at_destination
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     block_device_info)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7844,
> in post_live_migration_at_destination
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     self._host.get_guest(instance)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 526, in
> get_guest
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     return
> libvirt_guest.Guest(self._get_domain(instance))
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]   File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 546, in
> _get_domain
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]     raise
> exception.InstanceNotFound(instance_id=instance.uuid)
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound: Instance
> 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572]
> 2021-03-15 17:27:05.179 82382 WARNING nova.compute.resource_tracker
> [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca
> 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] Instance not resizing, skipping
> migration.: InstanceNotFound_Remote: Instance
> 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found.
> 2021-03-15 17:27:13.621 82382 INFO nova.compute.manager [-] [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] VM Stopped (Lifecycle Event)
> 2021-03-15 17:27:13.711 82382 INFO nova.compute.manager
> [req-0dcd030c-d520-4223-91a6-f2fbc9f2444a - - - - -] [instance:
> 25fbb55c-5991-49b2-885f-26adebaeb572] During the sync_power process the
> instance has moved from host podto2-kvmae to host podto2-kvm01
>
>
> The istances was on podto1-kvm01 and we try to migrate it on podto2-kvmae.
> It stops to migrate and goes in error state and stop to responding to ping
> requests.
> From logs is seems it returned on podto2-kvm01 but it is in error state on
> podto2-kvmae.
> After hard rebooting it starts on podto2-kvmae and then we can migrate it
> where we want without errors.
> Please, any help ?
> Thanks
> Ignazio
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210315/54e41afe/attachment-0001.html>