Open Stack

Hi Dmitriy,
I tried the fix anyways, i.e. I replaced the value of swift_fallocate_reserve as
shown here:
https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/782117/1/defaults/main.yml

Now I'm getting the same error with '%%%' instead of '%%':

Mar 22 15:04:49 bc1bl12 systemd[1]: swift-account-reaper.service: Main process
exited, code=exited, status=1/FAILURE
Mar 22 15:04:49 bc1bl12 systemd[1]: swift-account-reaper.service: Failed with
result 'exit-code'.
Mar 22 15:04:51 bc1bl12 systemd[1]: swift-account-reaper.service: Scheduled
restart job, restart counter is at 171887.
Mar 22 15:04:51 bc1bl12 systemd[1]: Stopped swift-account-reaper service.
Mar 22 15:04:51 bc1bl12 systemd[1]: Started swift-account-reaper service.
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: Traceback (most recent
call last):
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]:   File
"/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py",
line 803, in config_fallocate_val>
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]:     reserve_value =
float(reserve_value[:-1])
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: ValueError: could not
convert string to float: '1%%'
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: During handling of the
above exception, another exception occurred:
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: Traceback (most recent
call last):
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]:   File
"/openstack/venvs/swift-22.1.0/bin/swift-account-reaper", line 23, in <module>
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]:
    run_daemon(AccountReaper, conf_file, **options)
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]:   File
"/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/daemon.py",
line 304, in run_daemon
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]:
    utils.config_fallocate_value(conf.get('fallocate_reserve', '1%'))
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]:   File
"/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py",
line 809, in config_fallocate_val>
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]:     raise
ValueError('Error: %s is an invalid value for fallocate'
Mar 22 15:04:52 bc1bl12 swift-account-reaper[322101]: ValueError: Error: 1%%% is
an invalid value for fallocate_reserve.

Kind regards,
Oliver
 
> Message: 3
> Date: Mon, 22 Mar 2021 12:06:19 +0200
> From: Dmitriy Rabotyagov <noonedeadpunk at ya.ru>
> To: "openstack-discuss at lists.openstack.org"
> 	<openstack-discuss at lists.openstack.org>
> Subject: Re: [glance][openstack-ansible] Snapshots disappear during
> 	saving
> Message-ID: <984511616407178 at mail.yandex.ru>
> Content-Type: text/plain; charset=utf-8
> 
> Well, looking into the fix I suggested, I'm not sure if it's valid one.
> There's really be a mess in patches, and according to the log provided, config
> needs just `1%` instead of the `1%%` you currently have.
> 
> And it feels like that's what default behaviour should do with [1]. But I'm
> pretty sure that this error was making swift fail and thus having weird issues
> while operating.
> 
> So I'm not sure what specifily wrong with value of swift_fallocate_reserve -
> maybe we've missed some config to define it or it has been overriden
> somewhere, but it feels like current default should cover issue you see in
> swift...
> 
> [1]
> https://opendev.org/openstack/openstack-ansible-os_swift/src/branch/stable/victoria/defaults/main.yml#L162
> 
> 22.03.2021, 11:56, "Dmitriy Rabotyagov" <noonedeadpunk at ya.ru>:
> > Yes, 1%% is smth we're fighting for years, as this setting changes on the
> > swift side from time to time, and I really lost account which one is valid
> > at the moment.
> >
> > Here's related SWIFT bug:
> >
> > https://bugs.launchpad.net/swift/+bug/1844368
> >
> > I've just pushed
> > https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/782117 to
> > cover this issue. Can you try applying this change manually to see if this
> > works?
> >
> > 22.03.2021, 11:25, "Oliver Wenz" <oliver.wenz at dhbw-mannheim.de>:
> >> Hi Dmitriy,
> >> thanks for your answer! Yes, we do use swift and its use as glance backend
> >> is
> >> intentional.
> >>
> >> I got the following from the swift-proxy-server logs in the swift container
> >> on
> >> the infra host after taking a snapshot:
> >>
> >> Mar 22 08:43:43 infra1-swift-proxy-container-27169fa7 proxy-server[87]:
> >> Client
> >> disconnected without sending last chunk (txn:
> >> txa7c64547baf0450eb0034-006058588b) (client_ip: 192.168.110.
> >> 106)
> >> Mar 22 08:43:43 infra1-swift-proxy-container-27169fa7 proxy-server[87]:
> >> 192.168.110.106 192.168.110.211 22/Mar/2021/08/43/43 PUT
> >> /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/3ec63ec2-aa3b-4c3b-a904-b55d1a6ec878
> >> -00001 HTTP/1.0 499 - python-swiftclient-3.10.1 gAAAAABgWFiKMz9R...
> >> 204800000 89
> >> - txa7c64547baf0450eb0034-006058588b - 52.2991 - - 1616402571.623875856
> >> 1616402623.922997952 0
> >>
> >> On the swift host some services logs contain errors. E.g. the
> >> swift-container-updater service:
> >>
> >> Mar 22 08:42:19 bc1bl12 systemd[1]: swift-container-updater.service: Main
> >> process exited, code=exited, status=1/FAILURE
> >> Mar 22 08:42:19 bc1bl12 systemd[1]: swift-container-updater.service: Failed
> >> with
> >> result 'exit-code'.
> >> Mar 22 08:42:21 bc1bl12 systemd[1]: swift-container-updater.service:
> >> Scheduled
> >> restart job, restart counter is at 162982.
> >> Mar 22 08:42:21 bc1bl12 systemd[1]: Stopped swift-container-updater
> >> service.
> >> Mar 22 08:42:21 bc1bl12 systemd[1]: Started swift-container-updater
> >> service.
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: Traceback (most
> >> recent
> >> call last):
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File
> >> "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py",
> >> line 803, in config_fallocate_value
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: reserve_value =
> >> float(reserve_value[:-1])
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: ValueError: could
> >> not
> >> convert string to float: '1%'
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: During handling of
> >> the
> >> above exception, another exception occurred:
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: Traceback (most
> >> recent
> >> call last):
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File
> >> "/openstack/venvs/swift-22.1.0/bin/swift-container-updater", line 23, in
> >> <module>
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]:
> >>     run_daemon(ContainerUpdater, conf_file, **options)
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File
> >> "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/daemon.py",
> >> line 304, in run_daemon
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]:
> >>     utils.config_fallocate_value(conf.get('fallocate_reserve', '1%'))
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File
> >> "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py",
> >> line 809, in config_fallocate_value
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: raise
> >> ValueError('Error: %s is an invalid value for fallocate'
> >> Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: ValueError: Error:
> >> 1%%
> >> is an invalid value for fallocate_reserve.
> >>
> >> The same exit-code and traceback shows in the logs of
> >> swift-container-auditor,
> >> swift-account-auditor and swift-account-reaper services. Does this tell you
> >> anything useful?
> >>
> >> We didn't experience any problems when uploading files to containers, only
> >> when
> >> taking snapshots of instances.
> >>
> >> Kind regards,
> >> Oliver
> >>
> >>>  ------------------------------
> >>>
> >>>  Message: 4
> >>>  Date: Thu, 18 Mar 2021 12:44:47 +0200
> >>>  From: Dmitriy Rabotyagov <noonedeadpunk at ya.ru>
> >>>  To: "openstack-discuss at lists.openstack.org"
> >>>          <openstack-discuss at lists.openstack.org>
> >>>  Subject: Re: [glance][openstack-ansible] Snapshots disappear during
> >>>          saving
> >>>  Message-ID: <374941616064157 at mail.yandex.ru>
> >>>  Content-Type: text/plain; charset=utf-8
> >>>
> >>>  Hi Olver,
> >>>
> >>>  Am I right that you're also using OpenStack Swift and it's intentional to
> >>>  store images there?
> >>>  Since the issue is related to the upload process into the Swift. So also
> >>>  checking Swift logs
> >>>  be usefull as well.
> >
> > --
> > Kind Regards,
> > Dmitriy Rabotyagov
> 
> 
> -- 
> Kind Regards,
> Dmitriy Rabotyagov
> 
> 
> 
> ------------------------------
> 
> Message: 4
> Date: Mon, 22 Mar 2021 12:35:37 +0100
> From: Bernard Cafarelli <bcafarel at redhat.com>
> To: OpenStack Discuss <openstack-discuss at lists.openstack.org>
> Subject: Re: [neutron][stable] neutron-lib stable branches core
> 	reviewers
> Message-ID:
> 	<CABHdKwrEN99b3VS89U0M8BRYevV2oqGLSPkSfVzdjCZNFoKHpA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> On Thu, 18 Mar 2021 at 14:48, Slawek Kaplonski <skaplons at redhat.com> wrote:
> 
> > Hi,
> >
> > I just noticed that neutron-lib project has got own group
> > "neutron-lib-stable-maint" which has +2 powers in neutron-lib stable
> > branches [1]. As I see now in gerrit that group don't have any members.
> > Would it be maybe possible to remove that group and add
> > "neutron-stable-maint" to the neutron-lib stable branches instead? If yes,
> > should I simply propose patch to change [1] or is there any other way which
> > I should do it?
> >
> I guess we never spotted that "neutron-lib-stable-maint" before, as it
> seems neutron cores had +2 powers before:
> https://review.opendev.org/c/openstack/neutron-lib/+/717093
> 
> But yes it could be useful to have either of these groups (neutron cores or
> stable cores) in, backports to neutron-lib are rare but some happen - I
> remember one some time ago where it was valid to have a patch backported.
> 
> 
> > [1]
> > https://github.com/openstack/project-config/blob/master/gerrit/acls/openstack/neutron-lib.config
> >
> > --
> > Slawek Kaplonski
> > Principal Software Engineer
> > Red Hat
> 
> 
> 
> -- 
> Bernard Cafarelli
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210322/9edd75a0/attachment.html>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> openstack-discuss mailing list
> openstack-discuss at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
> 
> 
> ------------------------------
> 
> End of openstack-discuss Digest, Vol 29, Issue 126
> **************************************************