Open Stack

On 02/04/2018 10:44 PM, Qiming Teng wrote:
> Starting about 24 hours ago, we have been notified CI gate failure
> although we haven't changed anything to our project locally. Before that
> we have spent quite some time making the out-of-tree tempest plugins
> work on gate.
>
> After checking the log again and again ... we found the following logs
> from Keystone:
>
> Feb 05 03:31:12.609492 ubuntu-xenial-ovh-gra1-0002362092
> devstack at keystone.service[24845]: WARNING keystone.common.wsgi [None
> req-dfcbf106-fbf5-41bd-9012-3c65d1de5f9a None admin] Could not find
> project: service.: ProjectNotFound: Could not find project: service.
>
> Feb 05 03:31:13.845694 ubuntu-xenial-ovh-gra1-0002362092
> devstack at keystone.service[24845]: WARNING keystone.common.wsgi [None
> req-50feed46-7c15-425d-bec7-1b4a7ccf6859 None admin] Could not find
> service: clustering.: ServiceNotFound: Could not find service:
> clustering.
>
> Feb 05 03:31:12.552647 ubuntu-xenial-ovh-gra1-0002362092
> devstack at keystone.service[24845]: WARNING keystone.common.wsgi [None
> req-0a5e660f-dad6-4779-aea4-dd6969c728e6 None admin] Could not find
> domain: Default.: DomainNotFound: Could not find domain: Default.
>
> Feb 05 03:31:12.441128 ubuntu-xenial-ovh-gra1-0002362092
> devstack at keystone.service[24845]: WARNING keystone.common.wsgi [None
> req-7eb9ed90-28fc-40aa-8a41-d560f7a156c9 None admin] Could not find
> user: senlin.: UserNotFound: Could not find user: senlin.
>
> Feb 05 03:31:12.336572 ubuntu-xenial-ovh-gra1-0002362092
> devstack at keystone.service[24845]: WARNING keystone.common.wsgi [None
> req-19e52d02-5471-49a2-8acd-360199d8c6e0 None admin] Could not find
> role: admin.: RoleNotFound: Could not find role: admin.
>
> Feb 05 03:28:33.797665 ubuntu-xenial-ovh-gra1-0002362092
> devstack at keystone.service[24845]: WARNING keystone.common.wsgi [None
> req-544cd822-18a4-4f7b-913d-297716418239 None admin] Could not find
> user: glance.: UserNotFound: Could not find user: glance.
>
> Feb 05 03:28:29.993214 ubuntu-xenial-ovh-gra1-0002362092
> devstack at keystone.service[24845]: WARNING py.warnings [None
> req-dc411d9c-6ab9-44e3-9afb-20e5e7034f12 None admin]
> /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:865:
> UserWarning: Policy identity:create_endpoint failed scope check. The
> token used to make the request was project scoped but the policy
> requires ['system'] scope. This behavior may change in the future where
> using the intended scope is required
>
> Feb 05 03:28:29.920892 ubuntu-xenial-ovh-gra1-0002362092
> devstack at keystone.service[24845]: WARNING keystone.common.wsgi [None
> req-32a4a378-d6d3-411e-9842-2178e577af27 None admin] Could not find
> service: compute.: ServiceNotFound: Could not find service: compute.
These are all WARNINGS messages. If this is a tempest run, these are
probably from negative testing [0], in which case keystone is doing the
correct thing. The warnings you've pasted are also present in successful
tempest runs [1]. Can you provide a link to a patch that's failing? What
project do you work on?

[0]
https://github.com/openstack/tempest/blob/master/tempest/api/identity/admin/v3/test_projects_negative.py
[1]
http://logs.openstack.org/57/540557/2/check/tempest-full/bbd7cdd/controller/logs/screen-keystone.txt.gz?level=WARNING

>
> ....
>
> ----------------------
>
> So I'm wondering what the hack happened? Keystone version bump?
> Devstack changed? Tempest settings changed? 
> Why are we merging these changes near the end of a cycle when people are
> focusing on stabilizing things?
The original feature freeze date was 10 days ago [2] and with the
condition of the gate during that time, there were several projects
trailing with merging feature. Keystone was one of them and we issued
feature freeze exceptions for those efforts [3] [4] [5]. Based on the
warnings you've reported, I'm not convinced any of those efforts are
affecting CI in a negative way, especially since we're still getting
support into tempest to test those features.

[2] https://releases.openstack.org/queens/schedule.html#q-keystone-ffreeze
[3]
http://lists.openstack.org/pipermail/openstack-dev/2018-January/126587.html
[4]
http://lists.openstack.org/pipermail/openstack-dev/2018-January/126588.html
[5]
http://lists.openstack.org/pipermail/openstack-dev/2018-January/126589.html
> Any hints on these are highly appreciated.
>
> - Qiming
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180205/5ad1026a/attachment.sig>