November 2019 - openstack-discuss

[kolla][tripleo] Infra style images
by Marcin Juszkiewicz 26 Nov '19

26 Nov '19

One of things we have on a list of things to do during Ussuri cycle is implementation of 'infra' images. BP: https://blueprints.launchpad.net/kolla/+spec/infra-images # What are infra images? Images that are always built from binary packages (or are Java monsters). We have about 70 such ones from quick check. All those Ceph, Prometheus, MariaDB, cron, chrony, storm, sensu etc ones. ## libvirt image There is 'nova-libvirt' image. Contains libvirt daemon (with qemu and all required packages) so it would get renamed to 'libvirt'. # Building The idea is that 'infra' will be a new build type (like we have 'binary', 'source' etc). With all source base images marked as unbuildable so there will be no 'debian-infra-nova-compute' one. On the other hand building of 'binary'/'source' type images would lock out all 'infra' ones to not get images with names like 'debian-source-ceph-mon'. # Pros - Clean split between OpenStack components (binary/source) and infrastructure needed to get them running (infra). - Less images to publish on CI. Infra ones can be built weekly as they do not change much. - No more questions how did we built ceph-mon from source ;D # Cons - We need to change kolla-ansible, tripleo and maybe some other projects' code to use new type of images. - Migration from previous releases would be more complicated due to image renames.

2 1

openrc issue with keystone LDAP integration
by Satish Patel 26 Nov '19

26 Nov '19

I am running openstack stein and very strange issue going on, life was good until today when i finish my keystone + LDAP integration with multi-domain setup and all role assignment in SQL. when today one of user complained that his openrc isn't working correctly, look like something openrc doesn't like about LDAP integration, but same user can access everything from Horizon. my ldap domain is "eng" and here is my openrc file. # COMMON OPENSTACK ENVS export OS_ENDPOINT_TYPE=internalURL export OS_INTERFACE=internalURL export OS_USERNAME=spatel export OS_PASSWORD='MyLDAPPassword123' export OS_PROJECT_NAME=eng export OS_TENANT_NAME=eng export OS_AUTH_TYPE=password export OS_AUTH_URL=http://172.28.16.9:5000/v3 export OS_NO_CACHE=1 export OS_USER_DOMAIN_NAME=eng export OS_PROJECT_DOMAIN_NAME=eng export OS_REGION_NAME=RegionOne # For openstackclient export OS_IDENTITY_API_VERSION=3 export OS_AUTH_VERSION=3 [root@openstack ~]# source spatel.rc [root@openstack ~]# nova list ERROR (Unauthorized): The request you have made requires authentication. (HTTP 401) (Request-ID: req-5877deee-b8be-4b21-9ff6-855ae43e268e) but if i take same openrc file and add "admin" account it and "default" domain then it works so don't know why it doesn't like LDAP creds?

1 0

[keystone] Upcoming team meetings and PTGs
by Colleen Murphy 26 Nov '19

26 Nov '19

Hi team, Tomorrow, October 29, we are having our virtual pre-PTG meeting from 14:00-17:30 UTC in lieu of our regular 16:00 IRC meeting. Tuesday, November 5 some of us will be in Shanghai for the Forum and I will not be available to chair the meeting, so I propose we skip it. Tuesday, November 12, we will have our virtual post-PTG meeting from 15:00-17:30 UTC in lieu of our regular 16:00 IRC meeting. Details about the PTG sessions can be found on the planning etherpad: https://etherpad.openstack.org/p/keystone-shanghai-ptg Colleen

1 2

[ops] Running VMware atop of OpenStack (eg: esxi)
by Paul Belanger 26 Nov '19

26 Nov '19

Greetings, I wanted to ask if anybody in the community is running VMware a top of OpenStack in any capacity? In Ansible we are doing a POC as part of our testing platform and running into some common ops issue. For example, what do people usually do for things like using config-drive? Is there any specific tooling you us to customize esxi images to run atop OpenStack. Bascially, looking for humans to bounce questions off and see who else is doing it. Thanks! Paul

3 5

[keystone] PTG recap
by Colleen Murphy 26 Nov '19

26 Nov '19

The keystone team didn't formally meet at the IRL PTG in Shanghai, but we had virtual meetings before and after the event to discuss the upcoming cycle. Below is a summary of what we discussed, more detailed notes were kept in the main etherpad[1] and sub-pads linked from there. [1] https://etherpad.openstack.org/p/keystone-shanghai-ptg Cycle Retrospective ------------------- https://etherpad.openstack.org/p/keystone-train-retrospective We started the pre-PTG with our usual end-of-cycle retrospective. One outcome of this was reevaluating the effectiveness of structured weekly office hours: we weren't having them frequently because often no one proposed a topic for the meeting, so we decided to replace the weekly-but-only-sometimes face-to-face meetings with once per month bug squash meetings. We're also going to experiment with having a weekly rotation of team members who will act as the point person for helping triage incoming bugs and responding to users with support questions, in an effort to create some structure to spread the load of ensuring users have a positive encounter with the team. Although we tried to have milestone-ly checkpoints on feature work, that did not seem to be enough to keep up momentum throughout the cycle on important features, so we'll try to have more frequent checkpoints, more precise deadlines, and better decomposition of tasks. Policy ------ https://etherpad.openstack.org/p/PVG-keystone-forum-policy The purpose of the policy discussion was mainly to prepare for the forum session happening the following week, so most of the result is captured by the forum recap[2]. This was the discussion during which the idea of forming a pop-up team to work through an initial set of projects, rather than creating a community goal and attempting to get all projects done in one cycle, was proposed. We had also discussed creating some kind of "policy-ready" tool to help operators figure out which projects had completed their migration, but this seemed only marginally more useful than simply keeping a manually updated list. (We discussed other ideas for operator tooling at the forum.) [2] http://www.gazlene.net/shanghai-forum-ptg.html Limits ------ https://etherpad.openstack.org/p/keystone-ussuri-ptg-unified-limits We didn't have any design discussion on Unified Limits, since nothing much has changed since it was discussed at the last PTG. The only issue is that no one has been working on it in the last cycle, so the next steps were simply to followup with stakeholders and PoC authors, reassess the priority of the work, and reinstigate the work (this is now in progress). SCIM Extension -------------- We discussed a proposal to add the SCIM protocol[3 as an API extension. The desire is to be able to sync user state between the keystone SQL database (including shadow users) and an external identity provider. The main rationale is to be able to sync the state of the users in an external IdP with the state of the cloud resources they own, so that, for example, decommissioned users would not have orphaned cloud resources. However, resources are actually owned by keystone projects, not users, and there is a many-to-many relationship between users and projects, so there is no direct link between the state of a user in the keystone database and the state of a resource they touched. Moreover, although keystone has a v3-ext section in its API reference[4], we don't effectively support API extensions because we don't have a discovery mechanism for them, so for the sake of interoperability we discourage adding config-enabled API extensions. We suggested that this feature could instead potentially be implemented as a proxy service in front of keystone, and if such an implementation turned out to be infeasible, we could revisit adding this to keystone at a later time. [3] https://tools.ietf.org/html/rfc7644 [4] https://docs.openstack.org/api-ref/identity/v3-ext/ Cycle Planning -------------- https://tree.taiga.io/project/keystone-ussuri-roadmap/kanban This cycle will be lighter in keystone-specific feature work, as much of our focus will be on cross-project goals. The only new spec to be proposed for Ussuri will be the Alembic migration[5] which has been in the backlog for a while. We'll also continue work on unfinished specs from Train[6][7]. We're also accounting for time needed to work on the community goals[8], as well as improving our documentation and improving the CLI. [5] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/backlog/… [6] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/backlog/… [7] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/backlog/… [8] https://governance.openstack.org/tc/goals/selected/ussuri/index.html

1 0

[requirements][taskflow] networkx bump is blocked on what looks to be a taskflow issue.
by Matthew Thode 25 Nov '19

25 Nov '19

I've made a test review so people can see and test against, but it looks like the bump from 2.3 to 2.4 is causing some issues. hits nova,neutron,octavia,glance,cinder -- Matthew Thode

4 3

[oslo] Virtual PTG Planning
by Ben Nemec 25 Nov '19

25 Nov '19

Hi Osloers, Given that a lot of the team was not in Shanghai and we had a few topics proposed that didn't make sense to discuss as a result, I would like to try doing a virtual PTG the way a number of the other teams are. I've added a section to the PTG etherpad[0] with some proposed details, but in general I'm thinking we meet on Jitsi (it's open source) around the time of the Oslo meeting. It's possible we might be able to get through everything in the regularly scheduled hour, but if possible I'd like to keep the following hour (1600-1700 UTC) open as well. If everyone's available we could do it next week (the 18th) or possibly the following week (the 25th), although that runs into Thanksgiving week in the US so people might be out. I've created a Doodle poll[1] with selections for the next three weeks so please respond there if you can make it any of those days. If none of them work well we can discuss alternative options. Thanks. -Ben 0: https://etherpad.openstack.org/p/oslo-shanghai-topics 1: https://doodle.com/poll/8bqiv865ucyt8499

3 5

[keystone] Keystone Team Update - Week of 18 November 2019
by Colleen Murphy 25 Nov '19

25 Nov '19

# Keystone Team Update - Week of 18 November 2019 ## Meta I haven't done one of these updates in a while because of travel and $life. This week there were a few things worth sending a newsletter about, so here you go. However, as many of you are aware, my job focus is changing and I need to be more strategic and selective about the activities I put my time into. These weekly updates consume a not-insignificant amount of my time, and so from now on I'll not plan on continuing it. Many people have given me the feedback that they find this newsletter useful, and so I encourage anyone who has some pulse on the keystone team's activities to take up this weekly summary (need not be a core). The template is linked at the bottom of this email, and I've been using Lance's launchpad scripts[1] to generate bug statistics and some oneliners[2] to scrape Gerrit data. [1] https://github.com/lbragstad/launchpad-toolkit [2] https://gist.github.com/cmurphy/ee802fc0dc4bf57dffbea02265cc9e92 ## News ### PTG Recap We concluded our second virtual PTG session this week. Notes were kept in the etherpad[3] and I will send out a summary some time next week. [3] https://etherpad.openstack.org/p/keystone-shanghai-ptg ### Documentation for Upgrade Issues As operators start upgrading to Train[4] it's becoming clear that some of the changes we made last cycle aren't well documented and we need better tooling and documentation to help deal with them, especially immutable roles and deprecated policies. [4] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-k… ### User Support and Bug Duty To help share the responsibility of ensuring users and operators have a positive experience when coming to the keystone team for help, we started a weekly rotation of support duty, organized by etherpad[5]. Being on-duty means being responsible for triaging new bugs, and responding to support requests on the mailing list and on IRC. It does not mean having to solve people's problems right away, only ensuring that they know someone is listening and cares about their issues. If you would like to help, feel free to add your name to the list - you need not be a keystone developer, just willing to help triage user issues, confirm bug reports, ask for more information, and help give pointers to resources that may help. We'll proceed roughly down the bulletted list but will take time at each IRC meeting to confirm the next person on-duty. [5] https://etherpad.openstack.org/p/keystone-l1-duty ### Unified Limits Update Work on the Unified Limits initiative[6] had stalled out last cycle but work on the oslo.limit implementation, which was the main thing blocking further progress in Nova or other projects, has been reignited this week[7]. Thanks John for picking this back up! [6] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/ongoing/… [7] https://review.opendev.org/#/q/project:openstack/oslo.limit+is:open ### No Meeting Next Week Next week is the Thanksgiving holiday in the US so we'll skip the team meeting (Tuesday November 26). ## Open Specs Ussuri specs: https://bit.ly/2XDdpkU Ongoing specs: https://bit.ly/2OyDLTh ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 3 changes this week. ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 32 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ## Bugs This week we opened 2 new bugs and closed 1. Bugs opened (2) Bug #1853170 (oslo.policy:High) opened by Ben Nemec https://bugs.launchpad.net/oslo.policy/+bug/1853170 Bug #1853038 (oslo.policy:Wishlist) opened by Ben Nemec https://bugs.launchpad.net/oslo.policy/+bug/1853038 Bugs closed (1) Bug #1852547 (keystone:Undecided) https://bugs.launchpad.net/keystone/+bug/1852547 ## Milestone Outlook https://releases.openstack.org/ussuri/schedule.html Specs for Ussuri must be proposed and ready to review by Milestone 1 which is in about three weeks. ## Shout-outs Thanks for Adam Harwell and Zachary Buhman for joining the keystone PTG and participating in the design discussion, John Garbutt for picking up the unified limits work, and everyone who signed up to help with bug duty! ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter

2 1

[Kolla] RabbitMQ failure during deploy (Openstack with Kolla)
by Bhupathi, Ramakrishna 25 Nov '19

25 Nov '19

Hey , I am evaluating Openstack with Kolla (have the latest ) and following the steps. I see that rabbitmq fails to start up and (keeps restarting). Essentially the script fails in deploy. I have a single node all in one config. Can someone tell me what the cause for this failure? RUNNING HANDLER [rabbitmq : Waiting for rabbitmq to start on first node] **************************************************************************************************************** fatal: [localhost]: FAILED! => {"changed": true, "cmd": "docker exec rabbitmq rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbitmq.pid", "delta": "0:00:07.732346", "end": "2019-11-25 13:33:12.102342", "msg": "non-zero return code", "rc": 137, "start": "2019-11-25 13:33:04.369996", "stderr": "", "stderr_lines": [], "stdout": "Waiting for 'rabbit@kolla-ubuntu'\npid is 6", "stdout_lines": ["Waiting for 'rabbit@kolla-ubuntu'", "pid is 6"]} RUNNING HANDLER [rabbitmq : Restart rabbitmq container (rest of nodes)] ***************************************************************************************************************** NO MORE HOSTS LEFT ********************************************************************************************************************************************************************** PLAY RECAP ****************************************************************************************************************************************************************************** localhost : ok=95 changed=2 unreachable=0 failed=1 skipped=78 rescued=0 ignored=0 Command failed ansible-playbook -i ./all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /home/ubuntu/venv/share/kolla-ansible/ansible/site.yml Accessing the container logs for rabbitmq .. this is what I see BOOT FAILED =========== Error description: {could_not_start,rabbitmq_management, {rabbitmq_management, {bad_return, {{rabbit_mgmt_app,start,[normal,[]]}, {'EXIT', {{could_not_start_listener, [{port,15672}], {shutdown, {failed_to_start_child,ranch_acceptors_sup, {listen_error,rabbit_web_dispatch_sup_15672,eaddrinuse}}}}, {gen_server,call, [rabbit_web_dispatch_registry, {add,rabbit_mgmt, [{port,15672}], #Fun<rabbit_web_dispatch.0.49789095>, --RamaK E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

2 1

[kolla] Forum project update slides
by Mark Goddard 25 Nov '19

25 Nov '19

Hi, During the PTG there were some requests for the slides from the Kolla project update forum session given by Surya and Jeffrey. https://docs.google.com/presentation/d/19Lj9WcSjjcs8f5ljL1r4HMIQFl4Nokepsij… Please get in touch if you can't reach the above and would like to see the slides. Thanks, Mark

1 0