- openstack-discuss - lists.openstack.org

[nova][neutron][cyborg] Bandwidth (and accel) providers are broken if CONF.host is set
by Eric Fried 28 Nov '19

28 Nov '19

Hi all. Sean mentioned a "downstream bug" in IRC today [1], which we discussed a little bit, but without gibi; and then Matt and I discussed it more later [2], but without gibi *or* Sean. Since I don't know if there's a bug report I can comment on, I wanted to summarize here for now so I don't forget. The Problem =========== Neutron needs to know the compute node resource provider on which to hang the child providers for QoS bandwidth. Today it assumes CONF.host is the name of that provider. That's wrong. The name of the provider is `hypervisor_hostname`, which for libvirt [3] happens to match the *default* value of CONF.host [4]. Per the bug Sean describes, if you override CONF.host, neutron won't find the compute node provider, and things break. The problem will be the same for any non-nova wishing to discover the compute node RP -- e.g. cyborg for purposes of creating child providers for accelerators. The Right Solution ================== Neutron (and any $service) should look up the compute node provider by its UUID. That's returned by the /os-hypervisors APIs after microversion 2.53, e.g. [5], but, Catch-22, you can currently only filter those results on hypervisor_hostname. So you would have to e.g. GET /os-hypervisors/detail and then walk the list looking for service.host matching CONF.host. That's way heavy for your CERNs. So the proposal moving forward is to add (in a new microversion) a ?service_host=XXX qparam to those APIs to let you filter down to just the one entry for your CONF.host. The UUID of that entry will also be the UUID of the compute node resource provider. (At that point you don't even need to ask Placement for that provider; you can just use that UUID directly in the APIs that create the child providers. Yay, you got your extra API call back.) Now, that's not backportable, and this problem exists in stable releases (at least those that support QoS bandwidth). So we should totally do it, but we also need... The Backportable Solution ========================= Neutron should use `gethostname()` rather than CONF.host to discover the compute node resource provider. I don't consider this a viable permanent solution because it is tightly coupled to knowing that hypervisor_hostname == `gethostname()`, which happens to be true for libvirt, but not necessarily for other drivers. We can get away with it for stable because we happen to know that we're only supporting QoS bandwidth via Placement for libvirt. Upgrade Concerns ================ Matt and I didn't nail down whether neutron and compute are allowed to be at different versions on a given host, or what those are allowed to be. But things should be sane if neutron (or any $service) logics like this in >=ussuri: if new_nova_microversion_available: do_the_os_hypervisors_thing() elif using_new_non_libvirt_feature: raise YouCantDoThisWithOldNova() else: do_the_gethostname_thing() Action Summary ============== If the above sounds reasonable, it would entail the following actions: - Neutron(/Cyborg?): backportable patch to s/CONF.host/socket.gethostname()/ - Nova: GET /os-hypervisors*?service_host=X in a new microversion. - Neutron/Cyborg: master-only patch to do the logic described in `Upgrade Concerns`_ (though for now without the `elif` branch). Thanks, efried [1] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.… [2] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.… [3] https://opendev.org/openstack/nova/src/commit/1cd5563f2dd2b218db2422397c8aa… [4] https://opendev.org/openstack/nova/src/commit/1cd5563f2dd2b218db2422397c8aa… [5] https://docs.openstack.org/api-ref/compute/?expanded=list-hypervisors-detai…

6 19

Re: [lists.openstack.org代发][nova][ptg] Support re-configure deleted_on_termination in server
by Brin Zhang(张百林) 28 Nov '19

28 Nov '19

Hi Sylvain, Matthew: https://review.opendev.org/#/c/693828/ Here is the PoC code, please review this change, thanks. Sylvain Bauza [mailto:sbauza@redhat.com] Items: [lists.openstack.org代发][nova][ptg] Support re-configure deleted_on_termination in server Spec is https://review.opendev.org/#/c/580336/ Most people seem to think this makes sense but realize there are already other ways to do this (snapshot) and therefore it's not totally necessary. The agreement in the room was to post the code up for the change, as this will help sell people on it if it's trivial enough and document the use case (i.e. are there scenarios where this would make life 10x easier?)

1 0

[kolla] kolla-ceph future
by Michał Nasiadka 28 Nov '19

28 Nov '19

Hello, Ceph deployment functionality in Kolla-Ansible has been deprecated in Train (https://review.opendev.org/#/c/669214/1) Given the fact that Ceph deployment was (and is) important functionality in Kolla-Ansible - I’d like to propose following steps: 1) Improve external Ceph functionality in Kolla-Ansible (configurable keyrings, users, improved documentation) - in progress now 2) Replace current kolla-ceph CI jobs with an equivalent using ceph-ansible (and possibly ceph orchestrator after Octopus is out around March 2020) 3) Remove kolla-ceph functionality in Ussuri release from Kolla-Ansible code base - or (if there are volunteers) extract it as a separate repository with a separate set of maintainers/core team. 4) (OPTIONAL) Find a volunteer to work on ceph-ansible/ceph-orchestrator existing deployment takeover functionality for ceph-kolla (current core team has no spare cycles to work on that) Essentially this mail is a call for volunteers for 3) - if there is somebody that is interested in picking up the code and maintaining it - please speak up. If there are none - we will start preparing for removing the code in 2020. Thank you for your understanding in this not easy decision. Kind regards, Michal Nasiadka

1 0

[olso][pbr][i18n] Using setup.cfg [files] data_files to install localization files
by info＠dantalion.nl 27 Nov '19

27 Nov '19

Hello everyone :), I was wondering what the preferred method to install localization files is. I can think of some probably solutions such as: 1: Including the locale files as part of a package for the target systems package manager (pacman, yum, apt, etc). 2: adding the locale files to the [files] directive in setup.cfg: I hope someone can answer my question. Kind regards, Corne Lukken

3 5

Re: [keystone][nova][barbican][neutron][cinder][tc][policy] Proposal for policy popup team⁩
by yumeng bao 27 Nov '19

27 Nov '19

>> Please nominate Yumeng Bao (yumeng_bao(a)yahoo.com) as the liaison with >> Cyborg team. She will contribute the spec. We have informed Howard of >> this. > Thanks Sundar, I have replaced Howard with Yumeng as the liaison for cyborg. > (Side note - the wiki can be edited by anyone, and I am tracking changes in it, so anyone may feel free to change their project liaison or add or remove team members and I will be notified of the change.) > Colleen Thanks Colleen for the update and the note reminder ! I will update the cyborg team progress on wiki paget later. Regards, Yumeng

1 0

[meta][K8s][API][Extended Maintenance][Operation Docs] Change SIG to Advisory status
by Rico Lin 27 Nov '19

27 Nov '19

Dear SIG members and chairs To follow the discussion in Meta SIG PTG room [1]. I would like to propose to change the following SIGs to Advisory status [2] to represent the SIG stays around for provide help, make sure everything stays working and provide advice when needed. - K8s SIG - API SIG - Extended Maintenance SIG - Operation Docs SIG If you think your SIG should not belong to `advisory` statue. Please advise from the following statuses: - active: SIG reaches out for discussion and event, have plans for the current cycle, host meetings or send ML out regularly. - forming: SIG still setting up. - advisory: SIG stays around for help, make sure everything stays working and provide advice when needed. - complete: SIG completes its mission. If that sounds correct, I need at least one chair from each SIG +1 on [2], so we can make sure it's what SIGs agreed on. [1] https://etherpad.openstack.org/p/PVG-meta-sig [2] https://review.opendev.org/#/c/695625/ -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

3 4

[meta-sig] Add Thierry Carrez to core reviewer
by Rico Lin 27 Nov '19

27 Nov '19

I would like to propose adding Thierry to core reviewer for governance-sigs I don't think anyone will doubt his knowledge of maintaining governance-sigs repo. He will be able to do core review on all patches. To create SIG patch, it will require both UC and TC to give their blessing. That means besides Amy's +2, we only need Thierry or mine +2 to approve the workflow. Let me know what you think if you have other thoughts, otherwise, I will add him in core think week. -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

2 2

[keystone][nova][barbican][neutron][cinder][tc][policy] Proposal for policy popup team
by Colleen Murphy 27 Nov '19

27 Nov '19

Hi, At the Shanghai forum, we discussed forming a popup team around reforming default policies for several projects. I've formally proposed this team here: https://review.opendev.org/695993 I've also created a wiki page to document the effort and coordinate the work: https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… This is the best way I can think of to organize a cross-project initiative like this, since I think the openstack specs repo is kind of defunct? I'm open to other ideas for coordinating this. I've gone ahead and named project-specific liaisons based on discussions at the forum, and also created a member list based on the list of volunteers in the forum etherpad. If you weren't at the forum but would like to be involved, please go ahead and add your name to the list on the wiki page. To form the popup team, we need a second co-lead: please let me know if you're interested. As hinted in the wiki page, I'm not seeking to lead this long-term, so there are actually two lead positions open. Let me know your thoughts in this thread. Colleen

5 8

[oslo] Adoption of microversion-parse
by Thierry Carrez 27 Nov '19

27 Nov '19

Hi Osloites, A recent review of the OpenStack namespace yielded a library that no project team owns, and yet is specific to openstack and present in global-requirements: https://opendev.org/openstack/microversion-parse Ideally it should be adopted by one of our project teams. After discussing it with cdent, and since it is openstack-specific and depended on by most openstack projects, it feels like Oslo would be a good home for it. It would follow an "independent" release model and I don't think it would trigger much updates or maintenance. I can post patches to make it happen if you confirm you're OK to host it. Just keep me posted :) Cheers, -- Thierry Carrez (ttx)

7 13

[security] Security SIG Meeting Tomorrow Canceled
by Gage Hugo 27 Nov '19

27 Nov '19

Just a reminder that the security sig meeting tomorrow is cancelled, hope everyone has a great holiday weekend, and we will be back next week!

1 0

[ops] registration now open for OpenStack Ops Metup, Bloomberg London jan 7,8 2020
by Chris Morgan 27 Nov '19

27 Nov '19

Registration is now open for the OpenStack Ops Meetup at Bloomberg London, January 7,8 2020: https://go.bloomberg.com/attend/invite/openstack-operators-meetup-london202… Please sign up if attending. Agenda still under formation via normal community process (link on sign-up page). See you there! Chris -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[neutron][L3][OVN] add a time slot for OVN L3 functionality in the L3 meeting
by LIU Yulong 27 Nov '19

27 Nov '19

Hi all, Neutron team recently merged the convergence ML2+OVS and OVN [1], it was also discussed during the PTG in Shanghai [2]. And more about that is the train is now getting started [3][4]. So during the L3 meeting, we may want to add a new section in L3 meeting to talk about some OVN L3 related topics, such as feature gaps, bugs, new ideas or anything needs to draw the attention of the neutron team. OVN is also a long-developed project with various functions. So, an independent L3 meeting time looks appropriate, we cannot cover all the OVN stuff in one meeting. So if you are interested in OVN and some L3 stuff, do not be hesitant, please come to the L3 meeting with your thoughts, ideas or questions. The Neutron L3 Sub-team Meeting [5] will be held weekly on Wednesday at 14:00 UTC in irc channel #openstack-meeting. [1] http://specs.openstack.org/openstack/neutron-specs/specs/ussuri/ml2ovs-ovn-… [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010702… [3] https://blueprints.launchpad.net/neutron/+spec/neutron-ovn-merge [4] https://review.opendev.org/#/q/topic:bp/neutron-ovn-merge [5] http://eavesdrop.openstack.org/#Neutron_L3_Sub-team_Meeting Regards, LIU Yulong

1 0

[largescale-sig] First meeting summary and next actions
by Thierry Carrez 27 Nov '19

27 Nov '19

Hi everyone, As announced in [1], the Large Scale SIG had its first IRC meeting today. You can access summary and logs at: http://eavesdrop.openstack.org/meetings/large_scale_sig/2019/large_scale_si… This meeting was mostly focused on agreeing on team logistics, and an early discussion on short-term objectives for the group. As a result of this meeting I created: - Reference information about the SIG at https://wiki.openstack.org/wiki/Large_Scale_SIG - A change formally proposing the SIG at https://review.opendev.org/696302 - Two etherpads to further refine our two initial ideas for SIG short-term goals: Scaling within one cluster, and instrumentation of the bottlenecks: https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling Document large scale configuration and tips &tricks: https://etherpad.openstack.org/p/large-scale-sig-documentation Between now and next meeting, the idea is to validate those initial documents, get the SIG formally set up, and refine those goals on those two etherpads, ideally splitting them into early objectives and initial steps. Current proposal is to have our next meeting on Dec 18. Belmiro is unfortunately not available on that week, but the week before amorin and myself are not available, and the week(s) after are the end-of-year holidays for some of us, so that is still probably the best date before next year. Cheers, [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011062… -- Thierry Carrez (ttx)

1 0

[Kolla] FAILED - RETRYING: wait for MariaDB to be available via HAProxy (10 retries left).
by Bhupathi, Ramakrishna 27 Nov '19

27 Nov '19

Mark, Thanks for the tip. Now I get a failure (as below) during deploy (I have all-in-one) config. I am on stein. It appears to me that this has been presumably fixed . But I still running into this error. I do not see anything explicit in the HAProxy/MariaDB logs. TASK [mariadb : wait for MariaDB to be available via HAProxy] ************************************************************************************************** FAILED - RETRYING: wait for MariaDB to be available via HAProxy (10 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (9 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (8 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (7 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (6 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (5 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (4 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (3 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (2 retries left). FAILED - RETRYING: wait for MariaDB to be available via HAProxy (1 retries left). fatal: [localhost]: FAILED! => {"attempts": 10, "changed": false, "elapsed": 60, "msg": "Timeout when waiting for search string MariaDB in 10.1.0.250:3306"} NO MORE HOSTS LEFT ********************************************************************************************************************************************* PLAY RECAP ***************************************************************************************************************************************************** localhost : ok=76 changed=0 unreachable=0 failed=1 skipped=74 rescued=0 ignored=0 Command failed ansible-playbook -i ../../all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /home/cloud-user/kolla-ansible/ansible/site.yml --RamaK -----Original Message----- From: Mark Goddard [mailto:mark@stackhpc.com] Sent: Monday, November 25, 2019 11:08 AM To: Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi(a)charter.com> Cc: openstack-discuss(a)lists.openstack.org Subject: Re: [Kolla] RabbitMQ failure during deploy (Openstack with Kolla) On Mon, 25 Nov 2019 at 13:54, Bhupathi, Ramakrishna <C-Ramakrishna.Bhupathi(a)charter.com> wrote: > > Hey , I am evaluating Openstack with Kolla (have the latest ) and following the steps. I see that rabbitmq fails to start up and (keeps restarting). Essentially the script fails in deploy. I have a single node all in one config. > > > > Can someone tell me what the cause for this failure? > > > > > > RUNNING HANDLER [rabbitmq : Waiting for rabbitmq to start on first > node] > ********************************************************************** > ****************************************** > > fatal: [localhost]: FAILED! => {"changed": true, "cmd": "docker exec > rabbitmq rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbitmq.pid", > "delta": "0:00:07.732346", "end": "2019-11-25 13:33:12.102342", "msg": > "non-zero return code", "rc": 137, "start": "2019-11-25 > 13:33:04.369996", "stderr": "", "stderr_lines": [], "stdout": "Waiting > for 'rabbit@kolla-ubuntu'\npid is 6", "stdout_lines": ["Waiting for > 'rabbit@kolla-ubuntu'", "pid is 6"]} > > > > RUNNING HANDLER [rabbitmq : Restart rabbitmq container (rest of > nodes)] > ********************************************************************** > ******************************************* > > > > NO MORE HOSTS LEFT > ********************************************************************** > ********************************************************************** > ************************** > > > > PLAY RECAP > ********************************************************************** > ********************************************************************** > ********************************** > > localhost : ok=95 changed=2 unreachable=0 failed=1 skipped=78 rescued=0 ignored=0 > > > > Command failed ansible-playbook -i ./all-in-one -e > @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e > CONFIG_DIR=/etc/kolla -e kolla_action=deploy > /home/ubuntu/venv/share/kolla-ansible/ansible/site.yml > > > > > > Accessing the container logs for rabbitmq .. this is what I see > > > > BOOT FAILED > > =========== > > > > Error description: > > {could_not_start,rabbitmq_management, > > {rabbitmq_management, > > {bad_return, > > {{rabbit_mgmt_app,start,[normal,[]]}, > > {'EXIT', > > {{could_not_start_listener, > > [{port,15672}], > > {shutdown, > > {failed_to_start_child,ranch_acceptors_sup, > > > {listen_error,rabbit_web_dispatch_sup_15672,eaddrinuse}}}}, Hi RamaK, here is your issue. RabbitMQ management fails to start up due to the port already being in use (eaddrinuse). Perhaps RabbitMQ is running on the host already? > > {gen_server,call, > > [rabbit_web_dispatch_registry, > > {add,rabbit_mgmt, > > [{port,15672}], > > #Fun<rabbit_web_dispatch.0.49789095>, > > > > --RamaK > > The contents of this e-mail message and any attachments are intended > solely for the > addressee(s) and may contain confidential and/or legally privileged > information. If you are not the intended recipient of this message or > if this message has been addressed to you in error, please immediately > alert the sender by reply e-mail and then delete this message and any > attachments. If you are not the intended recipient, you are notified > that any use, dissemination, distribution, copying, or storage of this > message or any attachment is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

3 3

Communication problem between ironic-python-agent and CI server.
by Guannan GN2 Sun 27 Nov '19

27 Nov '19

Hi team, I'm now trying to use ironic deployed with devstack to manage baremetal machine. However when it run into deploying stage, I open the BM server terminal and see it successfully load ramdisk and boot into it. It get the ip I assigned and I can ping it from CI server side. But it then deploy failed just about 2 minutes later. When I check ironic-conductor log with command "sudo journalctl -a --unit devstack@ir-cond" and found error like this: ERROR ironic.drivers.modules.agent_client [None req-de37bc21-8d62-41db-8983-c06789939818 None None] Failed to connect to the agent running on node ea88ba26-756d-4d32-89f4-7ff086fa8868 for invoking command iscsi.start_iscsi_target. Error: HTTPConnectionPool(host='10.0.0.25', port=9999): Max retries exceeded with url: /v1/commands?wait=true (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f5461027f10>, 'Connection to 10.0.0.25 timed out. (connect timeout=60)')): ConnectTimeout: HTTPConnectionPool(host='10.0.0.25', port=9999): Max retries exceeded with url: /v1/commands?wait=true (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f5461027f10>, 'Connection to 10.0.0.25 timed out. (connect timeout=60)')) I can ping it from CI server side, so it is strange why the connection time out between ironic-python-agent and CI server. Does anyone meet similar problem or have idea about it? Thank you! Best Regards, Guannan

3 3

[tc][board][all] - Adding OpenStack community support to the savedotorg campaign
by Graham Hayes 26 Nov '19

26 Nov '19

Hey All, I am not sure if this has been seen by everyone or not - but there is a change in how the .org top level domain is being ran in the works, in a way that may not be in the best interests of the non profits that it was created to facilitate. [1] A lot of well known non profits have already joined, and as a community that has an interest in the internet as a whole, and uses a .org domain, I think we should add our voice in support. What do people think? Are we happy to have the TC use its voice on behalf of the OpenStack project, or do we think the board should use its voice on behalf of the entire foundation? - Graham 1 - https://savedotorg.org/

7 8

Freezer Project Update
by Deepa 26 Nov '19

26 Nov '19

Hello Team Good Day I am Deepa from Fingent Global Solutions and we are a big fan of Openstack and we do have 4 + openstack setup (including production) We have deployed Openstack using juju and Maas .So when we check for backup feasibility other than cinder-backup we were able to see Freezer Project. But couldn't find any charms for it in juju charms. Also there isn't a clear documentation on how to install freezer . https://docs.openstack.org/releasenotes/freezer/train.html. No proper release notes in the latest version as well. Can you please tell me whether this project is in developing state? Whether charms will be added to juju in future. Can you also share a proper documentation on how to install Freezer in cluster setup. Thanks for your help. Regards, Deepa K R

3 6

manila share group replication support
by Dong.Ding＠dell.com 26 Nov '19

26 Nov '19

Hi, experts, I'm trying to implement the share replication feature for EMC Unity Manila driver. Unity isn't capable to promote a single share. The share must be promoted together with its share server. The problem is that we have several shares exported from one single server. So, promoting a share will cause all shares being promoted. My question is that is there any solution to promote several shares as they are in a group? I was trying to find something useful about 'manila group replication' ,only find Ocata Doc mentioned it, but no detail information: https://specs.openstack.org/openstack/manila-specs/specs/ocata/manila-share… And there is no code or commit history matches 'group replication' in Manila. Do you have any suggestions for our situation? Thanks, Ding Dong

2 2

Re: [nova] [cyborg] Impact of moving bind to compute
by Nadathur, Sundar 26 Nov '19

26 Nov '19

Hi, We had a thread [1] on this subject from May of this year. The preference was that "ARQ creation happens in conductor and binding happens in compute" [2]. The ARQ binding involves device preparation and FPGA programming, which may take a while. So, it is done asynchronously. It is desirable to kickstart the binding ASAP, to maximize the overlap with other tasks needed for VM creation. We wound up doing all of binding in the compute for the following reason. If we call Cyborg to initiate ARQ binding and then wait for the notification event, we may miss the event if it comes in the window in between. So we had to call wait_for_instance_event() and, within its scope, call Cyborg for binding. This logic moved everything to compute. But now we are close to having an improved wait_for_instance_event() [3]. So I propose to: A. Start the binding in the conductor. This gets maximum concurrency between binding and other tasks. B. Wait for the binding notification in the compute manager (without losing the event). In fact, we can wait inside _build_resources, which is where Neutron/Cinder resources are gathered as well. That will allow for doing the cleanup in a consistent manner as today. C. Call Cyborg to get the ARQs in the virt driver, like today. Please LMK if you have any objections. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006541.html [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006979.html [3] https://review.opendev.org/#/c/695985/ Regards, Sundar

3 4

[scientific-sig] Meeting today, 2100UTC - Supercomputing, networking, monitoring, and more
by Stig Telfer 26 Nov '19

26 Nov '19

Hi All - We have an IRC meeting coming up at 2100 UTC (about 2.5 hours time) in channel #openstack-meeting. Everyone is welcome. Today’s agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_November_29th_20… <https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_November_29th_20…> It’s pretty packed today, we have a wash-up from SC to discuss, plus an update on Linuxbridge/ML2, plus an update on mixed virt and baremetal, plus a discussion on telemetry after Gnocchi. Loads to cover! Cheers, Stig

1 0

[sahara] Cancelling Sahara meeting November 28, and looking ahead to December 12
by Jeremy Freudberg 26 Nov '19

26 Nov '19

Hi all, Due to the Thanksgiving holiday in the US there will be no Sahara meeting 2019-11-28. Additionally I will be on PTO the date of the following meeting 2019-12-12. I may establish a special meeting for 2019-12-05 depending on interest. Thanks, Jeremy

1 0

Re: [ironic]: Timeout reached while waiting for callback for node
by fsbiz＠yahoo.com 26 Nov '19

26 Nov '19

Thanks Julia. In addition to what you mentioned this particular issue seems to have cropped up when we added 100 more baremetal nodes. I've also narrowed down the issue (TFTP timeouts) when 3-4 baremetal nodes are in "deploy" state and downloading the OS via iSCSI. Each iSCSI transfer takes about 6 Gbps and thus with four transfers we are over our 20Gbps capacity of the leaf-spine links. We are slowly migrating to iPXE so it should help. That being said is there a document on large scale ironic design architectures?We are looking into a DC design (primarily for baremetals) for upto 2500 nodes. thanks,Fred, On Wednesday, October 23, 2019, 03:19:41 PM PDT, Julia Kreger <juliaashleykreger(a)gmail.com> wrote: Greetings Fred! Reply in-line. On Tue, Oct 22, 2019 at 12:47 PM fsbiz(a)yahoo.com <fsbiz(a)yahoo.com> wrote: [trim] TFTP logs: shows TFTP client timed out (weird). Any pointers here? Sadly this is one of those things that comes with using TFTP. Issues like this is why the community tends to recommend using ipxe.efi to chainload as you can perform transport over TCP as opposed to UDP where in something might happen mid-transport. tftpd shows ramdisk_deployed completed. Then, it reports that the client timed out. Grub does tend to be very abrupt and not wrap up very final actions. I suspect it may just never be sending the ack back and the transfer may be completing. I'm afraid this is one of those things you really need to see on the console what is going on. My guess would be that your deploy_ramdisk lost a packet in transfer or that it was corrupted in transport. It would be interesting to know if the network card stack is performing checksum validation, but for IPv4 it is optional. [trim] This has me stumped here. This exact failure seems to be happening 3 to 4 times a week on different nodes.Any pointers appreciated. thanks,Fred.

3 3

[tc][stable] Changing stable branch policy
by Mohammed Naser 26 Nov '19

26 Nov '19

Hi everyone, At the PTG, the TC discussed what we can do about our stable branch policy and there was a few different ideas put on the table, however, something that I felt made a lot of sense was revisiting the way that we currently apply it. We all know that we're definitely a lot more resource limited as a community and it's important for us to start breaking down some of those ideas which made sense when the velocity of the project was very high. One of the things that used to make sense is maintaining a dedicated stable core team across all projects. At the current time: 1. Some projects seem to have some sort of power of their stable branches through historical reasons 2. Some projects don't have access to merging things into stable branches and need to rely on the stable maintenance team to do that 3. We are *really* thankful for our current stable team, but it seems that there is a lot of work that does bottleneck other teams (and really, stable reviews is a difficult task). The proposal that I had was that in mind would be for us to let teams self manage their own stable branches. I think we've reached a point where we can trust most of our community to be familiar with the stable branch policy (and let teams decide for themselves what they believe is best for the success of their own projects). I'd like to invite the community to comment on this change, the approach that we can take to do this (or other ideas) -- being mindful with the limited set of resources that we have inside the community. Thanks, Mohammed

12 26

[magnum] Kubernetes, multiple versions: kube-system pods in pending status on newly spun-up clusters (since last Thursday)
by Namrata Sitlani 26 Nov '19

26 Nov '19

Hello folks, As of last week (14 Nov), our Magnum (Rocky) environment has stopped spinning up working Kubernetes clusters. To be precise, Magnum does report the cluster status as CREATE_COMPLETE, but once it is up all its Kubernetes pods are stuck in the Pending state. We use the following commands to create the Kubernetes clusters: http://paste.openstack.org/show/786348/. All pods show pending status which shows cluster could not select a minion node for them. The deployment fails with the following output " 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate" For reference: http://paste.openstack.org/show/786717/. For reference output of kubectl get all -A http://paste.openstack.org/show/786729/ If we manually remove NoSchedule taint from the minion nodes we get all the pods running For reference: http://paste.openstack.org/show/786718/ But after the manual fix too openstack-cloud-controller-manager pods are missing so any interaction from the Kubernetes control plane to OpenStack services is non-functional. We are assuming that the missing openstack cloud controller manager pod is also the reason for the taint issue which we are encountering. For the node taint issue, https://ask.openstack.org/en/question/120442/magnum-kubernetes-noschedule-t… suggests to add [trust] cluster_user_trust = true to magnum.conf. But there is OSA variable named magnum_cluster_user_trust that can be set to true for this purpose. However, the default for this variable has been True and we have confirmed we have the parameter in our environment cluster_user_trust=True as well. Note: The kubernetes cluster deployed here uses kube_tag 1.14.8 but we are getting same result with other kube_tag versions v1.13.12, v1.14.8 also. Ideally, we should not remove taints manually so can you please confirm our findings and help us find a way forward We can provide more logs if needed Thanks Namrata Sitlani

1 0

[kolla][openstack-ansible][tripleo] CentOS 8 host migration
by Mark Goddard 26 Nov '19

26 Nov '19

Hi, During the recent kolla PTG discussions [1], we covered CentOS 8 in some detail. For kolla there are several aspects to CentOS 8 support: * kolla-build execution host * container images * kolla-ansible execution host * kolla-ansible remote hosts In this context I am discussing the last of these - kolla-ansible remote hosts, i.e. the OS on the controllers and computes etc. Given that this could be quite a thorny problem, I'd like to propose that we collaborate between deployment projects. It is my understanding that upgrades from CentOS 7 to 8 will not be supported, and a reinstall is required. We set out some goals for the migration: * Migrate hosts from CentOS 7 to CentOS 8 * Upgrade from Train CentOS 7 containers to Ussuri CentOS 8 containers * Note: this means Ussuri containers (CentOS 8) don't land on CentOS 7 * Decouple the CentOS 7 to 8 and OpenStack upgrades, to avoid operator pain * Avoid excessive downtime There is a Red Hat policy [2] that suggests that if you mix host and container OS versions, it is safer for the host to be ahead of the container. This makes sense if you think about userland accessing kernel features. This leads us to the following migration path: * Start with Train release, CentOS 7 containers * Redeploy (rolling) hosts with CentOS 8 * Provision * Configure host OS * Redeploy Train containers with CentOS 7 * Upgrade to Ussuri release, CentOS 8 containers The logic we used to come up with this procedure is as follows: * Host must be upgraded to CentOS 7 before containers (going on policy) * Train containers don't support CentOS 8 base due to RDO * Ussuri containers don't support CentOS 7 base due to RDO * Want to separate CentOS 7 -> 8 migration from OpenStack upgrade This leads us to the conclusion that at least one release of kolla-ansible must bridge these worlds, since we will be deploying Train containers on CentOS 8. We could do either of: 1. Train k-a supports CentOS 8 hosts 2. Ussuri k-a supports deploying Train containers While 1. may require us to backport more than we would like, supporting deployment of multiple OpenStack releases could be challenging, so we expect to go with 1 here. There are some risks involved, including: * some Train services may not work on CentOS 8 * ceph? * OVS & neutron agents? * libvirt? * Ansible minimum version 2.8 for CentOS 8 (ref?) Hopefully we can save ourselves some effort and solve this together. [1] https://etherpad.openstack.org/p/kolla-ussuri-ptg [2] https://access.redhat.com/support/policy/rhel-container-compatibility Thanks, Mark

3 4