- openstack-discuss - lists.openstack.org

[Octavia]-Seeking some high points on using Octavia
by Singh, Prabhjit 08 Sep '19

08 Sep '19

Hi Michael, I have been trying to get Octavia LbaaS up and running and get performance tested. It has taken me some time to get quite a few things working. While I continue to invest time in using Octavia and stay excited on some of the upcoming features. I have been asked the following questions by my leadership to which I do not have any direct answers. 1. What is the adoption of Octavia, are major organizations looking to adopt and invest in it. Can you provide some numbers 2. Roadmap wise is the Open community committed to investing in Octavia and why 3. Per your suggestion I tried to look up who are the primary companies using Octavia and haven't found a clear indication, any insight would be great. 4. Would features from haproxy 2.0 be included in Octavia 5. There are some open solutions from haproxy, Envoy, consul. How would Octavia compare. 6. Lastly, do you have enough encouragement to keep the project going, I guess I am looking for some motivation for continuing to choose to use Octavia when there are several turnkey solutions ( though offered at a price ). Currently I have been working with Redhat to answer the following questions, these are not for the community, hopefully Redhat will be able to pursue with your team. 1. How to offload logs to an external log/metrics collector 2. How to turn off logs during performance testing, I honestly do not want to do this because the performance tester is really generating live traffic which mimics a real time scenario. 3. How to set cron for rotating logs, I would think that this should be automatic. Would I need to do this everytime? 4. Do you have any way to increase performance of the amphora, my take is haproxy can handle several thousands of concurrent connections but in our case seems like we hit a threshold at 3500 - 4500 connections and then it starts to either send resets or the connections stay open for a long time. Thanks & Regards Prabhjit -----Original Message----- From: Singh, Prabhjit Sent: Tuesday, July 23, 2019 9:45 AM To: Michael Johnson <johnsomor(a)gmail.com> Cc: openstack-discuss(a)lists.openstack.org Subject: RE: [Octavia]-Seeking performance numbers on Octavia Thanks so much for the valuable insights Michael! Appreciate it and keep up the good work, as I ramp up with more dev know how hopefully I would start making contributions and can maybe convince my team to start as well. Thanks & Regards Prabhjit Singh -----Original Message----- From: Michael Johnson <johnsomor(a)gmail.com> Sent: Monday, July 22, 2019 5:48 PM To: Singh, Prabhjit <Prabhjit.Singh22(a)T-Mobile.com> Cc: openstack-discuss(a)lists.openstack.org Subject: Re: [Octavia]-Seeking performance numbers on Octavia [External] Hi Prabhjit, Comments in-line below. Michael On Sun, Jul 21, 2019 at 5:24 PM Singh, Prabhjit <Prabhjit.Singh22(a)t-mobile.com> wrote: > > Hi Michael, > > Thanks for taking the time out to send me your inputs and valuable suggestions. I do remember meeting you at the Denver Summit and hearing to a couple of your sessions. > If you wouldn't mind, I do have a few more questions and your answers would help me understand that should I continue to invest in having Octavia as one of our available LBs. > > 1. Based on your response and the amount of time you are investing in > supporting Octavia, what are some of the use cases, like for e.g. if load balancing web traffic how many transactions/connections minimum can be expected. I do understand you mentioned that it's hard to performance test Octavia but some real time situations from your testing and how customers have adopted Octavia would help me level set some expectations. This is really cloud and application specific. I would recommend you fire up an Octavia install and use your preferred tool to measure it. Some good tools are tsung, weighttp, and iperf3. > 2. We are thinking of Octavia as one of the offerings, that offers a self-serve type model. Do you know of any customers who have been able to use Octavia as one of their primary load balancers and any encouraging feedback you have gotten on Octavia. There are examples of organizations using Octavia available if you google Octavia. > 3. You suggested increasing the Ram size, I could go about making a whole new Flavor. Yes, to increase the allocated RAM for a load balancer, you would create an additional nova flavor with the specifications you would like. You can then either set this as the default nova flavor for amphora (amp_flavor_id is the setting) or you can create an Octavia flavor that specifies the nova compute flavor to use (See https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.open… for more information on Octavia flavors). > 4. I also noticed on the haproxy.conf the maxconns is set to 2000, should I increase this, does this affect the connection per server, which you said 64000 conns per server, so if I have 10 servers can I expect somewhere close to 640000 sessions? I think you are looking at the haproxy.conf file provided by your operating system package. Octavia does not use this file, it creates it's own HAProxy configuration files as needed under /var/lib/octavia inside the amphora. The default, if the user does not specify one at listener creation, is 1,000,000. > 5. Based on some of the limitations and the dev work in progress, I think the most important feature that would make Octavia a real solid offering would be the Active-Active and Autoscaling feature. I brought this up with you in our brief conversation at the summit, and you did mention that its not a top priority at this time and you are looking for some help. I have noticed a lot of documentation has been updated on this feature, do you think with the available document and progress I could spin up a distributor and manage sessions between Amphora or it's not complete yet. Active/Active is still on our roadmap, but unfortunately the people that were working on it had to stop for personal reasons. There may be some folks picking up this work again soon. At this point the Active/Active patches up for review are non-functional and still a work in progress. > 6. We have a Triple O setup, do you think I can make the above tweaks with the Triple O setup. I think you are able to make various adjustments to Octavia with Triple O, but I do not have specifics on that. > Thanks & Regards > > Prabhjit Singh > Systems Design and Strategy - Magentabox > | O: (973) 397-4819 | M: (973) 563-4445 > > > > -----Original Message----- > From: Michael Johnson <johnsomor(a)gmail.com> > Sent: Friday, July 19, 2019 6:00 PM > To: Singh, Prabhjit <Prabhjit.Singh22(a)T-Mobile.com> > Cc: openstack-discuss(a)lists.openstack.org > Subject: Re: [Octavia]-Seeking performance numbers on Octavia > > [External] > > > Hi Prabhjit, > > As you have mentioned, it is very challenging to get accurate performance results in cloud environments. There are a large number(very large in fact) of factors that can impact the overall performance of OpenStack and Octavia. > > In our OpenDev testing environment, we only have software emulation virtual machines available (Qemu running with the TCG engine) which performs extremely poorly. This means that the testing environment does not reflect how the software is used in real world deployments. > An example of this is simply booting a VM can take up to ten minutes on Qemu with TCG when it takes about twenty seconds on a real OpenStack deployment. > With this resource limitation, we cannot effectively run performance benchmarking test jobs on the OpenDev environment. > > Because of this, we don't publish performance numbers as they will not reflect what you can achieve in your environment. > > Let me try to speak to your bullet points: > 1. The Octavia team has never (to my knowledge) claimed the Amphora driver is "carrier grade". We do consider the Amphora driver to be "operator grade", which speaks to a cloud operator's perspective versus the previous offering that did not support high availability, have appropriate maintenance tooling, upgrade paths, performance, etc. > To me, "carrier grade" has an additional level of requirements including performance, latency, scale, and availability SLAs. This is not what the Octavia Amphora driver is currently ready for. That said, third party provider drivers for Octavia may be able to provide a "carrier grade" level of load balancing for OpenStack. > 2. As for performance tuning, much of this is either automatically handled by Octavia or are dependent on the application you are load balancing and your cloud deployment. For example we have many configuration settings to tune how many retries we attempt when interacting with other services. In performing and stable clouds, these can be tuned down, in others the defaults may be appropriate. If you would like faster failover, at the expense of slightly more network traffic, you can tune the health monitoring and keepalived_vrrp settings. We do not currently have a performance tuning guide for Octavia but would support someone authoring one. > 3. We do not currently have a guide for this. I will say with the version of HAproxy currently being shipped with the distributions, going beyond the 1vCPU per amphora does not gain you much. With the release of HAProxy 2.0 this has changed and we expect to be adding support for vertically scaling the Amphora in future releases. Disk space is only necessary if you are storing the flow logs locally, which I would not recommend for a performance load balancer (See the notes in the log offloading guide: > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.open…) > Finally, the RAM usage is a factor of the number of concurrent connections and if you are enabling TLS on the load balancer. For typical load balancing loads, the default is typically fine. However, if you have high connection counts and/or TLS offloading, you may want to experiment with increasing the available RAM. > 4. The source IP issue is a known issue > (https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstoryboar…) We have not prioritized addressing this as we have not had anyone come forward that they needed this in their deployment. If this is an issue impacting your use case, please comment on the story to that effect and provide a use case. This will help the team prioritize this work. > Also, patches are welcome! If you are interested in working on this issue, I can help you with information about how this could be added. > It should also be noted that it is a limitation of 64,000 connections per-backend server, not per load balancer. > 5. The team uses the #openstack-lbaas IRC channel on freenode and is happy to answer questions, etc. > > To date, we have had limited resources (people and equipment) available to do performance evaluation and tuning. There are definitely kernel and HAProxy tuning settings we have evaluated and added to the Amphora driver, but I know there is more work that can be done. If you are interested in help us with this work, please let us know. > > Michael > > P.S. Here are just a few considerations that can/will impact the performance of an Octavia Amphora load balancer: > > Hardware used for the compute nodes > Network Interface Cards (NICs) used in the compute nodes Number of > network ports enabled on the compute hosts Network switch > configurations (Jumbo frames, and so on) Cloud network topology > (leaf‐spine, fat‐tree, and so on) The OpenStack Neutron networking > configuration (ML2 and ML3 drivers) Tenant networking configuration > (VXLAN, VLANS, GRE, and so on) Colocation of applications and Octavia > amphorae Over subscription of the compute and networking resources > Protocols being load balanced Configuration settings used when > creating the load balancer (connection limits, and so on) Version of > OpenStack services (nova, neutron, and so on) Version of OpenStack > Octavia Flavor of the OpenStack Octavia load balancer OS and > hypervisor versions used Deployed security mitigations (Spectre, > Meltdown, and so on) Customer application performance Health of the > customer application > > On Fri, Jul 19, 2019 at 8:52 AM Singh, Prabhjit <Prabhjit.Singh22(a)t-mobile.com> wrote: > > > > Hi > > > > > > > > I have been trying to test Octavia with some traffic generators and > > my tests are inconclusive. Appreciate your inputs on the following > > > > > > > > It would be really nice to have some performance numbers that you guys have been able to achieve for this to be termed as carrier grade. > > Would also appreciate if you could share any inputs on performance > > tuning Octavia Any recommended flavor sizes for spinning up Amphorae, the default size of 1 core, 2 Gb disk and 1 Gig RAM does not seem enough. > > Also I noticed when the Amphorae are spun up, at one time only one > > master is talking to the backend servers and has one IP that its > > using, it has to run out of ports after 64000 TCP concurrent > > sessions, id there a way to add more IPs or is this the limitation > > If I needed some help with Octavia and some guidance around > > performance tuning can someone from the community help > > > > > > > > Thanks & Regards > > > > > > > > Prabhjit Singh > > > > > > > > > > > >

2 1

[tc] Campaign Question: Treating the Problem, not just the symptoms- Burnout, No Polling, etc
by Kendall Nelson 07 Sep '19

07 Sep '19

Hello :) Wanted to split the question Chris Dent asked here[1] into its own thread so people down the road and those tracking it now can find more easily. To kind of rephrase for everyone (Chris, correct me if I am wrong or not getting all of it): What do you think we, as a community, can do about the lack of candidates for roles like TC or PTL? How can we adjust, as a community, to make our governance structures fit better? In what wasy can we address and prevent burnout? I think JP[2] and Jay[3] already started to enumerate good ideas on the other thread, so to summarize/ expand/ add to their lists: - Reducing the number of TC members to 9, and maybe someday down to 7. When we were having polls for every election (maybe not every project) it was at a time where the electorate (and theoretically the number of possible candidates) was also huge. Since we have move past the hype curve and stabilized as a project, the number of polls we've (I say we because I used to be and still plan to help with elections) had to make have decreased. It seems to be a matter of proportions. - Continuing to improve education and onboarding process. Agreed 100%, but this should be an ongoing focus for everyone too- every contributor TC, PTL, or otherwise. The best way to get more people involved faster is a lower barrier to entry, but we all know that. Yes some things like gerrit and IRC are hard for people to get past and likely won't be changing for our community any time soon, but there are things like that with every community (I don't know if you have ever tried to push patches to k8s but their tagging of PRs is something they are working on making less complicated and better documented). Breaking down the onboarding process we have at the moment into smaller modules and clearly documenting the progression through those modules for new comers to easily find and work through is important. Also, though, having that be the only place that we, as a community, point to (meaning no duplicate information in multiple places like we have today) when new contributors have issues. - Better documentation of tribal knowledge. I proposed as a community goal for the U release[4], to formalize project specific onboarding information (some teams have already done this) and project specific guides for PTLs (I know we already have the broad strokes for all PTLs documented fairly well, but there's always project specific stuff) so that when there is a turn over mid release, its easier for someone new to step up. - Utilize the user survey to gather info about how/why contribution is happening or why they aren't contributing if that's the case. There are already several questions there from the TC about this topic in the survey, but perhaps they can be re-framed if we aren't getting the info we want from them. As a reminder, here they are: -- To which projects does your organization contribute maintenance resources, such as patches for bug fixes and code reviews on master or stable branches? -- What prevents you or your organization from contributing more maintenance resources, or makes contributing difficult? -- How do members of your organization contribute to OpenStack? I think the real issue is getting larger vendors of OpenStack to get their users to take the user survey. We have a pretty solid reach as it is, but there are a lot of people using OpenStack that don't take the survey that we don't know about even because they are confidential (their results can still be confidential if they take the survey). - Longer release cycle. I know this has come up a dozen or more times (and I'm a little sorry for bringing it up again), but I think OpenStack has stabilized enough that 6 months is a little short and now may finally be the time to lengthen things a bit. 9 months might be a better fit. With longer release cycles comes more time to get work done as well which I've heard has been a complaint of more part time contributors when this discussion has come up in the past. - Co-PTL type position? I've noticed and talked to several PTLs on a variety of projects that need a little extra help with the role. They either don't feel like they have all the experience they need to be PTL yet and so they want the previous PTL to help out still or maybe they want to do it, but there are enough variables in their day to day work (or lack of overlap tz wise with most of the other contributors to that project), that having a backup person to help out and backfill when they need help. - Talking to each other. I really honestly think just talking to one another could help too. When you find yourself in a conversation with someone about how unmotivated they are because they have a ton of work to do. You might offer to take something off their plate. Or help them see maybe they need to not take on anything new till some other work gets wrapped up. We are a community that succeeds together, so if you see someone burning themselves out do what you can to help lighten their load (helping directly is great, but there are plenty of other people in our community that you could call on to help too). Hopefully goes without saying, but don't burn yourself out trying to help someone else either. Some of these things are more actionable, others are still high level and need to have concrete actions tied to them, but I think there are plenty of things we can do to make progress here. -Kendall (diablo_rojo) [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-September/00908… [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-September/00908… [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-September/00910… [4] https://etherpad.openstack.org/p/PVG-u-series-goals

6 9

[neutron] CI issues
by Slawek Kaplonski 07 Sep '19

07 Sep '19

Hi neutrinos, We are currently having some issues in our gate. Please see [1], [2] and [3] for details. If Your Neutron patch failed on neutron-functional, neutron-functional-python27 or networking-ovn-tempest-dsvm-ovs-release jobs, please don’t recheck before all those issues will be solved. Recheck will not help and You will only use infra resources. [1] https://bugs.launchpad.net/neutron/+bug/1842659 [2] https://bugs.launchpad.net/neutron/+bug/1842482 [3] https://bugs.launchpad.net/bugs/1842657 — Slawek Kaplonski Senior software engineer Red Hat

1 1

[all][research] Survey for Openstack developers =)
by Farida El Zanaty 06 Sep '19

06 Sep '19

Hi!I am Farida from McGill University. I am trying to learn more about code reviews in the Openstack community, as I have been studying Openstack projects for a while. Please help me understand your perspective on design discussions during code reviews by filling up this 10-minute survey: https://forms.gle/Hhn191f6cxF5hVgG8 Survey participants will also be entered into a raffle for a $50 Amazon gift card. A little bit of context: Under the supervision of Prof. Shane McIntosh, my research aims to study design discussions that occur between developers during code reviews. Last year, we published a study about the frequency and types of such discussions that occur in OpenStack Nova and Neutron (http://rebels.ece.mcgill.ca/papers/esem2018_elzanaty.pdf<https://slack-redir.net/link?url=http%3A%2F%2Frebels.ece.mcgill.ca%2Fpapers…>).We are reaching out to Openstack developers to better understand their perspectives on design discussions during code reviews. Survey: https://forms.gle/Hhn191f6cxF5hVgG8<https://slack-redir.net/link?url=https%3A%2F%2Fforms.gle%2FHhn191f6cxF5hVgG8>Thanks for your time, Farida El-Zanaty =)

1 0

[all][tc] PDF Community Goal Update
by Akihiro Motoki 06 Sep '19

06 Sep '19

Hi all, I would like to share the status of the PDF community goal. We made great progresses these two weeks. Highlights ---------- It is time for working on the PDF community goal in your project! - openstack-tox-docs job supports PDF build now. AJaeger and I worked on the job support of PDF doc build and publishing and have completed it. - We confirmed PDF build with the sphinx latex builder works for several major projects like nova, cinder and neutron. The approach with the latex builder should work for most projects. - First PDF doc is now published from the i18n project. How to get started ------------------ - "How to get started" section in the PDF goal etherpad [1] explains the minimum steps. You can find useful examples there too. - To build PDF docs locally, you need to install LaTex related packages. See "To test locally" in the etherpad [1]. - If you hit problems during PDF build, check the common problems etherpad [2]. We are collecting knowledges there. - If you have questions, feel free to ask #openstack-doc IRC channel. Also Please sign up your name to "Project volunteers" in [1]. Useful links ------------ [1] https://etherpad.openstack.org/p/train-pdf-support-goal [2] https://etherpad.openstack.org/p/pdf-goal-train-common-problems [3] Ongoing reviews: https://review.opendev.org/#/q/topic:build-pdf-docs+(status:open+OR+status:… Thanks, Akihiro Motoki (amotoki)

9 15

[release][heat][nova][openstackclient][openstacksdk][nova] Pending final library releases
by Sean McGinnis 06 Sep '19

06 Sep '19

Hey everyone, Yesterday was the deadline for any non-client library releases. In order to make sure we include any unreleased commits, and to make sure those commits are able to make it onto a stable/train branch, releases were proposed by the Release Management team for any libs that looked like they needed it. We've had most of them +1'd by the team's PTL or release liaison and have been able to process those requests. There are still a few with no responses from the teams though. https://review.opendev.org/#/q/topic:train-3+status:open If you are a PTL or release liaison for one of the teams tagged in the subject line, please take a look and either +1 if things are ready, or if there are any last minute critical fixes going in, update the patches with a more appropriate commit hash to tag from. For any release requests not ack'd by the teams, we will need to proceed with these by Monday morning to make sure updates make it out and any dependency issues are flushed out before the client lib and other upcoming freezes. I will also submit patches to create the stable/train branch for any of these libs that have not already done so. If there are any questions or concerns about any of this, please reach out here or in the #openstack-release channel and we'll do what we can to help out. Thanks! Sean

1 0

[nova] Deprecating the XenAPI driver
by Matt Riedemann 06 Sep '19

06 Sep '19

After discussing this at the Train PTG and logging a quality warning in the driver 3 months ago [1] with no response, the nova team is now formally deprecating the XenAPI driver [2]. There has been no working third party CI for the driver for at least a release and no clear maintainers of the driver in nova anymore. If you're using the driver in production, please speak up now otherwise technically the driver could be removed as early as the Ussuri release. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006744.html [2] https://review.opendev.org/#/c/680732/ -- Thanks, Matt

1 0

[cinder][ops] Shanghai Forum - Cinder Topic Planning
by Brian Rosmaita 06 Sep '19

06 Sep '19

The Cinder Community would like to get in on some Forum action in Shanghai, but to do that we need to have some topics to propose. You don't have to actively be working on Cinder to propose a topic, and you don't have to be present to win. The point of the Forum sessions is to get feedback from operators and users about the current state of the software, get some ideas about what should be in the next release, and have some strategic discussion about The Future. So whether you can attend or not, if you have some ideas you'd like us to discuss, feel free to propose a topic: https://etherpad.openstack.org/p/cinder-shanghai-forum-proposals The deadline for proposals to the Foundation is 20 September, so if you could get your idea down on the etherpad before the Cinder weekly meeting on Wednesday 18 September 16:00 UTC, that will give the Cinder team time to look them over. thanks! brian

1 0

[tripleo] Proposing Damien Ciabrini as core on TripleO/HA
by Michele Baldessari 06 Sep '19

06 Sep '19

Hi all, Damien (dciabrin on IRC) has always been very active in all HA things in TripleO and I think it is overdue for him to have core rights on this topic. So I'd like to propose to give him core permissions on any HA-related code in TripleO. Please vote here and in a week or two we can then act on this. Thanks, -- Michele Baldessari <michele(a)acksyn.org> C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D

16 15

[goal][python3] Train unit tests weekly update (goal-1)
by Corey Bryant 06 Sep '19

06 Sep '19

This is the goal-1 weekly update for the "Update Python 3 test runtimes for Train" goal [1]. There is only 1 week remaining for completion of Train community goals [2]. == How can you help? == If your project has failing tests please take a look and help fix. Python 3.7 unit tests will be self-testing in Zuul. Failing patches: https://review.openstack.org/#/q/topic:python3-train +status:open+(+label:Verified-1+OR+label:Verified-2+) If your project has patches with successful tests please help get them merged. Open patches needing reviews: https://review.openstack.org/#/q/topic:python3 -train+is:open Patch automation scripts needing review: https://review.opendev.org/#/c/666934 == Ongoing Work == We're down to 8 projects with failing tests that need fixing, and 3 projects with successful tests that should be ready to merge. I've been working to contact PTLs for these projects to help finish them up. Thank you to all who have contributed their time and fixes to enable patches to land! == Completed Work == All patches have been submitted to all applicable projects for this goal. Merged patches: https://review.openstack.org/#/q/topic:python3-train +is:merged == What's the Goal? == To ensure (in the Train cycle) that all official OpenStack repositories with Python 3 unit tests are exclusively using the 'openstack-python3-train-jobs' Zuul template or one of its variants (e.g. 'openstack-python3-train-jobs-neutron') to run unit tests, and that tests are passing. This will ensure that all official projects are running py36 and py37 unit tests in Train. For complete details please see [1]. == Reference Material == [1] Goal description: https://governance.openstack.org/tc/goals/train/ python3-updates.html [2] Train release schedule: https://releases.openstack.org/train /schedule.html (see R-5 for "Train Community Goals Completed") Storyboard: https://storyboard.openstack.org/#!/story/2005924 Porting to Python 3.7: https://docs.python.org/3/whatsnew/3.7.html#porting-to-python-3-7 Python Update Process: https://opendev.org/openstack/governance/src/branch/master/resolutions/2018… Train runtimes: https://opendev.org/openstack/governance/src/branch/master/reference/runtim… train.rst Thanks, Corey

1 0

[monasca] Review Priority flag
by Witek Bedyk 06 Sep '19

06 Sep '19

Hello Team, now that we have the possibility to label our code changes with Review-Priority I would like to start the discussion about formalizing its usage. Right now every core reviewer can set its value, but we haven't defined any rules on how to use it. I suggest a process of proposing the changes which should be prioritized in weekly team meeting or in the mailing list. Any core reviewer, preferably from a different company, could confirm such proposed change by setting RV +1. I hope it's simple enough. What do you think? Another topic is exposing the prioritized code changes to the reviewers. We can list them using the filter [1]. We could add the link to this filter to Contributor Guide [2] and Priorities page [3]. We should also go through the list every week in the meeting. Any other ideas? Thanks Witek [1] https://review.opendev.org/#/q/(projects:openstack/monasca+OR+project:opens… [2] https://docs.openstack.org/monasca-api/latest/contributor/index.html [3] http://specs.openstack.org/openstack/monasca-specs/

2 1

[ironic] opensuse-15 jobs are temporary non-voting on bifrost
by Dmitry Tantsur 06 Sep '19

06 Sep '19

Hi all, JFYI we had to disable opensuse-15 jobs because they kept failing with repository issues. Help with debugging appreciated. Dmitry

2 2

[placement] update 19-35
by Chris Dent 06 Sep '19

06 Sep '19

HTML: https://anticdent.org/placement-update-19-35.html Let's have a placement update 19-35. Feature freeze is this week. We have a feature in progress (consumer types, see below) but it is not critical. # Most Important Three main things we should probably concern ourselves with in the immediate future: * We are currently without a PTL for Ussuri. There's some discussion about the options for dealing with this in an [email thread](http://lists.openstack.org/pipermail/openstack-discuss/2019-Septemb…. If you have ideas (or want to put yourself forward), please share. * We need to work on useful documentation for the features developed this cycle. * We need to create some [cycle highlights](http://lists.openstack.org/pipermail/openstack-discuss/2019-Sep…. To help with that I've started [an etherpad](https://etherpad.openstack.org/p/placement-train-cycle-highlights). If I've forgotten anything, please make additions. # What's Changed * osc-placement 1.7.0 has been [released](https://pypi.org/project/osc-placement/). This adds support for [managing allocation ratios](https://review.opendev.org/#/q/topic:allocation-ratios+(status:open… via aggregates, but adding a few different commands and args for inventory manipulation. * Work on consumer types exposed that placement needed to be first class in grenade to make sure database migrations are run. That [change has merged](https://review.opendev.org/679655). Until then placement was upgraded as part of nova. # Stories/Bugs (Numbers in () are the change since the last pupdate.) There are 24 (-1) stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 (0) are [untagged](https://storyboard.openstack.org/#!/worklist/580). 5 (0) are [bugs](https://storyboard.openstack.org/#!/worklist/574). 4 (0) are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 11 (-1) are [rfes](https://storyboard.openstack.org/#!/worklist/594). 4 (0) are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 17 (0). * Placement related nova [in progress bugs](https://goo.gl/vzGGDQ) on launchpad: 6 (0). # osc-placement * <https://review.opendev.org/666542> Add support for multiple member_of. There's been some useful discussion about how to achieve this, and a consensus has emerged on how to get the best results. * `--amend` and `--aggregate` on resource provider inventory has merged and been release 1.7.0 (see above). # Main Themes ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. * <https://review.opendev.org/#/q/topic:bp/support-consumer-types> I took this through to microversion and api-ref docs, so it is ready for wider review. If this doesn't make it in for Train, that's okay. The goal is to have it ready for Nova to start working with it when Nova is able. ## Cleanup Cleanup is an overarching theme related to improving documentation, performance and the maintainability of the code. The changes we are making this cycle are fairly complex to use and are fairly complex to write, so it is good that we're going to have plenty of time to clean and clarify all these things. Performance related explorations continue: * <https://review.opendev.org/#/c/679385/> Refactor initialization of research context. This puts the code that might cause an exit earlier in the process so we can avoid useless work. One outcome of the performance work needs to be something like a _Deployment Considerations_ document to help people choose how to tweak their placement deployment to match their needs. The simple answer is use more web servers and more database servers, but that's often very wasteful. # Other Placement Miscellaneous changes can be found in [the usual place](https://review.opendev.org/#/q/project:openstack/placement+status:op…. * <https://review.opendev.org/676982> Merge request log and request id middlewares is worth attention. It makes sure that _all_ log message from a single request use a global and local request id. There are three [os-traits changes](https://review.opendev.org/#/q/project:openstack/os-traits+status:… being discussed. And zero [os-resource-classes changes](https://review.opendev.org/#/q/project:openstack/os-resource-class…. # Other Service Users This week (because of feature freeze) I will not be adding new finds to the list, just updating what was already on the list. * <https://review.opendev.org/662229> helm: add placement chart * <https://review.opendev.org/#/q/topic:bp/virtual-persistent-memory> libvirt: report pmem namespaces resources by provider tree * <https://review.opendev.org/660852> Nova: Remove PlacementAPIConnectFailure handling from AggregateAPI * <https://review.opendev.org/670112> Nova: WIP: Add a placement audit command * <https://review.opendev.org/671793> Nova: libvirt: Start reporting PCPU inventory to placement A part of <https://review.opendev.org/#/q/topic:bp/cpu-resources> * <https://review.opendev.org/#/q/topic:bp/support-move-ops-with-qos-ports> Nova: support move ops with qos ports * <https://review.opendev.org/667952> nova: Support filtering of hosts by forbidden aggregates * <https://review.opendev.org/670696> tempest: Add placement API methods for testing routed provider nets * <https://review.opendev.org/672678> openstack-helm: Build placement in OSH-images * <https://review.opendev.org/674129> Correct global_request_id sent to Placement * <https://review.opendev.org/#/q/topic:bp/cross-cell-resize> Nova: cross cell resize * <https://review.opendev.org/674524> Nova: Scheduler translate properties to traits * <https://review.opendev.org/623558> Nova: single pass instance info fetch in host manager * <https://review.opendev.org/#/q/topic:bp/provider-config-file> Nova: using provider config file for custom resource providers # End 🐎 -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent

1 0

[keystone][horizon][zaqar][tempest][requirements] library updates breaking projects
by Matthew Thode 06 Sep '19

06 Sep '19

I emailed a while ago about problem updates and wanted to give an update. I'm hoping we can get these fixed before the freeze which is on Monday iirc. horizon This is a newer issue which e0ne and amotoki know about but no existing review to fix it. please test against https://review.opendev.org/680457 -semantic-version===2.8.1 +semantic-version===2.6.0 tempest STILL has failures I thought the following commit would fix it, but nope https://github.com/mtreinish/stestr/commit/136027c005fc437341bc23939a18a5f3… -stestr===2.5.1 +stestr===2.4.0 python-zaqarclient waiting on https://review.opendev.org/679842 may be merging today -jsonschema===3.0.2 +jsonschema===2.6.0 keystone a review is out there that seems to have tests passing https://review.opendev.org/677511/ -oauthlib===3.1.0 +oauthlib===3.0.2 -- Matthew Thode

2 1

[cloudkitty] Shift IRC meeting of september 6th
by Luka Peschke 06 Sep '19

06 Sep '19

Hello, Some CK cores are unavailable today, so we've decided to move today's meeting to next friday (the 13th) at 15h UTC / 17h CEST . Regards, Luka Peschke

1 0

[dev][tacker] Steps to setup tacker for testing VNF packages
by Uikey, Nitin 06 Sep '19

06 Sep '19

Hi All, Please find below the steps to set-up tacker for managing vnf packages. Steps to set-up tacker for managing vnf packages:- 1. Api-paste.ini [composite:tacker] /vnfpkgm/v1: vnfpkgmapi_v1 [composite:vnfpkgmapi_v1] use = call:tacker.auth:pipeline_factory noauth = request_id catch_errors extensions vnfpkgmapp_v1 keystone = request_id catch_errors authtoken keystonecontext extensions vnfpkgmapp_v1 [app:vnfpkgmapp_v1] paste.app_factory = tacker.api.vnfpkgm.v1.router:VnfpkgmAPIRouter.factory You can also copy api-paste.ini available in patch : https://review.opendev.org/#/c/675593 2. Configuration options changes : tacker.conf a) Periodic task to delete the vnf package artifacts from nodes and glance store. default configuration in tacker/tacker/conf/conductor.py vnf_package_delete_interval = 1800 b) Path to store extracted CSAR file on compute node default configuration in tacker/conf/vnf_package.py vnf_package_csar_path = /var/lib/tacker/vnfpackages/ vnf_package_csar_path should have Read and Write access (+rw) c) Path to store CSAR file at glance store default configuration in /devstack/lib/tacker filesystem_store_datadir = /var/lib/tacker/csar_files filesystem_store_datadir should have Read and Write access (+rw) 3. Apply python-tackerclient patches https://review.opendev.org/#/c/679956/ https://review.opendev.org/#/c/679957/ https://review.opendev.org/#/c/679958/ 4. Apply tosca parser changes https://review.opendev.org/#/c/675561/ 5. Sample CSAR file to create VNF package tacker/tacker/samples/vnf_packages/sample_vnf_pkg.zip 6. Commands to manage VNF packages To create a VNF package - openstack vnfpack create —user-data key=value <package_UUID> will be generated by this command which will be used in other commands to manage VNF Package. To upload the CSAR file 1. using direct path - openstack vnfpack upload <package_UUID> --upload-method direct-file --path <path of CSAR file> 2. using web - openstack vnfpack upload <package_UUID> --upload-method web-download --path <web URL> To list all the VNF Package - openstack vnfpack list To show a VNF package details - openstack vnfpack show <package_UUID> To delete a VNF package - openstack vnfpack delete <package_UUID> use `openstack vnfpack --help` command for more information Regards, Nitin Uikey Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.

1 2

Root Ara report removed from Zuul Jobs
by Clark Boylan 06 Sep '19

06 Sep '19

Hello Everyone, We have removed the top level Zuul job Ara reports from our Zuul jobs. This was done to reduce the total number of objects we are uploading to our swift/ceph object stores as some clouds have indicated the total object volume is a bit high. Analysis showed that Ara represented a significant chunk of that data. We did not remove that information though. Zuul's build dashboard is able to render a similar report for builds. You can find that by clicking on the "Console" tab of a build. For exampe, here is one for a nova tox job: http://zuul.openstack.org/build/8e581b24d38b4e5c8ff046be081c4525/console We hope this makes our log storage easier to support while still providing the information you need to debug your jobs. Note jobs that run a nested ara-report are not affected by this. I think TripleO, OSA, and others do this. Thank you, Clark

1 0

zuul and nodepool ansible roles
by David M Noriega 06 Sep '19

06 Sep '19

How do I go about contributing to the zuul and nodepool roles? They do not have either a launchpad or storyboard page.

3 3

Octavia LB flavor recommendation for Amphora VMs
by Pawel Konczalski 06 Sep '19

06 Sep '19

Hello everyone / Octavia Team, what is your experience / recommendation for a Octavia flavor with is used to deploy Amphora VM for small / mid size setups? (RAM / Cores / HDD) BR Pawel

2 1

[neutron] Open Infrastructure Summit Shanghai - forum topics ideas
by Slawek Kaplonski 06 Sep '19

06 Sep '19

Hi Neutrinos, We want to collect some ideas about potential topics which can be proposed as sessions on Forum in Shanghai. I created etherpad [1]. If You have any idea for such potential topic, please add it there. If You don’t have any ideas, please also check etherpad - maybe You will be interested in one of topics proposed by others. We don’t have much time for that as deadline for CFP is 20th of September, so please don’t wait too long with writing there Your ideas :) [1] https://etherpad.openstack.org/p/neutron-shanghai-forum-brainstorming — Slawek Kaplonski Senior software engineer Red Hat

1 0

[release] Release countdown for week R-5, September 9-13
by Sean McGinnis 05 Sep '19

05 Sep '19

Development Focus ----------------- We are getting close to the end of the Train cycle! Next week on September 12 is the train-3 milestone, also known as feature freeze. It's time to wrap up feature work in the services and their client libraries, and defer features that won't make it to the Ussuri cycle. General Information ------------------- This coming week is the deadline for client libraries: their last feature release needs to happen before "Client library freeze" on September 12. Only bugfix releases will be allowed beyond this point. When requesting those library releases, you can also include the stable/train branching request with the review (as an example, see the "branches" section here: https://opendev.org/openstack/releases/src/branch/master/deliverables/pike/…) September 12 is also the deadline for feature work in all OpenStack deliverables following the cycle-with-rc model. To help those projects produce a first release candidate in time, only bugfixes should be allowed in the master branch beyond this point. Any feature work past that deadline has to be approved by the team PTL. Finally, feature freeze is also the deadline for submitting a first version of your cycle-highlights. Cycle highlights are the raw data hat helps shape what is communicated in press releases and other release activity at the end of the cycle, avoiding direct contacts from marketing folks. See https://docs.openstack.org/project-team-guide/release-management.html#cycle… for more details. Upcoming Deadlines & Dates -------------------------- Train-3 milestone (feature freeze): September 12 (R-5 week) RC1 deadline: September 26 (R-3 week) Train final release: October 16 Forum+PTG at Shanghai summit: November 4

1 0

[Horizon] Paging and Angular...
by Marek Lyčka 05 Sep '19

05 Sep '19

Hi all, I took apart the Horizon paging mechanism while working on [1] and have a few of findings: - Paging is unimplemented/turned off for many (if not most) panels, not just Routers and Networks - Currently, single page data loads could potentially bump up against API hard limits - Sorting is also broken in places where paging is enabled (Old images...), see [2] - The Networks table loads data via three API calls due to neutron API limitations, which makes the marker based mechanism unusable - There is at least one more minor bug which breaks pagination, there may be more While some of these things may be fixable in different hacky and/or inefficient ways, we already have Angular implementations which fix many of them and make improving and fixing the rest easier. Since Angular ports would help with other unrelated issues as well and allow us to start deprecating old code, I was wondering two things: 1) What would it take to increase the priority of Angularization in general? 2) Can the Code Review process be modified/improved to increase the chance for Angularization changes to be code reviewed and merged if they do happen? My previous attempts in this area have failed because of lack of code reviewers... Since full Angularization is still the goal for Horizon as far as I know, I'd rather spend time doing that than hacking solutions to different problems in legacy code which is slated deprecation. Best Regards, Marek [1] https://bugs.launchpad.net/horizon/+bug/1746184 [2] https://bugs.launchpad.net/horizon/+bug/1782732 -- Marek Lyčka Linux Developer Ultimum Technologies s.r.o. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic marek.lycka(a)ultimum.io <stanislav.ulrych(a)ultimum.io> *https://ultimum.io <https://ultimum.io/>*

1 0

[goals][IPv6-Only Deployments and Testing] Week R-6 Update
by Ghanshyam Mann 05 Sep '19

05 Sep '19

Hello Everyone, Below is the progress on Ipv6 goal during R6 week. I am preparing the legacy base job for IPv6 deployment. NOTE: As first step, I am going to set up the job with all IPv6 deployment setting and basic verification whether service listens on IPv6 or not. As the second step, we will add post-run script to audit the tcpdump/logs etc for unwanted IPv4 traffic. Summary: * Number of Ipv6 jobs proposed Projects: 28 * Number of pass projects: 18 ** Number of project merged out of pass project: 13 * Number of failing projects: 10 Storyboard: ========= - https://storyboard.openstack.org/#!/story/2005477 Current status: ============ 1. Cinder and devstack fix are merged for cinder IPv6 job and I did recheck on cinder patch. 2. Preparing the legacy base job with IPv6 setting - https://review.opendev.org/#/c/680233/ 3. Zun, Watcher, Telemetry(thanks to zhurong) are merged. I have proposed to run telemetry ipv6 job on Panko and Aodh gate also. 4. This week new projects ipv6 jobs patch and status: - Tacker: link: https://review.opendev.org/#/c/676918/ status: Current functional jobs are n-v so I am not sure IPv6 job will pass or not. waiting for gate result. Need Help from the project team: 1. Monasca: waiting for new kafka client patches merge - https://review.opendev.org/#/c/674814/2 2. Sahara: https://review.opendev.org/#/c/676903/ Job is failing to start the sahara service. I could not find the logs for sahara service(it shows an empty log under apache). Need help from sahara team. 3. Searchlight: https://review.opendev.org/#/c/678391/ python-searchlightclient error, Trinh will be looking into this. 4. Senlin: https://review.opendev.org/#/c/676910/ Not able to connect on auth url - https://zuul.opendev.org/t/openstack/build/0ad3b4aac0424ad78171ca7546421f5e… 5. qinling: https://review.opendev.org/#/c/673506/1 logs are not there so i did recheck to get the fresh log for debugging. IPv6 missing support found: ===================== 1. https://review.opendev.org/#/c/673397/ 2. https://review.opendev.org/#/c/673449/ 3. https://review.opendev.org/#/c/677524/ How you can help: ============== - Each project needs to look for and review the ipv6 job patch. - Verify it works fine on ipv6 and no ipv4 used in conf etc - Any other specific scenario needs to be added as part of project IPv6 verification. - Help on debugging and fix the bug in IPv6 job is failing. Everything related to this goal can be found under this topic: Topic: https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(stat… How to define and run new IPv6 Job on project side: ======================================= - I prepared a wiki page to describe this section - https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing Review suggestion: ============== - Main goal of these jobs will be whether your service is able to listen on IPv6 and can communicate to any other services either OpenStack or DB or rabbitmq etc on IPv6 or not. So check your proposed job with that point of view. If anything missing, comment on patch. - One example was - I missed to configure novnc address to IPv6- https://review.opendev.org/#/c/672493/ - base script as part of 'devstack-tempest-ipv6' will do basic checks for endpoints on IPv6 and some devstack var setting. But if your project needs more specific verification then it can be added in project side job as post-run playbooks as described in wiki page[1]. [1] https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing -gmann

1 0

[zaqar][requirements] - release zaqarclient please?
by Matthew Thode 05 Sep '19

05 Sep '19

Hi Zaqar team, I tried to contact you via IRC but that didn't seem to go very well. The requirements team is looking for a release of the client so that we can move on updating jsonschema. We'd like to update it before the freeze time (starts Monday the 9th) so that all the other projects that use it can use it for a while in gate before release. held back because waiting for zaqarclient -jsonschema===3.0.2 +jsonschema===2.6.0 Is it possible to release a new version of the client (for instance as novaclient just recently did (among others))? Thanks, -- Matthew Thode

2 1

[dev][tosca-parser] review of toscadefinition1.2-support code
by Uikey, Nitin 05 Sep '19

05 Sep '19

Dear core-reviewers, We have submitted one patch regarding `Add support for tosca definition version 1.2` under topic `toscadefinition1.2-support`. Ref: https://review.opendev.org/#/c/675561/ We are adding a new feature in tacker to implement ETSI specs. In the beginning adding interface to manage vnf packages. For this feature spec [1] to merge, we need tosca-parser patch to be merged. We would appreciate if you can take a look at the patch and give your feedback. Thank you in advance. [1] : https://review.opendev.org/#/c/582930 Regards, Nitin Uikey Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.

1 0