August 2020 - openstack-discuss - lists.openstack.org

[nova][ops] Live migration and CPU features
by Belmiro Moreira 21 Aug '20

21 Aug '20

Hi, in our infrastructure we have always compute nodes that need a hardware intervention and as a consequence they are rebooted, bringing a new kernel, kvm, ... In order to have a good compromise between performance and flexibility (live migration) we have been using "host-model" for the "cpu_mode" configuration of our service VMs. We didn't expect to have CPU compatibility issues because we have the same hardware type per cell. The problem is that when a compute node is rebooted the instance domain is recreated with the new cpu features that were introduced because of the reboot (using centOS). If there are new CPU features exposed, this basically blocks live migration to all the non rebooted compute nodes (those cpu features are not exposed, yet). The nova-scheduler doesn't know about them when scheduling the live migration destination. I wonder how other operators are solving this issue. I don't like stopping OS upgrades. What I'm considering is to define a "custom" cpu_mode for each hardware type. I would appreciate your comments and learn how you are solving this problem. Belmiro

5 6

[glance][horizon][ops] Dashboard show Forbidden 403 but openstack cli works fine
by Amjad Kotobi 21 Aug '20

21 Aug '20

Hi, We are running Train release, currently facing below error when change to “Images” panel on the project which I have “User” role. The image is visibility is public. Error: Forbidden. Insufficient permissions of the requested operation Error: Unable to retrieve the project. By using openstack-cli everything works and I do not face “Forbidden” 403, but in dashboard “access.log” it shows "GET /dashboard/api/keystone/projects/7331defd55ef479fbf0a9a1ac3fe9055 HTTP/1.1” 403 http://xxxxxx/dashboard/project/images” I checked: 1. glance: policy.json from both dashboard and api/registry hosts are same. 2. From dashboard API images is 2 As soon as I change the OWNER of image to the project which my role = USER the error disappears. Any ideas or previous encounter similar to this issue? Thanks Amjad

1 0

Using os_token
by Eric K. Miller 21 Aug '20

21 Aug '20

Hi, It looks like the OS_TOKEN environment variable can be set so a token can be re-used instead of a new authentication for each CLI command with the OpenStack Client. I'm a little confused as to how this works and haven't found any good documentation on the subject. I would have expected to be able to: 0) set appropriate OS_* variables for password authentication 1) create a token using "openstack token issue" 2) unset all OS_* environment variables 3) set OS_TOKEN to the token's value provided in #1 4) set OS_AUTH_TYPE to "v3token" 5) set OS_AUTH_URL to the respective KeyStone endpoint 6) set OS_IDENTITY_API_VERSION to "3" 7) use the CLI as normal However, I get a "The service catalog is empty." message. Maybe I'm missing something above or am I completely misunderstanding the purpose of the OS_TOKEN variable? >From examples I have seen, it looks like the token can be used in a REST API call. Is there a way to use an existing token with the CLI, instead, so a new token is not issued for every CLI command instantiation? Thanks! Eric

1 0

Re: [requirements][oslo] Inclusion of CONFspirator in openstack/requirements
by Adrian Turjak 20 Aug '20

20 Aug '20

On 21/08/20 9:16 am, Ben Nemec wrote: > > In general, given the complexity of what you're talking about I think > a driver plugin would be the way to go, as opposed to trying to fit > this all in with the core oslo.config functionality (assuming you/we > decide to pursue integrating at all). > > There were a few other things you mentioned as features of the > library. The following are some off-the-cuff thoughts without having > looked too closely at the library, so if they're nonsense that's my > excuse. > > "because I have plugins that need to dynamically load config on > startup, which then has to be lazy_loaded" > > Something like this could probably be done. I believe this is kind of > how the castellan driver in oslo.config works. Config values are > looked up and cached on-demand, as opposed to all at once. This one is a little weird, but essentially the way this works this in Adjutant: I load the config so I can start the app and go through base logic and loading plugins, but the config groups that are pulled from plugins aren't added to my config group tree until AFTER the config has already been loaded. So part of the config is usable, but some hasn't yet fully been loaded until after the plugins are done, and then that subtree will lazy_load itself when first accessed. It means that until a given lazy_loaded group is actually accessed as config, the config tree underneath it can still have config options added. It's likely not too crazy to do this in oslo, and have groups only read from the cached source (loaded file dict) when first accessed. > > "I also have weird overlay logic for defaults that can be overridden" > > My knee-jerk reaction to this is that oslo.config already supports > overriding defaults, so I assume there's something about your use case > that didn't mesh with that functionality? Or is this part of > oslo.config that you reused? Sooo, this one is a little special because what this feature lets you do is take any group in the config tree once loaded, and call the overlay function on it with either a dict, or another group. The returned value will be a deep copy of the config tree with the values present in the given dict/group overlaid on that original group. As in a depth first dict update, where only keys that exist on the overriding dict will be updated in the copy of the original dict. I need to write the docs for this... with a sane example, but here is my unit test for it: https://gitlab.com/catalyst-cloud/confspirator/-/blob/master/confspirator/t… I use this in Adjutant by having some config groups which are a default for something, and another place where many things can override that default via another group in the config, so I create an overlay copy and pass that to the place that needs the config it it's most specific case when the code actually pulls values from the conf group I passed it. See: https://opendev.org/openstack/adjutant/src/branch/master/adjutant/actions/v… I hope that helps, because it is in my mind a little odd to explain, but it allows some useful things when reusing actions in different tasks. > "I also have nested config groups that need to be named dynamically to > allow plugin classes to be extended without subclasses sharing the > same config group name." > > Minus the nesting part, this is also something being done with > oslo.config today. The config driver feature actually uses dynamically > named groups, and I believe at least Cinder is using them too. They do > cause a bit of grief for some config tools, like the validator, but it > is possible to do something like this. Cool! yeah I implemented that feature in my code because I ran into a case of confliciting namespaces and needed to find a better way to handle subclasses that needed to have different dynamic names for their config groups. > > Now, as to the question of whether we should try to integrate this > library with oslo.config: I don't know. Helpful, right? That's mostly where we got to last time I asked in #openstack-oslo! > > I think answering that question definitively would take a deeper dive > into what the new library is doing differently than I can commit to. > As I noted above, I don't think the things you're doing are so far out > in left field that it would be unreasonable to consider integrating > into oslo.config, but the devil is in the details and there are a lot > of details here that I don't know anything about. For example, will > the oslo.config data model even accommodate nested groups? I suspect > it doesn't now, but could it? Probably, but I can't say how > difficult/disruptive it would be. I looked into it briefly, and to do what I wanted, while also maintaining oslo.config how it was... ended up a bit messy, so I gave up because it would take too long and the politics of trying to get it merged/reviewed wouldn't be worth the effort most likely. > > If someone wanted to make incremental progress toward integration, I > think writing a basic YAML driver for oslo.config would be a good > start. It would be generally useful even if this work doesn't go > anywhere, and it would provide a basis for a hypothetical future > YAML-based driver providing CONFspirator functionality. From there we > could start looking to integrate more advanced features one at a time. > > Apologies for the wall of text. I hope you got something out of this > before your eyes glazed over. > > -Ben > Thanks for the wall of text! It was useful! I think ultimately it may be safer just maintaining my own separate library. For people who don't want to use .ini and prefer yaml/toml it's simpler, and for people who prefer .ini and don't need nesting etc, it's safer to keep oslo.config as it is. If someone does find anything I've done that makes sense in oslo.config I'd be happy to work porting it over, but I don't want to make structural changes to it. I'll always keep an eye on oslo.config, and may occasionally steal the odd idea if you add something cool, but I think other than my use of types.py and some of the Opt classes as a base, my code has diverged quite a bit.

1 0

per user quota not working properly
by its-openstack＠zohocorp.com 20 Aug '20

20 Aug '20

Dear openstack, We are facing a peculiar issue with regards to users quota of resources. e.g: s.no project user instance quota no instance created 1 test - 10 2 test user1 2 2 3 test user2 2 error "quota over" 4 test user3 3 able to create only 1 instance 5 test user4 no user quota defined able to create 10 instance As you see from mentioned table. when user1,user2, has instance quota of 2 and when user1 has created 2 instance, user2 unable to create instance. but user3 able to create only 1 more instance, user 4 has no quota applied so project quota 10 will be applied and he can create 10 instance. the quota is applied to each user but not tracked for each user, so this defeats the purpose of per user quota. Please help us with resolving this issue. Regards, sysadmin team

1 0

[largescale-sig] RPC ping
by Arnaud Morin 20 Aug '20

20 Aug '20

Hey all, TLDR: I propose a change to oslo_messaging to allow doing a ping over RPC, this is useful to monitor liveness of agents. Few weeks ago, I proposed a patch to oslo_messaging [1], which is adding a ping endpoint to RPC dispatcher. It means that every openstack service which is using oslo_messaging RPC endpoints (almosts all OpenStack services and agents - e.g. neutron server + agents, nova + computes, etc.) will then be able to answer to a specific "ping" call over RPC. I decided to propose this patch in my company mainly for 2 reasons: 1 - we are struggling monitoring our nova compute and neutron agents in a correct way: 1.1 - sometimes our agents are disconnected from RPC, but the python process is still running. 1.2 - sometimes the agent is still connected, but the queue / binding on rabbit cluster is not working anymore (after a rabbit split for example). This one is very hard to debug, because the agent is still reporting health correctly on neutron server, but it's not able to receive messages anymore. 2 - we are trying to monitor agents running in k8s pods: when running a python agent (neutron l3-agent for example) in a k8s pod, we wanted to find a way to monitor if it is still live of not. Adding a RPC ping endpoint could help us solve both these issues. Note that we still need an external mechanism (out of OpenStack) to do this ping. We also think it could be nice for other OpenStackers, and especially large scale ops. Feel free to comment. [1] https://review.opendev.org/#/c/735385/ -- Arnaud Morin

10 31

[requirements][oslo] Inclusion of CONFspirator in openstack/requirements
by Adrian Turjak 20 Aug '20

20 Aug '20

Hey OpenStackers! I'm hoping to add CONFspirator to openstack/requirements as I'm using it Adjutant: https://review.opendev.org/#/c/746436/ The library has been in Adjutant for a while but I didn't add it to openstack/requirements, so I'm trying to remedy that now. I think it is different enough from oslo.config and I think the features/differences are ones that are unlikely to ever make sense in oslo.config without breaking it for people who do use it as it is, or adding too much complexity. I wanted to use oslo.config but quickly found that the way I was currently doing config in Adjutant was heavily dependent on yaml, and the ability to nest things. I was in a bind because I didn't have a declarative config system like oslo.config, and the config for Adjutant was a mess to maintain and understand (even for me, and I wrote it) with random parts of the code pulling config that may or may not have been set/declared. After finding oslo.config was not suitable for my rather weird needs, I took oslo.config as a starting point and ended up writing another library specific to my requirements in Adjutant, and rather than keeping it internal to Adjutant, moved it to an external library. CONFspirator was built for a weird and complex edge case, because I have plugins that need to dynamically load config on startup, which then has to be lazy_loaded. I also have weird overlay logic for defaults that can be overridden, and building it into the library made Adjutant simpler. I also have nested config groups that need to be named dynamically to allow plugin classes to be extended without subclasses sharing the same config group name. I built something specific to my needs, that just so happens to also be a potentially useful library for people wanting something like oslo.config but that is targeted towards yaml and toml, and the ability to nest groups. The docs are here: https://confspirator.readthedocs.io/ The code is here: https://gitlab.com/catalyst-cloud/confspirator And for those interested in how I use it in Adjutant here are some places of interest (be warned, it may be a rabbit hole): https://opendev.org/openstack/adjutant/src/branch/master/adjutant/config https://opendev.org/openstack/adjutant/src/branch/master/adjutant/feature_s… https://opendev.org/openstack/adjutant/src/branch/master/adjutant/core.py https://opendev.org/openstack/adjutant/src/branch/master/adjutant/api/v1/op… https://opendev.org/openstack/adjutant/src/branch/master/adjutant/actions/v… https://opendev.org/openstack/adjutant/src/branch/master/adjutant/actions/v… https://opendev.org/openstack/adjutant/src/branch/master/adjutant/tasks/v1/… https://opendev.org/openstack/adjutant/src/branch/master/adjutant/tasks/v1/… If there are strong opinions about working to add this to oslo.config, let's chat, as I'm not against merging this into it somehow if we find a way that make sense, but while some aspects where similar, I felt that this was cleaner without being part of oslo.config because the mindset I was building towards seemed different and oslo.config didn't need my complexity. Cheers, Adrian

2 1

[cloudkitty] Resuming CloudKitty IRC meetings
by Pierre Riteau 20 Aug '20

20 Aug '20

Hello, We are resuming IRC meetings for the CloudKitty project using the existing calendar schedule. The first meeting will be on Monday August 24 at 1400 UTC in #cloudkitty on freenode, then every two weeks. Everyone is welcome: contributors, users, and anyone who would like to contribute or use CloudKitty but doesn't know how to get started. The agenda is available on Etherpad [1]. The meeting description [2] was stating that meetings were on the first and third Monday of the month, but the calendar schedule was using odd weeks. I've submitted a change [3] to synchronise the description: let's meet on odd weeks instead of using a month-based schedule. Thanks in advance to all of you helping to keep the project going. Pierre Riteau (priteau) [1] https://etherpad.opendev.org/p/cloudkitty-meeting-topics [2] http://eavesdrop.openstack.org/#CloudKitty_Team_Meeting [3] https://review.opendev.org/#/c/747256/

3 6

[Neutron] Drivers meeting agenda
by Slawek Kaplonski 20 Aug '20

20 Aug '20

Hi, Here is agenda for tomorrow's drivers meeting. We have 2 RFEs to discuss: * https://bugs.launchpad.net/neutron/+bug/1891334 - [RFE] Enable change of CIDR on a subnet * https://bugs.launchpad.net/neutron/+bug/1892200 - Make keepalived healthcheck more configurable There are also some points from Rodolfo on the on demand agenda but Rodolfo is on PTO this week so probably we will discuss those topics finally next week. See You on the meeting tomorrow and have a nice day :) -- Slawek Kaplonski Principal software engineer Red Hat

1 0

[release] Release countdown for week R-7 Aug 24 - 28
by Sean McGinnis 20 Aug '20

20 Aug '20

Development Focus ----------------- We are entering the last weeks of the Victoria development cycle. From now until the final release, we'll send a countdown email like this every week. It's probably a good time for teams to take stock of their library and client work that needs to be completed yet. The non-client library freeze is coming up, followed closely by the client lib freeze. Please plan accordingly to avoid any last minute rushes to get key functionality in. General Information ------------------- Next week is the Extra-ATC freeze, in preparation for future elections. All contributions to OpenStack are valuable, but some are not expressed as Gerrit code changes. Please list active contributors to your project team who do not have a code contribution this cycle, and therefore won't automatically be considered an Active Technical Contributor and allowed to vote. This is done by adding extra-atcs to https://opendev.org/openstack/governance/src/branch/master/reference/projec… before the Extra-ATC freeze on August 28. A quick reminder of the upcoming freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (Sept 3). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (Sept 10) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, Sept 10. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (Sept 24) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (Oct 8) Finally, now is also a good time to start planning what highlights you want for your deliverables in the cycle highlights. The deadline to submit an initial version for those is set to Feature freeze (Sept 10). Background on cycle-highlights: http://lists.openstack.org/pipermail/openstack-dev/2017-December/125613.html Project Team Guide, Cycle-Highlights: https://docs.openstack.org/project-team-guide/release-management.html#cycle… knelson [at] openstack.org/diablo_rojo on IRC is available if you need help selecting or writing your highlights Upcoming Deadlines & Dates -------------------------- Non-client library freeze: September 3 (R-6 week) Client library freeze: September 10 (R-5 week) Victoria-3 milestone: September 10 (R-5 week) Victoria release: October 14

1 0