February 2019 - openstack-discuss

[placement][nova] What is the plan for tracking placement blueprints post-extraction?
by Matt Riedemann 21 Mar '19

21 Mar '19

I was updating the status on some nova blueprints in launchpad today and we have 4 placement API blueprints marked as blocked, with one actually in progress now [1]. The issue is none of the changes are being reflected back in the whiteboard for the blueprint in launchpad because the changes are made in the extracted placement repo which doesn't have a launchpad project, nor is there anything in storyboard for placement. Right now it's a small thing, but I'm wondering what, if any, plans are in place for tracking blueprints for the extracted placement repo. I'm assuming the answer is storyboard but maybe it hasn't been discussed much yet. [1] https://blueprints.launchpad.net/nova/+spec/alloc-candidates-in-tree -- Thanks, Matt

6 9

[dev][tc][ptl] Continuing Evaluating projects in relation to OpenStack cloud vision
by Chris Dent 21 Mar '19

21 Mar '19

Yesterday at the TC meeting [1] we decided that the in-progress task to make sure the technical vision document [2] has been fully evaluated by project teams needs a bit more time, so this message is being produced as a reminder. Back in January Julia produced a message [3] suggesting that each project consider producing a document where they compare their current state with an idealized state if they were in complete alignment with the vision. There were two hoped for outcomes: * A useful in-project document that could help guide future development. * Patches to the vision document to clarify or correct the vision where it is discovered to be not quite right. A few projects have started that process (see, for example, melwitt's recent message for some links [4]) resulting in some good plans as well as some improvements to the vision document [5]. In the future the TC would like to use the vision document to help evaluate projects applying to be "official" as well as determining if projects are "healthy". As such it is important that the document be constantly evolving toward whatever "correct" means. The process described in Julia's message [3] is a useful to make it so. Please check it out. Thanks. [1] http://eavesdrop.openstack.org/meetings/tc/2019/tc.2019-02-07-14.00.html [2] https://governance.openstack.org/tc/reference/technical-vision.html [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001417.… [4] http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002501… [5] https://review.openstack.org/#/q/project:openstack/governance+file:referenc… -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

5 4

[ironic][neutron] nf_conntrack_helper now disabled by default
by Derek Higgins 19 Mar '19

19 Mar '19

Hi All, Shortly before the holidays CI jobs moved from xenial to bionic, for Ironic this meant a bunch failures[1], all have now been dealt with, with the exception of the UEFI job. It turns out that during this job our (virtual) baremetal nodes use tftp to download a ipxe image. In order to track these tftp connections we have been making use of the fact that nf_conntrack_helper has been enabled by default. In newer kernel versions[2] this is no longer the case and I'm now trying to figure out the best way to deal with the new behaviour. I've put together some possible solutions along with some details on why they are not ideal and would appreciate some opinions 1. Why not enable the conntrack helper with echo 1 > /proc/sys/net/netfilter/nf_conntrack_helper The router namespace is still created with nf_conntrack_helper==0 as it follows the default the nf_conntrack module was loaded with 2. Enable it in modprobe.d # cat /etc/modprobe.d/conntrack.conf options nf_conntrack nf_conntrack_helper=1 This works but requires the nf_conntrack module to be unloaded if it has already been loaded, for devstack and I guess in the majority of cases (including CI nodes) this means a reboot stage or a potentially error prone sequence of stopping the firewall and unloading nf_conntrack modules. This also globally turns on the helper on the host reintroducing the security concerns it comes with 3. Enable the contrack helper in the router network namespace when it is created[3] This works for ironic CI, but there may be better solutions that can be worked within neutron that I'm not aware of. Of the 3 options above this would be most transparent to other operators as the original behaviour would be maintained. thoughts on any of the above? or better solutions? 1 - https://storyboard.openstack.org/#!/story/2004604 2 - https://kernel.googlesource.com/pub/scm/linux/kernel/git/horms/ipvs-next/+/… 3 - https://review.openstack.org/#/c/628493/1

6 10

Re: [heat] keystone endpoint configuration
by Zane Bitter 18 Mar '19

18 Mar '19

Sending to the list as I intended the first time... with edit to take into account new info from Rabi. On 27/02/19 5:29 PM, Jonathan Rosser wrote: > Conversely, heat itself needs to be able to talk to many other openstack components, defined in the [clients_*] config sections. It is reasonable to describe these interactions as being "Internal" - I may misunderstand some of this though. Yeah, that is reasonable, and in fact we give you the option in the config file to choose which set of endpoints to use from the catalog. But we assume that there is only one auth_url used to fetch the catalog. > So here lies the issue - appropriate entries in heat.conf to make internal interactions between heat and horizon (one example) work in real-world deployments results in the keystone internal URL being placed in callbacks, and then SoftwareDemployments never complete as the internal keystone URL is not usually accessible to a VM. Do you know which config options affect Horizon? It's surprising to me that anything in *Heat's* config would make a difference to Horizon. The only thing I can think of is that www_authenticate_uri might have to be on the internal network for Horizon to work, but Colleen already said that it should always be on the public network. And in any event it would be overridden for the purposes of signal URLs by setting [clients_keystone]/auth_uri to the public endpoint. > I suspect that there is not much coverage for this kind of network separation in gate tests. Yeah, I suspect only the deployment projects run tests with complex-enough networking setups to be able to verify this, and it's not their job to check that Heat supports this use case. > There are already examples of similar config options in heat.conf, such > as "heat_waitcondition_server_url" - would additonal config items such > as server_base_auth_url and signal_responder_auth_url be appropriate so > that we can be totally explicit about the endpoints handed on to created > VM? Yes, that's along the lines of what I was thinking too (although I think we'd only need one option, for URLs destined to be called from userspace). We already have an endpoint_type option (that defaults to PublicURL), so maybe we just need to be able to specify internal_auth_uri and public_auth_uri and we can select based on the endpoint type when we're using the clients internally, but always use the public one when gathering data to pass to a VM? [Edit: Rabi pointed out that X-Auth-Url is inserted by middleware, and gets its information from the same config options, so the following paragraph is a red herring.] Looking close at the code though, I wonder if part of the problem here is that we use the X-Auth-Url header from the request context whenever it is available (and only fall back to the config options when it is not), which should generally always be the public URL... but possibly is the internal URL if the request comes from Horizon... or Magnum?? In fact, I wonder if that could be the _whole_ problem in terms of the stuff you laid out at the beginning of the thread? Notwithstanding that there are still cases with completely isolated networks that cannot be covered by config options alone. e.g. as discussed in https://storyboard.openstack.org/#!/story/2004524 you can get the 'signal_url' attribute from OS::Heat::ScalingPolicy. That can either be passed to Aodh (should presumably use the internal auth URI) or e.g. given to a VM that might be running the user's own monitoring system (must use the public auth URI). The only way I can see to solve that would be to have separate 'internal_signal_url' and 'public_signal_url' attributes (are ordinary users even supposed to be able to see internal URLs in the catalog?). That's not an ideal experience for users since on the vast majority of clouds where the services are able to access external endpoints then the public URL will continue to work in either case (so extra work to figure out, and people will choose wrong leaving lots of non-interoperable templates floating around out there). But I guess it may be better than what we have now. cheers, Zane.

2 1

[kolla] Let's switch to Python 3
by Marcin Juszkiewicz 13 Mar '19

13 Mar '19

Python 2 is going bye bye. Everyone knows that, lot of us ignores that. But we need to move on. Kolla has few patches to make use of Python3 based binary packages. One for Fedora [1], other for Ubuntu and Debian [2]. 1. https://review.openstack.org/624838 2. https://review.openstack.org/625298 But those patches are just tip of iceberg... # Types of images There are two types of images in Kolla: - binary - source ## Binaries state in distributions Binary ones are built from distribution packages. For RHEL/CentOS/OracleLinux is means RDO packages, for Ubuntu it is UCA and for Debian it is whatever is in distribution repositories. RDO ones are Python 2 based because there is no other version of Python in RHEL7 (I do not count EPEL or SCL). For Ubuntu we have a mix of Python 2 and Python 3 (from what I heard). Debian needs update to Buster (current 'testing', currently in soft freeze) and then we will get Python 3 packages (currently for Rocky). ### Pip issue On binary distributions we tend to use 'pip' from distribution packages. Which means 8.1.2 for CentOS:7, 9.0.1 for Ubuntu:bionic and 18.1 for Debian:buster. ## Source images In my opinion this type of images is most common during development cycle. Here we install Python and binaries from distribution repositories and then bootstrap latest 'pip' and use it to install all Python modules we need. For now those images are Python 2 based. # WIP plans for T cycle For T cycle we should move whatever possible to Python 3. Which means Debian:buster, Ubuntu:bionic and source images. No idea yet does it involves CentOS:7 source images as well as this would require us to use EPEL repository for Python 3. ## Always bootstrap pip Due to 'pip' issue we may need to bootstrap latest version in each type of images. This will give us simple 'pip' command in each image nevermind which version of Python is used. That saves us from dealing with "is it 'pip' or 'pip3' we need to use" thing. ## Create new variables There will be few different versions of Python in use: 2.7 (RHEL7 family), 3.6 (Ubuntu:bionic and RHEL7/EPEL), 3.7 (Debian:buster). As we tend to modify some files or need to know where Python modules are present. We may use values from 'sys.path' list for it or create 'python_path' (or similar) variable and set it per distribution/image_type. # Some kind of work items list 1. Move Debian to 'buster' release. https://review.openstack.org/612681 2. Bootstrap 'pip' in all image types. 3. Move source images to Python 3 (Debian, Ubuntu for sure, no idea yet about CentOS). 4. Move binary images to Python 3 (Debian, Ubuntu, Fedora/RHEL8) Each step will require several build/test/fix cycles. We also need to identify which images still require Python 2. # Summary I know that this mail looks a bit chaotic but I think that we need to discuss and plan what we need to do and how. And cooperate with external teams which provide binary packages (that's why Thomas and Haïkel are in Cc:). What do you think?

5 10

[api][neutron] tagging support when creating a resource
by Akihiro Motoki 12 Mar '19

12 Mar '19

Hi API-SIG, neutron and tagging-related folks, This email ask opinions on tagging when creating a resource. We received a feature request to support tagging when creating a resource [1]. Neutron supports bulk creation of resources but the current tagging API recommended by the API-SIG does not define tagging when creating a resource. As a result, if we want to create 100 ports with specific tags, we need 1 (bulk creation) +100 (taggins per resource) API calls. It sounds nice to me to support tagging when creating a resource to address the above problem. What in my mind is to specify 'tags' attribute in a body of POST request like: {'port': {'name': 'foo', 'network_id': <network-id>, 'tags': ['red', 'blue'] } } I don't know the reason why the current API-SIG recommended tagging API does not support this model. The current tagging API defines "tags" as a subresource. Is there any reason this model was adopted? Best Regards, Akihiro Motoki (irc: amotoki) [1] https://bugs.launchpad.net/neutron/+bug/1815933 [RFE] Allow bulk-tagging of resources [2] https://specs.openstack.org/openstack/api-wg/guidelines/tags.html

4 5

[horizon][plugins][vitrage][heat][ironic][manila] Integration tests on gates
by Ivan Kolodyazhny 11 Mar '19

11 Mar '19

Hi team, A few weeks ago we enabled horizon-integration-tests job[1]. It's a set of selenium-based test cases to verify that Horizon works as expected from the user's perspective. Like any new job, it's added in a non-voting mode for now. During the PTG, I'd got several conversations with project teams that it would be good to have such tests in each plugin to verify that plugin works correctly with a current Horizon version. We've got about 30 plugins in the Plugin Registry [2]. Honestly, without any kind of testing in most of the plugins, we can't be sure that they work well with a current version of Horizon. That's why we decided to implement some kind of smoke tests for plugins based on Horizon integration tests framework. These tests should verify that a plugin is installed and pages could be opened in a browser. We will run these tests on the experimental queue and/or on some schedule on Horizon gates to verify that plugins are maintained and working properly. My idea is to have such a list of 'tested' plugins, so we can add 'Maintained' label to the Plugin Registry. Once these jobs become voting, we can add a label 'Verified'. I think such a schedule looks reasonable: * Stein-Train release cycles - add non-voting jobs for each maintained plugin and introduce "Maintained" label * Train-U release cycles - makes stable jobs voting and introduce "Verified" label in the Horizon Plugin registry I do understand that some teams don't have enough resources to maintain integration tests, so I'm stepping as a volunteer to introduce such tests and jobs for the project. I already published patches for Vitrage and Heat [3] plugins and will do the same for Ironic and Manila dashboards in a short time. Any help or feedback is welcome:). [1] https://review.openstack.org/#/c/580469/ [2] https://docs.openstack.org/horizon/latest/install/plugin-registry.html [3] https://review.openstack.org/#/q/topic:horizon-integration-tests+(status:op… Regards, Ivan Kolodyazhny, http://blog.e0ne.info/

3 3

[Interop-wg] [dev] [cinder] [qa] Strict Validation for Volume API using JSON Schema
by Ghanshyam Mann 09 Mar '19

09 Mar '19

Hello everyone, Tempest is planning to add the strict API validation using JSON schema for Volume test [1]. We do the same for all the compute test. With *Strict* JSON schema validation, all the API response will be validated with predefined schema with additionalProperties=False. additionalProperties=False will not allow any additional attribute in API response than what upstream APIs has in their current version. For example: If any vendor has modified the API response and returning additional attributes then Tempest tests going to fail. This will help: - To improve the OpenStack interoperability. Strict validation of API response is always helpful to maintain the interoperability. - To improve the volume API testing to avoid the backward compatible changes. Sometime we accidentally change the API in backward incompatible way and strict validation with JSON schema help to block those. We want to hear from cinder and interop team about any impact of this change to them. [1] https://blueprints.launchpad.net/tempest/+spec/volume-response-schema-valid… -gmann

4 6

[tc] forum planning
by Doug Hellmann 08 Mar '19

08 Mar '19

It's time to start planning the forum. We need a volunteer (or 2) to set up the etherpad and start collecting ideas for sessions the TC should be running. Who wants to handle that? -- Doug

2 3

[all][qa] tempest-multinode-full and tempest-slow job will start running on stable/* branches
by Ghanshyam Mann 08 Mar '19

08 Mar '19

Hi All, Till now Tempest jobs 'tempest-multinode-full' & 'tempest-slow' are running on master gate only because those are restricted by branch variant to mater-only[1]. Those jobs are supposed to be run for stable branches also. Many projects (for example Nova) have added them in their stable branch gate pipeline to test the backport changes. Below two patches are removing the master branch restriction from them so that they will run on stable branches also. Stable/pike and stable/queens were failing due to ssh issue which is fixed and merged in both branches. Both jobs are running fine now on all stable branches, I have tested them on Nova stable branches gate[2] - https://review.openstack.org/#/c/610938/ - https://review.openstack.org/#/c/620582/ I have opened the above two patches to merge now. tempest-slow job is voting in stable branch and tempest-multinode-full is non voting in most of the places. so once above two patches are merged, below are the change will happen in stable branch gate testing: 1. stable gate will start running the tempest-slow and tempest-multinode-full job where ever it is already added in stable branch pipeline. 2. If any of your job derived from either tempest-slow or tempest-multinode-full job and not supposed to run for stable branch (and till now it relies on master restriction of base job) then, you need to handle the same restriction in your job. 3. If you are looking for multinode/slow test coverage in stable branches then, they are available now. Feel free to ping me on IRC or via this mail thread for anything or if we need to wait for those patches to get merge (in case any projects is affected) [1] http://git.openstack.org/cgit/openstack/tempest/tree/.zuul.yaml#n202 http://git.openstack.org/cgit/openstack/tempest/tree/.zuul.yaml#n267 [2] https://review.openstack.org/#/q/topic:tempest-multinode-slow-stable+(statu… -gmann

1 1