- openstack-discuss - lists.openstack.org

apologize
by Bino Oetomo 09 Dec '18

09 Dec '18

Dear All. I apologize for my double post. I just realize after I write the second one. The first one is posted before I joint the list, it's http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000745… And the second one is : http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000760… Sincerely -bino-

1 0

Add port to router using python
by Bino Oetomo 09 Dec '18

09 Dec '18

Dear All. My openstack installation version is 'rocky'. I tried to create a new port and add it to existing router. Got no error messages, but the port status always 'DOWN'. -------------------------------------- >>> myrouter {'status': 'ACTIVE', 'external_gateway_info': {'network_id': 'd10dd06a-0425-49eb-a8ba-85abf55ac0f5', 'enable_snat': True, 'external_fixed_ips': [{'subnet_id': '4639e018-1cc1-49cc-89d4-4cad49bd4b89', 'ip_address': '10.10.10.2'}]}, 'availability_zone_hints': [], 'availability_zones': ['nova'], 'description': '', 'tags': [], 'tenant_id': 'c0b89f614b5a457cb5acef8fe8c2b320', 'created_at': '2018-12-07T07:24:41Z', 'admin_state_up': True, 'distributed': False, 'updated_at': '2018-12-07T07:58:02Z', 'project_id': 'c0b89f614b5a457cb5acef8fe8c2b320', 'flavor_id': None, 'revision_number': 10, 'routes': [], 'ha': False, 'id': 'a616dcc0-1f72-4424-9494-4d13b42445ee', 'name': 'Bino-rtr-01'} ```>>> mynetworks=[n for n in netlist['networks'] if n['name'].startswith('Bino')] >>> mynetworks [{'provider:physical_network': 'viavlan', 'ipv6_address_scope': None, 'revision_number': 7, 'port_security_enabled': True, 'mtu': 1500, 'id': '942675f6-fd5e-4bb2-ba43-487be992ff4e', 'router:external': False, 'availability_zone_hints': [], 'availability_zones': ['nova'], 'ipv4_address_scope': None, 'shared': False, 'project_id': 'c0b89f614b5a457cb5acef8fe8c2b320', 'status': 'ACTIVE', 'subnets': ['5373bc35-a90f-4793-a912-801920e47769'], 'description': '', 'tags': [], 'updated_at': '2018-12-07T07:50:27Z', 'provider:segmentation_id': 2037, 'name': 'Bino-net-01', 'admin_state_up': True, 'tenant_id': 'c0b89f614b5a457cb5acef8fe8c2b320', 'created_at': '2018-12-07T07:28:00Z', 'provider:network_type': 'vlan'}, {'provider:physical_network': 'viavlan', 'ipv6_address_scope': None, 'revision_number': 6, 'port_security_enabled': True, 'mtu': 1500, 'id': 'dfc8ed54-106d-48d0-8b45-cbd3cf0fbb79', 'router:external': False, 'availability_zone_hints': [], 'availability_zones': ['nova'], 'ipv4_address_scope': None, 'shared': False, 'project_id': 'c0b89f614b5a457cb5acef8fe8c2b320', 'status': 'ACTIVE', 'subnets': ['c71a86a3-f9a8-4e60-828e-5d6f87e58ac9'], 'description': '', 'tags': [], 'updated_at': '2018-12-07T07:49:59Z', 'provider:segmentation_id': 2002, 'name': 'Bino-net-02', 'admin_state_up': True, 'tenant_id': 'c0b89f614b5a457cb5acef8fe8c2b320', 'created_at': '2018-12-07T07:29:52Z', 'provider:network_type': 'vlan'}] >>> print([n['name'] for n in mynetworks]) ['Bino-net-01', 'Bino-net-02'] >>> mynetwork=mynetworks[1] >>> mynetwork['name'] 'Bino-net-02' >>> body_value = { ... 'port': { ... 'admin_state_up': True, ... 'device_id': myrouter['id'], ... 'name': 'Bino-rtr-01-02', ... 'network_id': mynetwork['id'], ... } ... } >>> response = nt.create_port(body=body_value) >>> response {'port': {'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'updated_at': '2018-12-10T01:44:10Z', 'device_owner': '', 'revision_number': 1, 'binding:profile': {}, 'port_security_enabled': True, 'fixed_ips': [{'subnet_id': 'c71a86a3-f9a8-4e60-828e-5d6f87e58ac9', 'ip_address': '192.168.202.6'}], 'id': 'a2a337af-3a37-4a9a-a4f1-ffaacdf9f881', 'security_groups': ['4bed540c-266d-4cc2-8225-3e02ccd89ff1'], 'binding:vif_details': {}, 'binding:vif_type': 'unbound', 'mac_address': 'fa:16:3e:05:aa:b5', 'project_id': 'c0b89f614b5a457cb5acef8fe8c2b320', 'status': 'DOWN', 'binding:host_id': '', 'description': '', 'tags': [], 'device_id': 'a616dcc0-1f72-4424-9494-4d13b42445ee', 'name': 'Bino-rtr-01-02', 'admin_state_up': True, 'network_id': 'dfc8ed54-106d-48d0-8b45-cbd3cf0fbb79', 'tenant_id': 'c0b89f614b5a457cb5acef8fe8c2b320', 'created_at': '2018-12-10T01:44:10Z', 'binding:vnic_type': 'normal'}} >>> response['port']['status'] 'DOWN' -------------------------------------- Kindly please give me some clues to fic this problem Sincerely -bino-

1 0

[neutron] [octavia] Manual deployment step for octavia on packstack
by Vikrant Aggarwal 09 Dec '18

09 Dec '18

Hello Team, Do we have the steps documented somewhere to install octavia manually like we have for zun [1]? I have done the openstack deployment using packstack and now I want to install the octavia manually on it. I have done the following steps: # groupadd --system octavia # useradd --home-dir "/var/lib/octavia" --create-home --system --shell /bin/false -g octavia octavia # cd /var/lib/octavia/ # git clone https://github.com/openstack/octavia.git # chown -R octavia:octavia * # pip install -r requirements.txt # python setup.py install # openstack user create --domain default --password-prompt octavia # openstack role add --project service --user octavia admin # openstack service create --name octavia --description "Octavia Service" " Octavia Load Balancing Servic" # openstack endpoint create --region RegionOne "Octavia Load Balancing Servic" public http://10.121.19.50:9876/v1 # openstack endpoint create --region RegionOne "Octavia Load Balancing Servic" admin http://10.121.19.50:9876/v1 # openstack endpoint create --region RegionOne "Octavia Load Balancing Servic" internal http://10.121.19.50:9876/v1 Made the following changes in the configuration file. [root@packstack1 octavia(keystone_admin)]# diff etc/octavia.conf /etc/ octavia/octavia.conf 20,21c20,21 < # bind_host = 127.0.0.1 < # bind_port = 9876 --- > bind_host = 10.121.19.50 > bind_port = 9876 38c38 < # api_v2_enabled = True --- > # api_v2_enabled = False 64c64 < # connection = mysql+pymysql:// --- > connection = mysql+pymysql://octavia:octavia@10.121.19.50/octavia 109c109 < # www_authenticate_uri = https://localhost:5000/v3 --- > www_authenticate_uri = https://10.121.19.50:5000/v3 111,114c111,114 < # auth_url = https://localhost:5000/v3 < # username = octavia < # password = password < # project_name = service --- > auth_url = https://10.121.19.50:35357/v3 > username = octavia > password = octavia > project_name = service 117,118c117,118 < # project_domain_name = Default < # user_domain_name = Default --- > project_domain_name = default > user_domain_name = default Generated the certificates using the script and copy the following certificates in octavia: [root@packstack1 octavia(keystone_admin)]# cd /etc/octavia/ [root@packstack1 octavia(keystone_admin)]# ls -lhrt total 28K -rw-r--r--. 1 octavia octavia 14K Dec 4 05:50 octavia.conf -rw-r--r--. 1 octavia octavia 1.7K Dec 4 05:55 client.key -rw-r--r--. 1 octavia octavia 989 Dec 4 05:55 client.csr -rw-r--r--. 1 octavia octavia 1.7K Dec 4 05:55 client.pem Can anyone please guide me about the further configuration? [1] https://docs.openstack.org/zun/latest/install/controller-install.html Thanks & Regards, Vikrant Aggarwal

1 0

[infra] A change to Zuul's queuing behavior
by corvus＠inaugust.com 09 Dec '18

09 Dec '18

Hi, We recently made a change to how Zuul and Nodepool prioritize node requests. Cloud resources are the major constraint in how long it takes Zuul to run test jobs on proposed changes. Because we're using more resources than ever before (but not necessarily because we're doing more work -- Clark has been helping to identify inefficiencies in other mailing list threads), the amount of time it takes to receive results on a change has been increasing. Since some larger projects consume the bulk of cloud resources in our system, this can be especially frustrating for smaller projects. To be sure, it impacts everyone, but while larger projects receive a continuous stream of results (even if delayed) smaller projects may wait hours before seeing results on a single change. In order to help all projects maintain a minimal velocity, we've begun dynamically prioritizing node requests based on the number of changes a project has in a given pipeline. This means that the first change for every project in the check pipeline has the same priority. The same is true for the second change of each project in the pipeline. The result is that if a project has 50 changes in check, and another project has a single change in check, the second project won't have to wait for all 50 changes ahead before it gets nodes allocated. As conditions change (requests are fulfilled, changes are added and removed) the priorities of any unfulfilled requests are adjusted accordingly. In the gate pipeline, the grouping is by shared change queue. But the gate pipeline still has a higher overall precedence than check. We hope that this will make for a significant improvement in the experience for smaller projects without causing undue hardship for larger ones. We will be closely observing the new behavior and make any necessary tuning adjustments over the next few weeks. Please let us know if you see any adverse impacts, but don't be surprised if you notice node requests being filled "out of order". -Jim

6 6

[infra] Update on test throughput and Zuul backlogs
by Clark Boylan 09 Dec '18

09 Dec '18

Hello everyone, I was asked to write another one of these in the Nova meeting today so here goes. TripleO has done a good job of reducing resource consumption and now represents about 42% of the total resource usage for the last month down from over 50% when we first started tracking this info. Generating the report requires access to Zuul's scheduler logs so I've pasted a copy at http://paste.openstack.org/show/736797/. There is a change, https://review.openstack.org/#/c/616306/, to report this data via statsd which will allow anyone to generate it off of our graphite server once deployed. Another piece of exciting (good) news is that we've changed the way the Zuul resource allocation scheme prioritizes requests. In the check pipeline a change's relative priority is based on how many changes for that project are already in check and in the gate pipeline it is relative to the number of changes in the shared gate queue. What this means is that less active projects shouldn't need to wait as long for their changes to be tested, but more active projects like tripleo-heat-templates, nova, and neutron may see other changes being tested ahead of their changes. More details on this thread, http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000482…. One side effect of this change is that our Zuul is now running more jobs per hour than in the past (because it can more quickly churn through changes for "cheap" projects). Unfortunately, this has increased the memory demands on the zuul-executors and we've found that we are using far more swap than we'd like. We'll be applying https://review.openstack.org/#/c/623245/ to reduce the amount of memory required by each job during the course of its run which we hope helps. We've also added one new executor with plans to add a second if this change doesn't help. All that said flaky tests are still an issue. One set of problems seems related to slower than expected/before test nodes in the BHS1 region. We've been debugging these with OVH (thank you amorin!) and think we've managed to make some improvements though so far the problems persist. Current theory is that we are acting as our own noisy neighbors starving the hypervisors of disk IO throughput. In order to test that we've halved the total number of resources we'll use there. More details at https://etherpad.openstack.org/p/bhs1-test-node-slowness including a list of e-r bugs that may be tied to this issue. One thing to keep in mind is that while the test nodes are slower than we'd like, they have also exposed some situations where our software is less efficient than we'd like. At least one bug, https://bugs.launchpad.net/nova/+bug/1807219, has been identified through this. I would encourage people debugging these slow tests to look to see if this exposes a deficiency in our software that can be fixed. CentOS 7.6 released this last Monday. Fallout from that has included needing to update ansible playbooks that ensure the latest version of a centos distro package without setting become: yes. Previously the package was installed at the latest version on our images which ansible could verify without root privileges. Additionally golang is no longer a valid package on the base OS as it was on 7.5 (side note, this doesn't feel incredibly stable for users if anyone from rhel is listening). If your jobs depend on golang on centos and were getting that from the distro packages on 7.5 you'll need to find somewhere else to get golang now. With the distro updates come broken nested virt. Unfortunately nested virt continues to be a back and forth of working today, not working tomorrow. It seem that our test node kernels play a big impact on that then a few days later the various clouds apply new hypervisor kernel updates and things work again. If your jobs attempt to use nested virt and you've seen odd behavior from them (like reboots) recently this may be the cause. These are the big issues that affect large numbers of projects (or even all of them), but there are still many project specific problems floating around as well. Unfortunately I haven't had much time to help dig into those recently (see broader issues above), but I think it would be helpful if projects can do some of that digging themselves. Also, a friendly reminder that we try to provide in cloud region mirrors and caches for commonly used resources like distro packages, pypi packages, dockerhub images, and so on. If your jobs aren't using these and you find they fail occasionally due to the Internet being flaky we'll be happy to help you update the jobs to use the in region resources instead. We'll keep pushing to fix the broader issues and are more than happy to help debug failures you hit within your projects as well. Hopefully this was helpful despite its length. Clark

5 7

[OpenStack][Neutron] WARNING neutron.pecan_wsgi.controllers.root
by Soheil Pourbafrani 08 Dec '18

08 Dec '18

Hi, I defined an external flat network and the Instances' network functionality is OK but the neutron logs continusly on the file server.log: *2018-12-04 03:21:22.206 2218 WARNING neutron.pecan_wsgi.controllers.root [req-0ac40043-ec93-490d-8770-5f4c49a11ea6 539929ca1549436cb4c4171e037e8df7 0a26c316d0d143229f7420cf7fa35bdc - default default] No controller found for: floatingips - returning response code 404: PecanNotFound* 2018-12-04 03:21:22.208 2218 INFO neutron.pecan_wsgi.hooks.translation [req-0ac40043-ec93-490d-8770-5f4c49a11ea6 539929ca1549436cb4c4171e037e8df7 0a26c316d0d143229f7420cf7fa35bdc - default default] GET failed (client error): The resource could not be found. 2018-12-04 03:21:22.209 2218 INFO neutron.wsgi [req-0ac40043-ec93-490d-8770-5f4c49a11ea6 539929ca1549436cb4c4171e037e8df7 0a26c316d0d143229f7420cf7fa35bdc - default default] 192.168.0.31 "GET /v2.0/floatingips?fixed_ip_address=192.168.0.205&port_id=15251089-950a-4a4b-998b-e7412350b8bc HTTP/1.1" status: 404 len: 309 time: 0.0077560 2018-12-04 03:21:22.284 2218 INFO neutron.wsgi [req-1177939c-e304-439a-a6bf-1a3d216a2336 539929ca1549436cb4c4171e037e8df7 0a26c316d0d143229f7420cf7fa35bdc - default default] 192.168.0.31 "GET /v2.0/subnets?id=9771f7f2-12f8-46bd-aca3-f1d26e0b9768 HTTP/1.1" status: 200 len: 835 time: 0.0708778 2018-12-04 03:21:22.368 2218 INFO neutron.wsgi [req-12dce42d-f6a0-4deb-902d-c46d95621013 539929ca1549436cb4c4171e037e8df7 0a26c316d0d143229f7420cf7fa35bdc - default default] 192.168.0.31 "GET /v2.0/ports?network_id=a90d7d71-1ff4-4a98-b2b9-68adaea7d1c4&device_owner=network%3Adhcp HTTP/1.1" status: 200 len: 1085 time: 0.0782819 2018-12-04 03:21:22.569 2218 INFO neutron.wsgi [req-e73d9ad4-4e70-4fb6-91a5-9bd02c606284 539929ca1549436cb4c4171e037e8df7 0a26c316d0d143229f7420cf7fa35bdc - default default] 192.168.0.31 "GET /v2.0/networks/a90d7d71-1ff4-4a98-b2b9-68adaea7d1c4?fields=segments HTTP/1.1" status: 200 len: 212 time: 0.1933858 2018-12-04 03:21:22.751 2218 INFO neutron.wsgi [req-79921bc1-7f1a-4eee-aeca-3b660aab0522 539929ca1549436cb4c4171e037e8df7 0a26c316d0d143229f7420cf7fa35bdc - default default] 192.168.0.31 "GET /v2.0/networks/a90d7d71-1ff4-4a98-b2b9-68adaea7d1c4?fields=provider%3Aphysical_network&fields=provider%3Anetwork_type HTTP/1.1" status: 200 len: 281 time: 0.1775169 2018-12-04 03:21:34.957 2218 INFO neutron.wsgi [req-50877132-2c57-4220-9689-5c739d415aa8 539929ca1549436cb4c4171e037e8df7 0a26c316d0d143229f7420cf7fa35bdc - default default] 192.168.0.32 "GET /v2.0/ports?tenant_id=8cd6e51308894e119d2ad90ed15e71d2&device_id=882901a0-e518-4b4b-bf08-e998c4c0c875 HTTP/1.1" status: 200 len: 1094 time: 0.1096959 2018-12-04 03:21:35.116 2218 INFO neutron.wsgi [req-0305018f-b085-46c2-86cd-1769432cda91 539929ca1549436cb4c4171e037e8df7 0a26c316d0d143229f7420cf7fa35bdc - default default] 192.168.0.32 "GET /v2.0/networks?id=a90d7d71-1ff4-4a98-b2b9-68adaea7d1c4 HTTP/1.1" status: 200 len: 877 time: 0.1516771 I checked the neutron.conf and there was not any web_framework attribute under the [Default] section. >From OpenStack forum, someone suggest to set *web_framework = legacy *in neutron.conf. I did but it had no effect and neutron still is logging the so-called warning. Is this a bug or I miss something in configuration? I use the OpenStack Rocky.

1 0

[dev] quick report on microstack
by Pete Vander Giessen 07 Dec '18

07 Dec '18

Hi All, On Tue, Nov 20, 2018, 8:30 AM Chris Morgan <mihalis68 at gmail.com wrote: > Mark Shuttleworth introduced 'microstack' at the conference last week. > It's a snap based openstack install suitable for getting a quick instance > up for playing with. > ... On Tue Nov 20 14:37:35 UTC 2018 Melvin Hillsman <mrhillsman at gmail.com> wrote: > I tried it as well, as a one off it seems useful, but after about an hour > and looking at some of the issues on GitHub it is definitely in need of > some TLC. Restarting and reconfiguring is a pickle to say the least. For > example, if you want to access it from other than localhost. I do however > like the simplicity of it out the gate. Thank you both for taking a look and raising bugs :-) We've spent our time after returning from the U.S. Thanksgiving Break hammering bits and pieces into shape. I just pushed a release to the candidate channel that should fix some of the hiccups on install and restart. (Instances you've spun up will still be stopped after a system reboot, but you should be able to start them up manually and continue using them without any issues.) You can install it with: sudo snap install microstack --classic --candidate And if you've already got microstack installed, you can do: sudo snap refresh microstack @Chris Morgan: I left a comment on the bug relating to theme switching that you raised (https://github.com/CanonicalLtd/microstack/issues/39) It looks like something is broken in the pathing for the scss around the roboto font face, but I'm not sure whether it's an issue with the way that the snap lays out the files (a missing symlink, maybe?), or if it's a more general issue when installing from a Rocky release tarball. We're troubleshooting, but if you'd weigh in with an opinion, I'd appreciate it. Thanks again! ~ PeteVG

1 0

[ops][docs] The Contributor Guide: Ops Feedback Session Summary
by Kendall Nelson 07 Dec '18

07 Dec '18

Hello! For the long version, feel free to look over the etherpad[1]. It should be noted that this session was in relation to the operator section of the contributor guide, not the operations guide, though they should be closely related and link to one another. Basically the changes requested can be boiled down to two types of changes: cosmetic and missing content. Cosmetic Changes: - Timestamps so people can know when the last change was made to a given doc (dhellmann volunteered to help here)[2] - Floating bug report button and some mechanism for auto populating which page a bug is on so that the reader doesn’t have to know what rst file in what repo has the issue to file a bug[3] - Maybe not cosmetic exactly, but it was requested that the entirety of the contributor guide be translated into as many languages as possible (stevenK volunteered to help get the framework setup so that things could be imported into Zanata for translation)[4] - Link to source somewhere on the page so those interested in fixing issues that don’t know the repo structure can find and fix issues right away[5] Missing Content: - Links to the docs in the Operator guide (HA guide, etc)[6] - A guide to ‘I fixed this bug locally, how do I get it upstream?’[7] If there are any unclaimed items you are interested in working on, please do! The contributor guide work items are tracked in StoryBoard here[8]! -Kendall (diablo_rojo) [1] https://etherpad.openstack.org/p/BER-Contrib-Guide-Ops [2] https://storyboard.openstack.org/#!/story/2004436 [3] https://storyboard.openstack.org/#!/story/2004437 [4] https://storyboard.openstack.org/#!/story/2004438 [5] https://storyboard.openstack.org/#!/story/2004439 [6] https://storyboard.openstack.org/#!/story/2004441 [7] https://storyboard.openstack.org/#!/story/2004442 [8]https://storyboard.openstack.org/#!/project/openstack/contributor-guide

5 7

[octavia] Routing between lb-mgmt-net and amphora
by Paul Bourke 07 Dec '18

07 Dec '18

Hi, This is mostly a follow on to the thread at[0], though due to the mailing list transition it was easier to start a new thread. I've been attempting to get Octavia setup according to the dev-quick-start guide[1], but have been struggling with the following piece: "Add appropriate routing to / from the ‘lb-mgmt-net’ such that egress is allowed, and the controller (to be created later) can talk to hosts on this network." In mranga's reply, they say: > -- Create an ovs port on br-int > -- Create a neutron port using the ovs port that you just created. > -- Assign the ip address of the neutron port to the ovs port > -- Use ip netns exec to assign a route in the router namespace of the LoadBalancer network. I have enough of an understanding of Neutron/OVS for this to mostly make sense, but not enough to actually put it into practice it seems. My environment: 3 x control nodes 2 x network nodes 1 x compute All nodes have two interfaces, eth0 being the management network - 192.168.5.0/24, and eth1 being used for the provider network. I then create the Octavia lb-mgmt-net on 172.18.2.0/24. I've read the devstack script[2] and have the following questions: * Should I add the OVS port to br-int on the compute, network nodes, or both? * What is the purpose of creating a neutron port in this scenario If anyone is able to explain this a bit further or can even point to some good material to flesh out the underlying concepts it would be much appreciated, I feel the 'Neutron 101' videos I've done so far are not quite getting me there :) Cheers, -Paul [0] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000544… [1] https://docs.openstack.org/octavia/latest/contributor/guides/dev-quick-star… [2] https://github.com/openstack/octavia/blob/master/devstack/plugin.sh

2 3

[nova] slides now available for project update and project onboarding sessions
by melanie witt 07 Dec '18

07 Dec '18

Howdy all, This is just a FYI that slides are now attached to the summit events for the Nova - Project Update and the Nova - Project Onboarding sessions: https://www.openstack.org/summit/berlin-2018/summit-schedule/events/22861/n… https://www.openstack.org/summit/berlin-2018/summit-schedule/events/22860/n… Cheers, -melanie

1 0