December 2018 - openstack-discuss

[TripleO] flattening tripleo-heat-templates: what about pacemaker?
by Cédric Jeanneret 18 Dec '18

18 Dec '18

Hello folks, While working on the service flattening in t-h-t, I have some question regarding the pacemaker things. If we take rabbitmq: - we have rabbitmq things in docker/services - we have rabbitmq things in puppet/services - we have rabbitmq things in docker/services/pacemaker - we have rabbitmq things in puppet/services/pacemaker If the plain services are easy to flatten, the pacemaker part is more annoying, at least on the naming/location. I see the following possibilities: - create a deployment/pacemaker and move all pacemaker-related things in there - name files something like rabbitmq-pacemaker-foo.yaml and push them in deployment/rabbitmq Both look valid. And both might have their downsides. Any advice, idea, feeling on that? Also, regarding the workflow: do we do that flattening in one, or two passes? i.e. we can move all rabbitmq related code at once, or only the plain service, and do a second patch for the pacemaker things. Here again, both look valid, but I'm more in the "2 passes" idea. So... yeah, I'd like to get some feedback on that in order to do things right, and avoid monkey-patching because paths/naming aren't good ;). Thanks! Cheers, C. -- Cédric Jeanneret Software Engineer DFG:DF

2 1

can't login to a compute node deployed using ironic
by Manuel Sopena Ballesteros 18 Dec '18

18 Dec '18

3 2

Meaning of role name: auditor versus reader
by Adam Young 18 Dec '18

18 Dec '18

We've recently come to accept reader as one of the default roles. However, one thing that is not clear to me is the intention: is this designed to be the readonly set of operations that an admin can do, or the read only set of operations that a member can do? Should we really have two read-only roles, one for each case? Perhaps the admin-read-only should be called auditor, and then reader is for member only operations?

4 4

[KOLLA][MONASCA] monasca containers not running
by Manuel Sopena Ballesteros 18 Dec '18

18 Dec '18

Dear Openstack and Kolla community, I am trying to deploy monasca using kolla-ansible 7.0.0 and realised a couple of containers are not working Container monasca_log_persister See: 'bin/logstash --help' + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting /etc/logstash/conf.d/log-persister.conf INFO:__main__:Copying /var/lib/kolla/config_files/log-persister.conf to /etc/logstash/conf.d/log-persister.conf INFO:__main__:Setting permission for /etc/logstash/conf.d/log-persister.conf INFO:__main__:Deleting /etc/logstash/elasticsearch-template.json INFO:__main__:Copying /var/lib/kolla/config_files/elasticsearch-template.json to /etc/logstash/elasticsearch-template.json INFO:__main__:Setting permission for /etc/logstash/elasticsearch-template.json INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/log/kolla/monasca INFO:__main__:Setting permission for /var/log/kolla/monasca/monasca-api-error.log INFO:__main__:Setting permission for /var/log/kolla/monasca/monasca-api-access.log INFO:__main__:Setting permission for /var/log/kolla/monasca/monasca-log-api-error.log INFO:__main__:Setting permission for /var/log/kolla/monasca/monasca-log-api-access.log ++ cat /run_command + CMD='/usr/share/logstash/bin/logstash --log-in-json --log /var/log/kolla/monasca/monasca-log-persister.log -f /etc/logstash/conf.d/log-persister.conf' + ARGS= + [[ ! -n '' ]] + . kolla_extend_start ++ [[ ! -d /var/log/kolla/logstash ]] +++ stat -c %a /var/log/kolla/logstash ++ [[ 755 != \7\5\5 ]] + echo 'Running command: '\''/usr/share/logstash/bin/logstash --log-in-json --log /var/log/kolla/monasca/monasca-log-persister.log -f /etc/logstash/conf.d/log-persister.conf'\''' Running command: '/usr/share/logstash/bin/logstash --log-in-json --log /var/log/kolla/monasca/monasca-log-persister.log -f /etc/logstash/conf.d/log-persister.conf' + exec /usr/share/logstash/bin/logstash --log-in-json --log /var/log/kolla/monasca/monasca-log-persister.log -f /etc/logstash/conf.d/log-persister.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults ERROR: Unrecognised option '--log-in-json' Container monasca_log_transformer See: 'bin/logstash --help' + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting /etc/logstash/conf.d/log-transformer.conf INFO:__main__:Copying /var/lib/kolla/config_files/log-transformer.conf to /etc/logstash/conf.d/log-transformer.conf INFO:__main__:Setting permission for /etc/logstash/conf.d/log-transformer.conf INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/log/kolla/logstash ++ cat /run_command + CMD='/usr/share/logstash/bin/logstash --log-in-json --log /var/log/kolla/logstash/monasca-log-transformer.log -f /etc/logstash/conf.d/log-transformer.conf' + ARGS= + [[ ! -n '' ]] + . kolla_extend_start ++ [[ ! -d /var/log/kolla/logstash ]] +++ stat -c %a /var/log/kolla/logstash ++ [[ 755 != \7\5\5 ]] + echo 'Running command: '\''/usr/share/logstash/bin/logstash --log-in-json --log /var/log/kolla/logstash/monasca-log-transformer.log -f /etc/logstash/conf.d/log-transformer.conf'\''' Running command: '/usr/share/logstash/bin/logstash --log-in-json --log /var/log/kolla/logstash/monasca-log-transformer.log -f /etc/logstash/conf.d/log-transformer.conf' + exec /usr/share/logstash/bin/logstash --log-in-json --log /var/log/kolla/logstash/monasca-log-transformer.log -f /etc/logstash/conf.d/log-transformer.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults ERROR: Unrecognised option '--log-in-json' Any thoughts? Thank you very much Manuel Sopena Ballesteros | Big data Engineer Garvan Institute of Medical Research The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010 T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb(a)garvan.org.au<mailto:manuel.sb@garvan.org.au> NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

2 1

Instance Unable to connect to Data/Management Network
by Zufar Dhiyaulhaq 18 Dec '18

18 Dec '18

Hi, I have OpenStack cluster with this following network environment. Global Configuration : Data & Management Network : 10.60.60.0/24 External Network: 10.61.61.0/24 - All my endpoint is created in the Data & Management Network - Instance get floating IP from External Network IP Address Allocation : Controller: 10.60.60.10 | 10.61.61.10 Compute0: 10.60.60.20 | 10.61.61.20 Compute1: 10.60.60.30 | 10.61.61.30 Floating IP range : 10.61.61.100 - 10.61.61.200 My problem is the Instance cannot connect (ping, ssh, all connection) into the data/management network. I am running a magnum cluster but the instance needs to communicate with the endpoint. How to allow an Instance to connect to the data/management network? Best Regards, Zufar Dhiyaulhaq

2 1

[dev] [tc][all] TC office hours is started now on #openstack-tc
by Ghanshyam Mann 18 Dec '18

18 Dec '18

Hello everyone, TC office hour is started on #openstack-tc channel. Feel free to reach to us for anything you want discuss/input/feedback/help from TC. - gmann & TC

1 0

[cyborg] IRC meeting for Dec 18th
by Li Liu 18 Dec '18

18 Dec '18

The IRC meeting will be held tomorrow at 0300 UTC, which is 10:00 pm est(Tuesday) / 7:00 pm pst(Tuesday) /11 am Beijing time (Wednesday) This week's agenda: 1. Try to merge https://review.openstack.org/#/c/615462/ 2. Review https://review.openstack.org/#/c/625630/ from CoCo 3. Review https://review.openstack.org/#/c/596691/ from Zhenghao 4. Track status updates. -- Thank you Regards Li

1 0

[all] Service-specific configuration options
by Stephen Finucane 18 Dec '18

18 Dec '18

I proposed a WIP change for oslo.config [1] last week that would allow us to specify what services (e.g. nova-compute, nova-scheduler, nova- api, etc.) an option applies to. This came about because of comments on another issue where we found no one could say with surety what services needed an option in their configuration option (which in turn brought up questions about why those services needed the option). It was suggested I bring the idea up on the mailing list before we proceeded any further with it so here I am. I'm interested in two things. Firstly, I'd like to know whether generation of per-service configuration files is actually something others find problematic? I'm imagining this is annoying for operators and those working on deployment tooling but I've never got any hard data to back up that assumption. Secondly, I didn't see anything in the likes of cinder, nova or neutron but has anyone else already worked on something like this? For what it's worth, I did explain why I've gone with the approach I have in the commit message, but that doesn't mean there aren't other options I could be missing. Stephen [1] https://review.openstack.org/#/c/625011/ PS: I've tagged this [all] as enumerating the projects this actually affected left me with at least a dozen tags before I was even done. Sorry for any possible noise.

3 2

[tosca-parser] Failing to get functions from capabilities
by Michael Still 18 Dec '18

18 Dec '18

Hi, I'm trying to use tosca-parser to parse CSAR files. Its working quite well, until I hit a wall with capabilities today. Specifically I am testing with the single instance wordpress example CSAR, which uses a get_input for the num_cpus argument for host capabilities for the server. Based on how properties work, I would expect to get a function back for anything which requires referencing another value, but in the case of capabilities I either get the hardcoded value (strings, ints etc), or a None for values which would be functions if we were talking about prototypes. Here's a snippet of example code: #!/usr/bin/python import sys from jinja2 import Template import toscaparser.functions from toscaparser.tosca_template import ToscaTemplate tosca = ToscaTemplate(sys.argv[1]) for nodetemplate in tosca.nodetemplates: print print('Processing node template %s'% nodetemplate.name) capabilities = nodetemplate.get_capabilities_objects() for cap in capabilities: propobjs = cap.get_properties_objects() if not propobjs: continue for po in propobjs: print(' %s: %s' %(po.name, po.value)) Which returns this: $ python _capabilities.py csar_wordpress.zip No handlers could be found for logger "tosca.model" Processing node template wordpress network_name: PRIVATE initiator: source protocol: tcp secure: False Processing node template webserver network_name: PRIVATE initiator: source protocol: tcp secure: False secure: True Processing node template mysql_dbms Processing node template mysql_database Processing node template server secure: True min_instances: 1 max_instances: 1 mem_size: 4096 MB num_cpus: None disk_size: 10 GB distribution: Fedora version: 18.0 type: Linux architecture: x86_64 I would expect num_cpus for the "server" node_template to be a GetInput() function based on its definition in the CSAR, but instead I get None. Is there an example somewhere of how to correctly access functions for capabilities? Thanks, Michael

2 7

Gate fracas (status) update
by Matt Riedemann 18 Dec '18

18 Dec '18

I wanted to follow up from Clark's last gate status update [1]. Lots of work has been going on the last few weeks to try and get the gate under control since it's hard to merge code when you can't merge code. Most of my update is specific to nova, but some of it might be interesting to others as well for general QA/infra FYI. I'll group this into a few categories of issue. Optimize node usage / reduce blast radius ----------------------------------------- * The nova-cells-1 job has been moved to the experimental queue [2] so that we can still test that environment but do it on-demand. This helps us avoid latent cellsv1-specific race failures which might reset the gate. * I'm trying to drop the nova-multiattach job which is partially redundant with tempest-full except it only runs tempest.compute.api tests and it also runs slow tests [3]. What is blocking this is test_volume_swap_with_multiattach in the tempest-slow job intermittently failing during volume cleanup [4]. I have put a lot of debug details into that bug and have a debug logging patch up as well to see if I can recreate the failure and sort out what is going wrong. The main difference between this test in the tempest-slow job and the nova-multiattach job is simply that the tempest-slow job is multinode and nova-multiattach is single node, and the single node aspect might have been hiding some weird race bugs when it came to disconnecting the volume to cleanup if the servers are all on the same host. Help debugging that would be appreciated. Zuul queuing changes -------------------- An infra thread [5] prompted some discussion in IRC which led to changes in how tripleo changes will be queued by zuul [6][7]. The idea here is to isolate tripleo changes into their own queue so failures in tripleo changes don't disrupt (or starve) changes in openstack projects (in general) from getting queued up for test nodes. tl;dr: nova changes should enqueue more like they used to before [5]. Gate bugs --------- * http://status.openstack.org/elastic-recheck/#1807518 A fix for devstack was merged on both master and stable/rocky but we're still seeing this, so it probably needs more investigation. * http://status.openstack.org/elastic-recheck/#1783405 We need another deep dive on tempest tests which aren't marked as slow but which might be slow and contributing to overall job timeouts. * http://status.openstack.org/elastic-recheck/#1808171 This is relatively new. Debug notes are in the launchpad bug. In this test, a server is being created with 7 ports and times out waiting to get the network-vif-plugged event from neutron on the first port only, the other 6 events are received. Regarding the port that doesn't get the event, there are some weird messages in the neutron agent logs so there could be a race there but we likely need help from the neutron team to investigate. I'm not sure what might have prompted this, but maybe the recent change to use Ubuntu Bionic nodes and a new version of OVS is buggy? * http://status.openstack.org/elastic-recheck/#1800472 The fix for this [8] merged a couple of hours ago and we're seeing it drop off the e-r graph. * http://status.openstack.org/elastic-recheck/#1806912 There are a couple of separate nova bugs for this [9][10]. The fixes for both changes are approved and should reduce the amount of time it takes to start nova-api which will help avoid timeouts on slower test nodes. The fix for [10] also fixes a long-standing rolling upgrade issue so we'll be backporting that one. * http://status.openstack.org/elastic-recheck/#1798688 There is a nova fix up for this [11] and has a +2, it's very simple and just needs another nova core (efried would get it but he's out until January). * http://status.openstack.org/elastic-recheck/#1807520 This has been fixed on all grenade branches [12] and was very latent (goes back to pike) but only showed up on slower test nodes. * http://status.openstack.org/elastic-recheck/#1808010 This is a real snowball issue where the cirros filesystem fills up so config drive fails, falling back to use the metadata API to get networking information but the metadata API response is too slow and cloud-init times out. I've got a related fix [13] but we likely need someone to help profile where our other inefficiencies are in responding the metadata API requests. * http://status.openstack.org/elastic-recheck/#1808063 This one is also relatively new and I'm not sure what might be causing it. ---- There are other bugs in the e-r page but the hits are low enough, or they are latent enough, that I won't bother trying to detail them here. [1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/thread… [2] https://review.openstack.org/#/c/623538/ [3] https://review.openstack.org/#/q/topic:drop-multiattach-job [4] https://bugs.launchpad.net/tempest/+bug/1807723 [5] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/thread… [6] https://review.openstack.org/#/c/623595/ - this is the zuul feature [7] https://review.openstack.org/#/c/624246/ - the tripleo-ci change [8] https://review.openstack.org/#/c/615347/ [9] https://bugs.launchpad.net/nova/+bug/1807219 [10] https://bugs.launchpad.net/nova/+bug/1807044 [11] https://review.openstack.org/#/c/623596 [12] https://review.openstack.org/#/q/I833d79ecc97ddc844bf156ab64477c7c77424f20 [13] https://review.openstack.org/#/c/624778 -- Thanks, Matt

2 1