May 2019 - openstack-discuss - lists.openstack.org

[Zun] openstack appcontainer run error
by Alejandro Ruiz Bermejo 02 May '19

02 May '19

I'm having troubles with the verify step of the Zun intallation at Openstack Queens on Ubuntu 18.04 LTS. I previously Posted a trouble with it and already fixed the error you guys pointed at. Now i still can't launch the app container. It freeze at container_creating task, the shows an error state root@controller /h/team# openstack appcontainer show 4a657ac5-058c-43eb-8cbf-7239ad3c4d76 +-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | addresses | {} | | links | [{u'href': u' http://controller:9517/v1/containers/4a657ac5-058c-43eb-8cbf-7239ad3c4d76', u'rel': u'self'}, {u'href': u' http://controller:9517/containers/4a657ac5-058c-43eb-8cbf-7239ad3c4d76', u'rel': u'bookmark'}] | | image | cirros | | labels | {} | | disk | 0 | | security_groups | [] | | image_pull_policy | None | | user_id | a16c6ef0319b4643a4ec8e56a1d025cb | | uuid | 4a657ac5-058c-43eb-8cbf-7239ad3c4d76 | | hostname | None | | environment | {} | | memory | None | | project_id | 59065d8f970b467aa94ef7b35f1edab5 | | status | Error | | workdir | None | | auto_remove | False | | status_detail | None | | host | None | | image_driver | docker | | task_state | None | | status_reason | *Docker internal error: 500 Server Error: Internal Server Error ("failed to update store for object type *libnetwork.endpointCnt: client: endpoint http://10.8.9.54:2379 <http://10.8.9.54:2379> exceeded header timeout"). * | | name | test1 | | restart_policy | {} | | ports | [] | | command | "ping" "8.8.8.8" | | runtime | None | | cpu | None | | interactive | False | +-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ I tried to launch another container without defining a network and without executing any command, and it also had the same error. I can launch container from the computer node cli with docker commands, the errors are when i try to launch them from the controller CLI. I run a docker run hello-world at the compute node and everything went fine Wen u runned openstack appcontainer create hello-world i had exactly the same error

2 1

[baremetal-sig] Planet for syndicating bare metal activity?
by Stig Telfer 02 May '19

02 May '19

Hi all - Good to see all the activity around bare metal this week. To keep information flowing, would it make sense to implement something like a planet feed for syndicating baremetal blog content from program members, linked to from the landing page https://www.openstack.org/bare-metal/ <https://www.openstack.org/bare-metal/> ? Cheers, Stig

1 0

[keystone][heat] security_compliance options and auto-created users
by Pavlo Shchelokovskyy 02 May '19

02 May '19

Hi all, I am currently looking at options defined in [security_compliance] section of keystone.conf [0] and trying to understand how enabling those security features may affect other services. The first thing I see is that any project that auto-creates some temporary users may be affected. Of the top of my head I can recall only Heat and Tempest doing this. For Tempest situation is easier as a) tempest can use static credentials instead of dynamic ones so it is possible to craft appropriate users beforehand and b) those users are relatively short-lived (required for limited time). In case of Heat though those users are used for deferred auth (like in autoscaling) which for long lived stacks can happen at arbitrary time in future - which is a problem. Below is breakdown of options/features possible to set and what problems that may pose for Heat and ideas on how to work those around: - disable_user_account_days_inactive - this may pose a problem for deferred auth, and it seems is not possible to override it via user "options". IMO there's a need to add a new user option to Keystone to ignore this setting for given user, and then use it in Heat to create temporary users. - lockout failure options (lockout_failure_attempts, lockout_duration) - can be overridden by user option, but Heat has to set it first. Also the question remains how realistically such problem may arise for an auto-created internal user and whether Heat should set this option at all - password expiry options (password_expires_days, unique_last_password_count, minimum_password_age) - poses a problem for deferred auth, but can be overridden by user option, so Heat should definitely set it IMO for users it creates - change_password_upon_first_use - poses problem for auto-generated users, can be overridden by a user option, but Heat must set it for its generated users - password strength enforcement (password_regex, password_regex_description) - now this is an interesting one. Currently Heat creates passwords for its temporary users with this function [1] falling back to empty password if a resource is not generating one for itself. Depending on regex_password setting in keystone, it may or may not be enough to pass the password strength check. I've looked around and (as expected) generating a random string which satisfies a pre-defined arbitrary regex is quite a non-trivial task, couple of existing Python libraries that can do this note that they support only a limited subset of full regex spec. So it seems that a most simple solution would be to add yet another user option to Keystone to ignore password strength enforcement for this given user, and amend Heat to set this option as well for internal users it creates. We in Heat may also think as to whether it would have any benefit to also set the 'lock_password' user option for the auto-created users which will prohibit such users to change their passwords via API themselves. I'd very like to hear opinion from Keystone community as most solutions I named are 'add new user option to Keystone' :-) [0] https://opendev.org/openstack/keystone/src/branch/master/keystone/conf/secu… [1] https://opendev.org/openstack/heat/src/branch/master/heat/common/password_g… Cheers, - Pavlo -- Dr. Pavlo Shchelokovskyy Principal Software Engineer Mirantis Inc www.mirantis.com

4 4

[sig][upgrades] job done
by James Page 02 May '19

02 May '19

Hi All After this mornings PTG session, I'm pleased to propose that we formally end the Upgrades SIG. That’s a “pleased” because we feel that our job as a SIG is done! Upgrades in OpenStack are no longer a "special interest"; they are now an integral part of the philosophy of projects within the OpenStack ecosystem and although there are probably still some rough edges, we don’t think we need a SIG to drive this area forward any longer. So thanks for all of the war stories, best practice discussion and general upgrade related conversation over the Forums, PTG’s and Summits over the last few years - it's been fun! Regards James

2 2

[nova][ptg] Summary: Stein Retrospective
by Eric Fried 02 May '19

02 May '19

Etherpad: https://etherpad.openstack.org/p/nova-ptg-train-retrospective Summary: - At least one newcomer was very happy with the welcome/support he received. Let's keep up the encouragement and make new long-term contributors - Placement extraction went pretty well. Forum session had no negative energy. (Is this because we planned really well, or because it wasn't that big a deal to begin with?) - Great work and collaboration on the bandwidth series. Framework will set us up nicely for other uses as well (e.g. cyborg). - Runways work. Let's keep using them. - Release themes: some people benefit from their existence; for others they are irrelevant but harmless. So let's keep doing them, since they benefit some. - Good coordination & communication with the TripleO team. - Long commit chains are hard. Things that have helped some people, which should be encouraged for the future: hangouts, videos, and/or emails, supplementing specs, acting as "review guidance". - The Stein release seemed a little less focused than usual. No cause or action was identified. - Pre-PTG emails around Placement were very effective for some, suggested to be used in the future for Nova, though noting the limitations and restricting to the parts of the design process not appropriate for other forums (like specs). Actions: - do themes (to be discussed Saturday 1200) - keep doing runways - (mriedem) hangout/video/review-guide-email for cross-cell resize work - Consider pre-PTG email threads for U efried .

1 0

[oslo] Proposing Hervé Beraud for Oslo Core
by Ben Nemec 02 May '19

02 May '19

Hi, Hervé has been working on Oslo for a while now and in that time has shown tremendous growth in his understanding of Oslo and OpenStack. I think he would make a good addition to the general Oslo core team. Existing Oslo team members (+Keystone, Castellan, and anyone else we co-own libraries with) please respond with +1/-1. If there are no objections I'll add him to the ACL next week and we can celebrate in person. :-) Thanks. -Ben

9 9

[openstack-dev] [neutron] Team picture
by Miguel Lavalle 02 May '19

02 May '19

Dear Neutrinos, Please remember that we will have out team picture taken at 11:50, NEXT TO THE PTG REGISTRATION DESK. Please be there on time Best regards Miguel

1 0

[requirements][qa][all] mock 3.0.0 released
by Doug Hellmann 02 May '19

02 May '19

There's a major version bump of one of our testing dependencies, so watch for new unit test job failures. Doug > Begin forwarded message: > > From: Chris Withers <chris(a)withers.org> > Subject: [TIP] mock 3.0.0 released > Date: May 2, 2019 at 2:07:34 AM MDT > To: "testing-in-python(a)lists.idyll.org" <testing-in-python(a)lists.idyll.org>, Python List <python-list(a)python.org> > > Hi All, > > I'm pleased to announce the release of mock 3.0.0: > https://pypi.org/project/mock/ > > This brings to rolling backport up to date with cpython master. > > It's been a few years since the last release, so I'd be surprised if there weren't some problems. > If you hit any issues, please pin to mock<3 and then: > > - If your issue relates to mock functionality, please report in the python tracker: https://bugs.python.org > > - If your issue is specific to the backport, please report here: https://github.com/testing-cabal/mock/issues > > If you're unsure, go for the second one and we'll figure it out. > > cheers, > > Chris > > _______________________________________________ > testing-in-python mailing list > testing-in-python(a)lists.idyll.org > http://lists.idyll.org/listinfo/testing-in-python

2 1

[stable] propose Tim Burke as stable core
by John Dickinson 02 May '19

02 May '19

Tim has been very active in proposing and maintaining patches to Swift’s stable branches. Of recent (non-automated) backports, Tim has proposed more than a third of them. --John

2 1

properly sizing openstack controlplane infrastructure
by Hartwig Hauschild 02 May '19

02 May '19

Hi everyone, A colleague and I have more or less recently been tasked with redesigning my employers openstack-infrastructure and it turned out that starting over will be easier than fixing the existing stack as we've managed to lock ourselves in a corner quite good. The requirements we've got are basically "here's 50 compute-nodes, make sure whatever you're building scales upwards from there". We have the existing pike-stack as a reference but we don't really know how the different services scale up with more compute-nodes to handle. The pike-stack has three servers as control-plane, each of them with 96G of RAM and they don't seem to have too much room left when coordinating 14 compute-nodes. We're thinking about splitting the control-nodes into infrastructure (db/rabbit/memcache) and API. What would I want to look for when sizing those control-nodes? I've not been able to find any references for this at all, just rather nebulous '8G RAM should do' which is around what our rabbit currently inhales. Also: We're currently running Neutron in OVS-DVR-VXLAN-Configuration. Does that properly scale up and above 50+ nodes or should we look into offloading the networking onto something else? I'm aware that the questions may sound a bit funny, but we have to spec out the controlplane-hardware before we can start testing and we'd prefer not to retrofit the servers because we goofed up. Thanks for your time, -- Cheers, Hardy

4 7

Kolla-ansible pike nova compute keeps restarting
by Shyam Biradar 02 May '19

02 May '19

Hi, I am setting up all-in-one ubuntu based kolla-ansible pike openstack. Deployment is failing at following ansible task: TASK [nova : include_tasks] ****************************** ************************************************************ **************************** included: /root/virtnev/share/kolla-ansible/ansible/roles/nova/tasks/discover_computes.yml for localhost TASK [nova : Waiting for nova-compute service up] ************************************************************ ************************************ FAILED - RETRYING: Waiting for nova-compute service up (20 retries left). FAILED - RETRYING: Waiting for nova-compute service up (19 retries left). FAILED - RETRYING: Waiting for nova-compute service up (18 retries left). FAILED - RETRYING: Waiting for nova-compute service up (17 retries left). FAILED - RETRYING: Waiting for nova-compute service up (16 retries left). FAILED - RETRYING: Waiting for nova-compute service up (15 retries left). FAILED - RETRYING: Waiting for nova-compute service up (14 retries left). FAILED - RETRYING: Waiting for nova-compute service up (13 retries left). FAILED - RETRYING: Waiting for nova-compute service up (12 retries left). FAILED - RETRYING: Waiting for nova-compute service up (11 retries left). FAILED - RETRYING: Waiting for nova-compute service up (10 retries left). FAILED - RETRYING: Waiting for nova-compute service up (9 retries left). FAILED - RETRYING: Waiting for nova-compute service up (8 retries left). FAILED - RETRYING: Waiting for nova-compute service up (7 retries left). FAILED - RETRYING: Waiting for nova-compute service up (6 retries left). FAILED - RETRYING: Waiting for nova-compute service up (5 retries left). FAILED - RETRYING: Waiting for nova-compute service up (4 retries left). FAILED - RETRYING: Waiting for nova-compute service up (3 retries left). FAILED - RETRYING: Waiting for nova-compute service up (2 retries left). FAILED - RETRYING: Waiting for nova-compute service up (1 retries left). fatal: [localhost -> localhost]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["docker", "exec", "kolla_toolbox", "openstack", "--os-interface", "internal", "--os-auth-url", "http://192.168.122.151:35357 ", "--os-identity-api-version", "3", "--os-project-domain-name", "default", "--os-tenant-name", "admin", "--os-username", "admin", "--os-password", " ivpu1km8qxnVQESvAF4cyTFstOvrbxGUHjFF15gZ", "--os-user-domain-name", "default", "compute", "service", "list", "-f", "json", "--service", "nova-compute"], "delta": "0:00:02.555356", "end": "2019-05-02 09:24:45.485786", "rc": 0, "start": "2019-05-02 09:24:42.930430", "stderr": "", "stderr_lines": [], "stdout": "[]", "stdout_lines": ["[]"]} -------------------------------------------------------------------- I can see following stack trace in nova-compute container log 4. 2019-05-02 08:21:30.522 7 INFO nova.service [-] Starting compute node (version 16.1.7) 2019-05-02 08:21:30.524 7 ERROR oslo_service.service [-] Error starting thread.: PlacementNotConfigured: This compute is not configured to talk to the placement service. Configure the [placement] section of nova.conf and restart the service. 2019-05-02 08:21:30.524 7 ERROR oslo_service.service Traceback (most recent call last): 2019-05-02 08:21:30.524 7 ERROR oslo_service.service File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_service/service.py", line 721, in run_service 2019-05-02 08:21:30.524 7 ERROR oslo_service.service service.start() 2019-05-02 08:21:30.524 7 ERROR oslo_service.service File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/service.py", line 156, in start 2019-05-02 08:21:30.524 7 ERROR oslo_service.service self.manager.init_host() 2019-05-02 08:21:30.524 7 ERROR oslo_service.service File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1155, in init_host 2019-05-02 08:21:30.524 7 ERROR oslo_service.service raise exception. PlacementNotConfigured() 2019-05-02 08:21:30.524 7 ERROR oslo_service.service PlacementNotConfigured: This compute is not configured to talk to the placement service. Configure the [placement] section of nova.conf and restart the service. 2019-05-02 08:21:30.524 7 ERROR oslo_service.service 2019-05-02 08:21:59.229 7 INFO os_vif [-] Loaded VIF plugins: ovs, linux_bridge --------------------------------------------------------------------- I saw nova-compute nova.conf has [placement] section configured well and it's same as nova_api's placement section. Other nova containers are started well. Any thoughts? Best Regards, Shyam Biradar Email Id: shyambiradarsggsit(a)gmail.com Contact: 8600266938

1 0

kube_cluster_deploy fails
by Pawel Konczalski 02 May '19

02 May '19

Hi, i try to deploy a kubernetes cluster on OpenStack rocky with Magnum. All Heat steps runs fine except for the last one "kube_cluster_deploy": kube_cluster_deploy c3a79e96-87d8-442e-83f1-91d7a1acd896 1 minute Create Failed CREATE aborted (Task create from SoftwareDeployment "kube_cluster_deploy" Stack "kubernetes-cluster9-l4ol4ger3key" [b17bfab3-f46a-477b-921f-4719bfd8df06] Timed out) The cluster contains three control / compute nodes, Octavia is also installed and works fine as far as I can see. Atomic Image: https://dl.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-2018041… The command was: openstack coe cluster template create kubernetes-cluster-template \ --image fedora-atomic-latest \ --external-network public \ --dns-nameserver 8.8.8.8 \ --master-flavor m1.small \ --flavor m1.large \ --coe kubernetes \ --docker-volume-size 10 \ --tls-disabled \ --floating-ip-enabled openstack coe cluster create kubernetes-cluster \ --cluster-template kubernetes-cluster-template \ --master-count 1 \ --node-count 1 \ --timeout 30 \ --keypair mykey Did someone run also into this Problem or know how to fix it? BR Pawel

3 4

[openstack-dev] [neutron] PTG agenda
by Miguel Lavalle 02 May '19

02 May '19

Hi Neutrinos,, I took your proposals for PTG topics and organized them in an agenda. Please look at https://etherpad.openstack.org/p/openstack-networking-train-ptg. Let's have a very productive meeting! Best regards Miguel

2 3

[baremetal-sig][ironic][ptg] Bare-metal whitepaper meeting at PTG
by Jacob Anders 02 May '19

02 May '19

Hi All, As discussed in the forum session earlier in the week, I would like to put together a session at the PTG for the Bare-metal SIG members to discuss the Bare Metal Whitepaper work and plan out next steps. Ironic schedule for the PTG is pretty tight but how about 4pm on the Friday? We could do this as a breakout in the main Ironic session. Who would be interested/available? Thanks, cheers, Jacob

1 0

[octavia] anchor discountinued?
by Volodymyr Litovka 02 May '19

02 May '19

Dear colleagues, it seems Anchor, which is used by Octavia as PKI system, is discontinued. Is there replacement for Anchor which can be used with Octavia? Thank you. -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison

3 3

[ETCD] client: etcd member http://controller:2379 has no leader
by Alejandro Ruiz Bermejo 02 May '19

02 May '19

Hi, i'm installing Zun in Openstack Queens with Ubuntu 18.04.1 LTS, i already have configured docker and kuyr-libnetwork. I'm following the guide at https://docs.openstack.org/zun/queens/install/index.html. I followed all the steps of the installation at controller node and everything resulted without problems. After finished the installation direction at compute node the *systemctl status zun-compute* have the following errors root@compute /h/team# systemctl status zun-compute ● zun-compute.service - OpenStack Container Service Compute Agent Loaded: loaded (/etc/systemd/system/zun-compute.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-04-30 16:46:56 UTC; 4h 26min ago Main PID: 2072 (zun-compute) Tasks: 1 (limit: 4915) CGroup: /system.slice/zun-compute.service └─2072 /usr/bin/python /usr/local/bin/zun-compute Apr 30 16:46:56 compute systemd[1]: Started OpenStack Container Service Compute Agent. Apr 30 16:46:57 compute zun-compute[2072]: 2019-04-30 16:46:57.929 2072 INFO zun.cmd.compute [-] Starting server in PID 2072 Apr 30 16:46:57 compute zun-compute[2072]: 2019-04-30 16:46:57.941 2072 INFO zun.container.driver [-] Loading container driver 'docker.driver.DockerDriver' Apr 30 16:46:58 compute zun-compute[2072]: 2019-04-30 16:46:58.028 2072 INFO zun.container.driver [-] Loading container driver 'docker.driver.DockerDriver' Apr 30 16:48:33 compute zun-compute[2072]: 2019-04-30 16:48:33.645 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'glance' Apr 30 16:48:33 compute zun-compute[2072]: 2019-04-30 16:48:33.911 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'glance' Apr 30 16:48:35 compute zun-compute[2072]: 2019-04-30 16:48:35.455 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'glance' Apr 30 16:48:35 compute zun-compute[2072]: 2019-04-30 16:48:35.939 2072 ERROR zun.image.glance.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Imae cirros was not found in glance: ImageNotFound: Image cirros could not be found. Apr 30 16:48:35 compute zun-compute[2072]: 2019-04-30 16:48:35.940 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'docker' Apr 30 16:48:55 compute zun-compute[2072]: 2019-04-30 16:48:55.011 2072 ERROR zun.compute.manager [req-7bfa764a-45b8-4e2f-ac70-84d8bb71b135 - - - - -] Error occurred while calling Docker create API: Docker internal error: 500 Server Error: Internal Server Error ("failed to update store for object typpe *libnetwork.endpointCnt: client: etcd member http://controller:2379 has no leader").: DockerError: Docker internal error: 500 Server Error: Internal Server Error ("failed to update store for object type *libnetwork.endpointtCnt: client: etcd member http://controller:2379 has no leader"). Also *systemctl status docker* show the next output root@compute /h/team# systemctl status docker ● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/docker.service.d └─docker.conf, http-proxy.conf, https-proxy.conf Active: active (running) since Tue 2019-04-30 16:46:25 UTC; 4h 18min ago Docs: https://docs.docker.com Main PID: 1777 (dockerd) Tasks: 21 CGroup: /system.slice/docker.service └─1777 /usr/bin/dockerd --group zun -H tcp://compute:2375 -H unix:///var/run/docker.sock --cluster-store etcd://controller:2379 Apr 30 16:46:20 compute dockerd[1777]: time="2019-04-30T16:46:20.815305836Z" level=warning msg="Your kernel does not support cgroup rt runtime" Apr 30 16:46:20 compute dockerd[1777]: time="2019-04-30T16:46:20.815933695Z" level=info msg="Loading containers: start." Apr 30 16:46:24 compute dockerd[1777]: time="2019-04-30T16:46:24.378526837Z" level=info msg="Default bridge (docker0) is assigned with an IP address 17 Apr 30 16:46:24 compute dockerd[1777]: time="2019-04-30T16:46:24.572558877Z" level=info msg="Loading containers: done." Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.198101219Z" level=info msg="Docker daemon" commit=e8ff056 graphdriver(s)=overlay2 vers Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.198211373Z" level=info msg="Daemon has completed initialization" Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.232286069Z" level=info msg="API listen on /var/run/docker.sock" Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.232318790Z" level=info msg="API listen on 10.8.9.58:2375" Apr 30 16:46:25 compute systemd[1]: Started Docker Application Container Engine. Apr 30 16:48:55 compute dockerd[1777]: time="2019-04-30T16:48:55.009820439Z" level=error msg="Handler for POST /v1.26/networks/create returned error: failed to update store for object type *libnetwork.endpointCnt: client: etcd member http://controller:2379 has no leader" When i try to launch an app container as the guide says it shows an Error state and when i run opentack appcontainer show this is the reason of the error status_reason | Docker internal error: 500 Server Error: Internal Server Error ("failed to update store for object type *libnetwork.endpointCnt: client: etcd member http://controller:2379 has no leader")

1 0

[magnm][ptg] Room for magnum this afternoon
by Spyros Trigazis 02 May '19

02 May '19

Hello everyone, Magnum will have two PTG sessions this afternoon [0] in Room 112. Note that magnum's track is not in the printed scheduled you have taken from the registration desk. You can join remotely in this etherpad [1]. Cheers, Spyros [0] http://ptg.openstack.org/ptg.html [1] https://etherpad.openstack.org/p/magnum-train-ptg

1 1

Re: [ops] how to get best io performance from my block devices
by Tim Bell 02 May '19

02 May '19

There are some hints in https://wiki.openstack.org/wiki/Documentation/HypervisorTuningGuide There are some tips in https://www.linux-kvm.org/page/Tuning_KVM too but you’d need to find the corresponding OpenStack flags on the guest/images/hosts/flavors. Overall, there are several options so it’s recommended to establish a baseline performance on a representative work load and try the various options. Tim From: Manuel Sopena Ballesteros <manuel.sb(a)garvan.org.au> Date: Wednesday, 1 May 2019 at 06:35 To: "openstack-discuss(a)lists.openstack.org" <openstack-discuss(a)lists.openstack.org> Subject: how to get best io performance from my block devices Dear Openstack community, I would like to have a high performance distributed database running in Openstack vms. I tried attaching dedicated nvme pci devices to the vm but the performance is not as good as I can get from bare metal. Bare metal: [root@zeus-54 data]# fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.1 Starting 1 process Jobs: 1 (f=1): [f(1)][100.0%][r=39.5MiB/s,w=39.6MiB/s][r=10.1k,w=10.1k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=50892: Wed May 1 22:22:45 2019 read: IOPS=9805, BW=38.3MiB/s (40.2MB/s)(4596MiB/120001msec) slat (usec): min=39, max=6678, avg=94.72, stdev=55.78 clat (nsec): min=450, max=18224, avg=525.83, stdev=120.10 lat (usec): min=39, max=6679, avg=95.36, stdev=55.79 clat percentiles (nsec): | 1.00th=[ 462], 5.00th=[ 478], 10.00th=[ 482], 20.00th=[ 486], | 30.00th=[ 490], 40.00th=[ 494], 50.00th=[ 502], 60.00th=[ 510], | 70.00th=[ 516], 80.00th=[ 532], 90.00th=[ 596], 95.00th=[ 676], | 99.00th=[ 860], 99.50th=[ 1048], 99.90th=[ 1384], 99.95th=[ 2480], | 99.99th=[ 3728] bw ( KiB/s): min= 720, max=40736, per=100.00%, avg=39389.00, stdev=5317.58, samples=239 iops : min= 180, max=10184, avg=9847.23, stdev=1329.39, samples=239 write: IOPS=9799, BW=38.3MiB/s (40.1MB/s)(4594MiB/120001msec) slat (nsec): min=2982, max=106207, avg=4220.09, stdev=980.04 clat (nsec): min=407, max=18130, avg=451.48, stdev=103.71 lat (usec): min=3, max=111, avg= 4.74, stdev= 1.03 clat percentiles (nsec): | 1.00th=[ 414], 5.00th=[ 418], 10.00th=[ 422], 20.00th=[ 430], | 30.00th=[ 434], 40.00th=[ 434], 50.00th=[ 438], 60.00th=[ 438], | 70.00th=[ 442], 80.00th=[ 446], 90.00th=[ 462], 95.00th=[ 588], | 99.00th=[ 700], 99.50th=[ 916], 99.90th=[ 1208], 99.95th=[ 1288], | 99.99th=[ 3536] bw ( KiB/s): min= 752, max=42608, per=100.00%, avg=39366.63, stdev=5355.73, samples=239 iops : min= 188, max=10652, avg=9841.64, stdev=1338.93, samples=239 lat (nsec) : 500=69.98%, 750=28.64%, 1000=0.90% lat (usec) : 2=0.42%, 4=0.04%, 10=0.01%, 20=0.01% cpu : usr=2.20%, sys=10.85%, ctx=1176675, majf=0, minf=1372 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=1176625,1175958,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=38.3MiB/s (40.2MB/s), 38.3MiB/s-38.3MiB/s (40.2MB/s-40.2MB/s), io=4596MiB (4819MB), run=120001-120001msec WRITE: bw=38.3MiB/s (40.1MB/s), 38.3MiB/s-38.3MiB/s (40.1MB/s-40.1MB/s), io=4594MiB (4817MB), run=120001-120001msec Disk stats (read/write): nvme9n1: ios=1174695/883620, merge=0/0, ticks=105502/72225, in_queue=192101, util=99.28% From vm: [centos@kudu-1 nvme0]$ sudo fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.1 Starting 1 process Jobs: 1 (f=1): [m(1)][100.0%][r=29.2MiB/s,w=29.7MiB/s][r=7487,w=7595 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=44383: Wed May 1 12:22:24 2019 read: IOPS=6994, BW=27.3MiB/s (28.6MB/s)(3278MiB/120000msec) slat (usec): min=54, max=20476, avg=115.27, stdev=71.45 clat (nsec): min=1757, max=31476, avg=2163.02, stdev=688.66 lat (usec): min=56, max=20481, avg=118.51, stdev=71.66 clat percentiles (nsec): | 1.00th=[ 1800], 5.00th=[ 1832], 10.00th=[ 1864], 20.00th=[ 1992], | 30.00th=[ 2040], 40.00th=[ 2064], 50.00th=[ 2064], 60.00th=[ 2096], | 70.00th=[ 2096], 80.00th=[ 2128], 90.00th=[ 2480], 95.00th=[ 2544], | 99.00th=[ 4448], 99.50th=[ 5536], 99.90th=[11072], 99.95th=[12736], | 99.99th=[18560] bw ( KiB/s): min= 952, max=31224, per=100.00%, avg=28153.51, stdev=4126.89, samples=237 iops : min= 238, max= 7806, avg=7038.23, stdev=1031.70, samples=237 write: IOPS=6985, BW=27.3MiB/s (28.6MB/s)(3274MiB/120000msec) slat (usec): min=7, max=963, avg=12.60, stdev= 6.24 clat (nsec): min=1662, max=199250, avg=2030.26, stdev=712.33 lat (usec): min=10, max=970, avg=15.68, stdev= 6.48 clat percentiles (nsec): | 1.00th=[ 1688], 5.00th=[ 1720], 10.00th=[ 1736], 20.00th=[ 1864], | 30.00th=[ 1928], 40.00th=[ 1944], 50.00th=[ 1944], 60.00th=[ 1960], | 70.00th=[ 1960], 80.00th=[ 1992], 90.00th=[ 2352], 95.00th=[ 2384], | 99.00th=[ 4048], 99.50th=[ 4768], 99.90th=[11456], 99.95th=[13120], | 99.99th=[19072] bw ( KiB/s): min= 912, max=31880, per=100.00%, avg=28119.64, stdev=4176.38, samples=237 iops : min= 228, max= 7970, avg=7029.75, stdev=1044.07, samples=237 lat (usec) : 2=51.56%, 4=47.17%, 10=1.03%, 20=0.22%, 50=0.01% lat (usec) : 250=0.01% cpu : usr=4.96%, sys=28.37%, ctx=839307, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=839283,838268,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3278MiB (3438MB), run=120000-120000msec WRITE: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3274MiB (3434MB), run=120000-120000msec Disk stats (read/write): nvme0n1: ios=838322/651596, merge=0/0, ticks=83804/22119, in_queue=104773, util=70.18% Is there a way I can get near bare metal performance from my nvme block devices? NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

1 0

Which are the most popular free open source OpenStack cloud operating systems or distros?
by Turritopsis Dohrnii Teo En Ming 02 May '19

02 May '19

Subject/Topic: Which are the most popular free open source OpenStack cloud operating systems or distros? Good afternoon from Singapore, First of all, I am very new to OpenStack. May I know which are the most popular free open source OpenStack cloud operating systems or distros? How do I download, install and deploy these OpenStack distros as private cloud, public cloud or hybrid cloud? Where can I find good and detailed documentation? Thank you very much for your advice. -----BEGIN EMAIL SIGNATURE----- The Gospel for all Targeted Individuals (TIs): [The New York Times] Microwave Weapons Are Prime Suspect in Ills of U.S. Embassy Workers Link: https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html ******************************************************************************************** Singaporean Mr. Turritopsis Dohrnii Teo En Ming's Academic Qualifications as at 14 Feb 2019 [1] https://tdtemcerts.wordpress.com/ [2] https://tdtemcerts.blogspot.sg/ [3] https://www.scribd.com/user/270125049/Teo-En-Ming -----END EMAIL SIGNATURE-----

4 3

[nova usage] openstack usage CLI output differs from horizon output
by Florian Engelmann 02 May '19

02 May '19

Hi, as far as I understood Horizon overview usage should give the same numbers as openstack usage show --project <project_name> --start 2019-03-01 --end 2019-03-31 But in our deployment (rocky) the Horizon numbers are higher (~15%). Any idea why? Could be a bug? All the best, Flo

1 0

[glance] [ops] Issue sharing an image with another project (something related to get_image_location)
by Massimo Sgaravatto 02 May '19

02 May '19

I have a small Rocky installation where Glance is configured with 2 backends (old images use the 'file' backend while new ones use the rbd backend, which is the default) show_multiple_locations is true but I have these settings in policy.json: # grep _image_location /etc/glance/policy.json "delete_image_location": "role:admin", "get_image_location": "role:admin", "set_image_location": "role:admin", This was done because of: https://wiki.openstack.org/wiki/OSSN/OSSN-0065 If an unpriv user tries to share a private image: $ openstack image add project 3194a04b-ffc8-4aaf-b6c8-adc24e3d3fe6 e81df4c0b493439abb8b85bfd4cbe071 403 Forbidden: Not allowed to create members for image 3194a04b-ffc8-4aaf-b6c8-adc24e3d3fe6. (HTTP 403) In the log file it looks like that the problem is related to the get_image_location operation: /var/log/glance/api.log:2019-04-29 16:06:54.523 8220 WARNING glance.api.v2.image_members [req-dd93cdc9-767d-4c51-8e5a-edf746c02264 ab573ba3ea014b778193b6922ffffe6d ee1865a76440481cbcff08544c7d580a - default default] Not allowed to create members for image 3194a04b-ffc8-4aaf-b6c8-adc24e3d3fe6.: Forbidden: You are not authorized to complete get_image_location action. But actually the sharing operation succeeded: $ glance member-list --image-id 3194a04b-ffc8-4aaf-b6c8-adc24e3d3fe6 +--------------------------------------+----------------------------------+---------+ | Image ID | Member ID | Status | +--------------------------------------+----------------------------------+---------+ | 3194a04b-ffc8-4aaf-b6c8-adc24e3d3fe6 | e81df4c0b493439abb8b85bfd4cbe071 | pending | +--------------------------------------+----------------------------------+---------+ Cheers, Massimo

2 3

[ptg] Interviews at PTG Denver
by Rain Leander 02 May '19

02 May '19

Hello all! I'm attending PTG this week to conduct project interviews [0]. These interviews have several purposes. Please consider all of the following when thinking about what you might want to say in your interview: * Tell the users/customers/press what you've been working on in Rocky * Give them some idea of what's (what might be?) coming in Stein * Put a human face on the OpenStack project and encourage new participants to join us * You're welcome to promote your company's involvement in OpenStack but we ask that you avoid any kind of product pitches or job recruitment In the interview I'll ask some leading questions and it'll go easier if you've given some thought to them ahead of time: * Who are you? (Your name, your employer, and the project(s) on which you are active.) * What did you accomplish in Rocky? (Focus on the 2-3 things that will be most interesting to cloud operators) * What do you expect to be the focus in Stein? (At the time of your interview, it's likely that the meetings will not yet have decided anything firm. That's ok.) * Anything further about the project(s) you work on or the OpenStack community in general. Finally, note that there are only 40 interview slots available, so please consider coordinating with your project to designate the people that you want to represent the project, so that we don't end up with 12 interview about Neutron, or whatever. I mean, love me some Neutron, but twelve interviews is a bit too many, eh? It's fine to have multiple people in one interview - Maximum 3, probably. Interview slots are 30 minutes, in which time we hope to capture somewhere between 10 and 20 minutes of content. It's fine to run shorter but 15 minutes is probably an ideal length. See you SOON! [0] https://docs.google.com/spreadsheets/d/1xZosqEL_iRI1Q-A5j-guRVh6Gc8-rRQ6SqK… -- K Rain Leander OpenStack Community Liaison Open Source and Standards Team https://www.rdoproject.org/ http://community.redhat.com

1 0

how to setup nvme pci-pasthrough to get close to native performance?
by Manuel Sopena Ballesteros 01 May '19

01 May '19

Dear Openstack community, I am configuring a high performance storage vms, I decided to go to the easy path (pci-passthrough), I can spin up vms and see the pci devices, however performance is below native/bare metal. Native/Bare metal performance: [root@zeus-54 data]# fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.1 Starting 1 process Jobs: 1 (f=1): [f(1)][100.0%][r=39.5MiB/s,w=39.6MiB/s][r=10.1k,w=10.1k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=50892: Wed May 1 22:22:45 2019 read: IOPS=9805, BW=38.3MiB/s (40.2MB/s)(4596MiB/120001msec) slat (usec): min=39, max=6678, avg=94.72, stdev=55.78 clat (nsec): min=450, max=18224, avg=525.83, stdev=120.10 lat (usec): min=39, max=6679, avg=95.36, stdev=55.79 clat percentiles (nsec): | 1.00th=[ 462], 5.00th=[ 478], 10.00th=[ 482], 20.00th=[ 486], | 30.00th=[ 490], 40.00th=[ 494], 50.00th=[ 502], 60.00th=[ 510], | 70.00th=[ 516], 80.00th=[ 532], 90.00th=[ 596], 95.00th=[ 676], | 99.00th=[ 860], 99.50th=[ 1048], 99.90th=[ 1384], 99.95th=[ 2480], | 99.99th=[ 3728] bw ( KiB/s): min= 720, max=40736, per=100.00%, avg=39389.00, stdev=5317.58, samples=239 iops : min= 180, max=10184, avg=9847.23, stdev=1329.39, samples=239 write: IOPS=9799, BW=38.3MiB/s (40.1MB/s)(4594MiB/120001msec) slat (nsec): min=2982, max=106207, avg=4220.09, stdev=980.04 clat (nsec): min=407, max=18130, avg=451.48, stdev=103.71 lat (usec): min=3, max=111, avg= 4.74, stdev= 1.03 clat percentiles (nsec): | 1.00th=[ 414], 5.00th=[ 418], 10.00th=[ 422], 20.00th=[ 430], | 30.00th=[ 434], 40.00th=[ 434], 50.00th=[ 438], 60.00th=[ 438], | 70.00th=[ 442], 80.00th=[ 446], 90.00th=[ 462], 95.00th=[ 588], | 99.00th=[ 700], 99.50th=[ 916], 99.90th=[ 1208], 99.95th=[ 1288], | 99.99th=[ 3536] bw ( KiB/s): min= 752, max=42608, per=100.00%, avg=39366.63, stdev=5355.73, samples=239 iops : min= 188, max=10652, avg=9841.64, stdev=1338.93, samples=239 lat (nsec) : 500=69.98%, 750=28.64%, 1000=0.90% lat (usec) : 2=0.42%, 4=0.04%, 10=0.01%, 20=0.01% cpu : usr=2.20%, sys=10.85%, ctx=1176675, majf=0, minf=1372 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=1176625,1175958,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=38.3MiB/s (40.2MB/s), 38.3MiB/s-38.3MiB/s (40.2MB/s-40.2MB/s), io=4596MiB (4819MB), run=120001-120001msec WRITE: bw=38.3MiB/s (40.1MB/s), 38.3MiB/s-38.3MiB/s (40.1MB/s-40.1MB/s), io=4594MiB (4817MB), run=120001-120001msec Disk stats (read/write): nvme9n1: ios=1174695/883620, merge=0/0, ticks=105502/72225, in_queue=192101, util=99.28% VM performance: [centos@kudu-1 nvme0]$ sudo fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.1 Starting 1 process Jobs: 1 (f=1): [m(1)][100.0%][r=29.2MiB/s,w=29.7MiB/s][r=7487,w=7595 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=44383: Wed May 1 12:22:24 2019 read: IOPS=6994, BW=27.3MiB/s (28.6MB/s)(3278MiB/120000msec) slat (usec): min=54, max=20476, avg=115.27, stdev=71.45 clat (nsec): min=1757, max=31476, avg=2163.02, stdev=688.66 lat (usec): min=56, max=20481, avg=118.51, stdev=71.66 clat percentiles (nsec): | 1.00th=[ 1800], 5.00th=[ 1832], 10.00th=[ 1864], 20.00th=[ 1992], | 30.00th=[ 2040], 40.00th=[ 2064], 50.00th=[ 2064], 60.00th=[ 2096], | 70.00th=[ 2096], 80.00th=[ 2128], 90.00th=[ 2480], 95.00th=[ 2544], | 99.00th=[ 4448], 99.50th=[ 5536], 99.90th=[11072], 99.95th=[12736], | 99.99th=[18560] bw ( KiB/s): min= 952, max=31224, per=100.00%, avg=28153.51, stdev=4126.89, samples=237 iops : min= 238, max= 7806, avg=7038.23, stdev=1031.70, samples=237 write: IOPS=6985, BW=27.3MiB/s (28.6MB/s)(3274MiB/120000msec) slat (usec): min=7, max=963, avg=12.60, stdev= 6.24 clat (nsec): min=1662, max=199250, avg=2030.26, stdev=712.33 lat (usec): min=10, max=970, avg=15.68, stdev= 6.48 clat percentiles (nsec): | 1.00th=[ 1688], 5.00th=[ 1720], 10.00th=[ 1736], 20.00th=[ 1864], | 30.00th=[ 1928], 40.00th=[ 1944], 50.00th=[ 1944], 60.00th=[ 1960], | 70.00th=[ 1960], 80.00th=[ 1992], 90.00th=[ 2352], 95.00th=[ 2384], | 99.00th=[ 4048], 99.50th=[ 4768], 99.90th=[11456], 99.95th=[13120], | 99.99th=[19072] bw ( KiB/s): min= 912, max=31880, per=100.00%, avg=28119.64, stdev=4176.38, samples=237 iops : min= 228, max= 7970, avg=7029.75, stdev=1044.07, samples=237 lat (usec) : 2=51.56%, 4=47.17%, 10=1.03%, 20=0.22%, 50=0.01% lat (usec) : 250=0.01% cpu : usr=4.96%, sys=28.37%, ctx=839307, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=839283,838268,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3278MiB (3438MB), run=120000-120000msec WRITE: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3274MiB (3434MB), run=120000-120000msec Disk stats (read/write): nvme0n1: ios=838322/651596, merge=0/0, ticks=83804/22119, in_queue=104773, util=70.18% This is my Openstack rocky configuration: nova.conf on controller node [pci] alias = { "vendor_id":"10de", "product_id":"1db1", "device_type":"type-PCI", "name":"nv_v100" } alias = { "vendor_id":"8086", "product_id":"0953", "device_type":"type-PCI", "name":"nvme"} nova.conf on compute node: [pci] passthrough_whitelist = [ {"address":"0000:84:00.0"}, {"address":"0000:85:00.0"}, {"address":"0000:86:00.0"}, {"address":"0000:87:00.0"} ] alias = { "vendor_id":"8086", "product_id":"0953", "device_type":"type-PCI", "name":"nvme"} This is how the nvmes are exposed to the vm <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x86' slot='0x00' function='0x0'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> Guest OS is centos 7.6 so I am guessing nvme drivers are included. Any help about what needs to my configuration to get close to native io performance? Thank you very much Manuel From: Manuel Sopena Ballesteros [mailto:manuel.sb@garvan.org.au] Sent: Wednesday, May 1, 2019 10:31 PM To: openstack-discuss(a)lists.openstack.org Subject: how to get best io performance from my block devices Dear Openstack community, I would like to have a high performance distributed database running in Openstack vms. I tried attaching dedicated nvme pci devices to the vm but the performance is not as good as I can get from bare metal. Bare metal: [root@zeus-54 data]# fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.1 Starting 1 process Jobs: 1 (f=1): [f(1)][100.0%][r=39.5MiB/s,w=39.6MiB/s][r=10.1k,w=10.1k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=50892: Wed May 1 22:22:45 2019 read: IOPS=9805, BW=38.3MiB/s (40.2MB/s)(4596MiB/120001msec) slat (usec): min=39, max=6678, avg=94.72, stdev=55.78 clat (nsec): min=450, max=18224, avg=525.83, stdev=120.10 lat (usec): min=39, max=6679, avg=95.36, stdev=55.79 clat percentiles (nsec): | 1.00th=[ 462], 5.00th=[ 478], 10.00th=[ 482], 20.00th=[ 486], | 30.00th=[ 490], 40.00th=[ 494], 50.00th=[ 502], 60.00th=[ 510], | 70.00th=[ 516], 80.00th=[ 532], 90.00th=[ 596], 95.00th=[ 676], | 99.00th=[ 860], 99.50th=[ 1048], 99.90th=[ 1384], 99.95th=[ 2480], | 99.99th=[ 3728] bw ( KiB/s): min= 720, max=40736, per=100.00%, avg=39389.00, stdev=5317.58, samples=239 iops : min= 180, max=10184, avg=9847.23, stdev=1329.39, samples=239 write: IOPS=9799, BW=38.3MiB/s (40.1MB/s)(4594MiB/120001msec) slat (nsec): min=2982, max=106207, avg=4220.09, stdev=980.04 clat (nsec): min=407, max=18130, avg=451.48, stdev=103.71 lat (usec): min=3, max=111, avg= 4.74, stdev= 1.03 clat percentiles (nsec): | 1.00th=[ 414], 5.00th=[ 418], 10.00th=[ 422], 20.00th=[ 430], | 30.00th=[ 434], 40.00th=[ 434], 50.00th=[ 438], 60.00th=[ 438], | 70.00th=[ 442], 80.00th=[ 446], 90.00th=[ 462], 95.00th=[ 588], | 99.00th=[ 700], 99.50th=[ 916], 99.90th=[ 1208], 99.95th=[ 1288], | 99.99th=[ 3536] bw ( KiB/s): min= 752, max=42608, per=100.00%, avg=39366.63, stdev=5355.73, samples=239 iops : min= 188, max=10652, avg=9841.64, stdev=1338.93, samples=239 lat (nsec) : 500=69.98%, 750=28.64%, 1000=0.90% lat (usec) : 2=0.42%, 4=0.04%, 10=0.01%, 20=0.01% cpu : usr=2.20%, sys=10.85%, ctx=1176675, majf=0, minf=1372 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=1176625,1175958,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=38.3MiB/s (40.2MB/s), 38.3MiB/s-38.3MiB/s (40.2MB/s-40.2MB/s), io=4596MiB (4819MB), run=120001-120001msec WRITE: bw=38.3MiB/s (40.1MB/s), 38.3MiB/s-38.3MiB/s (40.1MB/s-40.1MB/s), io=4594MiB (4817MB), run=120001-120001msec Disk stats (read/write): nvme9n1: ios=1174695/883620, merge=0/0, ticks=105502/72225, in_queue=192101, util=99.28% >From vm: [centos@kudu-1 nvme0]$ sudo fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.1 Starting 1 process Jobs: 1 (f=1): [m(1)][100.0%][r=29.2MiB/s,w=29.7MiB/s][r=7487,w=7595 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=44383: Wed May 1 12:22:24 2019 read: IOPS=6994, BW=27.3MiB/s (28.6MB/s)(3278MiB/120000msec) slat (usec): min=54, max=20476, avg=115.27, stdev=71.45 clat (nsec): min=1757, max=31476, avg=2163.02, stdev=688.66 lat (usec): min=56, max=20481, avg=118.51, stdev=71.66 clat percentiles (nsec): | 1.00th=[ 1800], 5.00th=[ 1832], 10.00th=[ 1864], 20.00th=[ 1992], | 30.00th=[ 2040], 40.00th=[ 2064], 50.00th=[ 2064], 60.00th=[ 2096], | 70.00th=[ 2096], 80.00th=[ 2128], 90.00th=[ 2480], 95.00th=[ 2544], | 99.00th=[ 4448], 99.50th=[ 5536], 99.90th=[11072], 99.95th=[12736], | 99.99th=[18560] bw ( KiB/s): min= 952, max=31224, per=100.00%, avg=28153.51, stdev=4126.89, samples=237 iops : min= 238, max= 7806, avg=7038.23, stdev=1031.70, samples=237 write: IOPS=6985, BW=27.3MiB/s (28.6MB/s)(3274MiB/120000msec) slat (usec): min=7, max=963, avg=12.60, stdev= 6.24 clat (nsec): min=1662, max=199250, avg=2030.26, stdev=712.33 lat (usec): min=10, max=970, avg=15.68, stdev= 6.48 clat percentiles (nsec): | 1.00th=[ 1688], 5.00th=[ 1720], 10.00th=[ 1736], 20.00th=[ 1864], | 30.00th=[ 1928], 40.00th=[ 1944], 50.00th=[ 1944], 60.00th=[ 1960], | 70.00th=[ 1960], 80.00th=[ 1992], 90.00th=[ 2352], 95.00th=[ 2384], | 99.00th=[ 4048], 99.50th=[ 4768], 99.90th=[11456], 99.95th=[13120], | 99.99th=[19072] bw ( KiB/s): min= 912, max=31880, per=100.00%, avg=28119.64, stdev=4176.38, samples=237 iops : min= 228, max= 7970, avg=7029.75, stdev=1044.07, samples=237 lat (usec) : 2=51.56%, 4=47.17%, 10=1.03%, 20=0.22%, 50=0.01% lat (usec) : 250=0.01% cpu : usr=4.96%, sys=28.37%, ctx=839307, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=839283,838268,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3278MiB (3438MB), run=120000-120000msec WRITE: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3274MiB (3434MB), run=120000-120000msec Disk stats (read/write): nvme0n1: ios=838322/651596, merge=0/0, ticks=83804/22119, in_queue=104773, util=70.18% Is there a way I can get near bare metal performance from my nvme block devices? NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed. NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

1 0

[cinder] [PTG] Room for Thursday Morning ...
by Jay Bryant 01 May '19

01 May '19

Team, There is some confusion with the schedule. Thought we were scheduled for room 203 in the morning but we weren't. Room 112, was free so I have booked that for our use Thursday morning. See you all there. Looking forward to a few productive days of discussion. Jay

1 0

[qa][ptg] QA Dinner on 2nd May @ 6.30 PM
by Ghanshyam Mann 01 May '19

01 May '19

Hi All, We have planned for QA dinner on 2nd May, Thursday 6.30 PM. Anyone is welcome to join. Here are the details: Restaurant: Indian Resturant ('Little India Downtown Denver') Map: shorturl.at/byDIJ Wednesday night, 6:30 PM Meeting at the restaurant directly. -gmann

1 0

[security-sig] Security SIG BoF Notes
by Gage Hugo 01 May '19

01 May '19

Thanks to everyone who attended the Security SIG BoF session! Attached are the notes taken from the discussion during the session with relevant links. If there was anything missed, please feel free to mention it here or reach out in #openstack-security. Board Picture: https://drive.google.com/open?id=1YWYdp9F5faGzlww1Cr7-i2TawDh60trg Topics: - Overall Security SIG - Links: - https://security.openstack.org/ - https://wiki.openstack.org/wiki/Security-SIG - Security SIG: https://wiki.openstack.org/wiki/Security-SIG - Weekly Agenda: https://etherpad.openstack.org/p/security-agenda - Meeting Time: Weekly on Thursday at 1500 UTC #openstack-meeting - IRC Server: irc.freenode.net - Key Lime: https://github.com/keylime/keylime - Integration with Ironic https://github.com/keylime/keylime/issues/101 - Bandit: https://github.com/PyCQA/bandit - Running bandit as part of tox gate - Keystone does this: https://github.com/openstack/keystone/blob/master/tox.ini#L40 - Run as a separate job - Example (not tox): https://github.com/openstack/openstack-helm/blob/master/zuul.d/jobs-opensta… - Host Intrusion - Wazuh was mentioned: https://wazuh.com/ - Ansible Hardening - OpenStack Ansible: https://docs.openstack.org/openstack-ansible/latest/ - Security SIG "Help Wanted" - https://docs.openstack.org/security-analysis/latest/ - Only has Barbican, missing other projects that have been added since - Multiple other libraries in review to be added - https://review.openstack.org/#/q/project:openstack/security-analysis+is:open - https://docs.openstack.org/security-guide/ - Security guide doesn’t seem to have been updated since Pike, so it’s a good 1.5 years behind - https://security.openstack.org/#secure-development-guidelines - Improve documentation of secure coding practices - improve coverage of bandit and syntribos jobs across projects, and look into other similar tools we could be using to better secure the software we write - https://wiki.openstack.org/wiki/Security_Notes - Help with writing security notes and triaging the backlog - https://wiki.openstack.org/wiki/Security/Security_Note_Process - https://bugs.launchpad.net/ossn - Security blog: http://openstack-security.github.io/ - VMT Public Bug Assistance - Many reports of suspected vulnerabilities start out as public bugs or are made public over the course of being triaged, and assistance with those is encouraged from the entire community - https://bugs.launchpad.net/ossa - Having someone who is familiar with the affected project provide context to a security bug really helps the VMT definine concrete impact statements and speeds up the overall process - Bootstrapping AWS / Windows Guest Domains / Guest VMs - nova-join: https://github.com/openstack/novajoin - application credentials: https://docs.openstack.org/keystone/latest/user/application_credentials.html - Barbican: https://wiki.openstack.org/wiki/Barbican - Policy - Cross-project policy effort: - https://governance.openstack.org/tc/goals/queens/policy-in-code.html - https://specs.openstack.org/openstack/keystone-specs/specs/keystone/ongoing…

1 0

[cloudkitty] May meeting is cancelled
by Luka Peschke 01 May '19

01 May '19

Hello everybody, Given that most of us won't be available on friday the 3rd, the cloudkitty IRC meeting that was planned at that date is cancelled. The next meeting will be held on june 7th at 15h UTC / 17h CET. Cheers, -- Luka Peschke

1 0

[OpenStack-Ansible][OSA] Team Dinner Denver
by Amy Marrich 01 May '19

01 May '19

We are looking at having our official team dinner Wednesday evening. Please visit this etherpad: https://etherpad.openstack.org/p/osa-team-dinner-plan To add your name and vote on restaurants so I can get a head count and make a reservation. Thanks, Amy (spotz)

2 1

[devstack] Identity URL problem
by Neil Jerram 01 May '19

01 May '19

Does anyone know what causes this problem at [1]: 2019-04-30 16:34:03.137 | +++ functions-common:oscwrap:2346 : command openstack role add admin --user neutron --project service --user-domain Default --project-domain Default 2019-04-30 16:34:03.139 | +++ functions-common:oscwrap:2346 : openstack role add admin --user neutron --project service --user-domain Default --project-domain Default 2019-04-30 16:34:04.331 | Failed to discover available identity versions when contacting http://104.239.175.234/identity. Attempting to parse version from URL. 2019-04-30 16:34:04.331 | Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Not Found (HTTP 404) [1] http://logs.openstack.org/79/638479/3/check/networking-calico-tempest-dsvm/… I think there are loads of uses of that URL, before where the networking-calico plugin uses it, so I can't see why the plugin's use hits that error. Thanks, Neil

2 3

[Nova][Neutron] Using Xen hypervisor On Openstack
by Gökhan IŞIK 01 May '19

01 May '19

Hi Team, I am trying to test hypervisors which OpenStack supported. So I installed 1 controller node , 1 xen compute node and 1 kvm compute node. I installed OpenStack Pike version. My kvm compute node works properly but I have problem about xen compute node. ı installed xen server 7.0 version to my server and on domU I created centos 7 virtual machine and ı installed nova-compute on it. For installing Openstack on xen I followed https://openstack-xenserver.readthedocs.io/en/latest/ guide. When I tried creating virtual machine , virtual machine is created but it has no ip. It didn't get any ip. I have no experince on xen server and ı don't know how ı can solve this problem. I looked at logs but I didn't see any errors. I need your help. I doubt of my neutron config. ı have 2 nics and one for management network and 1 for public network. These are my nics on dom0[ https://imgur.com/a/IrdLoCn] these are my nics on domU [https://imgur.com/a/5RliHa7] <https://imgur.com/a/5RliHa7> I am sending my ifconfig output on dom0[ http://paste.openstack.org/show/750146/] and domU[ http://paste.openstack.org/show/750147/] I am also sending my openvswitch_agent.ini file[ http://paste.openstack.org/show/750148/ ]. ı am waiting for your help. Thanks.

1 0

[Nova][Neutron] When Trying to Use Xen Hypervisor on OpenStack, Virtual machines can not get ip
by Gökhan IŞIK 01 May '19

01 May '19

Hi Team, I am trying to test hypervisors which OpenStack supported. So I installed 1 controller node , 1 xen compute node and 1 kvm compute node. I installed OpenStack Pike version. My kvm compute node works properly but I have problem about xen compute node. ı installed xen server 7.0 version to my server and on domU I created centos 7 virtual machine and ı installed nova-compute on it. For installing Openstack on xen I followed https://openstack-xenserver.readthedocs.io/en/latest/ guide. When I tried creating virtual machine , virtual machine is created but it has no ip. It didn't get any ip. I have no experince on xen server and ı don't know how ı can solve this problem. I looked at logs but I didn't see any errors. I need your help. I doubt of my neutron config. ı have 2 nics and one for management network and 1 for public network. These are my nics on dom0[ https://imgur.com/a/IrdLoCn] these are my nics on domU [https://imgur.com/a/5RliHa7] <https://imgur.com/a/5RliHa7> I am sending my ifconfig output on dom0[ http://paste.openstack.org/show/750146/] and domU[ http://paste.openstack.org/show/750147/] I am also sending my openvswitch_agent.ini file[ http://paste.openstack.org/show/750148/ ].

1 0

[ptg][sig][upgrades] Train PTG Upgrades SIG session
by James Page 01 May '19

01 May '19

Hi All I've finally found time to create an etherpad for the Upgrades SIG session at the upcoming PTG in Denver (on the train on my way to LHR to catch my flight). https://etherpad.openstack.org/p/upgrade-sig-ptg-train I've added a few proposed topics but if you're at the PTG (or summit) and have anything upgrade related to discuss please add your topic to the etherpad over the next few days - I'll then put together a rough schedule for our half day of upgrades discussion on Thursday morning in room 201. IRC meetings never really got restarted since the last PTG but I know that the promise of getting together to discuss upgrade successes and challenges generally appeals to us all based on prior sessions! Thanks in advance and see you all in Denver! Cheers James

1 1

how to get best io performance from my block devices
by Manuel Sopena Ballesteros 01 May '19

01 May '19

Dear Openstack community, I would like to have a high performance distributed database running in Openstack vms. I tried attaching dedicated nvme pci devices to the vm but the performance is not as good as I can get from bare metal. Bare metal: [root@zeus-54 data]# fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.1 Starting 1 process Jobs: 1 (f=1): [f(1)][100.0%][r=39.5MiB/s,w=39.6MiB/s][r=10.1k,w=10.1k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=50892: Wed May 1 22:22:45 2019 read: IOPS=9805, BW=38.3MiB/s (40.2MB/s)(4596MiB/120001msec) slat (usec): min=39, max=6678, avg=94.72, stdev=55.78 clat (nsec): min=450, max=18224, avg=525.83, stdev=120.10 lat (usec): min=39, max=6679, avg=95.36, stdev=55.79 clat percentiles (nsec): | 1.00th=[ 462], 5.00th=[ 478], 10.00th=[ 482], 20.00th=[ 486], | 30.00th=[ 490], 40.00th=[ 494], 50.00th=[ 502], 60.00th=[ 510], | 70.00th=[ 516], 80.00th=[ 532], 90.00th=[ 596], 95.00th=[ 676], | 99.00th=[ 860], 99.50th=[ 1048], 99.90th=[ 1384], 99.95th=[ 2480], | 99.99th=[ 3728] bw ( KiB/s): min= 720, max=40736, per=100.00%, avg=39389.00, stdev=5317.58, samples=239 iops : min= 180, max=10184, avg=9847.23, stdev=1329.39, samples=239 write: IOPS=9799, BW=38.3MiB/s (40.1MB/s)(4594MiB/120001msec) slat (nsec): min=2982, max=106207, avg=4220.09, stdev=980.04 clat (nsec): min=407, max=18130, avg=451.48, stdev=103.71 lat (usec): min=3, max=111, avg= 4.74, stdev= 1.03 clat percentiles (nsec): | 1.00th=[ 414], 5.00th=[ 418], 10.00th=[ 422], 20.00th=[ 430], | 30.00th=[ 434], 40.00th=[ 434], 50.00th=[ 438], 60.00th=[ 438], | 70.00th=[ 442], 80.00th=[ 446], 90.00th=[ 462], 95.00th=[ 588], | 99.00th=[ 700], 99.50th=[ 916], 99.90th=[ 1208], 99.95th=[ 1288], | 99.99th=[ 3536] bw ( KiB/s): min= 752, max=42608, per=100.00%, avg=39366.63, stdev=5355.73, samples=239 iops : min= 188, max=10652, avg=9841.64, stdev=1338.93, samples=239 lat (nsec) : 500=69.98%, 750=28.64%, 1000=0.90% lat (usec) : 2=0.42%, 4=0.04%, 10=0.01%, 20=0.01% cpu : usr=2.20%, sys=10.85%, ctx=1176675, majf=0, minf=1372 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=1176625,1175958,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=38.3MiB/s (40.2MB/s), 38.3MiB/s-38.3MiB/s (40.2MB/s-40.2MB/s), io=4596MiB (4819MB), run=120001-120001msec WRITE: bw=38.3MiB/s (40.1MB/s), 38.3MiB/s-38.3MiB/s (40.1MB/s-40.1MB/s), io=4594MiB (4817MB), run=120001-120001msec Disk stats (read/write): nvme9n1: ios=1174695/883620, merge=0/0, ticks=105502/72225, in_queue=192101, util=99.28% >From vm: [centos@kudu-1 nvme0]$ sudo fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.1 Starting 1 process Jobs: 1 (f=1): [m(1)][100.0%][r=29.2MiB/s,w=29.7MiB/s][r=7487,w=7595 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=44383: Wed May 1 12:22:24 2019 read: IOPS=6994, BW=27.3MiB/s (28.6MB/s)(3278MiB/120000msec) slat (usec): min=54, max=20476, avg=115.27, stdev=71.45 clat (nsec): min=1757, max=31476, avg=2163.02, stdev=688.66 lat (usec): min=56, max=20481, avg=118.51, stdev=71.66 clat percentiles (nsec): | 1.00th=[ 1800], 5.00th=[ 1832], 10.00th=[ 1864], 20.00th=[ 1992], | 30.00th=[ 2040], 40.00th=[ 2064], 50.00th=[ 2064], 60.00th=[ 2096], | 70.00th=[ 2096], 80.00th=[ 2128], 90.00th=[ 2480], 95.00th=[ 2544], | 99.00th=[ 4448], 99.50th=[ 5536], 99.90th=[11072], 99.95th=[12736], | 99.99th=[18560] bw ( KiB/s): min= 952, max=31224, per=100.00%, avg=28153.51, stdev=4126.89, samples=237 iops : min= 238, max= 7806, avg=7038.23, stdev=1031.70, samples=237 write: IOPS=6985, BW=27.3MiB/s (28.6MB/s)(3274MiB/120000msec) slat (usec): min=7, max=963, avg=12.60, stdev= 6.24 clat (nsec): min=1662, max=199250, avg=2030.26, stdev=712.33 lat (usec): min=10, max=970, avg=15.68, stdev= 6.48 clat percentiles (nsec): | 1.00th=[ 1688], 5.00th=[ 1720], 10.00th=[ 1736], 20.00th=[ 1864], | 30.00th=[ 1928], 40.00th=[ 1944], 50.00th=[ 1944], 60.00th=[ 1960], | 70.00th=[ 1960], 80.00th=[ 1992], 90.00th=[ 2352], 95.00th=[ 2384], | 99.00th=[ 4048], 99.50th=[ 4768], 99.90th=[11456], 99.95th=[13120], | 99.99th=[19072] bw ( KiB/s): min= 912, max=31880, per=100.00%, avg=28119.64, stdev=4176.38, samples=237 iops : min= 228, max= 7970, avg=7029.75, stdev=1044.07, samples=237 lat (usec) : 2=51.56%, 4=47.17%, 10=1.03%, 20=0.22%, 50=0.01% lat (usec) : 250=0.01% cpu : usr=4.96%, sys=28.37%, ctx=839307, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=839283,838268,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3278MiB (3438MB), run=120000-120000msec WRITE: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3274MiB (3434MB), run=120000-120000msec Disk stats (read/write): nvme0n1: ios=838322/651596, merge=0/0, ticks=83804/22119, in_queue=104773, util=70.18% Is there a way I can get near bare metal performance from my nvme block devices? NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

1 0

[PTG][I18n] Additional Virtual PTG scheduling
by Ian Y. Choi 01 May '19

01 May '19

Hello, Although I shared my priorities as I18n PTL during Train cycle [1], I couldn't attend to the PTG at this time. There will be Docs+I18n PTG tomorrow in Denver with lots of discussions for more on cross-project collaboration with Docs team, and other teams, and I wanna join as remote on 16:00-17:00 according to [2] (thanks a lot, Frank!). I want to design an additional virtual PTG event, which some of other teams also design something similar as, but would like to plan somewhat differently to reflect I18n team members' geographical & language diversity as much as possible. Any translators, language coordinators, and contributors are welcome with the following cadence: - Please allocate your 30 minutes on May 2 (according to Denver timezone). - Please visit https://ethercalc.openstack.org/i18n-virtual-ptg-train and grasp how I18n Virtual PTG operates. - Choose your best time and write your name, country, preferred comm method, and notes by filling out cells on H19-K66. I will be online on IRC or Zoom (or please share your best communication method - I will follow as much as possible). This might be something different from general cadence on PTG, but I really hope that I18n team will have better communication through such activities. Please join in the discussion - I will reflect all of opinions as much as possible for better I18n world during this cycle. Note that I purposely marked some of my unavailable time slots but can be adjusted well - believe me, since someone asks me when I sleep (although it is getting harder.. :p ) With many thanks, /Ian [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003757.ht… [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005668.ht…

1 0

[Ceilometer]: cpu_util meter not being calculated as expected leading to delay in scaling
by Arvind Kumar 01 May '19

01 May '19

Hi, A design issue is observed in ceilometer service of Openstack. Setup include multiple compute nodes and 3 controller nodes. Meters from each compute node are sent to all the 3 ceilometer instances via RabbitMQ in round robin fashion at an interval of 10 min. After transformation of cumulative cpu meter data, cpu_util is generated by ceilometer instance at controller node and is published to the http address configured in ceilometer pipeline configuration. cpu_util is used by the application to take the decision if scaling of VM needs to be triggered or not. Ceilometer instance calculates cpu_util for a VM from the difference between cumulative cpu usage of VM at two timestamp divided by the timestamp difference. Let's say 1 compute node send the cumulative cpu usage of a VM (C1, C2, C3, C4) at timestamp T1, T2, T3, T4 (difference between any two timestamp is 10 min). Now (C1,T1) & (C4,T4) tuple is received by ceilometer instance 1, (C2,T2) by instance 2, (C3,T3) by instance 3. Here even if CPU usage of VM is increased between T1 & T2, cpu_util is calculated for 30 min duration (T1 & T4) rather than as expected for 10 min. This leads to scaling getting triggered after T4 that too when CPU usage is consistently above the threshold between T1 and T4. Please suggest how could this issue could be resolved. Do we have any solution to bind VM or compute node meter data to specific ceilometer instance for processing? Regards, Arvind. "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

2 2

[placement][nova][ironic][blazar][ptg] Placement PTG Agenda
by Chris Dent 30 Apr '19

30 Apr '19

Near the top of the placement ptg etherpad [1] I've sketched out a schedule for the end of this week for those who happen to be in Denver. Since so many of the placement team will be required elsewhere, it is pretty thin. I think this is okay because a) we got quite a bit accomplished during the pre-PTG emails, b) the main things we need to discuss [2] will be strongly informed by other discussion in the week and need to be revisited several times. The summary of the schedule is: Thursday: 14:30-Beer: In the nova room doing cross project stuff Friday: Morning: wherever you need to be, often nova room Afternoon: In the placement room (for those who can) to capture and clarify results of the Thursday session and Friday morning and topics as people present allows. Saturday: Morning: Ironic/Blazar/Placement/Anyone else interested in using placement. Afternoon: Capture and clarify, retrospective, refactoring goals, hacking. The topics in [2] (and all the related emails) will be mixed into Thursday, Friday and Saturday afternoons. Thank you for whatever time you're able to make available. If you have conflicts, don't worry, everything will get summarized later and if it is properly important will come up again. [1] https://etherpad.openstack.org/p/placement-ptg-train [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005715.ht… -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

1 0

[Zun] openstack appcontainer create Error
by Alejandro Ruiz Bermejo 30 Apr '19

30 Apr '19

Hi, i'm installing Zun in Openstack Queens with Ubuntu 18.04.1 LTS, i already have configured docker and kuyr-libnetwork. I'm following the guide at https://docs.openstack.org/zun/queens/install/index.html. I followed all the steps of the installation at controller node and everything resulted without problems. After finished the installation direction at compute node the *systemctl status zun-compute* have the following errors root@compute /h/team# systemctl status zun-compute ● zun-compute.service - OpenStack Container Service Compute Agent Loaded: loaded (/etc/systemd/system/zun-compute.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-04-30 16:46:56 UTC; 4h 26min ago Main PID: 2072 (zun-compute) Tasks: 1 (limit: 4915) CGroup: /system.slice/zun-compute.service └─2072 /usr/bin/python /usr/local/bin/zun-compute Apr 30 16:46:56 compute systemd[1]: Started OpenStack Container Service Compute Agent. Apr 30 16:46:57 compute zun-compute[2072]: 2019-04-30 16:46:57.929 2072 INFO zun.cmd.compute [-] Starting server in PID 2072 Apr 30 16:46:57 compute zun-compute[2072]: 2019-04-30 16:46:57.941 2072 INFO zun.container.driver [-] Loading container driver 'docker.driver.DockerDriver' Apr 30 16:46:58 compute zun-compute[2072]: 2019-04-30 16:46:58.028 2072 INFO zun.container.driver [-] Loading container driver 'docker.driver.DockerDriver' Apr 30 16:48:33 compute zun-compute[2072]: 2019-04-30 16:48:33.645 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'glance' Apr 30 16:48:33 compute zun-compute[2072]: 2019-04-30 16:48:33.911 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'glance' Apr 30 16:48:35 compute zun-compute[2072]: 2019-04-30 16:48:35.455 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'glance' Apr 30 16:48:35 compute zun-compute[2072]: 2019-04-30 16:48:35.939 2072 ERROR zun.image.glance.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Imae cirros was not found in glance: ImageNotFound: Image cirros could not be found. Apr 30 16:48:35 compute zun-compute[2072]: 2019-04-30 16:48:35.940 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'docker' Apr 30 16:48:55 compute zun-compute[2072]: 2019-04-30 16:48:55.011 2072 ERROR zun.compute.manager [req-7bfa764a-45b8-4e2f-ac70-84d8bb71b135 - - - - -] Error occurred while calling Docker create API: Docker internal error: 500 Server Error: Internal Server Error ("failed to update store for object typpe *libnetwork.endpointCnt: client: etcd member http://controller:2379 has no leader").: DockerError: Docker internal error: 500 Server Error: Internal Server Error ("failed to update store for object type *libnetwork.endpointtCnt: client: etcd member http://controller:2379 has no leader"). Also *systemctl status docker* show the next output root@compute /h/team# systemctl status docker ● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/docker.service.d └─docker.conf, http-proxy.conf, https-proxy.conf Active: active (running) since Tue 2019-04-30 16:46:25 UTC; 4h 18min ago Docs: https://docs.docker.com Main PID: 1777 (dockerd) Tasks: 21 CGroup: /system.slice/docker.service └─1777 /usr/bin/dockerd --group zun -H tcp://compute:2375 -H unix:///var/run/docker.sock --cluster-store etcd://controller:2379 Apr 30 16:46:20 compute dockerd[1777]: time="2019-04-30T16:46:20.815305836Z" level=warning msg="Your kernel does not support cgroup rt runtime" Apr 30 16:46:20 compute dockerd[1777]: time="2019-04-30T16:46:20.815933695Z" level=info msg="Loading containers: start." Apr 30 16:46:24 compute dockerd[1777]: time="2019-04-30T16:46:24.378526837Z" level=info msg="Default bridge (docker0) is assigned with an IP address 17 Apr 30 16:46:24 compute dockerd[1777]: time="2019-04-30T16:46:24.572558877Z" level=info msg="Loading containers: done." Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.198101219Z" level=info msg="Docker daemon" commit=e8ff056 graphdriver(s)=overlay2 vers Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.198211373Z" level=info msg="Daemon has completed initialization" Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.232286069Z" level=info msg="API listen on /var/run/docker.sock" Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.232318790Z" level=info msg="API listen on 10.8.9.58:2375" Apr 30 16:46:25 compute systemd[1]: Started Docker Application Container Engine. Apr 30 16:48:55 compute dockerd[1777]: time="2019-04-30T16:48:55.009820439Z" level=error msg="Handler for POST /v1.26/networks/create returned error: failed to update store for object type *libnetwork.endpointCnt: client: etcd member http://controller:2379 has no leader" When i try to launch an app container as the guide says it shows an Error state and when i run opentack appcontainer show this is the reason of the error status_reason | Docker internal error: 500 Server Error: Internal Server Error ("failed to update store for object type *libnetwork.endpointCnt: client: etcd member http://controller:2379 has no leader") I had some troubles installing Kuryr-libnetwork besides that i didn't had any othet problem during the installation of Zun

2 1