May 2019 - openstack-discuss - lists.openstack.org

[Nova][Neutron] When Trying to Use Xen Hypervisor on OpenStack, Virtual machines can not get ip
by Gökhan IŞIK 01 May '19

01 May '19

Hi Team, I am trying to test hypervisors which OpenStack supported. So I installed 1 controller node , 1 xen compute node and 1 kvm compute node. I installed OpenStack Pike version. My kvm compute node works properly but I have problem about xen compute node. ı installed xen server 7.0 version to my server and on domU I created centos 7 virtual machine and ı installed nova-compute on it. For installing Openstack on xen I followed https://openstack-xenserver.readthedocs.io/en/latest/ guide. When I tried creating virtual machine , virtual machine is created but it has no ip. It didn't get any ip. I have no experince on xen server and ı don't know how ı can solve this problem. I looked at logs but I didn't see any errors. I need your help. I doubt of my neutron config. ı have 2 nics and one for management network and 1 for public network. These are my nics on dom0[ https://imgur.com/a/IrdLoCn]. these are my nics on domU [https://imgur.com/a/5RliHa7]. <https://imgur.com/a/5RliHa7> I am sending my ifconfig output on dom0[ http://paste.openstack.org/show/750146/] and domU[ http://paste.openstack.org/show/750147/]. I am also sending my openvswitch_agent.ini file[ http://paste.openstack.org/show/750148/ ].

1 0

[ptg][sig][upgrades] Train PTG Upgrades SIG session
by James Page 01 May '19

01 May '19

Hi All I've finally found time to create an etherpad for the Upgrades SIG session at the upcoming PTG in Denver (on the train on my way to LHR to catch my flight). https://etherpad.openstack.org/p/upgrade-sig-ptg-train I've added a few proposed topics but if you're at the PTG (or summit) and have anything upgrade related to discuss please add your topic to the etherpad over the next few days - I'll then put together a rough schedule for our half day of upgrades discussion on Thursday morning in room 201. IRC meetings never really got restarted since the last PTG but I know that the promise of getting together to discuss upgrade successes and challenges generally appeals to us all based on prior sessions! Thanks in advance and see you all in Denver! Cheers James

1 1

how to get best io performance from my block devices
by Manuel Sopena Ballesteros 01 May '19

01 May '19

Dear Openstack community, I would like to have a high performance distributed database running in Openstack vms. I tried attaching dedicated nvme pci devices to the vm but the performance is not as good as I can get from bare metal. Bare metal: [root@zeus-54 data]# fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.1 Starting 1 process Jobs: 1 (f=1): [f(1)][100.0%][r=39.5MiB/s,w=39.6MiB/s][r=10.1k,w=10.1k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=50892: Wed May 1 22:22:45 2019 read: IOPS=9805, BW=38.3MiB/s (40.2MB/s)(4596MiB/120001msec) slat (usec): min=39, max=6678, avg=94.72, stdev=55.78 clat (nsec): min=450, max=18224, avg=525.83, stdev=120.10 lat (usec): min=39, max=6679, avg=95.36, stdev=55.79 clat percentiles (nsec): | 1.00th=[ 462], 5.00th=[ 478], 10.00th=[ 482], 20.00th=[ 486], | 30.00th=[ 490], 40.00th=[ 494], 50.00th=[ 502], 60.00th=[ 510], | 70.00th=[ 516], 80.00th=[ 532], 90.00th=[ 596], 95.00th=[ 676], | 99.00th=[ 860], 99.50th=[ 1048], 99.90th=[ 1384], 99.95th=[ 2480], | 99.99th=[ 3728] bw ( KiB/s): min= 720, max=40736, per=100.00%, avg=39389.00, stdev=5317.58, samples=239 iops : min= 180, max=10184, avg=9847.23, stdev=1329.39, samples=239 write: IOPS=9799, BW=38.3MiB/s (40.1MB/s)(4594MiB/120001msec) slat (nsec): min=2982, max=106207, avg=4220.09, stdev=980.04 clat (nsec): min=407, max=18130, avg=451.48, stdev=103.71 lat (usec): min=3, max=111, avg= 4.74, stdev= 1.03 clat percentiles (nsec): | 1.00th=[ 414], 5.00th=[ 418], 10.00th=[ 422], 20.00th=[ 430], | 30.00th=[ 434], 40.00th=[ 434], 50.00th=[ 438], 60.00th=[ 438], | 70.00th=[ 442], 80.00th=[ 446], 90.00th=[ 462], 95.00th=[ 588], | 99.00th=[ 700], 99.50th=[ 916], 99.90th=[ 1208], 99.95th=[ 1288], | 99.99th=[ 3536] bw ( KiB/s): min= 752, max=42608, per=100.00%, avg=39366.63, stdev=5355.73, samples=239 iops : min= 188, max=10652, avg=9841.64, stdev=1338.93, samples=239 lat (nsec) : 500=69.98%, 750=28.64%, 1000=0.90% lat (usec) : 2=0.42%, 4=0.04%, 10=0.01%, 20=0.01% cpu : usr=2.20%, sys=10.85%, ctx=1176675, majf=0, minf=1372 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=1176625,1175958,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=38.3MiB/s (40.2MB/s), 38.3MiB/s-38.3MiB/s (40.2MB/s-40.2MB/s), io=4596MiB (4819MB), run=120001-120001msec WRITE: bw=38.3MiB/s (40.1MB/s), 38.3MiB/s-38.3MiB/s (40.1MB/s-40.1MB/s), io=4594MiB (4817MB), run=120001-120001msec Disk stats (read/write): nvme9n1: ios=1174695/883620, merge=0/0, ticks=105502/72225, in_queue=192101, util=99.28% >From vm: [centos@kudu-1 nvme0]$ sudo fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.1 Starting 1 process Jobs: 1 (f=1): [m(1)][100.0%][r=29.2MiB/s,w=29.7MiB/s][r=7487,w=7595 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=44383: Wed May 1 12:22:24 2019 read: IOPS=6994, BW=27.3MiB/s (28.6MB/s)(3278MiB/120000msec) slat (usec): min=54, max=20476, avg=115.27, stdev=71.45 clat (nsec): min=1757, max=31476, avg=2163.02, stdev=688.66 lat (usec): min=56, max=20481, avg=118.51, stdev=71.66 clat percentiles (nsec): | 1.00th=[ 1800], 5.00th=[ 1832], 10.00th=[ 1864], 20.00th=[ 1992], | 30.00th=[ 2040], 40.00th=[ 2064], 50.00th=[ 2064], 60.00th=[ 2096], | 70.00th=[ 2096], 80.00th=[ 2128], 90.00th=[ 2480], 95.00th=[ 2544], | 99.00th=[ 4448], 99.50th=[ 5536], 99.90th=[11072], 99.95th=[12736], | 99.99th=[18560] bw ( KiB/s): min= 952, max=31224, per=100.00%, avg=28153.51, stdev=4126.89, samples=237 iops : min= 238, max= 7806, avg=7038.23, stdev=1031.70, samples=237 write: IOPS=6985, BW=27.3MiB/s (28.6MB/s)(3274MiB/120000msec) slat (usec): min=7, max=963, avg=12.60, stdev= 6.24 clat (nsec): min=1662, max=199250, avg=2030.26, stdev=712.33 lat (usec): min=10, max=970, avg=15.68, stdev= 6.48 clat percentiles (nsec): | 1.00th=[ 1688], 5.00th=[ 1720], 10.00th=[ 1736], 20.00th=[ 1864], | 30.00th=[ 1928], 40.00th=[ 1944], 50.00th=[ 1944], 60.00th=[ 1960], | 70.00th=[ 1960], 80.00th=[ 1992], 90.00th=[ 2352], 95.00th=[ 2384], | 99.00th=[ 4048], 99.50th=[ 4768], 99.90th=[11456], 99.95th=[13120], | 99.99th=[19072] bw ( KiB/s): min= 912, max=31880, per=100.00%, avg=28119.64, stdev=4176.38, samples=237 iops : min= 228, max= 7970, avg=7029.75, stdev=1044.07, samples=237 lat (usec) : 2=51.56%, 4=47.17%, 10=1.03%, 20=0.22%, 50=0.01% lat (usec) : 250=0.01% cpu : usr=4.96%, sys=28.37%, ctx=839307, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=839283,838268,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3278MiB (3438MB), run=120000-120000msec WRITE: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3274MiB (3434MB), run=120000-120000msec Disk stats (read/write): nvme0n1: ios=838322/651596, merge=0/0, ticks=83804/22119, in_queue=104773, util=70.18% Is there a way I can get near bare metal performance from my nvme block devices? NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

1 0

[PTG][I18n] Additional Virtual PTG scheduling
by Ian Y. Choi 01 May '19

01 May '19

Hello, Although I shared my priorities as I18n PTL during Train cycle [1], I couldn't attend to the PTG at this time. There will be Docs+I18n PTG tomorrow in Denver with lots of discussions for more on cross-project collaboration with Docs team, and other teams, and I wanna join as remote on 16:00-17:00 according to [2] (thanks a lot, Frank!). I want to design an additional virtual PTG event, which some of other teams also design something similar as, but would like to plan somewhat differently to reflect I18n team members' geographical & language diversity as much as possible. Any translators, language coordinators, and contributors are welcome with the following cadence: - Please allocate your 30 minutes on May 2 (according to Denver timezone). - Please visit https://ethercalc.openstack.org/i18n-virtual-ptg-train and grasp how I18n Virtual PTG operates. - Choose your best time and write your name, country, preferred comm method, and notes by filling out cells on H19-K66. I will be online on IRC or Zoom (or please share your best communication method - I will follow as much as possible). This might be something different from general cadence on PTG, but I really hope that I18n team will have better communication through such activities. Please join in the discussion - I will reflect all of opinions as much as possible for better I18n world during this cycle. Note that I purposely marked some of my unavailable time slots but can be adjusted well - believe me, since someone asks me when I sleep (although it is getting harder.. :p ) With many thanks, /Ian [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003757.ht… [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005668.ht…

1 0

[Ceilometer]: cpu_util meter not being calculated as expected leading to delay in scaling
by Arvind Kumar 01 May '19

01 May '19

Hi, A design issue is observed in ceilometer service of Openstack. Setup include multiple compute nodes and 3 controller nodes. Meters from each compute node are sent to all the 3 ceilometer instances via RabbitMQ in round robin fashion at an interval of 10 min. After transformation of cumulative cpu meter data, cpu_util is generated by ceilometer instance at controller node and is published to the http address configured in ceilometer pipeline configuration. cpu_util is used by the application to take the decision if scaling of VM needs to be triggered or not. Ceilometer instance calculates cpu_util for a VM from the difference between cumulative cpu usage of VM at two timestamp divided by the timestamp difference. Let's say 1 compute node send the cumulative cpu usage of a VM (C1, C2, C3, C4) at timestamp T1, T2, T3, T4 (difference between any two timestamp is 10 min). Now (C1,T1) & (C4,T4) tuple is received by ceilometer instance 1, (C2,T2) by instance 2, (C3,T3) by instance 3. Here even if CPU usage of VM is increased between T1 & T2, cpu_util is calculated for 30 min duration (T1 & T4) rather than as expected for 10 min. This leads to scaling getting triggered after T4 that too when CPU usage is consistently above the threshold between T1 and T4. Please suggest how could this issue could be resolved. Do we have any solution to bind VM or compute node meter data to specific ceilometer instance for processing? Regards, Arvind. "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

2 2

[placement][nova][ironic][blazar][ptg] Placement PTG Agenda
by Chris Dent 01 May '19

01 May '19

Near the top of the placement ptg etherpad [1] I've sketched out a schedule for the end of this week for those who happen to be in Denver. Since so many of the placement team will be required elsewhere, it is pretty thin. I think this is okay because a) we got quite a bit accomplished during the pre-PTG emails, b) the main things we need to discuss [2] will be strongly informed by other discussion in the week and need to be revisited several times. The summary of the schedule is: Thursday: 14:30-Beer: In the nova room doing cross project stuff Friday: Morning: wherever you need to be, often nova room Afternoon: In the placement room (for those who can) to capture and clarify results of the Thursday session and Friday morning and topics as people present allows. Saturday: Morning: Ironic/Blazar/Placement/Anyone else interested in using placement. Afternoon: Capture and clarify, retrospective, refactoring goals, hacking. The topics in [2] (and all the related emails) will be mixed into Thursday, Friday and Saturday afternoons. Thank you for whatever time you're able to make available. If you have conflicts, don't worry, everything will get summarized later and if it is properly important will come up again. [1] https://etherpad.openstack.org/p/placement-ptg-train [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005715.ht… -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

1 0

[Zun] openstack appcontainer create Error
by Alejandro Ruiz Bermejo 01 May '19

01 May '19

Hi, i'm installing Zun in Openstack Queens with Ubuntu 18.04.1 LTS, i already have configured docker and kuyr-libnetwork. I'm following the guide at https://docs.openstack.org/zun/queens/install/index.html. I followed all the steps of the installation at controller node and everything resulted without problems. After finished the installation direction at compute node the *systemctl status zun-compute* have the following errors root@compute /h/team# systemctl status zun-compute ● zun-compute.service - OpenStack Container Service Compute Agent Loaded: loaded (/etc/systemd/system/zun-compute.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-04-30 16:46:56 UTC; 4h 26min ago Main PID: 2072 (zun-compute) Tasks: 1 (limit: 4915) CGroup: /system.slice/zun-compute.service └─2072 /usr/bin/python /usr/local/bin/zun-compute Apr 30 16:46:56 compute systemd[1]: Started OpenStack Container Service Compute Agent. Apr 30 16:46:57 compute zun-compute[2072]: 2019-04-30 16:46:57.929 2072 INFO zun.cmd.compute [-] Starting server in PID 2072 Apr 30 16:46:57 compute zun-compute[2072]: 2019-04-30 16:46:57.941 2072 INFO zun.container.driver [-] Loading container driver 'docker.driver.DockerDriver' Apr 30 16:46:58 compute zun-compute[2072]: 2019-04-30 16:46:58.028 2072 INFO zun.container.driver [-] Loading container driver 'docker.driver.DockerDriver' Apr 30 16:48:33 compute zun-compute[2072]: 2019-04-30 16:48:33.645 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'glance' Apr 30 16:48:33 compute zun-compute[2072]: 2019-04-30 16:48:33.911 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'glance' Apr 30 16:48:35 compute zun-compute[2072]: 2019-04-30 16:48:35.455 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'glance' Apr 30 16:48:35 compute zun-compute[2072]: 2019-04-30 16:48:35.939 2072 ERROR zun.image.glance.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Imae cirros was not found in glance: ImageNotFound: Image cirros could not be found. Apr 30 16:48:35 compute zun-compute[2072]: 2019-04-30 16:48:35.940 2072 INFO zun.image.driver [req-7e0b8325-1e09-4410-80f4-af807cbc0420 a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 default - -] Loading container image driver 'docker' Apr 30 16:48:55 compute zun-compute[2072]: 2019-04-30 16:48:55.011 2072 ERROR zun.compute.manager [req-7bfa764a-45b8-4e2f-ac70-84d8bb71b135 - - - - -] Error occurred while calling Docker create API: Docker internal error: 500 Server Error: Internal Server Error ("failed to update store for object typpe *libnetwork.endpointCnt: client: etcd member http://controller:2379 has no leader").: DockerError: Docker internal error: 500 Server Error: Internal Server Error ("failed to update store for object type *libnetwork.endpointtCnt: client: etcd member http://controller:2379 has no leader"). Also *systemctl status docker* show the next output root@compute /h/team# systemctl status docker ● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/docker.service.d └─docker.conf, http-proxy.conf, https-proxy.conf Active: active (running) since Tue 2019-04-30 16:46:25 UTC; 4h 18min ago Docs: https://docs.docker.com Main PID: 1777 (dockerd) Tasks: 21 CGroup: /system.slice/docker.service └─1777 /usr/bin/dockerd --group zun -H tcp://compute:2375 -H unix:///var/run/docker.sock --cluster-store etcd://controller:2379 Apr 30 16:46:20 compute dockerd[1777]: time="2019-04-30T16:46:20.815305836Z" level=warning msg="Your kernel does not support cgroup rt runtime" Apr 30 16:46:20 compute dockerd[1777]: time="2019-04-30T16:46:20.815933695Z" level=info msg="Loading containers: start." Apr 30 16:46:24 compute dockerd[1777]: time="2019-04-30T16:46:24.378526837Z" level=info msg="Default bridge (docker0) is assigned with an IP address 17 Apr 30 16:46:24 compute dockerd[1777]: time="2019-04-30T16:46:24.572558877Z" level=info msg="Loading containers: done." Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.198101219Z" level=info msg="Docker daemon" commit=e8ff056 graphdriver(s)=overlay2 vers Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.198211373Z" level=info msg="Daemon has completed initialization" Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.232286069Z" level=info msg="API listen on /var/run/docker.sock" Apr 30 16:46:25 compute dockerd[1777]: time="2019-04-30T16:46:25.232318790Z" level=info msg="API listen on 10.8.9.58:2375" Apr 30 16:46:25 compute systemd[1]: Started Docker Application Container Engine. Apr 30 16:48:55 compute dockerd[1777]: time="2019-04-30T16:48:55.009820439Z" level=error msg="Handler for POST /v1.26/networks/create returned error: failed to update store for object type *libnetwork.endpointCnt: client: etcd member http://controller:2379 has no leader" When i try to launch an app container as the guide says it shows an Error state and when i run opentack appcontainer show this is the reason of the error status_reason | Docker internal error: 500 Server Error: Internal Server Error ("failed to update store for object type *libnetwork.endpointCnt: client: etcd member http://controller:2379 has no leader") I had some troubles installing Kuryr-libnetwork besides that i didn't had any othet problem during the installation of Zun

2 1