Open Stack

Hi All

Last Wednesday I try to setup a devstack environment with latest openstack master code  in one Ubuntu 14.04 LTS system, after devstack installation finished I have met RabbitMQ beam.smp take high CPU usage issue in the system, the RabbitMQ beam.smp process will take high CPU usage which will block the CLI command execution speed and most openstack related CLI tool will take a much longer execution time, do ironic deploy will also failed for long response time.
I also did unstack and try to reinstall the devstack environment with stable Mitaka code version later, but still met this situation, does anyone has met it before? Or someone could give advice for debugging?



Below is the top log and rabbitmqctl list_queues output:

stack at ironic-5:/etc/swift$ top
top - 07:30:19 up  1:26, 24 users,  load average: 45.83, 52.45, 42.57
Tasks: 2023 total,  79 running, 1944 sleeping,   0 stopped,   0 zombie
%Cpu(s):  9.7 us, 44.7 sy,  0.0 ni, 41.8 id,  1.2 wa,  2.6 hi,  0.0 si,  0.0 st
KiB Mem:  13201025+total, 10573100 used, 12143715+free,   117756 buffers
KiB Swap:  3905532 total,        0 used,  3905532 free.   645744 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  4345 rabbitmq  20   0 4205468 444428   2652 S 995.3  0.3 181:28.18 beam.smp
  9751 stack     20   0  234892  68884   6112 R  66.4  0.1   5:32.26 ironic-conducto
  9786 stack     20   0   23708   3316   1108 R  53.0  0.0   6:46.15 top
10778 stack     20   0    9980   1032    860 R  51.5  0.0   0:04.47 ps
  4004 mysql     20   0 10.048g 103752   7576 S  48.0  0.1   6:45.56 mysqld
  1263 root      20   0       0      0      0 R  43.9  0.0   5:27.91 kworker/11:1
  1269 root      20   0       0      0      0 S  43.0  0.0   5:21.67 kworker/17:1
  1254 root      20   0       0      0      0 S  42.3  0.0   5:26.01 kworker/14:1
  1228 root      20   0       0      0      0 R  41.7  0.0   5:20.41 kworker/4:1
  1230 root      20   0       0      0      0 R  41.0  0.0   5:22.68 kworker/6:1
  1234 root      20   0       0      0      0 R  40.7  0.0   5:23.76 kworker/2:1
  1255 root      20   0       0      0      0 R  40.1  0.0   5:14.81 kworker/19:1
  1343 root      20   0       0      0      0 S  40.0  0.0   4:33.72 kworker/78:1
  1276 root      20   0       0      0      0 S  39.9  0.0   5:26.94 kworker/10:1
  1229 root      20   0       0      0      0 R  39.4  0.0   5:17.28 kworker/5:1
  1231 root      20   0       0      0      0 R  39.3  0.0   5:25.96 kworker/7:1
  1245 root      20   0       0      0      0 R  39.3  0.0   5:20.79 kworker/18:1


stack at ironic-5:~$ sudo rabbitmqctl list_queues
Listing queues ...
compute 0
compute.ironic-5        0
conductor       462
conductor.ironic-5      0
conductor_fanout_02312222e335461984a0e821034ae981       0
conductor_fanout_05bf60e9f9d04cffbfaba180560f240f       0
conductor_fanout_0b0da163a76445fe9719b7c3bf961747       0
conductor_fanout_0cad93609166481ba3139b2d29efe852       0
conductor_fanout_0f2d693f3101463a88959d6f0f560725       0
conductor_fanout_118cc43bad3243ffb159dce273809c36       0
conductor_fanout_128d669046a44c76b581cbf0d833d4c8       0
conductor_fanout_166983b8d404432da5fe4d0763c1d01b       0
conductor_fanout_1715879fc61e4ce9ab3450839129fa53       0
conductor_fanout_18c1c0bef7fa4e7a940a823b4cf1e9a7       0
conductor_fanout_1ed22d9007694ff89d8536eca88ecc35       0
conductor_fanout_1fa0812d84524f438e15cc33ee858519       0
conductor_fanout_262d8a934b444b0fab7a250e044d02be       0
conductor_fanout_345e0e245efd4736add8a655e5b20cd2       0
conductor_fanout_347a1ab5e2684b48b1545f0dfed89c5b       0
conductor_fanout_34a8f23517e74960a6a6ad2d4405bfbd       0
conductor_fanout_34fb90b34a3947d19679574221123828       0
conductor_fanout_3b247b2b5c3c451f8233a70d63d54b2c       0
conductor_fanout_3cef115a999b42e683d88081211a4260       0
conductor_fanout_3d85908c76db497f889aa4e7c078000f       0
conductor_fanout_478c24cfc27046ab82bad83e260462e0       0
conductor_fanout_50016371c6e34364a6ded9a6ca5acc38       0
conductor_fanout_62f4dfadbfc74628a7c391c9d2c04f9a       0
conductor_fanout_633182c537b14d96b7795f228f9a2180       0
conductor_fanout_7930b023e08e47878aa0ad18e721303b       0
conductor_fanout_818b7e9364924ead85366c6c8c1239ec       0
conductor_fanout_8332f27e65684a39b7835b530211a264       0
conductor_fanout_940bdd09b48b45ad9f75141899b0de40       0
conductor_fanout_9542728a1b774b23850befadf37254c8       0
conductor_fanout_b05a8d5de5a643d7ad959a8018ba1b77       0
conductor_fanout_b17c7e4dbca84383bbbdfd2817479f14       0
conductor_fanout_b5e8462e0baa4a639c99c66715f15cdc       0
conductor_fanout_b8b03b6f432c4ad4a312934adefd160f       0
conductor_fanout_bd3d04fbc0bc49f3b653184ae4d74665       0
conductor_fanout_d84e2b0ca87c467d9f3e3181174ce57d       0
conductor_fanout_dc1168b70b1e4896bdcff8648bd41ceb       0
conductor_fanout_ea4bace966774f25b225cc021edd5b75       0
conductor_fanout_ed94f40b142a45f3ad4718fd7fda06af       0
conductor_fanout_f07d9fe0c2a94320882a6693e5fa7ee4       0
consoleauth     0
consoleauth.ironic-5    0
consoleauth_fanout_acf3fff89b6140d0bd02b7e3769fd6d8     0
dhcp_agent      0
dhcp_agent.ironic-5     0
dhcp_agent_fanout_51c06e11f03b4392a88d5f0e7dfff68d      0
ironic.conductor_manager        0
ironic.conductor_manager.localhost      0
ironic.conductor_manager_fanout_1b49c8d574d24ad0a5938e726bc0051d        0
l3_agent        0
l3_agent.ironic-5       0
l3_agent_fanout_5fd74b3a793c43e58ebe4d8fd8a0ed2b        0
q-agent-notifier-dvr-update     0
q-agent-notifier-dvr-update.ironic-5    0
q-agent-notifier-dvr-update_fanout_ed0690fa41b44f89b70cf8d79f9b6313     0
q-agent-notifier-network-update 0
q-agent-notifier-network-update.ironic-5        0
q-agent-notifier-network-update_fanout_49086ac9f078430296e44a4ae748e354 0
q-agent-notifier-port-delete    0
q-agent-notifier-port-delete.ironic-5   0
q-agent-notifier-port-delete_fanout_12581a41a29a45a3bfd2d3b0b54fda13    0
q-agent-notifier-port-update    0
q-agent-notifier-port-update.ironic-5   0
q-agent-notifier-port-update_fanout_a2f1501a7e974a4788fb2dba1532290d    0
q-agent-notifier-security_group-update  0
q-agent-notifier-security_group-update.ironic-5 0
q-agent-notifier-security_group-update_fanout_d58a85063cef46d0a976f8614dd75037  0
q-agent-notifier-tunnel-delete  0
q-agent-notifier-tunnel-delete.ironic-5 0
q-agent-notifier-tunnel-delete_fanout_00e7f6f2bac44ba488d41eb3e0b1ab96  0
q-agent-notifier-tunnel-update  0
q-agent-notifier-tunnel-update.ironic-5 0
q-agent-notifier-tunnel-update_fanout_ab4ed650e1694ea9961cb8d2c9b9f8c8  0
q-l3-plugin     0
q-l3-plugin.ironic-5    0
q-l3-plugin_fanout_7343f9d5b16d481b8ecd5a51b97c182f     0
q-l3-plugin_fanout_8c774f054d3e4b8e8d0015a12a288dce     0
q-plugin        44
q-plugin.ironic-5       0
q-plugin_fanout_abdfdd70f83e40148babce324d1ecee5        0
q-reports-plugin        1135
q-reports-plugin.ironic-5       0
q-reports-plugin_fanout_bd08dc48882449bb8f27b45edd82ba67        0
q-reports-plugin_fanout_bf3fe0d2b1444927ad9e3cf12fe3348e        0
q-server-resource-versions      0
q-server-resource-versions.ironic-5     0
q-server-resource-versions_fanout_b8cfbb46829b4adb89001b9b789957ca      207
reply_04d902be93534661900185e8cbc819e2  0
reply_39bdce7dec824bcba2ba89799431741a  0
reply_3fcd2cb83d364221878a48d6c4488db5  1159
reply_8fc1af6ec318403fa7acd79fd4234a65  68456
reply_cc3fc1f53e48400aa316e13a3e6ee6bd  617
scheduler       0
scheduler.ironic-5      0
scheduler_fanout_e00486b8682a4e6d8c10e55f8824f96a       0
...done.


Best regards,
-Li

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160426/738cd7fb/attachment.html>