Open Stack

Hi Sean,

Thanks for replying back, response inline.

On Mon, Nov 9, 2015 at 8:24 PM, Mooney, Sean K <sean.k.mooney at intel.com>
wrote:
> Hi
> Can you provide some more information regarding your deployment?
>
> Can you check which kernel you are using.
>
> uname -a

Linux ubuntu 3.16.0-50-generic #67~14.04.1-Ubuntu SMP Fri Oct 2 22:07:51
UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

>
> If you are using a 3.19 kernel changes to some locking code in the kennel
broke synchronization dpdk2.0 and requires dpdk 2.1 to be used instead.
> In general it is not advisable to use a 3.19 kernel with dpdk as it can
lead to non-deterministic behavior.
>
> When devstack hangs can you connect with a second ssh session and run
>         sudo service ovs-dpdk status
> and
>         ps aux | grep ovs
>
sudo service ovs-dpdk status
sourcing config
/opt/stack/logs/ovs-vswitchd.pid is not running
Not all processes are running restart!!!
1
ubuntu at ubuntu:~/samta/devstack$ ps -ef | grep ovs
root     13385     1  0 15:17 ?        00:00:00 /usr/sbin/ovsdb-server
--detach --pidfile=/opt/stack/logs/ovsdb-server.pid
--remote=punix:/usr/local/var/run/openvswitch/db.sock
--remote=db:Open_vSwitch,Open_vSwitch,manager_options
ubuntu   24451 12855  0 15:45 pts/0    00:00:00 grep --color=auto ovs

>
> When the deployment hangs at sudo ovs-vsctl br-set-external-id br-ex
bridge-id br-ex
> It usually means that the ovs-vswitchd process has exited.
>
The above result shows that ovs-vswitchd is not running.
> This can happen for a number of reasons.
> The vswitchd process may exit if it  failed to allocate memory (due to
memory fragmentation or lack of free hugepages)
> if the ovs-vswitchd.log is not available can you check the the hugepage
mount point was created in
> /mnt/huge And that Iis mounted
> Run
>         ls -al /mnt/huge
> and
>         mount
>
ls -al /mnt/huge
total 4
drwxr-xr-x 2 libvirt-qemu kvm     0 Nov 11 15:18 .
drwxr-xr-x 3 root         root 4096 May 15 00:09 ..

ubuntu at ubuntu:~/samta/devstack$ mount
/dev/mapper/ubuntu--vg-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs
(rw,noexec,nosuid,nodev,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_prio type cgroup (rw,relatime,net_prio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb)
/dev/sda1 on /boot type ext2 (rw)
systemd on /sys/fs/cgroup/systemd type cgroup
(rw,noexec,nosuid,nodev,none,name=systemd)
hugetlbfs-kvm on /run/hugepages/kvm type hugetlbfs (rw,mode=775,gid=106)
nodev on /mnt/huge type hugetlbfs (rw,uid=106,gid=106)
nodev on /mnt/huge type hugetlbfs (rw,uid=106,gid=106)

> then checkout how many hugepages are mounted
>
>         cat /proc/meminfo | grep huge
>

cat /proc/meminfo | grep Huge
AnonHugePages:    292864 kB
HugePages_Total:       5
HugePages_Free:        5
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

>
> the vswitchd process may also exit if it  failed to initializes dpdk
interfaces.
> This can happen if no interface is  compatible with the igb-uio or
vfio-pci drivers
> (note in the vfio-pci case all interface in the same iommu group must be
bound to the vfio-pci driver and
> The iommu must be enabled in the kernel command line with VT-d enabled in
the bios)
>
> Can you  check which interface are bound to the dpdk driver by running
the following command
>
>         /opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py --status
>
/opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=p1p1 drv=ixgbe
unused=igb_uio
0000:02:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=p4p1 drv=i40e
unused=igb_uio
0000:03:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=p2p1 drv=i40e
unused=igb_uio
0000:06:00.0 'I350 Gigabit Network Connection' if=em1 drv=igb
unused=igb_uio *Active*
0000:06:00.1 'I350 Gigabit Network Connection' if=em2 drv=igb
unused=igb_uio

Other network devices
=====================
0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' unused=igb_uio

>
> Finally can you confim that ovs-dpdk compiled successfully by either
check the xstack.log or
> Checking for the BUILD_COMPLETE file in /opt/stack/ovs
BUILD_COMPLETE exist in /opt/stack/ovs though its empty.
>
> Regards
> sean
>
>
>
>
> -----Original Message-----
> From: Samta Rangare [mailto:samtarangare at gmail.com]
> Sent: Monday, November 9, 2015 2:31 PM
> To: Czesnowicz, Przemyslaw
> Cc: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [networking-ovs-dpdk]
>
> Thanks for replying Przemyslaw, there is no ovs-vswitchd.log in
/opt/stack/logs/. This is all contains inside (ovsdb-server.pid, screen).
>
> When I cancel stack .sh (ctr c), and try to rerun this $sudo ovs-vsctl
br-set-external-id br-ex bridge-id br-ex it didnt hang, that means vSwitch
was running isn't it ?
>
> But rerunning stack.sh after unstack hangs again.
>
> Thanks,
> Samta
>
> On Mon, Nov 9, 2015 at 7:50 PM, Czesnowicz, Przemyslaw <
przemyslaw.czesnowicz at intel.com> wrote:
>> Hi Samta,
>>
>> This usually means that the vSwitch is not running/has crashed.
>> Can you check in /opt/stack/logs/ovs-vswitchd.log ? There should be an
error msg there.
>>
>> Regards
>> Przemek
>>
>>> -----Original Message-----
>>> From: Samta Rangare [mailto:samtarangare at gmail.com]
>>> Sent: Monday, November 9, 2015 1:51 PM
>>> To: OpenStack Development Mailing List (not for usage questions)
>>> Subject: [openstack-dev] [networking-ovs-dpdk]
>>>
>>> Hello Everyone,
>>>
>>> I am installing devstack with networking-ovs-dpdk. The local.conf
>>> exactly looks like the one is available in /opt/stack/networking-ovs-
>>> dpdk/doc/source/_downloads/local.conf.single_node.
>>> So I believe all the necessary configuration will be taken care.
>>>
>>> However I am stuck at place where devstack is trying to set
>>> external-id ($ sudo ovs-vsctl br-set-external-id br-ex bridge-id
>>> br-ex). As soon as it hits at this place it's just hangs forever. I
>>> tried commenting this line from
>>> lib/neutron_plugin/ml2 (I know this is wrong) and then all services
>>> came up except ovs-dpdk agent and ovs agent.
>>>
>>> BTW I am deploying it in ubuntu 14.04. Any pointer will be really
helpful.
>>>
>>> Thanks,
>>> Samta
>>>
>>> __________________________________________________________
>>> ________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-
>>> request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151111/81e60f8e/attachment.html>