[openstack-dev] [ovs-discuss] [neutron][networking-ovn] OpenStack Newton using OVN as mechanism driver is giving error.
pranab boruah
pranabjyotiboruah at gmail.com
Thu Feb 2 12:35:47 UTC 2017
Numan,
Sorry for making this thread too long.
Kind of deadlock scenario here:
The process listening on port 6641 is ovsdb-server(not NB db).
Details:
Start OVS:
# /usr/share/openvswitch/scripts/ovs-ctl start --system-id="random"
# netstat -putna | grep 6641
tcp 0 0 192.168.10.10:6641 0.0.0.0:*
LISTEN 5118/ovsdb-server
# ps -afx | grep ovsdb
5118 ? S< 0:00 \_ ovsdb-server /etc/openvswitch/conf.db
-vconsole:emer -vsyslog:err -vfile:info
--remote=punix:/var/run/openvswitch/db.sock
--private-key=db:Open_vSwitch,SSL,private_key
--certificate=db:Open_vSwitch,SSL,certificate
--bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir
--log-file=/var/log/openvswitch/ovsdb-server.log
--pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
Start NB db
# /usr/share/openvswitch/scripts/ovn-ctl restart_northd
--db-sb-create-insecure-remote=yes --db-nb-create-insecure-remote=yes
# netstat -putna | grep 6641
tcp 0 0 192.168.10.10:6641 0.0.0.0:*
LISTEN 5118/ovsdb-server
# ps -afx | grep ovsdb
5118 ? S< 0:00 \_ ovsdb-server /etc/openvswitch/conf.db
-vconsole:emer -vsyslog:err -vfile:info
--remote=punix:/var/run/openvswitch/db.sock
--private-key=db:Open_vSwitch,SSL,private_key
--certificate=db:Open_vSwitch,SSL,certificate
--bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir
--log-file=/var/log/openvswitch/ovsdb-server.log
--pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
5215 ? S 0:00 \_ ovsdb-server --detach --monitor
-vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-nb.log
--remote=punix:/var/run/openvswitch/ovnnb_db.sock
--pidfile=/var/run/openvswitch/ovnnb_db.pid
--remote=db:OVN_Northbound,NB_Global,connections --unixctl=ovnnb_db.ctl
--private-key=db:OVN_Northbound,SSL,private_key
--certificate=db:OVN_Northbound,SSL,certificate
--ca-cert=db:OVN_Northbound,SSL,ca_cert --remote=ptcp:6641:192.168.10.10
/etc/openvswitch/ovnnb_db.db
5223 ? S 0:00 \_ ovsdb-server --detach --monitor
-vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-sb.log
--remote=punix:/var/run/openvswitch/ovnsb_db.sock
--pidfile=/var/run/openvswitch/ovnsb_db.pid
--remote=db:OVN_Southbound,SB_Global,connections --unixctl=ovnsb_db.ctl
--private-key=db:OVN_Southbound,SSL,private_key
--certificate=db:OVN_Southbound,SSL,certificate
--ca-cert=db:OVN_Southbound,SSL,ca_cert --remote=ptcp:6642:192.168.10.10
/etc/openvswitch/ovnsb_db.db
Northbound DB logs show the same error.
Well I am not supposed to kill the ovsdb-server process. Am I ?
Also when I kill the ovsdb-server process, the error goes away. But my
doubt is we need the ovsdb-server process on the Controller node. Right?
Thanks for your patience.
Pranab
On 2 February 2017 at 17:45, Numan Siddique <nusiddiq at redhat.com> wrote:
> In the ovsdb logs I see
>
> 2017-02-02T11:07:12.366Z|00020|socket_util|ERR|6641:192.168.10.10: bind:
> Address already in use
> 2017-02-02T11:07:14.869Z|00021|socket_util|ERR|6641:192.168.10.10: bind:
> Address already in use
> 2017-02-02T11:07:16.182Z|00022|socket_util|ERR|6641:192.168.10.10: bind:
> Address already in use
> 2017-02-02T11:07:16.183Z|00023|socket_util|ERR|6641:192.168.10.10: bind:
> Address already in use
>
> Probably you can use netstat and see which processes are listening on port
> 6641, kill those processes and restart ovn-northd service and see if it
> works.
>
> Thanks
> Numan
>
>
> On Thu, Feb 2, 2017 at 5:26 PM, pranab boruah <pranabjyotiboruah at gmail.com
> > wrote:
>
>> NB db is not listening on the port 6641:
>> # ovn-nbctl --db=tcp:192.168.10.10:6641 show
>> 2017-02-02T11:35:10Z|00001|ovsdb_idl|ERR|tcp:192.168.10.10:6641:
>> requested schema not found
>> ovn-nbctl: tcp:192.168.10.10:6641: database connection failed (No such
>> file or directory)
>>
>> # ovn-sbctl --db=tcp:192.168.10.10:6642 show
>> Chassis "e8968c35-11db-4b3b-ab24-c20866ac3dc8"
>> hostname: "compute-447"
>> Encap geneve
>> ip: "172.16.10.11"
>> options: {csum="true"}
>> Encap vxlan
>> ip: "172.16.10.11"
>> options: {csum="true"}
>> Chassis "9f20e7bd-5e16-4273-8ce8-96d0e0a12047"
>> hostname: "compute-397"
>> Encap geneve
>> ip: "172.16.10.12"
>> options: {csum="true"}
>> Encap vxlan
>> ip: "172.16.10.12"
>> options: {csum="true"}
>>
>> # ovsdb-client dump unix:/var/run/openvswitch/ovnnb_db.sock Connection
>> Connection table
>> _uuid external_ids inactivity_probe
>> is_connected max_backoff other_config status
>> target
>> ------------------------------------ ------------ ----------------
>> ------------ ----------- ------------ ---------------
>> -------------------------
>> c110ffff-5ba2-4125-8a51-3c3db583f267 {} []
>> false [] {} {state=BACKOFF} "ptcp:
>> 192.168.10.10:6641"
>>
>> # ovsdb-client dump unix:/var/run/openvswitch/ovnsb_db.sock Connection
>> Connection table
>> _uuid external_ids inactivity_probe
>> is_connected max_backoff other_config read_only status
>> target
>> ------------------------------------ ------------ ----------------
>> ------------ ----------- ------------ --------- ---------------
>> -------------------------
>> 1beb9cba-cc01-4981-8a67-a8b265326df9 {} []
>> false [] {} false {state=BACKOFF} "ptcp:
>> 192.168.10.10:6642"
>>
>> No change after restarting Controller node and restating Northd as
>> mentioned here:
>> https://github.com/openvswitch/ovs/commit/84d0ca5d00fe01b291
>> 63236d48fa0f9105687149
>>
>> Pranab
>>
>>
>>
>>
>>
>> On 2 February 2017 at 17:00, Numan Siddique <nusiddiq at redhat.com> wrote:
>>
>>>
>>>
>>> On Thu, Feb 2, 2017 at 4:44 PM, pranab boruah <
>>> pranabjyotiboruah at gmail.com> wrote:
>>>
>>>> Numan,
>>>>
>>>> Thanks, but, the above commands didn't changed anything. Checked the
>>>> logs and found this:
>>>>
>>>>
>>>
>>> Could you run the below two commands and see how it goes
>>>
>>> - "ovn-nbctl --db=tcp:192.168.10.10:6641 show" -> See if this command
>>> is successful or not. If not then ovsdb-server is either not listening on
>>> this port or it may be possible that you need to add iptables rule to allow
>>> this port.
>>>
>>> -
>>> "ovn-
>>> s
>>> bctl --db=tcp:192.168.10.10:664
>>> 2
>>> show"
>>> -
>>>
>>>
>>> On the controller node where you are running the ovsdb-server's run
>>> - "sudo ovsdb-client dump unix:/var/run/openvswitch/ovnnb_db.sock
>>> Connection"
>>> - "sudo ovsdb-client dump unix:/var/run/openvswitch/ovnsb_db.sock
>>> Connection"
>>>
>>> If the Connection table is empty, it means ovsdb-server is not listening
>>> on the tcp ports.
>>>
>>> Thanks
>>> Numan
>>>
>>>
>>> LOG of NB database:
>>>>
>>>> 2017-02-02T11:07:12.366Z|00020|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:07:14.869Z|00021|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:07:16.182Z|00022|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:07:16.183Z|00023|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:07:16.184Z|00024|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:09:55.403Z|00025|ovsdb_jsonrpc_server|INFO|ptcp:6642:
>>>> 192.168.10.10: remote deconfigured
>>>> 2017-02-02T11:09:55.403Z|00026|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:09:55.403Z|00027|ovsdb_jsonrpc_server|ERR|Dropped 10 log
>>>> messages in last 176 seconds (most recently, 159 seconds ago) due to
>>>> excessive rate
>>>> 2017-02-02T11:09:55.403Z|00028|ovsdb_jsonrpc_server|ERR|ptcp:6641:
>>>> 192.168.10.10: listen failed: Address already in use
>>>> 2017-02-02T11:09:55.403Z|00029|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:09:55.403Z|00030|ovsdb_jsonrpc_server|ERR|ptcp:6641:
>>>> 192.168.10.10: listen failed: Address already in use
>>>> 2017-02-02T11:09:57.460Z|00031|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:09:57.460Z|00032|ovsdb_jsonrpc_server|ERR|ptcp:6641:
>>>> 192.168.10.10: listen failed: Address already in use
>>>> 2017-02-02T11:09:57.460Z|00033|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:09:59.961Z|00034|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:10:02.464Z|00035|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:10:04.967Z|00036|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:10:07.469Z|00037|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:10:09.972Z|00038|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:10:12.475Z|00039|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:10:14.978Z|00040|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>> 2017-02-02T11:10:17.481Z|00041|socket_util|ERR|6641:192.168.10.10:
>>>> bind: Address already in use
>>>>
>>>> -Pranab
>>>>
>>>>
>>>> On 2 February 2017 at 16:36, Numan Siddique <nusiddiq at redhat.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Feb 2, 2017 at 4:00 PM, pranab boruah <
>>>>> pranabjyotiboruah at gmail.com> wrote:
>>>>>
>>>>>> Numan,
>>>>>> Details as requested:
>>>>>>
>>>>>> ___________
>>>>>> ml2_conf.ini:|
>>>>>> ```````````````````
>>>>>> ovn_nb_connection = tcp:192.168.10.10:6641
>>>>>> ovn_sb_connection = tcp:192.168.10.10:6642
>>>>>> ovn_l3_mode = True
>>>>>> ovn_l3_scheduler = chance
>>>>>> ovn_l3_admin_net_cidr = 169.254.128.0/30
>>>>>> ovn_native_dhcp = True
>>>>>>
>>>>>> Note: 192.168.10.10 is the management ip of controller. Firewall,
>>>>>> NetworkManager and SElinux are disabled on all nodes.
>>>>>> ____________________________
>>>>>> output of "ps -aef | grep ovsdb" : |
>>>>>> `````````````````````````````````````````````````
>>>>>> root 21220 21219 0 Feb01 ? 00:00:44 ovsdb-server
>>>>>> /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info
>>>>>> --remote=punix:/var/run/openvswitch/db.sock
>>>>>> --private-key=db:Open_vSwitch,SSL,private_key
>>>>>> --certificate=db:Open_vSwitch,SSL,certificate
>>>>>> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir
>>>>>> --log-file=/var/log/openvswitch/ovsdb-server.log
>>>>>> --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
>>>>>>
>>>>>> root 21324 21323 0 Feb01 ? 00:00:03 ovsdb-server --detach
>>>>>> --monitor -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-nb.log
>>>>>> --remote=punix:/var/run/openvswitch/ovnnb_db.sock
>>>>>> --pidfile=/var/run/openvswitch/ovnnb_db.pid
>>>>>> --remote=db:OVN_Northbound,NB_Global,connections
>>>>>> --unixctl=ovnnb_db.ctl --private-key=db:OVN_Northbound,SSL,private_key
>>>>>> --certificate=db:OVN_Northbound,SSL,certificate
>>>>>> --ca-cert=db:OVN_Northbound,SSL,ca_cert /etc/openvswitch/ovnnb_db.db
>>>>>>
>>>>>> root 21332 21331 0 Feb01 ? 00:00:03 ovsdb-server --detach
>>>>>> --monitor -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-sb.log
>>>>>> --remote=punix:/var/run/openvswitch/ovnsb_db.sock
>>>>>> --pidfile=/var/run/openvswitch/ovnsb_db.pid
>>>>>> --remote=db:OVN_Southbound,SB_Global,connections
>>>>>> --unixctl=ovnsb_db.ctl --private-key=db:OVN_Southbound,SSL,private_key
>>>>>> --certificate=db:OVN_Southbound,SSL,certificate
>>>>>> --ca-cert=db:OVN_Southbound,SSL,ca_cert /etc/openvswitch/ovnsb_db.db
>>>>>>
>>>>>> Note: NB and SB databases are supposed to be listening on a tcp
>>>>>> socket. But it isn't.
>>>>>>
>>>>>>
>>>>> Running the below commands should solve your problem
>>>>> - ovn-nbctl set-connection ptcp:6641:192.168.10.10
>>>>>
>>>>> - ovn-
>>>>> s
>>>>> bctl set-connection ptcp:664
>>>>> 2
>>>>> :192.168.10.10
>>>>>
>>>>> L
>>>>> et me know if you still face problems.
>>>>>
>>>>> The reason for this is this commit -
>>>>> https://github.com/openvswitch/ovs/commit/84d0ca5d00fe01b291
>>>>> 63236d48fa0f9105687149
>>>>>
>>>>> P
>>>>> lease have a look at it.
>>>>>
>>>>> Thanks
>>>>> Numan
>>>>>
>>>>>
>>>>>
>>>>>> _________________________
>>>>>> output of logs |
>>>>>> ```````````````````````````````````````````
>>>>>> Neutron server log :
>>>>>> - http://paste.openstack.org/show/597320/
>>>>>> ovs-nothd log:
>>>>>> - http://paste.openstack.org/show/597322/
>>>>>> ovs-vswitchd log on Controller:
>>>>>> - http://paste.openstack.org/show/597324/
>>>>>> _________________________________________
>>>>>> # ovn-nbctl and ovn-sbctl show no output. |
>>>>>> ````````````````````````````````````````````````````````````
>>>>>> ```````````
>>>>>> ________________________________________
>>>>>> OVN RPMS installation and download details: |
>>>>>> ````````````````````````````````````````````````````````````
>>>>>> ``````````
>>>>>> On controller:
>>>>>> # git clone https://github.com/openvswitch/ovs.git
>>>>>> # ./boot.sh
>>>>>> # ./configure
>>>>>> # make rpm-fedora RPMBUILD_OPT="--without check"
>>>>>> # make rpm-fedora-kmod
>>>>>>
>>>>>> ALL nodes:
>>>>>>
>>>>>> # rpm -i openvswitch-kmod-2.6.90-1.el7.centos.x86_64.rpm# rpm -U openvswitch-2.6.90-1.el7.centos.x86_64.rpm
>>>>>>
>>>>>> Only Controller:
>>>>>>
>>>>>> # rpm -i openvswitch-ovn-common-*.x86_64.rpm# rpm -i openvswitch-ovn-central-*.x86_64.rpm
>>>>>>
>>>>>> Only Compute:
>>>>>>
>>>>>> # rpm -i openvswitch-ovn-common-*.x86_64.rpm# rpm -i openvswitch-ovn-host-*.x86_64.rpm
>>>>>>
>>>>>> # lsmod | grep open
>>>>>> - shows all the required modules
>>>>>>
>>>>>> Let me know if you need anything else.
>>>>>>
>>>>>> Thanks.
>>>>>> pjb
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2 February 2017 at 15:23, Numan Siddique <nusiddiq at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Adding openstack-dev ML with [networking-ovn] tag.
>>>>>>>
>>>>>>> Please see comments inline
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 2, 2017 at 3:01 PM, pranab boruah <
>>>>>>> pranabjyotiboruah at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi guys,
>>>>>>>> We have a OpenStack multi-node lab. Everything was working, until I
>>>>>>>> tried integrating with OVN.
>>>>>>>> Followed the official guide for integration:
>>>>>>>>
>>>>>>>> http://docs.openstack.org/developer/networking-ovn/install.html
>>>>>>>>
>>>>>>>> Lab details:
>>>>>>>> 1 Controller, 2 computes
>>>>>>>>
>>>>>>>> CentOS-7.3, OpenStack Newton, separate network for mgmt and tunnel.
>>>>>>>>
>>>>>>>> Error details :
>>>>>>>> Neutron Server log shows :
>>>>>>>>
>>>>>>>> ERROR networking_ovn.ovsdb.impl_idl_ovn [-] OVS database
>>>>>>>> connection to OVN_Northbound failed with error: '{u'error': u'unknown
>>>>>>>> database', u'details': u'get_schema request specifies unknown database
>>>>>>>> OVN_Northbound', u'syntax': u'["OVN_Northbound"]'}'. Verify that the OVS
>>>>>>>> and OVN services are available and that the 'ovn_nb_connection' and
>>>>>>>> 'ovn_sb_connection' configuration options are correct.
>>>>>>>>
>>>>>>>> Stuck in this for more than 2 week. Starting to hate my life.
>>>>>>>> Please help. Before I tried installing in physical machines, followed the
>>>>>>>> same guide and installed in VMs. Everything was working then.
>>>>>>>>
>>>>>>>> Solutions I tried:
>>>>>>>> 1) Completely fresh installing everything.
>>>>>>>> 2) Rebuilding ovs
>>>>>>>> 3) Check SB and NB configuration options in plugin.ini are exactly
>>>>>>>> correct.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Could you please share
>>>>>>> - the values of ovn_nb_connection and ovn_sb_connection defined in
>>>>>>> ml2_conf.ini.
>>>>>>> - the output of "ps -aef | grep ovsdb" on the node where you are
>>>>>>> running the ovn-northd server.
>>>>>>> - If you could share the output of neutron-server logs in some
>>>>>>> paste link, it would be helpful.
>>>>>>>
>>>>>>> Are you able to run "ovn-nbctl show" ? I presume you have installed
>>>>>>> the appropriate ovn RPMS ? If not so, please let me know how you have
>>>>>>> installed it.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Numan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Let me know if additional details are required.
>>>>>>>>
>>>>>>>>
>>>>>>>> -pjb
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> discuss mailing list
>>>>>>>> discuss at openvswitch.org
>>>>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170202/5068b43e/attachment.html>
More information about the OpenStack-dev
mailing list