[openstack-dev] [ovs-discuss] [neutron][networking-ovn] OpenStack Newton using OVN as mechanism driver is giving error.
Numan Siddique
nusiddiq at redhat.com
Thu Feb 2 12:15:37 UTC 2017
In the ovsdb logs I see
2017-02-02T11:07:12.366Z|00020|socket_util|ERR|6641:192.168.10.10: bind:
Address already in use
2017-02-02T11:07:14.869Z|00021|socket_util|ERR|6641:192.168.10.10: bind:
Address already in use
2017-02-02T11:07:16.182Z|00022|socket_util|ERR|6641:192.168.10.10: bind:
Address already in use
2017-02-02T11:07:16.183Z|00023|socket_util|ERR|6641:192.168.10.10: bind:
Address already in use
Probably you can use netstat and see which processes are listening on port
6641, kill those processes and restart ovn-northd service and see if it
works.
Thanks
Numan
On Thu, Feb 2, 2017 at 5:26 PM, pranab boruah <pranabjyotiboruah at gmail.com>
wrote:
> NB db is not listening on the port 6641:
> # ovn-nbctl --db=tcp:192.168.10.10:6641 show
> 2017-02-02T11:35:10Z|00001|ovsdb_idl|ERR|tcp:192.168.10.10:6641:
> requested schema not found
> ovn-nbctl: tcp:192.168.10.10:6641: database connection failed (No such
> file or directory)
>
> # ovn-sbctl --db=tcp:192.168.10.10:6642 show
> Chassis "e8968c35-11db-4b3b-ab24-c20866ac3dc8"
> hostname: "compute-447"
> Encap geneve
> ip: "172.16.10.11"
> options: {csum="true"}
> Encap vxlan
> ip: "172.16.10.11"
> options: {csum="true"}
> Chassis "9f20e7bd-5e16-4273-8ce8-96d0e0a12047"
> hostname: "compute-397"
> Encap geneve
> ip: "172.16.10.12"
> options: {csum="true"}
> Encap vxlan
> ip: "172.16.10.12"
> options: {csum="true"}
>
> # ovsdb-client dump unix:/var/run/openvswitch/ovnnb_db.sock Connection
> Connection table
> _uuid external_ids inactivity_probe
> is_connected max_backoff other_config status
> target
> ------------------------------------ ------------ ----------------
> ------------ ----------- ------------ ---------------
> -------------------------
> c110ffff-5ba2-4125-8a51-3c3db583f267 {} []
> false [] {} {state=BACKOFF} "ptcp:
> 192.168.10.10:6641"
>
> # ovsdb-client dump unix:/var/run/openvswitch/ovnsb_db.sock Connection
> Connection table
> _uuid external_ids inactivity_probe
> is_connected max_backoff other_config read_only status
> target
> ------------------------------------ ------------ ----------------
> ------------ ----------- ------------ --------- ---------------
> -------------------------
> 1beb9cba-cc01-4981-8a67-a8b265326df9 {} []
> false [] {} false {state=BACKOFF} "ptcp:
> 192.168.10.10:6642"
>
> No change after restarting Controller node and restating Northd as
> mentioned here:
> https://github.com/openvswitch/ovs/commit/84d0ca5d00fe01b29163236d48fa0f
> 9105687149
>
> Pranab
>
>
>
>
>
> On 2 February 2017 at 17:00, Numan Siddique <nusiddiq at redhat.com> wrote:
>
>>
>>
>> On Thu, Feb 2, 2017 at 4:44 PM, pranab boruah <
>> pranabjyotiboruah at gmail.com> wrote:
>>
>>> Numan,
>>>
>>> Thanks, but, the above commands didn't changed anything. Checked the
>>> logs and found this:
>>>
>>>
>>
>> Could you run the below two commands and see how it goes
>>
>> - "ovn-nbctl --db=tcp:192.168.10.10:6641 show" -> See if this command
>> is successful or not. If not then ovsdb-server is either not listening on
>> this port or it may be possible that you need to add iptables rule to allow
>> this port.
>>
>> -
>> "ovn-
>> s
>> bctl --db=tcp:192.168.10.10:664
>> 2
>> show"
>> -
>>
>>
>> On the controller node where you are running the ovsdb-server's run
>> - "sudo ovsdb-client dump unix:/var/run/openvswitch/ovnnb_db.sock
>> Connection"
>> - "sudo ovsdb-client dump unix:/var/run/openvswitch/ovnsb_db.sock
>> Connection"
>>
>> If the Connection table is empty, it means ovsdb-server is not listening
>> on the tcp ports.
>>
>> Thanks
>> Numan
>>
>>
>> LOG of NB database:
>>>
>>> 2017-02-02T11:07:12.366Z|00020|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:07:14.869Z|00021|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:07:16.182Z|00022|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:07:16.183Z|00023|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:07:16.184Z|00024|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:09:55.403Z|00025|ovsdb_jsonrpc_server|INFO|ptcp:6642:
>>> 192.168.10.10: remote deconfigured
>>> 2017-02-02T11:09:55.403Z|00026|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:09:55.403Z|00027|ovsdb_jsonrpc_server|ERR|Dropped 10 log
>>> messages in last 176 seconds (most recently, 159 seconds ago) due to
>>> excessive rate
>>> 2017-02-02T11:09:55.403Z|00028|ovsdb_jsonrpc_server|ERR|ptcp:6641:
>>> 192.168.10.10: listen failed: Address already in use
>>> 2017-02-02T11:09:55.403Z|00029|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:09:55.403Z|00030|ovsdb_jsonrpc_server|ERR|ptcp:6641:
>>> 192.168.10.10: listen failed: Address already in use
>>> 2017-02-02T11:09:57.460Z|00031|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:09:57.460Z|00032|ovsdb_jsonrpc_server|ERR|ptcp:6641:
>>> 192.168.10.10: listen failed: Address already in use
>>> 2017-02-02T11:09:57.460Z|00033|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:09:59.961Z|00034|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:10:02.464Z|00035|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:10:04.967Z|00036|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:10:07.469Z|00037|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:10:09.972Z|00038|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:10:12.475Z|00039|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:10:14.978Z|00040|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>> 2017-02-02T11:10:17.481Z|00041|socket_util|ERR|6641:192.168.10.10:
>>> bind: Address already in use
>>>
>>> -Pranab
>>>
>>>
>>> On 2 February 2017 at 16:36, Numan Siddique <nusiddiq at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Feb 2, 2017 at 4:00 PM, pranab boruah <
>>>> pranabjyotiboruah at gmail.com> wrote:
>>>>
>>>>> Numan,
>>>>> Details as requested:
>>>>>
>>>>> ___________
>>>>> ml2_conf.ini:|
>>>>> ```````````````````
>>>>> ovn_nb_connection = tcp:192.168.10.10:6641
>>>>> ovn_sb_connection = tcp:192.168.10.10:6642
>>>>> ovn_l3_mode = True
>>>>> ovn_l3_scheduler = chance
>>>>> ovn_l3_admin_net_cidr = 169.254.128.0/30
>>>>> ovn_native_dhcp = True
>>>>>
>>>>> Note: 192.168.10.10 is the management ip of controller. Firewall,
>>>>> NetworkManager and SElinux are disabled on all nodes.
>>>>> ____________________________
>>>>> output of "ps -aef | grep ovsdb" : |
>>>>> `````````````````````````````````````````````````
>>>>> root 21220 21219 0 Feb01 ? 00:00:44 ovsdb-server
>>>>> /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info
>>>>> --remote=punix:/var/run/openvswitch/db.sock
>>>>> --private-key=db:Open_vSwitch,SSL,private_key
>>>>> --certificate=db:Open_vSwitch,SSL,certificate
>>>>> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir
>>>>> --log-file=/var/log/openvswitch/ovsdb-server.log
>>>>> --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
>>>>>
>>>>> root 21324 21323 0 Feb01 ? 00:00:03 ovsdb-server --detach
>>>>> --monitor -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-nb.log
>>>>> --remote=punix:/var/run/openvswitch/ovnnb_db.sock
>>>>> --pidfile=/var/run/openvswitch/ovnnb_db.pid
>>>>> --remote=db:OVN_Northbound,NB_Global,connections
>>>>> --unixctl=ovnnb_db.ctl --private-key=db:OVN_Northbound,SSL,private_key
>>>>> --certificate=db:OVN_Northbound,SSL,certificate
>>>>> --ca-cert=db:OVN_Northbound,SSL,ca_cert /etc/openvswitch/ovnnb_db.db
>>>>>
>>>>> root 21332 21331 0 Feb01 ? 00:00:03 ovsdb-server --detach
>>>>> --monitor -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-sb.log
>>>>> --remote=punix:/var/run/openvswitch/ovnsb_db.sock
>>>>> --pidfile=/var/run/openvswitch/ovnsb_db.pid
>>>>> --remote=db:OVN_Southbound,SB_Global,connections
>>>>> --unixctl=ovnsb_db.ctl --private-key=db:OVN_Southbound,SSL,private_key
>>>>> --certificate=db:OVN_Southbound,SSL,certificate
>>>>> --ca-cert=db:OVN_Southbound,SSL,ca_cert /etc/openvswitch/ovnsb_db.db
>>>>>
>>>>> Note: NB and SB databases are supposed to be listening on a tcp
>>>>> socket. But it isn't.
>>>>>
>>>>>
>>>> Running the below commands should solve your problem
>>>> - ovn-nbctl set-connection ptcp:6641:192.168.10.10
>>>>
>>>> - ovn-
>>>> s
>>>> bctl set-connection ptcp:664
>>>> 2
>>>> :192.168.10.10
>>>>
>>>> L
>>>> et me know if you still face problems.
>>>>
>>>> The reason for this is this commit -
>>>> https://github.com/openvswitch/ovs/commit/84d0ca5d00fe01b291
>>>> 63236d48fa0f9105687149
>>>>
>>>> P
>>>> lease have a look at it.
>>>>
>>>> Thanks
>>>> Numan
>>>>
>>>>
>>>>
>>>>> _________________________
>>>>> output of logs |
>>>>> ```````````````````````````````````````````
>>>>> Neutron server log :
>>>>> - http://paste.openstack.org/show/597320/
>>>>> ovs-nothd log:
>>>>> - http://paste.openstack.org/show/597322/
>>>>> ovs-vswitchd log on Controller:
>>>>> - http://paste.openstack.org/show/597324/
>>>>> _________________________________________
>>>>> # ovn-nbctl and ovn-sbctl show no output. |
>>>>> ````````````````````````````````````````````````````````````
>>>>> ```````````
>>>>> ________________________________________
>>>>> OVN RPMS installation and download details: |
>>>>> ``````````````````````````````````````````````````````````````````````
>>>>> On controller:
>>>>> # git clone https://github.com/openvswitch/ovs.git
>>>>> # ./boot.sh
>>>>> # ./configure
>>>>> # make rpm-fedora RPMBUILD_OPT="--without check"
>>>>> # make rpm-fedora-kmod
>>>>>
>>>>> ALL nodes:
>>>>>
>>>>> # rpm -i openvswitch-kmod-2.6.90-1.el7.centos.x86_64.rpm# rpm -U openvswitch-2.6.90-1.el7.centos.x86_64.rpm
>>>>>
>>>>> Only Controller:
>>>>>
>>>>> # rpm -i openvswitch-ovn-common-*.x86_64.rpm# rpm -i openvswitch-ovn-central-*.x86_64.rpm
>>>>>
>>>>> Only Compute:
>>>>>
>>>>> # rpm -i openvswitch-ovn-common-*.x86_64.rpm# rpm -i openvswitch-ovn-host-*.x86_64.rpm
>>>>>
>>>>> # lsmod | grep open
>>>>> - shows all the required modules
>>>>>
>>>>> Let me know if you need anything else.
>>>>>
>>>>> Thanks.
>>>>> pjb
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 2 February 2017 at 15:23, Numan Siddique <nusiddiq at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Adding openstack-dev ML with [networking-ovn] tag.
>>>>>>
>>>>>> Please see comments inline
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 2, 2017 at 3:01 PM, pranab boruah <
>>>>>> pranabjyotiboruah at gmail.com> wrote:
>>>>>>
>>>>>>> Hi guys,
>>>>>>> We have a OpenStack multi-node lab. Everything was working, until I
>>>>>>> tried integrating with OVN.
>>>>>>> Followed the official guide for integration:
>>>>>>>
>>>>>>> http://docs.openstack.org/developer/networking-ovn/install.html
>>>>>>>
>>>>>>> Lab details:
>>>>>>> 1 Controller, 2 computes
>>>>>>>
>>>>>>> CentOS-7.3, OpenStack Newton, separate network for mgmt and tunnel.
>>>>>>>
>>>>>>> Error details :
>>>>>>> Neutron Server log shows :
>>>>>>>
>>>>>>> ERROR networking_ovn.ovsdb.impl_idl_ovn [-] OVS database
>>>>>>> connection to OVN_Northbound failed with error: '{u'error': u'unknown
>>>>>>> database', u'details': u'get_schema request specifies unknown database
>>>>>>> OVN_Northbound', u'syntax': u'["OVN_Northbound"]'}'. Verify that the OVS
>>>>>>> and OVN services are available and that the 'ovn_nb_connection' and
>>>>>>> 'ovn_sb_connection' configuration options are correct.
>>>>>>>
>>>>>>> Stuck in this for more than 2 week. Starting to hate my life. Please
>>>>>>> help. Before I tried installing in physical machines, followed the same
>>>>>>> guide and installed in VMs. Everything was working then.
>>>>>>>
>>>>>>> Solutions I tried:
>>>>>>> 1) Completely fresh installing everything.
>>>>>>> 2) Rebuilding ovs
>>>>>>> 3) Check SB and NB configuration options in plugin.ini are exactly
>>>>>>> correct.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Could you please share
>>>>>> - the values of ovn_nb_connection and ovn_sb_connection defined in
>>>>>> ml2_conf.ini.
>>>>>> - the output of "ps -aef | grep ovsdb" on the node where you are
>>>>>> running the ovn-northd server.
>>>>>> - If you could share the output of neutron-server logs in some paste
>>>>>> link, it would be helpful.
>>>>>>
>>>>>> Are you able to run "ovn-nbctl show" ? I presume you have installed
>>>>>> the appropriate ovn RPMS ? If not so, please let me know how you have
>>>>>> installed it.
>>>>>>
>>>>>> Thanks
>>>>>> Numan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Let me know if additional details are required.
>>>>>>>
>>>>>>>
>>>>>>> -pjb
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> discuss mailing list
>>>>>>> discuss at openvswitch.org
>>>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170202/3ca2ff5b/attachment.html>
More information about the OpenStack-dev
mailing list