[neutron][OVN] Instance not getting metadata
Hi, I have set up OpenStack with OVN enabled (manual install) and I can create Instance, associate floating IP and test the ping. But my Instance not getting metadata from OpenStack. I check the reference architecture, https://docs.openstack.org/networking-ovn/queens/admin/refarch/refarch.html the compute nodes should have installed ovn-metadata-agent but I don't find any configuration document about ovn-metadata-agent. I have configured the configuration files like this: [DEFAULT] nova_metadata_ip = 10.100.100.10 [ovs] ovsdb_connection = unix:/var/run/openvswitch/db.sock [ovn] ovn_sb_connection = tcp:10.101.101.10:6642 But the agent always fails. Full logs: http://paste.openstack.org/show/757805/ Anyone know whats happen and how to fix this error? Best Regards, Zufar Dhiyaulhaq
Hi Zufar, The ovn-metadata-agent is trying to connection with the local OVSDB instance via UNIX socket. The same socket used by the "ovn-controller" process running on your compute nodes. For example: $ ps ax | grep ovn-controller 1640 ? S<sl 0:04 ovn-controller unix:/usr/local/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/opt/stack/logs/ovn-controller.log --pidfile=/usr/local/var/run/openvswitch/ovn-controller.pid --detach Can you see the "unix:/usr/local/var/run/openvswitch/db.sock" ? That's the UNIX socket path that needs to be passed to the ovn-metadata-agent via configuration option. You can find that configuration option at: $ grep ovsdb_connection /etc/neutron/networking_ovn_metadata_agent.ini ovsdb_connection = unix:/usr/local/var/run/openvswitch/db.sock Hope that helps, Lucas On Fri, Aug 16, 2019 at 5:53 AM Zufar Dhiyaulhaq <zufardhiyaulhaq@gmail.com> wrote:
Hi,
I have set up OpenStack with OVN enabled (manual install) and I can create Instance, associate floating IP and test the ping.
But my Instance not getting metadata from OpenStack. I check the reference architecture, https://docs.openstack.org/networking-ovn/queens/admin/refarch/refarch.html the compute nodes should have installed ovn-metadata-agent but I don't find any configuration document about ovn-metadata-agent.
I have configured the configuration files like this:
[DEFAULT] nova_metadata_ip = 10.100.100.10 [ovs] ovsdb_connection = unix:/var/run/openvswitch/db.sock [ovn] ovn_sb_connection = tcp:10.101.101.10:6642
But the agent always fails. Full logs: http://paste.openstack.org/show/757805/
Anyone know whats happen and how to fix this error?
Best Regards, Zufar Dhiyaulhaq
Hi Lucas, Its always same, my OVSDB local instance is listening in unix:/var/run/openvswitch/db.sock [root@zu-ovn-compute0 ~]# ps ax | grep ovn-controller 15398 ? S<s 0:00 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/openvswitch/ovn-controller.log --pidfile=/var/run/openvswitch/ovn-controller.pid --detach 15807 pts/0 S+ 0:00 grep --color=auto ovn-controller [root@zu-ovn-compute0 ~]# grep ovsdb_connection /etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini ovsdb_connection = unix:/var/run/openvswitch/db.sock [root@zu-ovn-compute0 ~]# tail -f /var/log/neutron/networking-ovn-metadata-agent.log 2019-08-16 10:29:08.391 15647 CRITICAL neutron [-] Unhandled error: Exception: Could not connect to unix:/var/run/openvswitch/db.sock 2019-08-16 10:29:08.391 15647 ERROR neutron Traceback (most recent call last): 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/bin/networking-ovn-metadata-agent", line 10, in <module> 2019-08-16 10:29:08.391 15647 ERROR neutron sys.exit(main()) 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/cmd/eventlet/agents/metadata.py", line 17, in main 2019-08-16 10:29:08.391 15647 ERROR neutron metadata_agent.main() 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata_agent.py", line 38, in main 2019-08-16 10:29:08.391 15647 ERROR neutron agt.start() 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 139, in start 2019-08-16 10:29:08.391 15647 ERROR neutron self.ovs_idl = ovsdb.MetadataAgentOvsIdl().start() 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/ovsdb.py", line 59, in start 2019-08-16 10:29:08.391 15647 ERROR neutron 'Open_vSwitch') 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 120, in get_schema_helper 2019-08-16 10:29:08.391 15647 ERROR neutron raise Exception("Could not connect to %s" % connection) 2019-08-16 10:29:08.391 15647 ERROR neutron Exception: Could not connect to unix:/var/run/openvswitch/db.sock 2019-08-16 10:29:08.391 15647 ERROR neutron [root@zu-ovn-compute0 ~]# systemctl status openvswitch ● openvswitch.service - Open vSwitch Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; enabled; vendor preset: disabled) Active: active (exited) since Jum 2019-08-16 10:27:01 EDT; 4min 22s ago Main PID: 15323 (code=exited, status=0/SUCCESS) Tasks: 0 CGroup: /system.slice/openvswitch.service Agu 16 10:27:01 zu-ovn-compute0 systemd[1]: Starting Open vSwitch... Agu 16 10:27:01 zu-ovn-compute0 systemd[1]: Started Open vSwitch. [root@zu-ovn-compute0 ~]# systemctl status ovn-controller ● ovn-controller.service - OVN controller daemon Loaded: loaded (/usr/lib/systemd/system/ovn-controller.service; enabled; vendor preset: disabled) Active: active (running) since Jum 2019-08-16 10:27:12 EDT; 4min 22s ago Main PID: 15398 (ovn-controller) Tasks: 1 CGroup: /system.slice/ovn-controller.service └─15398 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/openvswitch/ovn-controller... Agu 16 10:27:12 zu-ovn-compute0 systemd[1]: Starting OVN controller daemon... Agu 16 10:27:12 zu-ovn-compute0 ovn-ctl[15386]: Starting ovn-controller [ OK ] Agu 16 10:27:12 zu-ovn-compute0 systemd[1]: Started OVN controller daemon. Best Regards, Zufar Dhiyaulhaq On Fri, Aug 16, 2019 at 9:21 PM Lucas Alvares Gomes <lucasagomes@gmail.com> wrote:
Hi Zufar,
The ovn-metadata-agent is trying to connection with the local OVSDB instance via UNIX socket. The same socket used by the "ovn-controller" process running on your compute nodes. For example:
$ ps ax | grep ovn-controller 1640 ? S<sl 0:04 ovn-controller unix:/usr/local/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/opt/stack/logs/ovn-controller.log --pidfile=/usr/local/var/run/openvswitch/ovn-controller.pid --detach
Can you see the "unix:/usr/local/var/run/openvswitch/db.sock" ? That's the UNIX socket path that needs to be passed to the ovn-metadata-agent via configuration option.
You can find that configuration option at:
$ grep ovsdb_connection /etc/neutron/networking_ovn_metadata_agent.ini ovsdb_connection = unix:/usr/local/var/run/openvswitch/db.sock
Hope that helps, Lucas
On Fri, Aug 16, 2019 at 5:53 AM Zufar Dhiyaulhaq <zufardhiyaulhaq@gmail.com> wrote:
Hi,
I have set up OpenStack with OVN enabled (manual install) and I can
create Instance, associate floating IP and test the ping.
But my Instance not getting metadata from OpenStack. I check the
reference architecture, https://docs.openstack.org/networking-ovn/queens/admin/refarch/refarch.html the compute nodes should have installed ovn-metadata-agent but I don't find any configuration document about ovn-metadata-agent.
I have configured the configuration files like this:
[DEFAULT] nova_metadata_ip = 10.100.100.10 [ovs] ovsdb_connection = unix:/var/run/openvswitch/db.sock [ovn] ovn_sb_connection = tcp:10.101.101.10:6642
But the agent always fails. Full logs:
http://paste.openstack.org/show/757805/
Anyone know whats happen and how to fix this error?
Best Regards, Zufar Dhiyaulhaq
Hi Lucas, My networking-ovn-metadata-agent.service is running with user neutron [root@zu-ovn-compute1 ~]# cat /usr/lib/systemd/system/networking-ovn-metadata-agent.service [Unit] Description=OpenStack networking-ovn Metadata Agent After=syslog.target network.target openvswitch.service Requires=openvswitch.service [Service] Type=simple User=neutron But the sock is listen in openvswitch user. [root@zu-ovn-compute1 ~]# cd /var/run/openvswitch/ [root@zu-ovn-compute1 openvswitch]# ls -lh total 12K srwxr-x---. 1 openvswitch hugetlbfs 0 Agu 16 10:28 br-int.mgmt srwxr-x---. 1 openvswitch hugetlbfs 0 Agu 16 10:28 br-int.snoop srwxr-x---. 1 openvswitch hugetlbfs 0 Agu 16 10:28 br-provider.mgmt srwxr-x---. 1 openvswitch hugetlbfs 0 Agu 16 10:28 br-provider.snoop srwxr-x---. 1 openvswitch hugetlbfs 0 Agu 16 10:27 db.sock I have try to change the user in networking-ovn-metadata-agent.service from neutron to root and the service is running. [root@zu-ovn-compute0 openvswitch]# cat /usr/lib/systemd/system/networking-ovn-metadata-agent.service [Unit] Description=OpenStack networking-ovn Metadata Agent After=syslog.target network.target openvswitch.service Requires=openvswitch.service [Service] Type=simple User=root PermissionsStartOnly=true ExecStart=/usr/bin/networking-ovn-metadata-agent --config-file /etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini --config-dir /etc/neutron/conf.d/networking-ovn-metadata-agent --log-file /var/log/neutron/networking-ovn-metadata-agent.log PrivateTmp=false KillMode=process Restart=on-failure [Install] WantedBy=multi-user.target [root@zu-ovn-compute0 openvswitch]# systemctl status networking-ovn-metadata-agent.service ● networking-ovn-metadata-agent.service - OpenStack networking-ovn Metadata Agent Loaded: loaded (/usr/lib/systemd/system/networking-ovn-metadata-agent.service; enabled; vendor preset: disabled) Active: active (running) since Jum 2019-08-16 13:13:24 EDT; 4min 5s ago Main PID: 18692 (networking-ovn-) Tasks: 3 CGroup: /system.slice/networking-ovn-metadata-agent.service ├─18692 /usr/bin/python2 /usr/bin/networking-ovn-metadata-agent --config-file /etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini --conf... ├─18710 /usr/bin/python2 /usr/bin/networking-ovn-metadata-agent --config-file /etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini --conf... └─18711 /usr/bin/python2 /usr/bin/networking-ovn-metadata-agent --config-file /etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini --conf... Agu 16 13:13:24 zu-ovn-compute0 systemd[1]: Started OpenStack networking-ovn Metadata Agent. Agu 16 13:13:26 zu-ovn-compute0 sudo[18712]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/privsep-helper --config-file /etc/neutron/plugins/networking-o... Hint: Some lines were ellipsized, use -l to show in full. Are this is a bug in centos upstream? Best Regards, Zufar Dhiyaulhaq On Fri, Aug 16, 2019 at 9:31 PM Zufar Dhiyaulhaq <zufardhiyaulhaq@gmail.com> wrote:
Hi Lucas,
Its always same, my OVSDB local instance is listening in unix:/var/run/openvswitch/db.sock
[root@zu-ovn-compute0 ~]# ps ax | grep ovn-controller 15398 ? S<s 0:00 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/openvswitch/ovn-controller.log --pidfile=/var/run/openvswitch/ovn-controller.pid --detach 15807 pts/0 S+ 0:00 grep --color=auto ovn-controller
[root@zu-ovn-compute0 ~]# grep ovsdb_connection /etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini ovsdb_connection = unix:/var/run/openvswitch/db.sock
[root@zu-ovn-compute0 ~]# tail -f /var/log/neutron/networking-ovn-metadata-agent.log 2019-08-16 10:29:08.391 15647 CRITICAL neutron [-] Unhandled error: Exception: Could not connect to unix:/var/run/openvswitch/db.sock 2019-08-16 10:29:08.391 15647 ERROR neutron Traceback (most recent call last): 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/bin/networking-ovn-metadata-agent", line 10, in <module> 2019-08-16 10:29:08.391 15647 ERROR neutron sys.exit(main()) 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/cmd/eventlet/agents/metadata.py", line 17, in main 2019-08-16 10:29:08.391 15647 ERROR neutron metadata_agent.main() 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata_agent.py", line 38, in main 2019-08-16 10:29:08.391 15647 ERROR neutron agt.start() 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 139, in start 2019-08-16 10:29:08.391 15647 ERROR neutron self.ovs_idl = ovsdb.MetadataAgentOvsIdl().start() 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/ovsdb.py", line 59, in start 2019-08-16 10:29:08.391 15647 ERROR neutron 'Open_vSwitch') 2019-08-16 10:29:08.391 15647 ERROR neutron File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 120, in get_schema_helper 2019-08-16 10:29:08.391 15647 ERROR neutron raise Exception("Could not connect to %s" % connection) 2019-08-16 10:29:08.391 15647 ERROR neutron Exception: Could not connect to unix:/var/run/openvswitch/db.sock 2019-08-16 10:29:08.391 15647 ERROR neutron
[root@zu-ovn-compute0 ~]# systemctl status openvswitch ● openvswitch.service - Open vSwitch Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; enabled; vendor preset: disabled) Active: active (exited) since Jum 2019-08-16 10:27:01 EDT; 4min 22s ago Main PID: 15323 (code=exited, status=0/SUCCESS) Tasks: 0 CGroup: /system.slice/openvswitch.service
Agu 16 10:27:01 zu-ovn-compute0 systemd[1]: Starting Open vSwitch... Agu 16 10:27:01 zu-ovn-compute0 systemd[1]: Started Open vSwitch.
[root@zu-ovn-compute0 ~]# systemctl status ovn-controller ● ovn-controller.service - OVN controller daemon Loaded: loaded (/usr/lib/systemd/system/ovn-controller.service; enabled; vendor preset: disabled) Active: active (running) since Jum 2019-08-16 10:27:12 EDT; 4min 22s ago Main PID: 15398 (ovn-controller) Tasks: 1 CGroup: /system.slice/ovn-controller.service └─15398 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/openvswitch/ovn-controller...
Agu 16 10:27:12 zu-ovn-compute0 systemd[1]: Starting OVN controller daemon... Agu 16 10:27:12 zu-ovn-compute0 ovn-ctl[15386]: Starting ovn-controller [ OK ] Agu 16 10:27:12 zu-ovn-compute0 systemd[1]: Started OVN controller daemon.
Best Regards, Zufar Dhiyaulhaq
On Fri, Aug 16, 2019 at 9:21 PM Lucas Alvares Gomes <lucasagomes@gmail.com> wrote:
Hi Zufar,
The ovn-metadata-agent is trying to connection with the local OVSDB instance via UNIX socket. The same socket used by the "ovn-controller" process running on your compute nodes. For example:
$ ps ax | grep ovn-controller 1640 ? S<sl 0:04 ovn-controller unix:/usr/local/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/opt/stack/logs/ovn-controller.log --pidfile=/usr/local/var/run/openvswitch/ovn-controller.pid --detach
Can you see the "unix:/usr/local/var/run/openvswitch/db.sock" ? That's the UNIX socket path that needs to be passed to the ovn-metadata-agent via configuration option.
You can find that configuration option at:
$ grep ovsdb_connection /etc/neutron/networking_ovn_metadata_agent.ini ovsdb_connection = unix:/usr/local/var/run/openvswitch/db.sock
Hope that helps, Lucas
On Fri, Aug 16, 2019 at 5:53 AM Zufar Dhiyaulhaq <zufardhiyaulhaq@gmail.com> wrote:
Hi,
I have set up OpenStack with OVN enabled (manual install) and I can
create Instance, associate floating IP and test the ping.
But my Instance not getting metadata from OpenStack. I check the
reference architecture, https://docs.openstack.org/networking-ovn/queens/admin/refarch/refarch.html the compute nodes should have installed ovn-metadata-agent but I don't find any configuration document about ovn-metadata-agent.
I have configured the configuration files like this:
[DEFAULT] nova_metadata_ip = 10.100.100.10 [ovs] ovsdb_connection = unix:/var/run/openvswitch/db.sock [ovn] ovn_sb_connection = tcp:10.101.101.10:6642
But the agent always fails. Full logs:
http://paste.openstack.org/show/757805/
Anyone know whats happen and how to fix this error?
Best Regards, Zufar Dhiyaulhaq
participants (2)
-
Lucas Alvares Gomes
-
Zufar Dhiyaulhaq