[Openstack] Controller deployment fails - any advice é help appreciated
Mario Romascanu
romar at mindtrek.dnsalias.net
Fri Feb 24 02:33:28 UTC 2017
Hi
I was able to get to a certain point in deploying an environment on a
fresh installation of Mirantis OpenStack 9.0 but I am stuck with this
bizarre error in the controller deployment
It does not seem to be a communication issue - but rather a logical one
- an error 500 on a particular REST call URI to neutron and 200 OK
otherwise on different neutron URI.
This occurs after deploying Ubuntu and Openstack on the Controller node
*I am at a loss of what the issue is here - any suggestion welcome*
"Deployment has failed. All nodes are finished. Failed tasks:
Task[connectivity_tests/4] Stopping the deployment process!"
"Failed to deploy node 'MOS_Controller (1e:c1)': Task connectivity_tests
failed on node 4"
[root at fuel ~]# fuel node
id | status | name | cluster | ip |
mac | roles | pending_roles | online | group_id
---+---------+------------------------+---------+-----------+-------------------+------------+---------------+--------+---------
3 | stopped | MOS_Compute (46:4a) | 1 | 10.20.0.4 |
00:50:56:98:46:4a | compute | | 1 | 1
4 | error | MOS_Controller (1e:c1) | 1 | 10.20.0.3 |
00:50:56:98:1e:c1 | controller | | 1 | 1
An in the logs I see the traceback partially included - the root cause
is highlighted
Date Level Message
2017-02-23 20:56:14 ERROR [7fad42058880] (base) Unexpected
exception occured
Traceback (most recent call last):
File
"/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line
297, in handle_errors
return func(cls, *args, **kwargs)
File "<string>", line 2, in GET
File
"/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line
355, in validate
return func(cls, *args, **kwargs)
File "<string>", line 2, in GET
File
"/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line
381, in serialize
resp = func(cls, *args, **kwargs)
File
"/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/network_configuration.py",
line 92, in GET
return self.serializer.serialize_for_cluster(cluster)
File
"/usr/lib/python2.7/site-packages/nailgun/objects/serializers/network_configuration.py",
line 112, in serialize_for_cluster
result = cls.serialize_net_groups_and_vips(cluster, allocate_vips)
File
"/usr/lib/python2.7/site-packages/nailgun/objects/serializers/network_configuration.py",
line 42, in serialize_net_groups_and_vips
nets = cluster.network_groups + [
File
"/usr/lib/python2.7/site-packages/nailgun/db/sqlalchemy/models/cluster.py",
line 159, in network_groups
net_list.extend(ng.networks)
File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line
237, in __get__
return self.impl.get(instance_state(instance), dict_)
File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line
583, in get
value = self.callable_(state, passive)
File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/strategies.py", line
532, in _load_for_state
return self._emit_lazyload(session, state, ident_key, passive)
File "<string>", line 1, in <lambda>
File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/strategies.py", line
602, in _emit_lazyload
result = q.all()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
line 2588, in all
return list(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 86, in instances
util.raise_from_cause(err)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py",
line 200, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 71, in instances
rows = [proc(row) for row in fetch]
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 428, in _instance
loaded_instance, populate_existing, populators)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 486, in _populate_full
dict_[key] = getter(row)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/type_api.py",
line 1030, in process
return process_value(value, dialect)
File
"/usr/lib/python2.7/site-packages/nailgun/db/sqlalchemy/models/fields.py",
line 32, in process_result_value
value = jsonutils.loads(value)
File
"/usr/lib/python2.7/site-packages/oslo_serialization/jsonutils.py", line
229, in loads
return json.loads(encodeutils.safe_decode(s, encoding), **kwargs)
File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 381, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting ',' delimiter: line 1 column 320 (char 319)
*[pid: 3540|app: 0|req: 656/1262] 192.168.198.222 () {50 vars in 905
bytes} [Thu Feb 23 20:56:14 2017] GET
/api/clusters/1/network_configuration/neutron?_=1487883365500 =>
generated 39 bytes in 11 msecs (HTTP/1.1 500) 5 headers in 223 bytes (2
switches on core 0)*
[pid: 3539|app: 0|req: 607/1263] 192.168.198.222 () {50 vars in 841
bytes} [Thu Feb 23 20:56:15 2017] GET /api/clusters?_=1487883365502 =>
generated 627 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
switches on core 0)
[pid: 3539|app: 0|req: 608/1264] 192.168.198.222 () {50 vars in 835
bytes} [Thu Feb 23 20:56:15 2017] GET /api/tasks?_=1487883365504 =>
generated 286 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
switches on core 0)
[pid: 3540|app: 0|req: 657/1265] 192.168.198.222 () {50 vars in 835
bytes} [Thu Feb 23 20:56:15 2017] GET /api/nodes?_=1487883365503 =>
generated 2 bytes in 13 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
switches on core 0)
[pid: 3540|app: 0|req: 658/1266] 192.168.198.222 () {50 vars in 851
bytes} [Thu Feb 23 20:56:29 2017] GET /api/notifications?_=1487883365506
=> generated 769 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3539|app: 0|req: 609/1267] 192.168.198.222 () {50 vars in 869
bytes} [Thu Feb 23 20:56:29 2017] GET
/api/nodes/allocation/stats?_=1487883365505 => generated 30 bytes in 17
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 610/1268] 192.168.198.222 () {50 vars in 867
bytes} [Thu Feb 23 20:56:38 2017] GET
/api/clusters/1/attributes?_=1487883365508 => generated 35673 bytes in
20 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 611/1269] 192.168.198.222 () {50 vars in 871
bytes} [Thu Feb 23 20:56:38 2017] GET
/api/clusters/1/plugin_links?_=1487883365510 => generated 2 bytes in 4
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 612/1270] 192.168.198.222 () {50 vars in 861
bytes} [Thu Feb 23 20:56:38 2017] GET
/api/nodes?cluster_id=1&_=1487883365511 => generated 2 bytes in 12 msecs
(HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 613/1271] 192.168.198.222 () {50 vars in 845
bytes} [Thu Feb 23 20:56:38 2017] GET /api/clusters/1?_=1487883365507 =>
generated 625 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
switches on core 0)
[pid: 3539|app: 0|req: 614/1272] 192.168.198.222 () {50 vars in 861
bytes} [Thu Feb 23 20:56:38 2017] GET
/api/tasks?cluster_id=1&_=1487883365512 => generated 2 bytes in 3 msecs
(HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 615/1273] 192.168.198.222 () {50 vars in 871
bytes} [Thu Feb 23 20:56:38 2017] GET
/api/nodegroups?cluster_id=1&_=1487883365513 => generated 67 bytes in 3
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3540|app: 0|req: 659/1274] 192.168.198.222 () {50 vars in 857
bytes} [Thu Feb 23 20:56:38 2017] GET
/api/clusters/1/roles?_=1487883365509 => generated 7255 bytes in 67
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3540|app: 0|req: 660/1275] 192.168.198.222 () {50 vars in 845
bytes} [Thu Feb 23 20:56:38 2017] GET /api/releases/2?_=1487883365515 =>
generated 45381 bytes in 7 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
*[pid: 3539|app: 0|req: 616/1276] 192.168.198.222 () {50 vars in 905
bytes} [Thu Feb 23 20:56:38 2017] GET
/api/clusters/1/network_configuration/neutron?_=1487883365514 =>
generated 2441 bytes in 29 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)*
[pid: 3540|app: 0|req: 661/1277] 192.168.198.222 () {50 vars in 887
bytes} [Thu Feb 23 20:56:39 2017] GET
/api/v1/clusters/1/vmware_attributes?_=1487883365516 => generated 4068
bytes in 21 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on
core 0)
[pid: 3539|app: 0|req: 617/1278] 192.168.198.222 () {50 vars in 851
bytes} [Thu Feb 23 20:56:49 2017] GET /api/notifications?_=1487883365518
=> generated 769 bytes in 3 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3540|app: 0|req: 662/1279] 192.168.198.222 () {50 vars in 869
bytes} [Thu Feb 23 20:56:49 2017] GET
/api/nodes/allocation/stats?_=1487883365517 => generated 30 bytes in 16
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 618/1280] 192.168.198.222 () {50 vars in 851
bytes} [Thu Feb 23 20:57:09 2017] GET /api/notifications?_=1487883365520
=> generated 769 bytes in 6 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3540|app: 0|req: 663/1281] 192.168.198.222 () {50 vars in 869
bytes} [Thu Feb 23 20:57:09 2017] GET
/api/nodes/allocation/stats?_=1487883365519 => generated 30 bytes in 17
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 619/1282] 192.168.198.222 () {50 vars in 851
bytes} [Thu Feb 23 20:57:29 2017] GET /api/notifications?_=1487883365522
=> generated 769 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3540|app: 0|req: 664/1283] 192.168.198.222 () {50 vars in 869
bytes} [Thu Feb 23 20:57:29 2017] GET
/api/nodes/allocation/stats?_=1487883365521 => generated 30 bytes in 19
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 620/1284] 192.168.198.222 () {50 vars in 811
bytes} [Thu Feb 23 20:57:44 2017] GET /api/notifications?_=1487883365524
=> generated 769 bytes in 27 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3539|app: 0|req: 621/1285] 192.168.198.222 () {50 vars in 839
bytes} [Thu Feb 23 20:57:44 2017] GET /api/version?_=1487883365525 =>
generated 110 bytes in 2 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
switches on core 0)
[pid: 3539|app: 0|req: 622/1286] 192.168.198.222 () {50 vars in 841
bytes} [Thu Feb 23 20:57:44 2017] GET /api/settings?_=1487883365526 =>
generated 563 bytes in 3 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
switches on core 0)
[pid: 3539|app: 0|req: 623/1287] 192.168.198.222 () {50 vars in 841
bytes} [Thu Feb 23 20:57:44 2017] GET /api/clusters?_=1487883365527 =>
generated 627 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
switches on core 0)
[pid: 3539|app: 0|req: 624/1288] 192.168.198.222 () {50 vars in 835
bytes} [Thu Feb 23 20:57:44 2017] GET /api/tasks?_=1487883365529 =>
generated 286 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
switches on core 0)
[pid: 3539|app: 0|req: 625/1289] 192.168.198.222 () {50 vars in 835
bytes} [Thu Feb 23 20:57:44 2017] GET /api/nodes?_=1487883365528 =>
generated 2 bytes in 13 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
switches on core 0)
[pid: 3540|app: 0|req: 665/1290] 192.168.198.222 () {50 vars in 829
bytes} [Thu Feb 23 20:57:44 2017] GET
/api/nodes/allocation/stats?_=1487883365523 => generated 30 bytes in 81
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 626/1291] 192.168.198.222 () {50 vars in 851
bytes} [Thu Feb 23 20:57:44 2017] GET /api/notifications?_=1487883365531
=> generated 769 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3540|app: 0|req: 666/1292] 192.168.198.222 () {50 vars in 869
bytes} [Thu Feb 23 20:57:44 2017] GET
/api/nodes/allocation/stats?_=1487883365530 => generated 30 bytes in 16
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3540|app: 0|req: 667/1293] 192.168.198.222 () {50 vars in 867
bytes} [Thu Feb 23 20:57:46 2017] GET
/api/clusters/1/attributes?_=1487883365533 => generated 35673 bytes in
21 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3540|app: 0|req: 668/1294] 192.168.198.222 () {50 vars in 871
bytes} [Thu Feb 23 20:57:46 2017] GET
/api/clusters/1/plugin_links?_=1487883365535 => generated 2 bytes in 4
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3540|app: 0|req: 669/1295] 192.168.198.222 () {50 vars in 845
bytes} [Thu Feb 23 20:57:46 2017] GET /api/clusters/1?_=1487883365532 =>
generated 625 bytes in 3 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
switches on core 0)
[pid: 3540|app: 0|req: 670/1296] 192.168.198.222 () {50 vars in 861
bytes} [Thu Feb 23 20:57:46 2017] GET
/api/nodes?cluster_id=1&_=1487883365536 => generated 2 bytes in 12 msecs
(HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3540|app: 0|req: 671/1297] 192.168.198.222 () {50 vars in 861
bytes} [Thu Feb 23 20:57:46 2017] GET
/api/tasks?cluster_id=1&_=1487883365537 => generated 2 bytes in 3 msecs
(HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3540|app: 0|req: 672/1298] 192.168.198.222 () {50 vars in 871
bytes} [Thu Feb 23 20:57:46 2017] GET
/api/nodegroups?cluster_id=1&_=1487883365538 => generated 67 bytes in 2
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 627/1299] 192.168.198.222 () {50 vars in 857
bytes} [Thu Feb 23 20:57:46 2017] GET
/api/clusters/1/roles?_=1487883365534 => generated 7255 bytes in 70
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 628/1300] 192.168.198.222 () {50 vars in 845
bytes} [Thu Feb 23 20:57:46 2017] GET /api/releases/2?_=1487883365540 =>
generated 45381 bytes in 14 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3540|app: 0|req: 673/1301] 192.168.198.222 () {50 vars in 905
bytes} [Thu Feb 23 20:57:46 2017] GET
/api/clusters/1/network_configuration/neutron?_=1487883365539 =>
generated 2441 bytes in 40 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3540|app: 0|req: 674/1302] 192.168.198.222 () {50 vars in 887
bytes} [Thu Feb 23 20:57:46 2017] GET
/api/v1/clusters/1/vmware_attributes?_=1487883365541 => generated 4068
bytes in 19 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on
core 0)
[pid: 3539|app: 0|req: 629/1303] 192.168.198.222 () {50 vars in 851
bytes} [Thu Feb 23 20:58:04 2017] GET /api/notifications?_=1487883365543
=> generated 769 bytes in 3 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3540|app: 0|req: 675/1304] 192.168.198.222 () {50 vars in 869
bytes} [Thu Feb 23 20:58:04 2017] GET
/api/nodes/allocation/stats?_=1487883365542 => generated 30 bytes in 16
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 630/1305] 192.168.198.222 () {50 vars in 851
bytes} [Thu Feb 23 20:58:24 2017] GET /api/notifications?_=1487883365545
=> generated 769 bytes in 5 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3540|app: 0|req: 676/1306] 192.168.198.222 () {50 vars in 869
bytes} [Thu Feb 23 20:58:24 2017] GET
/api/nodes/allocation/stats?_=1487883365544 => generated 30 bytes in 19
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 631/1307] 192.168.198.222 () {50 vars in 851
bytes} [Thu Feb 23 20:58:44 2017] GET /api/notifications?_=1487883365547
=> generated 769 bytes in 3 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3540|app: 0|req: 677/1308] 192.168.198.222 () {50 vars in 869
bytes} [Thu Feb 23 20:58:44 2017] GET
/api/nodes/allocation/stats?_=1487883365546 => generated 30 bytes in 16
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3540|app: 0|req: 678/1309] 192.168.198.222 () {50 vars in 851
bytes} [Thu Feb 23 20:59:04 2017] GET /api/notifications?_=1487883365549
=> generated 769 bytes in 5 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
[pid: 3539|app: 0|req: 632/1310] 192.168.198.222 () {50 vars in 869
bytes} [Thu Feb 23 20:59:04 2017] GET
/api/nodes/allocation/stats?_=1487883365548 => generated 30 bytes in 22
msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 3539|app: 0|req: 633/1311] 192.168.198.222 () {50 vars in 851
bytes} [Thu Feb 23 20:59:24 2017] GET /api/notifications?_=1487883365551
=> generated 769 bytes in 5 msecs (HTTP/1.1 200) 4 headers in 185 bytes
(2 switches on core 0)
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20170223/a9ef86f0/attachment.html>
More information about the Openstack
mailing list