[openstack-dev] [Trove] Resource not found when creating db instances.

Wang Sen wangsen at linux.vnet.ibm.com
Thu Jan 19 03:51:57 UTC 2017


Hi all,

I met the resource not found error when I'm creating a database
instance. The instance stays on build status and turns to error status
after timeout.

I know trove is not ready for neuton. Is there a work around for this
issue ? Thanks in advance.

Below is the detailed information.

Error Log
=========

/var/log/trove/trove-taskmanager.log:
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task [-] Error during Manager.publish_exists_event
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task Traceback (most recent call last):
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     task(self, context)
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/trove/taskmanager/manager.py", line 429, in publish_exists_event
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     self.admin_context)
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/trove/extensions/mgmt/instances/models.py", line 178, in publish_exist_events
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     notifications = transformer()
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/trove/extensions/mgmt/instances/models.py", line 271, in __call__
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     client=self.nova_client)
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/trove/extensions/mgmt/instances/models.py", line 40, in load_mgmt_instances
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     mgmt_servers = client.servers.list(search_opts={'all_tenants': 1})
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 835, in list
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     "servers")
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 249, in _list
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     resp, body = self.api.client.get(url)
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 480, in get
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     return self._cs_request(url, 'GET', **kwargs)
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 436, in _cs_request
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     self.authenticate()
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 619, in authenticate
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     self._v2_auth(auth_url)
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 684, in _v2_auth
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     return self._authenticate(url, body)
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 697, in _authenticate
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     **kwargs)
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 431, in _time_request
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     resp, body = self.request(url, method, **kwargs)
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 425, in request
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task     raise exceptions.from_response(resp, body, url, method)
2017-01-19 11:27:31.666 22795 ERROR oslo_service.periodic_task NotFound: The resource could not be found. (HTTP 404)

Openstack Cluster
=================

openstack version: Neuton
trove version: 2.5.0
    $ root at kvm-215:~# trove --version
    2.5.0
    $ root at kvm-215:~# openstack --version
    openstack 3.2.0

Controller Node: ubuntu 16.04, 9.181.129.215
Compute Node: ubuntu 16.04, 9.181.129.213

Manage network: 192.168.1.0/24
Provider network: 9.181.129.0/24

endpoints:

    $ openstack endpoint list

+----------------------------------+-----------+--------------+--------------+---------+-----------+----------------------------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                          |
+----------------------------------+-----------+--------------+--------------+---------+-----------+----------------------------------------------+
| 08fa008d727643428800f10b31e32e3b | RegionOne | glance       | image        | True    | admin     | http://controller:9292                       |
| 26cd2828179e4ca58b4cda80127889c2 | RegionOne | cinder       | volume       | True    | admin     | http://controller:8776/v1/%(tenant_id)s      |
| 3148ed89e7e241b287d5dc68960d8052 | RegionOne | cinderv2     | volumev2     | True    | admin     | http://controller:8776/v2/%(tenant_id)s      |
| 42d5aac6c0a349c5a167280459b16549 | RegionOne | trove        | database     | True    | public    | http://controller:8779/v1.0/%(tenant_id)s    |
| 4600804165954819944a8c0ff9fe5f7d | RegionOne | trove        | database     | True    | admin     | http://controller:8779/v1.0/%(tenant_id)s    |
| 4b7e6ba4db9d406d9f008a5c71700d94 | RegionOne | nova         | compute      | True    | internal  | http://controller:8774/v2.1/%(tenant_id)s    |
| 4c35a14f4c8a4224aede30f91af25647 | RegionOne | glance       | image        | True    | public    | http://controller:9292                       |
| 5a2ea322cadb4a4881db689015dfa182 | RegionOne | neutron      | network      | True    | public    | http://controller:9696                       |
| 688688d146ee4d0eb93da67d108dd6a8 | RegionOne | neutron      | network      | True    | internal  | http://controller:9696                       |
| 6db0991bd4e9439a87ed08e696dfbf35 | RegionOne | cinderv2     | volumev2     | True    | internal  | http://controller:8776/v2/%(tenant_id)s      |
| 721c045a5acb44c8a67545bd880e05aa | RegionOne | cinder       | volume       | True    | internal  | http://controller:8776/v1/%(tenant_id)s      |
| 74261ec81ed642b680b35a101902f9c2 | RegionOne | swift        | object-store | True    | public    | http://controller:8080/v1/AUTH_%(tenant_id)s |
| 74f5a84c8bb344bda8f3dce5570001d4 | RegionOne | neutron      | network      | True    | admin     | http://controller:9696                       |
| 85a295d376584809ac2c8adde8c59afa | RegionOne | cinder       | volume       | True    | public    | http://controller:8776/v1/%(tenant_id)s      |
| 886c00a92bbf4b6d9244427d4f7a0e8d | RegionOne | swift        | object-store | True    | admin     | http://controller:8080/v1                    |
| 8ccce1c3736a4c0493ffee8f07f016b0 | RegionOne | keystone     | identity     | True    | admin     | http://controller:35357/v3/                  |
| b59590340879457cb6311d3044387635 | RegionOne | swift        | object-store | True    | internal  | http://controller:8080/v1/AUTH_%(tenant_id)s |
| b80a4a3dda8344ac992644d2c8dc167b | RegionOne | trove        | database     | True    | internal  | http://controller:8779/v1.0/%(tenant_id)s    |
| bcfc25d9ec6d4cbfb10746cbd3caf153 | RegionOne | keystone     | identity     | True    | internal  | http://controller:35357/v3/                  |
| bf565acdf1b64ea8a079a7e49f37331a | RegionOne | keystone     | identity     | True    | public    | http://controller:5000/v3/                   |
| dbcb99659b9f428aae909f77cb22bd1c | RegionOne | cinderv2     | volumev2     | True    | public    | http://controller:8776/v2/%(tenant_id)s      |
| de93f0d04ffa43b0bdc93da396044a06 | RegionOne | nova         | compute      | True    | admin     | http://controller:8774/v2.1/%(tenant_id)s    |
| f9e8afbd6fe747619fd325982930e782 | RegionOne | glance       | image        | True    | internal  | http://controller:9292                       |
| fb13a0896e724ac4b972531a3cc46853 | RegionOne | nova         | compute      | True    | public    | http://controller:8774/v2.1/%(tenant_id)s    |
+----------------------------------+-----------+--------------+--------------+---------+-----------+----------------------------------------------+

Trove Configuration
===================

trove.conf:
[DEFAULT]
debug = False
auth_strategy = keystone
add_addresses = True
bind_host = 0.0.0.0
bind_port = 8779
rpc_backend=rabbit
control_exchange = trove
db_api_implementation = "trove.db.sqlalchemy.api"
trove_auth_url = http://controller:5000/v3
nova_compute_url = http://controller:8774/v2.1
cinder_url = http://controller:8776/v1
swift_url = http://controller:8080/v1/AUTH_
notifier_queue_hostname = controller
os_region_name = RegionOne
network_label_regex = .*
trove_volume_support = True
block_device_mapping = vdb
device_path = /dev/vdb
max_accepted_volume_size = 10
max_instances_per_tenant = 5
max_volumes_per_tenant = 100
max_backups_per_tenant = 5
volume_time_out=30
http_get_rate = 200
http_post_rate = 200
http_put_rate = 200
http_delete_rate = 200
http_mgmt_post_rate = 200
trove_dns_support = False
dns_account_id = 123456
dns_auth_url = http://127.0.0.1:5000/v2.0
dns_username = user
dns_passkey = password
dns_ttl = 3600
dns_domain_name = 'trove.com.'
dns_domain_id = 11111111-1111-1111-1111-111111111111
dns_driver = trove.dns.designate.driver.DesignateDriver
dns_instance_entry_factory = trove.dns.designate.driver.DesignateInstanceEntryFactory
dns_endpoint_url = http://127.0.0.1/v1/
dns_service_type = dns
network_driver = trove.network.nova.NovaNetwork
default_neutron_networks =
taskmanager_queue = taskmanager
admin_roles = admin
agent_heartbeat_time = 10
agent_call_low_timeout = 5
agent_call_high_timeout = 150
reboot_time_out = 60
api_paste_config = /etc/trove/api-paste.ini
log_dir = /var/log/trove
[database]
connection = mysql+pymysql://trove:pass4trove@controller/trove
idle_timeout = 3600
[profiler]
[ssl]
[t_host=controller
rabbit_userid=openstack
rabbit_password=RABBITPASS
[mysql]
root_on_create = False
tcp_ports = 3306
volume_support = True
device_path = /dev/vdb
ignore_users = os_admin, root
ignore_dbs = mysql, information_schema, performance_schema
[redis]
tcp_ports = 6379
volume_support = False
[cassandra]
tcp_ports = 7000, 7001, 9042, 9160
volume_support = True
device_path = /dev/vdb
[couchbase]
tcp_ports = 8091, 8092, 4369, 11209-11211, 21100-21199
volume_support = True
device_path = /dev/vdb
[mongodb]
tcp_ports = 2500, 27017, 27019
volume_support = True
device_path = /dev/vdb
num_config_servers_per_cluster = 1
num_query_routers_per_cluster = 1
[vertica]
tcp_ports = 5433, 5434, 22, 5444, 5450, 4803
udp_ports = 5433, 4803, 4804, 6453
volume_support = True
device_path = /dev/vdb
cluster_support = True
cluster_member_count = 3
api_strategy = trove.common.strategies.cluster.experimental.vertica.api.VerticaAPIStrategy
[cors]
[cors.subdomain]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = trove
password = pass4trove
oslo_messaging_rabbit]


trove-taskmanager.conf:
[DEFAULT]
debug = False
update_status_on_fail = True
rpc_backend=rabbit
control_exchange = trove
db_api_implementation = trove.db.sqlalchemy.api
trove_auth_url = http://controller:5000/v3
nova_compute_url = http://controller:8774/v2.1
cinder_url = http://controller:8776/v1
swift_url = http://controller:8080/v1/AUTH_
notifier_queue_hostname = controller
os_region_name = RegionOne
trove_volume_support = True
block_device_mapping = vdb
device_path = /dev/vdb
mount_point = /var/lib/mysql
volume_time_out=30
server_delete_time_out=480
use_nova_server_config_drive = True
nova_proxy_admin_user = admin
nova_proxy_admin_pass = frobisher86
nova_proxy_admin_tenant_name = service
taskmanager_manager=trove.taskmanager.manager.Manager
exists_notification_transformer = trove.extensions.mgmt.instances.models.NovaNotificationTransformer
exists_notification_ticks = 30
notification_service_id = mysql:2f3ff068-2bfb-4f70-9a9d-a6bb65bc084b
trove_dns_support = False
dns_account_id = 123456
dns_auth_url = http://controller:5000/v2.0
dns_username = user
dns_passkey = password
dns_ttl = 3600
dns_domain_name = 'trove.com.'
dns_domain_id = 11111111-1111-1111-1111-111111111111
dns_driver = trove.dns.designate.driver.DesignateDriver
dns_instance_entry_factory = trove.dns.designate.driver.DesignateInstanceEntryFactory
dns_endpoint_url = http://controller/v1/
dns_service_type = dns
network_driver=trove.network.neutron.NeutronDriver
default_neutron_networks =
trove_security_groups_support = True
trove_security_group_rule_cidr = 0.0.0.0/0
agent_heartbeat_time = 10
agent_call_low_timeout = 5
agent_call_high_timeout = 150
agent_replication_snapshot_timeout = 36000
use_nova_server_volume = False
network_label_regex = .*
template_path = /etc/trove/templates/
pydev_debug = disabled
[database]
connection = mysql+pymysql://trove:pass4trove@controller/trove
idle_timeout = 3600
[profiler]
[oslo_messaging_rabbit]
rabbit_host=controller
rabbit_userid=openstack
rabbit_password=pass4rabbit
[mysql]
icmp = True
tcp_ports = 3306
volume_support = True
device_path = /dev/vdb
[redis]
tcp_ports = 6379
volume_support = False
[cassandra]
tcp_ports = 7000, 7001, 9042, 9160
volume_support = True
device_path = /dev/vdb
[couchbase]
tcp_ports = 8091, 8092, 4369, 11209-11211, 21100-21199
volume_support = True
device_path = /dev/vdb
[mongodb]
volume_support = True
device_path = /dev/vdb
[vertica]
tcp_ports = 5433, 5434, 22, 5444, 5450, 4803
udp_ports = 5433, 4803, 4804, 6453
volume_support = True
device_path = /dev/vdb
mount_point = /var/lib/vertica
taskmanager_strategy = trove.common.strategies.cluster.experimental.vertica.taskmanager.VerticaTaskManagerStrategy


trove-conductor.conf:
[DEFAULT]
debug = False
trove_auth_url = http://controller:5000/v3
notifier_queue_hostname = controller
log_dir = /var/log/trove
conductor_manager = trove.conductor.manager.Manager
rpc_backend=rabbit
control_exchange = trove
[profiler]
[database]
connection = mysql+pymysql://trove:pass4trove@controller/trove
[oslo_messaging_rabbit]
rabbit_host=controller
rabbit_userid=openstack
rabbit_password=RABBITPASS


trove-guestagent.conf:
[DEFAULT]
debug = True
log_file = trove-guestagent.log
log_dir = /var/log/trove
ignore_users = os_admin
control_exchange = trove

rabbit_host = controller
rabbit_userid = openstack
rabbit_password = RABBITPASS
nova_proxy_admin_user = admin
nova_proxy_admin_pass = ADMIN_PASS
nova_proxy_admin_tenant_name = service
trove_auth_url = http://controller:35357/v3




More information about the OpenStack-dev mailing list