Open Stack

Hi,

I'm looking at the backup and restore for Trove instances. I have a 
devstack setup running the stable/icehouse branch (see attached local.conf).

With some minor modifications trove-guestagent.conf on the guest image 
(attached) I can create a valid backup, but attempting to result results 
in a database instance in FAILED state. Investigation shows that it has 
booted ok, but not successfully applied the restore - the failure being 
a broken pipe during the backup being streamed to the guest.

 From the guest logs (see below), it looks like we've authenticated ok 
but just get a broken pipe for no obvious reason. I've disabled 
encryption and compression to see if they were causing issues (does not 
look like it).

The swift logs on the storage don't show anything odd (200 responses to 
the request to GET the backup). Any ideas?

Regards

Mark


---- guestagent log ----

2014-06-05 04:05:18.372 971 DEBUG urllib3.connectionpool [-] "GET 
/v1/AUTH_98274f70ea6f4eba938d673c05b4d82e/database_backups/c143c4a2-4f88-4084-9ec7-c4ff28dd7585 
HTTP/1.1" 200 538523 _make_request 
/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415
2014-06-05 04:05:18.373 971 DEBUG swiftclient [-] REQ: curl -i 
http://172.24.4.1:8080/v1/AUTH_98274f70ea6f4eba938d673c05b4d82e/database_backups/c143c4a2-4f88-4084-9ec7-c4ff28dd7585 
-X GET -H "X-Auth-Token: f37eb4dea8cc24caed09aef0ad8baad8" http_log 
/usr/lib/python2.7/dist-packages/swiftclient/client.py:88
2014-06-05 04:05:18.374 971 DEBUG swiftclient [-] RESP STATUS: 200 OK 
http_log /usr/lib/python2.7/dist-packages/swiftclient/client.py:89
2014-06-05 04:05:18.375 971 DEBUG swiftclient [-] RESP HEADERS: 
[('content-length', '538523'), ('accept-ranges', 'bytes'), 
('last-modified', 'Thu, 05 Jun 2014 03:58:15 GMT'), 
('x-object-manifest', 
'database_backups/c143c4a2-4f88-4084-9ec7-c4ff28dd7585_'), 
('x-timestamp', '1401940694.39874'), ('etag', 
'"9eb376f8260e6d870f9ebd16d7dd9683"'), ('x-trans-id', 
'txa37cde877eb2481486cc9-00538fec7e'), ('date', 'Thu, 05 Jun 2014 
04:05:18 GMT'), ('content-type', 'application/octet-stream')] http_log 
/usr/lib/python2.7/dist-packages/swiftclient/client.py:90
2014-06-05 04:05:18.390 971 ERROR trove.guestagent.backup.backupagent 
[-] [Errno 32] Broken pipe
2014-06-05 04:05:18.391 971 ERROR trove.guestagent.backup.backupagent 
[-] Error restoring backup c143c4a2-4f88-4084-9ec7-c4ff28dd7585
2014-06-05 04:05:18.391 971 ERROR 
trove.guestagent.datastore.mysql.manager 
[req-5b5629a7-6a6b-46f9-a523-7c6690e4c1cb 
6cf9836394ca41459a491ec37580fdaa 98274f70ea6f4eba938d673c05b4d82e - - -] 
[Errno 32] Broken pipe
-------------- next part --------------
[[local|localrc]]
ADMIN_PASSWORD=password
MYSQL_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=tokentoken
ENABLED_SERVICES+=,-n-net
ENABLED_SERVICES+=,trove,tr-api,tr-tmgr,tr-cond
ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account
ENABLED_SERVICES+=,heat,h-api,h-api-cfn,h-api-cw,h-eng
ENABLED_SERVICES+=,-n-novnc,-n-xvnc
NOVNC_FROM_PACKAGE=false
SWIFT_HASH=12go358snjw24501
ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,q-metering,tempest
SCREEN_LOGDIR=$DEST/logs/screen
-------------- next part --------------
[DEFAULT]
datastore_manager = mysql
rabbit_host = 172.24.4.1
verbose = True
debug = True
bind_port = 8778
bind_host = 0.0.0.0
nova_proxy_admin_user = admin
nova_proxy_admin_pass = ADMIN
nova_proxy_admin_tenant_name = service
trove_auth_url = http://172.24.4.1:35357/v2.0
control_exchange = trove
root_grant = ALL
root_grant_option = True
log_dir = /tmp
log_file = guest.log
ignore_users = os_admin
ignore_dbs = lost+found, mysql, information_schema

# Backups
backup_namespace = trove.guestagent.strategies.backup.mysql_impl
restore_namespace = trove.guestagent.strategies.restore.mysql_impl
storage_strategy = SwiftStorage
storage_namespace = trove.guestagent.strategies.storage.swift
swift_url = http://172.24.4.1:8080/v1/AUTH_
backup_swift_container = database_backups
backup_use_gzip_compression = False
backup_use_openssl_encryption = False
#backup_aes_cbc_key = "default_aes_cbc_key"
backup_use_snet = False
backup_chunk_size = 65536
backup_segment_max_size = 2147483648

[mysql]
backup_strategy = MySQLDump