Open Stack

On 13/06/14 13:22, Mark Kirkwood wrote:
> On 06/06/14 13:32, Mark Kirkwood wrote:
>> On 06/06/14 11:22, Mark Kirkwood wrote:
>>> On 06/06/14 07:58, Pete Zaitcev wrote:
>>>> On Thu, 05 Jun 2014 17:14:43 +1200
>>>> Mark Kirkwood <mark.kirkwood at catalyst.net.nz> wrote:
>>>>
>>>>> The swift logs on the storage don't show anything odd (200
>>>>> responses to
>>>>> the request to GET the backup). Any ideas?
>>>>
>>>> You aren't using nginx as your LB in front of Swift by any chance?
>>>> It almost looks like the LB can't handle the GET size and keels over.
>>>>
>>>> -- P
>>>>
>>>
>>> Hmmm, I *think* I'm talking directly to the Swift proxy - but it's a
>>> good place to start looking. Thanks!
>>
>> Retrying with a guest image built using exactly the same trove
>> guestagent version (i.e from stable/icehouse to match what I am using in
>> the devstack setup) makes restore work fine.
>>
>> I *was* using an image with the guestagent from Ubuntu 14.04 by mistake.
>> So I guess I've run into some python library version incompatibilities
>> (or maybe a bug libraries that 14.04 has...I'll investigate that soon
>> when I try to get backups working in that)!
>>
>
> ...sure enough, this reappeared as soon as I started using the 14.04
> packages and the corresponding guest image running the trove guestagent.
>
> The actual bit of code failing is indicated below:
>
> trove/guestagent/strategies/restore/base.py
>
>      def _unpack(self, location, checksum, command):
>          stream = self.storage.load(location, checksum)
>          process = subprocess.Popen(command, shell=True,
>                                     stdin=subprocess.PIPE,
>                                     stderr=subprocess.PIPE)
>          content_length = 0
>          for chunk in stream:
>              process.stdin.write(chunk)         <=== broken pipe here
>              content_length += len(chunk)
>          process.stdin.close()
>          utils.raise_if_process_errored(process, RestoreError)
>          LOG.info(_("Restored %s bytes from stream.") % content_length)
>
>          return content_length
>
>
> Reading the context - the 'stream' is the backup streaming out of Swift,
> and the 'command' is 'sudo mysql' - *so* it is the passing of the data
> to restore to mysql that is failing.
>
> Adding in some more debugging show that what was actually happening was:
>
> $ sudo mysql < stream
> error: 'Access denied for user 'root'@'localhost' (using password: NO)'
>
> I'd set an initial root password for mysql in /root/.mysql_secret, and
> while this seemed to work fine for *creating* an instance it does not
> seem to for restoring it. Altering the image to have no initial root
> password makes restore work.
>
> Now it is possible I've got something configured wrong in the mysql
> setup of the guest image, and that this *should* actually work - I'll
> investigate further.

Well sort of, it looks like the following:

If you set a root password while installing mysql in the guest image, 
then set the password in /root/.mysql_secret:

$ cat /root/.mysql_secret
# The random password set for the root user at Fri May 23 00:00:00 2014 
(local time): password

and /etc/mysql/my.cnf:

$ cat /etc/mysql/my.cnf
...
[client]
...
password       = password

then instance creation, backup and restore wi.ll work ok.

Also as a bonus, using mysql 5.6 packages in the guest image will work 
too - just need to use 5.*6* in the various version strings e.g:

$ trove-manage datastore_version_update mysql mysql-5.6 mysql 
378f301d-e2df-45df-b771-90e316db5732  mysql-server-5.6 1
$ trove-manage datastore_update mysql mysql-5.6

regards

Mark