Open Stack

On 06/06/14 13:32, Mark Kirkwood wrote:
> On 06/06/14 11:22, Mark Kirkwood wrote:
>> On 06/06/14 07:58, Pete Zaitcev wrote:
>>> On Thu, 05 Jun 2014 17:14:43 +1200
>>> Mark Kirkwood <mark.kirkwood at catalyst.net.nz> wrote:
>>>
>>>> The swift logs on the storage don't show anything odd (200 responses to
>>>> the request to GET the backup). Any ideas?
>>>
>>> You aren't using nginx as your LB in front of Swift by any chance?
>>> It almost looks like the LB can't handle the GET size and keels over.
>>>
>>> -- P
>>>
>>
>> Hmmm, I *think* I'm talking directly to the Swift proxy - but it's a
>> good place to start looking. Thanks!
>
> Retrying with a guest image built using exactly the same trove
> guestagent version (i.e from stable/icehouse to match what I am using in
> the devstack setup) makes restore work fine.
>
> I *was* using an image with the guestagent from Ubuntu 14.04 by mistake.
> So I guess I've run into some python library version incompatibilities
> (or maybe a bug libraries that 14.04 has...I'll investigate that soon
> when I try to get backups working in that)!
>

...sure enough, this reappeared as soon as I started using the 14.04 
packages and the corresponding guest image running the trove guestagent.

The actual bit of code failing is indicated below:

trove/guestagent/strategies/restore/base.py

     def _unpack(self, location, checksum, command):
         stream = self.storage.load(location, checksum)
         process = subprocess.Popen(command, shell=True,
                                    stdin=subprocess.PIPE,
                                    stderr=subprocess.PIPE)
         content_length = 0
         for chunk in stream:
             process.stdin.write(chunk)         <=== broken pipe here
             content_length += len(chunk)
         process.stdin.close()
         utils.raise_if_process_errored(process, RestoreError)
         LOG.info(_("Restored %s bytes from stream.") % content_length)

         return content_length


Reading the context - the 'stream' is the backup streaming out of Swift, 
and the 'command' is 'sudo mysql' - *so* it is the passing of the data 
to restore to mysql that is failing.

Adding in some more debugging show that what was actually happening was:

$ sudo mysql < stream
error: 'Access denied for user 'root'@'localhost' (using password: NO)'

I'd set an initial root password for mysql in /root/.mysql_secret, and 
while this seemed to work fine for *creating* an instance it does not 
seem to for restoring it. Altering the image to have no initial root 
password makes restore work.

Now it is possible I've got something configured wrong in the mysql 
setup of the guest image, and that this *should* actually work - I'll 
investigate further.

Regards

Mark