Open Stack

On 4/23/15 1:31 AM, Dan Smith wrote:
>> Sure, but for people doing continuous deployment, they clearly haven't
>> ran the migrate_flavor_data (or if they have, they haven't filed any
>> bugs about it not working[0]).
> Hence the usefulness of T-H here, right? The point of the migration
> check is to make sure that people _do_ run it before the leave kilo.
> Right now, they have nothing other than the big note in the release
> notes about doing it. Evidence seems to show that they're not seeing it,
> which is exactly why we need the check :)
>
>> I also found what I believe to be a bug with the flavor migration code.
>> I started working on a fix by my limited knowledge of nova's objects has
>> hindered me. Any thoughts on the legitimacy of the bug would be helpful
>> though: https://bugs.launchpad.net/nova/+bug/1447132 . Basically for
>> some of the datasets that turbo-hipster uses there are no entries in the
>> new instance_extra table stopping any flavor migration from actually
>> running. Then in your change (174480) you check the metadata table
>> instead of the extras table causing the migration to fail even though
>> migrate_flavor_data thinks there is nothing to do.
> I don't think this has anything to do with the objects, it's probably
> just a result of my lack of sqlalchemy-fu. Sounds like you weren't close
> to a fix, so I'll try to cook something up.
Yeah it was my sqlalchemy-fu letting me down too. However I mentioned 
the objects because I was down a rabbit hole trying to figure out all of 
the code surrounding loading flavours from either system-metadata or extras.

If I selected all the instance_type_id's from the system-metadata table 
and used those uuid's to load the object with something like:
         instance = objects.Instance.get_by_uuid(
             context, instance_uuid,
             expected_attrs=['system_metadata', 'flavor'])

The tests would fail at that point when trying to read in the flavor as 
json. http://paste.openstack.org/show/205158/

Basically without digging further it seems like I should be able to load 
an instance by uuid regardless of the state of my flavor(s). Since this 
fails it seems like there is something wrong with the flavor handling on 
the objects.

>
> So, a question here: These data sets were captured at some point in
> time, right? Does that mean that they were from say Icehouse era and
> have had nothing done to them since? Meaning, are there data sets that
> actually had juno or kilo running on them? This instance_extra thing
> would be the case for any instance that hasn't been touched in a long
> time, so it's legit. However, as we move to more online migration of
> data, I do wonder if taking an ancient dataset, doing schema migrations
> forward to current and then expecting it to work is really reflective of
> reality.
>
> Can you shed some light on what is really going on?

As mikal has followed up, that's correct. We have intended to refresh 
our datasets and will try and get some more recent ones, but testing the 
old ones has also proven useful.

Another interesting thing is that migrate_flavor_data avoids migrating 
instances that are in the middle of an operation. The snapshot of 
databases we have include instances in this state. Since turbo-hipster 
is just testing against a snapshot in time there is no way for those 
instances to leave their working state and hence migrate_flavor_data can 
never finish (every run will leave instances undone). This therefore 
blocks the migrations from ever finishing.

I don't think this is a real world problem though, so once we get this 
migration closer to merging we might have to force the instances to be 
migrated in our datasets. In fact, that could be a useful feature so I 
wrote it here: https://review.openstack.org/#/c/176574/

Cheers,
Josh

>
>> I think it's worth noting that your change (174480) will require
>> operators (particularly continuous deployers) to run migrate_flavor_data
>> and given the difficulties I've found I'm not sure it's ready to be ran.
> Right, that's the point.
>
>> If we encounter bugs against real datasets with migrate_flavor_data then
>> deployers will be unable to upgrade until migrate_flavor_data is fixed.
>> This may make things awkward if a deployer updates their codebase and
>> can't run the migrations. Clearly they'll have to roll-back the changes.
>> This is the scenario turbo-hipster is meant to catch - migrations that
>> don't work on real datasets.
> Right, that's why we're holding until we make sure that it works.
>
>> If migrate_flavor_data is broken a backport into Kilo will be needed so
>> that if Liberty requires all the flavor migrations to be finished they
>> can indeed be ran before upgrading to Liberty. This may give reason not
>> to block on having the flavors migrated until the M-release but I
>> realise that has other undersiable consequences (ie high code maintenance).
> Backports to fix this are fine IMHO, and just like any other bug found
> in actual running of things. It's too bad that none of our continuous
> deployment people seem to have found this yet, but that's a not uncommon
> occurrence. Obviously if we hit something that makes it too painful to
> get right in kilo, then we can punt for another cycle.
>
> Thanks!
>
> --Dan
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev