Open Stack

On Thu, Sep 23 2021 at 05:33:11 PM +0200, Rodolfo Alonso Hernandez 
<ralonsoh at redhat.com> wrote:
> Hello Balazs:
> 
> You are right: a DB migration is not necessary but a DB sanitization. 
> I did something similar in 
> https://review.opendev.org/c/openstack/neutron/+/789831. This patch 
> includes the script to, in one shot, modify the whole DB and an 
> upgrade check to test it.
> 
> About the code, if something like the script provided and the upgrade 
> check is included, that will ensure the DB is correctly migrated in Y 
> release. That means the code supporting both formats could be removed 
> in Z.

Thank you Rodolfo, I think we are on the same page now.

cheers,
gibi

> 
> Regards.
> 
> 
> On Wed, Sep 22, 2021 at 6:54 PM Balazs Gibizer 
> <balazs.gibizer at est.tech> wrote:
>> 
>> 
>>  On Wed, Sep 22 2021 at 05:09:17 PM +0200, Rodolfo Alonso Hernandez
>>  <ralonsoh at redhat.com> wrote:
>>  > Hello Balazs:
>>  >
>>  > About the group_id, I see this is built using the port_id and the
>>  > qos_rules. We have all of this in the DB and we can build it
>>  > statically (I think so, maybe I'm very optimistic).
>> 
>>  Yes, that is probably doable. But I let Przemek work out the 
>> details in
>>  the patch.
>>  >
>>  > About the code, that was something I was thinking about after 
>> sending
>>  > the last mail. For at least two releases, we need to support both 
>> RP
>>  > formats in the DB. If we read only the UUID (old format), then we
>>  > should convert it and store it in the new format.
>>  >
>>  > About the migration, we don't support contract migrations anymore.
>>  > But this is not true as we have done some migrations that have 
>> added
>>  > new restrictions in the DB schema. In any case, this could be 
>> done as
>>  > an expansion migration. If the code is in place, I don't see any
>>  > problem of doing this migration with the server running. Each
>>  > "ml2_port_bindings" register will be updated atomically, while the
>>  > Neutron server will be able to handle both versions.
>>  >
>> 
>>  If I understand correctly what you described is an online data
>>  migration where
>>  1) neutron does not even need an expand migration as no new field is
>>  needed in the database as we use the old field both for the old and 
>> the
>>  new data
>>  2) neutron server converts between data formats when the data is 
>> read
>>  3) neutron can drop the conversion code only after every register is
>>  upgraded this way. As there could be ports that are not touched 
>> between
>>  upgrades we cannot simply say that we are done with the migration 
>> after
>>  waiting a release cycle. I think we have to add an upgrade check in
>>  Yoga that warns if there are still ports with the old format. And 
>> also
>>  neutron needs to provide a tool for the deployer to trigger the
>>  conversion of those remaining port before the upgrade to X.
>> 
>>  Do I understand your suggestion correct? Do you agree with the above
>>  solution proposal?
>> 
>>  Cheers,
>>  gibi
>> 
>>  > Regards.
>>  >
>>  >
>>  > On Wed, Sep 22, 2021 at 3:44 PM Balazs Gibizer
>>  > <balazs.gibizer at est.tech> wrote:
>>  >>
>>  >>
>>  >>  On Tue, Sep 21 2021 at 06:30:46 PM +0200, Rodolfo Alonso 
>> Hernandez
>>  >>  <ralonsoh at redhat.com> wrote:
>>  >>  > Hello Balazs:
>>  >>  >
>>  >>  > Sorry for the late reply, I was on PTO.
>>  >>  >
>>  >>  > If I'm not wrong, now port['binding:profile']['allocation'] 
>> is a
>>  >> UUID
>>  >>  > and you need it to be a list of UUIDs. Am I correct?
>>  >>
>>  >>  It is a bit more complicated than that. The old value is a 
>> single RP
>>  >>  UUID. the new value is a dict where the key is the group_id and 
>> the
>>  >>  value is the RP UUID fulfilling that group. So the 
>> transformation
>>  >> needs
>>  >>  to access to the group_id.
>>  >>  The group_id is a stable UUID generated by neutron server as 
>> part of
>>  >>  the port.resource_request value, but it is not persisted.
>>  >>
>>  >>  >
>>  >>  > To make this change in the DB you should use the Alembic
>>  >> migrations,
>>  >>  > as you said. That should ensure all registers are translated. 
>> We
>>  >>  > should also include a sanity check to ensure the DB migration 
>> was
>>  >>  > done correctly.
>>  >>
>>  >>  I'm not 100% sure but I think such data migration can only be 
>> done
>>  >> in
>>  >>  the contract part as it needs to be done while the neutron 
>> server is
>>  >>  down as the old code can only use the old data format while the 
>> new
>>  >>  code can only use the new format. Is it OK to introduce a new
>>  >> contract
>>  >>  migration in Yoga in neutron?
>>  >>
>>  >>  Cheers,
>>  >>  gibi
>>  >>
>>  >>
>>  >>  >
>>  >>  > Is that what you needed? Don't hesitate to ping me in IRC if
>>  >> needed.
>>  >>  >
>>  >>  > Regards.
>>  >>  >
>>  >>  > On Fri, Sep 10, 2021 at 6:06 PM Balazs Gibizer
>>  >>  > <balazs.gibizer at est.tech> wrote:
>>  >>  >> Hi Neutrinos!
>>  >>  >>
>>  >>  >>  We found a technical challenge during implementing the
>>  >>  >>  port-resource-request-groups API extension[1]. That 
>> extension
>>  >>  >> changes
>>  >>  >>  the format of the port.resoruce_request as well as the 
>> format
>>  >> of the
>>  >>  >>  port.binding:profile.allocation. The former is a calculated
>>  >> field on
>>  >>  >>  the port so that is easy. However the bindig:profile is
>>  >> persisted in
>>  >>  >>  the database so data migration is needed. What is the 
>> canonical
>>  >> way
>>  >>  >> to
>>  >>  >>  do such DB data translation in Neutron? Can we translate the
>>  >> data in
>>  >>  >>  place during alembic migration? Or should we do some kind of
>>  >> online
>>  >>  >>  data migration when the data is translated by neutron when 
>> it is
>>  >>  >> read
>>  >>  >>  from the db?
>>  >>  >>
>>  >>  >>  cheers,
>>  >>  >>  gibi
>>  >>  >>
>>  >>  >>  [1] 
>> https://review.opendev.org/c/openstack/neutron/+/805637/5
>>  >>  >>
>>  >>  >>
>>  >>  >>
>>  >>
>>  >>
>> 
>>