Open Stack

Hi Jay,

Comments below..

On 7/24/2017 8:01 AM, Jay Pipes wrote:
> +Dan Smith
>
> Good morning Mike :) Comments inline...
>
> On 07/23/2017 08:05 PM, Michael Bayer wrote:
>> On Sun, Jul 23, 2017 at 6:10 PM, Jay Pipes <jaypipes at gmail.com> wrote:
>>> Glad you brought this up, Mike. I was going to start a thread about 
>>> this.
>>> Comments inline.
>>>
>>> On 07/23/2017 05:02 PM, Michael Bayer wrote:
>>> Well, besides that point (which I agree with), that is attempting to 
>>> change
>>> an existing database schema migration, which is a no-no in my book ;)
>>
>> OK this point has come up before and I always think there's a bit of
>> an over-broad kind of purism being applied (kind of like when someone
>> says, "never use global variables ever!" and I say, "OK so sys.modules
>> is out right ?" :)  ).
>
> I'm not being a purist. I'm being a realist :) See below...
>
>> I agree with "never change a migration" to the extent that you should
>> never change the *SQL steps emitted for a database migration*. That
>> is, if your database migration emitted "ALTER TABLE foobar foobar
>> foobar" on a certain target databse, that should never change. No
>> problem there.
>>
>> However, what we're doing here is adding new datatype logic for the
>> NDB backend which are necessarily different; not to mention that NDB
>> requires more manipulation of constraints to make certain changes
>> happen.  To make all that work, the *Python code that emits the SQL
>> for the migration* needs to have changes made, mostly (I would say
>> completely) in the form of new conditionals for NDB-specific concepts.
>>     In the case of the datatypes, the migrations will need to refer to
>> a SQLAlchemy type object that's been injected with the ndb-specific
>> logic when the NDB backend is present; I've made sure that when the
>> NDB backend is *not* present, the datatypes behave exactly the same as
>> before.
>
> No disagreement here.
>
>> So basically, *SQL steps do not change*, but *Python code that emits
>> the SQL steps* necessarily has to change to accomodate for when the
>> "ndb" flag is present - this because these migrations have to run on
>> brand new ndb installations in order to create the database. If Nova
>> and others did the initial "create database" without using the
>> migration files and instead used a create_all(), things would be
>> different, but that's not how things work (and also it is fine that
>> the migrations are used to build up the DB).
>
> So, I see your point here, but my concern here is that if we *modify* 
> an existing schema migration that has already been tested to properly 
> apply a schema change for MySQL/InnoDB and PostgreSQL with code that 
> is specific to NDB, we introduce the potential for bugs where users 
> report that the same migration works sometimes but fails other times.

I don't think that the testing issues should be a concern here because 
I've been working to make sure that the tests work with both InnoDB and 
NDB. It's a pain, but again, we are only talking about a handful of the 
services. Bottom line here is that if you are not using NDB, the changes 
have zero effect on your setup.

>
> I would much prefer to *add* a brand new schema migration that handles 
> conversion of the entire InnoDB schema at a certain point to an 
> NDB-compatible one *after* that point. That way, we isolate the NDB 
> changes to one specific schema migration -- and can point users to 
> that one specific migration in case bugs arise. This is the reason 
> that every release we add a number of "placeholder" schema migration 
> numbered files to handle situations such as these.

The only problem with this approach is that it assumes you are on InnoDB 
to start out with, which is not the use case here. This is for new 
installations or ones that started out with NDB, so we have to start out 
with the base schema in the scripts working.
>
> I understand that Oracle wants to support older versions of OpenStack 
> in their distribution and that's totally cool with me. But, the proper 
> way IMHO to do this kind of thing is to take one of the placeholder 
> migrations and use that as the NDB-conversion migration. I would posit 
> that since Oracle will need to keep some not-insignificant amount of 
> Python code in their distribution fork of Nova in order to bring in 
> the oslo.db and Nova NDB support, that it will actually be *easier* 
> for them to maintain a *separate* placeholder schema migration for all 
> NDB conversion work instead of changing an existing schema migration 
> with a new patch.

And this is the whole point of the work that I'm doing. Getting upstream 
so that others can benefit and so that we don't have to waste cycles 
maintaining custom code. Instead, we do all of the work upstream and 
that will enable our customers to more easily upgrade from one release 
to another. FYI, we have been using NDB since version 2 of our product. 
We are working on version 4 right now.

>
> All the best,
> -jay
>
> __________________________________________________________________________ 
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 

Oracle <http://www.oracle.com/>
Octave J. Orgeron | Sr. Principal Architect
Certified Enterprise Architect
Oracle Linux OpenStack
Mobile: +1-720-616-1550 <tel:+17206161550>
500 Eldorado Blvd. | Broomfield, CO 80021
TOGAF 9.1 Certified <http://www.opengroup.org/certifications/togaf> 
Certified Oracle Enterprise Architect: Systems Infrastructure 
<http://www.oracle.com/us/solutions/enterprise-architecture/index.html>
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to 
developing practices and products that help protect the environment

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170724/9f549282/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oracle_sig_logo.gif
Type: image/gif
Size: 658 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170724/9f549282/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: togaf9-certified_web.png
Type: image/png
Size: 12836 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170724/9f549282/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oea_logo.jpg
Type: image/jpeg
Size: 2398 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170724/9f549282/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: green-for-email-sig_0.gif
Type: image/gif
Size: 356 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170724/9f549282/attachment-0001.gif>