<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Drop. It is wasting cycles, and not
something we should use in production. Migrations specific to
SQLPlus are the most time consuming work-arounds we have. SQLPlus
does not suit our development approach. <br>
<br>
<br>
<br>
On 02/03/2015 01:32 PM, Georgy Okrokvertskhov wrote:<br>
</div>
<blockquote
cite="mid:CAG_6_on9jf34xLV21KQtqmjzP85tkhW805m9Lech8+f_J7unMg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>I think we should switch to clean migration path. We do
have production installations but we can handle initial db
uprgade case by case for customers. It is better to fix this
issue now when we have few customers rather then doing later
at larger scale.<br>
<br>
</div>
Thanks<br>
</div>
Georgy<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Feb 3, 2015 at 9:05 AM, Mike
Bayer <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:mbayer@redhat.com" target="_blank">mbayer@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex"><span
class=""><br>
<br>
Andrew Pashkin <<a moz-do-not-send="true"
href="mailto:apashkin@mirantis.com">apashkin@mirantis.com</a>>
wrote:<br>
<br>
> Mike Bayer wrote:<br>
</span><span class="">>> there’s always a naming
convention in place; all databases other than<br>
>> SQLite produce them on the fly if you don’t
specify one. The purpose<br>
>> of the Alembic/SQLAlchemy naming_convention
feature is so that you<br>
>> have *one* naming convention, rather than N
unpredictable conventions.<br>
>> I’m not sure if you’re arguing the feature should
not be used. IMHO<br>
>> it should definitely be used for an application
that is deploying<br>
>> cross-database. Otherwise you have no choice but
to hardcode the<br>
>> naming conventions of each target database
individually in all cases<br>
>> that you need to refer to them.<br>
> You can't just bring SA/Alembic "naming conventions"
into the project,<br>
> because they will collide with auto-generated
constraint names.<br>
<br>
</span>I was proposing a way to fix this for the murano
project which only appears to have four migrations so far,
but with the assumption that there are existing production
environments which cannot do a full rebuild.<br>
<span class=""><br>
><br>
> So you need to hardcode reverese-engineered
constrants names into the<br>
> old migrations and then add new migration that
renames constraint<br>
> according with "naming conventions”.<br>
> OR you need to drop old<br>
> migrations, and create new one with "naming
conventions" - that will<br>
> be backward incompatible, but cleaner.<br>
<br>
<br>
</span>My proposal was to essentially do both strategies.
Build out fully clean migrations from the start, but also
add an additional “conditional” migration that will repair a
Postgresql / MySQL database that is already at the head, and
is detected as having the older naming convention. Because
openstack does not appear to use offline migrations, this
would be doable, though not necessarily worth it.<br>
<br>
If Murano can afford to just restart with clean migrations
and has no production deployments yet which would be
disrupted by a full rebuild, then sure, just do this.<br>
<div class="HOEnZb">
<div class="h5"><br>
<br>
<br>
<br>
<br>
><br>
> On 03.02.2015 18:32, Mike Bayer wrote:<br>
>> Andrew Pashkin <<a moz-do-not-send="true"
href="mailto:apashkin@mirantis.com">apashkin@mirantis.com</a>>
wrote:<br>
>><br>
>>> Mike Bayer wrote:<br>
>>>> The patch seems to hardcode the
conventions for MySQL and Postgresql.<br>
>>>> The first thought I had was that in
order to remove the dependence<br>
>>>> on them here, you’d need to instead
simply turn off the<br>
>>>> “naming_convention” in the MetaData if
you detect that you’re on one<br>
>>>> of those two databases. That would be a
safer idea than trying to<br>
>>>> hardcode these conventions (and would
also work for other kinds<br>
>>>> of backends).<br>
>>> With your solution it is still will be
necessary for developers<br>
>>> to guess constraints names when writing new
migrations. And it will<br>
>>> be even harder, because they will need also
to handle case of<br>
>>> "naming conventions”.<br>
>><br>
>> there’s always a naming convention in place;
all databases other than SQLite produce them on the fly
if you don’t specify one. The purpose of the
Alembic/SQLAlchemy naming_convention feature is so that
you have *one* naming convention, rather than N
unpredictable conventions. I’m not sure if you’re
arguing the feature should not be used. IMHO it should
definitely be used for an application that is deploying
cross-database. Otherwise you have no choice but to
hardcode the naming conventions of each target database
individually in all cases that you need to refer to
them.<br>
>><br>
>><br>
>><br>
>><br>
>>> Mike Bayer wrote:<br>
>>>> However, it’s probably worthwhile to
introduce a migration that does<br>
>>>> in fact rename existing constraints on
MySQL and Postgresql.<br>
>>> Yes, that's what I want to do in case of
the first solution.<br>
>>><br>
>>> Mike Bayer wrote:<br>
>>>>> Another possible solution is to
drop all current migrations and<br>
>>>>> introduce new one with correct
names.<br>
>>>> you definitely shouldn’t need to do
that.<br>
>>> Why?<br>
>>><br>
>>> On 30.01.2015 22:00, Mike Bayer wrote:<br>
>>>> Andrew Pashkin <<a
moz-do-not-send="true"
href="mailto:apashkin@mirantis.com">apashkin@mirantis.com</a>>
wrote:<br>
>>>><br>
>>>>> Working on this issue I encountered
another problem.<br>
>>>>><br>
>>>>> Most indices in the project has no
names and because of that,<br>
>>>>> developer must reverse-engineer
them in every migration.<br>
>>>>> Read about that also here [1].<br>
>>>>><br>
>>>>> SQLAlchemy and Alembic provide
feature for generation constraint<br>
>>>>> names by pattern, specifically to
resolve that kind of issues [1].<br>
>>>>><br>
>>>>> I decided to introduce usage of
this feature in Murano.<br>
>>>>><br>
>>>>> I've implemented solution that
preserves backward-compatibility<br>
>>>>> for migration and allows to rename
all constraints according<br>
>>>>> to patterns safely [2]. With it
user, that have already deployed Murano<br>
>>>>> will be able to upgrade to new
version of Murano without issues.<br>
>>>>><br>
>>>>> There are downsides in this
solution:<br>
>>>>> - It assumes that all versions of
Postgres and MySQL uses the<br>
>>>>> same patterns for constraints names
generation.<br>
>>>>> - It is hard to implement a test
for this solution and it will be slow.<br>
>>>>> Because there is need to reproduce
such situation when user has old<br>
>>>>> versions of migrations applied, and
then tries to upgrade.<br>
>>>><br>
>>>> The patch seems to hardcode the
conventions for MySQL and Postgresql. The first
thought I had was that in order to remove the dependence
on them here, you’d need to instead simply turn off the
“naming_convention” in the MetaData if you detect that
you’re on one of those two databases. That would be a
safer idea than trying to hardcode these conventions
(and would also work for other kinds of backends).<br>
>>>><br>
>>>> However, I’m not actually sure that you
even need special behavior for these two backends. If
an operator runs these migrations on a clean database,
then the constraints are generated with the consistent
names on all backends. if a target database already
has these schema constructs present, then these
migrations are never run; it doesn’t matter that they
have the right or wrong names already.<br>
>>>><br>
>>>> I suppose then that the fear is that
some PG/MySQL databases will have constraints that are
named in one convention, and others will have
constraints using the native conventions. However,
the case now is that all deployments are using native
conventions, and being able to DROP these constraints is
already not very feasible unless you again were willing
to hardcode those naming conventions up forward. The
constraints in these initial migrations, assuming you
don’t regenerate them, might just need to be left alone,
and the project proceeds in the future with a consistent
convention.<br>
>>>><br>
>>>> However, it’s probably worthwhile to
introduce a migration that does in fact rename existing
constraints on MySQL and Postgresql. This would be a
migration script that emits DROP CONSTRAINT and CREATE
CONSTRAINT for all the above constraints that have an
old name and a new name. The script would need to check
the backend, as you’re doing now, in order to run, and
yes it would hardcode the names of those conventions,
but at least it would just be a one-time run against
only currently deployed databases. Since your
migrations are run “live”, the script can make itself a
“conditional” run by checking for the “old” names and
skipping those that don’t exist.<br>
>>>><br>
>>>>> Another possible solution is to
drop all current migrations and<br>
>>>>> introduce new one with correct
names.<br>
>>>><br>
>>>> you definitely shouldn’t need to do
that.<br>
>>>><br>
>>>><br>
>>>>> This brings us to new problem -
migrations and models are out of sync<br>
>>>>> right now in multiple places -
there are different field types in<br>
>>>>> migrations and models, migrations
introduces indices that is absent<br>
>>>>> in models, etc.<br>
>>>>><br>
>>>>> And this solution has great
downside - it is not backward-compatible,<br>
>>>>> so all old users will lost their
data.<br>
>>>>><br>
>>>>> We (Murano team) should decide,
what solution we want to use.<br>
>>>>><br>
>>>>><br>
>>>>> [1]<br>
>>>>> <a moz-do-not-send="true"
href="http://alembic.readthedocs.org/en/latest/naming.html#tutorial-constraint-names"
target="_blank">http://alembic.readthedocs.org/en/latest/naming.html#tutorial-constraint-names</a><br>
>>>>> [2] <a moz-do-not-send="true"
href="https://review.openstack.org/150818"
target="_blank">https://review.openstack.org/150818</a><br>
>>>>><br>
>>>>> --<br>
>>>>> With kind regards, Andrew Pashkin.<br>
>>>>> cell phone - <a
moz-do-not-send="true"
href="tel:%2B7%20%28985%29%20898%2057%2059"
value="+79858985759">+7 (985) 898 57 59</a><br>
>>>>> Skype - waves_in_fluids<br>
>>>>> e-mail - <a moz-do-not-send="true"
href="mailto:apashkin@mirantis.com">apashkin@mirantis.com</a><br>
>>>>><br>
>>>>>
__________________________________________________________________________<br>
>>>>> OpenStack Development Mailing List
(not for usage questions)<br>
>>>>> Unsubscribe: <a
moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>>>> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>><br>
>>>>
__________________________________________________________________________<br>
>>>> OpenStack Development Mailing List (not
for usage questions)<br>
>>>> Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>>> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>><br>
>>> --<br>
>>> With kind regards, Andrew Pashkin.<br>
>>> cell phone - <a moz-do-not-send="true"
href="tel:%2B7%20%28985%29%20898%2057%2059"
value="+79858985759">+7 (985) 898 57 59</a><br>
>>> Skype - waves_in_fluids<br>
>>> e-mail - <a moz-do-not-send="true"
href="mailto:apashkin@mirantis.com">apashkin@mirantis.com</a><br>
>>><br>
>>>
__________________________________________________________________________<br>
>>> OpenStack Development Mailing List (not for
usage questions)<br>
>>> Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
>>
__________________________________________________________________________<br>
>> OpenStack Development Mailing List (not for
usage questions)<br>
>> Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
> --<br>
> With kind regards, Andrew Pashkin.<br>
> cell phone - <a moz-do-not-send="true"
href="tel:%2B7%20%28985%29%20898%2057%2059"
value="+79858985759">+7 (985) 898 57 59</a><br>
> Skype - waves_in_fluids<br>
> e-mail - <a moz-do-not-send="true"
href="mailto:apashkin@mirantis.com">apashkin@mirantis.com</a><br>
><br>
>
__________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage
questions)<br>
> Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature">
<div dir="ltr"><font color="#999999"><span
style="background-color:rgb(255,255,255)">Georgy
Okrokvertskhov<br>
Architect,<br>
<span style="font-family:arial;font-size:small">OpenStack
Platform Products,</span><br>
Mirantis</span><br>
<a moz-do-not-send="true" href="http://www.mirantis.com/"
target="_blank">http://www.mirantis.com</a><br>
Tel. +1 650 963 9828<br>
Mob. +1 650 996 3284</font><br>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>