Open Stack

On 01/24/2017 04:49 AM, Kirill Proskurin wrote:
> HI!
>
> Thing is, running Galera without enforcing it very bad idea for
> production use. Permissive mode was added more or less for testing
> purposes, running real production with it is dangerous, since it's not
> enforcing the mandatory Galera requirement, one of them is a "All
> tables must have a primary key".

Any application that wishes to run Galera in PK-enforcing mode may do 
so, by simply adding a primary key constraint to the alembic_version 
table beforehand if one is not already present.   Openstack applications 
may add a migration that does this automatically as well.

But beyond that, I disagree with this characterization.  Galera's 
documentation does not state "All tables must have a primary key", it 
states, "Do not use tables without a primary key", and the reasons they 
list do not affect Alembic's use case except in one potential edge case 
(although the documentation is too vague to know if it's real or not), 
for which the above workaround that the Openstack application add a PK 
constraint up front would mitigate.


Reviewing Galera's documentation on this subject:


http://galeracluster.com/documentation-webpages/limitations.html

"Do not use tables without a primary key."

OK, nevertheless they are giving us the choice.   Let's see why that is:


"When tables lack a primary key, rows can appear in different order on 
different nodes in your cluster. "

The Alembic versioning table typically stores exactly one row, or in the 
case of an Openstack application using multiple branches like Neutron 
may store several rows, however there is no ordering requirement on this 
table when they are SELECTed.  Additionally, relational database tables 
have no defined ordering in *any* case, so a SELECT that has no ORDER BY 
yet which expects a specific ordering is already broken.

The other case for intrinsic ordering is that concurrent UPDATE 
statements on a table may produce different row locking behavior; so 
that an UPDATE applied to a table may apply differently depending on 
which galera node is accessed.   This is a non-issue for the alembic 
version table because this table is only accessed by a single process 
during an upgrade operation and is not subject to any concurrency.

"As such, queries like SELECT...LIMIT... can return different results."

You should *never* run SELECT..LIMIT without an ORDER BY on any 
relational database so this point is moot.

"Additionally, on such tables the DELETE statement is unsupported."

Here we have the first issue that can be a problem.   The 
alembic_version table is normally not subject to a DELETE operation. 
The feature in which DELETE is used is if a project is using branches 
that merge together - when an upgrade across a merge point is made, the 
additional rows in the table for the merged points are DELETEd.

Currently, I'm not sure if any openstack project is merging any branches 
together.   If they are, they should include a migration that manually 
adds a primary key constraint to the alembic_version table.  That is, 
each Openstack project can address this issue directly just by including 
a migration that adds a primary key constraint to the alembic_version table.

In my testing, I cannot locate the nature of the "is unsupported" 
statement.   The DELETE statement does in fact work "fine" on a Galera 
cluster table that has no primary key, the correct row is deleted and 
the correct result is replicated to the other nodes.   So if 
"unsupported" means, "may fail under load", "doesn't work with the 
AUTO_INCREMENT feature" (which Alembic does not use) "may fail 
randomly", or simply, "works fine we just don't want to test it", is not 
clear.   But again, the upstream Alembic fix isn't as essential as much 
as it is that a project that is merging branches together makes sure 
they perform an ALTER against the alembic_version table to add a primary 
key constraint if one does not exist already.


"Note - If you have a table without a primary key, it is always possible 
to add an AUTO_INCREMENT column to the table without breaking your 
application. "

this cryptic statement suggests that perhaps Galera's AUTO_INCREMENT 
feature is the concern they have with DELETE.  Alembic does not use the 
AUTO_INCREMENT feature on this table.

Overall, Galera's notes on this subject do not indicate that an 
application with an un-constrained alembic_version table is dangerous in 
any way, with the possible exception of a branch merge operation for 
which the risk can be mitigated by adding the primary key constraint to 
the table.    Additional detail as to the "DELETE is not supported" 
comment would be helpful.





  You cant disable a single check, you
> could only disable them all and this could lead to some serious
> problems, like data loss or corruption.
>
> If OS wants support Galera, it needs to comply with the Galera
> requirements.
>
> On Mon, Jan 23, 2017 at 9:59 PM, Ihar Hrachyshka <ihrachys at redhat.com
> <mailto:ihrachys at redhat.com>> wrote:
>
>     An alternative could also be, for Newton and earlier, to release a
>     note saying that operators should not run the code against ENFORCING
>     galera mode. What are the reasons to enable that mode in OpenStack
>     scope that would not allow operators to live without it for another
>     cycle?
>
>     Ihar
>
>     On Mon, Jan 23, 2017 at 10:12 AM, Anna Taraday
>     <akamyshnikova at mirantis.com <mailto:akamyshnikova at mirantis.com>> wrote:
>     > Hello everyone!
>     >
>     > Guys in our team faced an issue when they try to run alembic
>     migrations on
>     > Galera with ENFORCING mode. [1]
>     >
>     > This was an issue with Alembic [2], which was quickly fixed by
>     Mike Bayer
>     > (many thanks!) and new version of alembic was resealed [3].
>     > The global requirements are updated [4].
>     >
>     > I think that it is desired to fix this for Newton at least. We
>     cannot bump
>     > requirements for Newton, so hot fix can be putting pk on this
>     table in the
>     > first migration like proposed [5].  Any other ideas?
>     >
>     > [1] - https://bugs.launchpad.net/neutron/+bug/1655610
>     <https://bugs.launchpad.net/neutron/+bug/1655610>
>     > [2] - https://bitbucket.org/zzzeek/alembic/issues/406
>     <https://bitbucket.org/zzzeek/alembic/issues/406>
>     > [3] -
>     http://alembic.zzzcomputing.com/en/latest/changelog.html#change-0.8.10
>     <http://alembic.zzzcomputing.com/en/latest/changelog.html#change-0.8.10>
>     > [4] - https://review.openstack.org/#/c/423118/
>     <https://review.openstack.org/#/c/423118/>
>     > [5] - https://review.openstack.org/#/c/419320/
>     <https://review.openstack.org/#/c/419320/>
>     >
>     >
>     > --
>     > Regards,
>     > Ann Taraday
>     >
>     >
>     __________________________________________________________________________
>     > OpenStack Development Mailing List (not for usage questions)
>     > Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>     >
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>
>
> --
> Best regards,
> Proskurin Kirill
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>