Open Stack

On Sep 12, 2014, at 7:20 AM, Sean Dague <sean at dague.net> wrote:

> 
> Because we are in Feature Freeze. Now is the time for critical bug fixes
> only, as we start to stabalize the tree. Releasing dependent libraries
> that can cause breaks, for whatever reason, should be soundly avoided.
> 
> If this was August, fine. But it's feature freeze.

I agree with this, changing the MySQL driver now is not an option.    That train has left the station, I think it’s better we all take the whole Kilo cycle to get used to mysql-connector and its quirks before launching it on the world, as there will be many more.

However for Kilo, I think those “COMMIT” phrases should be removed and overall we need to make a very hard and fast rule that we *do not put multiple statements in an execute*.    I’ve seen a bunch of these come through so far, and for some of them (more the in-Python ones) it seems like the underlying reason is a lack of understanding of what exactly a SQLAlchemy “Engine” is and what features it supports.

So first, let me point folks to the documentation for this, which anyone writing code involving Engine objects should read first:

http://docs.sqlalchemy.org/en/rel_0_9/core/connections.html

Key to this is that while engine supports an “.execute()” method, in order to do anything that intends to work on a single connection and typically a single transaction, you procure a Connection and usually a Transaction from the Engine, most easily like this:

with engine.begin() as conn:
   conn.execute(statement 1)
   conn.execute(statement 2)
   conn.execute(statement 3)
   .. etc


Now let me apologize for the reason this misunderstanding exists in the first place:  it’s because in 2005 I put the “.execute()” convenience method on the Engine itself (well in fact we didn’t have the Engine/Connection dichotomy back then), and I also thought that “implicit execution”, e.g. statement.execute(), would be a great idea.    Tons of other people still think it’s a great idea and even though I’ve buried this whole thing in the docs, they still use it like candy….until they have the need to control the scope of connectivity.  

*Huge* mistake, it’s my fault, but not something that can really be changed now.   Also, in 2005, Python didn’t have context managers.    So we have all kinds of klunky patterns like “trans = conn.begin()”, kind of J2EE style, etc., but these days, the above pattern is your best bet when you want to invoke multiple statements.    engine.execute() overall should just be avoided as it only leads to misunderstanding.   When we all move all of our migrate stuff to Alembic, there won’t be an Engine provided to a migration script, it will be a Connection to start with.