Open Stack

On 09/30/2016 10:54 AM, Roman Podoliaka wrote:
> Michał,
>
> You are absolutely right: this exception is raised when you try to
> lazy-load instance attributes outside a Session scope. There is an
> obvious problem with that - instances do not communicate with a DB on
> their own - it's left up to Session [1].
>
> Unfortunately, it does not play nicely with the "classic" DB access
> layer we have in Cinder and other projects, when you have a notion of
> pluggable DB APIs and SQLAlchemy implementation that looks like:
>
> @require_context
> @handle_db_data_error
> def snapshot_create(context, values):
>     values['snapshot_metadata'] = _metadata_refs(values.get('metadata'),
>                                                  models.SnapshotMetadata)
>     if not values.get('id'):
>         values['id'] = str(uuid.uuid4())
>
>     session = get_session()
>     with session.begin():
>         snapshot_ref = models.Snapshot()
>         snapshot_ref.update(values)
>         session.add(snapshot_ref)
>
>         return _snapshot_get(context, values['id'], session=session)
>
> In this case a Session (and transaction) scope is bound to "public" DB
> API functions. There are a few problems with this:
>
> 1) once a public DB function returns an instance, it becomes prone to
> lazy-load errors, as the corresponding session (and DB transaction) is
> already gone and it's not possible to load missing data (without
> establishing a new session/transaction)
>
> 2) you have to carefully pass a Session object when doing calls to
> "private" DB API functions to ensure they all participate in the very
> same DB transaction. Otherwise snapshot_get() above would not see the
> row created by snapshot_create() due to isolation of transactions in
> RDBMS
>
> 3) if you do multiple calls to "public" DB API functions when handling
> a single HTTP request it's not longer easy to do a rollback as every
> function creates its own DB transaction
>
> Mixing of Session objects creation with the actual business logic is
> considered to be an anti-pattern in SQLAlchemy [2] due to problems
> mentioned above.
>
> At this point I suggest you take a look at [3] and start using in
> Cinder: in Kilo we did a complete redesign of EngineFacade in oslo.db
> - it won't solve all you problems with lazy-loading automatically, but
> what it can do is provide a tool for declarative definition of session
> (and transaction) scope, so that it's not longer limited to one
> "public" DB API function and you can extend it when needed: you no
> longer create a Session object explicitly, but rather mark methods
> with a decorator, that will inject a session into the context, and all
> callees will participate in the established session (thus, DB
> transaction) rather than create a new one (my personal opinion is that
> for web-services it's preferable to bind session/transaction scope to
> the scope of one HTTP request, so that it's easy to roll back changes
> on errors - we are not there yet, but some projects like Nova are
> already moving the session scope up the stack, e.g. to objects layer).


+1 thanks Roman !


>
> Thanks,
> Roman
>
> [1] http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#what-does-the-session-do
> [2] http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it
> [3] https://specs.openstack.org/openstack/oslo-specs/specs/kilo/make-enginefacade-a-facade.html
>
> On Thu, Sep 22, 2016 at 4:45 PM, Michał Dulko <michal.dulko at intel.com> wrote:
>> Hi,
>>
>> I've just noticed another Cinder bug [1], similar to past bugs [2], [3].
>> All of them have a common exception causing them:
>>
>> sqlalchemy.orm.exc.DetachedInstanceError: Parent instance
>> <{$SQLAlchemyObject} at {$MemoryLocation}> is not bound to a Session;
>> lazy load operation of attribute '{$ColumnName}' cannot proceed
>>
>> We've normally fixed them by simply making the $ColumnName eager-loaded,
>> but as there's another similar bug report, I'm starting to think that we
>> have some issue with how we're managing our DB connections and
>> SQLAlchemy objects are losing their sessions too quickly, before we'll
>> manage to lazy-load required stuff.
>>
>> I'm not too experienced with SQLAlchemy session management, so I would
>> welcome any help with investigation.
>>
>> Thanks,
>> Michal
>>
>>
>> [1] https://bugs.launchpad.net/cinder/+bug/1626499
>> [2] https://bugs.launchpad.net/cinder/+bug/1517763
>> [3] https://bugs.launchpad.net/cinder/+bug/1501838
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>