Open Stack

Michał,

You are absolutely right: this exception is raised when you try to
lazy-load instance attributes outside a Session scope. There is an
obvious problem with that - instances do not communicate with a DB on
their own - it's left up to Session [1].

Unfortunately, it does not play nicely with the "classic" DB access
layer we have in Cinder and other projects, when you have a notion of
pluggable DB APIs and SQLAlchemy implementation that looks like:

@require_context
@handle_db_data_error
def snapshot_create(context, values):
    values['snapshot_metadata'] = _metadata_refs(values.get('metadata'),
                                                 models.SnapshotMetadata)
    if not values.get('id'):
        values['id'] = str(uuid.uuid4())

    session = get_session()
    with session.begin():
        snapshot_ref = models.Snapshot()
        snapshot_ref.update(values)
        session.add(snapshot_ref)

        return _snapshot_get(context, values['id'], session=session)

In this case a Session (and transaction) scope is bound to "public" DB
API functions. There are a few problems with this:

1) once a public DB function returns an instance, it becomes prone to
lazy-load errors, as the corresponding session (and DB transaction) is
already gone and it's not possible to load missing data (without
establishing a new session/transaction)

2) you have to carefully pass a Session object when doing calls to
"private" DB API functions to ensure they all participate in the very
same DB transaction. Otherwise snapshot_get() above would not see the
row created by snapshot_create() due to isolation of transactions in
RDBMS

3) if you do multiple calls to "public" DB API functions when handling
a single HTTP request it's not longer easy to do a rollback as every
function creates its own DB transaction

Mixing of Session objects creation with the actual business logic is
considered to be an anti-pattern in SQLAlchemy [2] due to problems
mentioned above.

At this point I suggest you take a look at [3] and start using in
Cinder: in Kilo we did a complete redesign of EngineFacade in oslo.db
- it won't solve all you problems with lazy-loading automatically, but
what it can do is provide a tool for declarative definition of session
(and transaction) scope, so that it's not longer limited to one
"public" DB API function and you can extend it when needed: you no
longer create a Session object explicitly, but rather mark methods
with a decorator, that will inject a session into the context, and all
callees will participate in the established session (thus, DB
transaction) rather than create a new one (my personal opinion is that
for web-services it's preferable to bind session/transaction scope to
the scope of one HTTP request, so that it's easy to roll back changes
on errors - we are not there yet, but some projects like Nova are
already moving the session scope up the stack, e.g. to objects layer).

Thanks,
Roman

[1] http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#what-does-the-session-do
[2] http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it
[3] https://specs.openstack.org/openstack/oslo-specs/specs/kilo/make-enginefacade-a-facade.html

On Thu, Sep 22, 2016 at 4:45 PM, Michał Dulko <michal.dulko at intel.com> wrote:
> Hi,
>
> I've just noticed another Cinder bug [1], similar to past bugs [2], [3].
> All of them have a common exception causing them:
>
> sqlalchemy.orm.exc.DetachedInstanceError: Parent instance
> <{$SQLAlchemyObject} at {$MemoryLocation}> is not bound to a Session;
> lazy load operation of attribute '{$ColumnName}' cannot proceed
>
> We've normally fixed them by simply making the $ColumnName eager-loaded,
> but as there's another similar bug report, I'm starting to think that we
> have some issue with how we're managing our DB connections and
> SQLAlchemy objects are losing their sessions too quickly, before we'll
> manage to lazy-load required stuff.
>
> I'm not too experienced with SQLAlchemy session management, so I would
> welcome any help with investigation.
>
> Thanks,
> Michal
>
>
> [1] https://bugs.launchpad.net/cinder/+bug/1626499
> [2] https://bugs.launchpad.net/cinder/+bug/1517763
> [3] https://bugs.launchpad.net/cinder/+bug/1501838
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev