Open Stack

+1 for trying things differently :)

On 8/19/13 12:14 AM, "Jay Pipes" <jaypipes at gmail.com> wrote:

>On 08/18/2013 10:33 PM, Joe Gordon wrote:
>>         An alternative I think would be better would be to scrap the
>>use of
>>         the SQLAlchemy ORM; keep using the DB engine abstraction
>>support.
>>
>> +1, I am hoping this will provide noticeable performance benefits while
>> being agnostic of what DB back-end is being used.  With the way we use
>>   SQLALchemy being 25x slower then MySQL we have lots of room for
>> improvement (see http://paste.openstack.org/show/44143/ from
>> https://bugs.launchpad.net/nova/+bug/1212418).
>
>@require_admin_context
>def compute_node_get_all(context):
>     return model_query(context, models.ComputeNode).\
>             options(joinedload('service')).\
>             options(joinedload('stats')).\
>             all()
>
>Well, yeah... I suppose if you are attempting to create 115K objects in
>memory in Python (Need to collate each ComputeNode model object and each
>of its relation objects for Service and Stats) you are going to run into
>some performance problems. :)
>
>Would be interesting to see what the performance difference would be if
>you instead had dicts instead of model objects and did something like
>this instead (code not tested, just off top of head...):
>
># Assume a method to_dict() that takes a Model
># and returns a dict with appropriate empty dicts for
># relationship fields.
>
>qr = session.query(ComputeNode).join(Service).join(Stats)
>
>results = {}
>
>for record in qr:
>   node_id = record.ComputeNode.id
>   service_id = record.Service.id
>   stat_id = record.ComputeNodeStat.id
>   if node_id not in results.keys():
>     results[node_id] = to_dict(record.ComputeNode)
>   if service_id not in results[node_id]['services'].keys():
>     results[node_id]['services'][service_id] = to_dict(record.Service)
>   if stat_id not in results[node_id]['stats'].keys():
>     results[node_id]['stats'][stat_id] = to_dict(record.ComputeNodeStat)
>
>return results
>
>Whether it would be any faster than SQLAlchemy's joinedload...
>
>Besides that, though, probably is a good idea to look at even the
>existence of DB calls that potentially do that kind of massive query
>returning as A Bad Thing...
>
>Best,
>-jay
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev