Open Stack

Hi Neutrinos,
As we are progressing with object implementation for neutron database resources, I would like to summarize the status.

Please find the etherpad as up-to-date work status on object progress, TODOs/In progress/Done items:
https://etherpad.openstack.org/p/newton-ovo-progress
This etherpad should collect all the work that need to be done to address the objectification. Please add any items that you see are missing.

1. Object implementation
Since last report, we have manage to merge following patches:
- Allow unique keys to be used with get_object https://review.openstack.org/322024
- objects: loading synthetic fields from defined ORM relationships. https://review.openstack.org/334380
- objects: better apply filters for objects/db/api/get_object query. https://review.openstack.org/334381
- objects: Add update_fields method in base class. https://review.openstack.org/337539
- objects: Add RBAC to Subnet OVO https://review.openstack.org/337634
- objects: Adjust Subnet fields, add tenant_id and segment_id https://review.openstack.org/331009
- objects: introduce NetworkPortSecurity object https://review.openstack.org/327257
- trunk: avoid redundant refetch of subports on create https://review.openstack.org/348396
- tests: enable test_get_objects_queries_constant for trunk ports https://review.openstack.org/348378

Please check for following patches that are still under review:
- Introduces ovo objects for security groups https://review.openstack.org/284738
- objects: forbid updates for project_id field for subnets https://review.openstack.org/#/c/347787
- SUPER WIP OVO port object https://review.openstack.org/253641
- objects: expose database model for NeutronDbObject instances https://review.openstack.org/#/c/348279
- objects: remove support for multiple db models in from_db_object https://review.openstack.org/#/c/348271
- WIP: get_subnet(s), update, delete converted in db_base_plugin - https://review.openstack.org/#/c/321001/
- devref: docs about how to use NeutronDbObject. https://review.openstack.org/336518
- Print out specific filter that failed in object filtering unit test https://review.openstack.org/#/c/347884/


2. About object integration in neutron base code:
Tenant_id to project_id approach in objects:
For object implementation, we expose both project_id and tenant_id, but only the project_id is regular field in object 'fields' property. The tenant_id is declared (in automated manner for every object that has project_id [3]) as read-only value, available in to_dict() and in dictionary access to the objects fields. It does not appear in obj_to_primitive() method, so for example the tenant_id would not be sent over the RPC callback wire.
Currently, we are performing the automated translation from tenant_id to project_id when fetching the data from DB [4]. When the tenant_id column would be renamed into project_id, it would not have any effect on object implementation, the [4] would be not used and can be removed, but from object user perspective, nothing will change. There still will be tenant_id access [3] so we will be backward compatible.
When talking about REST API to OVO passing filters, we should be able to handle both tenant_id and project_id for filtering. Current approach is to translate the tenant_id to project_id in filters before passing it to get_objects() method [5][6]

REST API filtering:
I have encounter the problem when passing any random string as filters from REST API to get_objects OVO implementation [7]. At object layer, we do not accept unknown filters and errors out. The general approach should be to handle it at API level, to pass to plugins only valid filters. Current way to handle it is to remove all unknown filters in DB layer, just before passing the filters to get_objects() (work TBD).

Extensions and object implementation:
Replacing direct access to DB by OVO means that every extension fields should be declared in object definition, in order to include the data in the object structure and fetch the data from DB . In-tree extensions can be done in using OVO, like synthetic field, field that is loaded from other place in DB than current object DB model.
That situation can be extremely difficult for out of tree plugins/drivers that are extending the neutron resource like port, subnet, network. Since dropping the support for them is not possible, we will have DB model as an object field and we will pass it in make_<resource>_dict and extension functions, just like it is done today. [10][11]

Returning all object fields for QoS and trunk port in REST API:
Current implementation for QoS and trunk port API is returning all object fields in REST API [8][9]. We need to limit the info return to the API consumer, because the internal data is leaking like revision number of DB state. This work is TBD.

Circular import issue:
To break the circular dependency when implementing the OVO in db classes that are having the model and mixin in the same file, we have started the effort to refactor the code. [1] ML discussion selected the option to move the models into neutron/db/models/<name>.py
Also it would be good to add the debt collector like in [2].



===
Team info:
Upgrades Subteam has the weekly meetings on Mondays, 3PM UTC, wiki page: https://wiki.openstack.org/wiki/Meetings/Neutron-Upgrades-Subteam

New patches are generally tracked under the following topic: https://review.openstack.org/#/q/topic:bp/adopt-oslo-versioned-objects-for-db

Cheers,
Artur

[1] http://lists.openstack.org/pipermail/openstack-dev/2016-July/100276.html
[2] https://review.openstack.org/#/c/330870/13/neutron/db/agents_db.py@88
[3] https://review.openstack.org/#/c/331009/8/neutron/objects/base.py@210
[4] https://review.openstack.org/#/c/331009/8/neutron/objects/subnet.py@177
[5] https://review.openstack.org/#/c/321001/13/neutron/db/db_base_plugin_common.py@278
[6] https://review.openstack.org/#/c/284738/49/neutron/db/securitygroups_db.py@65
[7] http://lists.openstack.org/pipermail/openstack-dev/2016-July/100286.html
[8] https://github.com/openstack/neutron/blob/9.0.0.0b1/neutron/services/qos/qos_plugin.py#L110
[9] https://github.com/openstack/neutron/blob/master/neutron/services/trunk/plugin.py#L101
[10] https://review.openstack.org/348279
[11] https://review.openstack.org/348478