[openstack-dev] [Neutron] Fix to agents race condition creates another issue

Roman Podoliaka rpodolyaka at mirantis.com
Thu Dec 12 10:22:36 UTC 2013


Hi,

So actually these two are different issues:
1) https://review.openstack.org/#/c/61677 fixes the issue with a few
mandatory migrations not applying when ML2 plugin is used (the only
reason we hadn't seen it before is that those migrations were applied
on Jenkins CI nodes because LBaaS service plugin was used. Eugene,
thanks for pointing this out!)
2) https://review.openstack.org/#/c/61663 fixes the issue of
https://review.openstack.org/#/c/58814/ not working properly for core
plugins which don't use agents extension (e.g bigswitch, plumgrid)

Roman

On Thu, Dec 12, 2013 at 11:13 AM, Edgar Magana <emagana at plumgrid.com> wrote:
> Ok, I just did it.
> https://bugs.launchpad.net/neutron/+bug/1260232
>
> It seems that Yong found the same problem for ML2:
> https://bugs.launchpad.net/bugs/1260224
>
> I am just wondering, how come we did not catch this with ML2 when we are
> claiming that it is our default plugin.
>
> Edgar
>
>
>
> On Wed, Dec 11, 2013 at 10:38 PM, Eugene Nikanorov <enikanorov at mirantis.com>
> wrote:
>>
>> Hi, Edgar,
>>
>> Please file a bug on this. Obviously this was introduced by that commit.
>>
>> Thanks,
>> Eugene.
>>
>>
>> On Thu, Dec 12, 2013 at 7:45 AM, Edgar Magana <emagana at plumgrid.com>
>> wrote:
>>>
>>> In commit:
>>> https://review.openstack.org/#/c/58814/
>>>
>>> There is an assumption that all "plugins" creates the
>>> plumgrid_neutron.agents which is not the case. I just tested big switch and
>>> plumgrid and they are failing:
>>>
>>> INFO  [alembic.migration] Running upgrade havana -> e197124d4b9, add
>>> unique constraint to members
>>> INFO  [alembic.migration] Running upgrade e197124d4b9 -> 1fcfc149aca4,
>>> Add a unique constraint on (agent_type, host) columns to prevent a race
>>> condition when an agent entry is 'upserted'.
>>> Traceback (most recent call last):
>>>   File "/usr/local/bin/neutron-db-manage", line 10, in <module>
>>>     sys.exit(main())
>>>   File "/opt/stack/neutron/neutron/db/migration/cli.py", line 143, in
>>> main
>>>     CONF.command.func(config, CONF.command.name)
>>>   File "/opt/stack/neutron/neutron/db/migration/cli.py", line 80, in
>>> do_upgrade_downgrade
>>>     do_alembic_command(config, cmd, revision, sql=CONF.command.sql)
>>>   File "/opt/stack/neutron/neutron/db/migration/cli.py", line 59, in
>>> do_alembic_command
>>>     getattr(alembic_command, cmd)(config, *args, **kwargs)
>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/command.py", line
>>> 124, in upgrade
>>>     script.run_env()
>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/script.py", line
>>> 193, in run_env
>>>     util.load_python_file(self.dir, 'env.py')
>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line
>>> 177, in load_python_file
>>>     module = load_module(module_id, path)
>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py", line
>>> 39, in load_module
>>>     return imp.load_source(module_id, path, fp)
>>>   File
>>> "/opt/stack/neutron/neutron/db/migration/alembic_migrations/env.py", line
>>> 105, in <module>
>>>     run_migrations_online()
>>>   File
>>> "/opt/stack/neutron/neutron/db/migration/alembic_migrations/env.py", line
>>> 89, in run_migrations_online
>>>     options=build_options())
>>>   File "<string>", line 7, in run_migrations
>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/environment.py",
>>> line 652, in run_migrations
>>>     self.get_context().run_migrations(**kw)
>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/migration.py",
>>> line 224, in run_migrations
>>>     change(**kw)
>>>   File
>>> "/opt/stack/neutron/neutron/db/migration/alembic_migrations/versions/1fcfc149aca4_agents_unique_by_type_and_host.py",
>>> line 50, in upgrade
>>>     local_cols=['agent_type', 'host']
>>>   File "<string>", line 7, in create_unique_constraint
>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/operations.py",
>>> line 539, in create_unique_constraint
>>>     schema=schema, **kw)
>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/ddl/impl.py", line
>>> 135, in add_constraint
>>>     self._exec(schema.AddConstraint(const))
>>>   File "/usr/local/lib/python2.7/dist-packages/alembic/ddl/impl.py", line
>>> 76, in _exec
>>>     conn.execute(construct, *multiparams, **params)
>>>   File
>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
>>> 1449, in execute
>>>     params)
>>>   File
>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
>>> 1542, in _execute_ddl
>>>     compiled
>>>   File
>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
>>> 1698, in _execute_context
>>>     context)
>>>   File
>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
>>> 1691, in _execute_context
>>>     context)
>>>   File
>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line
>>> 331, in do_execute
>>>     cursor.execute(statement, parameters)
>>>   File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174,
>>> in execute
>>>     self.errorhandler(self, exc, value)
>>>   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line
>>> 36, in defaulterrorhandler
>>>     raise errorclass, errorvalue
>>> sqlalchemy.exc.ProgrammingError: (ProgrammingError) (1146, "Table
>>> 'plumgrid_neutron.agents' doesn't exist") 'ALTER TABLE agents ADD CONSTRAINT
>>> uniq_agents0agent_type0host UNIQUE (agent_type, host)' ()
>>> ++ failed
>>> ++ local r=1
>>> +++ jobs -p
>>> ++ kill
>>> ++ set +o xtrace
>>>
>>>
>>>
>>> Is this known issues? If not, let me know and I can properly reported.
>>>
>>> Thanks,
>>>
>>> Edgar
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



More information about the OpenStack-dev mailing list