[openstack-dev] [Openstack] Errors in neutron-server while launching VM

Hinds, Luke (Nokia - GB/Bristol) luke.hinds at nokia.com
Fri Oct 16 15:17:30 UTC 2015


On Fri, 2015-10-16 at 20:04 +0530, EXT Rahul Arora wrote:
HI Nitish,

Actually i am running openstack kilo on my powerpc platform now.I am able to run it on the x86 successfully.

On the running setup i.e on x86 machine also i am having same output of the status command in neutron database,See below.

mysql> use neutron
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> status
--------------
mysql  Ver 14.14 Distrib 5.5.44, for debian-linux-gnu (i686) using readline 6.3

Connection id:        381
Current database:    neutron
Current user:        root at localhost
SSL:            Not in use
Current pager:        stdout
Using outfile:        ''
Using delimiter:    ;
Server version:        5.5.44-0ubuntu0.14.04.1 (Ubuntu)
Protocol version:    10
Connection:        127.0.0.1 via TCP/IP
Server characterset:    latin1
Db     characterset:    utf8
Client characterset:    utf8
Conn.  characterset:    utf8
TCP port:        3306
Uptime:            1 day 3 hours 52 min 27 sec

Threads: 12  Questions: 576409  Slow queries: 0  Opens: 1600  Flush tables: 1  Open tables: 374  Queries per second avg: 5.744


I am wondering what is the issue in powerpc.So i guess  cause is i am not able to run below command successfully on the powerpc.


neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head

But this command is successful in x86 machine.

On powerpc i am getting below error while running it,

raise errorclass, errorvalueegards
sqlalchemy.exc.OperationalError: (OperationalError) (1005, "Can't create table 'neutron.nuage_routerroutes_mapping' (errno: 121)") '\nCREATE TA
BLE nuage_routerroutes_mapping (\n\tdestination VARCHAR(64) NOT NULL, \n\tnexthop VARCHAR(64) NOT NULL, \n\trouter_id VARCHAR(36) NOT NULL, \n\
tnuage_route_id VARCHAR(36), \n\tPRIMARY KEY (destination, nexthop, router_id), \n\tFOREIGN KEY(router_id) REFERENCES routers (id) ON DELETE CA
SCADE\n)ENGINE=InnoDB\n\n' ()


I am not able to solve this.Please help me on this,

..

Regards

Rahul Arora


How are you installing juno? Is it a packaged version (devstack  / ansible etc) or manually? It might be that you have MariaDB compiled for x86 on there (different instruction sets).


On Fri, Oct 16, 2015 at 7:56 PM, nithish B <bestofnithish at gmail.com<mailto:bestofnithish at gmail.com>> wrote:
Rahul,
Clearly, neutron didn't create the tables right. You may have to re-install/re-configure neutron.

Thanks.

Regards,
Nitish B.

On Fri, Oct 16, 2015 at 7:55 PM, Rahul Arora <rahul1991.arora at gmail.com<mailto:rahul1991.arora at gmail.com>> wrote:
Hi Nitish.

Below is the output of status of neutron database.Yes i am using "use neutron" before running the command.

MariaDB [neutron]> status
--------------
mysql  Ver 15.1 Distrib 5.5.39-MariaDB, for Linux (powerpc) using readline 5.1

Connection id:          42
Current database:       neutron
Current user:           root at localhost
SSL:                    Not in use
Current pager:          stdout
Using outfile:          ''
Using delimiter:        ;
Server:                 MariaDB
Server version:         5.5.39-MariaDB Source distribution
Protocol version:       10
Connection:             Localhost via UNIX socket
Server characterset:    utf8
Db     characterset:    utf8
Client characterset:    latin1
Conn.  characterset:    latin1
UNIX socket:            /var/lib/mysql/mysql.sock
Uptime:                 1 hour 15 min 15 sec

Threads: 11  Questions: 24713  Slow queries: 5  Opens: 164  F

On Fri, Oct 16, 2015 at 7:51 PM, nithish B <bestofnithish at gmail.com<mailto:bestofnithish at gmail.com>> wrote:
Rahul,
Please check the version of MySQL/MariaDB which you are using. For quick reference, you can use the following command in mysql: "SHOW VARIABLES LIKE "%version%";"

Also, before you run "show tables", you need to select the DB to use, using "use <db_name>".

Thanks.


Regards,
Nitish B.

On Fri, Oct 16, 2015 at 6:41 PM, Rahul Arora <rahul1991.arora at gmail.com<mailto:rahul1991.arora at gmail.com>> wrote:
Hi Nasir,

While doing installation/configuration when i was trying to run neutron-server service.I got following errors.

2015-10-05 06:48:12.549 17209 TRACE neutron   File "/usr/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-10-05 06:48:12.549 17209 TRACE neutron     raise errorclass, errorvalue
2015-10-05 06:48:12.549 17209 TRACE neutron ProgrammingError: (ProgrammingError) (1146, "Table 'neutron.ml2_gre_allocations' doesn't exist") 'SELECT ml2_gre_allocations.gre_id AS ml2_gre_allocations_gre_id, ml2_gre_allocations.allocated AS ml2_gre_allocations_allocated \nFROM ml2_gre_allocations' ()
2015-10-05 06:48:12.549 17209 TRACE neutron

To fix this from internet i got one command to upgrade database.Below is the command i used.

$ neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head

INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> havana, havana_initial
INFO  [alembic.runtime.migration] Running upgrade havana -> e197124d4b9, add unique constraint to members
/usr/lib/python2.7/site-packages/alembic/util/langhelpers.py:122: UserWarning: Argument 'name' is now named 'constraint_name' for method create
_unique_constraint().
  oldname, newname, fn_name
/usr/lib/python2.7/site-packages/alembic/util/langhelpers.py:122: UserWarning: Argument 'source' is now named 'table_name' for method create_un
ique_constraint().
  oldname, newname, fn_name
/usr/lib/python2.7/site-packages/alembic/util/langhelpers.py:122: UserWarning: Argument 'local_cols' is now named 'columns' for method create_u
nique_constraint().
  oldname, newname, fn_name
INFO  [alembic.runtime.migration] Running upgrade e197124d4b9 -> 1fcfc149aca4, Add a unique constraint on (agent_type, host) columns to prevent
 a race
condition when an agent entry is 'upserted'.
INFO  [alembic.runtime.migration] Running upgrade 1fcfc149aca4 -> 50e86cb2637a, nsx_mappings
INFO  [alembic.runtime.migration] Running upgrade 50e86cb2637a -> 1421183d533f, NSX DHCP/metadata support
INFO  [alembic.runtime.migration] Running upgrade 1421183d533f -> 3d3cb89d84ee, nsx_switch_mappings
INFO  [alembic.runtime.migration] Running upgrade 3d3cb89d84ee -> 4ca36cfc898c, nsx_router_mappings
INFO  [alembic.runtime.migration] Running upgrade 4ca36cfc898c -> 27cc183af192, ml2_vnic_type
INFO  [alembic.runtime.migration] Running upgrade 27cc183af192 -> 50d5ba354c23, ml2 binding:vif_details
INFO  [alembic.runtime.migration] Running upgrade 50d5ba354c23 -> 157a5d299379, ml2 binding:profile
INFO  [alembic.runtime.migration] Running upgrade 157a5d299379 -> 3d2585038b95, VMware NSX rebranding
INFO  [alembic.runtime.migration] Running upgrade 3d2585038b95 -> abc88c33f74f, lb stats
INFO  [alembic.runtime.migration] Running upgrade abc88c33f74f -> 1b2580001654, nsx_sec_group_mapping
INFO  [alembic.runtime.migration] Running upgrade 1b2580001654 -> e766b19a3bb, nuage_initial
INFO  [alembic.runtime.migration] Running upgrade e766b19a3bb -> 2eeaf963a447, floatingip_status
INFO  [alembic.runtime.migration] Running upgrade 2eeaf963a447 -> 492a106273f8, Brocade ML2 Mech. Driver
INFO  [alembic.runtime.migration] Running upgrade 492a106273f8 -> 24c7ea5160d7, Cisco CSR VPNaaS
INFO  [alembic.runtime.migration] Running upgrade 24c7ea5160d7 -> 81c553f3776c, bsn_consistencyhashes
INFO  [alembic.runtime.migration] Running upgrade 81c553f3776c -> 117643811bca, nec: delete old ofc mapping tables
INFO  [alembic.runtime.migration] Running upgrade 117643811bca -> 19180cf98af6, nsx_gw_devices
INFO  [alembic.runtime.migration] Running upgrade 19180cf98af6 -> 33dd0a9fa487, embrane_lbaas_driver
INFO  [alembic.runtime.migration] Running upgrade 33dd0a9fa487 -> 2447ad0e9585, Add IPv6 Subnet properties
INFO  [alembic.runtime.migration] Running upgrade 2447ad0e9585 -> 538732fa21e1, NEC Rename quantum_id to neutron_id
INFO  [alembic.runtime.migration] Running upgrade 538732fa21e1 -> 5ac1c354a051, n1kv segment allocs for cisco n1kv plugin
/usr/lib/python2.7/site-packages/alembic/util/langhelpers.py:122: UserWarning: Argument 'source' is now named 'source_table' for method create_
foreign_key().
  oldname, newname, fn_name
/usr/lib/python2.7/site-packages/alembic/util/langhelpers.py:122: UserWarning: Argument 'referent' is now named 'referent_table' for method cre
ate_foreign_key().
  oldname, newname, fn_name
INFO  [alembic.runtime.migration] Running upgrade 5ac1c354a051 -> icehouse, icehouse
INFO  [alembic.runtime.migration] Running upgrade icehouse -> 54f7549a0e5f, set_not_null_peer_address
INFO  [alembic.runtime.migration] Running upgrade 54f7549a0e5f -> 1e5dd1d09b22, set_not_null_fields_lb_stats
INFO  [alembic.runtime.migration] Running upgrade 1e5dd1d09b22 -> b65aa907aec, set_length_of_protocol_field
INFO  [alembic.runtime.migration] Running upgrade b65aa907aec -> 33c3db036fe4, set_length_of_description_field_metering
INFO  [alembic.runtime.migration] Running upgrade 33c3db036fe4 -> 4eca4a84f08a, Remove ML2 Cisco Credentials DB
INFO  [alembic.runtime.migration] Running upgrade 4eca4a84f08a -> d06e871c0d5, set_admin_state_up_not_null_ml2
INFO  [alembic.runtime.migration] Running upgrade d06e871c0d5 -> 6be312499f9, set_not_null_vlan_id_cisco
INFO  [alembic.runtime.migration] Running upgrade 6be312499f9 -> 1b837a7125a9, Cisco APIC Mechanism Driver
INFO  [alembic.runtime.migration] Running upgrade 1b837a7125a9 -> 10cd28e692e9, nuage_extraroute
INFO  [alembic.runtime.migration] Running upgrade 10cd28e692e9 -> 2db5203cb7a9, nuage_floatingip
INFO  [alembic.runtime.migration] Running upgrade 2db5203cb7a9 -> 5446f2a45467, set_server_default
INFO  [alembic.runtime.migration] Running upgrade 5446f2a45467 -> db_healing, Include all tables and make migrations unconditional.
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.autogenerate.compare] Detected server default on column 'cisco_ml2_apic_epgs.provider'
INFO  [alembic.autogenerate.compare] Detected server default on column 'cisco_n1kv_vxlan_allocations.allocated'
INFO  [alembic.autogenerate.compare] Detected server default on column 'cisco_network_profiles.multicast_ip_index'
INFO  [alembic.autogenerate.compare] Detected server default on column 'meteringlabelrules.excluded'
INFO  [alembic.autogenerate.compare] Detected server default on column 'ml2_port_bindings.host'
INFO  [alembic.autogenerate.compare] Detected removed foreign key (router_id)(id) on table nuage_floatingip_pool_mapping
INFO  [alembic.autogenerate.compare] Detected added column 'nuage_routerroutes_mapping.destination'
INFO  [alembic.autogenerate.compare] Detected added column 'nuage_routerroutes_mapping.nexthop'
INFO  [alembic.autogenerate.compare] Detected server default on column 'poolmonitorassociations.status'
INFO  [alembic.autogenerate.compare] Detected added index 'ix_quotas_tenant_id' on '['tenant_id']'
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 238, in main
    CONF.command.func(config, CONF.command.name<http://CONF.command.name>)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 106, in do_upgrade
    do_alembic_command(config, cmd, revision, sql=CONF.command.sql)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 72, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/alembic/command.py", line 174, in upgrade
    script.run_env()
  File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 397, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 109, in <module>
    run_migrations_online()
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 100, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/lib/python2.7/site-packages/alembic/runtime/environment.py", line 797, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/lib/python2.7/site-packages/alembic/runtime/migration.py", line 312, in run_migrations
    step.migration_fn(**kw)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/versions/1d6ee1ae5da5_db_healing.py", line 32, in upgrade
    heal_script.heal()
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/heal_script.py", line 76, in heal
    execute_alembic_command(el)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/heal_script.py", line 88, in execute_alembic_command
    METHODS[command[0]](*command[1:])
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/heal_script.py", line 173, in add_column
    op.add_column(table_name, column.copy(), schema=schema)
  File "<string>", line 8, in add_column
  File "<string>", line 3, in add_column
  File "/usr/lib/python2.7/site-packages/alembic/operations/ops.py", line 1535, in add_column
    return operations.invoke(op)
  File "/usr/lib/python2.7/site-packages/alembic/operations/base.py", line 318, in invoke
    return fn(self, operation)
  File "/usr/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 123, in add_column
    schema=schema
  File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 172, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 841, in execute
    return meth(self, multiparams, params)
  File "/usr/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 69, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 895, in _execute_ddl
    compiled
  File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1070, in _execute_context
    context)
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 261, in _handle_dbapi_exception
    e, statement, parameters, cursor, context)
  File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1267, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/usr/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1063, in _execute_context
    context)
  File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python2.7/site-packages/MySQLdb/cursors.py", line 201, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
oslo_db.exception.DBConnectionError: (OperationalError) (2013, 'Lost connection to MySQL server during query') 'ALTER TABLE nuage_routerroutes_
mapping ADD COLUMN destination VARCHAR(64) NOT NULL' ()
root at t1040rdb:~#
 CTRL-A Z for help |115200 8N1 | NOR | Minicom 2.5    | VT102 |      Offline

I got errors after running this command but my error in RED part got solved after running this command.

Is this the cause of my error which i mentioned in another email?

Thanks for the help

..

Regards

Rahul Arora



On Fri, Oct 16, 2015 at 4:25 PM, Nasir Mahmood <nasir.mahmood at gmail.com<mailto:nasir.mahmood at gmail.com>> wrote:
Rahul,

You may need to re-create neutron DB...this error should never be seen in your logs if your installation/configuration is gone good,

On Fri, Oct 16, 2015 at 3:22 PM, nithish B <bestofnithish at gmail.com<mailto:bestofnithish at gmail.com>> wrote:
Hi Rahul,
Is the table "networks" available in the DB? Looks like it does not exist and thus when neutron queries the DB, it returns the error.
Let me know.

Thanks.
Nitish B.

Regards,
Nitish B.

On Fri, Oct 16, 2015 at 3:10 PM, Rahul Arora <rahul1991.arora at gmail.com<mailto:rahul1991.arora at gmail.com>> wrote:
Hi Team.

I am trying to run Openstack KILO release on my Ubuntu 14.04 x86 machine.I am able to run all the services.i.e glance,nova.keystone,neutron etc successfully.

But when i am trying to launch VM using below command manually.

nova boot --image 8f6e2973-c048-4445-8c31-38e159c930fc --flavor m1.tiny "rahul"

I am getting below error in neutron-server service.
=========================================


2015-10-06 02:29:53.894 8064 TRACE neutron.api.v2.resource     self.errorhandler(self, exc, value)
2015-10-06 02:29:53.894 8064 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-10-06 02:29:53.894 8064 TRACE neutron.api.v2.resource     raise errorclass, errorvalue
2015-10-06 02:29:53.894 8064 TRACE neutron.api.v2.resource OperationalError: (OperationalError) (1054, "Unknown column 'networks.mtu' in 'field list'") 'SELECT networks.tenant_id AS networks_tenant_id, networks.id<http://networks.id> AS networks_id, networks.name<http://networks.name> AS networks_name, networks.status AS networks_status, networks.admin_state_up AS networks_admin_state_up, networks.shared AS networks_shared, networks.mtu AS networks_mtu, networks.vlan_transparent AS networks_vlan_transparent, ipallocationpools_1.id<http://ipallocationpools_1.id> AS ipallocationpools_1_id, ipallocationpools_1.subnet_id AS ipallocationpools_1_subnet_id, ipallocationpools_1.first_ip AS ipallocationpools_1_first_ip, ipallocationpools_1.last_ip AS ipallocationpools_1_last_ip, dnsnameservers_1.address AS dnsnameservers_1_address, dnsnameservers_1.subnet_id AS dnsnameservers_1_subnet_id, subnetroutes_1.destination AS subnetroutes_1_destination, subnetroutes_1.nexthop AS subnetroutes_1_nexthop, subnetroutes_1.subnet_id AS subnetroutes_1_subnet_id, subnets_1.tenant_id AS subnets_1_tenant_id, subnets_1.id<http://subnets_1.id> AS subnets_1_id, subnets_1.name<http://subnets_1.name> AS subnets_1_name, subnets_1.network_id AS subnets_1_network_id, subnets_1.subnetpool_id AS subnets_1_subnetpool_id, subnets_1.ip_version AS subnets_1_ip_version, subnets_1.cidr AS subnets_1_cidr, subnets_1.gateway_ip AS subnets_1_gateway_ip, subnets_1.enable_dhcp AS subnets_1_enable_dhcp, subnets_1.shared AS subnets_1_shared, subnets_1.ipv6_ra_mode AS subnets_1_ipv6_ra_mode, subnets_1.ipv6_address_mode AS subnets_1_ipv6_address_mode, externalnetworks_1.network_id AS externalnetworks_1_network_id \nFROM networks LEFT OUTER JOIN externalnetworks ON networks.id<http://networks.id> = externalnetworks.network_id LEFT OUTER JOIN subnets AS subnets_1 ON networks.id<http://networks.id> = subnets_1.network_id LEFT OUTER JOIN ipallocationpools AS ipallocationpools_1 ON subnets_1.id<http://subnets_1.id> = ipallocationpools_1.subnet_id LEFT OUTER JOIN dnsnameservers AS dnsnameservers_1 ON subnets_1.id<http://subnets_1.id> = dnsnameservers_1.subnet_id LEFT OUTER JOIN subnetroutes AS subnetroutes_1 ON subnets_1.id<http://subnets_1.id> = subnetroutes_1.subnet_id LEFT OUTER JOIN externalnetworks AS externalnetworks_1 ON networks.id<http://networks.id> = externalnetworks_1.network_id \nWHERE networks.tenant_id IN (%s) AND networks.shared IN (%s)' ('75ee0e2b66e04287920904eb40351f57', 0)
2015-10-06 02:29:53.894 8064 TRACE neutron.api.v2.resource

NOTE: I am using ml2 plugins to run  the neutron-server service.

Please help me solving this issue.

Thanks for the help.


--

Regards


Rahul Arora

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack




--
Nasir Mahmood











__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151016/2eac1521/attachment-0001.html>


More information about the OpenStack-dev mailing list