Open Stack

Hi Tim,
You are great. It works! Thanks a lot!
I've also solved the problem with py27. The unit test seems to work.
The only thing that seems not to work is populate the 'clusters_links' and 'cluster_templates_links' tables: they are empty.
Also, the 'labels' table is empty.
I've no errors anyway.
Are these problems according to you?

Should I to try to add the translation of pods, containers and services?

I've add the code to review.

Ruben
----- Original Message -----
From: "Tim Hinrichs" <tim at styra.com>
To: "Ruben" <r.manganiello1 at studenti.unisa.it>
Cc: openstack-dev at lists.openstack.org, "timothy l hinrichs" <timothy.l.hinrichs at gmail.com>
Sent: Friday, November 25, 2016 10:36:29 PM
Subject: Re: [Congress] Magnum_driver

Hi Ruben,

Glad you got that worked out.  Once in a while I end up deleting my .tox
dir because it gets out of date.  I guess that's what the --recreate option
to tox does.  So I learned something.  :)

The new error message looks to be saying that an object of type 'Cluster'
cannot be iterated.  Congress's _translate_clusters method assumes that
what you give it is basically JSON (arrays, dictionaries, numbers, strings,
bools).   In this case Congress is trying to iterate over the keys of a
dictionary but is getting a 'Cluster' Python object instead.  What's
happening is that when you do the cluster.list() (or whatever it's called)
on the magnum-python you're getting back a list of Cluster Python objects
instead of a list of dictionaries.

To fix the problem, you'll want to take the results of cluster.list() and
translate it into a list of dictionaries, and then hand that off to
_translate_clusters.  Probably you'll need to do the same for the
cluster_template.  So instead of ...

        clusters_method = lambda: self._translate_clusters(
            {'clusters': self.magnum.clusters.list()})

You'll want to do something like ...

        clusters_method = lambda: self._translate_clusters(
            {'clusters': [x.__dict__ for x in self.magnum.clusters.list()]
 })

Or at least, that's what I'd start trying.  __dict__ grabs all the fields
of an object and returns a dictionary.  (If some of the values of that
dictionary are also Python objects, then you might need something that's
more complicated to recursively walk the result of clusters.list() and
translate everything into lists and dictionaries.  And I'm not sure that
__dict__ will give you exactly what you want, but it's worth a try.).

Tim


On Thu, Nov 24, 2016 at 12:11 PM Ruben <r.manganiello1 at studenti.unisa.it>
wrote:

> Hi Tim,
> I solved the problem with:
>
> tox --recreate -e py27
>
> Now I no have the error on the import. The unit test still has some
> errors, but I don't know why..
>
> Anyway, I've this error when I try to add the magnum_driver:
>
> 2016-11-24 20:56:27.191 INFO congress.datasources.datasource_driver [-]
> magnum:: polling
> 2016-11-24 20:56:27.192 DEBUG congress.datasources.datasource_driver [-]
> update table clusters. from (pid=18720) update_from_datasource
> /opt/stack/congress/congress/datasources/datasource_driver.py:1370
> 2016-11-24 20:56:27.427 DEBUG congress.datasources.magnum_driver [-]
> CLUSTERS: {'clusters': [<Clusters {u'status': u'CREATE_FAILED',
> u'cluster_template_id': u'8d25a1ed-faa6-4305-a6a1-6559708c805b', u'uuid':
> u'1634beb9-25de-4cdd-bafa-67537069f0cc', u'links': [{u'href': u'
> http://10.0.2.15:9511/v1/clusters/1634beb9-25de-4cdd-bafa-67537069f0cc',
> u'rel': u'self'}, {u'href': u'
> http://10.0.2.15:9511/clusters/1634beb9-25de-4cdd-bafa-67537069f0cc',
> u'rel': u'bookmark'}], u'stack_id':
> u'17f1248d-286a-4fd4-9639-af5773670f03', u'master_count': 1, u'keypair':
> u'testkey', u'node_count': 1, u'create_timeout': 60, u'name':
> u'k8s-cluster'}>]} from (pid=18720) _translate_clusters
> /opt/stack/congress/congress/datasources/magnum_driver.py:165
> 2016-11-24 20:56:27.435 ERROR congress.datasources.datasource_driver [-]
> Datasource driver raised exception
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> Traceback (most recent call last):
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  File "/opt/stack/congress/congress/datasources/datasource_driver.py", line
> 1384, in poll
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  self.update_from_datasource()  # sets self.state
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  File "/opt/stack/congress/congress/datasources/datasource_driver.py", line
> 1371, in update_from_datasource
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  self.update_methods[registered_table]()
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  File "/opt/stack/congress/congress/datasources/magnum_driver.py", line
> 150, in <lambda>
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  {'clusters': self.magnum.clusters.list()})
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  File "/opt/stack/congress/congress/datasources/datasource_utils.py", line
> 57, in inner
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  result = f(self, raw_data, *args, **kw)
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  File "/opt/stack/congress/congress/datasources/magnum_driver.py", line
> 167, in _translate_clusters
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  obj['clusters'], MagnumDriver.clusters_translator)
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  File "/opt/stack/congress/congress/datasources/datasource_driver.py", line
> 1048, in convert_objs
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  rows, _ = DataSourceDriver.convert_obj(o, translator)
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  File "/opt/stack/congress/congress/datasources/datasource_driver.py", line
> 1021, in convert_obj
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  parent_row_dict)
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  File "/opt/stack/congress/congress/datasources/datasource_driver.py", line
> 994, in _populate_translator_data_hdict
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  return cls._populate_hdict(translator, obj, parent_row_dict)
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  File "/opt/stack/congress/congress/datasources/datasource_driver.py", line
> 924, in _populate_hdict
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  cls._get_value(obj, field, selector), extract_fn)
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  File "/opt/stack/congress/congress/datasources/datasource_driver.py", line
> 693, in _get_value
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
>  if field in o:
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> TypeError: argument of type 'Cluster' is not iterable
> 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> 2016-11-24 20:56:27.435 INFO congress.datasources.datasource_driver [-]
> magnum:: finished polling
>
>
> Any suggestions to solve it?
> I've add the code to review.
>
> Ruben
>
> ----- Original Message -----
> From: "Ruben" <r.manganiello1 at studenti.unisa.it>
> To: "Tim Hinrichs" <tim at styra.com>
> Cc: openstack-dev at lists.openstack.org, "timothy l hinrichs" <
> timothy.l.hinrichs at gmail.com>
> Sent: Thursday, November 24, 2016 12:44:38 AM
> Subject: Re: [Congress] Magnum_driver
>
> Hi Tim,
> I already have 'pyhton-magnumclient' in the requirements.txt file, but I
> still have the error.
>
> This is the /opt/stack/congress/requirements.txt file:
>
> "# The order of packages is significant, because pip processes them in the
> order
> # of appearance. Changing the order has an impact on the overall
> integration
> # process, which may cause wedges in the gate later.
>
> Babel>=2.3.4 # BSD
> eventlet!=0.18.3,>=0.18.2 # MIT
> PuLP>=1.4.1 # MIT
> keystoneauth1>=2.14.0 # Apache-2.0
> keystonemiddleware!=4.5.0,>=4.2.0 # Apache-2.0
> Paste # MIT
> PasteDeploy>=1.5.0 # MIT
> pbr>=1.6 # Apache-2.0
> python-keystoneclient>=3.6.0 # Apache-2.0
> python-heatclient>=1.5.0 # Apache-2.0
> python-magnumclient
> python-muranoclient>=0.8.2 # Apache-2.0
> python-novaclient!=2.33.0,>=2.29.0 # Apache-2.0
> python-neutronclient>=5.1.0 # Apache-2.0
> python-ceilometerclient>=2.5.0 # Apache-2.0
> python-cinderclient!=1.7.0,!=1.7.1,>=1.6.0 # Apache-2.0
> python-swiftclient>=2.2.0 # Apache-2.0
> python-ironicclient>=1.6.0 # Apache-2.0
> alembic>=0.8.4 # MIT
> python-dateutil>=2.4.2 # BSD
> python-glanceclient>=2.5.0 # Apache-2.0
> Routes!=2.0,!=2.1,!=2.3.0,>=1.12.3;python_version=='2.7' # MIT
> Routes!=2.0,!=2.3.0,>=1.12.3;python_version!='2.7' # MIT
> six>=1.9.0 # MIT
> oslo.concurrency>=3.8.0 # Apache-2.0
> oslo.config!=3.18.0,>=3.14.0 # Apache-2.0
> oslo.context>=2.9.0 # Apache-2.0
> oslo.db!=4.13.1,!=4.13.2,>=4.11.0 # Apache-2.0
> oslo.messaging>=5.2.0 # Apache-2.0
> oslo.policy>=1.15.0 # Apache-2.0
> oslo.serialization>=1.10.0 # Apache-2.0
> oslo.service>=1.10.0 # Apache-2.0
> oslo.utils>=3.18.0 # Apache-2.0
> oslo.middleware>=3.0.0 # Apache-2.0
> oslo.vmware>=2.11.0 # Apache-2.0
> oslo.log>=3.11.0 # Apache-2.0
> WebOb>=1.6.0 # MIT"
>
> ----- Original Message -----
> From: "Tim Hinrichs" <tim at styra.com>
> To: "Ruben" <r.manganiello1 at studenti.unisa.it>,
> openstack-dev at lists.openstack.org
> Cc: "timothy l hinrichs" <timothy.l.hinrichs at gmail.com>
> Sent: Wednesday, November 23, 2016 11:44:40 PM
> Subject: Re: [Congress] Magnum_driver
>
> Ruben,
>
> All the software that gets imported by your code needs to be listed in
> requirements.txt so that when tox runs, it installs that software.  When I
> did some debugging for you I made the necessary change, so if you add that
> back into your change, that error should disappear.
>
> https://review.openstack.org/#/c/397150/
>
> Tim
>
>
> On Wed, Nov 23, 2016 at 12:29 PM Ruben <r.manganiello1 at studenti.unisa.it>
> wrote:
>
> > Hi everybody,
> > I've trying to run the unit test of the magnum_driver.
> >
> > I make:
> > -cd /opt/stack/congress/
> > -tox -epy27
> >
> > but I have error with the import.
> > This is the output:
> >
> > "py27 develop-inst-noop: /opt/stack/congress
> > py27 installed:
> >
> alabaster==0.7.9,alembic==0.8.8,amqp==1.4.9,anyjson==0.3.3,appdirs==1.4.0,Babel==2.3.4,cachetools==2.0.0,cffi==1.9.1,cliff==2.2.0,cmd2==0.6.9,-e
> > git+
> >
> http://git.openstack.org/openstack/congress@b2d96b56f721c941e85db565d203df008c455b19#egg=congress,contextlib2==0.5.4,coverage==4.2,cryptography==1.5.3,debtcollector==1.9.0,decorator==4.0.10,docutils==0.12,dogpile.cache==0.6.2,enum34==1.1.6,eventlet==0.19.0,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,flake8==2.2.4,funcsigs==1.0.2,functools32==3.2.3.post2,futures==3.0.5,futurist==0.19.0,greenlet==0.4.10,hacking==0.10.2,idna==2.1,ipaddress==1.0.17,iso8601==0.1.11,Jinja2==2.8,jsonpatch==1.14,jsonpointer==1.10,jsonschema==2.5.1,keystoneauth1==2.15.0,keystonemiddleware==4.10.0,kombu==3.0.37,linecache2==1.0.0,lxml==3.6.4,Mako==1.0.6,MarkupSafe==0.23,mccabe==0.2.1,mock==2.0.0,monotonic==1.2,mox3==0.18.0,msgpack-python==0.4.8,netaddr==0.7.18,netifaces==0.10.5,openstacksdk==0.9.9,os-client-config==1.22.0,osc-lib==1.2.0,oslo.concurrency==3.15.0,oslo.config==3.19.0,oslo.context==2.10.0,oslo.db==4.14.0,oslo.i18n==3.10.0,oslo.log==3.17.0,oslo.messaging==5.12.0,oslo.middleware==3.20.0,oslo.policy==1.16.0,oslo.serialization==2.14.0,oslo.service==1.17.0,oslo.utils==3.18.0,oslo.vmware==2.15.0,oslosphinx==4.8.0,oslotest==2.11.0,Paste==2.0.3,PasteDeploy==1.5.2,pbr==1.10.0,pep8==1.5.7,pika==0.10.0,pika-pool==0.1.3,ply==3.9,positional==1.1.1,prettytable==0.7.2,PuLP==1.6.1,pyasn1==0.1.9,pycadf==2.4.0,pycparser==2.17,pyflakes==0.8.1,Pygments==2.1.3,pyinotify==0.9.6,pyOpenSSL==16.2.0,pyparsing==1.5.7,python-ceilometerclient==2.7.0,python-cinderclient==1.9.0,python-dateutil==2.6.0,python-editor==1.0.1,python-glanceclient==2.5.0,python-heatclient==1.5.0,python-ironicclient==1.8.0,python-keystoneclient==3.6.0,python-mimeparse==1.6.0,python-muranoclient==0.11.1,python-neutronclient==6.0.0,python-novaclient==6.0.0,python-openstackclient==3.3.0,python-subunit==1.2.0,python-swiftclient==3.1.0,pytz==2016.7,PyYAML==3.12,reno==1.8.0,repoze.lru==0.6,requests==2.11.1,requests-mock==1.1.0,requestsexceptions==1.1.3,retrying==1.3.3,rfc3986==0.4.1,Routes==2.3.1,simplejson==3.10.0,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.3.6,sphinx-rtd-theme==0.1.9,SQLAlchemy==1.1.3,sqlalchemy-migrate==0.10.0,sqlparse==0.2.2,stevedore==1.18.0,suds-jurko==0.6,Tempita==0.5.2,tenacity==3.3.0,testrepository==0.0.20,testscenarios==0.5.0,testtools==2.2.0,traceback2==1.4.0,unicodecsv==0.14.1,unittest2==1.1.0,urllib3==1.19,warlock==1.2.0,WebOb==1.6.2,wrapt==1.10.8,yaql==1.1.1
> > py27
> > <
> http://git.openstack.org/openstack/congress@b2d96b56f721c941e85db565d203df008c455b19#egg=congress,contextlib2==0.5.4,coverage==4.2,cryptography==1.5.3,debtcollector==1.9.0,decorator==4.0.10,docutils==0.12,dogpile.cache==0.6.2,enum34==1.1.6,eventlet==0.19.0,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,flake8==2.2.4,funcsigs==1.0.2,functools32==3.2.3.post2,futures==3.0.5,futurist==0.19.0,greenlet==0.4.10,hacking==0.10.2,idna==2.1,ipaddress==1.0.17,iso8601==0.1.11,Jinja2==2.8,jsonpatch==1.14,jsonpointer==1.10,jsonschema==2.5.1,keystoneauth1==2.15.0,keystonemiddleware==4.10.0,kombu==3.0.37,linecache2==1.0.0,lxml==3.6.4,Mako==1.0.6,MarkupSafe==0.23,mccabe==0.2.1,mock==2.0.0,monotonic==1.2,mox3==0.18.0,msgpack-python==0.4.8,netaddr==0.7.18,netifaces==0.10.5,openstacksdk==0.9.9,os-client-config==1.22.0,osc-lib==1.2.0,oslo.concurrency==3.15.0,oslo.config==3.19.0,oslo.context==2.10.0,oslo.db==4.14.0,oslo.i18n==3.10.0,oslo.log==3.17.0,oslo.messaging==5.12.0,oslo.middleware==3.20.0,oslo.policy==1.16.0,oslo.serialization==2.14.0,oslo.service==1.17.0,oslo.utils==3.18.0,oslo.vmware==2.15.0,oslosphinx==4.8.0,oslotest==2.11.0,Paste==2.0.3,PasteDeploy==1.5.2,pbr==1.10.0,pep8==1.5.7,pika==0.10.0,pika-pool==0.1.3,ply==3.9,positional==1.1.1,prettytable==0.7.2,PuLP==1.6.1,pyasn1==0.1.9,pycadf==2.4.0,pycparser==2.17,pyflakes==0.8.1,Pygments==2.1.3,pyinotify==0.9.6,pyOpenSSL==16.2.0,pyparsing==1.5.7,python-ceilometerclient==2.7.0,python-cinderclient==1.9.0,python-dateutil==2.6.0,python-editor==1.0.1,python-glanceclient==2.5.0,python-heatclient==1.5.0,python-ironicclient==1.8.0,python-keystoneclient==3.6.0,python-mimeparse==1.6.0,python-muranoclient==0.11.1,python-neutronclient==6.0.0,python-novaclient==6.0.0,python-openstackclient==3.3.0,python-subunit==1.2.0,python-swiftclient==3.1.0,pytz==2016.7,PyYAML==3.12,reno==1.8.0,repoze.lru==0.6,requests==2.11.1,requests-mock==1.1.0,requestsexceptions==1.1.3,retrying==1.3.3,rfc3986==0.4.1,Routes==2.3.1,simplejson==3.10.0,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.3.6,sphinx-rtd-theme==0.1.9,SQLAlchemy==1.1.3,sqlalchemy-migrate==0.10.0,sqlparse==0.2.2,stevedore==1.18.0,suds-jurko==0.6,Tempita==0.5.2,tenacity==3.3.0,testrepository==0.0.20,testscenarios==0.5.0,testtools==2.2.0,traceback2==1.4.0,unicodecsv==0.14.1,unittest2==1.1.0,urllib3==1.19,warlock==1.2.0,WebOb==1.6.2,wrapt==1.10.8,yaql==1.1.1py27
> >
> > runtests: PYTHONHASHSEED='4235171505 <(423)%20517-1505>
> <(423)%20517-1505>'
> > py27 runtests: commands[0] | find . -type f -name *.py[c|o] -delete
> > py27 runtests: commands[1] | python setup.py testr --slowest
> --testr-args=
> > running testr
> > running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> > OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> > OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \
> > ${PYTHON:-python} -m subunit.run discover -t ./ ./congress/tests --list
> > --- import errors ---
> > Failed to import test module:
> congress.tests.datasources.test_magnum_driver
> > Traceback (most recent call last):
> >   File
> >
> "/opt/stack/congress/.tox/py27/local/lib/python2.7/site-packages/unittest2/loader.py",
> > line 456, in _find_test_path
> >     module = self._get_module_from_name(name)
> >   File
> >
> "/opt/stack/congress/.tox/py27/local/lib/python2.7/site-packages/unittest2/loader.py",
> > line 395, in _get_module_from_name
> >     __import__(name)
> >   File "congress/tests/datasources/test_magnum_driver.py", line 18, in
> > <module>
> >     from congress.datasources import magnum_driver
> >   File "congress/datasources/magnum_driver.py", line 16, in <module>
> >     from magnumclient import client as magnum_client
> > ImportError: No module named magnumclient
> > Non-zero exit code (2) from test listing.
> > error: testr failed (3)
> > ERROR: InvocationError: '/opt/stack/congress/.tox/py27/bin/python
> setup.py
> > testr --slowest --testr-args='
> >
> __________________________________________________________________________________________________
> > summary
> >
> __________________________________________________________________________________________________
> > ERROR:   py27: commands failed"
> >
> >
> > Pep8 is ok.
> > What should I do to solve the error above?
> >
> > Anyway I've errors with the translators..
> > I've add the code to review.
> >
> > Ruben
> >
> >
>