Open Stack

Hi Tim,
I've added the code of magnum_driver and its unit test to review.
It seems everything works.

Ruben

----- Original Message -----
From: "Tim Hinrichs" <tim at styra.com>
To: "Ruben" <r.manganiello1 at studenti.unisa.it>
Cc: openstack-dev at lists.openstack.org, "timothy l hinrichs" <timothy.l.hinrichs at gmail.com>
Sent: Saturday, November 26, 2016 12:48:12 AM
Subject: Re: [Congress] Magnum_driver

Definitely push that code up into Gerrit so we can all take a look.  Data
like pods and containers is probably the most valuable data from Magnum, so
I'd definitely recommend adding that.  But push the code you have to Gerrit
first.  (As long as you leave the ChangeId the same each time you push to
Gerrit, Gerrit will keep all of the versions you pushed organized together,
yet keep the versions separate.)

Tim

On Fri, Nov 25, 2016 at 3:06 PM Ruben <r.manganiello1 at studenti.unisa.it>
wrote:

> Hi Tim,
> You are great. It works! Thanks a lot!
> I've also solved the problem with py27. The unit test seems to work.
> The only thing that seems not to work is populate the 'clusters_links' and
> 'cluster_templates_links' tables: they are empty.
> Also, the 'labels' table is empty.
> I've no errors anyway.
> Are these problems according to you?
>
> Should I to try to add the translation of pods, containers and services?
>
> I've add the code to review.
>
> Ruben
> ----- Original Message -----
> From: "Tim Hinrichs" <tim at styra.com>
> To: "Ruben" <r.manganiello1 at studenti.unisa.it>
> Cc: openstack-dev at lists.openstack.org, "timothy l hinrichs" <
> timothy.l.hinrichs at gmail.com>
> Sent: Friday, November 25, 2016 10:36:29 PM
> Subject: Re: [Congress] Magnum_driver
>
> Hi Ruben,
>
> Glad you got that worked out.  Once in a while I end up deleting my .tox
> dir because it gets out of date.  I guess that's what the --recreate option
> to tox does.  So I learned something.  :)
>
> The new error message looks to be saying that an object of type 'Cluster'
> cannot be iterated.  Congress's _translate_clusters method assumes that
> what you give it is basically JSON (arrays, dictionaries, numbers, strings,
> bools).   In this case Congress is trying to iterate over the keys of a
> dictionary but is getting a 'Cluster' Python object instead.  What's
> happening is that when you do the cluster.list() (or whatever it's called)
> on the magnum-python you're getting back a list of Cluster Python objects
> instead of a list of dictionaries.
>
> To fix the problem, you'll want to take the results of cluster.list() and
> translate it into a list of dictionaries, and then hand that off to
> _translate_clusters.  Probably you'll need to do the same for the
> cluster_template.  So instead of ...
>
>         clusters_method = lambda: self._translate_clusters(
>             {'clusters': self.magnum.clusters.list()})
>
> You'll want to do something like ...
>
>         clusters_method = lambda: self._translate_clusters(
>             {'clusters': [x.__dict__ for x in self.magnum.clusters.list()]
>  })
>
> Or at least, that's what I'd start trying.  __dict__ grabs all the fields
> of an object and returns a dictionary.  (If some of the values of that
> dictionary are also Python objects, then you might need something that's
> more complicated to recursively walk the result of clusters.list() and
> translate everything into lists and dictionaries.  And I'm not sure that
> __dict__ will give you exactly what you want, but it's worth a try.).
>
> Tim
>
>
> On Thu, Nov 24, 2016 at 12:11 PM Ruben <r.manganiello1 at studenti.unisa.it>
> wrote:
>
> > Hi Tim,
> > I solved the problem with:
> >
> > tox --recreate -e py27
> >
> > Now I no have the error on the import. The unit test still has some
> > errors, but I don't know why..
> >
> > Anyway, I've this error when I try to add the magnum_driver:
> >
> > 2016-11-24 20:56:27.191 INFO congress.datasources.datasource_driver [-]
> > magnum:: polling
> > 2016-11-24 20:56:27.192 DEBUG congress.datasources.datasource_driver [-]
> > update table clusters. from (pid=18720) update_from_datasource
> > /opt/stack/congress/congress/datasources/datasource_driver.py:1370
> > 2016-11-24 20:56:27.427 DEBUG congress.datasources.magnum_driver [-]
> > CLUSTERS: {'clusters': [<Clusters {u'status': u'CREATE_FAILED',
> > u'cluster_template_id': u'8d25a1ed-faa6-4305-a6a1-6559708c805b', u'uuid':
> > u'1634beb9-25de-4cdd-bafa-67537069f0cc', u'links': [{u'href': u'
> > http://10.0.2.15:9511/v1/clusters/1634beb9-25de-4cdd-bafa-67537069f0cc',
> > u'rel': u'self'}, {u'href': u'
> > http://10.0.2.15:9511/clusters/1634beb9-25de-4cdd-bafa-67537069f0cc',
> > u'rel': u'bookmark'}], u'stack_id':
> > u'17f1248d-286a-4fd4-9639-af5773670f03', u'master_count': 1, u'keypair':
> > u'testkey', u'node_count': 1, u'create_timeout': 60, u'name':
> > u'k8s-cluster'}>]} from (pid=18720) _translate_clusters
> > /opt/stack/congress/congress/datasources/magnum_driver.py:165
> > 2016-11-24 20:56:27.435 ERROR congress.datasources.datasource_driver [-]
> > Datasource driver raised exception
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> > Traceback (most recent call last):
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  File "/opt/stack/congress/congress/datasources/datasource_driver.py",
> line
> > 1384, in poll
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  self.update_from_datasource()  # sets self.state
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  File "/opt/stack/congress/congress/datasources/datasource_driver.py",
> line
> > 1371, in update_from_datasource
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  self.update_methods[registered_table]()
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  File "/opt/stack/congress/congress/datasources/magnum_driver.py", line
> > 150, in <lambda>
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  {'clusters': self.magnum.clusters.list()})
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  File "/opt/stack/congress/congress/datasources/datasource_utils.py",
> line
> > 57, in inner
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  result = f(self, raw_data, *args, **kw)
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  File "/opt/stack/congress/congress/datasources/magnum_driver.py", line
> > 167, in _translate_clusters
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  obj['clusters'], MagnumDriver.clusters_translator)
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  File "/opt/stack/congress/congress/datasources/datasource_driver.py",
> line
> > 1048, in convert_objs
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  rows, _ = DataSourceDriver.convert_obj(o, translator)
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  File "/opt/stack/congress/congress/datasources/datasource_driver.py",
> line
> > 1021, in convert_obj
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  parent_row_dict)
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  File "/opt/stack/congress/congress/datasources/datasource_driver.py",
> line
> > 994, in _populate_translator_data_hdict
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  return cls._populate_hdict(translator, obj, parent_row_dict)
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  File "/opt/stack/congress/congress/datasources/datasource_driver.py",
> line
> > 924, in _populate_hdict
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  cls._get_value(obj, field, selector), extract_fn)
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  File "/opt/stack/congress/congress/datasources/datasource_driver.py",
> line
> > 693, in _get_value
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> >  if field in o:
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> > TypeError: argument of type 'Cluster' is not iterable
> > 2016-11-24 20:56:27.435 TRACE congress.datasources.datasource_driver
> > 2016-11-24 20:56:27.435 INFO congress.datasources.datasource_driver [-]
> > magnum:: finished polling
> >
> >
> > Any suggestions to solve it?
> > I've add the code to review.
> >
> > Ruben
> >
> > ----- Original Message -----
> > From: "Ruben" <r.manganiello1 at studenti.unisa.it>
> > To: "Tim Hinrichs" <tim at styra.com>
> > Cc: openstack-dev at lists.openstack.org, "timothy l hinrichs" <
> > timothy.l.hinrichs at gmail.com>
> > Sent: Thursday, November 24, 2016 12:44:38 AM
> > Subject: Re: [Congress] Magnum_driver
> >
> > Hi Tim,
> > I already have 'pyhton-magnumclient' in the requirements.txt file, but I
> > still have the error.
> >
> > This is the /opt/stack/congress/requirements.txt file:
> >
> > "# The order of packages is significant, because pip processes them in
> the
> > order
> > # of appearance. Changing the order has an impact on the overall
> > integration
> > # process, which may cause wedges in the gate later.
> >
> > Babel>=2.3.4 # BSD
> > eventlet!=0.18.3,>=0.18.2 # MIT
> > PuLP>=1.4.1 # MIT
> > keystoneauth1>=2.14.0 # Apache-2.0
> > keystonemiddleware!=4.5.0,>=4.2.0 # Apache-2.0
> > Paste # MIT
> > PasteDeploy>=1.5.0 # MIT
> > pbr>=1.6 # Apache-2.0
> > python-keystoneclient>=3.6.0 # Apache-2.0
> > python-heatclient>=1.5.0 # Apache-2.0
> > python-magnumclient
> > python-muranoclient>=0.8.2 # Apache-2.0
> > python-novaclient!=2.33.0,>=2.29.0 # Apache-2.0
> > python-neutronclient>=5.1.0 # Apache-2.0
> > python-ceilometerclient>=2.5.0 # Apache-2.0
> > python-cinderclient!=1.7.0,!=1.7.1,>=1.6.0 # Apache-2.0
> > python-swiftclient>=2.2.0 # Apache-2.0
> > python-ironicclient>=1.6.0 # Apache-2.0
> > alembic>=0.8.4 # MIT
> > python-dateutil>=2.4.2 # BSD
> > python-glanceclient>=2.5.0 # Apache-2.0
> > Routes!=2.0,!=2.1,!=2.3.0,>=1.12.3;python_version=='2.7' # MIT
> > Routes!=2.0,!=2.3.0,>=1.12.3;python_version!='2.7' # MIT
> > six>=1.9.0 # MIT
> > oslo.concurrency>=3.8.0 # Apache-2.0
> > oslo.config!=3.18.0,>=3.14.0 # Apache-2.0
> > oslo.context>=2.9.0 # Apache-2.0
> > oslo.db!=4.13.1,!=4.13.2,>=4.11.0 # Apache-2.0
> > oslo.messaging>=5.2.0 # Apache-2.0
> > oslo.policy>=1.15.0 # Apache-2.0
> > oslo.serialization>=1.10.0 # Apache-2.0
> > oslo.service>=1.10.0 # Apache-2.0
> > oslo.utils>=3.18.0 # Apache-2.0
> > oslo.middleware>=3.0.0 # Apache-2.0
> > oslo.vmware>=2.11.0 # Apache-2.0
> > oslo.log>=3.11.0 # Apache-2.0
> > WebOb>=1.6.0 # MIT"
> >
> > ----- Original Message -----
> > From: "Tim Hinrichs" <tim at styra.com>
> > To: "Ruben" <r.manganiello1 at studenti.unisa.it>,
> > openstack-dev at lists.openstack.org
> > Cc: "timothy l hinrichs" <timothy.l.hinrichs at gmail.com>
> > Sent: Wednesday, November 23, 2016 11:44:40 PM
> > Subject: Re: [Congress] Magnum_driver
> >
> > Ruben,
> >
> > All the software that gets imported by your code needs to be listed in
> > requirements.txt so that when tox runs, it installs that software.  When
> I
> > did some debugging for you I made the necessary change, so if you add
> that
> > back into your change, that error should disappear.
> >
> > https://review.openstack.org/#/c/397150/
> >
> > Tim
> >
> >
> > On Wed, Nov 23, 2016 at 12:29 PM Ruben <r.manganiello1 at studenti.unisa.it
> >
> > wrote:
> >
> > > Hi everybody,
> > > I've trying to run the unit test of the magnum_driver.
> > >
> > > I make:
> > > -cd /opt/stack/congress/
> > > -tox -epy27
> > >
> > > but I have error with the import.
> > > This is the output:
> > >
> > > "py27 develop-inst-noop: /opt/stack/congress
> > > py27 installed:
> > >
> >
> alabaster==0.7.9,alembic==0.8.8,amqp==1.4.9,anyjson==0.3.3,appdirs==1.4.0,Babel==2.3.4,cachetools==2.0.0,cffi==1.9.1,cliff==2.2.0,cmd2==0.6.9,-e
> > > git+
> > >
> >
> http://git.openstack.org/openstack/congress@b2d96b56f721c941e85db565d203df008c455b19#egg=congress,contextlib2==0.5.4,coverage==4.2,cryptography==1.5.3,debtcollector==1.9.0,decorator==4.0.10,docutils==0.12,dogpile.cache==0.6.2,enum34==1.1.6,eventlet==0.19.0,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,flake8==2.2.4,funcsigs==1.0.2,functools32==3.2.3.post2,futures==3.0.5,futurist==0.19.0,greenlet==0.4.10,hacking==0.10.2,idna==2.1,ipaddress==1.0.17,iso8601==0.1.11,Jinja2==2.8,jsonpatch==1.14,jsonpointer==1.10,jsonschema==2.5.1,keystoneauth1==2.15.0,keystonemiddleware==4.10.0,kombu==3.0.37,linecache2==1.0.0,lxml==3.6.4,Mako==1.0.6,MarkupSafe==0.23,mccabe==0.2.1,mock==2.0.0,monotonic==1.2,mox3==0.18.0,msgpack-python==0.4.8,netaddr==0.7.18,netifaces==0.10.5,openstacksdk==0.9.9,os-client-config==1.22.0,osc-lib==1.2.0,oslo.concurrency==3.15.0,oslo.config==3.19.0,oslo.context==2.10.0,oslo.db==4.14.0,oslo.i18n==3.10.0,oslo.log==3.17.0,oslo.messaging==5.12.0,oslo.middleware==3.20.0,oslo.policy==1.16.0,oslo.serialization==2.14.0,oslo.service==1.17.0,oslo.utils==3.18.0,oslo.vmware==2.15.0,oslosphinx==4.8.0,oslotest==2.11.0,Paste==2.0.3,PasteDeploy==1.5.2,pbr==1.10.0,pep8==1.5.7,pika==0.10.0,pika-pool==0.1.3,ply==3.9,positional==1.1.1,prettytable==0.7.2,PuLP==1.6.1,pyasn1==0.1.9,pycadf==2.4.0,pycparser==2.17,pyflakes==0.8.1,Pygments==2.1.3,pyinotify==0.9.6,pyOpenSSL==16.2.0,pyparsing==1.5.7,python-ceilometerclient==2.7.0,python-cinderclient==1.9.0,python-dateutil==2.6.0,python-editor==1.0.1,python-glanceclient==2.5.0,python-heatclient==1.5.0,python-ironicclient==1.8.0,python-keystoneclient==3.6.0,python-mimeparse==1.6.0,python-muranoclient==0.11.1,python-neutronclient==6.0.0,python-novaclient==6.0.0,python-openstackclient==3.3.0,python-subunit==1.2.0,python-swiftclient==3.1.0,pytz==2016.7,PyYAML==3.12,reno==1.8.0,repoze.lru==0.6,requests==2.11.1,requests-mock==1.1.0,requestsexceptions==1.1.3,retrying==1.3.3,rfc3986==0.4.1,Routes==2.3.1,simplejson==3.10.0,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.3.6,sphinx-rtd-theme==0.1.9,SQLAlchemy==1.1.3,sqlalchemy-migrate==0.10.0,sqlparse==0.2.2,stevedore==1.18.0,suds-jurko==0.6,Tempita==0.5.2,tenacity==3.3.0,testrepository==0.0.20,testscenarios==0.5.0,testtools==2.2.0,traceback2==1.4.0,unicodecsv==0.14.1,unittest2==1.1.0,urllib3==1.19,warlock==1.2.0,WebOb==1.6.2,wrapt==1.10.8,yaql==1.1.1
> > > py27
> > > <
> >
> http://git.openstack.org/openstack/congress@b2d96b56f721c941e85db565d203df008c455b19#egg=congress,contextlib2==0.5.4,coverage==4.2,cryptography==1.5.3,debtcollector==1.9.0,decorator==4.0.10,docutils==0.12,dogpile.cache==0.6.2,enum34==1.1.6,eventlet==0.19.0,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,flake8==2.2.4,funcsigs==1.0.2,functools32==3.2.3.post2,futures==3.0.5,futurist==0.19.0,greenlet==0.4.10,hacking==0.10.2,idna==2.1,ipaddress==1.0.17,iso8601==0.1.11,Jinja2==2.8,jsonpatch==1.14,jsonpointer==1.10,jsonschema==2.5.1,keystoneauth1==2.15.0,keystonemiddleware==4.10.0,kombu==3.0.37,linecache2==1.0.0,lxml==3.6.4,Mako==1.0.6,MarkupSafe==0.23,mccabe==0.2.1,mock==2.0.0,monotonic==1.2,mox3==0.18.0,msgpack-python==0.4.8,netaddr==0.7.18,netifaces==0.10.5,openstacksdk==0.9.9,os-client-config==1.22.0,osc-lib==1.2.0,oslo.concurrency==3.15.0,oslo.config==3.19.0,oslo.context==2.10.0,oslo.db==4.14.0,oslo.i18n==3.10.0,oslo.log==3.17.0,oslo.messaging==5.12.0,oslo.middleware==3.20.0,oslo.policy==1.16.0,oslo.serialization==2.14.0,oslo.service==1.17.0,oslo.utils==3.18.0,oslo.vmware==2.15.0,oslosphinx==4.8.0,oslotest==2.11.0,Paste==2.0.3,PasteDeploy==1.5.2,pbr==1.10.0,pep8==1.5.7,pika==0.10.0,pika-pool==0.1.3,ply==3.9,positional==1.1.1,prettytable==0.7.2,PuLP==1.6.1,pyasn1==0.1.9,pycadf==2.4.0,pycparser==2.17,pyflakes==0.8.1,Pygments==2.1.3,pyinotify==0.9.6,pyOpenSSL==16.2.0,pyparsing==1.5.7,python-ceilometerclient==2.7.0,python-cinderclient==1.9.0,python-dateutil==2.6.0,python-editor==1.0.1,python-glanceclient==2.5.0,python-heatclient==1.5.0,python-ironicclient==1.8.0,python-keystoneclient==3.6.0,python-mimeparse==1.6.0,python-muranoclient==0.11.1,python-neutronclient==6.0.0,python-novaclient==6.0.0,python-openstackclient==3.3.0,python-subunit==1.2.0,python-swiftclient==3.1.0,pytz==2016.7,PyYAML==3.12,reno==1.8.0,repoze.lru==0.6,requests==2.11.1,requests-mock==1.1.0,requestsexceptions==1.1.3,retrying==1.3.3,rfc3986==0.4.1,Routes==2.3.1,simplejson==3.10.0,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.3.6,sphinx-rtd-theme==0.1.9,SQLAlchemy==1.1.3,sqlalchemy-migrate==0.10.0,sqlparse==0.2.2,stevedore==1.18.0,suds-jurko==0.6,Tempita==0.5.2,tenacity==3.3.0,testrepository==0.0.20,testscenarios==0.5.0,testtools==2.2.0,traceback2==1.4.0,unicodecsv==0.14.1,unittest2==1.1.0,urllib3==1.19,warlock==1.2.0,WebOb==1.6.2,wrapt==1.10.8,yaql==1.1.1py27
> > >
> > > runtests: PYTHONHASHSEED='4235171505 <(423)%20517-1505>
> <(423)%20517-1505>
> > <(423)%20517-1505>'
> > > py27 runtests: commands[0] | find . -type f -name *.py[c|o] -delete
> > > py27 runtests: commands[1] | python setup.py testr --slowest
> > --testr-args=
> > > running testr
> > > running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> > > OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> > > OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \
> > > ${PYTHON:-python} -m subunit.run discover -t ./ ./congress/tests --list
> > > --- import errors ---
> > > Failed to import test module:
> > congress.tests.datasources.test_magnum_driver
> > > Traceback (most recent call last):
> > >   File
> > >
> >
> "/opt/stack/congress/.tox/py27/local/lib/python2.7/site-packages/unittest2/loader.py",
> > > line 456, in _find_test_path
> > >     module = self._get_module_from_name(name)
> > >   File
> > >
> >
> "/opt/stack/congress/.tox/py27/local/lib/python2.7/site-packages/unittest2/loader.py",
> > > line 395, in _get_module_from_name
> > >     __import__(name)
> > >   File "congress/tests/datasources/test_magnum_driver.py", line 18, in
> > > <module>
> > >     from congress.datasources import magnum_driver
> > >   File "congress/datasources/magnum_driver.py", line 16, in <module>
> > >     from magnumclient import client as magnum_client
> > > ImportError: No module named magnumclient
> > > Non-zero exit code (2) from test listing.
> > > error: testr failed (3)
> > > ERROR: InvocationError: '/opt/stack/congress/.tox/py27/bin/python
> > setup.py
> > > testr --slowest --testr-args='
> > >
> >
> __________________________________________________________________________________________________
> > > summary
> > >
> >
> __________________________________________________________________________________________________
> > > ERROR:   py27: commands failed"
> > >
> > >
> > > Pep8 is ok.
> > > What should I do to solve the error above?
> > >
> > > Anyway I've errors with the translators..
> > > I've add the code to review.
> > >
> > > Ruben
> > >
> > >
> >
>