<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jul 16, 2014 at 11:28 PM, Steve Baker <span dir="ltr"><<a href="mailto:sbaker@redhat.com" target="_blank">sbaker@redhat.com</a>></span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000"><div class="">
    <div>On 12/07/14 09:25, Joe Gordon wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr"><br>
        <div class="gmail_extra"><br>
          <br>
          <div class="gmail_quote">On Fri, Jul 11, 2014 at 4:42 AM,
            Jeremy Stanley <span dir="ltr"><<a href="mailto:fungi@yuggoth.org" target="_blank">fungi@yuggoth.org</a>></span> wrote:<br>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
              <div>On 2014-07-11 11:21:19 +0200 (+0200),
                Matthias Runge wrote:<br>
                > this broke horizon stable and master; heat stable
                is affected as<br>
                > well.<br>
              </div>
              [...]<br>
              <br>
              I guess this is a plea for applying something like the
              oslotest<br>
              framework to client libraries so they get backward-compat
              jobs run<br>
              against unit tests of all dependant/consuming software...
              branchless<br>
              tempest already alleviates some of this, but not the case
              of changes<br>
              in a library which will break unit/functional tests of
              another<br>
              project.<br>
            </blockquote>
            <div><br>
            </div>
            <div>We actually do have some tests for backwards
              compatibility, and they all passed. Presumably because
              both heat and horizon have poor integration test.</div>
            <div><br>
            </div>
            <div>
              We ran </div>
            <div><br>
            </div>
            <div>
              <ul style="color:rgb(0,0,0);font-family:sans-serif">
                <li style="list-style-type:none"><span style="display:inline-block;min-width:20em;padding-bottom:2pt;width:auto!important"><a href="http://logs.openstack.org/66/94166/3/check/check-tempest-dsvm-full-havana/8e09faa" style="text-decoration:none;color:rgb(6,84,172)" target="_blank">check-tempest-dsvm-full-havana</a></span> <span><span style="color:rgb(0,127,0)">SUCCESS</span> in
                    40m 47s (non-voting)</span></li>
                <li style="list-style-type:none"><span style="display:inline-block;min-width:20em;padding-bottom:2pt;width:auto!important"><a href="http://logs.openstack.org/66/94166/3/check/check-tempest-dsvm-neutron-havana/b4ad019" style="text-decoration:none;color:rgb(6,84,172)" target="_blank">check-tempest-dsvm-neutron-havana</a></span> <span><span style="color:rgb(0,127,0)">SUCCESS</span> in
                    36m 17s (non-voting)</span></li>
                <li style="list-style-type:none"><span style="display:inline-block;min-width:20em;padding-bottom:2pt;width:auto!important"><a href="http://logs.openstack.org/66/94166/3/check/check-tempest-dsvm-full-icehouse/c0c62e5" style="text-decoration:none;color:rgb(6,84,172)" target="_blank">check-tempest-dsvm-full-icehouse</a></span> <span><span style="color:rgb(0,127,0)">SUCCESS</span> in
                    53m 05s</span></li>
                <li style="list-style-type:none"><span style="display:inline-block;min-width:20em;padding-bottom:2pt;width:auto!important"><a href="http://logs.openstack.org/66/94166/3/check/check-tempest-dsvm-neutron-icehouse/a54aedb" style="color:rgb(6,84,172)" target="_blank">check-tempest-dsvm-neutron-icehouse</a></span> <span><span style="color:rgb(0,127,0)">SUCCESS</span> in
                    57m 28s</span></li>
              </ul>
            </div>
            <div><br>
            </div>
            <div>on the offending patches (<a href="https://review.openstack.org/#/c/94166/" target="_blank">https://review.openstack.org/#/c/94166/</a>)</div>
            <div> </div>
            <div><br>
            </div>
            <div>Infra patch that added these tests: <a href="https://review.openstack.org/#/c/80698/" target="_blank">https://review.openstack.org/#/c/80698/</a><br>
            </div>
            <div><br>
            </div>
            <br>
          </div>
        </div>
      </div>
    </blockquote></div>
    Heat-proper would have continued working fine with novaclient
    2.18.0. The regression was with raising novaclient exceptions, which
    is only required in our unit tests. I saw this break coming and
    switched to raising via from_response
    <a href="https://review.openstack.org/#/c/97977/22/heat/tests/v1_1/fakes.py" target="_blank">https://review.openstack.org/#/c/97977/22/heat/tests/v1_1/fakes.py</a><br>
    <br>
    Unit tests tend to deal with more internals of client libraries just
    for mocking purposes, and there have been multiple breaks in unit
    tests for heat and horizon when client libraries make internal
    changes.<br>
    <br>
    This could be avoided if the client gate jobs run the unit tests for
    the projects which consume them.</div></blockquote><div> </div><div>That may work but isn't this exactly what integration testing is for? <br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>