Instead of an unstable trunk, i think code should just be better vetted before it lands in the trunk. If the difference between trunk and your proposed unstable trunk is a set of automated tests, then those tests can just as easily be run on a LP branch before it gets into current trunk. We just need the tests. No matter how many stability levels of trunk we have, without better tests we can't guarantee stability.<br>
<br><div class="gmail_quote">On Thu, Feb 24, 2011 at 11:15 AM, Justin Santa Barbara <span dir="ltr"><<a href="mailto:justin@fathomdb.com">justin@fathomdb.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi Jay,<div><br></div><div>I couldn't agree more. I had another bug come up yesterday on another of my patches (I know - not a good day for me!) where I again broke the OpenStack API by requiring the metadata attribute.</div>
<div><br></div><div>In this case, it was missed by the unit tests. I believe I was always passing metadata, so simply missed the real world case. Here's the bug report:</div><div><a href="https://bugs.launchpad.net/nova/+bug/724143" target="_blank">https://bugs.launchpad.net/nova/+bug/724143</a></div>
<div><br></div><div>This brings up a number of points though:</div><div><br></div><div>On testing</div><div><br></div><div><ol><li>The bug reporter apparently knows how to program, but instead of us getting a test case which we could immediately use, we got a Ruby test case. I think we should do whatever we can so that people that are moderately comfortable with code also feel comfortable submitting a failing test case that we can use. I think this means having some version of an API client, or even a Reference-Implementation API client, in the source tree, and using it.</li>
<li>It's not clear to me how we deal with unit test cases that are failing - cases that represent a found bug that is not yet fixed. Maybe it should be submitted on a bugNNN branch, with failing tests, and then whoever works on the bug can branch from it? (Of course, we'd be lucky if all bugs went like this, but it does happen. I often find myself fixing tangential issues, and I'd like to know the 'right thing to do' is)</li>
<li>On this particular metadata bug, it was my fault. I therefore submitted a very rapid hotfix which just fixes the issue. I then coded up unit tests that use the OpenStack API, and did in fact hit the issue naturally, and then included my fix which resolved it. I submitted the 'full patch' as a separate branch.</li>
<li>This 'full patch' branch includes unit tests that bring up the OpenStack API and various services in-process, and runs tests just like a user. It would have caught the metadata issue. </li><li>The hope is that we can reuse the same tests as smoke tests, by simply tweaking them to work against real services instead of bringing up in-process stub-services. These could be (some of) the smoke tests in Hudson.</li>
<li>I'd hope that we could have two levels of these smoke tests, one that runs on a single configuration (e.g. KVM, OpenISCSI, Glance); another that runs a matrix of configurations (and might take an hour or more to run)</li>
<li>Ideally we'd have a torture test that would run overnight and be randomized and try to find obscure bugs, even if issues found are not necessarily repeatable in the way that non-randomized tests are.</li></ol></div>
<div><br></div><div>We need an unstable trunk:</div><div><br></div><div><ol><li>In general, it seems that our end-users are using trunk for unreleased functionality and treating it as if it were released. I don't think we should be encouraging that, because I know I'll make more mistakes in future and some of them will make it past the reviewers' defensive line into trunk; it's also simply not realistic to require reviewers to review every combination - e.g. how can a reviewer really review my HP SAN patch without an HP SAN? There will be issues in trunk, and if we have to revert them rather than just fixing them it will slow us down. The current situation is bad for our users and bad for developers. </li>
<li>One way we could keep everyone happy is by using our test suite to auto-merge from an 'unstable trunk' into 'stable trunk', only once code passes tests. Commits would initially merge into 'unstable trunk', and we would try to keep that branch moving forwards rather than reverting things that go wrong. Of course, maintaining a good 'stable trunk' relies on having good tests, but I think we're getting there. It's also great incentive to write good smoke tests.</li>
<li>Jay: I believe you've done this to great success on the Drizzle project?</li></ol></div><div><br></div><font color="#888888"><div>Justin</div></font><div><div></div><div class="h5"><div><br></div><div><br><br><div class="gmail_quote">
On Thu, Feb 24, 2011 at 6:13 AM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
I'd like to bring up an alternate reason why it was approved and<br>
subsequently reverted.<br>
<br>
The test cases for the OpenStack API (and much of the EC2 API) assume<br>
way too many things and mock out too many things. In addition, since<br>
there are zero smoketests for the OpenStack API, there were no<br>
functional tests that would have *immediately* highlighted this<br>
problem (and many other recent EC2 vs OS API problems).<br>
<br>
In other words, sure, we should revert the patch to "fix things",<br>
however the priority should *not* be on refactoring the auth API or<br>
the way the auth layer in Nova is handled. The priority should be on<br>
writing a smoketest for the OpenStack API so that we can link it into<br>
Hudson and these types of issues can be automatically caught.<br>
<font color="#888888"><br>
-jay<br>
</font><div><div></div><div><br>
On Wed, Feb 23, 2011 at 10:03 PM, Paul Voccio <<a href="mailto:paul.voccio@rackspace.com" target="_blank">paul.voccio@rackspace.com</a>> wrote:<br>
> Justin,<br>
> I think you hit upon the reason of why I think it was approved and reverted.<br>
> Because it hadn't been talked about in a blueprint or a mail sent to the<br>
> list (I think I'm up to date on the threads) and a patch landed means other<br>
> alternatives weren't considered before pushing it through to begin with. I<br>
> think we're all open to talking about how to better the auth system and make<br>
> improvements. Dragon has already discussed some alternatives and suggestions<br>
> on the BP page below. I think this is the right way to continue the dialog<br>
> and we all can agree on a good way forward.<br>
> I'm confident we can figure it out.<br>
> If I missed a conversation, my apologies.<br>
> pvo<br>
> From: Vishvananda Ishaya <<a href="mailto:vishvananda@gmail.com" target="_blank">vishvananda@gmail.com</a>><br>
> Date: Wed, 23 Feb 2011 18:19:41 -0800<br>
> To: Justin Santa Barbara <<a href="mailto:justin@fathomdb.com" target="_blank">justin@fathomdb.com</a>><br>
> Cc: <<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>><br>
> Subject: Re: [Openstack] Should the OpenStack API re-use the EC2<br>
> credentials?<br>
><br>
> Hey Justin,<br>
> Does it make any difference that the way the auth is (theoretically)<br>
> supposed to work with the os api is that the user gets an auth token from an<br>
> external auth server and then uses username / authtoken to actually contact<br>
> the api? I think it is just faked out right now to use the access_key<br>
> instead of doing external auth, but I think the reason it works like it does<br>
> is because the plan was to switch to external auth eventually.<br>
> Vish<br>
> On Feb 23, 2011, at 5:56 PM, Justin Santa Barbara wrote:<br>
><br>
> I previously fixed OpenStack authentication so it would use the same<br>
> credentials as EC2. This bugfix was just reverted, because it caused<br>
> OpenStack API users to have to enter in different credentials (sorry!), but<br>
> primarily because it hadn't been discussed on the mailing list. So here<br>
> goes!<br>
> Here's a<br>
> blueprint: <a href="https://blueprints.launchpad.net/nova/+spec/authentication-consistency" target="_blank">https://blueprints.launchpad.net/nova/+spec/authentication-consistency</a><br>
> Here's an overview of the problem:<br>
> EC2 uses an (api_key, api_secret) pair. Post-revert, OpenStack uses the<br>
> api_key(!) as the password, but a different value entirely as the username:<br>
> (username, api_key). The bugfix made it so that both APIs used the EC2<br>
> credentials (api_key, api_secret) . This did mean that anyone that had<br>
> saved the 'bad' OpenStack credentials was unable to continue to use those<br>
> credentials. I also overlooked exporting the updated credentials in novarc<br>
> (though a merge request was pending).<br>
> I actually thought originally that this was a straight-up bug, rather than a<br>
> design 'decision', so I should definitely have flagged it better. Again,<br>
> sorry to those I impacted.<br>
> As things stand now, post-revert, this is probably a security flaw, because<br>
> the EC2 API does not treat the api_key as a secret. The EC2 API can<br>
> (relatively) safely be run over non-SSL, because it uses signatures instead<br>
> of passing the shared secret directly.<br>
> This is also not very user-friendly. Post-revert, an end-user must know<br>
> whether any particular cloud tool uses the EC2 API or the OpenStack API, so<br>
> that they can enter in the correct pair of credentials. That doesn't seem<br>
> like a good idea; I think there should be one set of credentials.<br>
><br>
> There is some discussion about the idea of having the api_key be<br>
> user-friendly. I don't think it buys us anything, because the api_secret is<br>
> still going to be un-friendly, but I have no objection as long as it is does<br>
> in a way that does not break existing users of the EC2 API.<br>
> I propose that:<br>
> (1) the OpenStack API and EC2 credentials should be the same as each other<br>
> (whatever they are) for the sake of our collective sanity and<br>
> (2) we have to change the current configuration anyway for security<br>
> reasons.<br>
> (3) We should not change the EC2 credentials, because we've shipped the EC2<br>
> API and our users have an expectation that we won't break them without good<br>
> reason, so<br>
> (4) we must change the credentials for users of the (non-shipped) OpenStack<br>
> API.<br>
> Estimated user impact: I believe there are two people that will be affected,<br>
> and it will take them ~1 minute each, so total impact ~2 minutes.<br>
> The longer we delay fixing this, the more people we break and the bigger the<br>
> impact. It seems that we have no choice but to do a<br>
> non-backwards-compatible authentication change, but I believe this is OK at<br>
> the moment because the OpenStack API is not yet stable/released - i.e. we<br>
> can still make fixes without worrying about backwards compatibility shims.<br>
> We're not in "The Old New Thing" land yet :-)<br>
><br>
><br>
> As an aside, I am very unhappy about the way this revert was pushed through<br>
> by Rackspace team-members, seemingly without much consideration of<br>
> alternatives. Perhaps we should consider changing from needing two<br>
> core-approves, to needing one Rackspace core-approve and one non-Rackspace<br>
> core-approve.<br>
><br>
> Justin<br>
><br>
><br>
> _______________________________________________<br>
> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
><br>
> _______________________________________________ Mailing list:<br>
> <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a> More help :<br>
> <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
><br>
> Confidentiality Notice: This e-mail message (including any attached or<br>
> embedded documents) is intended for the exclusive and confidential use of<br>
> the<br>
> individual or entity to which this message is addressed, and unless<br>
> otherwise<br>
> expressly indicated, is confidential and privileged information of<br>
> Rackspace.<br>
> Any dissemination, distribution or copying of the enclosed material is<br>
> prohibited.<br>
> If you receive this transmission in error, please notify us immediately by<br>
> e-mail<br>
> at <a href="mailto:abuse@rackspace.com" target="_blank">abuse@rackspace.com</a>, and delete the original message.<br>
> Your cooperation is appreciated.<br>
><br>
> _______________________________________________<br>
> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
><br>
><br>
</div></div></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div><br><PRE>
Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is prohibited.
If you receive this transmission in error, please notify us immediately by e-mail
at abuse@rackspace.com, and delete the original message.
Your cooperation is appreciated.
</PRE>