Open Stack

On Wed, Jul 9, 2014 at 7:53 AM, Daniel P. Berrange <berrange at redhat.com> wrote:
> On Wed, Jul 09, 2014 at 07:39:39AM -0400, Sean Dague wrote:
>> On 07/03/2014 01:52 PM, Doug Hellmann wrote:
>> >> oslo code, by definition, is going to be used a lot, inside of tight
>> >> loops. Which means extremely verbose. Actually
>> >> nova.openstack.common.policy currently represents ~40% of all bytes in
>> >> nova-api logs (15 MB uncompressed content during a Tempest run).
>> >>
>> >> Both these are bugs that should be fixed, however, I'd like to figure
>> >> out how we come up with more systematic approach to keep the quality of
>> >> these things up, especially as an oslo fix is expensive in terms of
>> >> labor, as we have to fix in oslo first, then sync to projects. And as
>> >> more syncs have caused issues like this an oslo sync is not longer a
>> >> fast approve as it used to be.
>> >
>> > Do you have some automation in mind, or just focusing more on logging
>> > in reviews?
>>
>> I don't know. I'm just concerned that some really bad bugs are slipping
>> through oslo incubator, which is then copy / pasted into projects before
>> they get noticed (because there is no forward testing), then we have to
>> run around and replicate fixes everywhere.
>>
>> Stuff like this is generating a ton of extra work on everyone that's not
>> oslo.
>
> This is yet another example of the many problems inherant in the cut+paste
> approach used by olso incubator. I'd personally like to see olso-incubator
> killed and everything there moved into its own module with whatever API
> they have now. If we later find we need to break the APIs of some of the
> split out modules, so be it. IMHO it would be less pain overall than we
> have with the cut+paste approach. We'd be able to have fully integrated
> CI testing running against the olso changes as they go on, rather than
> only after copy+paste, whic would ease detection/avoid problems like you're
> highlighting here.

The problem is not that we are copying code from the incubator. The
problem is that too many of us are copying code into too many projects
before it is mature.

The lifecycle for incubated code is:

1. Someone in a project writes a module that works in that application.
2. Someone in another project wants something similar.
3. The code is moved to the incubator, and then copied into both apps.
4. The code is changed to decouple it from either application
(removing access to global state, naming configuration options more
generically and putting them in groups, adding configuration hooks so
we don't have nova calls in glance, etc.).
5. The API for the code is minimized so that implementation details
aren't exposed.
6. The code is turned into a proper library.

A lot of contributions to Oslo have stopped at step 3, with people
moving on to other work.

Steps 4 & 5 cannot be done in a vacuum. They have to happen in the
context of putting the code back into the application where those
changes can be tested in place. As soon as a library is released, that
API is effectively frozen for a long time, because if any project is
using a piece of the API we can't remove it. This is why step 5 is so
important.

SemVer has been proposed as a way to make backwards-incompatible
changes. SemVer lets you declare when those changes have happened, but
it doesn't meet our requirements for installing all of OpenStack on a
single server and handling rolling updates. Our commitment to
supporting single-node installations means we need to have all of
OpenStack installed with a single version of each library, from system
packages, on one box. Managing rolling updates with different versions
of two services are installed on the same server means we need to
replace one of those libraries while keeping both services working.

Those are seriously constraining compatibility requirements, and that
is why the incubator exists and won't just go away.

Think of using the incubator as "static linking" to a library. You get
the benefit of code sharing, along with the benefit that if the API of
that code changes you can decide when to take the changes. When the
module's API is stable, it can be moved to a library and shared more
dynamically.

The Oslo team has worked through the graduation process with some of
the bigger libraries (config and messaging), and we are doing so with
the database code and some of the smaller modules this cycle. Along
the way we're learning just how many assumptions these modules make
about the context in which they are used (lots of global state), how
many edge cases are not documented clearly (nova cells relies on
locking semantics of a thread locking library provided by *file* locks
on shared filesystems), where we have deployer-specific code (the
incubated database code uses an eventlet threadpool implementation
only available in a patched and unreleased version of eventlet), and
how many places we have unit tests for application code mocking out
internal bits of incubated modules (yes).

All of those challenges can be overcome, but it's not as simple as
packaging up what we have and calling it a library. The result of
doing that wouldn't be usable by anyone.

Doug


>
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev