Open Stack

On Aug 29, 2014, at 6:28 PM, Ben Nemec <openstack at nemebean.com> wrote:

> On 08/28/2014 11:14 AM, Doug Hellmann wrote:
>> Before Juno we set a deprecation policy for graduating libraries that said the incubated versions of the modules would stay in the incubator repository for one full cycle after graduation. This gives projects time to adopt the libraries and still receive bug fixes to the incubated version (see https://wiki.openstack.org/wiki/Oslo#Graduation).
>> 
>> That policy worked well early on, but has recently introduced some challenges with the low level modules. Other modules in the incubator are still importing the incubated versions of, for example, timeutils, and so tests that rely on mocking out or modifying the behavior of timeutils do not work as expected when different parts of the application code end up calling different versions of timeutils. We had similar issues with the notifiers and RPC code, and I expect to find other cases as we continue with the graduations.
>> 
>> To deal with this problem, I propose that for Kilo we delete graduating modules as soon as the new library is released, rather than waiting to the end of the cycle. We can update the other incubated modules at the same time, so that the incubator will always use the new libraries and be consistent.
> 
> So from a consumer perspective, this means projects will need to sync
> from stable/juno until they adopt the new libs and then they need to
> sync from master, which will also be using the new libs.

That’s right. I would expect the sync to be part of the adoption process.

> One thing I think is worth noting is the fact that this will require
> projects to adopt all of the libs at once (or at least all of the libs
> that need to match incubator, but that's not always obvious so probably
> safest to just say "all").  It might be possible to sync some modules
> from master and some from stable, but that sounds like a mess waiting to
> happen. :-)

Quite.

> 
> I guess my concern here is that I don't think projects have been
> adopting all of the oslo libs at once, so if, for example, a project was
> looking at adopting oslo.i18n and oslo.utils they may have to do both at
> the same time since adopting one will require them to start syncing from
> master, and then they won't have the ability to use the graduated
> modules anymore.
> 
> This may be a necessary evil, but it does raise the short-term bar for
> adopting any oslo lib, even if the end result will be the same (all of
> the released libs adopted).

True, more below.

> 
>> 
>> We have not had a lot of patches where backports were necessary, but there have been a few important ones, so we need to retain the ability to handle them and allow projects to adopt libraries at a reasonable pace. To handle backports cleanly, we can “freeze” all changes to the master branch version of modules slated for graduation during Kilo (we would need to make a good list very early in the cycle), and use the stable/juno branch for backports.
>> 
>> The new process would be:
>> 
>> 1. Declare which modules we expect to graduate during Kilo.
>> 2. Changes to those pre-graduation modules could be made in the master branch before their library is released, as long as the change is also backported to the stable/juno branch at the same time (we should enforce this by having both patches submitted before accepting either).
>> 3. When graduation for a library starts, freeze those modules in all branches until the library is released.
>> 4. Remove modules from the incubator’s master branch after the library is released.
>> 5. Land changes in the library first.
>> 6. Backport changes, as needed, to stable/juno instead of master.
>> 
>> It would be better to begin the export/import process as early as possible in Kilo to keep the window where point 2 applies very short.
>> 
>> If there are objections to using stable/juno, we could introduce a new branch with a name like backports/kilo, but I am afraid having the extra branch to manage would just cause confusion.
>> 
>> I would like to move ahead with this plan by creating the stable/juno branch and starting to update the incubator as soon as the oslo.log repository is imported (https://review.openstack.org/116934).
>> 
>> Thoughts?
> 
> I think the obvious concern for me is the extra overhead of trying to
> keep one more branch in sync with all the others.  With this we will
> require two commits for each change to incubator code that isn't
> graduating.  Backporting to Havana would require four changes.  I guess
> this is no worse than the situation with graduating code (one commit to
> the lib and one to incubator), but that's temporary pain for specific
> files.  This would continue indefinitely for all files in incubator.
> 
> We could probably help this by requiring changes to be linked in their
> commit messages so reviewers can vote on both changes at once, but it's
> still additional work for everyone so I think it's worth bringing up.
> 
> I don't have a better solution to the incubator-lib mismatch issue so
> I'm okay with going forward on this, but it will introduce some new
> issues that I think we should be aware of going in.

Absolutely. The extra overhead of always back-porting everything in modules not being graduated wasn’t something I considered. I’ll have to look at some stats about the changes we landed in juno before I can get an idea of how much work that might really be.

I see basically 3 options:

1. Keep doing what we’re doing, and wait to delete code 1 full cycle after it graduates. This requires changes to graduated libraries be backported to the incubator only when a project that wants to delay adopting a library but needs a fix, but has the side-effect of conflicts with incubated and graduated versions of some modules described above.

2. Accelerate slightly so we delete graduated code at the end of the cycle when it graduates. This reduces the amount of time where the conflicts need to be resolved and where backports are needed, but doesn’t actually eliminate either problem since resolving the conflicts through code changes within the incubator assumes the libraries are available.

3. Follow the proposal above, and delete code within a cycle as it graduates. This increases our backport load, but rewards projects who adopt libraries during a cycle by making their syncs easier. Depending on how much extra load that really causes, we could mitigate it within oslo by only backporting changes on request. We could also, for example, stop deleting modules after K-3 to make wrapping up the release a little easier.

Are there pros/cons for these options that I’m missing?

Doug