Open Stack

On Mon, Jul 7, 2014 at 6:33 AM, Sean Dague <sean at dague.net> wrote:
> On 07/04/2014 10:54 AM, Mark McLoughlin wrote:
>> On Fri, 2014-07-04 at 15:31 +0200, Ihar Hrachyshka wrote:
>>> Hi all,
>>> at the moment we have several bot jobs that sync contents to affected
>>> projects:
>>>
>>> - translations are copied from transifex;
>>> - requirements are copied from global requirements repo.
>>>
>>> We have another source of common code - oslo-incubator, though we
>>> still rely on people manually copying the new code from there to
>>> affected projects. This results in old, buggy, and sometimes
>>> completely different versions of the same code in all projects.
>>>
>>> I wonder why don't we set another bot to sync code from incubator? In
>>> that way, we would:
>>> - reduce work to do for developers [I hope everyone knows how boring
>>> it is to fill in commit message with all commits synchronized and
>>> create sync requests for > 10 projects at once];
>>> - make sure all projects use (almost) the same code;
>>> - ensure projects are notified in advance in case API changed in one
>>> of the modules that resulted in failures in gate;
>>> - our LOC statistics will be a bit more fair ;) (currently, the one
>>> who syncs a large piece of code from incubator to a project, gets all
>>> the LOC credit at e.g. stackalytics.com).
>>>
>>> The changes will still be gated, so any failures and incompatibilities
>>> will be caught. I even don't expect most of sync requests to fail at
>>> all, meaning it will be just a matter of two +2's from cores.
>>>
>>> I know that Oslo team works hard to graduate lots of modules from
>>> incubator to separate libraries with stable API. Still, I guess we'll
>>> live with incubator at least another cycle or two.
>>>
>>> What are your thoughts on that?
>>
>> Just repeating what I said on IRC ...
>>
>> The point of oslo-incubator is that it's a place where APIs can be
>> cleaned up so that they are ready for graduation. Code living in
>> oslo-incubator for a long time with unchanging APIs is not the idea. An
>> automated sync job would IMHO discourage API cleanup work. I'd expect
>> people would start adding lots of ugly backwards API compat hacks with
>> their API cleanups just to stop people complaining about failing
>> auto-syncs. That would be the opposite of what we're trying to achieve.
>
> The problem is in recent times we've actually seen the opposite happen.
> Code goes into oslo-incubator working. It gets "cleaned up". It syncs
> back to the projects and break things. olso.db was a good instance of that.
>
> Because during the get "cleaned up" phase it's not being tested in the
> whole system. It's only being unit tested.
>
> Basically code goes from working in place, drops 95% of it's testing,
> then gets refactored. Which is exactly what you don't want to be doing
> to refactor code.
>
> So I think the set of trade offs for oslo looked a lot different when
> only a couple projects were using it, and the amount of code is small,
> vs. where we stand now.

That's exactly the problem. We have too many projects copying code
that should be moved to stable libraries. We're addressing that
problem for existing code, but it takes a lot of coordination work and
time for each library and we have a small review team. In the future,
I would like for newly incubated code to graduate more quickly and
before more than a couple of projects have adopted it. That will make
evolving APIs more difficult, but as you point out we can use
compatibility layers where needed.

>
> What it's produced is I think the opposite of what we're trying to
> achieve (as people are pretty gunshy now on oslo syncs), because the
> openstack/ tree across projects is never the same. So you'll have 12
> different versions of log.py in a production system.
>
> What I really want is forward testing of oslo interfaces. Because most

The spec for the testing oslo lib changes with other projects' unit
tests is up for review https://review.openstack.org/#/c/95885/

We're already running the integration tests using master for the libraries.

> of the breaks in oslo weren't because there was a very strong view that
> an certain interface or behavior needed to change. It was because after
> all the testing was removed from the code, and the people working on it
> in oslo didn't have the context on how the code was used in a project,
> behavior changed. Not intentionally, just as a side effect.
>
> I think the goal of oslo is really common code for OpenStack. I would
> much rather have all the projects running the same oslo code, even if it
> meant a few compat interfaces in there, than having the wild divergence
> of olso code in the current model.
>
>         -Sean
>
> --
> Sean Dague
> http://dague.net
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>