Open Stack

On 4/7/20 5:15 AM, Dmitry Tantsur wrote:
> Hi,
> 
> On Mon, Apr 6, 2020 at 11:06 PM Ben Nemec <openstack at nemebean.com 
> <mailto:openstack at nemebean.com>> wrote:
> 
>     At the risk of getting defensive, I do want to make some points
>     relating
>     to Oslo specifically here.
> 
> 
> I really do hope nothing I've said comes as offensive to the Oslo team. 
> I'm not blaming you, we're all in this together :)
> 
> 
>     TLDR at the bottom if your eyes glaze over at the wall of text.
> 
>     On 4/6/20 3:10 AM, Dmitry Tantsur wrote:
>      > With absolutely no disrespect meant to the awesome Oslo team, I
>     think
>      > the existence of Oslo libraries is a bad sign. I think as a
>     strong FOSS
>      > community we shouldn't invest into libraries that are either
>     useful only
>      > to us or at least are marketed this way. For example:
> 
>     I think it's relevant to keep in mind that Oslo didn't start out as a
>     collection of libraries, it started out as a bunch of code forklifted
>     from Nova for use by other OpenStack services. It was a significant
>     effort just to get the Nova-isms out of it, much less trying to
>     sanitize
>     everything OpenStack-specific.
> 
>     I also don't think it's fair to characterize Oslo as only focused on
>     OpenStack today. In reality, many of our new libraries since the
>     initial
>     incubator split have been general purpose as often as not. Where they
>     haven't, it's things like oslo.limit that are explicitly dependent
>     on an
>     OpenStack service (Keystone in this case). We believe in being good OSS
>     citizens as much as anyone else.
> 
>     There's also a boil the ocean problem with trying to generalize every
>     solution to every problem. It's a question we ask every time a new
>     library is proposed, but in some cases it just doesn't make sense to
>     write a library for an audience that may or may not exist. And in some
>     cases when such an audience appears, we have refactored libraries to
>     make them more general purpose, while often keeping an
>     OpenStack-specific layer to ease integration with OpenStack services.
>     See oslo.concurrency and fasteners.
> 
>     In fact, that kind of split is a pretty common pattern, even in cases
>     where the underlying library didn't originate in Oslo/OpenStack. Think
>     sqlalchemy/oslo.db, dogpile/oslo.cache, kombu/oslo.messaging. A lot of
>     Oslo is glue code to make it easier to use something in a common way
>     across OpenStack services.
> 
>      > 1) oslo.config is a fantastic piece of software that the whole
>     python
>      > world could benefit from. Same for oslo.service, probably.
> 
>     oslo.config is an interesting case because there is definitely interest
>     outside OpenStack thanks to things like the Castellan config backend,
>     but as Doug mentioned oslo.config is heavily opinionated (global config
>     object, anyone?) and that's an issue for a lot of people. I will also
>     point out that the only oslo.* dependency that oslo.config has is
>     oslo.i18n, which itself has no other oslo dependencies, so there's not
>     much barrier to entry to using it standalone today.
> 
> 
> oslo.config currently pulls in 16 dependencies to a clean venv, 
> including oslo.18n, Babel (why?) and requests (WHY?).

Well, it's got translated strings and thus i18n, Babel is used for lazy 
translation, and requests is used for the remote file driver. It's 
posssible the latter two could/should be optional deps, but without them 
you do lose functionality.

> 
> 
>     I would not inflict oslo.service on anyone I liked. :-P
> 
> 
> So, you don't like us? :-P

I don't _think_ I'm responsible for anyone using oslo.service. If I am, 
it happened years ago before I had dug into oslo.service much. ;-)

> 
> 
>     Seriously though, I would advocate for using cotyledon if you're
>     looking
>     for a general purpose service library. It's not eventlet-based and
>     provides a lot of the functionality of oslo.service, at least as I
>     understand it. It was also written by an ex-Oslo contributor outside of
>     OpenStack so maybe it's an interesting case study for this
>     discussion. I
>     don't know how much contribution it gets from other sources, but that
>     would be good to look into.
> 
> 
> Good to know, thanks!
> 
> 
>      > 2) oslo.utils as a catch-all repository of utilities should IMO be
>      > either moved to existing python projects or decomposed into small
>      > generally reusable libraries (essentially, each sub-module could
>     be a
>      > library of its own). Same for oslo.concurrency.
> 
>     oslo.concurrency has already been decomposed into general purpose code
>     (fasteners) and OpenStack-specific, at least for the most part. I'm
>     sure
>     the split isn't perfect, but it's not like we haven't recognized the
>     need for better concurrency libraries in Python as a whole. Note that
>     fasteners also lives outside Oslo in another ex-Osloers personal repo.
>     Again, I'm not sure whether that was beneficial or not but it might be
>     worth reaching out to Mehdi and Josh to see how they feel about it.
> 
> 
> I *think* we mostly use oslo.concurrency for its execute() wrapper, 
> which seems like it could be a very useful library on its own (can I 
> suggest better-execute as a name?).

Certainly something we could explore. Note that quite a few projects 
also use interprocess locks from o.c, in which case they're implicitly 
picking up some standard config opts for setting things like lock_path. 
I don't know if that's the case for Ironic though.

> 
> 
>     I would probably -2 any attempt to split oslo.utils. You'd end up
>     with a
>     bunch of single file libraries and a metric ****-ton of administrative
>     overhead. It's bad enough managing 40 some repos as it is. Also, that's
>     another library with minimal cross-dependencies with the rest of Oslo
>     (just oslo.i18n again), which was an intentional design decision to
>     make
>     it relatively painless to pull in.
> 
> 
> I guess I have a personal dislike of libraries without a simple goal 
> (I've been also fighting with various "utils" modules inside Ironic - 
> with only limited luck).

I guess I should have noted that I agree "utils" and "misc" modules tend 
to be a code smell, but in this case we had two bad options available 
and had to pick one. IMHO we've done a reasonable job of keeping 
oslo.utils pretty minimal, but obviously I'm a bit biased. :-)

> 
> I do suspect that at least some of oslo.utils contents could find a new 
> home, some maybe even in the stdlib. I also don't want us to end up in 
> the leafpad situation, on the other hand.

That's entirely possible. We kind of lost our stdlib guy when Victor 
left the Oslo team so there may be things we've overlooked that could 
have been upstreamed.

> 
> 
>      > 3) I'm genuinely not sure why oslo.log and oslo.i18n exist and which
>      > parts of their functionality cannot be moved upstream.
> 
>     oslo.log basically provides convenience functions for OpenStack
>     services, which is kind of what the oslo.* libraries should do. It
>     provides built-in support for things like context objects, which are
>     fairly domain-specific and would be difficult to generalize. It also
>     provides a common set of configuration options so each project doesn't
>     have to write their own. We don't need 20 different ways to enable
>     debug
>     logging. :-)
> 
>     oslo.i18n is mostly for lazy translation, which apparently is a thing
>     people still use even though the company pushing for it in the first
>     place no longer cares. We've also had calls to remove it completely
>     because it's finicky and tends to break things if the consuming
>     projects
>     don't follow the best practices with translated strings. So it's in a
>     weird limbo state.
> 
> 
> May I join these calls please? :) Will I break Zanata (assuming anybody 
> still translates ironic projects) if I switch to the upstream gettext?

I don't think you'll break Zanata, I think you'll break anyone who was 
using lazy translation. I know there are users of it out there, but 
maybe we should look into getting a user survey question for Oslo that 
would give us an idea of how widely its used.

> 
> 
>     I did talk with JP in Shanghai about possibly making it optional
>     because
>     it pulls in a fair amount of translation files which can bloat minimal
>     containers. I looked into it briefly and I think it would be possible,
>     although I don't remember a lot of details because nobody was really
>     pushing for it anymore so it's hard to justify spending a bunch of time
>     on it.
> 
> 
> I'd be curious to hear more.

Upon reflection, I think the thing we were looking to make optional was 
the Babel dep of oslo.i18n. It pulls in something like 6 MB of 
translation files and is only used to look up the available languages 
for lazy translation. I think code changes were needed though, it wasn't 
as simple as moving Babel to optional. There's probably an argument that 
we shouldn't be making that call when lazy translation is disabled 
anyway though.

> 
> Dmitry
> 
> 
>     TLDR: I think Oslo provides value both in and out of OpenStack
>     (shocking, I know!). I'm sure the separation isn't perfect, but what is?
> 
>     There are some projects that have been split out of it that might be
>     interesting case studies for this discussion if anyone wants to follow
>     up. Not it. ;-)
> 
>     -Ben
>