On Thu, Apr 16, 2020 at 10:50 AM Dmitry Tantsur <dtantsur@redhat.com> wrote:
Hi,
On Wed, Apr 15, 2020 at 9:55 PM Ben Nemec <openstack@nemebean.com> wrote:
On 4/7/20 5:15 AM, Dmitry Tantsur wrote:
Hi,
On Mon, Apr 6, 2020 at 11:06 PM Ben Nemec <openstack@nemebean.com <mailto:openstack@nemebean.com>> wrote:
At the risk of getting defensive, I do want to make some points relating to Oslo specifically here.
<snip>
> 1) oslo.config is a fantastic piece of software that the whole python > world could benefit from. Same for oslo.service, probably.
oslo.config is an interesting case because there is definitely
interest
outside OpenStack thanks to things like the Castellan config
backend,
but as Doug mentioned oslo.config is heavily opinionated (global
config
object, anyone?) and that's an issue for a lot of people. I will
also
point out that the only oslo.* dependency that oslo.config has is oslo.i18n, which itself has no other oslo dependencies, so there's
not
much barrier to entry to using it standalone today.
oslo.config currently pulls in 16 dependencies to a clean venv, including oslo.18n, Babel (why?) and requests (WHY?).
Well, it's got translated strings and thus i18n, Babel is used for lazy translation, and requests is used for the remote file driver. It's posssible the latter two could/should be optional deps, but without them you do lose functionality.
Please pardon my ignorance, it's the first time I hear about this driver? How many people are using it in the wild? Will it hurt much if we treat this dependencies similar to database/tooz backend, i.e. as optional?
More on Babel below.
I would not inflict oslo.service on anyone I liked. :-P
So, you don't like us? :-P
I don't _think_ I'm responsible for anyone using oslo.service. If I am, it happened years ago before I had dug into oslo.service much. ;-)
Seriously though, I would advocate for using cotyledon if you're looking for a general purpose service library. It's not eventlet-based and provides a lot of the functionality of oslo.service, at least as I understand it. It was also written by an ex-Oslo contributor
outside of
OpenStack so maybe it's an interesting case study for this discussion. I don't know how much contribution it gets from other sources, but
that
would be good to look into.
Good to know, thanks!
> 2) oslo.utils as a catch-all repository of utilities should IMO
be
> either moved to existing python projects or decomposed into small > generally reusable libraries (essentially, each sub-module could be a > library of its own). Same for oslo.concurrency.
oslo.concurrency has already been decomposed into general purpose
code
(fasteners) and OpenStack-specific, at least for the most part. I'm sure the split isn't perfect, but it's not like we haven't recognized the need for better concurrency libraries in Python as a whole. Note
that
fasteners also lives outside Oslo in another ex-Osloers personal
repo.
Again, I'm not sure whether that was beneficial or not but it might
be
worth reaching out to Mehdi and Josh to see how they feel about it.
I *think* we mostly use oslo.concurrency for its execute() wrapper, which seems like it could be a very useful library on its own (can I suggest better-execute as a name?).
Certainly something we could explore. Note that quite a few projects also use interprocess locks from o.c, in which case they're implicitly picking up some standard config opts for setting things like lock_path. I don't know if that's the case for Ironic though.
We only use in-process locks.
I would probably -2 any attempt to split oslo.utils. You'd end up with a bunch of single file libraries and a metric ****-ton of
administrative
overhead. It's bad enough managing 40 some repos as it is. Also,
that's
another library with minimal cross-dependencies with the rest of
Oslo
(just oslo.i18n again), which was an intentional design decision to make it relatively painless to pull in.
I guess I have a personal dislike of libraries without a simple goal (I've been also fighting with various "utils" modules inside Ironic - with only limited luck).
I guess I should have noted that I agree "utils" and "misc" modules tend to be a code smell, but in this case we had two bad options available and had to pick one. IMHO we've done a reasonable job of keeping oslo.utils pretty minimal, but obviously I'm a bit biased. :-)
I guess I'm fine with that if we stop pulling in Babel :)
By the way, only oslo.i18n imports Babel directly. I'm proposing patches to various projects that move it out of requirements.txt. If you see one - please review.
E.g. a shameless ping for osc-lib folks: https://review.opendev.org/717737
I do suspect that at least some of oslo.utils contents could find a new home, some maybe even in the stdlib. I also don't want us to end up in the leafpad situation, on the other hand.
That's entirely possible. We kind of lost our stdlib guy when Victor left the Oslo team so there may be things we've overlooked that could have been upstreamed.
> 3) I'm genuinely not sure why oslo.log and oslo.i18n exist and
which
> parts of their functionality cannot be moved upstream.
oslo.log basically provides convenience functions for OpenStack services, which is kind of what the oslo.* libraries should do. It provides built-in support for things like context objects, which are fairly domain-specific and would be difficult to generalize. It also provides a common set of configuration options so each project
doesn't
have to write their own. We don't need 20 different ways to enable debug logging. :-)
oslo.i18n is mostly for lazy translation, which apparently is a
thing
people still use even though the company pushing for it in the first place no longer cares. We've also had calls to remove it completely because it's finicky and tends to break things if the consuming projects don't follow the best practices with translated strings. So it's in
a
weird limbo state.
May I join these calls please? :) Will I break Zanata (assuming anybody still translates ironic projects) if I switch to the upstream gettext?
I don't think you'll break Zanata, I think you'll break anyone who was using lazy translation. I know there are users of it out there, but maybe we should look into getting a user survey question for Oslo that would give us an idea of how widely its used.
I meant changing ironic, which honestly doesn't have real translations any more..
I did talk with JP in Shanghai about possibly making it optional because it pulls in a fair amount of translation files which can bloat
minimal
containers. I looked into it briefly and I think it would be
possible,
although I don't remember a lot of details because nobody was really pushing for it anymore so it's hard to justify spending a bunch of
time
on it.
I'd be curious to hear more.
Upon reflection, I think the thing we were looking to make optional was the Babel dep of oslo.i18n. It pulls in something like 6 MB of translation files
Oh, yeah. When building our ramdisk, we have to manually delete these files to make it smaller.
and is only used to look up the available languages for lazy translation. I think code changes were needed though, it wasn't as simple as moving Babel to optional. There's probably an argument that we shouldn't be making that call when lazy translation is disabled anyway though.
Is it something I can help with?
It seems that Babel is only used in get_available_languages. Can we make it an optional dependency for projects that do use this call (ironic does not)?
I've prepared a patch: https://review.opendev.org/#/c/720404/ comments are welcome. It seems that we need to update all projects using get_available_languages (mostly "core" projects) to depend on Babel explicity. Dmitry
Dmitry
Dmitry
TLDR: I think Oslo provides value both in and out of OpenStack (shocking, I know!). I'm sure the separation isn't perfect, but
what is?
There are some projects that have been split out of it that might be interesting case studies for this discussion if anyone wants to
follow
up. Not it. ;-)
-Ben