Open Stack

Excerpts from Sean Dague's message of 2017-05-03 16:16:29 -0400:
> Screen is going away in Queens.
> 
> Making the dev / test runtimes as similar as possible is really
> important. And there is so much weird debt around trying to make screen
> launch things reliably (like random sleeps) because screen has funny
> races in it.
> 
> It does mean some tricks people figured out in screen are going away.

It sounds like maybe we should start building a shared repository of new
tips & tricks for systemd/journald.

Doug

> 
> Journalctl provides some pretty serious improvements in querying logs
> https://www.freedesktop.org/software/systemd/man/journalctl.html - you
> can search in time ranges, filter by units (one or more of them), and if
> we get to the bottom of the eventlet interaction, we'll be able to
> search by things like REQUEST_ID as well.
> 
> Plus every modern Linux system uses systemd now, so skills learned
> around systemd and journalctl are transferable both from OpenStack to
> other systems, as well as for new people coming in that understand how
> this works outside of OpenStack. So it helps remove a difference from
> the way we do things from the rest of the world.
> 
>     -Sean
> 
> On 05/03/2017 04:02 PM, Hongbin Lu wrote:
> > Hi Sean,
> > 
> > I tried the new systemd devstack and frankly I don't like it. There are several handy operations in screen that seems to be impossible after switching to systemd. For example, freeze a process by "Ctrl + a + [". In addition, navigating though the logs seems difficult (perhaps I am not familiar with journalctl).
> > 
> > From my understanding, the plan is dropping screen entirely in devstack? I would argue that it is better to keep both screen and systemd, and let users choose one of them based on their preference.
> > 
> > Best regards,
> > Hongbin
> > 
> >> -----Original Message-----
> >> From: Sean Dague [mailto:sean at dague.net]
> >> Sent: May-03-17 6:10 AM
> >> To: openstack-dev at lists.openstack.org
> >> Subject: Re: [openstack-dev] [devstack] [all] systemd in devstack by
> >> default
> >>
> >> On 05/02/2017 08:30 AM, Sean Dague wrote:
> >>> We started running systemd for devstack in the gate yesterday, so far
> >>> so good.
> >>>
> >>> The following patch (which will hopefully land soon), will convert
> >> the
> >>> default local use of devstack to systemd as well -
> >>> https://review.openstack.org/#/c/461716/. It also includes
> >>> substantially updated documentation.
> >>>
> >>> Once you take this patch, a "./clean.sh" is recommended. Flipping
> >>> modes can cause some cruft to build up, and ./clean.sh should be
> >>> pretty good at eliminating them.
> >>>
> >>> https://review.openstack.org/#/c/461716/2/doc/source/development.rst
> >>> is probably specifically interesting / useful for people to read, as
> >>> it shows how the standard development workflows will change (for the
> >>> better) with systemd.
> >>>
> >>>     -Sean
> >>
> >> As a follow up, there are definitely a few edge conditions we've hit
> >> with some jobs, so the following is provided as information in case you
> >> have a job that seems to fail in one of these ways.
> >>
> >> Doing process stop / start
> >> ==========================
> >>
> >> The nova live migration job is special, it was restarting services
> >> manually, however it was doing so with some copy / pasted devstack code,
> >> which means it didn't evolve with the rest of devstack. So the stop
> >> code stopped working (and wasn't robust enough to make it clear that
> >> was the issue).
> >>
> >> https://review.openstack.org/#/c/461803/ is the fix (merged)
> >>
> >> run_process limitations
> >> =======================
> >>
> >> When doing the systemd conversion I looked for a path forward which was
> >> going to make 90% of everything just work. The key trick here was that
> >> services start as the "stack" user, and aren't daemonizing away from
> >> the console. We can take the run_process command and make that the
> >> ExecStart in a unit file.
> >>
> >> *Except* that only works if the command is specified by an *absolute
> >> path*.
> >>
> >> So things like this in kuryr-libnetwork become an issue
> >> https://github.com/openstack/kuryr-
> >> libnetwork/blob/3e2891d6fc5d55b3712258c932a5a8b9b323f6c2/devstack/plugi
> >> n.sh#L148
> >>
> >> There is also a second issue there, which is calling sudo in the
> >> run_process line. If you need to run as a user/group different than the
> >> default, you need to specify that directly.
> >>
> >> The run_process command now supports that -
> >> https://github.com/openstack-
> >> dev/devstack/blob/803acffcf9254e328426ad67380a99f4f5b164ec/functions-
> >> common#L1531-L1535
> >>
> >> And lastly, run_process really always did expect that the thing you
> >> started remained attached to the console. These are run as "simple"
> >> services in systemd. If you are running a thing which already
> >> daemonizes systemd is going to assume (correctly in this simple mode)
> >> the fact that the process detatched from it means it died, and kill and
> >> clean it up.
> >>
> >> This is the issue the OpenDaylight plugin ran into.
> >> https://review.openstack.org/#/c/461889/ is the proposed fix.
> >>
> >>
> >>
> >> If you run into any other issues please pop into #openstack-qa (or
> >> respond to this email) and we'll try to work through them.
> >>
> >>     -Sean
> >>
> >> --
> >> Sean Dague
> >> http://dague.net
> >>
> >> _______________________________________________________________________
> >> ___
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: OpenStack-dev-
> >> request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
>