Open Stack

Hi Josh,

sorry for the double reply, I seem to be having ML bounce issues.

comments in-line,

On Thu, May 12, 2016 at 4:04 PM, Joshua Harlow <harlowja at fastmail.com> wrote:
>
> Hi there all-ye-operators,
>
> I am investigating how to help move godaddy from rpms to a container-like solution (virtualenvs, lxc, or docker...) and a set of questions that comes up is the following (and I would think that some folks on this mailing list may have some useful insight into the answers):
>
> * Have you done the transition?
Back in the Havana timeframe I attempted a package to source
conversion and it was fugly. I was attempting to convert nodes
in-place and found that the distro packages we're applying "value
added" out of tree patches and those patches caused me no end of pain.

> * How did the transition go?
As mentioned, the transition was painful while cleaning up packages
leaves all sorts of crufty bits on a host the biggest problem I ran
into during that time period was related to the DB. Some of the distro
packages pulled in patches that added DB migrations and those
migrations made going to the OpenStack source hard. While the
conversion was a great learning experience in the end I abandoned the
effort.

> * Was/is kolla used or looked into? or something custom?
Full disclosure, I work for Rackspace on the OpenStack-Ansible
project. The OSA project uses both containers (LXC) and python
virtual-environments. We do both because we want user, file system,
and process isolation which containers gives us and we want to isolate
OpenStack from the operating system which has python dependencies. The
idea is to allow the host operating system to do what it does best and
keep OpenStack as far away from it as possible. This has some great
advantages which we've seen in our ability to scale services
independently of a given hosts while keeping it fully isolated. In the
end our solution is a hybrid one as we run the on metal for
Cinder-Volume (when using the reference LVM driver), Nova-Compute
(when using Linux+KVM), Swift-.* (except proxies).

> * How long did it take to do the transition from a package based solution (with say puppet/chef being used to deploy these packages)?
I cant remember specifically but I think I worked on the effort for ~2
weeks before I decided it wasn't worth continuing. If I were to try it
all again I'd likely have a better time today that I did then but I
still think it'd be a mess. My basic plan of attack today would be to
add nodes to the environment using the new infrastructure and slowly
decommission the old deployment. You'll likely need to identify the
point in time your distro packages currently are and take stock of all
of the patches they may have applied. Then with all of that fully
understood you will need to deploy a version of OpenStack just ahead
of your release which "should" pulled the environment in line with the
community.

>   * Follow-up being how big was the team to do this?
It was just me, I was working on a 15+ node lab.

> * What was the roll-out strategy to achieve the final container solution?
My team at Rackspace created the initial release of what is now known
as the OpenStack-Ansible project. In our evaluation of container
technologies found docker to be a cute tool that was incapable of
doing what we needed while remaining stable/functional so we went with
LXC. Having run LXC for a few years now, 2 of which has been with
production workloads, I've been very happy with the results. This is
our basic reference architecture: [
http://docs.openstack.org/developer/openstack-ansible/install-guide/overvie=
w-hostlayout.html
]. Regardless of your choices in container technologies your going to
have to deal with some limitations. Before going full "container all
the things" I'd look into what your current deployment needs are and
see if there are known limitations going to cause you major headaches
(like AF_NET_LINK not being namespace aware making it impossible to
mount an iscsi target within a container
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1226855 unless you
drop the network namespace).

> Any other feedback (and/or questions that I missed)?
  * Dependency problems are still problems in every container
technology. Having a reliable build environment or a package mirror is
still a must.
  * Docker files are not packages no matter how many docker people
tell you they are. However, a docker file is a really nice way to
express a container runtime and if you treat it like a runtime
expression engine and stay within those lines it works rather well.
  * OverlayFS has had some problems with various under mounts
(example: https://bugzilla.redhat.com/show_bug.cgi?id=3D1319507). We're
using LVM + EXT4 for the container root devices which I've not seen
reliability issues with.
  * BTRFS may not be a good solution either (see the gotchas for more
on that https://btrfs.wiki.kernel.org/index.php/Gotchas)
  * ZFS on linux looks really promising and has a long history of
being awesome but I'm reserving full judgment on that until I can have
a good long look at it.
  * If you're using containers and dropping all of the namespaces to
work around problems or to make your life easier just run a chroot and
save yourself a lot of frustration.
  * Generally a containerized solution will require you to re-imagine
your infrastructure. I know containers are all the hype and everyone
says it's so simple but there's nothing simple about running services
in production that people rely on especially when you consider the
network topology.

If your deployment and operator teams are already accustom to a chef
or puppet ecosystem, I'm assuming you're a chef or puppet shop based
on one of the questions, I'd personally recommend popping into those
communities IRC channels (if you're not already) to see what other
folks are doing. It's likely others are working on similar things and
you may be able to get what you need while helping out other deployers
all without reinventing many wheels.

That's about all I can think of right now and I hope it helps. Best of
luck to you and if you have other questions or just want to chat drop
by the #openstack-ansible channel (even if you don't use
OpenStack-Ansible) I'm always happy to help and there are lots of
other deployers running lots of different things and they may too be
able to give you insight or perspective.

> Thanks,
>
> Josh
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


--

Kevin Carter
IRC: Cloudnull