Open Stack

Still confused by:
[base] -> [service] -> [+ puppet]
not:
[base] -> [puppet]
and
[base] -> [service]
?

Thanks,
Kevin
________________________________________
From: Bogdan Dobrelya [bdobreli at redhat.com]
Sent: Friday, November 30, 2018 5:31 AM
To: Dan Prince; openstack-dev at lists.openstack.org; openstack-discuss at lists.openstack.org
Subject: Re: [openstack-dev] [TripleO][Edge] Reduce base layer of containers for security and size of images (maintenance) sakes

On 11/30/18 1:52 PM, Dan Prince wrote:
> On Fri, 2018-11-30 at 10:31 +0100, Bogdan Dobrelya wrote:
>> On 11/29/18 6:42 PM, Jiří Stránský wrote:
>>> On 28. 11. 18 18:29, Bogdan Dobrelya wrote:
>>>> On 11/28/18 6:02 PM, Jiří Stránský wrote:
>>>>> <snip>
>>>>>
>>>>>> Reiterating again on previous points:
>>>>>>
>>>>>> -I'd be fine removing systemd. But lets do it properly and
>>>>>> not via 'rpm
>>>>>> -ev --nodeps'.
>>>>>> -Puppet and Ruby *are* required for configuration. We can
>>>>>> certainly put
>>>>>> them in a separate container outside of the runtime service
>>>>>> containers
>>>>>> but doing so would actually cost you much more
>>>>>> space/bandwidth for each
>>>>>> service container. As both of these have to get downloaded to
>>>>>> each node
>>>>>> anyway in order to generate config files with our current
>>>>>> mechanisms
>>>>>> I'm not sure this buys you anything.
>>>>>
>>>>> +1. I was actually under the impression that we concluded
>>>>> yesterday on
>>>>> IRC that this is the only thing that makes sense to seriously
>>>>> consider.
>>>>> But even then it's not a win-win -- we'd gain some security by
>>>>> leaner
>>>>> production images, but pay for it with space+bandwidth by
>>>>> duplicating
>>>>> image content (IOW we can help achieve one of the goals we had
>>>>> in mind
>>>>> by worsening the situation w/r/t the other goal we had in
>>>>> mind.)
>>>>>
>>>>> Personally i'm not sold yet but it's something that i'd
>>>>> consider if we
>>>>> got measurements of how much more space/bandwidth usage this
>>>>> would
>>>>> consume, and if we got some further details/examples about how
>>>>> serious
>>>>> are the security concerns if we leave config mgmt tools in
>>>>> runtime
>>>>> images.
>>>>>
>>>>> IIRC the other options (that were brought forward so far) were
>>>>> already
>>>>> dismissed in yesterday's IRC discussion and on the reviews.
>>>>> Bin/lib bind
>>>>> mounting being too hacky and fragile, and nsenter not really
>>>>> solving the
>>>>> problem (because it allows us to switch to having different
>>>>> bins/libs
>>>>> available, but it does not allow merging the availability of
>>>>> bins/libs
>>>>> from two containers into a single context).
>>>>>
>>>>>> We are going in circles here I think....
>>>>>
>>>>> +1. I think too much of the discussion focuses on "why it's bad
>>>>> to have
>>>>> config tools in runtime images", but IMO we all sorta agree
>>>>> that it
>>>>> would be better not to have them there, if it came at no cost.
>>>>>
>>>>> I think to move forward, it would be interesting to know: if we
>>>>> do this
>>>>> (i'll borrow Dan's drawing):
>>>>>
>>>>>> base container| --> |service container| --> |service
>>>>>> container w/
>>>>> Puppet installed|
>>>>>
>>>>> How much more space and bandwidth would this consume per node
>>>>> (e.g.
>>>>> separately per controller, per compute). This could help with
>>>>> decision
>>>>> making.
>>>>
>>>> As I've already evaluated in the related bug, that is:
>>>>
>>>> puppet-* modules and manifests ~ 16MB
>>>> puppet with dependencies ~61MB
>>>> dependencies of the seemingly largest a dependency, systemd
>>>> ~190MB
>>>>
>>>> that would be an extra layer size for each of the container
>>>> images to be
>>>> downloaded/fetched into registries.
>>>
>>> Thanks, i tried to do the math of the reduction vs. inflation in
>>> sizes
>>> as follows. I think the crucial point here is the layering. If we
>>> do
>>> this image layering:
>>>
>>>> base| --> |+ service| --> |+ Puppet|
>>>
>>> we'd drop ~267 MB from base image, but we'd be installing that to
>>> the
>>> topmost level, per-component, right?
>>
>> Given we detached systemd from puppet, cronie et al, that would be
>> 267-190MB, so the math below would be looking much better
>
> Would it be worth writing a spec that summarizes what action items are
> bing taken to optimize our base image with regards to the systemd?

Perhaps it would be. But honestly, I see nothing biggie to require a
full blown spec. Just changing RPM deps and layers for containers
images. I'm tracking systemd changes here [0],[1],[2], btw (if accepted,
it should be working as of fedora28(or 29) I hope)

[0] https://review.rdoproject.org/r/#/q/topic:base-container-reduction
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1654659
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1654672


>
> It seems like the general consenses is that cleaning up some of the RPM
> dependencies so that we don't install Systemd is the biggest win.
>
> What confuses me is why are there still patches posted to move Puppet
> out of the base layer when we agree moving it out of the base layer
> would actually cause our resulting container image set to be larger in
> size.
>
> Dan
>
>
>>
>>> In my basic deployment, undercloud seems to have 17 "components"
>>> (49
>>> containers), overcloud controller 15 components (48 containers),
>>> and
>>> overcloud compute 4 components (7 containers). Accounting for
>>> overlaps,
>>> the total number of "components" used seems to be 19. (By
>>> "components"
>>> here i mean whatever uses a different ConfigImage than other
>>> services. I
>>> just eyeballed it but i think i'm not too far off the correct
>>> number.)
>>>
>>> So we'd subtract 267 MB from base image and add that to 19 leaf
>>> images
>>> used in this deployment. That means difference of +4.8 GB to the
>>> current
>>> image sizes. My /var/lib/registry dir on undercloud with all the
>>> images
>>> currently has 5.1 GB. We'd almost double that to 9.9 GB.
>>>
>>> Going from 5.1 to 9.9 GB seems like a lot of extra traffic for the
>>> CDNs
>>> (both external and e.g. internal within OpenStack Infra CI clouds).
>>>
>>> And for internal traffic between local registry and overcloud
>>> nodes, it
>>> gives +3.7 GB per controller and +800 MB per compute. That may not
>>> be so
>>> critical but still feels like a considerable downside.
>>>
>>> Another gut feeling is that this way of image layering would take
>>> longer
>>> time to build and to run the modify-image Ansible role which we use
>>> in
>>> CI, so that could endanger how our CI jobs fit into the time limit.
>>> We
>>> could also probably measure this but i'm not sure if it's worth
>>> spending
>>> the time.
>>>
>>> All in all i'd argue we should be looking at different options
>>> still.
>>>
>>>> Given that we should decouple systemd from all/some of the
>>>> dependencies
>>>> (an example topic for RDO [0]), that could save a 190MB. But it
>>>> seems we
>>>> cannot break the love of puppet and systemd as it heavily relies
>>>> on the
>>>> latter and changing packaging like that would higly likely affect
>>>> baremetal deployments with puppet and systemd co-operating.
>>>
>>> Ack :/
>>>
>>>> Long story short, we cannot shoot both rabbits with a single
>>>> shot, not
>>>> with puppet :) May be we could with ansible replacing puppet
>>>> fully...
>>>> So splitting config and runtime images is the only choice yet to
>>>> address
>>>> the raised security concerns. And let's forget about edge cases
>>>> for now.
>>>> Tossing around a pair of extra bytes over 40,000 WAN-distributed
>>>> computes ain't gonna be our the biggest problem for sure.
>>>>
>>>> [0]
>>>> https://review.rdoproject.org/r/#/q/topic:base-container-reduction
>>>>
>>>>>> Dan
>>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Jirka
>>>>>
>>>>> _______________________________________________________________
>>>>> ___________
>>>>>
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>> ___________________________________________________________________
>>> _______
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsu
>>> bscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>


--
Best regards,
Bogdan Dobrelya,
Irc #bogdando