Open Stack

Hi Kevin.
Puppet not only creates config files but also executes a service 
dependent steps, like db sync, so neither '[base] -> [puppet]' nor 
'[base] -> [service]' would not be enough on its own. That requires some 
services specific code to be included into *config* images as well.

PS. There is a related spec [0] created by Dan, please take a look and 
propose you feedback

[0] https://review.openstack.org/620062

On 11/30/18 6:48 PM, Fox, Kevin M wrote:
> Still confused by:
> [base] -> [service] -> [+ puppet]
> not:
> [base] -> [puppet]
> and
> [base] -> [service]
> ?
> 
> Thanks,
> Kevin
> ________________________________________
> From: Bogdan Dobrelya [bdobreli at redhat.com]
> Sent: Friday, November 30, 2018 5:31 AM
> To: Dan Prince; openstack-dev at lists.openstack.org; openstack-discuss at lists.openstack.org
> Subject: Re: [openstack-dev] [TripleO][Edge] Reduce base layer of containers for security and size of images (maintenance) sakes
> 
> On 11/30/18 1:52 PM, Dan Prince wrote:
>> On Fri, 2018-11-30 at 10:31 +0100, Bogdan Dobrelya wrote:
>>> On 11/29/18 6:42 PM, Jiří Stránský wrote:
>>>> On 28. 11. 18 18:29, Bogdan Dobrelya wrote:
>>>>> On 11/28/18 6:02 PM, Jiří Stránský wrote:
>>>>>> <snip>
>>>>>>
>>>>>>> Reiterating again on previous points:
>>>>>>>
>>>>>>> -I'd be fine removing systemd. But lets do it properly and
>>>>>>> not via 'rpm
>>>>>>> -ev --nodeps'.
>>>>>>> -Puppet and Ruby *are* required for configuration. We can
>>>>>>> certainly put
>>>>>>> them in a separate container outside of the runtime service
>>>>>>> containers
>>>>>>> but doing so would actually cost you much more
>>>>>>> space/bandwidth for each
>>>>>>> service container. As both of these have to get downloaded to
>>>>>>> each node
>>>>>>> anyway in order to generate config files with our current
>>>>>>> mechanisms
>>>>>>> I'm not sure this buys you anything.
>>>>>>
>>>>>> +1. I was actually under the impression that we concluded
>>>>>> yesterday on
>>>>>> IRC that this is the only thing that makes sense to seriously
>>>>>> consider.
>>>>>> But even then it's not a win-win -- we'd gain some security by
>>>>>> leaner
>>>>>> production images, but pay for it with space+bandwidth by
>>>>>> duplicating
>>>>>> image content (IOW we can help achieve one of the goals we had
>>>>>> in mind
>>>>>> by worsening the situation w/r/t the other goal we had in
>>>>>> mind.)
>>>>>>
>>>>>> Personally i'm not sold yet but it's something that i'd
>>>>>> consider if we
>>>>>> got measurements of how much more space/bandwidth usage this
>>>>>> would
>>>>>> consume, and if we got some further details/examples about how
>>>>>> serious
>>>>>> are the security concerns if we leave config mgmt tools in
>>>>>> runtime
>>>>>> images.
>>>>>>
>>>>>> IIRC the other options (that were brought forward so far) were
>>>>>> already
>>>>>> dismissed in yesterday's IRC discussion and on the reviews.
>>>>>> Bin/lib bind
>>>>>> mounting being too hacky and fragile, and nsenter not really
>>>>>> solving the
>>>>>> problem (because it allows us to switch to having different
>>>>>> bins/libs
>>>>>> available, but it does not allow merging the availability of
>>>>>> bins/libs
>>>>>> from two containers into a single context).
>>>>>>
>>>>>>> We are going in circles here I think....
>>>>>>
>>>>>> +1. I think too much of the discussion focuses on "why it's bad
>>>>>> to have
>>>>>> config tools in runtime images", but IMO we all sorta agree
>>>>>> that it
>>>>>> would be better not to have them there, if it came at no cost.
>>>>>>
>>>>>> I think to move forward, it would be interesting to know: if we
>>>>>> do this
>>>>>> (i'll borrow Dan's drawing):
>>>>>>
>>>>>>> base container| --> |service container| --> |service
>>>>>>> container w/
>>>>>> Puppet installed|
>>>>>>
>>>>>> How much more space and bandwidth would this consume per node
>>>>>> (e.g.
>>>>>> separately per controller, per compute). This could help with
>>>>>> decision
>>>>>> making.
>>>>>
>>>>> As I've already evaluated in the related bug, that is:
>>>>>
>>>>> puppet-* modules and manifests ~ 16MB
>>>>> puppet with dependencies ~61MB
>>>>> dependencies of the seemingly largest a dependency, systemd
>>>>> ~190MB
>>>>>
>>>>> that would be an extra layer size for each of the container
>>>>> images to be
>>>>> downloaded/fetched into registries.
>>>>
>>>> Thanks, i tried to do the math of the reduction vs. inflation in
>>>> sizes
>>>> as follows. I think the crucial point here is the layering. If we
>>>> do
>>>> this image layering:
>>>>
>>>>> base| --> |+ service| --> |+ Puppet|
>>>>
>>>> we'd drop ~267 MB from base image, but we'd be installing that to
>>>> the
>>>> topmost level, per-component, right?
>>>
>>> Given we detached systemd from puppet, cronie et al, that would be
>>> 267-190MB, so the math below would be looking much better
>>
>> Would it be worth writing a spec that summarizes what action items are
>> bing taken to optimize our base image with regards to the systemd?
> 
> Perhaps it would be. But honestly, I see nothing biggie to require a
> full blown spec. Just changing RPM deps and layers for containers
> images. I'm tracking systemd changes here [0],[1],[2], btw (if accepted,
> it should be working as of fedora28(or 29) I hope)
> 
> [0] https://review.rdoproject.org/r/#/q/topic:base-container-reduction
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1654659
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1654672
> 
> 
>>
>> It seems like the general consenses is that cleaning up some of the RPM
>> dependencies so that we don't install Systemd is the biggest win.
>>
>> What confuses me is why are there still patches posted to move Puppet
>> out of the base layer when we agree moving it out of the base layer
>> would actually cause our resulting container image set to be larger in
>> size.
>>
>> Dan
>>
>>
>>>
>>>> In my basic deployment, undercloud seems to have 17 "components"
>>>> (49
>>>> containers), overcloud controller 15 components (48 containers),
>>>> and
>>>> overcloud compute 4 components (7 containers). Accounting for
>>>> overlaps,
>>>> the total number of "components" used seems to be 19. (By
>>>> "components"
>>>> here i mean whatever uses a different ConfigImage than other
>>>> services. I
>>>> just eyeballed it but i think i'm not too far off the correct
>>>> number.)
>>>>
>>>> So we'd subtract 267 MB from base image and add that to 19 leaf
>>>> images
>>>> used in this deployment. That means difference of +4.8 GB to the
>>>> current
>>>> image sizes. My /var/lib/registry dir on undercloud with all the
>>>> images
>>>> currently has 5.1 GB. We'd almost double that to 9.9 GB.
>>>>
>>>> Going from 5.1 to 9.9 GB seems like a lot of extra traffic for the
>>>> CDNs
>>>> (both external and e.g. internal within OpenStack Infra CI clouds).
>>>>
>>>> And for internal traffic between local registry and overcloud
>>>> nodes, it
>>>> gives +3.7 GB per controller and +800 MB per compute. That may not
>>>> be so
>>>> critical but still feels like a considerable downside.
>>>>
>>>> Another gut feeling is that this way of image layering would take
>>>> longer
>>>> time to build and to run the modify-image Ansible role which we use
>>>> in
>>>> CI, so that could endanger how our CI jobs fit into the time limit.
>>>> We
>>>> could also probably measure this but i'm not sure if it's worth
>>>> spending
>>>> the time.
>>>>
>>>> All in all i'd argue we should be looking at different options
>>>> still.
>>>>
>>>>> Given that we should decouple systemd from all/some of the
>>>>> dependencies
>>>>> (an example topic for RDO [0]), that could save a 190MB. But it
>>>>> seems we
>>>>> cannot break the love of puppet and systemd as it heavily relies
>>>>> on the
>>>>> latter and changing packaging like that would higly likely affect
>>>>> baremetal deployments with puppet and systemd co-operating.
>>>>
>>>> Ack :/
>>>>
>>>>> Long story short, we cannot shoot both rabbits with a single
>>>>> shot, not
>>>>> with puppet :) May be we could with ansible replacing puppet
>>>>> fully...
>>>>> So splitting config and runtime images is the only choice yet to
>>>>> address
>>>>> the raised security concerns. And let's forget about edge cases
>>>>> for now.
>>>>> Tossing around a pair of extra bytes over 40,000 WAN-distributed
>>>>> computes ain't gonna be our the biggest problem for sure.
>>>>>
>>>>> [0]
>>>>> https://review.rdoproject.org/r/#/q/topic:base-container-reduction
>>>>>
>>>>>>> Dan
>>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Jirka
>>>>>>
>>>>>> _______________________________________________________________
>>>>>> ___________
>>>>>>
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>> ___________________________________________________________________
>>>> _______
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsu
>>>> bscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
> 
> 
> --
> Best regards,
> Bogdan Dobrelya,
> Irc #bogdando
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando