Open Stack

On 12/3/18 10:34 AM, Bogdan Dobrelya wrote:
> Hi Kevin.
> Puppet not only creates config files but also executes a service 
> dependent steps, like db sync, so neither '[base] -> [puppet]' nor 
> '[base] -> [service]' would not be enough on its own. That requires some 
> services specific code to be included into *config* images as well.
> 
> PS. There is a related spec [0] created by Dan, please take a look and 
> propose you feedback
> 
> [0] https://review.openstack.org/620062

I'm terribly sorry, but that's a corrected link [0] to that spec.

[0] https://review.openstack.org/620909

> 
> On 11/30/18 6:48 PM, Fox, Kevin M wrote:
>> Still confused by:
>> [base] -> [service] -> [+ puppet]
>> not:
>> [base] -> [puppet]
>> and
>> [base] -> [service]
>> ?
>>
>> Thanks,
>> Kevin
>> ________________________________________
>> From: Bogdan Dobrelya [bdobreli at redhat.com]
>> Sent: Friday, November 30, 2018 5:31 AM
>> To: Dan Prince; openstack-dev at lists.openstack.org; 
>> openstack-discuss at lists.openstack.org
>> Subject: Re: [openstack-dev] [TripleO][Edge] Reduce base layer of 
>> containers for security and size of images (maintenance) sakes
>>
>> On 11/30/18 1:52 PM, Dan Prince wrote:
>>> On Fri, 2018-11-30 at 10:31 +0100, Bogdan Dobrelya wrote:
>>>> On 11/29/18 6:42 PM, Jiří Stránský wrote:
>>>>> On 28. 11. 18 18:29, Bogdan Dobrelya wrote:
>>>>>> On 11/28/18 6:02 PM, Jiří Stránský wrote:
>>>>>>> <snip>
>>>>>>>
>>>>>>>> Reiterating again on previous points:
>>>>>>>>
>>>>>>>> -I'd be fine removing systemd. But lets do it properly and
>>>>>>>> not via 'rpm
>>>>>>>> -ev --nodeps'.
>>>>>>>> -Puppet and Ruby *are* required for configuration. We can
>>>>>>>> certainly put
>>>>>>>> them in a separate container outside of the runtime service
>>>>>>>> containers
>>>>>>>> but doing so would actually cost you much more
>>>>>>>> space/bandwidth for each
>>>>>>>> service container. As both of these have to get downloaded to
>>>>>>>> each node
>>>>>>>> anyway in order to generate config files with our current
>>>>>>>> mechanisms
>>>>>>>> I'm not sure this buys you anything.
>>>>>>>
>>>>>>> +1. I was actually under the impression that we concluded
>>>>>>> yesterday on
>>>>>>> IRC that this is the only thing that makes sense to seriously
>>>>>>> consider.
>>>>>>> But even then it's not a win-win -- we'd gain some security by
>>>>>>> leaner
>>>>>>> production images, but pay for it with space+bandwidth by
>>>>>>> duplicating
>>>>>>> image content (IOW we can help achieve one of the goals we had
>>>>>>> in mind
>>>>>>> by worsening the situation w/r/t the other goal we had in
>>>>>>> mind.)
>>>>>>>
>>>>>>> Personally i'm not sold yet but it's something that i'd
>>>>>>> consider if we
>>>>>>> got measurements of how much more space/bandwidth usage this
>>>>>>> would
>>>>>>> consume, and if we got some further details/examples about how
>>>>>>> serious
>>>>>>> are the security concerns if we leave config mgmt tools in
>>>>>>> runtime
>>>>>>> images.
>>>>>>>
>>>>>>> IIRC the other options (that were brought forward so far) were
>>>>>>> already
>>>>>>> dismissed in yesterday's IRC discussion and on the reviews.
>>>>>>> Bin/lib bind
>>>>>>> mounting being too hacky and fragile, and nsenter not really
>>>>>>> solving the
>>>>>>> problem (because it allows us to switch to having different
>>>>>>> bins/libs
>>>>>>> available, but it does not allow merging the availability of
>>>>>>> bins/libs
>>>>>>> from two containers into a single context).
>>>>>>>
>>>>>>>> We are going in circles here I think....
>>>>>>>
>>>>>>> +1. I think too much of the discussion focuses on "why it's bad
>>>>>>> to have
>>>>>>> config tools in runtime images", but IMO we all sorta agree
>>>>>>> that it
>>>>>>> would be better not to have them there, if it came at no cost.
>>>>>>>
>>>>>>> I think to move forward, it would be interesting to know: if we
>>>>>>> do this
>>>>>>> (i'll borrow Dan's drawing):
>>>>>>>
>>>>>>>> base container| --> |service container| --> |service
>>>>>>>> container w/
>>>>>>> Puppet installed|
>>>>>>>
>>>>>>> How much more space and bandwidth would this consume per node
>>>>>>> (e.g.
>>>>>>> separately per controller, per compute). This could help with
>>>>>>> decision
>>>>>>> making.
>>>>>>
>>>>>> As I've already evaluated in the related bug, that is:
>>>>>>
>>>>>> puppet-* modules and manifests ~ 16MB
>>>>>> puppet with dependencies ~61MB
>>>>>> dependencies of the seemingly largest a dependency, systemd
>>>>>> ~190MB
>>>>>>
>>>>>> that would be an extra layer size for each of the container
>>>>>> images to be
>>>>>> downloaded/fetched into registries.
>>>>>
>>>>> Thanks, i tried to do the math of the reduction vs. inflation in
>>>>> sizes
>>>>> as follows. I think the crucial point here is the layering. If we
>>>>> do
>>>>> this image layering:
>>>>>
>>>>>> base| --> |+ service| --> |+ Puppet|
>>>>>
>>>>> we'd drop ~267 MB from base image, but we'd be installing that to
>>>>> the
>>>>> topmost level, per-component, right?
>>>>
>>>> Given we detached systemd from puppet, cronie et al, that would be
>>>> 267-190MB, so the math below would be looking much better
>>>
>>> Would it be worth writing a spec that summarizes what action items are
>>> bing taken to optimize our base image with regards to the systemd?
>>
>> Perhaps it would be. But honestly, I see nothing biggie to require a
>> full blown spec. Just changing RPM deps and layers for containers
>> images. I'm tracking systemd changes here [0],[1],[2], btw (if accepted,
>> it should be working as of fedora28(or 29) I hope)
>>
>> [0] https://review.rdoproject.org/r/#/q/topic:base-container-reduction
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1654659
>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1654672
>>
>>
>>>
>>> It seems like the general consenses is that cleaning up some of the RPM
>>> dependencies so that we don't install Systemd is the biggest win.
>>>
>>> What confuses me is why are there still patches posted to move Puppet
>>> out of the base layer when we agree moving it out of the base layer
>>> would actually cause our resulting container image set to be larger in
>>> size.
>>>
>>> Dan
>>>
>>>
>>>>
>>>>> In my basic deployment, undercloud seems to have 17 "components"
>>>>> (49
>>>>> containers), overcloud controller 15 components (48 containers),
>>>>> and
>>>>> overcloud compute 4 components (7 containers). Accounting for
>>>>> overlaps,
>>>>> the total number of "components" used seems to be 19. (By
>>>>> "components"
>>>>> here i mean whatever uses a different ConfigImage than other
>>>>> services. I
>>>>> just eyeballed it but i think i'm not too far off the correct
>>>>> number.)
>>>>>
>>>>> So we'd subtract 267 MB from base image and add that to 19 leaf
>>>>> images
>>>>> used in this deployment. That means difference of +4.8 GB to the
>>>>> current
>>>>> image sizes. My /var/lib/registry dir on undercloud with all the
>>>>> images
>>>>> currently has 5.1 GB. We'd almost double that to 9.9 GB.
>>>>>
>>>>> Going from 5.1 to 9.9 GB seems like a lot of extra traffic for the
>>>>> CDNs
>>>>> (both external and e.g. internal within OpenStack Infra CI clouds).
>>>>>
>>>>> And for internal traffic between local registry and overcloud
>>>>> nodes, it
>>>>> gives +3.7 GB per controller and +800 MB per compute. That may not
>>>>> be so
>>>>> critical but still feels like a considerable downside.
>>>>>
>>>>> Another gut feeling is that this way of image layering would take
>>>>> longer
>>>>> time to build and to run the modify-image Ansible role which we use
>>>>> in
>>>>> CI, so that could endanger how our CI jobs fit into the time limit.
>>>>> We
>>>>> could also probably measure this but i'm not sure if it's worth
>>>>> spending
>>>>> the time.
>>>>>
>>>>> All in all i'd argue we should be looking at different options
>>>>> still.
>>>>>
>>>>>> Given that we should decouple systemd from all/some of the
>>>>>> dependencies
>>>>>> (an example topic for RDO [0]), that could save a 190MB. But it
>>>>>> seems we
>>>>>> cannot break the love of puppet and systemd as it heavily relies
>>>>>> on the
>>>>>> latter and changing packaging like that would higly likely affect
>>>>>> baremetal deployments with puppet and systemd co-operating.
>>>>>
>>>>> Ack :/
>>>>>
>>>>>> Long story short, we cannot shoot both rabbits with a single
>>>>>> shot, not
>>>>>> with puppet :) May be we could with ansible replacing puppet
>>>>>> fully...
>>>>>> So splitting config and runtime images is the only choice yet to
>>>>>> address
>>>>>> the raised security concerns. And let's forget about edge cases
>>>>>> for now.
>>>>>> Tossing around a pair of extra bytes over 40,000 WAN-distributed
>>>>>> computes ain't gonna be our the biggest problem for sure.
>>>>>>
>>>>>> [0]
>>>>>> https://review.rdoproject.org/r/#/q/topic:base-container-reduction
>>>>>>
>>>>>>>> Dan
>>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Jirka
>>>>>>>
>>>>>>> _______________________________________________________________
>>>>>>> ___________
>>>>>>>
>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>> Unsubscribe:
>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>> ___________________________________________________________________
>>>>> _______
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsu
>>>>> bscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>
>>
>> -- 
>> Best regards,
>> Bogdan Dobrelya,
>> Irc #bogdando
>>
> 
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando