Open Stack

On 12/18/2013 12:27 PM, Tim Simpson wrote:
>>> Please provide proof of that assumption or at least a general hypothesis that we can test.
> I can't prove that the new agent will be larger as it doesn't exist yet.
>
>>> Since nothing was agreed upon anyway, I don't know how you came to that conclusion.  I would suggest that any agent framework be held to an extremely high standard for footprint for this very reason.
> Sorry, I formed a conclusion based on what I'd read so far. There has been talk to add Salt to this Unified Agent along with several other things. So I think its a valid concern to state that making this thing small is not as high on the list of priorities as adding extra functionality.
>
> The C++ agent is just over 3 megabytes of real memory and takes up less than 30 megabytes  of virtual memory. I don't think an agent has to be *that* small. However it won't get near that small unless making it tiny is made a priority, and I'm skeptical that's possible while also deciding an agent will be capable of interacting with all major OpenStack projects as well as Salt.
>
>>> Nobody has suggested writing an agent that does everything.
> Steven Dake just said:
>
> "A unified agent addresses the downstream viewpoint well, which is 'There is only one agent to package and maintain, and it supports all the integrated OpenStack Program projects'."
>
> So it sounds like some people are saying there will only be one. Or that it is at least an idea.
>
>>> If Trove's communication method is in fact superior to all others, then perhaps we should discuss using that in the unified agent framework.
> My point is every project should communicate to an agent in its own interface, which can be swapped out for whatever implementations people need.
>
>>>   In fact I've specifically been arguing to keep it focused on facilitating guest<->service communication and limiting its in-guest capabilities to narrowly focused tasks.
> I like this idea better than creating one agent to rule them all, but I would like to avoid forcing a single method of communicating between agents.
>
>>> Also I'd certainly be interested in hearing about whether or not you think the C++ agent could made generic enough for any project to use.
> I certainly believe much of the code could be reused for other projects. Right now it communicates over RabbitMQ, Oslo RPC style, so I'm not sure how much it will fall in line with what the Unified Agent group wants. However, I would love to talk more about this. So far my experience has been that no one wants to pursue using / developing an agent that was written in C++.
>
> Thanks,
>
> Tim
> ________________________________________
> From: Clint Byrum [clint at fewbar.com]
> Sent: Wednesday, December 18, 2013 11:36 AM
> To: openstack-dev
> Subject: Re: [openstack-dev] [trove] My thoughts on the Unified Guest Agent
>
> Excerpts from Tim Simpson's message of 2013-12-18 07:34:14 -0800:
>> I've been following the Unified Agent mailing list thread for awhile
>> now and, as someone who has written a fair amount of code for both of
>> the two existing Trove agents, thought I should give my opinion about
>> it. I like the idea of a unified agent, but believe that forcing Trove
>> to adopt this agent for use as its by default will stifle innovation
>> and harm the project.
>>
> "Them's fightin words". ;)
>
> That is a very strong position to take. So I am going to hold your
> statements of facts and assumptions to a very high standard below.
>
>> There are reasons Trove has more than one agent currently. While
>> everyone knows about the "Reference Agent" written in Python, Rackspace
>> uses a different agent written in C++ because it takes up less memory. The
>> concerns which led to the C++ agent would not be addressed by a unified
>> agent, which if anything would be larger than the Reference Agent is
>> currently.
>>
> "Would be larger..." - Please provide proof of that assumption or at least
> a general hypothesis that we can test. Since nothing was agreed upon
> anyway, I don't know how you came to that conclusion. I would suggest
> that any agent framework be held to an extremely high standard for
> footprint for this very reason.
>
>> I also believe a unified agent represents the wrong approach
>> philosophically. An agent by design needs to be lightweight, capable
>> of doing exactly what it needs to and no more. This is especially true
>> for a project like Trove whose goal is to not to provide overly general
>> PAAS capabilities but simply installation and maintenance of different
>> datastores. Currently, the Trove daemons handle most logic and leave
>> the agents themselves to do relatively little. This takes some effort
>> as many of the first iterations of Trove features have too much logic
>> put into the guest agents. However through perseverance the subsequent
>> designs are usually cleaner and simpler to follow. A community approved,
>> "do everything" agent would endorse the wrong balance and lead to
>> developers piling up logic on the guest side. Over time, features would
>> become dependent on the Unified Agent, making it impossible to run or
>> even contemplate light-weight agents.
>>
> Nobody has suggested writing an agent that does everything. A
> framework for agents to build on is what has been suggested. In fact
> I've specifically been arguing to keep it focused on facilitating
> guest<->service communication and limiting its in-guest capabilities to
> narrowly focused tasks.
>
>> Trove's interface to agents today is fairly loose and could stand to be
>> made stricter. However, it is flexible and works well enough. Essentially,
>> the duck typed interface of the trove.guestagent.api.API class is used
>> to send messages, and Trove conductor is used to receive them at which
>> point it updates the database. Because both of these components can be
>> swapped out if necessary, the code could support the Unified Agent when
>> it appears as well as future agents.
>>
>> It would be a mistake however to alter Trove's standard method of
>> communication to please the new Unified Agent. In general, we should
>> try to keep Trove speaking to guest agents in Trove's terms alone to
>> prevent bloat.
>>
> If Trove's communication method is in fact superior to all others,
> then perhaps we should discuss using that in the unified agent framework.
>
> Also I'd certainly be interested in hearing about whether or not you
> think the C++ agent could made generic enough for any project to use.
> That would be a nice win.
If footprint is the concern, I am certain you would be better off with a 
C implementation.  C++ is full of fail in a variety of ways and offers 
no useful advantage for something as small as an agent ;-)

We wrote our heat agents in python because they are not long-lived 
processes, hence they can bleed into the guest OS memory on demand and 
are only typically used during bootstrapping, vs always taking a chunk 
off the top as a long-lived agent process would do.

Just for comparison sake, the basic footprint of an operation like 
sleep(100) (eg the runtime)

python:
consumes 12.5MB of virt memory and 4.3MB of resident memory.

C:
4MB of virt memory and 328k of resident memory

C++:
12.5MB of virt memory and 784k of resident memory

C++ buys 3MB of ram, compared to all the losses of the great 
functionality the infrastructure team has put into place.  But C is a 
real winner for footprint, especially since the C library (the virt 
part) will almost always be demanded by some application via the dynamic 
loader.

Regards
-steve

> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev