Open Stack

Cool, so we aren't alone here (I knew it!)

Has the DNS generation ever been a problem for you, as its out of band 
and can have problems like if it ever stops working, the DNS name may be 
reported in the instance, but that hostname may not actually be in DNS 
and thus this equals a confused user?

Same goes with cleanup as well, when it's out of band it's hard to 
handle the failure of DNS to actually perform the delete and therefore 
it makes it hard/impossible to stop the instance from being deleted if 
this happens (and if a DNS deletion failure occurs and the user tries to 
reuse the same hostname in a short period of time before that problem is 
fixed then you get a doubly confused user).

It almost makes me desire for a more customizable boot/delete-workflow 
where these kinds of additional actions can be tied into the larger 
workflow for those that want to do them (in a way there would be a 
standard workflow that would be the accepted openstack way, but it would 
be relatively easy to add a new <unit> in that to call out to DNS or to 
do ...).

Ben Morrice wrote:
> Hello,
>
> We do something similar. Our requirements are:
>
> * VM is registered in CMDB (foreman as we are a puppet shop)
> * VM is registered in DNS
> * Cleanup of the above is performed on VM termination
>
> Our Horizon is customised, so that when a VM is spawned the user can
> associate it to a foreman hostgroup and environment. If the user doesn't
> select a hostgroup/environment the record is still created in foreman as
> 'unmanaged'.
> If the record is not able to be created in Foreman, Horizon will fail at
> this point and the VM will not be spawned.
>
> We also have a bit of python that listens and reacts to
> compute.instance.create.end and compute.instance.delete.end messages in
> the notification queue.
>
> For creations we do a bit of sanity checking and create A records based
> off the VM name
> For deletions we remove the DNS A records, remove from Foreman as well
> as checking/removing from other systems that the VM may have been a be a
> member of (example: GPFS cluster member invoked through puppet).
>
> We decided to react on compute.instance.create.end messages for DNS
> creations as we wanted to be able to use HEAT and automatically
> generated hostnames.
>
> I'm not sure what the 'best practices' are, however this setup has been
> working nicely for us for the past 2 years.
>
> Kind regards,
>
> Ben Morrice
>
> ______________________________________________________________________
> Ben Morrice | e: ben.morrice at epfl.ch | t: +41-21-693-9670
> EPFL ENT CBS BBP
> Biotech Campus
> Chemin des Mines 9
> 1202 Geneva
> Switzerland
>
> On 06/07/16 20:13, Joshua Harlow wrote:
>> Hi folks (operators and devs),
>>
>> I was digging into some of godaddy code yesterday/everyday and it got
>> me thinking about how other operators are handling customized internal
>> integrations and what some of the common patterns are for how these
>> integrations are typically performed (overall it doesn't feel like
>> there are that many patterns that people can be doing in that many
>> different ways). It got me thinking that there really isn't a best
>> practices document that I know for these types of common integrations
>> (perhaps this can be the start of one?) that operators and developers
>> can provide feedback on (because perhaps there is a new or to-be new
>> API that replaces an older style of integration).
>>
>> This hopefully falls into the pattern (and/or is similar to) that is
>> described in:
>>
>> https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/api-no-more-extensions.html#alternatives-for-extension-owners
>>
>>
>> '''
>> Please bring forward the conversation to the wider community anyway.
>> There are a lot of OpenStack deploys, and issues that you think are
>> specific to your environment may not be. And in conversation with the
>> upstream operator and developer communities we could probably come up
>> with a generic facility that supports your use.
>> '''
>>
>> So let me describe some of the internal integrations we've had to do
>> and see if these sound familiar to what others have had to do and it'd
>> be nice to share how those integrations were performed (or even if the
>> integration that was done is similar or done in a different manner,
>> that's cool to):
>>
>> (1) Requirement we have is to register virtual machines booted with a
>> in-house dns solution and to also delete dns names registered on
>> deletion being triggered.
>>
>> The steps this will do is on reception of a
>> 'compute.instance.create.end' event being received (the program that
>> does this listens on the notifications queue of nova) logic will
>> proceed that will (direct comment from routine here):
>>
>> 'Creates A and PTR records based off of the instance's hostname and
>> fixed ip.'
>>
>> This goes about and has further logic to realize this statement (its
>> error robustness is a different discussion, but I digress) and the
>> 'compute.instance.delete.end' event does the reverse:
>>
>> 'Removes instance's DNS entries. Will remove A and PTR records.'
>>
>> Now sidestepping the choice of using the notifications queue (and
>> having a consumer on that) for this which IMHO has issues in that it's
>> out of band, has no way to stop a VM from being created, or set a
>> state of the VM like ERROR_DNS_SETUP if it fails and so-on (ie if a
>> create dns record fails, how does the delete know that it shouldn't
>> try to delete and such...) I was wondering how others are doing this
>> kind of similar actions. I believe that they might be more native
>> integration with neutron that is possible here to solve this using a
>> native solution instead?
>>
>> (2) Requirement that for all created instances we register into a
>> home-grown cmdb solution with the 'fixed_ips' that the instance was
>> built with, the environment it was built with, information about the
>> instances image (family) and so-on (the following is a more complete
>> gathering of info that we send):
>>
>> server = sr_manager.build_server_object(
>> environmentID=cmdb_env_id,
>> hostname=instance.hostname,
>> serialNumber=instance.instance_id,
>> fqdn=fqdn,
>> rack=_short_hostname(instance.host),
>> operatingSystemIdentifier=os_spec,
>> operatingSystemFamily=os_family,
>> ramMB=instance.memory_mb,
>> teamID=cmdb_team_id,
>> teamName=cmdb_team_name,
>> drivesGB=instance.disk_gb,
>> ...
>> nics=cmdb_nics
>> )
>>
>> So similar to (1) we have also plugged into the notifications queue
>> with the same queue consumer and hook-in to do this out of band system
>> registration (which is really a eventually/best-effort population into
>> the home grown cmdb due to mentioned issues about about robustness and
>> failures). I was wondering what others are doing for this kind of in
>> house cmdb (which most companies have) and how they are doing VM
>> registration into that (are people even bothering to do that?); oh
>> btw, on removal it triggers a removal/retirement process from that
>> same system using a function like:
>>
>> sr_manager.retire_server(instance.hostname, instance.user_id)
>>
>> (3) On deletion (only) we in the same consumer connect into corporate
>> LDAP and delete the instance hostname from that system and delete the
>> instance from registration into a variant of
>> https://fedorahosted.org/spacewalk/ (the registration of the instance
>> into these systems happens on instance boot); so this got me wondering
>> how do others register/deregister virtual machines created into such
>> systems (we can't be the only one using corporate LDAP/spacewalk in
>> this manner)?
>>
>> (4) A few more, but this is long enough for now ;)
>>
>> -Josh
>>
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators