Open Stack

On Wed, Jun 22, 2016 at 6:33 PM, Thierry Carrez <thierry at openstack.org>
wrote:

> Jeremy Stanley wrote:
>
>> On 2016-06-21 17:34:07 +0000 (+0000), Jeremy Stanley wrote:
>>
>>> On 2016-06-21 18:16:49 +0200 (+0200), Thierry Carrez wrote:
>>>
>>>> It hurts a lot when it's down because of so many services being served
>>>> from
>>>> it. We could also separate the published websites (status.o.o,
>>>> governance.o.o, security.o.o, releases.o.o...) which require limited
>>>> resources and grow slowly, from the more resource-hungry storage sites
>>>> (logs.o.o, tarballs.o.o...).
>>>>
>>>
>>> Agreed, that's actually a pretty trivial change, comparatively
>>> speaking.
>>>
>>
>> Oh, though it bears mention that the most recent extended outage
>> (and by far longest we've experienced in a while) would have been
>> just as bad either way. It had nothing to do with recovering
>> attached volumes/filesystems, but rather was a host outage at the
>> provider entirely outside our sphere of control. That sort of issue
>> can potentially happen with any of our servers/services no matter
>> how much we split them up.
>>
>
> I don't think it would have been just as bad... Even in the unlucky case
> where the VMs end up on the same machine and are all affected, IIUC
> rebuilding some of them would have been much faster if they were split up
> (less data to rsync) ?



I believe only the VM was rsync'd to a new server and the cinder volumes
reattached. What did take a while though was the fsck that ran after it
rebooted. This would have been faster for some services if they were on a
separate VM with smaller disks.

Another gain from separating the services is the surface area affected
should a node go down is smaller. If we're lucky just one service would go
down at a time (for example job logs, but tarballs stays up).

Cheers,
Josh



>
>
> --
> Thierry Carrez (ttx)
>
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20160622/27659cf3/attachment.html>