<div dir="ltr">Hello,<div><br></div><div>I would like to continue this insightful discussion by dropping a couple of suggestions inline.</div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 5, 2013 at 3:47 PM, Caitlin Bestler <span dir="ltr"><<a href="mailto:caitlin.bestler@nexenta.com" target="_blank">caitlin.bestler@nexenta.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><span style="color:rgb(34,34,34)">While we don't want to solve every possible topology, I think we really need to pay attention to what multi-site really requires.</span><br>
</div>
<br>
I haven't done any studies of the entire market, but in my experience inter-site replication used by storage services is almost always<br>
via dedicated or VPN tunnels, and when VPN tunnels are used they are traffic shaped.<br>
<br>
This is not just a matter of connecting a bunch of IP addresses on the internet and then form a vague impression as to which ones<br>
are "far" away. It is more like the type of discovery routers do where each tunnel is a "link".<br>
<br>
A proper remote replication solution will be aware of these links, and take that into account in its replication strategy. One example<br>
topology that I believe is very likely is a distributed corporate intranet. The branch offices are very unlikely to connect with each<br>
other, but rather mostly connect with the central office (and maybe one alternate location).<br>
<br>
If the communications capacity favors communicating with certain sites, then we should favor replicating to those sites. Communications<br>
capacity between corporate sites is typically provisioned (whether with dedicated lines or just VPN) and not something you will be able<br>
to just increase on demand instantly. Inter-site bandwidth is still expensive.<br>
<br>
That said, there are still two important things to reach a consensus on:<br>
<br>
* Are we talking about enabling the Swift Proxy to access content that is at multiple sites, but each object is linked to a specific site.<br>
Or are we creating a global namespace with eventual consistency, and smart assignment of objects to the sites where they are<br>
actually referenced? The first goal is certainly easier.<br></blockquote><div> </div><div>Our initial idea was to create a global namespace, i.e. have a single ring shared across all regions and containing all devices, and have proxy-servers accessing data based on the ring location with preference of local servers. Now, after some a work done on replication network feature, we understand that the most likely deployment topology is regions with replication networks connected by VPN of some sort and storage networks totally isolated. In such deployment, no proxy server will ever access remote region's storage server, thus no need in global namespace for accessing data. What we're actually need global namespace for is the inter-region replication, which brings us to the second question:</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* What forms of site-to-site replication are we going to support? Is this something each system administrator specifies (such as<br>
by adding policies along the lines of "all new objects created at a branch office will be replicated to the two central sites on<br>
a daily basis. Only objects actually referenced at a branch office will be cached there.") or something more akin to how Swift<br>
operates locally where the user does not specify where specific things are stored?</blockquote><div><br></div><div>It looks like we need a kind of 'ring-of-rings' and a server(s) controlling inter-region replication in every region. This server might be represented as a device with very high weight, or some special device, which basically has at least one replica of most partitions (or each partition) in the cluster. This ensures the local replicators report number of replicas in local cluster to inter-region replicator. Inter-region replicators, in turn, compare value of replicas to recorded in 'ring-of-rings' and initiate cross-region replication if local region lost all configured replicas of partition.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br></div><div class="gmail_extra">--</div><div class="gmail_extra">Best regards,</div><div class="gmail_extra">Oleg Gelbukh</div><div class="gmail_extra">Mirantis, Inc.</div>
</div>