Open Stack

Dear Alan:

 Indeed, I didn't provide enough detail of the problem we are trying to solve. 

We are looking at the general problem of remote instruments, or data 
loggers, generating large amounts of data, potentially very noisy 
and/or redundant, connected to a consolidation backbone data 
store/compute core via a low bandwidth network. Frequently, the data 
generated is several orders of magnitude bigger than the network 
bandwidth. So we are trying to take advantage of the knowledge about 
the signals so we can reduce the noise, and remove potential 
redundancy, before hitting the network. This organization follows 
traditional edge computing architectures that you see in CDNs and 
Industrial Embedded systems, but we are talking about instruments that 
can generate hundreds of GBytes per day, and edge computes that may 
require hundreds of compute hours when executed on COTS servers. 

We are trying to augment OpenStack to manage these edge computes. It 
feels it is an extension of the SDN push, as you can think of these 
data reduction pipelines as dynamic network services, with the 
differentiation that they require deep computes. 

Any further questions and/or ideas are greatly appreciated. 

Theo

On Wed, 11 Dec 2013 15:40:43 +0000, "Sill, Alan" <alan.sill at ttu.edu> wrote:
Theo,
>
> There are many possible solutions here that are used heavily in HPC 
> environments (xrootd, sci-hadoop, dcache) so probably we would need 
> to hear more about what your specific hybrid cloud data management 
> needs are. 
>
> Alan
>
> On Dec 11, 2013, at 9:12 AM, Theodore Omtzigt <theo at stillwater-sc.com> wrote:
>
> > We are working on an OpenStack-based HPC on-demand service, but run 
> into data management problems. We are working in the context of a 
> hybrid cloud configuration, tailored to collaborative HPC, and are 
> looking for similar minded folks and projects, to solve data 
> management, and in particular data migration services for said hybrid 
> clouds. 
> > > Theo
> > Stillwater Supercomputing, Inc. "Accelerating Innovation"
> > _______________________________________________
> > OpenStack-HPC mailing list
> > OpenStack-HPC at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc
>
>
>