Dear Alan: Indeed, I didn't provide enough detail of the problem we are trying to solve. We are looking at the general problem of remote instruments, or data loggers, generating large amounts of data, potentially very noisy and/or redundant, connected to a consolidation backbone data store/compute core via a low bandwidth network. Frequently, the data generated is several orders of magnitude bigger than the network bandwidth. So we are trying to take advantage of the knowledge about the signals so we can reduce the noise, and remove potential redundancy, before hitting the network. This organization follows traditional edge computing architectures that you see in CDNs and Industrial Embedded systems, but we are talking about instruments that can generate hundreds of GBytes per day, and edge computes that may require hundreds of compute hours when executed on COTS servers. We are trying to augment OpenStack to manage these edge computes. It feels it is an extension of the SDN push, as you can think of these data reduction pipelines as dynamic network services, with the differentiation that they require deep computes. Any further questions and/or ideas are greatly appreciated. Theo On Wed, 11 Dec 2013 15:40:43 +0000, "Sill, Alan" <alan.sill@ttu.edu> wrote: Theo,
There are many possible solutions here that are used heavily in HPC environments (xrootd, sci-hadoop, dcache) so probably we would need to hear more about what your specific hybrid cloud data management needs are.
Alan
On Dec 11, 2013, at 9:12 AM, Theodore Omtzigt <theo@stillwater-sc.com> wrote:
We are working on an OpenStack-based HPC on-demand service, but run into data management problems. We are working in the context of a hybrid cloud configuration, tailored to collaborative HPC, and are looking for similar minded folks and projects, to solve data management, and in particular data migration services for said hybrid clouds.
Theo Stillwater Supercomputing, Inc. "Accelerating Innovation"
OpenStack-HPC mailing list OpenStack-HPC@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc