We are working on an OpenStack-based HPC on-demand service, but run into data management problems. We are working in the context of a hybrid cloud configuration, tailored to collaborative HPC, and are looking for similar minded folks and projects, to solve data management, and in particular data migration services for said hybrid clouds. Theo Stillwater Supercomputing, Inc. "/Accelerating Innovation"/
Theo, There are many possible solutions here that are used heavily in HPC environments (xrootd, sci-hadoop, dcache) so probably we would need to hear more about what your specific hybrid cloud data management needs are. Alan On Dec 11, 2013, at 9:12 AM, Theodore Omtzigt <theo@stillwater-sc.com> wrote:
We are working on an OpenStack-based HPC on-demand service, but run into data management problems. We are working in the context of a hybrid cloud configuration, tailored to collaborative HPC, and are looking for similar minded folks and projects, to solve data management, and in particular data migration services for said hybrid clouds.
Theo Stillwater Supercomputing, Inc. "Accelerating Innovation" _______________________________________________ OpenStack-HPC mailing list OpenStack-HPC@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc
Dear Alan: Indeed, I didn't provide enough detail of the problem we are trying to solve. We are looking at the general problem of remote instruments, or data loggers, generating large amounts of data, potentially very noisy and/or redundant, connected to a consolidation backbone data store/compute core via a low bandwidth network. Frequently, the data generated is several orders of magnitude bigger than the network bandwidth. So we are trying to take advantage of the knowledge about the signals so we can reduce the noise, and remove potential redundancy, before hitting the network. This organization follows traditional edge computing architectures that you see in CDNs and Industrial Embedded systems, but we are talking about instruments that can generate hundreds of GBytes per day, and edge computes that may require hundreds of compute hours when executed on COTS servers. We are trying to augment OpenStack to manage these edge computes. It feels it is an extension of the SDN push, as you can think of these data reduction pipelines as dynamic network services, with the differentiation that they require deep computes. Any further questions and/or ideas are greatly appreciated. Theo On Wed, 11 Dec 2013 15:40:43 +0000, "Sill, Alan" <alan.sill@ttu.edu> wrote: Theo,
There are many possible solutions here that are used heavily in HPC environments (xrootd, sci-hadoop, dcache) so probably we would need to hear more about what your specific hybrid cloud data management needs are.
Alan
On Dec 11, 2013, at 9:12 AM, Theodore Omtzigt <theo@stillwater-sc.com> wrote:
We are working on an OpenStack-based HPC on-demand service, but run into data management problems. We are working in the context of a hybrid cloud configuration, tailored to collaborative HPC, and are looking for similar minded folks and projects, to solve data management, and in particular data migration services for said hybrid clouds.
Theo Stillwater Supercomputing, Inc. "Accelerating Innovation"
OpenStack-HPC mailing list OpenStack-HPC@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc
As Alan says, I think you will have to be a bit more specific about the problem! As an aside, there is a news item about Intel's Lustre working well on AWS: http://www.enterprisetech.com/2013/12/06/lustre-running-better-expected-aws/ The contents of this e-mail are confidential and for the exclusive use of the intended recipient. If you are not the intended recipient you should not read, copy, retransmit or disclose its contents. If you have received this email in error please delete it from your system immediately and notify us either by email or telephone. The views expressed in this communication may not necessarily be the views held by McLaren Racing Limited. McLaren Racing Limited | McLaren Technology Centre | Chertsey Road | Woking | Surrey | GU21 4YH | UK | Company Number: 01517478
participants (4)
-
Hearns, John
-
Sill, Alan
-
theo＠stillwater-sc.com
-
Theodore Omtzigt