Meta-skeleton Framework Overview
Extreme data volumes as well as the costs (in terms of latencies and energy/power) associated with data movement are requiring a rethinking of the simulation workflows and necessitating in-situ formulations where the coupled simulation codes as well as the data analytics and uncertainty quantification happen close of where the data resides, on the extreme scale system itself.
As a result, it is critical that the co-design processes go beyond individual computational methods an their skeletons, to include the interactions and data exchanges between these skeletons as well as with analytics components. The goal of the meta-skeleton framework is enable such a co-design of end-to-end data-intensive simulations workflows that include coupled simulations and analytics components (which may have very different characteristics from the simulation codes). It enables scientist to reason about the rich design spaces available for the placement and scheduling of computation and data in space while considering performance and energy/power constraints and associated tradeoffs. The meta-skeleton integrates skeletons of the interacting components and architecture-independent representation of their behaviors (e.g., machine independent characteristics, such as memory access patterns obtained from Byfl and communication patterns obtained from MPI traces) with system levels empirical as well as analytic models and simulators. It then allows users to explore combinations of algorithmic, runtime placement and scheduling, data movement patterns, and system architecture design choices (including hypothetical system designs) and evaluate relative performance and energy/power behaviors.
Related publications are listed below:
- M. Gamell, I. Rodero, M. Parashar, S. Poole, "Exploring Energy and Performance Behaviors of Data-Intensive Scientific Workflows on Systems with Deep Memory Hierarchies", 20th IEEE International Conference on High Performance Computing (HiPC), Hyderabad, India, December 2013 - to appear. (Acceptance 25%)
- M. Gamell, I. Rodero, M. Parashar, J.C. Bennett, H. Kolla, J. Chen, P. Bremer, A.G. Landge, A. Gyulassy, P. McCormick, S. Pakin, V. Pascucci, "Exploring Power Behaviors and Tradeoffs of In-situ Data Analytics",
International Conference on High Performance Computing Networking, Storage and Analysis (SC), Denver, Colorado, November 2013 - to appear. (Acceptance 20%)
A schematic overview of our current prototype of the meta-skeleton framework process is presented in the PDF document below:
A short video demonstration of the protoype is presented below: