Autonomic Management of Instrumented Datacenters

Principle Researchers: Nanyan Jiang, Manish Parashar and Daria Pompili

Technical advances are leading to a pervasive computational ecosystem that integrates computing infrastructures with embedded sensors and actuators, and giving rise to a new paradigm for monitoring, understanding, and managing natural and engineered systems - one that is information/data-driven and autonomic. The overarching goal of this research is to develop sensor system middle-ware and programming support that will enable distributed networks of sensors to function, not only as passive measurement devices, but as intelligent data processing instruments, capable of data quality assurance, statistical synthesis and hypotheses testing as they stream data from the physical environment to the computational world.

This research investigates a programming system to support the development of in-network data processing mechanisms, and enable applications to discover, query, interact with, and control instrumented physical systems, such as autonomic datacenter management system using semantically meaning-ful abstractions. This includes abstractions and runtime mechanisms for integrating sensor systems with applications processes, as well as for in-network data processing such as aggregation, adaptive interpolation and assimilation.

The proposed system enables sensor-driven autonomic management of instrumented datacenters at two levels. First, it provides programming abstractions for integrating sensor systems with computational models for applications processes and with other application components in an end-to-end experiment. Second, it provides programming abstractions and system soft-ware support for developing in-network data processing mechanisms. Specifically for the latter, we explore the temporal and spatial correlation of sensor measurements in the targeted application domains to tradeoff between the complexity of coordination among sensor clusters and the savings that result from having fewer sensors for in-network processing, while maintaining an acceptable error threshold. Experimental results show that the proposed in-network mechanisms can facilitate the efficient usage of constraint resources and satisfy data requirement in the presence of dynamics and uncertainty.

In such a scenario, the programming system is used to enable the dynamic data-driven management of an instrumented data center system, to optimize power consumption, efficiency of cooling system and job throughput. Sensor networks monitor temperature, humidity, and airflow in real time, and provide non-intrusive and fine-grained data collection, and enable real-time processing. And these sensors are integrated with computational processes and job schedulers to take phenomenon, such as heat distribution and air flows into consideration, and to optimize data center performance in terms of energy consumption and throughput. More specifically, the scheduler uses real time information about data center operational conditions (e.g., temperature, humidity) from the sensors to generate optimal management policies. Furthermore, to enable timely response to environmental changes, in-network analysis uses localized temperature distribution to decide when and where to migrate jobs at runtime. Experimental results show that the provided programming system reduces overheads while achieving near optimal and timely management and control.

References:

N. Jiang and M. Parashar, ―Programming Support for Sen-sor-based Scien-tific Applica-tions,‖ Proceed-ings of the Next Generation Soft-ware (NGS) Workshop, (IPDPS 2008), Miami, FL, USA, IEEE Computer Society Press, April, 2008.

N. Jiang and M. Parashar, ―In-network Data Estimation Mechanisms for Sensor-driven Scientific Applica-tions,‖ Proceed-ings of the 15th IEEE Interna-tional Conference on High Perform-ance Computing (HiPC 2008), Bangalore, India, December, 2008.