What are some of the most commonly used but time consumptive operations (primitives) encountered in data mining, knowledge discovery, and decision support systems? How are the performance and scalability of these primitives affected by key hardware and software attributes of parallel systems? Can currently available algorithms be improved? Are these primitives candidates for inclusion in datamining and decision support software libraries? Should they be implemented in silicon?
The relevance of the planned data-mining research is that in the past, high performance computer hardware and software tended to be optimized for numerically rather than data intensive applications. The basic vector, matrix, and other mathematical operators prevalent in scientific computing have been well understood for decades and this knowledge has been used in many studies devoted to the investigation of factors affecting the performance of high end machines in such application areas. The results of these studies have led to the development of important new architectures, algorithms, and entire mathematical libraries all highly optimized for high speed execution of the most commom mathematical operations in scientific and technical computing. Over the past 20 years, they have led to the doubling in floating point performance every 18-24 months.
This has not been the case for the processing and I/O performance needed by data intensive applications encounterd not only in datamining and decision support systems, but also in many scientific codes where output increases rapidly with the dimensionality of the problem. An example is the ARPI3D weather model which spends 69 the elapsed time in doing various I/O operations (initialization, message passing and output) and only 31 including computation, when using 64 nodes on UNM's Roadrunner SuperCluster. Consequently, studies such as the one planned here are important to understand the characteristics of REAL (as opposed to synthetic) I/O workloads, the I/O constraints of today's machines, and the need for architectural changes in order to effectively support fast data movement as well as commonly occurring but complex operations during knowledge discovery, business intelligence, and decision-support applications. That understanding should lead to the design of new hardware and software systems with more effective data caching and prefetching strategies, greater use of file striping, much improved parallel file systems, as well as new instruction, architectures, and libraries to handle complex operations on very large data sets. Such systems would have more balanced compute and I/O performance and therfore be more effective for running a much wider range of applications than the less balanced systems of today.
The planned studies will be conducted using the wide variety of hardware, interconnects, and software resources currently existing at UNM, including those requested under this award. This includes the existing Linux Roadrunner SuperCluster, as well as IBM Power3 SMP and the Netfinity clusters. The resources also include a wide array of scientific application codes for computational physics, chemistry, and fluid dynamics to study scientific I/O workloads, while the UNM data warehouse could provide similar information for data mining and decision support systems.