A lack of efficient system software is an increasing impediment to deploying large-scale parallel and distributed systems. Systemically addressing operating system-induced performance anomalies requires accurate, low-overhead, whole-system monitoring, something that is currently unavailable in large tightly-coupled systems. In this paper, we present the design of IMPuLSE—Integrated Monitoring and Profiling for Large-Scale Environments—a system we are developing to meet this need. IMPuLSE’s innovative messagecentric profiling approach trades off of centralized global knowledge for low overhead, while retaining relatively fine-grained information about important cross-host performance interactions. The goal of this approach is to enable both large-scale system software adaptation and continuous system performance auditing.