The complex architectures of HEC systems present difficult challenges for performance optimization of scientific applications. Tools are needed that collect and present relevant information on application performance in a scalable manner so as to enable developers to easily identify and determine the causes of performance bottlenecks. This paper describes KOJAK, a suite of performance analysis tools that collect and analyze runtime data from high performance applications. Performance data are collected automatically using a combination of source code annotaions or binary instrumentation and hardware counters. The analysis tools use pattern recognition to convert the raw performance data into information about performance bottlenecks relevant to developers. Such automated approaches to performance instrumentation and analysis promise to increase programmer productivity and reduce time to solution by reducing both development and execution time.