The goal of this research is to develop performance profiles of parallel and distributed applications in order to predict their execution time under different network conditions. This paper measures the resource requirements of the NAS benchmark programs and characterizes their performance in a shared heterogeneous environment. The programs in the benchmark suite were executed on a controlled testbed and their usage of CPU, bandwidth, and memory were measured. The performance of the benchmark programs was also measured under controlled sharing of CPU and bandwidth. The results are used to characterize the behavior of the NAS benchmark programs with resource sharing. The paper demonstrates that the core system activity of a program can be accurately measured by passive probing, and that this measured system activity is the key to the prediction of program performance when resources must be shared. Our methods rely on system level measurements alone, and therefore, application knowledge or access to the source code, is not required. Hence, the techniques apply across programming languages and models. This paper is an important step towards building an automated framework to infer execution characteristics and estimate performance on shared networks. Such a framework has an important role in resource selection in shared clusters and grid computing environments.