The understanding of application characteristics such as hardware resource requirements and communication patterns is key in building highly utilized high performance computing systems for target workloads at a reasonable cost and with available technology. The characterization drives the design decision of both hardware and software. Memory access pattern is a key factor as data movement is a major challenge in today's system design. In this paper, we describe how memory characterization unveils key differences among applications and helps to shape the system design space. With examples, it is shown how memory characterization can be used in the hardware architecture design and application enablement to achieve better performance and system utilization.