As environmental and multi-display configurations become more common, HCI research is becoming increasingly concerned with actions around these displays. Tracking human activity is challenging, and there is currently no single solution that reliably handles all scenarios without excessive instrumentation. In this paper we present a novel tracking and analysis approach using a top-down 3D camera. Our hierarchical tracking approach models local and global affinities, scene constraints and motion patterns to find and track people in space, and a novel salience occupancy pattern (SOP) is used for action recognition. We present experiences applying our approach to build a proxemics-aware tabletop display prototype, and to create an exhibit combining a large vertical display with an interactive floor-projection.