Smart Homes have come a long way: From research laboratories in the early days, through (almost) neglect, to their recent revival in real-world environments enabled through the existence of commodity devices and robust, standardized software frameworks. With such availability, human activity recognition (HAR) in smart homes has become attractive for many real-world applications, especially in the domain of Ambient Assisted Living. Yet, getting started with an activity recognition system in specific smart homes, which are highly specialized spaces inhabited by individuals with idiosyncratic behaviors and habits, is a non-trivial endeavor. We present an approach for bootstrapping HAR systems for individual smart homes from scratch. At the beginning of the life cycle of a smart home, our system passively observes activities and derives rich representations for sensor data-action units-which are then aggregated into activity models through motif learning with minimal supervision. The resulting HAR system is then capable of recognizing relevant, most frequent activities in a smart home. We demonstrate the effectiveness of our bootstrapping procedure through experimental evaluations on CASAS datasets that show the practical value of our approach.