Network embedding (NE) focuses on discovering low-dimensional embeddings of nodes while retaining their intrinsic features and structure of nodes. It is essential for many practical applications, containing text mining, community detection, and node classification. However, the great majority of existing systems are incapable of combining structural and attribute information. To tackle the above-mentioned problem, considering the information diffusion process, we present a novel model for attribute NE (ANE), namely influential node diffusion-based matrix factorization (INDMF), which contains topology level and attribute level. In detail, we first propose a novel method to extract high-order information via influential node diffusion sequences. Then, we regard the optimization of our proposed structure-based and attribute-based loss functions as a matrix factorization problem. Furthermore, this model can be used to generate final node embedding by aggregating the topology level and attribute level hierarchically. Experiments are conducted on four real-world datasets, which indicates that INDMF beats all competing algorithms in node categorization, community detection, and graph visualization.