With the advance of computer technology and smart device, many technologies and applications have been developed to enhance the efficiency of human-computer interaction (HCI). For human, the hand is a natural and direct way in communication. Hand-held Object Recognition (HHOR), which is to predict the label for the object people hold in hand, can help machines in understanding the environment and people's intentions. However, it has not been well studied in the community. So, in this paper, we proposed a novel feature fusion based method for hand-held object recognition with RGB-D data. First, the skeleton information is used to initially locate the object and with depth map we extract object region in a region-growing manner. Then on the corresponding object point cloud, we use Multiple Kernel Learning (MKL) to fuse the shape feature with color feature to obtain the advantages of them. Specially, we collected a dataset, which contains 12800 video frames of 16 categories and each frame captures the visual image, depth map and user skeleton data. The experiment shows promising results in both segmentation and recognition.