In this paper a new method based on fuzzy inference is proposed for recognizing the human interaction in RGBD images. Due to differences in interactions of different individuals based on age, sex and physical built, their activity for a specific sequence vary greatly from each other, therefore using fuzzy system will contribute to interaction recognizing. In the proposed method, first four key frames are selected, skeleton-based features are then extracted and finally, an Adaptive Neuro-Fuzzy System is applied to recognize the interaction. This method was tested on SBU data set and the results showed the best performance compared to the state of the art methods.