Affective education has been the new educational pattern under modern ubiquitous learning environment. Especially in mobile learning, how to effectively construct affective education to optimize and enhance the teaching effectiveness has attracted many scholars attention. This paper presents the framework of affective education based on learner's interest and emotion recognition. Learner's voice, text and behavior log data are firstly preprocessed, then association rules analysis, SO-PMI (Semantic Orientation-Pointwise Mutual Information) and ANN-DL (Artificial Neural Network with Deep Learning) methods are used to learner's interest mining and emotion recognition. The experimental results show that these methods can effectively recognize the emotion of learners in mobile learning and satisfy the requirements of affective education.