One important problem in MOOCs is the lack of personalized support from teachers. Conversational agents arise as one possible solution to assist MOOC learners and help them to study. For example, conversational agents can help review key concepts of the MOOC by asking questions to the learners and providing examples. JavaPAL, a voice-based conversational agent for supporting learners on a MOOC on programming with Java offered on edX. This paper evaluates JavaPAL from different perspectives. First, the usability of JavaPAL is analyzed, obtaining a score of 74.41 according to a System Usability Scale (SUS). Second, learners’ performance is compared when answering questions directly through JavaPAL and through the equivalent web interface on edX, getting similar results in terms of performance. Finally, interviews with JavaPAL users reveal that this conversational agent can be helpful as a complementary tool for the MOOC due to its portability and flexibility compared to accessing the MOOC contents through the web interface.