Graph ranking is one popular and successful technique for image retrieval, but its effectiveness is often limited by the well-known semantic gap. To bridge this gap, one of the current trends is to leverage the click-through data associated with images to facilitate the graph-based image ranking. However, the sparse and noisy properties of the image click-through data make the exploration of such resource challenging. Towards this end, this paper propose a novel click-boosted graph ranking framework for image retrieval, which consists of two coupled components. Concretely, the first one is a click predictor based on matrix factorization with visual regularization, in order to alleviate the sparseness of the click-through data. The second component is a soft-label graph ranker that conducts the image ranking by using the enriched click-through data noise-tolerantly. Extensive experiments for the tasks of click predicting and image ranking validate the effectiveness of the proposed methods in comparison to several existing approaches.