Halftone image is widely used in printing and scanning equipment. It is significant for the halftone image to be preserved and processed. For the different resolution of the display devices, the processing and displaying of halftone image are faced with great challenges, such as Moore pattern and image blurring. The inverse halftone technique is required to remove the halftone screen. In this paper, we propose an inverse halftone algorithm based on sparse representation with the dictionary learned by two steps: deconvolution and sparse optimization in the transform domain to remove the noise. The main contributions of this paper include three aspects: first, we analysis the denoising effects for different training sets and the dictionary; Then we propose the denoising algorithm through adaptively learning the dictionary, which iteratively remove the noise of the training set and improve the dictionary; Then the inverse halftone algorithm is proposed. Finally, we verify that the noise level in the error diffusion linear model is fixed, and the noise level is only related to the diffusion operator. Experimental results show that the proposed algorithm has better PSNR and visual performance than state-of-the-art methods. The codes and constructed models are available at https://github.com/juneryoung2022/IH-WNNM.