Earthquake-damaged buildings recognition of the high-resolution remote sensing images has been an indispensable technical means in the post-earthquake emergency response. In view of the difficulties and constraints caused by the lack of pre-earthquake information, this article proposed a novel damaged buildings recognition of high-resolution remote sensing images based on feature space and decision tree optimization. By only using post-earthquake information, the potential building object set is extracted by combining WJSEG segmentation and a group of non-building screening rules. On this basis, an adaptive decision tree number extraction strategy based on the discrimination of classification accuracy by the curve fluctuation is applied. In addition, the spectrum, texture and geometric morphology features are selected according to the feature importance index to form symbolized sets of damaged buildings. Finally, based on the optimized random forest (RF) model, buildings are separated into three categories as undamaged building, partly damaged building and ruin. Experiments on four different datasets show that the overall accuracy all exceed 85% with the proposed method, which is significantly better than the other compared methods in both visual inspection and quantitative analysis.