Abandoned houses have become a common feature of the local landscapes: the rising number of abandoned houses is a major challenge facing many counties in South Korea. Their presence negatively influences the neighborhood by undermining its aesthetic quality, depreciating the perception of safety in the neighborhood properties, and deepening the fiscal deficit of local financing. The detection of abandoned houses is the first step toward adequate housing management by local governments. This study aims to provide a cost-effective and prompt approach to identifying abandoned houses in rural areas. Multi-source data, that is, images and building registry data are utilized and a multi-input neural network is designed to adopt these heterogeneous datasets. Trained by the two source datasets, the proposed network achieves 86.2% accuracy in classifying abandoned houses, which is an acceptable performance level in administrative practice. The database of abandoned houses identified in this manner is expected to promote effective housing management by governments and ultimately contribute to mitigating vacancies in rural areas.