Data-driven machine learning algorithms triggered a fundamental change in hedonic real estate pricing. However, their adaptive nonparametric structure makes inference and out-of-sample prediction challenging. This study introduces an explainable approach to interpreting machine learning predictions, which has not been done before in the local market context. Specifically, Random Forest and Extreme Gradient Boosting models are developed for residential real estate price prediction in Warsaw in 2021 on 10,827 property transactions. Model-agnostic Explainable Artificial Intelligence (XAI) methods are then used to investigate the black box decision making. The results show the practicability of applying XAI frameworks in the real estate market context to decode the rationale behind data-driven algorithms. Information about the relationships between input variables is extracted in greater detail. Accurate, reliable and transparent real estate valuation support tools can offer substantial advantages to participants in the real estate market, including banks, insurers, pension and sovereign wealth funds, as well public authorities and private individuals.