Abstract : Neural networks (NNs) normally do not allow any insight into the reasoning behind their predictions. We demonstrate how influence functions can unravel the black box of NN when trained to predict the phases of the one-dimensional extended spinless Fermi-Hubbard model at half-filling. Results are the first indication that the NN correctly learns an order parameter describing the transition. Moreover, we demonstrate that influence functions not only allow to check that the network trained to recognize known quantum phases can predict new unknown ones but even discloses information about the type of phase transition. This presentation is a continuation of the talk given within the Center for Theoretical Physics Colloquium. I will repeat (this time briefly) the introduction to the topic of interpretable machine learning. The detailed description of how we prepared the dataset, chose, and trained the NN will follow. Finally, I will present our results of applying influence functions to the trained NN. ______________________________________________ Zoom meeting details Topic: Quantum Information and Quantum Computing Working Group Time: Apr 30, 2020, 12:00 PM Warsaw Join Zoom Meeting: QIQCWG-ZOOM Meeting ID: 640 279 690Password: tp4C,kERx If you encounter any problems with connecting to the Zoom meeting, please email email@example.com directly.