It is a challenging task to perform the non-linear system state recognition and safety monitoring under strong noise and complex excitation, to tackle this problem, we innovatively propose the variational mode decomposition multiscale holographic transfer entropy statistics (VMTES) method based on the energy transfer relationship between non-linear system signals. The VMTES is a method that measures the information flow direction and coupling degree between non-linear systems, which can precisely measure the slight changes of energy transfer of a mechanical system, accurately assess the slight mutation of dynamical behaviors and status change of a mechanical system and therefore realize fault location and quantification of the non-linear rotating machinery system. The damage severity and direction of the measure point can be precisely described with the VMTES damage assessment indicator, providing the reliable basis for the structural health monitoring and fault diagnosis of the mechanical system. By applying the result to the state recognition of a chaotic system and the structural health monitoring of a rotating machinery system, we can see from the experimental results that the VMTES can effectively detect the fault of gears and rolling bearings under several working conditions. Based on the experiment, we also explain how this method simultaneously locate and quantify the non-linear vibration caused by the fault. (c) 2020 Elsevier Ltd. All rights reserved.