In this work the self-excited vibrations of the non-linear rotor are determined. The mathematical model of the rotor is a second-order nonlinear differential equation with a complex function and strong and weak nonlinear terms. The oscillations are determined analytically using Krylov-Bogoliubov methods and Jacobian elliptic functions. The solution of the differential equations is also calculated numerically. The analytical and numerical results are comparable. The amplitude of the vibrations increases and the rotation of the rotor is unstable.