In this paper we prove that if $G$ is a finite $p$-group of class $c$ with $G/G'$ of exponent $p^r$ and $L_i/L_{i+}$ is cyclic of order $p^r$ for $i= 1, 2,\dots, c-1$, where $L_i$, $i=0,1,\dots,c$ is the lower central series of $G$, then the order of $G$ divides the order of the group $A(G)$ of automorphisms of $G$.