Let $G$ be a finite group and $m,n$ two positive coprime integers. We prove that the set of conjugacy class sizes of primary and biprimary elements of $G$ is $\{1,m,n\}$ if and only if $G$ is quasi-Frobenius with abelian kernel and complement.