In this paper, the group $IA(G)$ of all $IA$-automorphisms of a group $G$ is studied. We prove some results regarding non-triviality, polycyclicity and commutativity of $IA(G)$ in addition to proving some basic results. We also prove some results analogous to a result by Schur and a weak form of its converse in the context of $IA$-automorphisms.