We introduce and investigate two types of the space $\mathcal{U}^*$ of $s$-ultra\-distri\-butions meant as equivalence classes of suitably defined fundamental sequences of smooth functions; we prove the existence of an isomorphism between $\mathcal{U}^*$ and the respective space $\mathcal{D}'^*$ of ultradistributions: of Beurling type if $*=(p!^t)$ and of Roumieu type if $*=\{p!^t\}$. We also study the spaces $\mathcal{T}^*$ and $\tilde{\mathcal{T}}^*$ of $t$-ultradistributions and $\tilde t$-ultradistributions, respectively, and show that these spaces are isomorphic with the space $\mathcal{S}'^*$ of tempered ultradistributions both in the Beurling and the Roumieu case.