In this paper, we consider all the non-metabelian groups of order $162$ and characterize the structure of the unit group of the corresponding group algebras. Overall, there are $55$ non-isomorphic groups having order $162$ and $11$ among them are non-metabelian. We study the unit group of the semisimple group algebras over any finite field whose characteristic does not divide the order of these eleven groups.