B.-Y. Chen and O. J. Garay studied pointwise slant submanifolds of almost Hermitian manifolds. By using the notion of pointwise slant submanifolds, we investigate the geometry of pointwise semi-slant submanifolds and their warped products in Sasakian and cosymplectic manifolds. We prove that there exist no proper pointwise semi-slant warped product submanifold other than contact CR-warped products in Sasakian manifolds. We give non-trivial examples of such submanifolds in cosymplectic manifolds and obtain several fundamental results, including a characterization for warped product pointwise semi-slant submanifolds.