For a finite group G, let Z(G) denote the center of G and cs∗(G) be the set of non-trivial conjugacy class sizes of G. In this paper, we show that if G is a finite group such that for some odd prime power q ≥ 4, cs∗(G) = cs∗(PGL2(q)), then either G PGL2(q) × Z(G) or G contains a normal subgroup N and a non-trivial element t ∈ G such that N PSL2(q)×Z(G), t2 ∈ N and G = N.〈t〉. This shows that the almost simple groups cannot be determined by their set of conjugacy class sizes (up to an abelian direct factor).