We present the $p$-adic probability logic $LpPP$ based on the paper \cite{5} of A. Khrennikov et al. The logical language contains formulas such as $P_{=s}(\alpha)$ with the intended meaning ``the probability of $\alpha$ is equal to $s$", where $\alpha$ is a propositional formula. We introduce a class of Kripke-like models that combine properties of the usual Kripke models and finitely additive $p$-adic probabilities. We propose an infinitary axiom system and prove that it is sound and strongly complete with respect to the considered class of models. In the paper the terms finitary and infinitary concern the meta language only, i.e., the logical language is countable, formulas are finite, while only proofs are allowed to be infinite. We analyze decidability of $LpPP$ and provide a procedure which decides satisfiability of a given probability formula.