Phosphate is normally soluble because water forms hydrogen bonds with the oxygens of phosphate. Remember that like dissolves like.
Group 2 cations, like hydrogens, are positively charged. They, however, are larger than protons, and more effecient at neutralizing the negatively charged oxygens.
Calcium finds the negative charge of phosphate and surrounds it, preventing water from forming hydrogen bonds and solvating.