By using soft-x-ray linear and magnetic dichroism on La0.825Sr0.175MnO3/PbZr0.2Ti0.8O3 ferromagnetic-ferroelectric heterostructures we demonstrate a nonvolatile modulation of the Mn 3d orbital anisotropy and magnetic moment. X-ray absorption spectroscopy at the Mn L2;3 edges shows that the ferroelectric polarization direction modifies the carrier density, the spin moment, and the orbital splitting of t2g and eg Mn 3d states. These results are consistent with polar distortions of the oxygen octahedra surrounding the Mn ions induced by the switching of the ferroelectric polarization.