I accidentally ate moldy bread, will it affect my milk?

I'm in a total tissy, was cleaning up dinner and came to the horrifying realization that I consumed a sandwhich made with moldy bread (it was along the bottom crust so I didn't notice at first). I'm worried it will affect my milk, does any one know if it will do so?