While studies have shown that children raised bilingual do take a little longer to start talking than those raised in monolingual households, this delay is temporary and certainly not a general rule. Alot of us grew up in bilingual households and grew up to be fluent in both.
My view is that you proceed with raising baby with both languages. Being able to speak Filipino will help him identify with his roots -- especially when he visits the Philippines in the future.
Magbasa pa