Talk to grandpa not to give your son whatever he wants. For example, if he wants the ipad and pester grandpa for it, grandpa may tell him its not the time to use ipad now and may offer an alternative toy/book/activity for him to play. If grandpa needs to be away to do some chores/task, get grandpa to tell him that he would be away, but would come back within xx minutes and stick to it. (eg: I need to hang up the laundry, you can follow me to the laundry or you can stay here and play with the lego and I will be back in xx mins.)
It may be difficult at first because your child is used to getting what he wants. The key is really to be consistent in your approach among all the caregivers, grandpa, you and dad. By being firm and consistent, your child will slowly understand. It will get better! Good luck!
Charu Saini