i think it is really important for you to talk to him and get things straight.
if he forced you out of the job, there is no way he can behave with you now. make sure he knows it loud and clear. also, for your own benefit, why dont you look for a work from home option?