Impolite? Me?

Sat in office at co-working space and vague entrepreneurial acquaintance wanders in to say “Hello.”

His phone rings, and he states: “Yeah, could you step outside for a minute? I need to take this.”

I sit there, blink a few times, and say “You do realize this is my office, right?”

And without missing a beat he replies “Yes, but I need to take this call in private so you need to leave.”

Anybody want to guess what my response was?