I don't think it is wrong to have a bit of banter with refs. My Dad would always manage games by talking to players and so did I when I reffed.