This actually happened to me at work the other day. It was far more embarrassing because I sit at a pod like circular desk with 5 other people. So as you can imagine, it was quite clear to the others that I was talking to someones shirt.
Ironically I was more relieved then embarrassed. His silence caused me to think I had offended him, but more importantly that he wasn’t impressed by my amazing wit.