I think I hear this statement more often than any other conversational opt-out; and it always has the same effect of turning my crank.
In communication, it is of utmost importance to be as unambiguous as possible. This means, among other things, that it is very important to use the proper words in the proper context.
People seem to get thrown all off track and lost when I suggest a better way of saying what they are saying. Like I've insulted them or something like that.
So they take the pseudo-intellectual highway outta there and dismiss the conversation as just "arguing over semantics", which it never was; I was just saying I understood what you meant but it may be clearer/more efficient to say (...) .
Sigh.
This is not only in conversation, or candid debate either. One of my professional tasks is web-based UI (User Interface) design. Try explaining the subtle, almost sub-conscious difference between "On/Off" and "Enabled/Disabled" to a software engineer who sees only the underlying "0/1" toggle (not fair, there's a whole context missing here, but anyways).
So, †I think I shall write a book one day called "Semantics ARE improtant, dammit! Why else would they be called "semantics!?""
Hehehehe. Then people will really think I'm insane.