Or is it? There are a couple of ways in which this apparent contradiction could be resolved. The most obvious would be if you had a special pronoun used only when the speaker was also the addressee; but, as such a form would be used only in talking to oneself, it would be unlikely to catch on enough to become part of the language. Less obviously, however, you could have a singular pronoun being used in a sufficiently vague way to refer to both the speaker and the addressee (but not to an uninvolved third person.)
Soon after moving to France, I realised that, in public announcements, this is in fact what French does with its 1st person singular pronoun je. The realisation was prompted by a poster in a medical insurance office saying, in big letters, something like:
Je choisis le générique, je ne fais pas d'avance de frais.
(I choose generic drugs, I pay no advance.)
This was clearly not a piece of self-observation someone had put up; rather, it was intended to tell us "Choose generic drugs, and pay no advance". Over the following days, I noticed that concealed exhortations of this form were everywhere: Oui je vote (Yes I vote), En car comme en voiture, je boucle ma ceinture (In a coach as in a car, I buckle my seatbelt), ... All easily understandable as conveying the message is "I do this, and so should you". But in English, you consistently cast such messages in the imperative, with no "I" at all: "Please take a moment to cast your vote in this important election" or "Buckle up, it's the law", and so on. One obvious side effect is that the slogan "Je suis Charlie" has at least one reading directly accessible to French speakers but not to English speakers who understand it word for word: namely, "I am Charlie, and you'd better be Charlie too".
The difference between the two languages in this respect is at the level of pragmatics, for now. But if such hortatives become sufficiently common in French, one could well imagine the construction grammaticalising further and even eventually becoming distinct from ordinary 1st person marking. In that case, we might end up with a true 1st person singular inclusive pronoun: a pronoun that simultaneously means "I" and "you", while taking strict singular agreement. Give it another 500 years...
Are you familiar with another language that does this?