Unclear use of the word "our"
Solution 1:
The ability to distinguish between the inclusive we and the exclusive we is referred to as clusivity and is a feature lacking in English. Without additional context, you're correct that it's impossible to determine whether you intended to include the listener in the pronoun or not, although in many cases knowledge of the basic scenario you're describing can be enough to clarify (e.g. if you and the listener were never on a team together or at least a hard working one). However, should it fail to be clear based on such knowledge you're pretty much stuck either adding in a clarifying phrase or simply avoiding the use of our by spelling out who you mean.