James clearly said that you would not figure it out immediately. But after you'd heard the word used in a variety of circumstances you'd figure out that a phoonwaffle is a rock, not a generic word for paperweights nor the given name of his individual low-maintenance housepet.
People learn language by immersion all the time. In fact, I'll let you in on a linguists' secret: The vast majority of language learning on this planet is not just done by immersion, but it's accomplished by
tiny children who don't even realize what they're doing and have no formal training in the discipline. They seem to do just fine with it, don't they?
But to return to your original question, yes indeed we are all constrained by the limits of our language. Unless you happen to be a professional sculptor, athlete, photographer, musician, carpenter, etc., most of your thoughts are formed in words. So clearly the language in which you think not only shapes your thoughts but constrains their domain.
This is why it's so important for people to learn a second language, preferably in childhood when it's so ridiculously easy. The less closely related the two languages are, the better. It gives you two different ways of thinking, so you can reality-test your thoughts in one language against the paradigms of the other.
For example, Chinese has no gender. Once you've studied Chinese thoroughly enough to have thoughts in Chinese (which can occur long before you're fluent, take my word for it) you start to wonder why you've always unconsciously assumed that all doctors are male and all teachers are female. "Ask your doctor if he will..." "Ask your children's teacher if she has ever..." In Chinese "he" and "she" are one word.