The opening of Ham on Rye where Buk is under the dinner table and wondering what is being said goes on to state:
"I was 'Henry Jr.' These people spoke German most of the time and in the beginning I did too."
Take that with a grain of salt. I doubt he actually spoke much, if any, German. Recounting 3-5 years of age, it may have been literary license, IMO. But, possibly...