"English not spoken here..."
Reluctance to use English does not equate to being xenophobic. Japanese speakers often struggle with English as a second language, and it is important to consider the reasons behind this phenomenon.
For various political and non-political reasons, people love to push the agenda that the majority of Japanese hate all foreigners and treat them badly. But it's just not true. A major component of Japanese people's hesitation in dealing with foreign visitors to their land can be summed up in one word: fear. They fear their own inability to speak other languages than Japanese. They fear making mistakes and upsetting the people they are speaking with, too. Since their own language is so complex to master, many Japanese never invest the time and effort in becoming fluent in any other language.
Japanese language education is a key focus in Japanese schools, with students learning to read and write in Japanese from a young age. Japanese is a complex language to acquire, with three distinct writing systems to learn - hiragana, katakana, and kanji - and then its own unique grammar structure and a severe context dependency. The language is an important part of culture and identity in the country.
[The three forms of Japanese writing, all in use in one ordinary sentence. Can you read this?]
Most foreigners who travel seem to expect basic English, at least, to be spoken by all shops and restaurants everywhere they go. But while English is a widely spoken language around the world, it is not so in Japan. While Japanese people may know a few basic English words or phrases, the majority become quite uncomfortable stepping outside their native language.
English is a language with vastly different grammar and pronunciation than Japanese, making it challenging for Japanese speakers to achieve any level of fluency. English is not widely spoken or practiced in Japan, especially compared to countries where it is a primary or secondary language, which further hinders language acquisition.
[ This sign's English and Chinese say there is no room. But in Japanese it is, "Those guests who can read Japanese, please enter!" If your Japanese allows you to read a simple sign, you are welcome in their shop. Doesn't seem very 'xenophobic' to me.]
Xenophobia is a fear or dislike of people from other countries or cultures. Far from being xenophobic, most Japanese are openly curious and friendly toward visitors from other countries. They are intrigued by differing languages and cultures, appearances and actions. It is important to recognize that not every Japanese who is hesitant to speak English is necessarily against any other cultures or people!
The number-one reason why someone is reluctant to use English, is their own lack of confidence in their language skills. It is unfair to label anyone as "xenophobic" based on a language preference without understanding the context behind their choice. Try to approach the issue with empathy and an open mind, instead. Learn a bit of Japanese yourself! Smiles, attempted greetings, ordinary politeness and saying thank you, go a long way toward "melting" any hesitation you see forming.