The sticking point is English ability. It's genuinely a very difficult thing to learn a second language to the point of being workplace-functional in it, and an additional level of daily stress and mental effort to do everything in your non-native language. The culture here doesn't exactly reward risk-taking and failing in public, and unfortunately becoming proficient in a language requires that you stumble through things a lot while you practice. Furthermore, most Japanese people have experienced fairly ineffective methods of teaching English in schools and ended up rather demoralized about their ability to learn it. (And if you're currently in a job that demands a lot of overtime, when do you study?)
This is why you see plenty of Japanese (and Korean) expats across China and ASEAN. A mid-level Japanese employee can become a senior manager in Thailand or Vietnam while earning a similar salary to Japan ($20-40k).