Japanese workers turn to resignation agencies

So how is it the company can “refuse to let them quit” or “force” an employee to go to a temple? What is the actual enforcement mechanism other than guilt?

Apart from the paperwork that you need to pass on to your next employer, salaries can be structured in a way that makes the base pay rather low, but the yearly income is boosted up to a reasonable level by bonuses, overtime allowance (fixed monthly amount paid whether you work the overtime or not), etc. If the company doesn't want to let someone quit, they can make it financially painful by withholding these things, or subtracting "damages" caused by the employee leaving.

EDIT: Come to think of it I'm actually not 100% sure about the legality of this, but they sure try it!

They can reduce or plain not pay your bonus, yeah. But making the employee pay for "damages", even by subtracting them from remaining pay or bonuses, is very very illegal.