Why were unions specifically to blame for your family leaving?
Union versus non-union is a quality versus quantity problem. Unions restrict the labour pool to increase wages. When that protects specialisation, it increases productivity. When it artificially constrains the labour pool, it decreases it.
A unionised job market showing unemployment (or underemployment) is usually an indication of the latter.