The bar for legal consequences is expected to be much higher than mere association.
It has never been perfect, nor uniformly applied in all circumstances, but it is and should remain a nominal goal of the justice system. For that to no longer be considered the case, even in a casual conversation like this, is a devastating shift of the Overton window towards authoritarianism as the norm.
It may not literally be guilt by association, but they’re two parts of the same whole in this case, right or wrong.