If you set up your ML so that it works x% of the time, you might very well have a profitable business even if you end up accidentally screwing over a bunch of folks. But no competitor can challenge you in the marketplace because the human cost of answering phones and emails to find that last little bit of efficiency is overwhelmingly disproportionate to any economic value the business would gain.
Many of us like to bang on businesses as being amoral and impersonal, but most are trying to do something people want, only better and more efficiently. ML may be providing an upper limit to efficiency by taking out any opportunity to do some serious analysis. Because in many cases removing that last 1-5% in inefficiency is the bit that leads to a completely new way of working, in many areas we may be boxing ourselves in to a very long-term status quo.