They can hold onto it, and never return it. They can deface it. All of that is a possibilty.
You could argue, a sufficiently locked down phone is a better alternative. If they do something, you'll only lose $$
But they can't potentially look at your banking app, read private notes, messages and emails, operate your home automation, look at your calendar, etc. if all they have is a plastic card.
Only for it's "original" use case - traffic laws enforcement. I don't think any other entity can validate if this piece of plastic is invalidated or not. Also, it's not like information on lost ID gets erased when you get a new one: still has your address, DOB and other info that can be misused.
I once had three valid drivers' licenses, because my wallet was stolen (later returned), and I left my ID at a bar. All three were valid for use at the same time despite being reported lost/stolen - they had identical barcodes, etc.