I wonder if there's ways to get the benefits of this while avoiding privacy issues. The best way I can think of preserve others privacy in an unobtrusive way would be to detect & remove other people's voices from any recordings (private by default). The only way to override this would be if others say "its ok to record" or equivalent phrase.
That way you get meeting notes (where people consent) but not conversations in confidence (where people don't). For the in-between cases that users wanted to remember, they could repeat back what others say (e.g. other: 'that is $59' user: 'ok $59').
If this were accurate enough it seems like it'd address most of the privacy concerns, and is a pattern that can be used by all types of recording devices we'll be seeing in the coming years (and may be a good basis for legal regulation of this class of device).
Zooming out, we're early in the cycle of this kind of tech, where forgetting is seen as a problem. There's a chance we over-shoot and remember too much, which is going to introduce its own problems because tech doesn't have a sense for what things are important psychologically for us to move on from and not have access to. Much like in other domains where tech gives us too much of what we want, I worry the response will be a dismissive "just don't look at it" personal responsibility argument. But that's further down the road from where we are.