In my opinion, the real gap in the market isn’t “better safety models”. it’s turn-key orchestration platforms that provide:
- A web portal for manual moderation and data-labeling workflows
- Multi-tier moderation checks (e.g., if a keyword is detected, escalate to an LLM)
- Simple integration of custom, business-specific models (e.g., blocking competitor mentions)
- A rules engine that combines all model outputs and issues the appropriate treatments
Two Hat and Azure kinda had this, but they didn't support custom models or rules engine.
While I love the idea of redacting/auto-correcting media, e-commerce / social media companies are structurally setup against this. They'd rather stick with the status quo of rejection, than using nano-banana to remove non-compliant features (like pii) from the images.
Once, I had to anonymize student data, so we could have a prod copy on staging. So maybe there is a use-case there...