Long story short, it's completely over-engineered by a bunch of intellectual engineers with no focus, no discipline, and no oversight. It ended up not delivering on any promises it made, and there were a lot of them.
I was warned left and right before presentations and meetings, "this customer hates your product because of ...." I started off every meeting with saying, "we're rearchitecting the product, this is how we're doing it, this is the tech we are using." Immediately there was a sense of relief from customers, followed by questions like, "why can't <current product> deliver <feature> that was promised?" I'm completely honest with bad decisions that were made and how it impacted the feature. Sure, there is skepticism on what we are doing, and I tell them they should absolutely be skeptical based on our track record. The result has been customers who have hated my product now offering to work with us on development.
I've also been completely forthcoming on configuration, security, resources, and setup issues I am finding, many of them are absolutely freakin' insane. I've flat out told customers it's frankly embarrassing and never let us do something like this in the future. The best feedback on this was, "At least you're telling us something. We usually get silence from this team."
God, this is the most depressing job ever.
Partially so I could learn from mistakes and partially since I’m a sucker for post-mortems :)
That's just to take a database log, put it into JSON, and zip it up.
That all for just one step in the data pipeline. The others are slightly less hairy, but 75% of this pipeline is literally just moving data around. It goes from JSON to MySQL to Postgres to Parquet. There is no data enrichment at all during these steps. It literally just unpacks from one format, packs to another and repeat.
The whole fucking thing is just one big masturbation circlejerk for a bunch of engineers that have thankfully been RIFed/forced out...