Reliability: It’s not great

I'm not a user of Fly.io. I can't help but notice how remarkable the effect of open communication on potential end users like me. I remember reading about their reliability problems on HN some time ago. That biased my view of the company. After reading this, the open communication and transparency restored my trust in them, and would make them again a potential candidate for future projects. Because now I know that they acknowledge the problem and that they are trying to improve things.

mrkurt[1] is also active here and has been very transparent in his comments about scaling issues.

Similar to this post he commented a week ago:

> In a year we'll either be ahead of those, or not growing anymore due to ongoing capacity issues. I'm hoping for the former.

I am rooting for Fly! Great team. The company reminds me of early HashiCorp.

[1] https://news.ycombinator.com/user?id=mrkurt

Agreed, this is how company communication should be.

I don't use Fly but would consider them in the future even given their recent issues.

I look at this in contrast to Twitter who had/has? an outage today. Their leadership is opaque and doesn't take responsibility for the issues they are causing.

Me too.

However, this is a double edge sword. Their key value proposition is scale / speed which makes it concerning that they haven't "solved" that yet.

This post is carefully worded corporate messaging, but because they write for their developer audience it has an informal "oh shucks we messed up bad y'all" vibe to it. But make no mistake, this is 100% corporate messaging.

I get that growing is super hard. And maybe fly will grow up to be a good platform some day. But that's the future. Today, they're flying by the seat of their pants and I mostly feel sorry for people who were tricked into thinking this platform is ready for production use.

I'm not sure why the cynicism around their candor. Do you think it's not genuine just because it was posted by a company employee?

Your post implies corporate messaging is bad. And anything posted by a company—or at least I don't know where you draw the line—can be considered corporate messaging. Am I just reading too much into your phrasing?

professionals don't get tricked into thinking a platform is ready for production use

If you don't have SLOs and SLAs, then you get what you get, essentially. Even a company with a great reputation can completely reverse course with a single bad incident, and you get nothing in return if there's not a contract.

I don’t get what you’re saying, this isn’t a brag disguised as a confession, they are actually admitting to poor performance, of course it’s to eventually make users trust them, but a) I don’t see nothing bad with that b) they are choosing the hard route.

They are being open and transparent (afaik) even if carefully worded, which I also don’t blame them for.

This is probably therapy, but your message and fly.io's post resonates a lot with what I'm going through. I took a product owner role about 6 months ago, my first, with a company that has turned out to be just a mired mess, and a product universally hated both internally and externally.

Long story short, it's completely over-engineered by a bunch of intellectual engineers with no focus, no discipline, and no oversight. It ended up not delivering on any promises it made, and there were a lot of them.

I was warned left and right before presentations and meetings, "this customer hates your product because of ...." I started off every meeting with saying, "we're rearchitecting the product, this is how we're doing it, this is the tech we are using." Immediately there was a sense of relief from customers, followed by questions like, "why can't <current product> deliver <feature> that was promised?" I'm completely honest with bad decisions that were made and how it impacted the feature. Sure, there is skepticism on what we are doing, and I tell them they should absolutely be skeptical based on our track record. The result has been customers who have hated my product now offering to work with us on development.

I've also been completely forthcoming on configuration, security, resources, and setup issues I am finding, many of them are absolutely freakin' insane. I've flat out told customers it's frankly embarrassing and never let us do something like this in the future. The best feedback on this was, "At least you're telling us something. We usually get silence from this team."

God, this is the most depressing job ever.

It's strategic messaging. It can't be genuine, because of what it is. The benefit they get is publicity and damage control, and as you can tell by the many responses here, it buys them time because many developers are willing to give them the benefit of the doubt.

Companies that engage in this kind of candor are careful not to disclose those things that would really hurt their business. Those things are still kept secret. If the CEO accidentally sexually harassed an employee that's not getting disclosed. A mea culpa is only offered for the issues that are already known regarding scaling, downtime, and missing features. Struggles they have because they're choosing to grow so fast.

Architectural astronauts.

In fairness, a CEO who has basically been Kanye-ing himself and his company into irrelevance is a low bar.

This is huge. Even as a member of a larger company, this stuff matters. If you have a vendor that doesn't bullshit you when things go wrong, you can actually trust. This is how you avoid companies having the "hmmm they seem to be having lots of issues recently, let's consider moving off them" conversation.

I feel this. I hope you get over the hump and your job gets fun. We've had flashes, at least, but I do think what we're doing (and probably what you're doing) require some irrational behavior.

Open communication is great when there are incidents, but even better is having no incidents. (of course there are nuances depending on specific context)

Can you help me in a detailed sense - what did you tell customers? did you literally say there's product is "completely over-engineered by a bunch of intellectual engineers with no focus, no discipline, and no oversight"? That seems a little over-honest to me but of course I wasn't there.

Part of what I hated about Product Management at my last role was the consistent helplessness I felt when I was on calls with our customers. I could tell our product wasn't meeting their needs but all I could do was try my best to give the engineers context on how best to eventually meet them.

I remember my first few days on the job just being ripped to shreds by our customers who (understandably) were slighted. Don't miss those days at all.

Your job sure does sound depressing, and it's not one I would succeed at, but if you can power through and turn this product around that's a hell of an accomplishment you'll have to be proud of.

I'm curious what you'd like to do next. You could probably have a great career doing these sorts of turnarounds repeatedly across companies, maybe even as a consultant, but would you want to?

Sorry, what? Do you expect that no company can think about what to write before they post it, or that any post about anything internal must cover all internal issues? Posts must be either all roses or a no-thought laundry list of everything bad?

Honestly, if you are a small fish to AWS... what is an SLA?

They can trot out a low level person to stall you with questions, or an AI question generator that maximizes the amount of time you waste on your end, and call that "SLA met".

And even if they DON'T meet the SLA on occasion, you built your stack on AWS. You are laying in the bed you made.

SO, what, AWS throws some free credits (that their 30-40% margin easily absorbs)?

The only big stick in these types of things is having dual-cloud capability, where you can move your service quickly from one cloud to the other. Stateless API servers? Maybe. Database servers? ouch. Cassandra could reliably span two clouds, man would AWS kill you on their ludicrously overpriced network costs.

Has anyone does Postgres replication across providers as a useful production system? Doubt it.

> Do you expect that no company can think about what to write before they post it...

I guess, you and GP are in agreement for the strategic part of the argument at least, if not the genuine part of it.

As someone who's been active on Fly's community forums for close to 18 months now, I think Fly employs some of the most genuine and helpful engs you'll see, so I'll give them the benefit of the doubt.

Of course not. The point is that the reader should recognize corporate communication for what it is: fundamentally self-serving. Corporate communication should therefore be met with thoughtful skepticism, and not with naivety or cynicism.

I don't agree. Fly.io took on 40m in venture capital and is now on a do-or-die mission towards a $400 million exit. This might be obvious to many here on HN but most developers don't understand these startup dynamics and the consequences of a growth-at-any-cost strategy.

Yeah I agree with you. Corporate messaging is still corporate messaging. If you go the candor route, it's a tactic of many you can employ.

I’m not [directly, at present] in the market for their product but the very frank and real introduction basically moved Fly from my “I think I’ve heard of that but I’ll need my memory jogged for any further recognition” category to my “potentially good stuff to reference when relevant” category.

This kind of frank and human communication is vulnerable, but it’s good for establishing credibility… with me at least!

When I was in an engineering IC version of this role, I longed for someone like you to have my back in management and have customers’ backs too. If I had a time machine and a magic lamp I’d team up with you.

As a future representation of past me, I can tell you:

1. Everything it’s making you feel is valid.

1b. If you’re feeling burnt out, please listen to it. It gets worse if you let it.

2. While I can’t hire you now, I can already tell you’re eminently hireable. If you have any cautious inclination to move, you will probably be better served by greener pastures.

3. Just take care of yourself.

4. When 3 contradicts 2, favor 3.

There is a world of difference between the tone and content of that Fly post and the tone and content that most people expect from cover-your-ass corporate blog posts.

Sure, both are examples of "self-serving corporate communication" - but it's clear that the way Fly communicate here is more valuable and trustworthy than so many other examples of this kind of thing handled poorly.

> that's a hell of an accomplishment you'll have to be proud of.

It's hinted by the C-level that if I can pull this off, it would be nothing short of a miracle. I'm pretty sure I can negotiate salary, education, bonus, and what not if I can pull this off.

As far as next, I've thought about that. It would be funny to call myself a turnaround specialist. This would be quite a remarkable feat, but I really don't know if I would have taken this job if I knew what a mess this was...

I have a POC I'm trying to get out in a couple of weeks. It's for the #1 feature that was promised in this product, never possible due to the architecture, and we've gotten raked over the coals for it. Wish me luck, because if I can get it out soon, it's going to be downhill for a while.

It depends on the audience.

First off, it helps I've been 15+ years as an engineer, 5 as an engineering manager, and throughout have the community contributions in the field on my resume. I instantly spotted the problems when I was given an architecture diagram on day 1 and discussed what I would do differently. All that gives credibility.

If it's internal audience, I am brutally honest. The organization needs to know this wasn't happenstance and bad luck that put us where we are now. It was a deliberate series of bad decisions based on a poor engineering and product culture. Now, for better or for worse, we are tasked with paying the debt.

There's a certain class of customers that are sister companies under the same parent. I'm honest with them, too, but go on the offensive. They have abused my team and our company in the past, and unfortunately, we have let them. I am more than happy to fire back and go toe to toe with bad behavior, and at the same time working to fix critical support issues.

For external customers, I've had remarkably good response in listening to their complaints. I am honest in discussing, in deep engineering detail, how the new product will address their problems, where issues might still be, and development timeline. I like to think the credibility portion comes into play here. In the past, customers were just told, "We'll look at it" and "We'll fix it" but nothing was ever planned.

Man, you don't know how this impacted me.

My boss is supportive, but he's also under heavy fire. Like I mentioned, my peers are rightfully skeptical. My team are a bunch of sharp, good guys, but haven't had any good guidance of mentorship in years, if not decades. They're all different, but what the have in common is that they've been screwed and judged unfairly thanks to past incompetence. That just pisses me off.

There's hope from those around me, but it's a pretty darn lonely job. You just gave me the fuel to not feel already beat up when I walk in the door tomorrow.

it is.

i do enjoy programming, working with other devs, but as soon as i stepped into a product management role it's a hellishly different set of problems. you're in the middle of the tech, the developers, the problems, and the customers. lots of lessons in there. tiring, but worthwhile.

> I'm pretty sure I can negotiate salary, education, bonus, and what not if I can pull this off.

Do this up front. Do it as soon as you possibly can. You will lose a huge amount of negotiating leverage if you "wait until you show them". I cannot stress this enough.

Really? For me it's the contrary, posts like this from companies are a signal to say "Jump out of the ship while you still can".

Kinda like when crypto exchanges tweet "Yo we're definitely not blocking withdrawals, we're perfectly healthy".

You know what would make me consider a company? The fact that they don't have a bad reputation to begin with, and don't need to make posts like that to try to save their reputation.

> This is how you avoid companies having the "hmmm they seem to be having lots of issues recently, let's consider moving off them" conversation.

Yeah, this is how you get companies to have the "well, it SEEMED they were having lots of issues, now it's clear that they indeed did, moving off of them is priority #1".

This is why engineers need to be on some customer calls. They can be told not to talk at first or be trained in customer etiquette, but nothing makes a difference like hearing this from the users.

Just to reinforce this suggestion, this was my first thought also. Go get your comp mate.

Could you further elaborate on “intellectual engineers”? What mistakes were made? Esoteric languages? Obfuscating design patterns?

Partially so I could learn from mistakes and partially since I’m a sucker for post-mortems :)

I don't think this means that it can't be genuine. I mean, the intent was to buy some good will with their customers who have experienced problems, that's for sure. But it's not a "bad" motive I think.

If I had a bad day and didn't get to complete something within my estimate, I'll tell my boss I had a bad day and ask for more time. Does that mean I have ulterior some ulterior motives? No, I just had a shit day, and needed some compassion.

They have been going through a rough patch recently with their scalability problems. And they realised they might not address it as easily or as quickly as they'd like. So they just wanted to buy time. I think that's better than "bunkering" and not letting your customers know what's up.

They do have the benefit that their audience is tech savvy as they are, so they can go into more details (and be less formal, I suppose) to get some understanding from their customers. As in, most devs have struggled at some point with a problem that exceeded the initial scope/time estimate. It sucks, and we know it sucks. So, why not give them the benefit of the doubt here?

Like, I think understand what you mean: their goal was to buy more time, and they achieved that. But even though it was corporate messaging, I still think it was genuine. I assume they felt a bit like "ok shit, we gotta talk to our customers, they deserve to know what's going on".

I guess they wouldn't air most their internal issue, since those don't aren't felt by the customers. So there's no need to apologise and explain themselves.

There is a part of the app that extracts and parses a database log. There is C, Java, Perl, Go, and bash scripts all over the repo. The bootstrapper is written in Java, but the core work is done in Go. Commandline arguments that are documented in comments may or may not even work. The Go section is one big state machine.

That's just to take a database log, put it into JSON, and zip it up.

That all for just one step in the data pipeline. The others are slightly less hairy, but 75% of this pipeline is literally just moving data around. It goes from JSON to MySQL to Postgres to Parquet. There is no data enrichment at all during these steps. It literally just unpacks from one format, packs to another and repeat.

The whole fucking thing is just one big masturbation circlejerk for a bunch of engineers that have thankfully been RIFed/forced out...

Agree with GP - your post makes me want to hire you.

One trick you might try: write future press releases. This helps you look beyond the immediate problems and focus on the destination. For example:

“Q3 2023: ACME CO released version Z today, which dramatically simplifies our engine to focus on core user needs. ‘It does thing I want an doesn’t crash anymore’ says Key Buyer #1”

By writing this down, you can put the vision in front of everyone. Then check it against actual progress to see how you’re doing.

If you have any interest in writing a series of blog posts about your experiences in the “turnaround” and linking to them in your profile I believe you would have an audience. Your explanations are clear and your experience is worthwhile. Just food for thought.

Same! It’s the same principle as CEOs and stock options. For instance Musk that set several goals (x% market share, x unit sold…) and the hardest they get the more he will receive compensations.

I am not sure how much you could negotiate but you can have something like that and being metric based. X% customers happy, x% rating change, x% customers retained when they were close to leave. Then you make the math of the revenue and profit and it’s hard to say no.