←back to thread

Fly.io Postgres cluster down for 3 days, no word from them about it

(webcache.googleusercontent.com)
797 points burnerbob | 8 comments | 20 Jul 23 23:42 UTC | HN request time: 0.972s | source | bottom
Show context
tptacek ◴[21 Jul 23 05:33 UTC] No.36810326[source]▶
Y'all, this is going to be deeply unsatisfying, but it's what I can report personally:

I have no earthly clue why this thread on our community site is unlisted.

We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread (the site is public, anybody can sign up for an account and see the thread).

Say what you want about us, but hiding out from stuff like this isn't one of our flaws. When I find out more about what happened with this thread, I'll let you know (or Kurt will reply here and tell me I'm wrong).

I don't know enough about what happened with this Sydney server to be helpful to people who had instances running on it. When I know more about it, I'll be helpful, but I'm just learning about this stuff right now, after getting back in from a night out.

Almost immediately afterwards

It looks like... all the posts in the app-not-working category are "private"? Like it's some setting on the category itself? "Private" here means you need to have signed up for a Discourse account to see them?

replies(12): >>36810339 #>>36810345 #>>36810393 #>>36810467 #>>36810497 #>>36810498 #>>36810755 #>>36810983 #>>36812367 #>>36812723 #>>36812856 #>>36834726 #
marcinzm ◴[21 Jul 23 12:06 UTC] No.36812856[source]▶
Honest advice, probably to Kurt rather than you, is you need better processes, accountability and (probably) communication in your company. The tone of your reply (and other communications from fly.io) is reflective of the lack of those things given the public sentiment regarding fly.io. At 60+ employees and so many issues that tone goes from humanly endearing to indicative of a non-scaling business. Other replies indicate you don't want the things (process, oversight, etc.) that a growing B2B business needs to really succeed which is not a good sign. Sure there's a cost to that corporate-ness and you want to minimize that cost but it's also a necessary evil for the business you're in at the scale you're at.

If something breaks once it's an accident, if it breaks twice it's bad luck but if it breaks down three times it's broken processes. Based on the comment here things break at fly.io a lot more often than three times.

replies(2): >>36814258 #>>36820707 #
1. tptacek ◴[21 Jul 23 14:32 UTC] No.36814258[source]▶
I'm just a person on Hacker News that happens to be at Fly.io; as I've said before, it's probably reasonable to think of me as an HN person first, and a Fly.io person second. My tone is my tone, and has been for the many years I've participated in this community. I got back from an evening out, saw that we were on the front page, poked around a little to find out what the hell was going on, and did my best to add some context. That's all.

If you're reading my comments on HN as some kind of official response from the company, you've misconstrued them.

replies(3): >>36814552 #>>36815698 #>>36816713 #
2. Aurornis ◴[21 Jul 23 14:55 UTC] No.36814552[source]▶
> If you're reading my comments on HN as some kind of official response from the company, you've misconstrued them.

For what it’s worth, this is the reason most companies eventually restrict their employees from making statements about the company; It doesn’t matter if you thought it was clear that is was unofficial, any statement from an employee in a position of power (such as someone with access to the control panel) will be perceived as a communication from the company.

You may have intended it to be a personal remark about your job, but there are a lot of people in this thread looking for any communication they can get about the company.

When you step in to fill that void as a person who appears to have access and power within the company, you are the official communication whether you intend to be or not.

replies(1): >>36814658 #
3. tptacek ◴[21 Jul 23 15:03 UTC] No.36814658[source]▶
Maybe I'll get restricted someday!
replies(1): >>36814947 #
4. urduntupu ◴[21 Jul 23 15:29 UTC] No.36814947{3}[source]▶
For the sake of fly.io, you should either restrict yourself and not respond or, if you can't resist, make it crystal clear, that you DO NOT represent fly.io. Your first message can and will be misunderstood and it DOES throw a poor light on fly.io.

I am a paying customer of fly.io, on the Scale plan.

replies(1): >>36816090 #
5. fdsadsaf ◴[21 Jul 23 16:19 UTC] No.36815698[source]▶
TBH I thought you were replying as the CEO of fly.io since 1) I've seen them post here before, 2) I have no idea how big fly.io's staff is and 3) your post didn't otherwise describe who you were. It doesn't look like I was the only one to be confused.

If you had said "thoughts are my own; I just work there" or something I think it would have been more clear.

6. tptacek ◴[21 Jul 23 16:42 UTC] No.36816090{4}[source]▶
Please feel free to reach out directly with your concerns. I'll certainly read any email you send me.
7. marcinzm ◴[21 Jul 23 17:26 UTC] No.36816713[source]▶
It seems you took my comment personally but it was about not just your comments but the overall tone of the fly.io communication (see recent blog post regarding funding) and approach to issues (three days of silence on a dead instance). You view processes and guidelines as chains versus as a ladder to help you climb a cliff. If the processes and communication was good then you'd know when you should self-restrict and when you shouldn't. You'd be empowered to make decisions within a framework that benefits fly.io the most versus being left to guess yourself. You'd understand why you should do that sometimes and why it's a better option for everyone.
replies(1): >>36816754 #
8. tptacek ◴[21 Jul 23 17:29 UTC] No.36816754[source]▶
I don't, but that's fine: it's not important that we understand each other all that clearly here, since all I'm talking about is how our public forum works.