Discussion: Server Failure on Sunday, September 17, 2023

[Nikiski Dave]

David I was so close to calling my shrink. But at $100 an  hour he is not not my friend..   Where do you want me to send you the check?

[TGO David]

10 minutes ago, bubbadavis said:

I really want to hear the nitty-gritty details of the old hosting company - after you get some rest! I'm used to servers having at least RAID arrays; while not foolproof, they usually help with data loss. 

We've used them for a few years.  Last year they were purchased by another company.  The prices went up but they [quietly] abandoned the Managed Service Provider model and went to pure bare-bones hosting.

When I reached out to them for assistance yesterday, I was advised that they no longer did that sort of thing and that it was up to me to handle every aspect of the recovery.

So, I hired a new company last night -- one that does do managed services -- and fired the old company today.

 

[TGO David]

6 minutes ago, Nikiski Dave said:

David I was so close to calling my shrink. But at $100 an  hour he is not not my friend..   Where do you want me to send you the check?

Had it taken another day to rebuild, I would have told you to just send your money to the local liquor store under my name. 

 

[Nikiski Dave]

Just now, TGO David said:

Had it taken another day to rebuild, I would have told you to just send your money to the local liquor store under my name. 

 

Is  Bourbon covered under medical?

[TGO David]

2 minutes ago, Nikiski Dave said:

Is  Bourbon covered under medical?

It really should be. 

[Alleycat72]

I assumed FBI....because FBI.

[papa61]

glad we are back,,,I was in a panic. I may not post a lot but I do read a lot (too much)?

[TGO David]

22 minutes ago, papa61 said:

glad we are back,,,I was in a panic. I may not post a lot but I do read a lot (too much)?

Honestly that's me these days.  I read more than I post.  With shorter days and longer nights around the corner, I'm sure that my activity will pick back up. 

 

[bubbadavis]

43 minutes ago, TGO David said:

We've used them for a few years.  Last year they were purchased by another company.  The prices went up but they [quietly] abandoned the Managed Service Provider model and went to pure bare-bones hosting.

When I reached out to them for assistance yesterday, I was advised that they no longer did that sort of thing and that it was up to me to handle every aspect of the recovery.

So, I hired a new company last night -- one that does do managed services -- and fired the old company today.

 

Good choice! Nice they decided to cut services while increasing prices...

[DWARREN123]

Sometimes the bad stuff happens because of the MSP, other times it's because of the MSP!

[RED333]

I was sure the .gov had knocked the site off line, ya know guns and all. Then I saw the post about what had happened and was thinking that FUJIMO had posted something that had done the hardware in.

Just glad the site is back up and running!

I do see my user name in the upper right corrner twice, a bit off set. Fixed after I reloaded my avatar pic.

Edited September 19, 2023 by RED333

[Ronald_55]

Too soon?

 

From someone that has been there with corrupt backups and a failed mission critical server.. I feel you.. Glad you were able to ditch the, as a coworker once put it, black hole of a support team and move to another hosting company.

[jpx2rk]

Never had to do anything like what happened and don't want to.  Glad you got it fixed, but I bet it was not fun times for all.

[peejman]

Thanks for providing this community. I also read a lot more than I post these days. 

[NoBanStan]

@TGO Davidas a fellow computer dude, you mentioned hardware failure. Is TGO running on dedicated hardware? I don't know the complexity of the infra, but would have assumed it would be all virtualized/cloud hosted.

definitely not to question your solution or anything, i'm just curious

[TGO David]

3 minutes ago, NoBanStan said:

@TGO Davidas a fellow computer dude, you mentioned hardware failure. Is TGO running on dedicated hardware? I don't know the complexity of the infra, but would have assumed it would be all virtualized/cloud hosted.

definitely not to question your solution or anything, i'm just curious

We were on dedicated hardware since around 2013 or 2014, but we are back to a virtualized solution now.  MacGyver and I are starting to research just moving it all to Azure for better scalability, tough.

 

[NoBanStan]

Just now, TGO David said:

We were on dedicated hardware since around 2013 or 2014, but we are back to a virtualized solution now.  MacGyver and I are starting to research just moving it all to Azure for better scalability, tough.

 

Azure guy here. Holler if needed.

[RED333]

I am guessing a few of the newest PMs were lost. Not a big deal, just letting y’all know.

[TGO David]

4 minutes ago, RED333 said:

I am guessing a few of the newest PMs were lost. Not a big deal, just letting y’all know.

Yes - anything done after August 17th until yesterday would be gone.  PMs included.

[RED333]

2 hours ago, TGO David said:

Yes - anything done after August 17th until yesterday would be gone.  PMs included.

Thanks for the answer and all that y’all do to keep this site up and running.

[gun sane]

[TGO David]

7 hours ago, gun sane said:

Thank you. 

[bersaguy]

I am just glad we are back up and operational and Thank you for all you do to keep us running!!! Like a few guys have said, I don't post as much as I use to but do come and read about everyday.