It was an epic of greek proportions...
We paid the server renewal on Saturday, but their system apparently flagged it as requiring manual intervention.
They then disabled the server when it expired on Sunday (why would you not prevent the auto-disabling in this scenario?)
No response from their email support on sunday.
Their canadian call centres didn't answer any phone calls at all on monday, nor requests to their english or french language twitter support accounts.
On Tuesday evening they replied to email and both twitter requests at basically the same time. They said it was an error on their side, and they had to get their control panel developers to fix it. The system reappeared a few minutes later.
I didn't want to spin up a backup on another server unless it was definitely not coming back, because I didn't really want to cause desync issues for the authentication/subscription database which is maintained via API calls from our subscription processor, otherwise we could probably have made svn work on sunday.
I'm also currently on vacation in the USA, so trying to sort this stuff out in between wasting ammo, eating corndogs and climbing mountains was frustrating!
Sorted now, though we may well look at alternative solutions to host the svn and user management stuff.
adam.
On 2018-07-24 08:51, Simon Mousey Smith wrote:
All back on :)
See reply below…
—————
Simon,
Yeah, it's back up now. The trouble was more administrative rather than technical I think. But Adam got it sorted :-)
Tom
On 7/24/2018 3:26 PM, Simon Mousey Smith wrote:
Hi,
Well the docs and website now appear back online!
FYI: http://status.ovh.co.uk/vms/ shows a nice grid of all there servers if you no ur rack num it can identify weather its off or on :)
FYI: I also dont use OVH so I dont no if anything is possible monitoring wise…
Regards
Simon
On 24 Jul 2018, at 14:16, Tom Laermans tom.laermans@powersource.cx wrote:
Hey Simon,
The server is hosted at OVH and they seem to be having some trouble... at least with that specific server.
We're alive, but all things you mention are hosted on that one single box... ;-/ Adam's working on it.
Tom
On 7/24/2018 2:35 PM, Simon Mousey Smith wrote: Hi Tom
Observium seems offline, is everything ok?
No website
No email
No jira
No updates?
Regards
Simon - Hestor
On 24 Jul 2018, at 15:50, Tom Funken t.funken@srizbi.com wrote:
Looks like everything is offline..
FROM: observium observium-bounces@observium.org ON BEHALF OF Simon Mousey Smith SENT: 23 July 2018 12:55 TO: observium@observium.org SUBJECT: [Observium] Svn update failed
Hi All
Anybody else having problems doing an SVN update today?
Mon Jul 23 00:04:01 BST 2018 Updating '/opt/observium': svn: E170013: Unable to connect to a repository at URL 'http://svn.observium.org/svn/observium/trunk' svn: E000110: Error running context: Connection timed out
Regards
Simon
Sent from my iPhone_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium