Powered by eUKhost®

Announcement

Collapse
No announcement yet.

Don't allow muppets to mess with client servers

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Don't allow muppets to mess with client servers

    Long story, but here is the short version...

    EUKHost force us to change IP addresses on our dedicated server, we try to say we really don't want this to happen as a lot of important things hosted on this server, but the response in the end is if we keep putting it off: "your server will go out of network on 7th August".

    So, we agree to a time-line for the server to have both sets of IPs so we can move everything over. They go in and add the new IPs, and tell us all we need to do is change the IP in Plesk for each domain. So we start with a non-important domain, switch the IP address over to a new one, and it stops working completely, in IIS Manager it shows the state as [Stopped] and gives a pretty generic error if you try to start it.

    Here is the very short version from there:
    - we update the ticket stating the problem
    - we think we fix the problem (within an hour), so update ticket again saying what we did:
    "I think we have managed to do this for this website, maybe by adding an ipaddress to the iplisten using:
    httpcfg set iplisten -i NEWIPADDRESS"
    - they reply saying no, we don't have to do anything but change IP
    - we reply saying we did that and it didn't work, what else are we missing then?

    Then the next day I'm getting messages all over the place from our monitoring systems saying all our sites are offline. I log in to the server, and EUK have someone logged in as admin, I ask what's going on and he tells me he's updated all our domain IPs for us. We did NOT ask for this, and certainly would never have expected anything like this from the ticket posts.

    Turns out he made a massive mess... First of all, the problem above still existed, our main (and most important) website was in [Stopped] state, wouldn't restart, just like the first one we tested. I told him about the httpcfg thing and if it was required for this as well (as this site is on an exclusive IP), he dismissed that and insisted on fixing it all for us himself. Many long conversations later, he'd messed with all sorts of things (we have a lot of subdomains), finally ended up trying to revert his changes.

    5 whole hours later, after him insisting that he (and his system admin) are working on fixing it, he comes back and tells me he fixed our problem and to get on chat to talk to him (he refused to ever actually call me like I asked many times). After another long wait while he is not available, I ask the story is, and here you can see the result:
    Muppet: The netwok error has been fixed
    Muppet: network*
    Dave: what was causing it?
    Dave: and what do we need to be aware of from here onwards?
    Muppet: I will let you know
    Muppet: Please wait
    (quite a long time later...)
    Muppet: The new IP's was not binded on the server properly
    Muppet: I have re binded the IP's using following command
    Muppet: httpcfg set iplisten -i “IP”
    Muppet: Usually this issue occur on windows 2003 server
    Dave: great, so this is EXACTLY what we suggested to you the problem was in the ticket...
    Muppet: we can connect the server but the IP's can't be used in the IIS for websites
    Dave: and again in our chat earlier
    Muppet: Right
    Dave: and we were told no that wasn't the problem
    Dave: then you decided to go and switch all our websites without solving this first???
    Muppet: But first I needed to investigate the exact issue
    Dave: as you can tell I'm not at all impresseed
    Dave: what??
    Muppet: Yes, Our admin was working on this issue before
    Dave: we tell you the issue, you disagree, mess up all our websites, and then hours later finally agree with us?
    Dave: honestly, that's very poor
    Muppet: yes, one of our senior admin MUPPET2 was working on your issue, but his machine was crashed
    Muppet: hence he properly explained me all your issues
    Muppet: Then I have investigated and fix the network error
    Dave: that's one of the worst excuses I've heard - his machine crashed... which caused you (and / or him) to not be able to read the ticket?
    Muppet: Dave we have properly fixed your issue, however our admin has given complete updates to me
    Muppet: Otherwise issue can be complicated and confusing
    Muppet: anyway
    Dave: sure, I understand it is fixed. however, if you'd taken any notice of the actual ticket you first wouldn't have caused our websites to go offline, and second fixed it right away


    As for fixing the issue, after all this we were back to where we started, still on old IPs that would be turned off / not work very shortly, hours of downtime on the vast majority of sites, all because Muppet went in and messed with things he was not asked to touch, and obviously should not have gone anywhere near. On top of all this, Plesk now comes up with errors when trying to change to the new IP, which resulted in the IP address not being changed properly any more through that, and none of our FTP logins working. So Plesk now useless for this domain and have to do everything through IIS Manager. I would have gone back to their support to fix it and the FTP problems, but am way too scared I'll end up with Muppet3 bringing everything down yet again. I've now (again, I've had to do this before last time they screwed things up) changed our admin password so they don't know what it is and can't cause further problems.

    So, in summary, here are my suggestions to EUKHost from this latest episode:
    1. Don't go forcing clients to change IP addresses on dedicated servers - particularly when your staff obviously don't know what is required to do so
    2. Don't allow Muppets to mess with client servers when they have no idea what they're doing
    3. Get Muppets to actually read tickets before jumping in and fixing** problems
    4. Try teaching your Muppets to listen to clients when they suggest what a problem might be when they caused everything to break

    **where 'fixing' in this case = doing something they weren't asked to do causing all domains to stop working for a long period of time

    Disclaimer: If you feel the term 'Muppet' is abusive, that is not the intention (although I do feel I could quite easily have used some much stronger words here). I use this phrase quite commonly for people that mess things up, so in this case it is completely warranted. I also thought you might prefer this than the staff member's actual name.

  • #2
    Hello David,

    I have personally taken updates from fellow team members who were working on IP switchover. A follow up of the same have been updated to you in ticket FAK-318-66640.

    Regards,
    euksud
    Technical Support Department.

    Comment


    • #3
      Which explains why you thought it necessary to switch IPs, so thanks for that (although it would have been much easier/better to tell us this when I was asking why/if this was necessary in the first place).

      However it doesn't go any way to explaining why your support thought it at all appropriate to go in and mess with our server/domains, or why it took so long for them to revert/fix afterwards when I'd suggested a number of times what the problem AND the solution might be.

      Comment


      • #4
        Originally posted by dave_s View Post
        Which explains why you thought it necessary to switch IPs, so thanks for that (although it would have been much easier/better to tell us this when I was asking why/if this was necessary in the first place).

        However it doesn't go any way to explaining why your support thought it at all appropriate to go in and mess with our server/domains, or why it took so long for them to revert/fix afterwards when I'd suggested a number of times what the problem AND the solution might be.
        Hi David, I've updated your ticket. Please let us know if you need any assistance with the same..
        Rock _a.k.a._ Jack Daniel

        Follow eUKhost on Twitter || Join eUKhost Community on Facebook

        Comment

        ban-img
        Working...
        X