Announcement

Collapse
No announcement yet.

R!Soft CDP Services

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • ejsolutions
    replied
    Re: R1Soft CDP Services

    R1soft CDP Backup Managers remain pitifully poor in performance, along with buggy software.
    Examples:
    r1softbackup2.specialservers.com - Time In Queue 2h 2m 40s
    r1softbackup3.specialservers.com -
    Replication Run Time 46m 4s (File Excludes 12s)
    Total Deltas 1.6 GB -> 1.5 GB (ratio 1:1)
    Average Throughput 604 KB/s

    (Thanks to Max T. for taking some time to look at this on Live Chat. Further investigation is to be carried out - allegedly.)

    Perhaps these should be renamed from specialservers.com to specialneedsservers.com
    Last edited by ejsolutions; 25-02-2014, 20:03.

    Leave a comment:


  • ejsolutions
    replied
    Re: R1Soft CDP Services

    Originally posted by Rock View Post
    Are you still getting these delays even now?
    Note the date of this post. Still an ongoing issue with the performance of the backup server(s).

    Leave a comment:


  • ejsolutions
    replied
    Re: R1Soft CDP Services

    Originally posted by Rock View Post
    Are you still getting these delays even now?
    This morning's Daily failed on R1Soft #2:
    Scheduled November 20, 2013 5:58:04 AM GMT
    Start Time November 20, 2013 7:37:21 AM GMT
    End Time November 20, 2013 7:37:39 AM GMT
    Run Time 17s
    Time In Queue 1h 39m 16s
    17 seconds runtime due to an hourly task running at the time (hardly surprising).

    EJ
    ejSolutions CDP Backup Monitoring Service

    Leave a comment:


  • Rock
    replied
    Re: R1Soft CDP Services

    Originally posted by ejsolutions View Post
    You appear to have missed the salient point: Time In Queue 48m 49s
    Are you still getting these delays even now?

    Leave a comment:


  • ejsolutions
    replied
    Re: R1Soft CDP Services

    Originally posted by Rock View Post
    We're glad to know that. Please report to us if you notice any anomaly & we too are monitoring the backup servers such that they function properly.
    You appear to have missed the salient point: Time In Queue 48m 49s

    Leave a comment:


  • Rock
    replied
    Re: R1Soft CDP Services

    We're glad to know that. Please report to us if you notice any anomaly & we too are monitoring the backup servers such that they function properly.

    Leave a comment:


  • ejsolutions
    replied
    Re: R1Soft CDP Services

    FYI, by pure chance, the Daily backup on r1softbackup2 ran today - it didn't overlap an Hourly Task on r1softbackup3.

    Scheduled November 17, 2013 5:58:07 AM GMT
    Start Time November 17, 2013 6:46:56 AM GMT
    End Time November 17, 2013 7:28:18 AM GMT
    Run Time 41m 21s
    Time In Queue 48m 49s
    Note length of time in queue, before the job ran.


    EJ
    ejSolutions CDP Backup Monitoring Service

    Leave a comment:


  • Rock
    replied
    Re: R1Soft CDP Services

    Originally posted by ejsolutions View Post
    Thanks. Actually, this morning's Report on hourly backups was 'clean' for the first time in ages. Daily failed but I haven't checked in detail, yet.
    You're welcome. We're still working on it & will update you about on any progress done, shortly.

    Leave a comment:


  • ejsolutions
    replied
    Re: R1Soft CDP Services

    Originally posted by Rock View Post
    ... I've asked one of our admins to check this for you & move your existing backup services to another server if that can help fix the ongoing issues..
    Thanks. Actually, this morning's Report on hourly backups was 'clean' for the first time in ages. Daily failed but I haven't checked in detail, yet.

    Leave a comment:


  • Rock
    replied
    Re: R1Soft CDP Services

    Hi EJ,

    I've asked one of our admins to check this for you & move your existing backup services to another server if that can help fix the ongoing issues.

    Your patience is appreciated in the meantime.

    Leave a comment:


  • ejsolutions
    replied
    Re: R1Soft CDP Services

    I ask very little of you (eUKHost) in terms of operational support for R1Soft. My concerns lie solely with the poorly performing/management of the CDP Manager servers. On numerous occasions it is only through my reporting that any action appears to happen. Having spent about an hour on Live Chat yesterday and not achieved one iota of improvement, I am not really motivated to report server issues to you.

    Anyone that may be considering your Remote Backup solution should be extremely cautious and definitely have their own alternative arrangements in place.


    Originally posted by Kieran View Post
    Now, your backup policies are working fine.
    R1Soft #3
    Replication Run Time 30m 13s (File Excludes 3m 59s)
    Total Deltas 176.3 MB -> 43.9 MB (ratio 4:1)
    Average Throughput 808 KB/s
    Though this does appear to be an anomaly, as others are taking about 10 minutes (approx. double of what they had been.)

    Disk Safe Verification
    Start Time November 13, 2013 12:04:03 AM GMT
    End Time November 13, 2013 1:22:19 AM GMT
    Run Time 1h 18m 16s
    Once again appears to be a one-off.

    At least I am now getting backups, after a number of DAYS without any (excepting my own extended 3-month offsite website backups).

    R1Soft #2 - a couple of example of which there are plenty!
    Scheduled November 13, 2013 5:58:01 AM GMT
    Start Time November 13, 2013 8:34:43 AM GMT
    End Time November 13, 2013 9:12:12 AM GMT
    Run Time 37m 29s
    Time In Queue 2h 36m 42s
    Scheduled November 12, 2013 10:52:43 PM GMT
    Start Time November 13, 2013 1:30:28 AM GMT
    End Time November 13, 2013 1:30:48 AM GMT
    Run Time 20s
    Time In Queue 2h 37m 44s
    The above Task couldn't complete due to a conflict with another Task. Hardly surprising when it had been in the Queue for so long and could've started at any moment.


    EJ
    ejSolutions CDP Backup Monitoring Service

    Leave a comment:


  • Kieran
    replied
    Re: R1Soft CDP Services

    Dear EJ !

    You can be sure that the backup provisioning and monitoring task is always taken seriously and we take appropriate
    action as soon as possible for such unexpected incidents. Although, we have 24/7 monitoring team which monitor and audit
    the backup servers and each user's policy, there is a limit in scope of support that we can provide to the end user regarding some case(s).

    So if you notice any issue with your backup policy or the CDP service then please report it to live chat support or
    open a support ticket, so that we can have a look as soon as possible.

    This time there was an issue with the hard drive which resulted in corruption of disk volume and hence the backup failed.
    We have replaced the disk and got everything working as it is. Now, your backup policies are working fine.

    Your patience and understanding is highly appreciated.

    Leave a comment:


  • ejsolutions
    replied
    Re: R1Soft CDP Services

    No hourly backups have been running since 10th Nov. 13:34, from r1softbackup3
    AFTER I had raised a Ticket for R1Soft Disc Safe Corruption [MON #RIB-168-39013], a status page miraculously appeared stating RAID degraded on backup server r1softbackup3! (11/11/2013 At 9:16 AM)
    Degraded? My Disc Safe has gone/has wrong permissions/is not attached - that's more than just performance degradation!
    RAID == data does not go offline.
    Once again it was down to ejSolutions CDP Backup Monitoring Service to report an issue.

    Look at the time of this posting, I have still no word on if I have ANY hourly backups (plus a daily one). Looking at the R1Soft console, I can see that the situation hasn't changed since yesterday:
    Failed to find disk cache entry for '/backup5/EUK-xxxxx/a90653db-dccd-409f-9axx-357d16axxxxxx'
    Cannot open disk safe at /backup5/EUK-xxxxx/a90653db-dccd-409f-9axx-357d16xxxxxx because the path does not exist
    Thank goodness for having a split Backup Manager! At least I have SOME backups, that I pay for.
    [Note: I also take/store my own offsite backups of major websites - just as well.]
    Who said business continuity? (Rhetorical.)


    Please take backup provision much more seriously; backups appear to be insignificant until you need them the most.


    EJ

    Leave a comment:


  • ejsolutions
    replied
    Re: R1Soft CDP Services

    To steal a word..
    Furthermore, I also run a CDP Server across The Pond, whose source VM is a rubbishy wee Xen Cloud VPS, with a minimal specification (has an effect on Agent performance). Typical figures are shown below - note the amount of time taken, the quantity of data and the throughput.

    Replication Run Time 26m 19s (File Excludes 4m 49s)
    Total Deltas 3.6 GB -> 3.5 GB (ratio 1:1)
    Average Throughput 2.9 MB/s
    The source data is actually heavily compressed already, hence the 1:1 ratio, which slows down the apparent throughput.

    Leave a comment:


  • ejsolutions
    replied
    Re: R1Soft CDP Services

    Originally posted by eUK-Victor View Post
    Dear EJ,

    I've checked your r1soft backup issue and found that you've only one server and it is being backed up on two of our r1soft backup servers "r1softbackup2.specialservers.com" and "r1softbackup3.specialservers.com".

    Further more, r1softbackup2 is configured for daily backup retention whereas r1softbackup3 is for hourly backup.

    The policy execution time for daily backup is set to execute daily at 5:54 AM. On the same time, at 6 AM the hourly policy will start to execute. It means that the backup agent will get only a few minutes [6 minutes max] to complete its execution.

    From the task history, it is clear that hourly policy takes 10 minutes approximately to complete its backup process and merging recovery points process. No sooner the hourly policy is on its way to complete its task, than the daily policy will try to start its execution. As there is a very few minutes of gap between these policy executions, they get overlapped and backup replication fails.

    To overcome this problem, I'd like to suggest that you should try changing the execution time of daily policy from 5:54 AM to may be at 5:30 or 6:30 AM so that the hourly policy will get sufficient time to complete its execution and post-backup merging task.
    3 Points here.

    1. Yes, I have a "belt & braces" approach to the backups by using two individual backup servers. This is primarily due to the overall poorly monitored CDP service. On the one occasion that I actually needed to do an urgent restore, the backup server was 'unavailable' (I don't recall the details).

    2. The hourly backup has always been set at "34 minutes past the hour" and under normal/usual circumstances, ten minutes has been ample - the snapshot can be quicker than 2 minutes, though averages around 5. This can be readily seen if one cares to look back over the historical runs i.e. not just the past 20 items of history. there is 69 pages of the stuff!
    Now, previously, the daily backup had been set around 20 minutes after the scheduled start of the hourly tasks. Thus giving some time for the task queue to be processed plus a good deal of leeway. Once again, looking further back in the historical (hysterical?) tasks, one can see that the backup window has grown increasingly. This is primarily down to the the poor performance of r1softbackup2.specialservers.com, as can be seen by some of the stats that I have posted previously. (The server has just taken 20 seconds purely to save a change to the scheduled time of the daily task.)

    (Sorry for this but it appears that this is the only way to get my point across.)
    [BLUNT]
    With a little common sense/lateral thinking, it should be possible to grasp that a daily backup that takes longer than an hour, will ALWAYS overlap/clash with an hourly task.
    [/BLUNT]

    Another set of figures.. (my emboldening, to highlight)
    r1softbackup2.specialservers.com/TaskHistory/
    Start Time September 30, 2013 5:54:02 AM BST
    End Time September 30, 2013 6:29:49 AM BST
    Replication Statistics
    Replication Run Time 35m 29s (File Excludes 15s)
    Total Deltas 1.9 GB -> 1.5 GB (ratio 1.2:1)
    Average Throughput 1 MB/s
    The above, even with an appalling average throughput (network/backup server Load & disc), shows that it can complete within 40 minutes. Remember, 10 minutes is required to an hourly backup to complete.

    I have now set the daily backup to begin at 5:58, though clearly, if this task takes over 36 minutes to complete, then it will clash with the hourly task. Also, this doesn't take into account any time lost whilst awaiting a task start in the queue.

    3. The merging of previous backup snapshots does not affect taking of fresh snapshots, where there are two different backup servers involved. This task is Backup Manager based and is separate to the Agent on the VM. For example, the daily backup server can take nearly all day to do a merge, with absolutely no performance penalty on the hourly backups that are run from the other Backup Manager.

    Ultimately, it is the poorly performing Backup Managers that are the root cause to the backup failures. I am surprised that other clients are not making any comments (publicly, or otherwise). As you may be aware CDP has a provision for 15 minute snapshots; I hate to think how many failures/warnings would be triggered by that frequency. Perhaps no-one else really monitors their backups, in detail. It's pointless finding an issue after the fact, when you might need it the most.

    [Let me state here, that I used to provide consultancy to some of the UK's larger companies, on backup provision and Disaster Recovery. I haven't even got around to exploring the R1soft DR capabilities.]

    Tutorial over.



    [I've likely missed something, after all that rambling!]
    Last edited by ejsolutions; 03-11-2013, 01:24. Reason: clarification and typos

    Leave a comment:

Working...
X