I haven't given up yet :) ^^^^^^^^

@JackDouglas @Phil if FRA is filling up and there's an issue moving the archive logs off - I could just 'delete archivelog all' to free up space in FRA right? Obviously, we assume the recovery failure at that point though

@rfusca you can delete the logs, but what do you mean exactly "we assume the recovery failure at that point though"?

can you reduce the recovery window?

@JackDouglas i'm saying that as it is right now, we back up logs every hour. Obviously if we delete the logs, we can't use them in the event of a point in time recovery

@rfusca OK

how far back do your logs go?

@JackDouglas well the ones in FRA currently are about back to midnight

ahhh

we take a backup every night, then back the archive logs up and FRA fills up. Something happen with the backup device and we can't backup them up right now, so FRA is filling up with archive logs

you do backups nightly then?

@JackDouglas ya

sounds like you are space-constrained in your FRA

lots of logging?

ya, its a very active system

there's only about 16 hours on average worth of logging space available

what % of the FRA does a full backup occupy?

i didn't think a full backup got written to the FRA...I may be confused there. we write our backups off to an Exagrid

it's not really an FRA if it doesn't contain a backup ;)

to my mind, the backups and archivelogs belong together: unless your normal recovery mode is not expected to be PITR

any reason the archivelog backups can't go to the Exagrid?

hmmm

@JackDouglas the exagrid is the issue

they normally go there

but its having...issues

oh dear

so tonight's backup won't work either?

that's a fun Easter you have lined up :)

well, it should be resolved by tonight, but it won't be resolved before FRA fills up at the current rate

are your archivelogs going anywhere else?

before you delete them I mean

what's the cost tomorrow if you have to revert to last nights backup and can't PITR subsequently?

I'd be desperately looking for another way to archive them off the FRA before deleting

@JackDouglas baaaaaaad

super duper mega insanely rediciously bad

you need another way out then

how big are today's logs so far?

let me look

@rfusca even if the risk is low (and I wonder if it is given what you've said about the Exagrid), (low risk)*(super duper mega insanely rediciously bad consequence)=(dont do it)

@JackDouglas indeed

you may even need to weigh up the relative cost of stopping the service until the recovery issues are resolved (once the FRA is getting close to full)

ya

@JackDouglas so if you ever use exagrid - don't let it get low on space. Not like 0% low or anything like that. 20% was it for us. According to exagrid engineers, when it gets 'low' - it changes to only allow 1 Mb/s incoming and tries to dedup 'live'.

@rfusca ouch

ya

ouch indeed

actually good news: you have your Easter back if you can delete some stuff?

if it was faulty that'd be bad news

one of our guys had set the exagrid to replication across a 10 Mb/s network....for about 2 TB. So it wasn't deleting stuff because it wasn't replicated across yet or such

so the sys admin responsible is going to hook it up on a gigabit network and it should catch up faster, free up space on the exagrid, and bring the exagrid back up to normal speed

55 hours

@JackDouglas for the 10 network?

yes

ya, well it wasn't dedicated, it was probably getting more like 4

(I had told him not to do that)

now you'll be limited by the write speed on the Exagrid I guess

ya

will it be quick enough that you get there before your FRA fills up?

but in the meantime, the VP over our division choose to assume the risk of deleting the archive logs. Despite me telling the app team to turn off everything that externally writes, they didn't. So a flurry of activity shot it up in the 95% range. We had to delete or risk worse

his decision I guess, I hope you got it in writing ;)

best not to mention who you work for in here for a while!

lol indeed

none of this should come back on me though luckily

the sysadmin was scrambling on the phone with Exagrid engineers "So, this would have happened anyway...right? right!?" "No sir...this happened because you turned on the replication and it couldn't catch up."

@rfusca hmm, my maths sucks. Got bits and bytes mixed up, you meant 4Mbit/s didn't you? That'd be over 1000 hours calctool.org/CALC/prof/computing/transfer_time

or 44 days :)

@JackDouglas yes

ya i know i tried to explain the math to him but he kept insisting "It won't be that bad, lets just try it."