And here I was thinking it was some kind of miracle.

@AndrasDeak I mean, if that was the case, "bit rot" wouldn't be a cause for concern...

bit rot can essentially happen even on hard drive that aren't used for an extended period of time (that is, never being powered), like say, 10 years...

I'm guessing it might just happen because of multiple factors, which might be what you meant when saying "magic"?

when I get bit rot I just cut off the rotten part and eat the rest

Most hard drives can store data for longer than 10 years.

Flash and EEPROMs are more likely to lose data after a decade or so.

(They leak information far faster than spinning rust)

@forest I mean yes, under totally normal conditions, but anything can happen...especially things you wouldn't know (eg: naked to the eye) unless you had the right equipment to verify this

Yeah. The biggest danger is just the drive mechanically breaking down.

I wouldn't say it's the only biggest, but yeah, definitely one of them

What would be a bigger danger?

As far as I understand it, hard drives' main failure mode is mechanical.

Whereas solid state drives' main failure mode is flash page failure.

I guess [other particle name here] interference would happen too, although it would take a longer exposure and stronger signal to do any lasting damage

Wouldn't those have to be fairly close? Modern HDD encoding is full of ECC.

@forest I mean, I never bought one nor made one so i wouldn't know, but i know some on youtube that made really powerful ones (eg: just point it in the direction of a car roughly 10-100 meter and it work)

Neat. But I kind of doubt most HDD owners will lose their data because of that. :P

A head crash or failing actuator is much more likely to be the culprit.

The second amendment guarantees my right to wipe my neighbor's hard drives with my emp gun

@forest yeah, indeed. That's why i said "you never know" :D (unless the hdd make a weird sound or doesn't actually get powered on properly)

@forest True

And for external drives, the temptation to pick it up and move it around to experience the fun gyroscopic effects of the disk while it's spinning!

@forest that remind me, some peoples who install Androidx86 on laptop noticed it actually use the hdd controller to know when it is in a particular position, and rotate the screen following that

wow

How does the ATA protocol even let the OS know that?

I'm unsure, but I know it's an actual thing (knew some Android ROM maintainer and they told me this without giving too much detail)

Only way to know would be knowing stuff in hardrive hardware or know what Androidx86 actually use to know that

You'd have to install it directly though (without virtualization) since I'm unsure if qemu/[whatever vm program] support whatever it use to do this

If i knew java i would probably try, but meh

@NordineLotfi I wouldn't worry about a flipped bit or two. We're talking about trashed backups. More importantly: we're talking about "SSD or HDD for backups" rather than "how do I build the Arctic Code Vault?". SSDs will lost information over time. HDDs might.

I'm also somewhat skeptical regarding EMP guns vs hard drives. But that's another matter.

@forest Do you have a reference for this? I'm not saying you are wrong, but it's a remarkable statement.

Then again, I've discovered that many remarkable things are true. In this best of all possible worlds.

@AndrasDeak Magnetic storage as in traditional spinning hard drives?

I don't follow the discussion about EMP guns. Are they supposed to destroy magnetic storage, like HDDs?

@FaheemMitha yes

Not necessarily, because future magnetic storage devices will also be less volatile. But for the time being we mostly have HDDs.

@FaheemMitha yes. HDDs have several drawbacks compared to SSDs. Main ones are catastrophic mechanical failure with no warning, and vulnerability to stray magnetic fields. The idea is that a strong enough electromagnetic pules will either fry the circuitry with induced currents, or the magnetic field of the pulse will directly flip the bits in the HDD. Neither are terribly likely situations in my opinion unless Mossad is out to get you, but then again I haven't invetigated current EMP technology.

But if it's in a youtube video it has to work.

of course the "fry the circuitry" part would also kill SSDs and motherboards and whatnot

my uneducated guess is that you'd need way more power than reasonable to make actual damage beyond point-blank range

@AndrasDeak Thanks for the information and thoughts.

Do you have opinions about SSDs vs HDDS, and external hard drives vs internal hard drive + enclosure?

I have an opinion about everything but nothing of substance

I have both by-design external drives, and former laptop HDDs made external with enclosures

And neither are something that I trust my backups with. I've been using a dumb RAID1 pair of HDDs in a standalone PC.

The plural is a lie; I have the one by-design external drive, which had to be replaced once after a year or so of use.

I'd love to have SSDs in my laptop but I can't justify the cost.

In theory SSDs would be more reliable than traditional HDDs, because there are no moving parts.

The drives inside my desktop/workstation are SSDs as well. It's been a few years since I installed them.

I don't know how long this computer is going to last. It was built in 2013. It's now been 8 years.

What you're saying is not "more reliable", it's "more resilient to mechanical failure". There are still other failure modes. But they are probably less catastrophic. And the added bonus is that SSDs are shock-resistant too.

This is also possibly the longest I've used a computer for.

But I really need to get a new computer. But we've been mostly locked down for over a year now. And widespread chaos. Sigh.

@AndrasDeak Mechanical failure seems like the most likely failure possibility.

I guess "more reliable" might also depend on how similar SSDs are to SD cards. SD cards are especially prone to random failure.

@FaheemMitha for an HDD, yes :P

Though I don't know how long lived SSDs are. Especially externally located ones.

It's like saying that submarines are safer than airplanes because they can't crash into mountains.

@AndrasDeak SSDs are more reliable to SDs, I think.

@FaheemMitha SSDs for instance have a set amount of writes for each bit. If you do too much IO you wear them out too soon. That's not a thing for HDDs.

It's hard enough to get anything done in India, without worldwide public health crises.

@AndrasDeak I'm aware. Might be an issue for backups, actually. Though the number of writes is enormous in practice. Does anyone have data about this?

The second answer from the top actually seems quite helpful.

Though of course I don't know if the writer knows what he or she is talking about.

But that's a general problem, of course.

I wouldn't expect backups to be huge IO load. Something like logging from an OS much more so.

@AndrasDeak Not incremental backups, anyway. And these days those are the only sensible kind of backups.

Deduplicating ones, to be precise.

> Secondly, hard drives usually give plenty of warning before they die and the data is usually recoverable when it happens. But when SSDs fail, they go from zero to dead quite quickly. One day they work, and the next day they don't -- sometimes for no reason at all. They also have a tendency to eat your data like some kind of bit-monster that feeds on zeroes and ones when they die.

that's actually mostly the opposite of what I expected

The top answer seems very misleading. It looks like it's about drives that are under constant use. What we're talking about is discharge of data in an unpowered drive.

or perhaps I'm the only one talking about that

@AndrasDeak That might be true as stated. But also SSDs don't normally fail until they have seen a lot of use.

There is an article on the net somewhere of someone who tried to make SSDs fail by writing data to them. Huge amounts of data.

He had to really to get them to fail. Though when they failed they didn't give much notice.

techreport.com/review/27909/…

This one, I think. ^^

As he records, no drive showed any problems before 200 TB. A level I'm very unlikely to reach.

I bet both data written and time elapsed matter

if you spread that load out over years you might need less than 200 TB

but this is again just uneducated musing

Also, note that in most cases the drives died quite abruptly.

@AndrasDeak My current external HDD, which I'm using as we speak, was purchased in 2014, on Amazon India. The previous one was also purchased on Amazon, I think. But this time on Amazon US.

I suppose the first question I should ask is whether I should replace it. But it's showing 2 sector errors now. Before it was showing 1. Opinions, uneducated or otherwise, are welcome.

If there's a single error you should replace it

and if there's a single reallocated sector in the SMART output you should replace it

@AndrasDeak OK. How uneducated is that opinion? :-)

unless this is part of a set of redundant disks for backup

I once had a drive that was reporting errors for over 10 years and I continued to use it without issue

It never actually failed in fact I just got rid of it when I got rid of that NAS

@FaheemMitha it stems from my extreme risk evasion. If all my eggs were in this one basket I wouldn't gamble. First sign of trouble, off you go.

There's few things worse than the cognitive dissonance when the drive fails, you lose your precious data, and you know there were warning signs.

@AndrasDeak Hmm. Fair enough.

@jesse_b How many errors did it report?

Are these errors coming during IO? Have you checked the SMART report?

@AndrasDeak No, I haven't checked the SMART report. How do I do that?

@FaheemMitha I forget what was actually wrong with it. It had two different SMART metrics that were over their acceptable thresholds but I think they were more performance issues than reliability

@jesse_b Those aren't errors as such.

Just indications of use, I think.

@FaheemMitha depends on your linux, maybe. I have smartmontools installed and I can do sudo smartctl -a /dev/sda

@AndrasDeak OK, I tried that. The SMART output isn't logging any errors. Regardless, I get daily emails about sectors. Maybe in my log. I'll look.

It might be this one:

> May 5 02:56:28 orwell smartd[1033]: Device: /dev/sdg [SAT], 2 Currently unreadable (pending) sectors

But if so, maybe all this fuss is for nothing, because the external HDD in question is actually sdh.

I'll take a closer look the next time I get an email.

I don't think I'm using sdg any more. Those drives are from 2007.

From lsblk, sdg isn't mounted.

@FaheemMitha I don't think it's supposed to log errors

it has a PASSED/FAILED(?) verdict, and a huge table detailing the health of your drive

@AndrasDeak SMART isn't?

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   050    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   100   001    Pre-fail  Always       -       1300
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       0

stuff like that ^

0 reallocated sectors and 0 read error rate are good

I'm actually confused what sdg is. I have a through f, which I think includes the 2007 drives. And h is the external HDD. Can someone remind me how to query that?

but you can see all the other (and vendor-specific) attributes, their values and limits

needless to say, you should do smartctl -a /dev/sdh :P

@AndrasDeak Let me just paste the output.

OK but I only know the straightforward stuff

gist.github.com/71a0ce6a7cb5a05cb7078598d72cc9d5

@AndrasDeak Well, I'm regularly convinced I know nothing.

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       25

 197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -      2

OK. And in human-speak?

Not perfect.

Ideally you have 0 error rate and 0 pending (and reallocated) sectors. How bad these nonzeros are, I can't tell.

@AndrasDeak Yes, but what do those numbers actually mean? They're ideally supposed to be 0?

Normally a drive functions without errors, hence 0 error rate.

Someone went to a lot of trouble to analyze a similar situation, at superuser.com/q/677438

And I believe reallocated sectors are ones that went bad, and their content ha to be copied over elsewhere. The firmware considered these sectors dead. If the drive is not full they don't do harm in themselves, but their appearance often means that other sectors are going to start failing soon.

I don't know what pending sectors are, but my guess would be that they are considered for reallocation. But again, uneducated guess.

but the actual experts in the room will eventually see this, and will correct me :P

> A sector is marked pending when a read fails. The pending sector will be marked reallocated if a subsequent write fails. If the write succeeds, it is removed from current pending sectors and assumed to be ok. (The exact behavior could differ slightly and I'll go into that later, but this is a close enough approximation for now.)

From an answer in that thread I just posted.

neat

@AndrasDeak Don't count on it. Few people read everything. Actually, I suspect that count is likely 0.

"If you want something answered on the internet, post a wrong answer"

@AndrasDeak So in summary, you think replacement is reasonable?

@AndrasDeak People have to read it first. People don't read chat backlog for the most part.

@FaheemMitha please read from chat.stackexchange.com/transcript/message/57884086#57884086

@AndrasDeak So that would be a yes, then.

With all the "I don't know what I'm doing" caveats, but yes

OK. So maybe the emails I get are actually for the external HDD after all.

Though the message about sdg is confusing.

If sdg exists, shouldn't fdisk know about it?

I just ran an interactive docker container without the -i flag and now I feel like the saw guy is trying to play a game with me

@FaheemMitha I don't have anything on hand, but it's a common tactic and pretty well known in the tech industry. For example, most Intel i5 processors are actually i7s that failed QA (certain components failed, so instead of scrapping it, they disable a bunch of components including the failed one, and sell it at a lower price). That's why i7s are more expensive: they contain components that are less likely to pass through QA.

That's probably not just why they are more expensive :P

Well, that's one reason why. Of course, capitalism is the main reason.

The same thing is true with RAM. Most RAM that comes in varying models with varying speeds are actually all the same model, but the slower ones were merely unstable on the test bed at higher rates, so their maximum speed was reduced in its internal configuration and it was sold as a slower model.

it's more of a reason why the company makes more of a profit on these

Same with some 2 core processors that are actually 4 core but with one of the cores running poorly and disabled by the manufacturer. AMD once infamously released a 4 core with only 3 active cores.

For some reason I'm reminded of kitkat factories, where rejected kitkats get ground up, forming the filling of the other kitkats.

3-vote closure being tested on 13 sites, but U&L is not on the list

I guess you guys have low crap rate and high tolerance for the little crap that's left

I'm hoping they'll add more to the list soon.

We want that on Cryptography as well.

If I understand correctly there are two dozen sites planned, and one dozen for the 45-day test run

it's worked out great on SO

@NordineLotfi i cant remember what exactly it is i wrote that you are referring to, which probably means i had been drinking at the tme

If it was in regard to anything i said regarding the banjo playing savant in deliverance, i can apologize but that doesnt mean my abilities to be socially appropriate are ever going to improve

the nature of my circumstances involve one tab with uber sensitive PC expectations in what i can and cant say to another tab that is a literal death threat that someone is sending me, so yes my ability to be appropriate for the situation is already flawed, and things are not made easy for me for the reasons ive said

and if it was bad grammar, well sure i dont make sense half the time on a good day

What happened?

@forest it's best just to let it happen

???

@forest don't engage

hm

reads scroll log

poor Jeff

O.o

Unfortunately half-conversations involving, ehm, suboptimally pleasant users are still suboptimally pleasant.

Jeff has always been optimally pleasant to me

I haven't had any encounters with him.

@forest you should, Jeff is great ;)

Unless you're a weed sympathizer

hah

Uh oh.

I'm in the "legalize everything 420 blaze it" crowd.

Just don't throw the seeds in his yard and you'll be fine

Won't get seeds if you cut off the buds. :P

Jeff would wack your weeds into oblivion

Wait, does he not like weed, or weeds?

Bees

shh, he's back

Nobody wants to ignore Jeff, that's why he's ok as the example