Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

Geek Culture / Any ideas for high capacity long term digital backup storage?

Author
Message
PAGAN_old
13
Years of Service
User Offline
Joined: 28th Jan 2006
Location: Capital of the Evil Empire
Posted: 20th Aug 2016 15:06 Edited at: 20th Aug 2016 15:47
It has been 6 years since i moved to Russia and bought my first harddrive here. Over these last several years my personal storage has grown up to 10 TB and it keeps on growing.
Obviously, their time has come and in the last year i lost 3 harddrives with valuable data. This is why i began to think about a more reliable long-term storage for my data.
A few years ago, i would have simply bought more harddrives but since the HDD factory in Malaysia burned down in 2013, harddrives have become expensive.
The second problem with harddrives would be the fact that since i need a long term backup storage meaning the hard drives will probably not be used for years before i need them, there is no guarantee that after a few years it will demagnetize resulting in corrupt data.
I have been looking for some solutions for a while and i have some ideas as well as pros and cons and will list them here.

Online cloud storage. Far from my preferred method, but i am willing to settle for this as my third backup but not my primary one. Here is why:
First and most obvious reason is i am the kind of guy who is always strapped for money to the point where i almost never pay my bills on time (yes i already had my electricity shut down a few times and i am constantly in debt to the utility companies) and this might result in my data being deleted from the cloud because i either forgot, or couldnt afford to pay for a few months.
Second reason is i simply dont trust companies like that because anything could happen from that company getting bankrupt and my data getting lost/sold to god knows who, censored by the Russian government because of a current state of digital paranoia and informational war with the west (tons of this is going on here from banning pokemon go to establishing a massive digital surveillance infrustructure over the russian internet in the name of anti-terrorism. Proxies these dont work more often than they do and if they do, they are slow and glitchy as hell and it wont be long before things like VPN will become illegal.
To make my point, 5 years ago i wrote a short story about a future digital dystopyan/orwellian-like state and now i get to see it being established

Moving on: SSDs The finally good trend ive been noticing in the last few years is that SSDs are finally becoming cheaper BUT WAIT! Russian government wants to establish import fees of up to 60% on all electronics in order to stimulate its own electronics industry and to persuade investors to open factories withing the countriy. Little to be said that living in this country is like playing the Russian roulette and i see no point in hoping for the best.
And besides, SSDs of a 10Tb size are still pretty expensive.
And one last thing: (i am not sure how true this is or not so fill me in here if anybody knows this better than me) SSDs to some degree have the same problem as regular harddrives in terms of magnetic fields within the flash memory (apparentley flash memory is also based on micro-magnetic fields inside the chip) Will get demagnetized and corrupt my data in few years. And to be fair, ive dealt with corrupt data on flash drives more often than on harddrives. So i too am skeptical of this solution.

Optical disks (DVDs Blue rays etc) a good hard copy solution which isnt magnet based but first problem is that i will need tons of these and unfourtunatley, this kind of media physically degrades over time. (i wish optical media came in protective cartriges like the PSP disks or some of the first cd roms back in the late 80s)

Streamers/tape drives. By far the most promising method altho from what i have read and from what people told me, there are a lot of varying opinions on this ranging from the most unsecure long term storage to the most solid method.
I did some research a while back on streamers and i dont really remember the names of different types of technologies but i had my eyes on 2 types. One of which is where the actual cassette drive tends to be extremley expensive- like from $1000 up to $5000. BUT the actual casettes to those types of drives are extremley cheap, around $20-$50 per TB (these kinds of casettes come at 500gb max) and even cheaper if you buy them used, (i am down with that because there are tons of them being sold, I saw an ad where a dude is selling 50TB for $200)...
The other type of streamer i researched is pretty much the opposite. The device itself is pretty affortable (up to $500) but the cassette cartridges for them are pretty expensive.
Then there is an issue of how good the actual streamer is, some record data with tons of errors, others claim its totally solid. And there is also a veriety of opinions concerning the long term storage as well. Ive been told that i dont have the proper conditions for storing such tape drives without the data being corrupted over the years.

I actually own an old seagate streamer but its too old and the cartriges to such devices are no longer sold and are low capacity anyway (1gb-5gb/casette)

Ultimately my core problem here is a financial one, which is why in the recent few weeks i ditched the field of low-paying technology and trying to switch over to real-estate/bankruptcy financial area where i have real chances of earning $5000-$15000 per sale (besides, i am tired of working with other-people's technology.

I am still interested in researching this topic and by all means feel free to enlighten me with things i dont know or might be wrong about, Maybe i am not the only one concerned about the ramifications of harddrive death making this an interesting topic for other people on this forum. The goal so far is high capacity and cost effective solution over a long term period.

Your signature has been erased by a mod
Ortu
DBPro Master
11
Years of Service
User Offline
Joined: 21st Nov 2007
Location: Austin, TX
Posted: 20th Aug 2016 18:57
ssd or hdd in a raid array will give you redundancy and allow you to replace a failed disk before losing the data.

Raid 6 or 10 are good setups, people like 5 for the disk space, but it is risky when it comes time to recover from a degraded state.

You can get an 8 bay disk enclosure with esata or USB 3.0/3.1 connector at pretty affordable rates.

Windows 10 has built in handling for raid through storage spaces, freenas is a good open source *nix based os for this as well that can run easily off a usb


A single player RPG featuring a branching, player driven storyline of meaningful choices and multiple endings alongside challenging active combat and intelligent AI.
PAGAN_old
13
Years of Service
User Offline
Joined: 28th Jan 2006
Location: Capital of the Evil Empire
Posted: 20th Aug 2016 21:44
Yeah, raid is a good solution for active systems, Not ruling it out for any future builds.
Looking more of a low-maintenance solution where i backup the data by hand and can forget about for years. All hard drives raid or not have a lifespan of 3-7 years before they either break or demagnetize. My first tb WD green which i bought 5 years ago is already making scary sounds.

Your signature has been erased by a mod
Phaelax
DBPro Master
15
Years of Service
User Offline
Joined: 16th Apr 2003
Location: Metropia
Posted: 20th Aug 2016 22:55
10TB and you're running out of storage? Sounds like you need to do some spring cleaning and get all the old useless junk out. Your drives really should be lasting longer than a few years. I have drives from the 90s I bet I could still pull data from. I think raid is your best option right now. If anything fails you just replace the drive without losing any data.

"I like offending people, because I think people who get offended should be offended." - Linus Torvalds
PAGAN_old
13
Years of Service
User Offline
Joined: 28th Jan 2006
Location: Capital of the Evil Empire
Posted: 21st Aug 2016 01:56 Edited at: 21st Aug 2016 01:57
getting rid of useless junk will free up maybe 15% of my space. Among memes, games, music, movies and shows i dont want to get rid of a lot of space is taken up by software and rare operating system distros as well as tools that allow me to make custom ones (like windows PE and stuff) Also rare custom firmwares for gadgets and rare electronics Its just a pain to track down these things when you need them. Like when i was trying to find a Bios for a military tablet from 2003, pretty much all links i found were dead for 5 years.

As for harddrives from the 90s, the weird thing is i also have drives from the 90s that still work. Very possibly because they used to be much higher quality because those things can take a beating and still work unlike modern ones. Altho the ones that have been lying around for years are full of corrupt data already.

Maybe ill get around to setting up raid once i get enough money and stable enough income to afford replacing harddrives whenever i need to. Altho again, i have multiple computers and getting raid on every one of them will be a pain in the ass. What i am looking for is a data-dump where i can dump my stuff to from all my computers every few months and not have to worry about it dissapearing.

According to some article from 2000s that ive read. the average storage per user by 2016 will be 3-5Tb. So yeah, i went above and beyond.

As i said, i am hoarding data because i like having access to it when i need to without having to go look for it because the internet in Russia is getting worse every year.

They recent mass surveillance law they passed here involves complete overhaul of the entire network infrastructure, installation of FSB spying equipment on every service provider on the territorry of the country (Edward Snowden must be rolling in his grave ) The IT and networking industry is in an uproar over this, because they will have to pay for it. Internet tariffs will skyrocket, the speed will slow down to a crawl, providers will go bankrupt etc.

I am just trying to set up a data bunker for the next 5-10 years because something tells me that streaming Netflicks in HD 1080 1920 4K MK ULTRA will no longer be an option and my 20 tb steam library might get blocked.
Your signature has been erased by a mod
Ortu
DBPro Master
11
Years of Service
User Offline
Joined: 21st Nov 2007
Location: Austin, TX
Posted: 21st Aug 2016 06:38
You wouldn't need to set it up on each computer, assuming they are all networked, set it up on one machine and dump everything to it through file shares.

For the amount of data you are talking about, you really only have 3 basic options: cloud, local raid, and tape.

tape might actually be something to consider if you won't access it all the time. It will last, as long as it is stored well, but access is a bit of a pain, and it probably won't be much if any cheaper than just going with the raid.

LTO3 tapes can hold 800GB of compressed data each


A single player RPG featuring a branching, player driven storyline of meaningful choices and multiple endings alongside challenging active combat and intelligent AI.
Phaelax
DBPro Master
15
Years of Service
User Offline
Joined: 16th Apr 2003
Location: Metropia
Posted: 21st Aug 2016 17:09
Quote: "he average storage per user by 2016 will be 3-5Tb. So yeah, i went above and beyond."


We're hardly average users :p


I've started storing all my data on a NAS. Gives me raid and lets all my devices access it.

"I like offending people, because I think people who get offended should be offended." - Linus Torvalds
PAGAN_old
13
Years of Service
User Offline
Joined: 28th Jan 2006
Location: Capital of the Evil Empire
Posted: 21st Aug 2016 17:24
I have been looking at LTO3 a while back and just took a look at it again, The new streamers cost $3000-$5000 but used ones are $500-$800 which is very affordable for me. and the tapes are anywhere from $30 for used to $100 new (800 gb)
any thoughts on why might the used streamers have such a huge markdown price?
Your signature has been erased by a mod
Ortu
DBPro Master
11
Years of Service
User Offline
Joined: 21st Nov 2007
Location: Austin, TX
Posted: 21st Aug 2016 21:20 Edited at: 21st Aug 2016 21:23
Used used may have some wear and tear, particularly on the assembly and head, but refurbished used should be fine. In general anything used is going to be a lot cheaper because a lot of people don't want used stuff.

I'm working towards a nas myself, it is definitely my top pick of the 3 options.


A single player RPG featuring a branching, player driven storyline of meaningful choices and multiple endings alongside challenging active combat and intelligent AI.
PAGAN_old
13
Years of Service
User Offline
Joined: 28th Jan 2006
Location: Capital of the Evil Empire
Posted: 21st Aug 2016 21:49 Edited at: 21st Aug 2016 21:51
Well, i am going to have a NAS anyway (for fast access). The thing is i am taking into account my negligence and procrastination. (before i can finally get around to replacing my failed Raid drives, my primary one will start to die). And usually with the used things even wear and tear doesn't justify an 80% drop in price even with electronics. How badly could wear and tear affect the quality of the LTO 3 tape recordings? Because i head that was an issue. But the tapes are cheap enough for me to make 3-4 backups of all of my data just in case. Also yes i did encounter a few new LTO3 streamers for a suspiciosly low price of $450 http://www.ict.com.ru/catalog/strimeri/eh842a/?ymclid=71812332039039383600014 (Its in russian i know sorry) HP StorageWorks Ultrium 920E SCSI LTO3 Tape Drive (OEM) 443584-001 EH842A This is suspiciosly low price for a new streamer which i am guessing is refurbished.
Your signature has been erased by a mod

Login to post a reply

Server time is: 2019-03-23 17:41:51
Your offset time is: 2019-03-23 17:41:51