VLOG: The BIG Data Storage Problem
February 5, 2016 by warzan
For some website features, you will need a FREE account and for some others, you will need to join the Cult of Games.
Or if you have already joined the Cult of Games Log in now
What difference will having a FREE account make?
Setting up a Free account with OnTableTop unlocks a load of additional features and content (see below). You can then get involved with our Tabletop Gaming community, we are very helpful and keen to hear what you have to say. So Join Us Now!
Free Account Includes
- Creating your own project blogs.
- Rating and reviewing games using our innovative system.
- Commenting and ability to upvote.
- Posting in the forums.
- Unlocking of Achivments and collectin hobby xp
- Ability to add places like clubs and stores to our gaming database.
- Follow games, recommend games, use wishlist and mark what games you own.
- You will be able to add friends to your account.
What's the Cult of Games?
Once you have made a free account you can support the community by joing the Cult of Games. Joining the Cult allows you to use even more parts of the site and access to extra content. Check out some of the extra features below.
Cult of Games Membership Includes
- Reduced ads, for a better browsing experience (feature can be turned on or off in your profile).
- Access to The Cult of Games XLBS Sunday Show.
- Extra hobby videos about painting, terrain building etc.
- Exclusive interviews with the best game designers etc.
- Behind the scenes studio VLogs.
- Access to our live stream archives.
- Early access to our event tickets.
- Access to the CoG Greenroom.
- Access to the CoG Chamber of Commerce.
- Access the CoG Bazarr Trading Forum.
- Create and Edit Records for Games, Companies and Professionals.
wooohoo keep it up boys
All the best with that guys, wish I had the expertise to help you.
@warzan For your IDE drives, you can purchase IDE/SATA to USB adaptors which you can use to access those old IDE drives. Locally they sell for $20-$30 Canadain. Most good computer supply stores should carry them. I know USB is not an ideal solution for permanent mounting, but if you are physically archiving the data and then accessing it infrequently, it might be the best bang for buck option.
http://www.bestbuy.com/site/thermaltake-blacx-hard-drive-enclosure-docking-station-black/9062055.p?id=1219699418029&skuId=9062055
USB IDE Dock…lets you hot swap drives to transfer data. A slight improvement over enclosures, you don’t need to assemble anything around it. Also reads smaller laptop drives.
whem u crack open those external HDs open guys, expect you obsolete pile to increase, because some have none standard HDs inside them 🙁
I need an edit button. sorry for the bad sentence…
You should be able to use Adobe Bridge to search a file or volume by tag. If that’s the case you just find some poor volunteer to pick through your archives to standardize the metadata in Bridge as you duplicate your drives.
For off-site storage, is it worth looking at some kind of cloud storage?
I don’t think so for their needs, a gig is fine and free but after that, your looking at at a monthly cost which adds up when it is per gig. Also most companies operate out of the US, the cost of uploading is phenomenal.
May be I am wrong.
You are not wrong. Personal levels of cloud storage are free, with costs incurred after certain levels, but business levels of storage are beyond what BoW could reasonably afford.
I am hybridising on-prem to IaaS just now, costs are mental.
DO NOT OPEN THE HDDS. You’ll kill them instantly. * Unless you mean just taking the drive out of the consumer case. But taking platters out of the HDD will turn them into drink coasters. *
If you don’t need to retrieve data very often I would use Amazon’s AWS Glacier service. Also, if you need local storage I would start using solid state laptop drives. They’re a bit pricier but the size is smaller, there’s no moving parts, they’re not magnetic storage.
Honestly though, I would use hosted services like AWS, or Rackspace. Or even stuff like Dropbox. A few benefits to hosted solutions: you don’t have to deal with maintaining the hardware, upgrading hardware, dealing with the power costs, et. al. Look into the Dropbox Business pricing. It’s mega affordable and great for data that is currently being used. Then use Glacier for deep storage.
Experience: I work for a leading digital product design company, have worked for one of the most well respected digital product design agency, and work closely with a lot of the big well known tech companies. Not bragging/humblebragging, just providing credentials to backup opinion. 🙂
Cloud storage is probably the way to go these days not sure i would risk Dropbox they have a few security issues. I am guessing that it’s not just the image and video stuff that needs to be approriatly stored but all the business support data aswell.
Dropbox has definitely had security issues in the past, I think they’ve corrected that. Their business + enterprise offering is definitely better than the free offering. But, I’m probably biased as I’ve got friends employed there as developers.
But, there’s lots of alternatives to Dropbox, I only mention it because it’s well known and super easy to use. Google offers similar services for similar pricing but with significantly more storage (order of magnitude more). You could also just something like a Drobo in the office for working files and then AWS Glacier for archives. Or build your own completely hosted Dropbox clone on AWS, Rackspace, or some other hosting service.
Wouhouuuu, keep it up boys.
I am a professional Information manager with nearly 20 years experience if thats any help @warzan i dropped you an email.
This may seem like an amateur question but once a video has been released, why do you need to keep all of the production footage? Is there a legal mandate for you to maintain records of all production footage or could you get away with just keeping the videos themselves? It may be a tough decision but unless you really need to keep the production footage, you could significantly reduce your data footprint by just backing up copies of the videos as they are when they get released.
My company uses Amazon but there are other options.
https://aws.amazon.com/
In particular check out the archive storage options as those tend to be cheaper. I think we use glacier for long term data storage.
https://aws.amazon.com/glacier/
Why don’t you burn every video on a cd-rom?
Hi guys,
From someone who works in the industry, can I suggest Amazon Web Services (AWS)? They are a massive industry leader for cloud computing and storage. Even giants like Netflix use them.
They offer several different cloud storage methods – but for archival in particular they offer something called Amazon Glacier. It’s specifically designed for data you’ll rarely need to access, but you’re talking incredibly cheap prices for storage – approx $0.01 (1 cent) a month per GB.
As it will all be hosted in the cloud, you’d use AWS’s own console / website to access everything so this would also eliminate the need for finding software to catalogue all your drives.
They offer data replication to different regions / zones (including their main European region being hosted in Ireland) should you be concerned about loss, and you don’t have to worry about physical security / future obsolescence. Their virtual security is also top notch.
For reference, I work in tech support for a software development and consultancy company who uses AWS as their primary data and computing host.
So taking a picture of each page on your phone wouldn’t help?
Taking photos just creates more data so you’re not solving anything. You need to get someone to hand sketch a copy of every page onto paper and the compile them into books.
Machine spirits is the way forward…
New BoW Archive Solution:
I had the problem some time ago of accessing an old IDE drive. Could i suggest one of these:
http://www.amazon.co.uk/AKORD%C2%AE-Drive-Reader-Convertor-Adapter/dp/B00EQ7UEK4/ref=sr_1_2
Here are the storage boxes you are talking about @warzan
http://www.aquaphoenix.com/application/sales/index.cgi?item=18
4 boxes are going to hold your 80 discs when you crack open the ones from the case. Many of the newer ones will be SATA inside anyway. (Maybe some of the cases and internal gubbins will go towards the terrain bitz box 🙂
The problem of the technology moving on is already at the door SSD drives are getting cheaper and with 100 times the access speed they are doing away with the bottleneck that has exsisted in computng for a long time.
I know it’s time consuming but I would recomend getting a SATA IDE bridge and getting everything off the old drives.
Interesting (Things you wouldnt think about) behind the scenes
Cheers
My work used to throw a bunch of those exact same boxes/foam away every day. I’ll check tomorrow if that’s still the case. They recently got a lot more “green” and implemented processes to eliminate waste, but if they’re destined for a recycling plant somewhere I could probably lay my hands on a few of these.
Don’t envy you guys on this one. Having worked in IT contracted to several motor manufacturing groups, I can tell you storage and archiving has never been an easy task, especially given how much data you’re trying to store.
I’m guessing a ballpark figure of 100Tb roughly? With cloud storage, you’re then looking at several thousand pounds a year at the very least, pretty much the same as a JBOD array.
So alternatively we’re back to storing the hard drives outside of the casing but that can be just as much of a pain. The polystyrene boxes are fine for physically protecting the drives, but I hope you’ve protected those drives from moisture and static because they’ll do as much damage as any physical knocks. I’d suggest wrapping them up in anti-static bags if you don’t already do so.
well you asked for it. Woo Hoo Good luck guys your awesome. Keep it up Boys
Tough task to do on a budget. Woohoo! Keep it up boys!
Thanks for the peeps behind the curtain. Woohoo keep it up boys 🙂
wooohoo keep it up boys
You have another problem you did not think of. Magnetic medium only have a very limited shelf life. We are having issues getting the 7 year requirement for PCI data at our company, magnetic tapes have a shelf life of 2-5 years. The suggestion is that the data get refreshed, rewrote to the medium every 2 years. So you will also have to consider that too in your storage of data. Blueray/DVD written disks only have a very limited shelf life too.
The Beast can NOT be contained, it just keeps growing and growing.
you guys will think of something.
Well it would be good to know what this is but it buffers so badly I have given up on it. Which is a shame guys.
Woohoo! Keep it up boys!
Woo Hoo, keep it up boys!
I can be of no further use to you.
Hi,
I think cloud storage costs would rule this option out as (while Glacier is a quarter of the cost of S3 for 160TB it’s still going to come in at 1120 USD per month and there will be data transfer costs on top of that and 90 USD per TB to download the file again might be dealbeaker.
I’d be reluctant to use Hard drives as longer term archival storage as it isn’t exactly what they were designed for – you will probably have to think about how long you are expecting the data that they hold to last as they do deteriorate over far less time than tapes (e.g. lto 6 or 7) which are designed for a 15-30 year storage period or optical discs which as you say are too small.
you can buy ide to usb cables on ebay, mine cost about £3.
I’m with @onlyonepinman on this one. Why do you need to keep everything? It needs to be a non-emotional and objective exercise in “do we need this” just like clearing a house. Especially as you don’t actually know what you have archived on each disk. Not saying you’re wrong to keep everything, just putting the thought out there.
@warzan
I am an IT professional working in a datacenter. I am not really an expert when it comes to storage because my expertise is elsewhere but I may be able to give you some general advice.
Being a small business like yours any professional storage solutions are out of the question due to the costs involved. Cloud storage is also not really an option due to monthly costs and it needs a hell of a lot bandwidth on your internet lines which results in even more costs.
So the only viable alternative is what you are currently already doing: The poor mans storage system with external HDD’s.
I can’t really give you advice on how to organize your data/discs, it all depends on how you work and operate. In general: Don’t keep everything. Identify important from not so important data and simply delete stuff once in a while to fight the ever growing mountain of data a little bit.
VERY IMPORTANT: Keep in mind that HDD’s are not really suited for long time cold storage. It may sound strange but I am not kidding you with the following information:
HDD’s are based on discs which surface is magnetically charged. Those charges can deteriorate over time (Physics 101, magnetic field breakdown). When it happens it’s called “Bit flipping” (because a bit may flip from 1 to 0 or vice versa). The whole process is also known as “bit rot”. Flipping Bits happen all the time but the HDD’s firmware can usually compensate that and repair it via algorithms and a checksum so the user usually never notices anything. When you have a HDD in cold storage for many years it may have several flipped bits accumulated over time beyond a point where the firmware can’t recover it anymore. At this point your OS will try to fix it by using similar mechanisms which are usually built into the file system. If this doesn’t help either, all or parts of your data is corrupted and possibly unusable anymore.
To prevent that the data needs to be actively written to the HDD again to “refresh” the magentic charge. There might be tools out there that will do this for you, but what will work for sure is when you copy all data from one drive to another. Using a rotation pattern you should copy all data every couple years at least once to keep them “fresh”.
“Flipping bits” sounds like what should be on a t-shirt of Moss from the IT Crowd. 😀
You can do eeet! 😀
As you are not competitors but colleagues of guys like Leo Laporte of This Week in Tech fame you may want to reach out and see if there are solutions that they have found for the long term storage. His programs are very techcentric and have kept him in touch with the leading professionals in the industry. He is also a radio guy and he has been very approachable to many of us in that industry, back when I was doing that work, and may help find you a solution that could get paid for by a trade advertising deal. There are lots of good people out there that have the same issues that you do and maybe there is a way to reach them.
Keep up the good work and as usual love the hobby stuff but it is nice to know that other small business owners have issues as well. 🙂
As a specialist consultant for the leading Global IT research firm, i’ve access to quite some posible solutions and the documentation to back it up. I’ll send you some pdf’s for you to look over, just wanted to drop a note here incase they get lost.
WooHoo!! Keep it up boys!!! 😉
If budget prevents you from using services like AWS Glacier ($150usd/m for 200tb ish) than I would recommend doing what Facebook does for cold storage. Write everything to Blu-ray Discs. Sony currently produces discs designed for large volume long life storage.
$1000 will probably get you a good writer and enough discs to last a lifetime. Make 2 or 3 copies of each disc to avoid loss of data to disc failure. Keep each set in different locations (not different locations in the same property).
Wish you’d have shared this with the public, not just backstage. both of my brothers work in IT (GoDaddy and at a university). They’d be good people to share this video with. Do you mind sending me a copy to swing over to them? Cheers!
http://www.amazon.co.uk/Tooploo-eSATA-Reader-Docking-Station/dp/B00GIBYAHY/ref=sr_1_6?s=computers&ie=UTF8&qid=1454890945&sr=1-6&keywords=ide+hard+drive+docking+station
http://www.amazon.co.uk/SATA-Anti-Static-Storage-Tank-Case/dp/B003OBZQUQ
Hope this helps but I have a couple of old PC’s that use IDE drives if your interested. Don’t want anything for them yours if they help 🙂
I now have no idea what everyone is talking about
My suggestion is going to be to wait. Been in IT since college in the late 80s early 90s. There is a new Technology coming out this year called XPoint (cross point is how it is said) and it is a new memory/storage architecture. It is 1000 times more dense than current SSD and RAM DIMM module memory chips. It is going to be a revolution in storage, and old storage methods will drop in price like a stone. Options that are out of reach today, could be viable in 9 months to a year. NAS box or some other similar storage method.
I would suggest Cloud storage, at least as an offsite Backup. Glacier as mentioned is a cheap option for that, just be aware that Amazon charged for data transmission FROM their datacenters. I’m not sure if ongoing operational costs would fit your business either. (200TB of Glacier is about 1500.00 a month) If you’re looking for just a nice method of storage your lovely collection of spinning disks: https://www.getprostorage.com/ would get your organized.
@warzan
Hi,
Another thing to consider with the HDD solution is also SSD do degrade over time. I would recommend getting a tape backup system with the LTO 5/6 Tapes as they are far the best solution and can be cold stored for up to 20 + years. Plus the other benefit for this system it can run independently so you do not tie up all your systems as it can run overnight.
IDE solution it to get a caddy as some people have shown above
Hope this helps