Tech talk, social media, blogging, computing tips and tricks

← →

Peer-to-peer data storage

April 28th, 2010 by David Bradley >> No Comments

This post was chosen as an Editor's Selection for ResearchBlogging.orgWhenever anyone mentions P2P file systems, the first thought that pops into the n00bs head is probably – piracy – and an image of teens downloading free copies of the latest young person’s popular music tracks from teh interwebs using an illicit file sharing system. Of course, Bit Torrent and other related systems can be used perfectly legitimately for sharing files that are not restricted by copyright, such as your company data and distributions that can be shared among staff or a workgroup so much more rapidly using a P2P system (perhaps the biggest reason, aside from privacy and a lack of due process) that should be used to ban ISPs (with RIAA et al pulling the strings) from throttling or blocking torrent traffic.

Anyway, that aside, peer-to-peer systems could also be a useful way to create distributed online data storage, that could be essentially free. A research paper from Taiwan landed on my desk that discusses Malugo, a system that connects peers (e.g. an individual computer, server, cluster, or supercomputer) and lets the owners of those computers upload (encrypted) chunks of their data to other peers, redundantly for storage purposes, backup being just one possible application. There are at least a couple of similar P2P file storage systems out there, among them Wuala, which by sheer coincidence today I also saw mentioned on Ghacks. I also recalled a document from almost a decade ago that discussed a similar system from Microsoft called PAST).

However, the Malugo system does have a novel workaround for several of the issues facing anyone trying to run a peer-to-peer storage system, such as ensuring redunancy of data chunks that avoids data retrieval problems when some peers are not connected.

One might imagine that Malugo is something akin to a Bit Torrent network, but there are important differences. BT is a file-sharing protocol. Malugo is a distributed storage system for grid computing. It uses the P2P technique to link all the distributed storage centers into a single grid system. BT or other kinds of file sharing might be layered on top of Malugo, in order to manipulate files, a database system could also be used to handle data.

The Taiwanese research team explains further that, the Malugo system, is a P2P storage system aimed at those involved in large-scale collaborative projects and perhaps the enterprise. “The Malugo system can cluster peers by routing locality automatically, partitioning files to different peers to achieve load balancing and replicating files to different groups to achieve geographical properties without the global information,” the researchers, who are based in the Department of Computer Science, at the National Tsing Hua University, in Hsinchu, explain.

I asked team member Yeh-Ching Chung to explain the difference between Malugo and Wuala, as at first glance they seem to duplicate purpose. “Wuala is a social distributed file-system for online storage, Chung says, “Malugo is a more closed system compared to Wuala. In other words, Wuala is a public cloud storage system while Malugo is a private cloud storage system.”

“Malugo is the data grid system we designed for the medicare-grid project from National Science Council (NSC), a prototype for which was completed in October 2008. We are now working on a home-grid project, also from NSC, for personal health care.,” Chung told me.

For the home-grid system we are currently developing, the functionalities related to file management we built on top of Malugo are similar to those of Wuala. Since the home-grid is for personal health care, all APIs provided follow Miscrosoft health common unser interface.”

Sign up for Wuala and it gives you 1Gb of free storage. That perhaps does not compare well to the 25Gb you get with each Windows Live account or the 7Gb or so you get with a Google Mail account. Both of which can be accessed as if they were virtual drives using Gladinet or a similar app and SkyDrive for the Microsoft service. So, is there an advantage of P2P over cloud storage? Well, in the case of Wuala and any that follow the big advantage could be that if you are willing to offer several gigabytes of capacity on your own hard drives and can sustain them as accessible online for at least 17% of the time, you can get the equivalent space on other drives in the P2P network. So, if you have a spare 500Gb external network attached storage device you could grab 500Gb of space in the cloud from fellow users of the system.

External hard drives are so inexpensive these days that you could easily save hundreds of dollars in online stroage costs with a conventional cloud storage service.

That said, given that external drives are so cheap, why not simply by two or three, backup to all of them, relocate them to a remote, but trusted place. Family or friends’ house perhaps, or your remote office safe. Keep one accessible, for daily backups and redundantly cycle them round their locations on a weekly or monthly basis. Sounds inconvenient, but this is your precious data we are talking about. It’s always after a catastrophic failure that people whinge about wishing they’d backed up, so do it now.

I’d also recommend running a scheduled backup for recent data files that stores to your Windows Live account(s) using Gladinet, which can mount all these cloud storage spaces as if they were just one more hard disk connected to your PC. Aside from the cloud security issues, this could be a much better option than external drives, although I opt for using both simultaneously. As to whether I’d want to store my data (encrypted or otherwise) on the machines of my peers…I’m undecided. Belt and braces for backup is important though.

Research Blogging IconYu-Wei Chan, Tsung-Hsuan Ho, Po-Chi Shih, & Yeh-Ching Chung (2010). Malugo: A peer-to-peer storage system Int. J. Ad Hoc and Ubiquitous Computing, 5 (4), 209-218

Footnote: The makers of Gladinet let me have a try out of the full version of their program, which can scheduled backups.

Related articles on cloud storage