Roll Your own 2TB Dropbox on mac/unix

Posted: 28th May 2012 by hackengineer in networking, scripting
Tags: , , ,

I have a lot of friends who are against having all of there data off in some cloud with who knows snooping  around.  For me this isnt an issue as I dont usually put sensitive material online and if I do its encrypted with disk utility.  It wasn’t until I had to get a laptop to be able to have all my work with me while going back to school full time that I realized I needed much more synced space than the 7 free GB dropbox provided (2GB default + accrued referral bonus).  To upgrade to a 100GB dropbox would cost $20 a month, no thanks.  Since I pretty much always leave my desktop as a SVN server, I decided it was going to become my personal 2TB dropbox.

There are a bunch of programs out there that have plenty of bells and whistles to help with this.  All I needed was something to sync specified folders from my desktop to laptop. It had to be a bi-directional sync meaning if I added or removed a file on one side the same would happen on the other.  I didn’t need backups (that was time-machine is for) or versioning (all code is kept in svn repos).  Goodsync seemed to be good but I didnt feel like paying $30 + $10 per license.  I saw freeware with poor reviews or not really bi-directional (no remote deletion).  I had been looking for a good excuse to work on my bash scripting abilities and decided this would make for a good project.

All macs come with a handy program called rsync.  It is really easy to use and solved half of the problem up front, one way sync.  If I only needed to mirror a folder I would have been done, but that is not very dropbox like.  I want to add and delete to both folders at any given time.  Rsync’s conundrum is that it cannot tell if a local missing file is because it was deleted locally or added remotely.  Linux pinguin gives a good run though of this issue.

1
2
3
4
5
#local->remote
rsync -auv -e ssh mySyncFolder/ remoteHost.com:/remoteSyncFolder

#remote->local
rsync -auv -e ssh remoteHost.com:/remoteSyncFolder mySyncFolder/

My solution uses rsync to power the transfer and adds logic to keep track of deleted files.  A list of all local deleted files is maintained on both sides (as of now no GUI exists so files must be deleted via terminal).  When a sync occurs the remote list of deleted files is pulled and all local copies are locally deleted if they have the identical size/mod time.  Now all of the files that have been deleted on the remote side are in sync locally but the locally deleted files still exist on the remote side.  These files will not be deleted until the remote computer initializes the sync and does the same process.  To avoid downloading files that have been deleted locally and not remotely, the local list of deleted files is used to tell rysnc not to blacklist these files. The same script is ran on the remote to sync its local files.

mySync flow diagram
All this is seamlessly automated by adding the script to a cronjob set for every 15 minutes ( or any time interval of you choice) and a very basic dropbox has been implemented!  Now I can transition from my laptop to desktop and back with out having to worry about manually tranfering large amounts of data.  Best of all its free with practically unlimited space!

1
*/15 * * * * /usr/loca/bin/syncMe.sh

If your interested in doing this yourself I will be posting part II with a walk through of the code and how to set up DDNS/port forwarding/sftp/authentication on the mac.  All you need to sync data securely from anywhere!

next page —>


Be Sociable, Share!