Download syncMe Beta 0.1 Here (668)

To get two synced is pretty easy once you have the networking figured out. You might want to put the scripts in your /Users/local/bin/ directory but you can put them where ever you like. Add the path to the synced directories in BOTH scripts. Now you are ready to sync. To test run the script in the terminal:

1
2
3
4
5
6
7
8
9
10
11
$ syncMe.sh
Initializing local sync folder
Initializing remote sync folder none->1.0
Setting Remote Delete Lock
Checking Remote Sync Lock
Checking for remote Deletions
Deleted files in Sync
Clearing Remote Delete Lock
Receiving
receiving file list ... done
EngineeringProjects/mySync/

sent 38 bytes received 398811 bytes 159539.60 bytes/sec
total size is 55629350557 speedup is 139474.71
building file list … done
EngineeringProjects/mySync/DDNS_PortForwarding.png
EngineeringProjects/mySync/DDNS_PortForwarding.vsd

sent 516249 bytes received 2302 bytes 345700.67 bytes/sec
total size is 55629350567 speedup is 107278.46

Thats it! Do the same steps on the remote side for a full bi-directional sync. If you want to delete files

1
2
$ syncDel.sh deleteme
Removing deleteme

When the remote initializes a sync the file deletme will be deleted. Last thing is to automate the sync with a cronjob. On the terminal type contab -e and add

1
2
RUNNINGFROMCRON=1
*/15 * * * * /usr/local/bin/syncMe.sh

This set a variable so the script knows its being run from a cronjob and will not ask for user input and will sync the folder every 15 minutes.  And there you have it, virtually unlimited sync space for free!

Projects Home


Coding a Simple Dropbox with Bash

Posted: 29th May 2012 by hackengineer in networking, scripting
Tags: , , , ,

I wrote this bash script very recently and I would say its still in “beta”. USE AT YOUR OWN RISK!! It has been working great but I recommend backing up your synced data just in case.

With that said lets get into the code! I have two bash scripts, one to sync the data and one to properly delete files. In order to get two folders synced for the first time is enter the path to the local sync folder, the path to the remote sync folder, and the port. Remember that by default rsync does not have access to the remotes root folder only the user home directory that you log in as. The remote sync directory is relative to this directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
#!/bin/sh

#Copyright (2012 hackengineer.com)
#Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
#The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

VAR_LOCAL_SYNC_DIR="/Users/myUserName/mySyncDir" #ADD root sync directory here without trailing /
VAR_REMOTE_SYNC_DIR="www.remoteHost.com:remoteSyncDir" #ADD remote sync directory relative to ssh directory here without trailing /
#VAR_REMOTE_SYNC_DIR="mac.local:remoteSyncDir" #local network example
VAR_PORT=22
VAR_RSYNC="/usr/bin/rsync" #specify where rsync lives if needed
#--------------------------------------------------

VAR_VERSION=0.1


function deleteFile {
    echo "removing $VAR_FULL_FILE"
    VAR_DATE=`date`
    echo "$VAR_FULL_FILE,$VAR_TIMESTAMP,$VAR_SIZE,$VAR_DATE" >> $VAR_LOCAL_SYNC_DIR/.rsync/rsyncLocalDeleteLog
    `rm -r $VAR_FULL_FILE`
}

#Will not delete from a cron
function proptDelete {
    if [ "$RUNNINGFROMCRON" != 1 ]; then
        echo "$VAR_FULL_FILE changed, Rem->Loc Size:$VAR_SIZE->$VAR_MY_SIZE ModTime:$VAR_TIMESTAMP->$VAR_MY_TIMESTAMP"
        echo "Delete?(y/n) "
        read word1
        if [ $word1 = "y" ]; then
            deleteFile
        fi
    else
        echo "$VAR_FULL_FILE changed!! Size:$VAR_SIZE->$VAR_MY_SIZE ModTime:$VAR_TIMESTAMP->$VAR_MY_TIMESTAMP" >> $VAR_LOCAL_SYNC_DIR/.rsync/rsyncErrorLog
    fi
}


#Verify local sync dir exists
if [ ! -e $VAR_LOCAL_SYNC_DIR/.rsync ]; then
    echo "<initializing local sync folder>"
   
    #make sure root dir exists
    if [ ! -e $VAR_LOCAL_SYNC_DIR ]; then
        mkdir $VAR_LOCAL_SYNC_DIR
    fi

    #create .rsync dir
    `mkdir $VAR_LOCAL_SYNC_DIR/.rsync`

    #add neccisary files.
    touch $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDelete
    touch $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteTemp
    touch $VAR_LOCAL_SYNC_DIR/.rsync/rsyncExcludeList
    echo 'clear' > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLock
    touch $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLockTemp
    echo 'clear' > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncSyncLock
    touch $VAR_LOCAL_SYNC_DIR/.rsync/rsyncSyncLockTemp
    touch $VAR_LOCAL_SYNC_DIR/.rsync/rsyncVersion
    touch $VAR_LOCAL_SYNC_DIR/.rsync/rsyncVersionTemp
    touch > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncErrorLog
    echo 'Filename,ModifiedTime,Size,DeleteTime' > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncLocalDeleteLog
    echo $VAR_VERSION > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncVersion
fi

#Verify remote sync dir exists suppress error if rsync fails
echo "none" > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncVersionTemp
$VAR_RSYNC -q -e "ssh -p $VAR_PORT" $VAR_REMOTE_SYNC_DIR/.rsync/rsyncVersion $VAR_LOCAL_SYNC_DIR/.rsync/rsyncVersionTemp 2> /dev/null
#echo "1" >>$VAR_LOCAL_SYNC_DIR/.rsync/rsyncVersionTemp
VAR_REMOTE_VERSION=`cat $VAR_LOCAL_SYNC_DIR/.rsync/rsyncVersionTemp`

if [ $VAR_REMOTE_VERSION != $VAR_VERSION ]; then # if versions DONT match
    echo "</initializing><initializing remote sync folder $VAR_REMOTE_VERSION->$VAR_VERSION>"
    # copy all files in .rsync
    $VAR_RSYNC -r -e "ssh -p $VAR_PORT" $VAR_LOCAL_SYNC_DIR/.rsync $VAR_REMOTE_SYNC_DIR/
    # init delete/exclude list
    echo "" > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLockTemp
    $VAR_RSYNC -r -e "ssh -p $VAR_PORT" $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLockTemp $VAR_REMOTE_SYNC_DIR/.rsync/rsyncDelete
    $VAR_RSYNC -r -e "ssh -p $VAR_PORT" $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLockTemp $VAR_REMOTE_SYNC_DIR/.rsync/rsyncExcludeList
    # init locks to clear
    echo "clear" > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLockTemp
    $VAR_RSYNC -r -e "ssh -p $VAR_PORT" $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLockTemp $VAR_REMOTE_SYNC_DIR/.rsync/rsyncDeleteLock    
    $VAR_RSYNC -r -e "ssh -p $VAR_PORT" $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLockTemp $VAR_REMOTE_SYNC_DIR/.rsync/rsyncSyncLock
fi

#Send Lock
echo "<setting Remote Delete Lock>\r"
echo 'lock' > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLockTemp
$VAR_RSYNC -e "ssh -p $VAR_PORT" $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLockTemp $VAR_REMOTE_SYNC_DIR/.rsync/rsyncDeleteLock

#Check Remote SyncLock Before Sync
echo  "<checking Remote Sync Lock>\r"
$VAR_RSYNC -e "ssh -p $VAR_PORT" $VAR_REMOTE_SYNC_DIR/.rsync/rsyncSyncLock $VAR_LOCAL_SYNC_DIR/.rsync/rsyncSyncLockTemp
VAR_LOCK=`cat $VAR_LOCAL_SYNC_DIR/.rsync/rsyncSyncLockTemp`
while [ $VAR_LOCK = 'lock' ];
do
    echo "remote Delete in progress.  please wait..."
    sleep 2;
    $VAR_RSYNC -e "ssh -p $VAR_PORT" $VAR_REMOTE_SYNC_DIR/.rsync/rsyncSyncLock $VAR_LOCAL_SYNC_DIR/.rsync/rsyncSyncLockTemp
    VAR_LOCK=`cat $VAR_LOCAL_SYNC_DIR/.rsync/rsyncSyncLockTemp`
done


#Get Delete List
echo  "</checking><checking for remote Deletions>\r"
$VAR_RSYNC -e "ssh -p $VAR_PORT" $VAR_REMOTE_SYNC_DIR/.rsync/rsyncDelete $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteTemp


#Delete Files that match name/size/mod date
for i in `cat $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteTemp`
do
    VAR_FILE=`echo $i | awk -F, '{print $1}'`
    VAR_TIMESTAMP=`echo $i | awk -F, '{print $2}'`
    VAR_SIZE=`echo $i | awk -F, '{print $3}'`

    VAR_FULL_FILE="$VAR_LOCAL_SYNC_DIR/$VAR_FILE"
    if [ -e $VAR_FULL_FILE ]; then  

        #Get size for directory or file
        if [ -d $VAR_FULL_FILE ]; then
            VAR_MY_SIZE=`du -s $VAR_FULL_FILE | awk {'print $1'}`
        else
            VAR_MY_SIZE=`ls -laT $VAR_FULL_FILE | awk {'print $5'}`
        fi
        VAR_MY_TIMESTAMP=`stat -f "%m" -t "%s"  $VAR_FULL_FILE`
       
        #Verify Local and romote files are identical.
        if [ $VAR_TIMESTAMP != $VAR_MY_TIMESTAMP ]; then
            proptDelete
        elif [ $VAR_SIZE != $VAR_MY_SIZE ]; then
            proptDelete
        else
            deleteFile
        fi
    else
        echo "$VAR_FULL_FILE < --File does not exist localy"
    fi
done

#Send Clear Delete List
echo  "<Deleted files in Sync>\r"
`echo > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteTemp`
$VAR_RSYNC -e "ssh -p $VAR_PORT" $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteTemp $VAR_REMOTE_SYNC_DIR/.rsync/rsyncDelete
$VAR_RSYNC -e "ssh -p $VAR_PORT" $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteTemp $VAR_REMOTE_SYNC_DIR/.rsync/rsyncExcludeList

#Send Lock release
echo  "<clearing Remote Delete Lock>\r"
echo 'clear' > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLockTemp
$VAR_RSYNC -e "ssh -p $VAR_PORT" $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLockTemp $VAR_REMOTE_SYNC_DIR/.rsync/rsyncDeleteLock

#Bi Directional Sync
echo "<receiving>"
$VAR_RSYNC -auv --copy-links --keep-dirlinks -e "ssh -p $VAR_PORT" --exclude '.rsync' --exclude-from $VAR_LOCAL_SYNC_DIR/.rsync/rsyncExcludeList $VAR_REMOTE_SYNC_DIR/ $VAR_LOCAL_SYNC_DIR/

echo "\n\n<sending>"
$VAR_RSYNC -auv --copy-links --keep-dirlinks -e "ssh -p $VAR_PORT" --exclude '.rsync' $VAR_LOCAL_SYNC_DIR/ $VAR_REMOTE_SYNC_DIR/
</sending></receiving></clearing></checking></setting></initializing>

The code is well commented but here is a quick rundown. When the script is ran for the first time it will create the specified directorys if they dont exist and initializes a .rsync folder which holds the system files.

Next it sets the remote “deleteLock file” to prevent any remote files from being deleted while a sync is taking place. In case files were being deleted before the lock was set, a synclock file is checked prior to checking out the deleted file list.

Once cleared the delete file list is downloaded. The files are checked to make sure the modification date and size are identical before deleting the files. This helps ensure only identical files are deleted in case of the match had been modified after a long period without syncing. After the deleted files have been removed locally, the remote list is cleared.

Finally the remote modified files are downloaded using rsync. The –copy-links and –keep-dirlinks options have been enabled as I like to selectively chose which folders to sync while maintaining the original directory structure. My sync folder typically contains soft links (ln -s ) aka shortcuts to folders I would like to have synced. Also the -a options ensures that the permissions, modified date, for example are preserved. The -u option only transfers files that are newer.

Rsync has many options which can be customized to your situation. Be sure to be careful as it can delete files on both sides.

For the delete script all you need to do is again add the local sync directory. It take the files you would like to delete as command line arguments and DOES NOT PROMPT for each deletion, so be sure you have the right files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
#!/bin/sh

#Copyright (2012 hackengineer.com)
#Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
#The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

#ADD root sync directory Here
VAR_LOCAL_SYNC_DIR="/Users/userName/sync"
#--------------------------------------------------


#make sure sync directory has been inititalized
if [ ! -e $VAR_LOCAL_SYNC_DIR/.rsync ]; then
    echo "Please run 2 way sync to initialize folder prior to deleting files"
    exit
fi

#Set Sync Lock
echo 'lock' > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncSyncLock

#Verify no remote sync is in progress
VAR_LOCK=`cat $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLock`
while [ $VAR_LOCK = "lock" ];
do
    echo "remote sync in progress.  please wait..."
    sleep 2;
    VAR_LOCK=`cat $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDeleteLock`
done

#Log relative file path, size and mod date prior to deletion
VAR_LOCAL_SYNC_DIR_E=`echo $VAR_LOCAL_SYNC_DIR | perl -pe 's|\/|\\\/|g'`
for i in $@
do
    #Verify file exists
    if [ -e $i ]; then

        #Check for directory
        if [ -d $i ]; then
            VAR_SIZE=`du -s $i | awk {'print $1'}`
        else
            VAR_SIZE=`ls -laT $i | awk {'print $5'}`
        fi
       
        #if in root dir
        if [ `pwd` = $VAR_LOCAL_SYNC_DIR ]; then
            VAR_RELATIVE_DIR=""
        else
            VAR_RELATIVE_DIR=`pwd | perl -pe "s/.*($VAR_LOCAL_SYNC_DIR_E\/?)(.*)/"'\$2'"/"`"/"
           
        fi

        VAR_TIMESTAMP=`stat -f "%m" -t "%s"  $i`
        #Log files for remote deletion
        echo "$VAR_RELATIVE_DIR$i,$VAR_TIMESTAMP,$VAR_SIZE" >> $VAR_LOCAL_SYNC_DIR/.rsync/rsyncDelete
        #Log blocked sync list until remote has deleted local copies
        echo "$VAR_RELATIVE_DIR$i" >> $VAR_LOCAL_SYNC_DIR/.rsync/rsyncExcludeList
        #Remove file
        echo "Removing $VAR_RELATIVE_DIR$i"
        `rm -r $i`
    else
        echo "$i does not exist"
    fi
done

#Clear Sync Lock
echo 'clear' > $VAR_LOCAL_SYNC_DIR/.rsync/rsyncSyncLock

It is worth noting that this script does use “rm -r”! Be sure of what your deleting when using this script

next page —>


Alright, if you want to sync two computers you will need to enable sftp.  On a Mac this is easy.  Open system preferences, sharing and enable Remote Login.  This enables ssh on your mac on port 22.  If you have two machines using the same router each machine will need to have a different port. On mac you need to open /System/Library/LaunchDaemons/ssh.plist and change serviceName and bounjour string from ssh to the desired port. I use 22 on my desktop and 26 on my laptop. Here is a list of standard ports.

Now that ssh is enabled on both computers you need to know how to find it on the internet.  The process isnt hard and can be done for free.  The process will start with the host domain name in a nice human readable format (www.remoteHost.com), but this is no good for a computer, it likes numbers.  So using DDNS it can translate the human readable domain name to the actual ip address.  I use no-ip’s free service and run its dynamic updater on each of my machines.  This will get us to the remote router, but now the router needs to know what computer to send the request to as there usually are many devices connected to any given router.  Using port forwarding the router can be configured to forward any given port to a given computer.  We will be using sftp (ssh) which is by default on port 22 but this can can be specified.  Therefore the router will need to forward any port 22 requests to the “remote” computer illustrated below.  This is done differently depending on your router.


Now that we know how to find the remote destination we need to do this securely.  By using rsync with the -e option we can specify the use of a secure shell (SSH).  This is great but we want to do it without having to enter the password every time we want to transfer data.  To get around this we will first have to generate a set of authentication keys using ssh-keygen.  Open a terminal and enter the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ cd ~/.ssh/
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/*****/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/*****/.ssh/.
Your public key has been saved in is_rsa.pub.
The key fingerprint is:
sd:34:56:4j:56:hg:45:fg:45:4g:5h:42:df:9c:f3:dc
The key's randomart image is:
+--[ RSA 2048]----+
|        . o + ..o|
|     . o o . o o.|
|       o + o . o |
|       . o . o   |
|      . . S . .  |
|       o . +.    |
|       0 . o+    |
|       . .=      |
|        E        |
+-----------------+

Leave the passphrase blank. This will generate a keypair (private and public). Leave the private key in the .ssh folder and copy the public key contents into the authorized_users file in the remote .ssh folder. For example enter the following in the terminal:

1
2
3
4
5
$ rsync ~/.ssh/id_rsa.pub user@www.remoteHost.com:.ssh/id_rsa.pub
$ ssh user@www.remoteHost.com
password:********
$ cat ~/.ssh/id_rsa.pub &gt;&gt; ~/.ssh/authorized_keys
$ exit

You also may need to adjust the permissions to 600 if you have problems. Now you should be able to ssh into the remote computer without entering a password. Authentication is done by matching the private key with the public key automatically. Do not share the private key!

next page—>


Roll Your own 2TB Dropbox on mac/unix

Posted: 28th May 2012 by hackengineer in networking, scripting
Tags: , , ,

I have a lot of friends who are against having all of there data off in some cloud with who knows snooping  around.  For me this isnt an issue as I dont usually put sensitive material online and if I do its encrypted with disk utility.  It wasn’t until I had to get a laptop to be able to have all my work with me while going back to school full time that I realized I needed much more synced space than the 7 free GB dropbox provided (2GB default + accrued referral bonus).  To upgrade to a 100GB dropbox would cost $20 a month, no thanks.  Since I pretty much always leave my desktop as a SVN server, I decided it was going to become my personal 2TB dropbox.

There are a bunch of programs out there that have plenty of bells and whistles to help with this.  All I needed was something to sync specified folders from my desktop to laptop. It had to be a bi-directional sync meaning if I added or removed a file on one side the same would happen on the other.  I didn’t need backups (that was time-machine is for) or versioning (all code is kept in svn repos).  Goodsync seemed to be good but I didnt feel like paying $30 + $10 per license.  I saw freeware with poor reviews or not really bi-directional (no remote deletion).  I had been looking for a good excuse to work on my bash scripting abilities and decided this would make for a good project.

All macs come with a handy program called rsync.  It is really easy to use and solved half of the problem up front, one way sync.  If I only needed to mirror a folder I would have been done, but that is not very dropbox like.  I want to add and delete to both folders at any given time.  Rsync’s conundrum is that it cannot tell if a local missing file is because it was deleted locally or added remotely.  Linux pinguin gives a good run though of this issue.

1
2
3
4
5
#local->remote
rsync -auv -e ssh mySyncFolder/ remoteHost.com:/remoteSyncFolder

#remote->local
rsync -auv -e ssh remoteHost.com:/remoteSyncFolder mySyncFolder/

My solution uses rsync to power the transfer and adds logic to keep track of deleted files.  A list of all local deleted files is maintained on both sides (as of now no GUI exists so files must be deleted via terminal).  When a sync occurs the remote list of deleted files is pulled and all local copies are locally deleted if they have the identical size/mod time.  Now all of the files that have been deleted on the remote side are in sync locally but the locally deleted files still exist on the remote side.  These files will not be deleted until the remote computer initializes the sync and does the same process.  To avoid downloading files that have been deleted locally and not remotely, the local list of deleted files is used to tell rysnc not to blacklist these files. The same script is ran on the remote to sync its local files.

mySync flow diagram
All this is seamlessly automated by adding the script to a cronjob set for every 15 minutes ( or any time interval of you choice) and a very basic dropbox has been implemented!  Now I can transition from my laptop to desktop and back with out having to worry about manually tranfering large amounts of data.  Best of all its free with practically unlimited space!

1
*/15 * * * * /usr/loca/bin/syncMe.sh

If your interested in doing this yourself I will be posting part II with a walk through of the code and how to set up DDNS/port forwarding/sftp/authentication on the mac.  All you need to sync data securely from anywhere!

next page —>


Matlab provides a great environment to work with images and therefore a great place to develop the Structured Light Algorithm.  The beagle board was used to project the structured light images and capture the resulting scene.  These images were then loaded into matlab providing data needed to develop the algorithm.  Working with Hana Wang and Rajoshi Biswas, the majority of the algorithm was developed as follows:

At this point we were able to create decent point clouds with matlab.  The resulting pointclouds can be shown in matlab using the ‘mesh’ function.  To add color us the ‘caxis’ function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
%% Total number of Images n = 20 %%
n = 18;
height = 480;
width = 640;
%scan = zeros(height, width, 3, n);
figure;

l=[1,3,5,7,9,11,13,15,17,19];       %% Two sequences, l and r
r=[2,4,6,8,10,12,14,16,18,20];
binProjCol = zeros(height,width,9);
bright = zeros(height,width,9);
projCol = zeros(height,width);
projColNormalized = zeros(height,width);
dright = zeros(height,width);
camCol = zeros (height, width);
camColNormalized = zeros (height, width);
%y = zeros (height, width);
depth = zeros (height, width);
%% Read the images obtained by scanning %%
for i = 1 : n
    scan(:,:,:,i) = imread(strcat([num2str(i) '.png']));
    imshow(scan(:,:,:,i));
    %pause;
end

display('Images read')

%% To decode bit sequence %%
%grayscan = zeros(height,width,n);
grayProjCol = zeros(height,width,n);
binary = zeros(height,width);

fid = fopen('pointcloud_Steven.xyz','w');
%% Convert from rgb2gray
for i=1:n
    %grayscan(:,:,i)= mat2gray(scan(:,:,:,i)); % used to verify sl patterns
    grayscan(:,:,i)= rgb2gray(scan(:,:,:,i)); %grayscan is a double
    imshow(grayscan(:,:,i));
    %    pause;
end

%% Extract bitsequence, find column number and the final coordinates
for k = 1 : height
    for j = 1: width
        %% To extract bitsequence from the images
        for i=1:(n/2)
            if abs(grayscan(k,j,l(i))-grayscan(k,j,r(i))>50)
                if grayscan(k,j,l(i)) > grayscan(k,j,r(i))
                    grayProjCol(k,j,l(i))=1;
                    %grayProjCol(k,j,r(i))=0;
                else
                    grayProjCol(k,j,l(i))=0;
                    %grayProjCol(k,j,r(i))=1;
                end
            end
        end
    end
end
for i=1:n
    imshow(grayProjCol(:,:,i));
    %pause;
end
disp('bitsequence extracted');
%% Convert the gray code sequence to decimal number = column number

for k = 1 : height
    for j = 1: width
        binProjCol(k,j,1) = grayProjCol(k,j,l(1));
        projCol(k,j) = binProjCol(k,j,1)*2^((n/2)-1);
        for i = 2 :(n/2)
            binProjCol(k,j,i) = xor(grayProjCol(k,j,l(i)), binProjCol(k,j,i-1) );
            projCol(k,j) = projCol(k,j) + ((2^((n/2)-i))*(binProjCol(k,j,i)));
        end
    end
end

tempMax = 0;
tempMin = 2^(n/2);
projColMean = mean(projCol);
for k = 1 : height
    for j = 1: width
        if(projCol(k,j)>projColMean(j)+150)
            projCol(k,j) = NaN;
        elseif (projCol(k,j)> tempMax)
            tempMax = projCol(k,j);
        end
        if(projCol(k,j)<projcolmean (j)-20)
            projCol(k,j) = NaN;
        elseif (projCol(k,j)< tempMin)
            tempMin = projCol(k,j);
        end
    end
end

       
[r,firstCamCol,z] = find(projCol,1,'first');
[r,lastCamCol,z] = find(projCol,1,'last');
[r,c,projColList] = find(projCol);
numOfCamCol = lastCamCol-firstCamCol;
firstProjCol = min(projColList);
lastProjCol = max(projColList);
numOfProjCol = lastProjCol - firstProjCol;


for k = 1 : height
    for j = 1: width

        %y(k,j) = k;
        if (projCol(k,j) ~= NaN)
           
            %% Calculate the coordinates of the point
            projColNormalized(k,j) = (projCol(k,j)-(firstProjCol-1))/(numOfProjCol); %480
            camColNormalized(k,j) = ((j-(firstCamCol-1))/numOfCamCol); %width
           
            if ( abs(((camColNormalized(k,j)- projColNormalized(k,j))))&lt;0.10)
                depth(k,j) = (camColNormalized(k,j)-projColNormalized(k,j));
                %depth(k,j) = ((projColNormalized(k,j)-(camColNormalized(k,j))));
            end
        else
            depth(k,j) = NaN;
        end
        fprintf(fid,'%4.1f %4.1f %4.9f %d \r\n',j,k,depth(k,j),projCol(k,j));
    end
end

The code above works but notice it does not handle cleaning up any erroneous points where no Structured Light is found (fuzzy black and white areas).   This was implemented after the code was ported to C.  Since the noise has a gaussian distribution it can be filtered out using a gaussian filter.

1
2
3
4
5
6
7
8
9
disp('Calculated depth');
zInv = flipud(depth);
H2 = fspecial('gaussian',5,.6);
zInvFilt = imfilter(zInv,H2);

figure;
mesh(zInvFilt);
caxis ([0.001 0.013]);
%caxis ([-0.007 0.0125])

This cleans up the noise and provides good results.  The video below shows a better representation of the 3D model.  The noise has been greatly reduced in the blue area where no SL patterns were projected. There are a few areas that were not the best surfaces to reflect the projected light such as the glossy surface that is close to being parallel with the camera.

<–Previous     Project Home     Next –>


Rendering results with Meshlab

Posted: 26th February 2012 by hackengineer in Computer Vision
Tags: , ,

Meshlab is pretty great for 3D point clouds and its free!  Here are a few steps that help really make the point clouds look good.

  • Open Meshlab and open the .xyz file from the 3D camera
  • Delete any points that look like they dont belong ( if you only see a small group of points you are probably zoomed out really far dude to a rogue point; delete it and zoom in)

  • Orient the pointcloud so that it represents the original scene (looking straight at it).  We will now compute the normals for each point.
    • Filters->point Set->Compute Normals For Point Sets
      • # of neighbors = 100
      • Check flip normals w.r.t viewpoint
    • Render->Lighting->Light On
    • The points should now have a shading effect depending on there normal.  To verify render->Show Vertex Normals

  • Lets add some color to the depth to make it stand out (darker as depth increases).  Meshlab has a tool to set color per vertex.  Use the z coordinate as a variable as shown below.
    • Filters->Color Creation and Processing->Per Vertex Color Function
    • For each of the colors adjust this formula for good results; z*0.78+125

  • We should have a good looking pointcloud at this point.  We can also generate a surface mesh with our point cloud.
    • Filters->Point Set->Surface Reconstruction:Poisson
      • Set Octree Depth to 10 or so
    • The toolbar has a list of different view options.  Click the ones with cylinders on them to view in surface mode.

And there you have it!  3D picture taken with the beagleboard.  The results look pretty good for a completely portable setup and HVGA projector.  I look forward to seeing what others can do with high resolution hardware!  Thanks for reading!

<–Previous     Project Home


The source code is posted on google code.  You can download it straight from the interwebs with beagleboard.  Download the folder, run the Makefile, run ./slGPIO.cpp, and push the button to start scanning!  Once completed if the program finds a usbdrive it will save the point cloud there.  If not it will be saved in the results folder.

Contents

  • Graycoded Structured Light images [20]
  • Makefile  // includes links to openCV cflags and libs.  Use this to natively compile SL code.
  • sl.cpp  //sl program
  • slGPIO.cpp  // starting point.  Initializes GPIO interrupt and calls sl.cpp
  • results folder //result go here if no SDcard is found
Makefile
Makefiles are a timesaver, so Ill post the entire contents here for copy/paste purposes (myself included).  Notice the shell call to pkg-config.  This is a really handy way to link to the location of the openCV cflags and libs
# 'make'        build executable file 'mycc'
# 'make clean'  removes all .o and executable files

.PHONY: depend clean

SCAN = slGPIO
DISPLAY = display

CC = gcc
CFLAGS := -g -Wall -DLINUX $(shell pkg-config opencv --cflags)
LIBs := $(shell pkg-config opencv --libs)
LFLAGS =

# Source files
SRCs := \
	slGPIO.cpp
	#sl.cpp \

# List of object files to compile
OBJs = $(SRCs:.cpp=.o)

all: $(SCAN)

$(DISPLAY): display.o
	gcc -g -Wall 

$(SCAN): $(OBJs)
	$(CC) $(CFLAGS) $(INCLUDES) -o $(SCAN) $(OBJs) $(LFLAGS) $(LIBs)

# 	cp $(SCAN) Release/

%.o: %.cpp %.h
	$(CC) $(CFLAGS) $(INCLUDES) -c $<  -o $@

clean:
	$(RM) *.o *~ $(SCAN)
sl.cpp

The sl code was written by myself and Peter Hokanson.  Its not to long and is broken up into 3 sections.  The source code will likely be easier to read with proper highlighting.  OpenCV was only used to manipulate the camera and projector.  Part 1 below projects and captures the images:

#include <stdlib.h>
#include <stdio.h>
#include <math.h>
#include <assert.h> // For my own paranoia's sake
#include <opencv/cv.h>
#include <opencv/highgui.h>
#define WHITE_THRESHOLD 5
#define BLACK_THRESHOLD -5
#define NUM_IMAGES 18
#define CAM_WIDTH 640
#define PROJ_WIDTH 640
#define Z_SCALE 500   //250 works

//#define TEST_RECOGNIZER

int gray2bin(int gray);
bool file_exists(const char * filename);
int slScanner(void);

int slScanner(void)
{
	//*********PART1 - Project SL and Cap Imgs************
	//Initialize
	IplImage* slPat[NUM_IMAGES] = {0};
	IplImage* capImg[NUM_IMAGES + 1] = {0};
// 	int height,width,step,channels;
	uchar *data;
	char s[128];
	int i,outFileNum=0;

	//Usage
	//if(argc>1){
	//	printf("Usage: sl images are in same folder named grayIMGx.png and range from 0-19\n");
	//	exit(0);
	//}

	//Load sl images
#ifndef TEST_RECOGNIZER
	for(int i=0;i<NUM_IMAGES;i++){
		sprintf(s,"%d.png",i+1);
		slPat[i]=cvLoadImage(s);
		if(!slPat[i]){
			printf("Could not load image file: %s\n",s);
			exit(0);
		}
	}

	// create a window
	cvStartWindowThread();
	cvNamedWindow("mainWin", CV_WINDOW_NORMAL);
	cvSetWindowProperty("mainWin", CV_WND_PROP_FULLSCREEN, CV_WINDOW_FULLSCREEN);

	// pause
	cvWaitKey(500);

	// set up camera
	CvCapture* capture = cvCaptureFromCAM( CV_CAP_ANY );
	if (!capture){
		fprintf( stderr, "Camera not found \n");
		getchar();
		exit(0);
	}
	cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT, 480);
	cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH, 640);
	//cvSetCaptureProperty(capture, CV_CAP_PROP_FPS, 5);
	//cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_COUNT, 1);
	//Maybe this will fix the frame delay?

	//Display images and capture
	for(int i=0;i<NUM_IMAGES;i++){

		// show the image
		cvShowImage("mainWin", slPat[i] );

		// pause
		cvWaitKey(5);

		// capture the image
		cvGrabFrame(capture);
		cvGrabFrame(capture);
		cvGrabFrame(capture);
		cvGrabFrame(capture);
		cvGrabFrame(capture); //Get to last frame in buffer
		capImg[NUM_IMAGES + 1] = cvRetrieveFrame(capture);   

		// write the image
		//cvWriteFrame(writer, capImg[i]);
		capImg[i] = cvCloneImage(capImg[NUM_IMAGES + 1]);
		sprintf(s,"results/%d.png",i+1);
		cvSaveImage(s,capImg[i]);
	}

	cvReleaseCapture(&capture);

	//Display captured images for debug
	for(int i=0;i<NUM_IMAGES;i++){

		// show the image (for debug)
		cvShowImage("mainWin", capImg[i] );

		// pause
		cvWaitKey(100);
	}

#else // TEST_RECOGNIZER
	for(int i=0;i<NUM_IMAGES;i++) {
		sprintf(s,"setupp/%d.png",i+1);
		capImg[i]=cvLoadImage(s);
		if(!capImg[i]){
			printf("Could not load image file: %s\n",s);
			exit(0);
		}
	}
#endif

Part 2 contains the SL decoding and depth redering algorithm:

	//*********PART2 - Decode captured images************

	// First initialize a matrix for showing the decoded values, needs 10 bits,
	// so could move to short* if memory becomes an issue.
	int *columnmap = (int*)calloc(capImg[0]->imageSize, sizeof(int));
	if(columnmap == NULL) {
		printf("Allocation for column map failed! Aborting\n");
		return 1;
	}

	// We're going to step over matching inverse pairs of projected patterns, and
	// perform a threshold operation to determine whether a given pixel is white
	// or not.
	for(int i=0;i<NUM_IMAGES;i+=2){
		printf("Processing scan patterns %d, %d\n",i,i+1);
		uchar* data1 = (uchar*)capImg[i]->imageData;
		uchar* data0 = (uchar*)capImg[i+1]->imageData;

		for(int y=0;y<capImg[i]->height;y++){
			for(int x=0;x<capImg[i]->width;x++) {
				int u = y*capImg[i]->widthStep+x*capImg[i]->nChannels;
				if(columnmap[u] == -1){
					// We have discarded this pixel
					continue;
				}
				int gray1 = data1[u] + data1[u+1] + data1[u+2];
				int gray0 = data0[u] + data0[u+1] + data0[u+2]; 

				// Shift our Gray code bit vector
				columnmap[u] = columnmap[u] << 1;
				if(gray1 - gray0 > WHITE_THRESHOLD){
					// We have a white pixel!
					assert((columnmap[u] & 1) == 0);
					columnmap[u] |= 1;
				} else if(gray1 - gray0 < BLACK_THRESHOLD){
					// A black pixel
					assert((columnmap[u] & 1) == 0);
					columnmap[u] |= 0;
				} else {
					// Not quite sure about this pixel: Punt!
					columnmap[u] = -1;
				}
			}
		}

		// Avoid memory leaks!
// 		printf("Releasing images %d, %d\n", i,i+1);
// 		cvReleaseImage(&capImg[i]);
// 		cvReleaseImage(&capImg[i+1]);
	}

	// Convert Gray code to decimal
	for(i = 0; i < capImg[0]->imageSize; i++){
// 		int tmp = columnmap[i];
		columnmap[i] = gray2bin(columnmap[i]);
// 		printf("Gray: %4x -> Binary %4x\n",tmp,columnmap[i]);
	}

	// TODO: Test against projector patterns -> 0 disparity
	// TODO: Test with SL images

    //Find available filename on jumpdrive
    sprintf(s,"/media/sda1/result%d.xyz",outFileNum);
    while (file_exists(s)) {
        outFileNum++;
        sprintf(s,"/media/sda1/result%d.xyz",outFileNum);
    }

	// Write the pointcloud
	FILE *f = fopen(s,"w");
	if(f == NULL) {
	  printf("Error opening %s:  Trying local directory...\n",s);
	  FILE *f = fopen("result.xyz","w");
	  if(f == NULL) {
            printf("Error opening result.xyz:  Aborting!\n");
            return 1;
	  }
	}

	printf("Writing pointcloud...\n");
	for(int y = 0;y < capImg[0]->height; y++){
		for(int x = 0;x < capImg[0]->width; x++) {
			int u = y*capImg[0]->widthStep+x*capImg[0]->nChannels;

			if(columnmap[u] == -1) continue; // Discarded!

			float disp = (float)columnmap[u]/(PROJ_WIDTH) - (float)x/CAM_WIDTH;
// 			printf("%f\n",disp);
			if(disp == 0.0) continue;
			float z = (float)Z_SCALE/(disp);

			fprintf(f,"%d %d %f\n",x,y,z);
		}
	}

	fclose(f);

Part 3 is a work in progress.  All thats contained here is some cleanup.

	//*******Part3 - Project 3D object with openGL*********

	for(int i = 0; i < NUM_IMAGES; i++){
		cvReleaseImage(&capImg[i]);
		//cvReleaseImage(&slPat[i]);
	}
	//Exit
	cvDestroyWindow("mainWin");
	free(columnmap);
	return 0;
}

<–Previous     Project Home     Next –>


The pushbutton allows the user to start a 3D scan by the push of a button, similar to how a real portable camera works.  I wasnt able to access the beagleboards user button with software, so I used a simple SPST switch and wired one side to GND and the other to GPIO 139.  I mounted the button on the topside of the polycarb housing.  With project deadlines luming, I didn’t add any hardware de-bouncing, but this can be done later in software.

The guys over at ridgerun have a great example on GPIO and interrupts, which I have modified to suit my purposes.  I have posted modifications below:

GPIO Setup:

gpio = 139; // SPST connected to GPIO 139
gpio_export(gpio); //export to user space
gpio_set_dir(gpio, 0); //set direction to input
gpio_set_edge(gpio, “falling”);  // Set interrupt edge, can be rising, falling or both.
gpio_fd = gpio_fd_open(gpio);

Upon interrupt:

printf("Starting Scann");
slScanner(); // run the sl code
printf("Scan Completen");
printf("Press button to scan again or Ctrl C to quit");

<–Previous     Project Home     Next –>


Installing your camera module and ENV.txt

Posted: 26th February 2012 by hackengineer in Computer Vision
Tags: , ,

Setting up the Leopardboard 365 VGA Camera board is pretty straight forward.  Connect it to the beagleboards camera header.  Edit the UENV.TXT file with the following line.  If it does not exist create it in /media/mmcblk0p1/UENV.TXT

  • camera=lbcmvga

We also need to build a jig to mount the lens to the camera.  Since the lens uses a magnetism to adhere a metal washer will do the trick.  A hot-glue gun works great for attaching the washer to the camera board.

<–Previous     Project Home     Next –>


The easiest way I found to get openCV up and running on the beagleboard was to build an image from scatch using the narcissus tool.  It works and it works really well in just a few steps.

  1. Select Beagleboard
  2. Select the Packages you want
    • OpenCV – Libraries will be readily available to include in your code
    • Toolchain – Ability to compile code natively on beagleboard
  3. Build ( this step takes a while so be patient)

I find it useful to select the build type as an SD image under the advanced option.  Once downloaded you can download the image onto the sd card easily with win32DiskImager.  After the image has been loaded, plug the SD card into the beagle board and boot it up.  This step also takes a long while to install but once it boots your all set!

<–Previous     Project Home     Next –>