[Dar-discussions] Differential backup based on catalogue file

[T o n g]

Hi,

I want to do a differential backup based on a given catalogue
file. I think it should work, but test shows otherwise [1]:

I.e., I see that a file is already backed up, and show up in the
catalogue file, but dar still wants to back it up, even everything
seems to be the same.

What's happening? How can I do further investigation?

Thanks

[1]

$ dar -l da-name_2009-12-11-cat | head -15
[data ][ EA ][compr] | permission | user | group | size | date | filename
----------------------+------------+-------+-------+-------+-------------------------------+------------
[InRef] [-----] drwxrwxr-x 9999 tong 0 Fri Jan 4 13:46:39 2008 db
[InRef] [-----] drwxrwxr-x tong tong 0 Fri Jan 4 13:46:39 2008 db/dict
[InRef] [-----] drwxrwxr-x tong tong 0 Fri Jan 4 13:46:39 2008 dev
[InRef] [-----] drwxrwxr-x tong tong 0 Fri Jan 4 13:46:39 2008 fun
[InRef] [-----] drwxrwxr-x tong tong 0 Fri Jan 4 13:46:39 2008 fun/games
[InRef] [-----] drwxrwxr-x tong tong 0 Fri Jan 4 13:46:39 2008 inet
[InRef] [-----] drwxrwxr-x tong tong 0 Fri Jan 4 13:46:39 2008 inet/lan
[InRef] [-----] drwxrwxr-x tong tong 0 Fri Jan 4 13:47:17 2008 inet/web
[InRef] [-----] -rwxrwxr-x tong tong 8510 Fri Jan 4 13:47:05 2008 inet/web/larve.2006.add-to-social-bookmarking-services.htm
[InRef] [-----] -rwxrwxr-x tong tong 12385 Fri Jan 4 00:00:00 2008 inet/web/sb.zip
[InRef] [-----] -rwxrwxr-x tong tong 199 Fri Jan 4 13:47:17 2008 inet/web/11getlog.lst
[InRef] [-----] drwxrwxr-x tong tong 0 Fri Jan 4 13:46:39 2008 inet/agent
[InRef] [-----] drwxrwxr-x tong tong 0 Fri Jan 4 13:46:39 2008 inet/email


$ dar -e -v -R . -c /path/to/da-name_2010-09-05-diff -A da-name_2009-12-11-cat -s 10M
Extracting contents of the archive...
Adding file to archive: /path/to/pwd/db
Adding file to archive: /path/to/pwd/db/dict
Adding file to archive: /path/to/pwd/dev
Adding file to archive: /path/to/pwd/fun
Adding file to archive: /path/to/pwd/fun/games
Adding file to archive: /path/to/pwd/inet
Adding file to archive: /path/to/pwd/inet/lan
Adding file to archive: /path/to/pwd/inet/web
Adding file to archive: /path/to/pwd/inet/web/larve.2006.add-to-social-bookmarking-services.htm
Adding file to archive: /path/to/pwd/inet/web/sb.zip
Adding file to archive: /path/to/pwd/inet/web/11getlog.lst
Adding file to archive: /path/to/pwd/inet/agent
Adding file to archive: /path/to/pwd/inet/email
[ . . . ]

[T o n g]

Post by T o n g
I.e., I see that a file is already backed up, and show up in the
catalogue file, but dar still wants to back it up, even everything seems
to be the same.

Oh,

- The catalogue file's original name is different. It was not da-
name_2009-12-11-cat, which is a symlink pointing to the original name.

- The root backup directory might be different. I might have moved it,
but not sure.

Still, I don't think the above 2 things should have negative impact on dar
differential backup, but just making the story complete.

[T o n g]

Post by T o n g
[InRef] [-----] -rwxrwxr-x tong tong 8510 Fri Jan 4 13:47:05 2008 inet/web/larve.2006.add-to-social-bookmarking-services.htm
[InRef] [-----] -rwxrwxr-x tong tong 12385 Fri Jan 4 00:00:00 2008 inet/web/sb.zip
[InRef] [-----] -rwxrwxr-x tong tong 199 Fri Jan 4 13:47:17 2008 inet/web/11getlog.lst

$ ls -l inet/web/larve.*.htm inet/web/sb.zip inet/web/11getlog.lst
-rwxrwxr-x 1 tong tong 199 2008-01-04 09:47 inet/web/11getlog.lst
-rwxrwxr-x 1 tong tong 8510 2008-01-04 09:47 inet/web/larve.2006.add-to-social-bookmarking-services.htm
-rwxrwxr-x 1 tong tong 12385 2008-01-03 20:00 inet/web/sb.zip

But having add '-H6' (ideally -H4 should be enough) to the dar
command line, dar only considers the 'sb.zip' file to be backed
up before, and still add the other 2 files. Why is that?

Thanks

[Denis Corbin]

Hello Tong,

there is an easy way to know why dar saved a file for differential
backup, it is to use the archive of reference and make a diff (dar -d)
against the fileystem you want to backup. For each file that differs
between archive and fileystem, dar will output what causes the difference.

In the hope it will help you understand,
Regards,
Denis.

Post by T o n g

Post by T o n g
[InRef] [-----] -rwxrwxr-x tong tong 8510 Fri Jan 4 13:47:05 2008 inet/web/larve.2006.add-to-social-bookmarking-services.htm
[InRef] [-----] -rwxrwxr-x tong tong 12385 Fri Jan 4 00:00:00 2008 inet/web/sb.zip
[InRef] [-----] -rwxrwxr-x tong tong 199 Fri Jan 4 13:47:17 2008 inet/web/11getlog.lst

$ ls -l inet/web/larve.*.htm inet/web/sb.zip inet/web/11getlog.lst
-rwxrwxr-x 1 tong tong 199 2008-01-04 09:47 inet/web/11getlog.lst
-rwxrwxr-x 1 tong tong 8510 2008-01-04 09:47 inet/web/larve.2006.add-to-social-bookmarking-services.htm
-rwxrwxr-x 1 tong tong 12385 2008-01-03 20:00 inet/web/sb.zip
But having add '-H6' (ideally -H4 should be enough) to the dar
command line, dar only considers the 'sb.zip' file to be backed
up before, and still add the other 2 files. Why is that?
Thanks

[T o n g]

Post by Denis Corbin
there is an easy way to know why dar saved a file for differential
backup, it is to use the archive of reference and make a diff (dar -d)
against the fileystem you want to backup.

Thanks for your answer Denis.

Sorry to have picked the wrong mlist,
gmane.comp.sysutils.backup.dar.general
instead of gmane.comp.sysutils.backup.dar.support.

Will follow up there.

Thanks again.

[T o n g]

Hi Denis,

I decided to still follow up here because it is more a bug report now.

To recap all related info first:

$ dar -l /path/to/da-name_2009-12-11-cat | head -15
[ . . . ]
[InRef] [-----] -rwxrwxr-x tong tong 8510 Fri Jan 4 13:47:05 2008 inet/web/larve.2006.add-to-social-bookmarking-services.htm
[InRef] [-----] -rwxrwxr-x tong tong 12385 Fri Jan 4 00:00:00 2008 inet/web/sb.zip
[InRef] [-----] -rwxrwxr-x tong tong 199 Fri Jan 4 13:47:17 2008 inet/web/11getlog.lst
[ . . . ]

$ ls -l inet/web/larve.*.htm inet/web/sb.zip inet/web/11getlog.lst
-rwxrwxr-x 1 tong tong 199 2008-01-04 09:47 inet/web/11getlog.lst
-rwxrwxr-x 1 tong tong 8510 2008-01-04 09:47 inet/web/larve.2006.add-to-social-bookmarking-services.htm
-rwxrwxr-x 1 tong tong 12385 2008-01-03 20:00 inet/web/sb.zip

I.e., all files are time-shifted by exactly 4 hours. Here's what happened:

$ dar -e -v -R /path/to/d_arch -c /path/to/da-name_2010-09-05-diff -A /path/to/da-name_2009-12-11-cat -s 10M
Extracting contents of the archive...
Adding file to archive: /path/to/pwd/db
Adding file to archive: /path/to/pwd/db/dict
Adding file to archive: /path/to/pwd/dev
Adding file to archive: /path/to/pwd/fun
Adding file to archive: /path/to/pwd/fun/games
Adding file to archive: /path/to/pwd/inet
Adding file to archive: /path/to/pwd/inet/lan
Adding file to archive: /path/to/pwd/inet/web
Adding file to archive: /path/to/pwd/inet/web/larve.2006.add-to-social-bookmarking-services.htm
Adding file to archive: /path/to/pwd/inet/web/sb.zip
Adding file to archive: /path/to/pwd/inet/web/11getlog.lst
Adding file to archive: /path/to/pwd/inet/agent
Adding file to archive: /path/to/pwd/inet/email
[ . . . ]

I.e., all files are put again into the differential backup.

$ dar -e -v -R /path/to/d_arch -c /path/to/da-name_2010-09-05-diff -A /path/to/da-name_2009-12-11-cat -s 10M -H5
Extracting contents of the archive...
[ . . . ]
Adding file to archive: /path/to/pwd/inet/web
Adding file to archive: /path/to/pwd/inet/web/larve.2006.add-to-social-bookmarking-services.htm
Adding file to archive: /path/to/pwd/inet/web/11getlog.lst
Adding file to archive: /path/to/pwd/inet/agent
Adding file to archive: /path/to/pwd/inet/email
[ . . . ]

I.e., having add '-H5'/'-H6' (ideally -H4 should be enough) to the dar
command line, dar only considers the 'sb.zip' file to be backed
up before, and still add the other 2 files.

Post by T o n g
What's happening? How can I do further investigation?

,-----
| there is an easy way to know why dar saved a file for differential
| backup, it is to use the archive of reference and make a diff (dar -d)
| against the fileystem you want to backup. For each file that differs
| between archive and fileystem, dar will output what causes the difference.
`-----

OK, here it goes:

$ /usr/bin/dar -v -R /path/to/d_arch -d /path/to/da-name_2009-12-11-cat
[ . . . ]
OK /path/to/d_arch/inet/lan
OK /path/to/d_arch/inet/web
OK /path/to/d_arch/inet/web/larve.2006.add-to-social-bookmarking-services.htm
OK /path/to/d_arch/inet/web/sb.zip
OK /path/to/d_arch/inet/web/11getlog.lst
OK /path/to/d_arch/inet/agent
OK /path/to/d_arch/inet/email
[ . . . ]

I.e., 'dar -d' thinks and reports that all 3 files are OK, and
shouldn't be put into the differential backup, even the -Hx
parameter is NOT given. There is sure some inconsistency here.

FYI, I've also executed the actual backup command without the '-e'
parameter, and the result is same. I.e., the behaviour is
consistent w/ or w/o '-e'.

Please investigate.

PS. I don't know how easily for you to duplicate a time-zone
shifting scenario (ie, whether changing the time-zone will have immediate
affect without rebooting), but this is how it happened to me.
http://thread.gmane.org/gmane.linux.debian.user/386125/focus=386199
Long story short, the files are on vfat partitions, you get
different file time when flipping BIOS CMOS clock between UTC and
localtime. Again, hope there is an easier way for you to
duplicate a time-zone shifting scenario.

Thanks

[Denis Corbin]

Post by T o n g
Hi Denis,

Hi Tong,

Post by T o n g
I decided to still follow up here because it is more a bug report now.

Well, this is not a bug mailing-list, you could have followed to
dar-support or open a bug in the bug tracker for dar at sourceforge.

Anyway, I have open a bug report at
https://sourceforge.net/tracker/?func=detail&aid=3064839&group_id=65612&atid=511612
I ask you to follow up there. But also read below.

Post by T o n g
$ dar -l /path/to/da-name_2009-12-11-cat | head -15
[ . . . ]
[InRef] [-----] -rwxrwxr-x tong tong 8510 Fri Jan 4 13:47:05 2008 inet/web/larve.2006.add-to-social-bookmarking-services.htm
[InRef] [-----] -rwxrwxr-x tong tong 12385 Fri Jan 4 00:00:00 2008 inet/web/sb.zip
[InRef] [-----] -rwxrwxr-x tong tong 199 Fri Jan 4 13:47:17 2008 inet/web/11getlog.lst
[ . . . ]
$ ls -l inet/web/larve.*.htm inet/web/sb.zip inet/web/11getlog.lst
-rwxrwxr-x 1 tong tong 199 2008-01-04 09:47 inet/web/11getlog.lst
-rwxrwxr-x 1 tong tong 8510 2008-01-04 09:47 inet/web/larve.2006.add-to-social-bookmarking-services.htm
-rwxrwxr-x 1 tong tong 12385 2008-01-03 20:00 inet/web/sb.zip
$ dar -e -v -R /path/to/d_arch -c /path/to/da-name_2010-09-05-diff -A /path/to/da-name_2009-12-11-cat -s 10M
Extracting contents of the archive...
Adding file to archive: /path/to/pwd/db
Adding file to archive: /path/to/pwd/db/dict
Adding file to archive: /path/to/pwd/dev
Adding file to archive: /path/to/pwd/fun
Adding file to archive: /path/to/pwd/fun/games
Adding file to archive: /path/to/pwd/inet
Adding file to archive: /path/to/pwd/inet/lan
Adding file to archive: /path/to/pwd/inet/web
Adding file to archive: /path/to/pwd/inet/web/larve.2006.add-to-social-bookmarking-services.htm
Adding file to archive: /path/to/pwd/inet/web/sb.zip
Adding file to archive: /path/to/pwd/inet/web/11getlog.lst
Adding file to archive: /path/to/pwd/inet/agent
Adding file to archive: /path/to/pwd/inet/email
[ . . . ]
I.e., all files are put again into the differential backup.
$ dar -e -v -R /path/to/d_arch -c /path/to/da-name_2010-09-05-diff -A /path/to/da-name_2009-12-11-cat -s 10M -H5
Extracting contents of the archive...
[ . . . ]
Adding file to archive: /path/to/pwd/inet/web
Adding file to archive: /path/to/pwd/inet/web/larve.2006.add-to-social-bookmarking-services.htm
Adding file to archive: /path/to/pwd/inet/web/11getlog.lst
Adding file to archive: /path/to/pwd/inet/agent
Adding file to archive: /path/to/pwd/inet/email
[ . . . ]
I.e., having add '-H5'/'-H6' (ideally -H4 should be enough) to the dar
command line, dar only considers the 'sb.zip' file to be backed
up before, and still add the other 2 files.

Post by T o n g
What's happening? How can I do further investigation?

,-----
| there is an easy way to know why dar saved a file for differential
| backup, it is to use the archive of reference and make a diff (dar -d)
| against the fileystem you want to backup. For each file that differs
| between archive and fileystem, dar will output what causes the difference.
`-----
$ /usr/bin/dar -v -R /path/to/d_arch -d /path/to/da-name_2009-12-11-cat
[ . . . ]
OK /path/to/d_arch/inet/lan
OK /path/to/d_arch/inet/web
OK /path/to/d_arch/inet/web/larve.2006.add-to-social-bookmarking-services.htm
OK /path/to/d_arch/inet/web/sb.zip
OK /path/to/d_arch/inet/web/11getlog.lst
OK /path/to/d_arch/inet/agent
OK /path/to/d_arch/inet/email
[ . . . ]
I.e., 'dar -d' thinks and reports that all 3 files are OK, and
shouldn't be put into the differential backup, even the -Hx
parameter is NOT given. There is sure some inconsistency here.

Well, I was a bit wrong, the archive difference does not show all the
differences that dar can see at differential backup time. After having
review the code, -d option, do compare file type, user and group
ownership, permission, file size when applicable and date of EA if
present. But it does not compare last access time nor last modification
time. This is also for that reason that -H option is useless with -d
option as you might have noticed the following warning:

-H is only useful with -c or -x

where from the bug I have openned. It should let us better see what is
the reason that lead dar to save the files during the differential backup.

Regards,
Denis.

Post by T o n g
FYI, I've also executed the actual backup command without the '-e'
parameter, and the result is same. I.e., the behaviour is
consistent w/ or w/o '-e'.
Please investigate.
PS. I don't know how easily for you to duplicate a time-zone
shifting scenario (ie, whether changing the time-zone will have immediate
affect without rebooting), but this is how it happened to me.
http://thread.gmane.org/gmane.linux.debian.user/386125/focus=386199
Long story short, the files are on vfat partitions, you get
different file time when flipping BIOS CMOS clock between UTC and
localtime. Again, hope there is an easier way for you to
duplicate a time-zone shifting scenario.
Thanks