Comments on Individuation Redux: A study of file duplication in the Debian archive

You might be interested in a similar project calle...

2007-09-08T01:45:35.628Z

You might be interested in a similar project called CDR (checksums done right) that I'm involved with. It's basically tripwire using vendor checksums (but since most debs don't have sha256 we generate them ourselves). We use virtualization to verify a system while it is online.

We currently have support for deb and rpm and we're looking at windows support as well. Ubuntu mainly but it could easily be extended to Debian.

A few stats:

* Architectures i386 and amd64 only
* Centos 4,5 and Ubuntu dapper-gutsy
* 7,033,495 unique files (via sha256)
* 134,134 unique packages

We haven't done much with tracking maintainers but that's a good idea and we'll likely look into some sort of trust system based on the maintainer. More for revocation reasons that actual trust. If you'd like to contribute to collaborate email me (scott at cse ucdavis education).

It would be easy to do, but it requires getting th...

2007-08-22T06:05:36.237Z

It would be easy to do, but it requires getting the sizes from the data tarball, which has a significant CPU cost (it must be uncompressed first).

I like all the statistics, but you left out the on...

2007-08-21T23:04:27.995Z

I like all the statistics, but you left out the one I was most interested in seeing: the amount of wasted space due to duplication, both as an absolute value and as a % of the total archive. Although from your description of the the duplicated files, I'm guessing it will be low.

Well it certainly uses inodes ;)

2007-08-21T21:28:37.188Z

Well it certainly uses inodes ;)