The .GOV test collection

Modified: 2 April 2002

.GOV is a TREC test collection.

• A crawl of .gov Web sites (early 2002).
• Stopped after 1 million text/html pages
• Also including text/plain and the extracted text of pdf, doc and ps.
• Supplied with duplicate tables (URL==URL) and redirect (URL->URL) tables, in case these are useful e.g. for link-based ranking.
• Documents truncated to 100k (reducing size from 35G to 18G)
• Strict file type checking (few or no binaries)
• Fewer documents than wt10g, but larger average docsize
• Thanks to Charlie Clarke for the crawl and Ed Fox for hosting the crawl machine.

Stats:

Documents:	1247753 (1.25 million)
text/html	1053372 (mime types reported by server)
application/pdf	131333
text/plain	43754
application/msword	13842
application/postscript	5673
other stuff which turned out to be text	44
Bundles:	4613
Total size:	19455030550 = 18.1G (without 100k limit was 35.3G)
Average bunsize:	4217435 = 4.0M
Average docsize:	15592 = 15.2k (higher due to pdf+word+ps I think)
Doc truncation length:	100kb (turned out to be "roughly 100k")
Docs without words:	55 (e.g. due to pdftotext producing much text followed by truncation)

Distribution information: http://ir.dcs.gla.ac.uk/test_collections/access_to_data.html