GOV2 Test
Collection
GOV2 is a TREC test collection for use in
the Terabyte Track.
- A crawl of .gov
sites (early 2004)
- A large proportion of the
available (crawlable) pages in .GOV
- Including html and text, plus
the extracted text of pdf, word and postscript.
- Not including binary files.
However pdf, word, postscript, gif, jpg and png have been crawled for NIST’s
use in relevance judging.
- Documents truncated to 256 kb
- Content-based file type
checking gives us accurate mime type information and avoids inclusion of
binaries
- 20 times more documents than
.GOV, and slightly higher average docsize (17.7
kb vs 15.2 kb).
- It was crawled
by Charlie Clarke and Ian Soboroff, using NIST hardware and network, then
formatted by Nick Craswell and distributed
by the University of Glasgow.
Document/corpus
info
|
|
|
25,205,179 |
documents (see below for mime types) |
|
17.7 kb |
average document size |
|
426 |
total gigs |
|
0.42 |
total tb |
Gzipped bundle info
|
|
|
80.7 |
total gigs of gz
bundles |
|
27,204 |
bundles |
|
3.04 |
megs per bundle (16 megs uncompressed) |
|
0.19 |
compression (gzipsize/fullsize) |
True mime
types
|
|
|
23,111,957 |
text/html (91.7%) |
|
2,030,339 |
application/pdf (8.1%) |
|
60,176 |
text/plain (0.2%) |
|
2,253 |
application/msword
(0.009%) |
|
454 |
application/postscript (0.002%) |
|
25,205,179 |
Total documents |
Server-reported
mime types
|
|
|
23,173,646 |
text/html |
|
2,031,571 |
application/pdf |
|
2,134 |
application/msword |
|
882 |
image/gif |
|
558 |
application/postscript |
|
417 |
image/jpeg |
|
5 |
text/html |
|
2 |
application/x-javascript |
|
2 |
text/css |
|
1 |
x-application/pdf |
|
25,209,218 |
Total (slightly wrong!) |
Updated:
2004-05-26
Nick Craswell ( 
)