TREC Blog Track: Crawling Current Status

We are currently fetching blogs and news feeds on the WWW to facilitate the TREC Blog Track. Our fetch rate is once every 2 seconds per crawler per unique host IP Address (1 second for very large blog hosts). The '/robots.txt' file is collected once every 25 days. Feeds are polled at least once every week.

We will are collecting:

  • Feeds (RSS or Atom), including comment feeds
  • Blog homepages
  • Blog permalinks

    If you have any question or concerns about this activity, please feel free to email me craigm{@at@}dcs.gla.ac.NOSPAM.uk