dcrawl – Web Crawler For Unique Domains
iSpeech.org
dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.
How does dcrawl work?
dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.
dcrawl Web Crawler Features
- Branching out only to predefined number of links found per one hostname.
- Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
- Can be restarted with same list of domains – last saved domains are added to the URL queue.
- Crawls only sites that return text/html Content-Type in HEAD response.
- Retrieves site body of maximum 1MB size.
- Does not save inaccessible domains.
dcrawl Usage
[adsense size='1' ]
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
___ __
__| _/________________ __ _ _| |
/ __ |/ ___\_ __ \__ \\ \/ \/ / |
/ /_/ \ \___| | \// __ \\ /| |__
\____ |\___ >__| (____ /\/\_/ |____/
\/ \/ \/ v.1.0
usage: dcrawl -url URL -out OUTPUT_FILE -t THREADS
-ms int
maximum different subdomains for one domain (def. 10) (default 10)
-mu int
maximum number of links to spider per hostname (def. 5) (default 5)
-out string
output file to save hostnames to
-t int
number of concurrent threads (def. 8) (default 8)
-url string
URL to start scraping from
-v bool
verbose (default false)
|
Example:
|
1
2
|
go build dcrawl.go
./dcrawl -url http://wired.com -out ~/domain_lists/domains1.txt -t 8
|
There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused.
You can download dcrawl go web crawler here:
Or read more here.
Gloss