Web Crawlers' Efficiency: A Comparative Analysis (University Report)

This report provides a comparative analysis of open-source web crawlers, focusing on their efficiency and key features. It begins with an introduction to web crawlers, explaining their functionality and applications in data mining, price comparison, and web page maintenance. The report then details the techniques used in web scraping, emphasizing the importance of bandwidth management, adherence to robot exclusion standards, and efficient disk space utilization. Several open-source crawlers, including Scrapy, Apache Nutch, Heritrix, and others, are examined, with a focus on their respective advantages and disadvantages. The report assesses the features that make a good web crawler, such as robustness, adherence to web server policies, distributed execution capabilities, scalability, and quality of data retrieved. The report compares the efficiency of these crawlers in various scenarios and discusses the challenges faced by crawlers, including scalability issues, content selection problems, and social implications. Finally, the report highlights the importance of selecting the right crawler based on specific project requirements and provides an overview of the challenges and considerations in web crawling.

Running head: WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY
WebCrawler’s and comparison of their efficiency
Name of the Student
Name of the University

1WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY
Introduction
Web crawlers are the software programs that helps the analysts or developers to
browse the web pages on internet in an automated manner. These web crawlers are also
helpful in automated maintenance of the websites in order to check different link to the
internal and external destinations as well as validation of the HTML code of the website
(Farag, Lee and Fox 2018). In addition to that, the crawlers are mostly used in order to gather
specific types of data from the different Web pages of a site. Mostly this information
includes collecting e-mail addresses.
Functionality of the web crawlers
The web crawlers somehow behave like the librarian in real life. It looks for certain
data on the internet and when it is found it is categorized by the crawler. The type of
information needs to be defined by particular predefined instructions.
Main usage of the open source web crawlers
The web crawlers are mainly used for price comparison on different e-commerce
portals search in order to get information about some specific products. In this way the prices
on different websites can be compared precisely in real time. In addition to that, in case of
data mining, a crawler is also capable of collecting publicly available e-mail or other similar
kind of data about the different organizations. Moreover, it is very helpful in order to collect
information about page views for a web page, incoming or outbound links to the pages.
Techniques to use in web scraping
Crawling and scraping Websites and pages requires ability to use the available
network bandwidth, conforming robot exclusion standards or policies of the web sites,
refreshing the web pages logically to get the required data, selection of the high quality and

2WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY
significant pages in order to explore the available data, use of the available disk space in an
efficient manner (Farag, Lee and Fox 2018). In addition to that, the crawler should also
continue working on the pages or websites even if the crawler counters large sized
documents, slow responsive servers, multiple URLs that are leads to the similar documents,
broken links as well as corrupted files.
Furthermore, the crawlers also let the users to invoke and direct for a text search and
get the results accordingly. In addition to that, the hypertext nature is also helpful for getting
the result in an enhanced manner over the text-search engine, through the use of the link-text
indexing and analysis.
Open sources crawlers
Following are the list of open source crawlers that are used widely by the
organizations and individuals;
Scrapy
Apache Nutch
Heritrix
WebSphinix
JSPider
GNUWGet
WIRE
Pavuk
Teleport

3WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY
WebCopierPro
Web2disk
Features for a good web crawler
There are certain features that must be satisfied by any web crawler to be one of the
best crawlers.
Being robust: The web crawlers must be intended to be flexible as well as resilient
against the traps that are used by the different web servers (Agre and Mahajan 2015). This
traps direct the crawlers in fetching the boundless number of pages from a specific domain of
the website. Some such traps are noxious which results in erogenous development of the site.
Maintaining the web server policies: most of the web servers have some policies or
approaches for the crawlers which visit them so as to abstain the crawlers from over-
burdening the websites that leads degradation in performance.
Distributed execution: An efficient crawler ought to be able to be executed in a
distributed manner over various servers or systems (Farag, Lee and Fox 2018).
Scalability of the operation: The architecture of the crawler must allow scaling up the
rate of crawling by including additional computers/systems as well as bandwidth or
transmission capacity to the targeted website.
Execution and productivity: The crawler framework should utilize different
framework resources including processor, system band-width and storage efficiently.
Quality: Quality characterizes how significant the crawled pages are fetched by the
crawlers. Crawler attempts to download the significant pages in the first attempt.
Features of Scrapy

4WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY
This crawler is developed using python. Scrappy comes along with some of the well-
known data export formats that like XML, JSON and CSV. This crawler is comparatively
easy to set up than the other crawler tools. This crawler was intended to extract some
specific information from the pages and not to collect a whole dump of HTML page content
(Farag, Lee and Fox 2018). In contrast to the above mentioned advantages, there are certain
drawbacks of this tool some of them are lack of support to the distributed crawling, difficulty
in exporting the data when the size crosses a certain limit.
Features of Apache Nutch
This open source crawler is highly scalable compared to the Scrappy crawler.
The Nutch is robust as well as scalable and can be executed on a cluster that may
include up to 100 computers.
The Nutch can be configured in such a way that it can be made biased in order to
fetch significant pages of the website first.
As it belongs to the Apache family thus it can be easily integrated with the Hadoop
platform. Besides this it can also integrated with the other parts of the Apache ecosystem.
The crawled data can be stored in distributed key value store technique similar to the
HBase.
Compare to the other tools this tool is Dynamically scalable and is fault-tolerant for
any kind of failure that may interrupt the crawling. Furthermore, it also provides flexible
plugin system to improve the functionality of the application.
The user interface of Nutch is provided on the browser as the Java Server Page. After
the user determines the subject to crawl it parses textual query and sequentially invokes
search method in NutchBean. In case Nutch is executed on the single computer or server,

5WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY
then the NutchBean converts the query in Lucene query in order to get a definite list of hits
from Lucene. After receiving the response which is rendered into HTML. On the other hand,
if the Nutch crawler is deployed in a distributed manner on multiple servers then the
NutchBean's search method remotely raises other search methods from the different
computers. This functionality can configured with different options such as perform search
locally or farm pieces of the crawling on different computers/servers.
Features of Heritrix
The Heritrix is another open source crawler that can be deployed in a distributed
environment through the hashing of targeted URL hosts on multiple systems. From the
distributed deployment it can be defined considered as scalable, but this is not dynamically
scalable for accommodating the increased data capture load (Farag, Lee and Fox 2018). Thus
the user has to decide the number of systems on which the crawler will be deployed before
commencing the crawling. If one of the machines goes down during your crawl you are out
of luck.
The output of the crawled web pages is stored in WARC (web archive) format file in
the local file system. This format is considered as the best format in order to write and store
multiple resources in the HTML format (Agre and Mahajan 2015). Moreover, after the
analysis the metadata about the collected data is stored in an archive file. Another feature is
secure user console. This crawler provides secure user control console for controlling the
complete interaction. The HTTPS protocol is utilized in order to access as well as manipulate
the console.
Pavuk web crawler
This tool is very helpful in providing detailed timing information for the fetched data
from the web pages.

6WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY
This tool can be utilized as the full featured FTP mirroring tool, as this tool is able to
preserve the permissions, modification time as well as symbolic links.
Users can also control the transfer speed through limiting the maximum or minimum
data speed.
In addition to that this tools is also helpful in optional multithreading, uses HTTP
proxies through the utilization of the round-robin scheduling.
Furthermore, it also supports NTLM authorization and support to the JavaScript
binding. Due to the support to the scripting it helps in completion of particular tasks.
WebSphinix
This tool is helpful in visualization of the collection of web pages that are crawled in
graphical format or charts. Similar to the other above mentioned tool this also supports
saving the pages on the local disk storage in order to browse offline (Farag, Lee and Fox
2018). Furthermore, it also helps in concatenation of the crawled pages together in order
view or print them as one document. WebSphinix also helpful in the extraction of text that
matches a certain provided pattern from different pages of a website.
Comparison of efficiency of the crawlers
The use of the different crawlers depends on the requirement of the scenario. For
some basic requirement a user can utilize Scrapy as it is faster. On the contrary, the Scarpy is
not scalable when compared to Heritrix and Nutch. Scrapy crawler is also an exceptional
choice in case of focused crawls on targeted website.
On the other hand, even though Heritrix is scalable but this cannot be not done in a
dynamic way. This tool is very useful in distributed environment when deployed on the
cluster of computers. The lack of dynamic nature of other crawlers is mitigated by the Nutch

7WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY
which belongs to apache family (Agre and Mahajan 2015). The Nutch is scalable as well as
the user can be achieved dynamically through the use of Hadoop. Another crawler WIRE is
also very useful in terms of scalability. WebCopier, Heritrix gives the best result in terms of
proposed scale. Various studies and experiment mentioned in this research paper highlights
the various facts about different crawlers which will help the users to select crawler which
will satisfy their needs.
Challenges faced by the crawlers
Scalability: The web is huge in size and it is continuously evolving with time. The crawlers
that requires great coverage as well as fresh crawled data then it have to achieve
tremendously high rate of throughput that leads to different difficult development and
compatibility issues (Farag, Lee and Fox 2018). The reason behind this can be stated as
search engine in the modern day’s deploys huge number of servers as well as multiple
number of high-speed network links.
Issues in Content Selection-It is found that the crawlers that has the highest-throughput are
even not able manage the implication that are faced while doing the same while crawling the
whole web and not even able keep up with changes on the targeted page. Thus crawling of
the web is carried out selectively as well as in a controlled order.
The main objective of this technique is to get the high-value and significant content faster
while ensuring subsequent coverage of the reasonable content as well as avoid the low-
quality, redundant, irrelevant content available on the web (Agre and Mahajan 2015).
Therefore, it is important that the used crawler must balance the objectives while crawling
that includes complete coverage as well as freshness of the content. Along with that, the
crawler also needs to follow the constraints like per-site rate limitations for retrieving the

8WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY
data. In addition to that, balance must be met between exploration and retrieval of the
potentially useful content and content that are already known to be significant.
Social issues due to the crawling: The crawlers need to be responsible while retrieving data
from the web as imposing too much burden on a web site may harm the performance
adversely (Farag, Lee and Fox 2018). It can be stated here that without having right
mechanisms in place any high-throughput crawler is capable of carrying out a denial-of-
service attack on the targeted website.
Conclusion
The web crawlers are in use from the time the web is widely used by the worldwide users in
order to provide relevant data to the users according to the user’s requirement and search
string. This can be considered as a method in order to collect data as well as and keeping up
newly available data in the rapidly growing Internet. This can be also represented as graph
search problem if the web/internet considered as large graph in which the webpages nodes
and edges are hyperlinks among them. Here it can be also stated that, content providers inject
misleading content in the corpus of the website that is often driven by financial reasons such
as directing web traffic to some specific web sites which needs to be filtered by the crawlers.

9WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY
References
Agre, G.H. and Mahajan, N.V., 2015, February. Keyword focused web crawler. In 2015 2nd
International Conference on Electronics and Communication Systems (Icecs) (pp. 1089-
1092). IEEE.
Du, Y., Liu, W., Lv, X. and Peng, G., 2015. An improved focused crawler based on semantic
similarity vector space model. Applied Soft Computing, 36, pp.392-407.
Farag, M.M., Lee, S. and Fox, E.A., 2018. Focused crawler for events. International Journal
on Digital Libraries, 19(1), pp.3-19.
Fetzer, C., Felber, P., Rivière, É., Schiavoni, V. and Sutra, P., 2015, June. Unicrawl: A
practical geographically distributed web crawler. In 2015 IEEE 8th International Conference
on Cloud Computing (pp. 389-396). IEEE.
Gupta, A. and Anand, P., 2015, February. Focused web crawlers and its approaches. In 2015
International Conference on Futuristic Trends on Computational Analysis and Knowledge
Management (ABLAZE) (pp. 619-622). IEEE.
Seyfi, A., Patel, A. and Júnior, J.C., 2016. Empirical evaluation of the link and content-based
focused Treasure-Crawler. Computer Standards & Interfaces, 44, pp.54-62.
Tsai, C.H., Ku, T. and Chien, W.F., 2015. Object Architected Design and Efficient Dynamic
Adjustment Mechanism of Distributed Web Crawlers. International Journal of
Interdisciplinary Telecommunications and Networking (IJITN), 7(1), pp.57-71.
Zhao, F., Zhou, J., Nie, C., Huang, H. and Jin, H., 2015. SmartCrawler: a two-stage crawler
for efficiently harvesting deep-web interfaces. IEEE transactions on services
computing, 9(4), pp.608-620.

10WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY

11WEBCRAWLER’S AND COMPARISON OF THEIR EFFICIENCY