Essay on The Crawling Module and Web Pages

The first versions of WWW ((what most people call “The Web”))) provide means for people around the world to exchange information between, to work together, to communicate, and to share documentation more efficiently. Tim Berners-Lee wrote the first browser (called WWW browser) and Web server in March 1991, allowing hypertext documents to be stored, fetched, and viewed. The Web can be seen as a tremendous document store where these documents (web pages) can be fetched by typing their address into a web browser. To do that, two im- portant techniques have been developed. First, a language called Hypertext Markup Languag (HTML) tells the computers how to display documents which contain texts, photos, sounds, visuals (video), and animation, interactive

For this assignment, I was allowed to improvise on a provided base code to develop a functioning web crawler. The web crawler needed to accept a starting URL and then develop a URL frontier queue of “out links” to be further explored. The crawler needed to track the number of URLs and stop adding them once the queue had reached 500 links. The crawler needed to also extract text and remove HTML tags and formatting. The assignment instructions offered using the BeautifulSoup module to achieve those goals, which I chose to do. Finally, the web crawler program needed to report metrics including the number of documents (web pages), the number of tokens extracted and processed, and the number of unique terms added to the term dictionary.

In these paper author focus on finding the gap in understanding how complex individual Web sites are and how this complexity impacts on the usersperformance. Also characterize the Web site both at content level (like, number and size of images) and service level (like, number of servers/origins). It may happen that some categories are more complex than other such as 'News '. Out of hundred 60% of Web sites fetched content from minimum five non-origin sources, and these give more than 35% of the bytes downloaded. In addition, they examine which metrics are most suitable for predicting page render and load times and catch that the number of objects requested is the most important factor. With respect to variability in load times, however, they alsofind number of servers is the best indicator.

A Web crawler, sometimes called a spider, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing. Crawlers can validate hyperlinks and HTML code. They can also be used for web scraping. Crawlers consume resources on the systems they visit and often visit sites without approval. Issues of schedule, load, and "politeness" come into play when large collections of pages are accessed. Mechanisms exist for public sites not wishing to be crawled to make this known to the crawling

Crawler must avoid the overloading of Web sites or network links while doing its task. Unless it has unlimited computing resources and unlimited time, it must carefully decide what URLs to scan and in what order as it deals with huge volumes of data .Crawler must decide how frequently to revisit pages it has already seen, in order to keep its client informed of changes on the

(King-Lup Liu, 2001) Given countless motors on the Internet, it is troublesome for a man to figure out which web search tools could serve his/her data needs. A typical arrangement is to build a metasearch motor on top of the web indexes. After accepting a client question, the metasearch motor sends it to those fundamental web indexes which are liable to give back the craved archives for the inquiry. The determination calculation utilized by a metasearch motor to figure out if a web index ought to be sent the inquiry ordinarily settles on the choice in light of the web search tool agent, which contains trademark data about the database of a web search tool. Be that as it may, a hidden web index may not will to give the required data to the metasearch motor. This paper demonstrates that the required data can be evaluated from an uncooperative web crawler with great exactness. Two bits of data which license precise web crawler determination are the quantity of reports filed by the web index and the greatest weight of every term. In this paper, we display systems for the estimation of these two bits of data.

One of the most important programming languages for building a search engine is HTML, or Hypertext Markup Language. Hypertext Markup Language is the programming language used to make generally every webpage. It is used to create text boxes, hyperlinks, images, et cetera. Sometimes another markup language, PHP, or Hypertext Preprocessor, is used which has the benefit of also being a server-side scripting language.

Nevertheless, it has obtained gigantic awareness best in the up to date years [41-58, 60-64]. Targeted crawlers avoid the crawling method on a certain set of issues that characterize a narrow area of the online. A focused or a topical internet crawler makes an attempt to download websites critical to a suite of pre-outlined subject matters. Hyperlink context varieties and most important part of web headquartered understanding retrieval assignment. Topical crawlers follow the hyperlinked constitution of the online making use of the supply of understanding to direct themselves towards topically relevant pages. For deriving the proper expertise, they mine the contents of pages which are already fetched to prioritize the fetching of unvisited pages. Topical crawlers depend especially on contextual understanding. This is considering that topical crawlers need to predict the advantage of downloading unvisited pages based on the understanding derived from pages that have been downloaded. One of the vital fashioned predictors is the anchor textual content of the hyperlinks [59]. The area targeted search engines like google and yahoo use these targeted crawlers to download selected

When the Internet and World Wide Web were first created, they were designed a research tools and for the distribution of information through information systems networks. But as the use of the Web has become increasingly more complex, the focus on Web pages and their design has initiated a number of major changes. Initially, static Web pages were common, but the focus in recent years has been on the development of dynamic Web pages which are linked to databases and allow for the integration of information on a number of different levels.

URL stands for “Uniform Resource Locator”. A URL is a formatted text string used by Web browsers, email clients and other software to identify a network resource on the Internet. Network resources are files that can be plain Web pages, other text documents, graphics, or programs. URL is the unique address for a file that is accessible on the Internet. A common way to get to a Web site is to enter the URL of its home page file in your Web browser 's address line. However, any file within that Web site can also be specified with a URL. Such a file might be any Web page other than the home page, an image file, or a program such as a common gateway interface application or Java applet. The URL contains the name of the protocol to be used to access the file resource, a domain name that identifies a specific computer on the Internet, and a pathname, a hierarchical description that specifies the location of a file in that computer.

Everyone who used the web undoubtedly have seen URL as a sight word in the internet world, and also have used URLs to get web pages and access websites. Actually, most of people term URL as “website address” habitually and think of an URL as the name of a file on the World Wide Web. If we consider the web world as the same as the real world, then URL would be the very unique physical address of every build on the earth that helps people to locate the accurate place. However, it is not the entire understanding of URL. URLs could also lead to other resources on the web, such as databases queries and command output.

Web servers are characterized mainly by low CPU utilization with spikes during peak periods, with disk performance as a consideration if the website is delivering dynamic content (Advanced Micro Devices, 2008). Traditional web servers only delivered static HTML pages, or pages that had no interactive or data-input elements – merely a send and read operation. Dynamic websites may utilize forms and databases, which would be an additional consideration to a high traffic website.

In this present web-savvy era, URL is a genuinely basic abbreviation which is broadly utilized as a word as a part of itself, without much thought for what it really remains for or what it is included. In this paper, the fundamental ideas of URLs and internet Cookies are discussed about with spotlight on its significance in Analytics perspective.

The Internet archive preserves the live web by saving snapshots of the websites made with a specific date which can be browsed or searched for various reasons. Its object is to save the whole web without being in favor of a specific lan- guage, domain or geographical location. The importance of archiving made it important to check its coverage. In this paper, we try to determine how well Arabic websites are archived and indexed, and if the number of archived and indexed websites is affected by by country code top level domain, geographic location, creation date and depth. We also crawled for Arabic hyperlinks and checked its archiving and indexing.

Internet archiving preserves the live web by saving snap- shots of the websites made with a specific date which can be browsed or searched for various reasons. Its object is to save the whole web without being in favor of a specific language, domain or geographical location. The importance of archiving made it important to check its coverage. In this paper, we try to determine how well Arabic websites are archived and indexed, and if the number of archived and indexed websites is affected by by country code top level domain, geographic location, creation date and depth. We also crawled for Arabic hyperlinks and checked its archiving and indexing.