CS 121 FINAL STUDY GUIDE
Universal Resource Locators (URL) - Answers - - URL (locator) vs URN (name)
- Locator must specify where the resource is
Anatomy of URL - Answers - - Web pages are stored on web servers that use HTTP to
exchange information with client software
URL: General Syntax - Answers - -
scheme://domain:port/path?query_string#fragment_id
URL Syntax: Scheme - Answers - - http(s), ftp, mailto, file, data, ...
URL Syntax: Domain:port - Answers - - Authority part. Domain = host
URL Syntax: Path - Answers - - map to a file system path
URL Syntax: Query_String - Answers - - sequence of attribute-value pairs separated by
a delimiter
URL Syntax: Fragment_id - Answers - - Provides direction to a secondary resource or
element
General Web Page Retrieval Process - Answers - 1. Web Crawler client program
connects to a DNS server
2. DNS server translates hostname into internet protocol (IP) address
3. Crawler attempts to connect to server host using specific port
4. Crawler sends an HTTP request to the web server to request a page (usually a GET
request)
Different flavors of data collection - Answers - 1. Data Dumps
2. URL Downloads
3. Web APIs
4. Web Crawling
Web Data Collection: Data Dumps - Answers - - Sites may package their data
periodically and provide it as a "dump"
Web Data Collection: URL Downloads - Answers - - Two Step process:
- Crawl to find out the URLs of specific resources
- Run a downloaded that takes that list and downloads the resources
Web Data Collection: Web APIs - Answers - - Sites may provided REST interface for
getting at their data
- Usually higher-level: avoids having to parse HTML
, - Usually restrictive: only part of the data
Web Data Collection: Web Crawling - Answers - - Like people, getting HTML pages and
other documents and discovering new URLs as it goes
- Web admins don't like crawlers
Web Crawling: Algorithm - Answers - 1. Initialize a queue of URLs (seeds)
2. Repeat until no more URLs in queue:
-a. Get one URL From the queue
-b. If the page can be crawled, fetch associated page
-c. Store Representation of page
-d. Parse and extract URLs from page and add them to the queue
- Queue = "Frontier"
Web Crawling: Freshness - Answers - - Freshness referes to the proportion of pages
that are fresh
- Not possible to constantly check all pages
- Optimizing for Freshness vs. Relevance can lead to bad decision, such as not crawling
popular sites
Web Crawling: Sitemaps - Answers - - Sitemaps are xml files hosted on the web
- Allow web masters to send info to crawlers
Web Crawling Challenges - Answers - - Performance
- Politeness
- Crawler Traps
- Duplicate Detection
- Data Noise
- Content Missed
Web Crawling Challenges: Performance - Answers - - Back of the envelope calculation
- 1 Page Fetch = 500 ms
- 1 DNS lookup = 800 ms
- Crawler thread waits for responses to request
- Solution: multi-threaded or distributed (servers) crawling
Performance Challenges: Distributed Crawling - Answers - - Helps to put the crawler
closer to the sites it crawls
- Reduces the number of sites the crawler has to remember
- Reduces computing Resources required
Performance Challenges: Focused Crawling - Answers - - Focus on pages about a
particular topic (vertical search)
- Rely on the fact that pages about a topic tent to have links to other pages on the same
topic
Universal Resource Locators (URL) - Answers - - URL (locator) vs URN (name)
- Locator must specify where the resource is
Anatomy of URL - Answers - - Web pages are stored on web servers that use HTTP to
exchange information with client software
URL: General Syntax - Answers - -
scheme://domain:port/path?query_string#fragment_id
URL Syntax: Scheme - Answers - - http(s), ftp, mailto, file, data, ...
URL Syntax: Domain:port - Answers - - Authority part. Domain = host
URL Syntax: Path - Answers - - map to a file system path
URL Syntax: Query_String - Answers - - sequence of attribute-value pairs separated by
a delimiter
URL Syntax: Fragment_id - Answers - - Provides direction to a secondary resource or
element
General Web Page Retrieval Process - Answers - 1. Web Crawler client program
connects to a DNS server
2. DNS server translates hostname into internet protocol (IP) address
3. Crawler attempts to connect to server host using specific port
4. Crawler sends an HTTP request to the web server to request a page (usually a GET
request)
Different flavors of data collection - Answers - 1. Data Dumps
2. URL Downloads
3. Web APIs
4. Web Crawling
Web Data Collection: Data Dumps - Answers - - Sites may package their data
periodically and provide it as a "dump"
Web Data Collection: URL Downloads - Answers - - Two Step process:
- Crawl to find out the URLs of specific resources
- Run a downloaded that takes that list and downloads the resources
Web Data Collection: Web APIs - Answers - - Sites may provided REST interface for
getting at their data
- Usually higher-level: avoids having to parse HTML
, - Usually restrictive: only part of the data
Web Data Collection: Web Crawling - Answers - - Like people, getting HTML pages and
other documents and discovering new URLs as it goes
- Web admins don't like crawlers
Web Crawling: Algorithm - Answers - 1. Initialize a queue of URLs (seeds)
2. Repeat until no more URLs in queue:
-a. Get one URL From the queue
-b. If the page can be crawled, fetch associated page
-c. Store Representation of page
-d. Parse and extract URLs from page and add them to the queue
- Queue = "Frontier"
Web Crawling: Freshness - Answers - - Freshness referes to the proportion of pages
that are fresh
- Not possible to constantly check all pages
- Optimizing for Freshness vs. Relevance can lead to bad decision, such as not crawling
popular sites
Web Crawling: Sitemaps - Answers - - Sitemaps are xml files hosted on the web
- Allow web masters to send info to crawlers
Web Crawling Challenges - Answers - - Performance
- Politeness
- Crawler Traps
- Duplicate Detection
- Data Noise
- Content Missed
Web Crawling Challenges: Performance - Answers - - Back of the envelope calculation
- 1 Page Fetch = 500 ms
- 1 DNS lookup = 800 ms
- Crawler thread waits for responses to request
- Solution: multi-threaded or distributed (servers) crawling
Performance Challenges: Distributed Crawling - Answers - - Helps to put the crawler
closer to the sites it crawls
- Reduces the number of sites the crawler has to remember
- Reduces computing Resources required
Performance Challenges: Focused Crawling - Answers - - Focus on pages about a
particular topic (vertical search)
- Rely on the fact that pages about a topic tent to have links to other pages on the same
topic
