Web Scraping

Lecture 13

Dr. Colin Rundel

Hypertext Markup Language

Most of the data on the web is still largely available as HTML - while it is structured (hierarchical) it often is not available in a form useful for analysis (flat / tidy).

<html>
  <head>
    <title>This is a title</title>
  </head>
  <body>
    <p align="center">Hello world!</p>
    <br/>
    <div class="name" id="first">John</div>
    <div class="name" id="last">Doe</div>
    <div class="contact">
      <div class="home">555-555-1234</div>
      <div class="home">555-555-2345</div>
      <div class="work">555-555-9999</div>
      <div class="fax">555-555-8888</div>
    </div>
  </body>
</html>

rvest

rvest is a package from the tidyverse that makes basic processing and manipulation of HTML data straight forward. It provides high level functions for interacting with html via the xml2 library.

Core functions:

  • read_html() - read HTML data from a url or character string.

  • html_elements() / html_nodes() - select specified elements from the HTML document using CSS selectors (or xpath).

  • html_element() / html_node() - select a single element from the HTML document using CSS selectors (or xpath).

  • html_table() - parse an HTML table into a data frame.

  • html_text() / html_text2() - extract tag’s text content.

  • html_name - extract a tag/element’s name(s).

  • html_attrs - extract all attributes.

  • html_attr - extract attribute value(s) by name.

html, rvest, & xml2

html = 
'<html>
  <head>
    <title>This is a title</title>
  </head>
  <body>
    <p align="center">Hello world!</p>
    <br/>
    <div class="name" id="first">John</div>
    <div class="name" id="last">Doe</div>
    <div class="contact">
      <div class="home">555-555-1234</div>
      <div class="home">555-555-2345</div>
      <div class="work">555-555-9999</div>
      <div class="fax">555-555-8888</div>
    </div>
  </body>
</html>'
read_html(html)
{html_document}
<html>
[1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
[2] <body>\n    <p align="center">Hello world!</p>\n    <br><div class="name" ...

Selecting elements

read_html(html) |> html_elements("p")
{xml_nodeset (1)}
[1] <p align="center">Hello world!</p>
read_html(html) |> html_elements("p") |> html_text()
[1] "Hello world!"
read_html(html) |> html_elements("p") |> html_name()
[1] "p"
read_html(html) |> html_elements("p") |> html_attrs()
[[1]]
   align 
"center" 
read_html(html) |> html_elements("p") |> html_attr("align")
[1] "center"

More selecting tags

read_html(html) |> html_elements("div")
{xml_nodeset (7)}
[1] <div class="name" id="first">John</div>
[2] <div class="name" id="last">Doe</div>
[3] <div class="contact">\n      <div class="home">555-555-1234</div>\n       ...
[4] <div class="home">555-555-1234</div>
[5] <div class="home">555-555-2345</div>
[6] <div class="work">555-555-9999</div>
[7] <div class="fax">555-555-8888</div>
read_html(html) |> html_elements("div") |> html_text()
[1] "John"                                                                                  
[2] "Doe"                                                                                   
[3] "\n      555-555-1234\n      555-555-2345\n      555-555-9999\n      555-555-8888\n    "
[4] "555-555-1234"                                                                          
[5] "555-555-2345"                                                                          
[6] "555-555-9999"                                                                          
[7] "555-555-8888"

Nesting tags

read_html(html) |> html_elements("body div")
{xml_nodeset (7)}
[1] <div class="name" id="first">John</div>
[2] <div class="name" id="last">Doe</div>
[3] <div class="contact">\n      <div class="home">555-555-1234</div>\n       ...
[4] <div class="home">555-555-1234</div>
[5] <div class="home">555-555-2345</div>
[6] <div class="work">555-555-9999</div>
[7] <div class="fax">555-555-8888</div>
read_html(html) |> html_elements("body>div")
{xml_nodeset (3)}
[1] <div class="name" id="first">John</div>
[2] <div class="name" id="last">Doe</div>
[3] <div class="contact">\n      <div class="home">555-555-1234</div>\n       ...

read_html(html) |> html_elements("body div div")
{xml_nodeset (4)}
[1] <div class="home">555-555-1234</div>
[2] <div class="home">555-555-2345</div>
[3] <div class="work">555-555-9999</div>
[4] <div class="fax">555-555-8888</div>

css selectors

We will be using a tool called selector gadget to help up identify the html elements of interest - it does this by constructing a css selector which can be used to subset the html document.

Some examples of basic selector syntax is below,

Selector Example Description
.class .title Select all elements with class=“title”
#id #name Select all elements with id=“name”
element p Select all <p> elements
element element div p Select all <p> elements inside a <div> element
element>element div > p Select all <p> elements with <div> as a parent
[attribute] [class] Select all elements with a class attribute
[attribute=value] [class=title] Select all elements with class=“title”

There are also a number of additional combinators and pseudo-classes that improve flexibility, see examples here.

CSS classes and ids

read_html(html) |> html_elements(".name")
{xml_nodeset (2)}
[1] <div class="name" id="first">John</div>
[2] <div class="name" id="last">Doe</div>
read_html(html) |> html_elements("div.name")
{xml_nodeset (2)}
[1] <div class="name" id="first">John</div>
[2] <div class="name" id="last">Doe</div>
read_html(html) |> html_elements("#first")
{xml_nodeset (1)}
[1] <div class="name" id="first">John</div>

Mixing it up

read_html(html) |> html_elements("[align]")
{xml_nodeset (1)}
[1] <p align="center">Hello world!</p>
read_html(html) |> html_elements(".contact div")
{xml_nodeset (4)}
[1] <div class="home">555-555-1234</div>
[2] <div class="home">555-555-2345</div>
[3] <div class="work">555-555-9999</div>
[4] <div class="fax">555-555-8888</div>

html_text() vs html_text2()

html = read_html(
  "<p>  
    This is the first sentence in the paragraph.
    This is the second sentence that should be on the same line as the first sentence.<br>This third sentence should start on a new line.
  </p>"
)
html |> html_text()
[1] "  \n    This is the first sentence in the paragraph.\n    This is the second sentence that should be on the same line as the first sentence.This third sentence should start on a new line.\n  "
html |> html_text2()
[1] "This is the first sentence in the paragraph. This is the second sentence that should be on the same line as the first sentence.\nThis third sentence should start on a new line."

html |> html_text() |> cat(sep="\n")
  
    This is the first sentence in the paragraph.
    This is the second sentence that should be on the same line as the first sentence.This third sentence should start on a new line.
  
html |> html_text2() |> cat(sep="\n")
This is the first sentence in the paragraph. This is the second sentence that should be on the same line as the first sentence.
This third sentence should start on a new line.

html tables

html_table = 
'<html>
  <head>
    <title>This is a title</title>
  </head>
  <body>
    <table>
      <tr> <th>a</th> <th>b</th> <th>c</th> </tr>
      <tr> <td>1</td> <td>2</td> <td>3</td> </tr>
      <tr> <td>2</td> <td>3</td> <td>4</td> </tr>
      <tr> <td>3</td> <td>4</td> <td>5</td> </tr>
    </table>
  </body>
</html>'
read_html(html_table) |>
  html_elements("table") |> 
  html_table()
[[1]]
# A tibble: 3 × 3
      a     b     c
  <int> <int> <int>
1     1     2     3
2     2     3     4
3     3     4     5

SelectorGadget

This is a javascript based tool that helps you interactively build an appropriate CSS selector for the content you are interested in.

selectorgadget.com

Web scraping considerations

“Can you?” vs “Should you?”

“Can you?” vs “Should you?”

Scraping permission & robots.txt

There is a standard for communicating to users if it is acceptable to automatically scrape a website via the robots exclusion standard or robots.txt.

You can find examples at all of your favorite websites: google, facebook, etc.

These files are meant to be machine readable, but the polite package can handle this for us (and much more).

polite::bow("http://google.com")
<polite session> http://google.com
    User-agent: polite R package
    robots.txt: 525 rules are defined for 5 bots
   Crawl delay: 5 sec
  The path is scrapable for this user-agent
polite::bow("http://facebook.com")
<polite session> http://facebook.com
    User-agent: polite R package
    robots.txt: 719 rules are defined for 33 bots
   Crawl delay: 5 sec
  The path is not scrapable for this user-agent

Scraping with polite

Beyond the bow() function, polite also has a scrape() function that helps you scrape a website while maintaining the three pillars of a polite session:

  • seek permission,

  • take slowly

  • never ask twice.

This is achieved by using the session object from bow() within the scrape() function to make the request (this is equivalent to rvest’s read_html() and returns a parsed html object).

New paths within the website can be accessed by using the nod() function before using scrape().

Example - Rotten Tomatoes

For the movies listed in Popular Streaming Movies list on rottentomatoes.com create a data frame with the Movies’ titles, their tomatometer score, and whether the movie is fresh or rotten, and the movie’s url.

Exercise 1

Using the url for each movie, now go out and grab the mpaa rating, the runtime and number of user ratings.

If you finish that you can then try to scrape the tomato meter and audience scores for each movie.