Scraping Data off a Web Site

I’m taking the Data Analysis class through Coursera and one of the topics we’ve covered so far is how to “scape” data off a web site. The idea is to programmatically got through the source code of a web page, pull out some data, and then clean it up so you can analyze it. This may seem like overkill at first glance. After all, why not just select the data with your mouse and copy-and-paste into a spreadsheet? Well, for one, there may be dozens (or hundreds) of pages to visit and copying-and-pasting from each one would be time-consuming and impractical. Second, rarely does a copy-and-paste off a web site produce data ready for analysis. You have to tidy it up, sometimes quite a bit. Clearly these are both tasks we would like to automate.

To put this idea to use, I decided to scrape some data from the box scores of Virginia Tech football games. I attended Tech and love watching their football team, so this seemed like a fun exercise. Here’s an example of one of their box scores. You’ll see it is has everything but what songs the band played during halftime. I decided to start simple and just scrape the Virginia Tech Drive Summaries. This summarizes each drive, including things like number of plays, number of yards gained, and time of possession. Here’s the function I wrote in R, called vtFballData:

vtFballData <- function(start,stop,season){
    dsf <- c()
    # read the source code
    for (i in start:stop){
    url <- paste("http://www.hokiesports.com/football/stats/showstats.html?",i,sep="")
    web_page <- readLines(url)

    # find where VT drive summary begins
    dsum <- web_page[(grep("Virginia Tech Drive Summary", web_page) - 2):
                         (grep("Virginia Tech Drive Summary", web_page) + 18)]
    dsum2 <- readHTMLTable(dsum)
    rn <- dim(dsum2[[1]])[1]
    cn <- dim(dsum2[[1]])[2]
    ds <- dsum2[[1]][4:rn,c(1,(cn-2):cn)]
    ds[,3] <- as.character(ds[,3]) # convert from factor to character
    py <- do.call(rbind,strsplit(sub("-"," ",ds[,3])," "))
    ds2 <- cbind(ds,py)
    ds2[,5] <- as.character(ds2[,5]) # convert from factor to character
    ds2[,6] <- as.character(ds2[,6]) # convert from factor to character
    ds2[,5] <- as.numeric(ds2[,5]) # convert from character to numeric
    ds2[,6] <- as.numeric(ds2[,6]) # convert from character to numeric
    ds2[,3] <- NULL # drop original pl-yds column

    names(ds2) <-c("quarter","result","top","plays","yards")
    # drop unused factor levels carried over from readlines
    ds2$quarter <- ds2$quarter[, drop=TRUE]
    ds2$result <- ds2$result[, drop=TRUE]

    # convert TOP from factor to character
    ds2[,3] <- as.character(ds2[,3])
    # convert TOP from M:S to just seconds
    ds2$top <- sapply(strsplit(ds2$top,":"),
          function(x) {
            x <- as.numeric(x)
            x[1]*60 + x[2]})

    # need to add opponent
    opp <- web_page[grep("Drive Summary", web_page)]
    opp <- opp[grep("Virginia Tech", opp, invert=TRUE)] # not VT
    opp <- strsplit(opp,">")[[1]][2]
    opp <- sub(" Drive Summary</td","",opp)
    ds2 <- cbind(season,opp,ds2)
    dsf <- rbind(dsf,ds2)
    }
return(dsf)
}

I’m sure this is three times longer than it needs to be and could be written much more efficiently, but it works and I understand it. Let’s break it down.

My function takes three values: start, stop, and season. Start and stop are both numerical values needed to specify a range of URLs on hokiesports.com. Season is simply the year of the season. I could have scraped that as well but decided to enter it by hand since this function is intended to retrieve all drive summaries for a given season.

The first thing I do in the function is define an empty variable called “dsf” (“drive summaries final”) that will ultimately be what my function returns. Next I start a for loop that will start and end at numbers I feed the function via the “start” and “stop” parameters. For example, the box score of the 1st game of the 2012 season has a URL ending in 14871. The box score of the last regular season game ends in 14882. To hit every box score of the 2012 season, I need to cycle through this range of numbers. Each time through the loop I “paste” the number to the end of “http://www.hokiesports.com/football/stats/showstats.html?” and create my URL. I then feed this URL to the readLines() function which retrieves the code of the web page and I save it as “web_page”.

Let’s say we’re in the first iteration of our loop and we’re doing the 2012 season. We just retrieved the code of the box score web page for the Georgia Tech game. If you go to that page, right click on it and view source, you’ll see exactly what we have stored in our “web_page” object. You’ll notice it has a lot of stuff we don’t need. So the next part of my function zeros in on the Virginia Tech drive summary:

# find where VT drive summary begins
dsum <- web_page[(grep("Virginia Tech Drive Summary", web_page) - 2):
                 (grep("Virginia Tech Drive Summary", web_page) + 18)]

This took some trial and error to assemble. The grep() function tells me which line contains the phrase “Virginia Tech Drive Summary”. I subtract 2 from that line to get the line number where the HTML table for the VT drive summary begins (i.e., where the opening <table> tag appears). I need this for the upcoming function. I also add 18 to that line number to get the final line of the table code. I then use this range of line numbers to extract the drive summary table and store it as “dsum”. Now I feed “dsum” to the readHTMLTable() function, which converts an HTML table to a dataframe (in a list object) and save it as “dsum2″. The readHTMLTable() function is part of the XML package, so you have download and install that package first and call library(XML) before running this function.

At this point we have a pretty good looking table. But it has 4 extra rows at the top we need to get rid of. Plus I don’t want every column. I only want the first column (quarter) and last three columns (How lost, Pl-Yds, and TOP). This is a personal choice. I suppose I could have snagged every column, but decided to just get a few. To get what I want, I define two new variables, “rn” and “cn”. They stand for row number and column number, respectively. “dsum2″ is a list object with the table in the first element, [[1]]. I reference that in the call to the dim () function. The first element returned is the number of rows, the second the number of columns. Using “rn” and “cn” I then index dsum2 to pull out a new table called “ds”. This is pretty much what I wanted. The rest of the function is mainly just formatting the data and giving names to the columns.

The next three lines of code serve to break up the “Pl-Yds” column into two separate columns: plays and yards. The following five lines change variable classes and remove the old “Pl-Yds” column. After that I assign names to the columns and drop unused factor levels. Next up I convert TOP into seconds. This allows me to do mathematical operations, such as summing and averaging.

The final chunk of code adds the opponent. This was harder than I thought it would be. I’m sure it can be done faster and easier than I did it, but what I does works. First I use the grep() function to identify the two lines that contain the phrase “Drive Summary”. One will always have Virginia Tech and the other their opponent. The next line uses the invert parameter of grep to pick the line that does not contain Virginia Tech. The selected line looks like this for the first box score of 2012: “<td colspan=\”9\”>Georgia Tech Drive Summary</td>”. Now I need to extract “Georgia Tech”. To do this I split the string by “>” and save the second element:

opp <- strsplit(opp,">")[[1]][2]

It looks like this after I do the split:

[[1]]
[1] "<td colspan=\"9\""              "Georgia Tech Drive Summary</td"

Hence the need to add the “[[1]][2]” reference. Finally I substitute ” Drive Summary</td” with nothing and that leaves me with “Georgia Tech”. Finally I add the season and opponent to the table and update the “dsf” object. The last line is necessary to allow me to add each game summary to the bottom of the previous table of game summaries.

Here’s how I used the function to scrape all VT drive summaries from the 2012 regular season:

dsData2012 <- vtFballData(14871,14882,2012)

To identify start and stop numbers I had to go to the VT 2012 stats page and hover over all the box score links to figure out the number sequence. Fortunately they go in order. (Thank you VT athletic dept!) The bowl game is out of sequence; its number is 15513. But I could get it by calling vtFballData(15513,15513,2012). After I call the function, which takes about 5 seconds to run, I get a data frame that looks like this:

 season          opp quarter result top plays yards
   2012 Georgia Tech       1   PUNT 161     6    24
   2012 Georgia Tech       1     TD 287    12    56
   2012 Georgia Tech       1  DOWNS 104     5    -6
   2012 Georgia Tech       2   PUNT 298     7    34
   2012 Georgia Tech       2   PUNT  68     4    10
   2012 Georgia Tech       2   PUNT  42     3     2

Now I’m ready to do some analysis! There are plenty of other variables I could have added, such as whether VT won the game, whether it was a home or away game, whether it was a noon, afternoon or night game, etc. But this was good enough as an exercise. Maybe in the future I’ll revisit this little function and beef it up.


Source: http://www.clayford.net/statistics/scraping-data-off-a-web-site/

Is Data Scraping Unethical?

Perhaps the biggest challenges that website owners face, in addition to attracting visitors, is coming up with original content to publish on their websites.

Search engines are ravenously hungry creatures. They are constantly scraping the web, seeking content that they can add to the index, and if your site publishes good quality original content, the chances are very likely that you will receive higher ranking on the SERPs.

The process may not be as simple as it sounds, as there are perhaps millions of competitors in the same area, who may be competing for ranking on the same keyword(s).

Because of the challenges that are faced, with the very time-consuming and labor intensive tasks of continually creating and publishing original content, website owners may often seek shortcuts or use methods and applications that are frowned on by the search engines.

In order to remain competitive, another of the tasks, with which website owners are faced, is with keeping an eye on the competition. You need to know what the competition is up to, and you need to be able to react to, or else you can easily get left behind. One of the ways that you can do this is by developing applications that focus on data scraping. Obtaining the data may be harmless, but how it is used is where the questions often arise.

While the practice may appear to fairly innocuous and can be useful, there are several instances where it may be questioned.

There is now rapidly expanding industry for data mining. Reports are that we now create more data every day than we did in the over the last two decades, and the market for data mining continues to expand exponentially. Marketers are constantly scraping the web to build profiles of consumers, and we may be making easier for them, by leaving trails that they can easily follow.

It may be it bit disconcerting to know that every website that you ever visit is actually logged, and the data can be used to build a profile of your habits. Many uses may find it intrusive, to find that information that should be considered as private, is now available for public consumption.

Scraping can involve not only personal data, but also your buying behaviour as well as customary habits or hobbies. All of your online activities can be tracked, and although it may be stated as otherwise, there are ways that your data can be shared by other third parties, with contravening any laws

It is easy to collect detailed information, such as cell-phone numbers, email addresses and even your posts on the social networks, can easily be tracked and analysed.

There is considerable debate as to the ownership of data that is posted on the social networks. To whom does it really belong, and who should be allowed access to it?

It is also not surprising that media outlets are using data scraping methods by employing what are called listening devices to monitor what is being said on the social networks in real time. It is one of the ways that they can observe what is being said about specific organisations, products or people.

The debate is sure to continue, but there is no doubt that it can be useful.

Source: http://www.twm.co.nz/is-data-scraping-unethical/

Web Scraping is actually pretty easy

For some of our clients we worked on extracting or submitting data automatically from websites which didn’t have an API we could use. This and more is called web scraping. Since our announcement of SellerScout, which relies heavily on this, I received a list of questions how we actually do this. So here are some thoughts on how to get started in the interesting web scraping world.

This article talks about some basics, which will work fine for most cases. This is probably not even remotely close to how Google does this or how we do it at SellerScout. The reason being both of those systems work in much larger scale and use cases are different. For example relying on machine learning, text analysis and semantic search algorithms etc. are all the things you might be doing if you want to build something big.
Downloading the web
Scraper

It's all just scraping

Spiders are the small applications you are going to be writing. Usually they are self-contained and CLI-friendly scripts, which have some internal logic how to extract information from a specific website or websites. As an example, the script might go to website’s homepage, download all the category pages, download products list for each of them and extract a list of products in the store.

If you are a Python guy, you might want to look at Twisted or Scrapy, later being very easy to use. If it’s PHP you are using, combination of cURL and libxml will allow doing the same; I’m not aware of any PHP frameworks for this. For any other language, you should give a look at Google.

Depending on your task you will need to support different functionality. If the website is for logged in users only, you should configure cURL to use cookies jar and initialize the scraping with a request to login page. If you need to extract thousands of documents, have some logic to pause and resume the script, so if it crashes it can start from the last completed document rather than from the start. In any case, try to replicate the natural user behaviour on the site.

Is it legal? Depends. There is no strict answer and it varies on what data you are trying to extract. Some data can be copyrighted, for example original texts, so if you are scraping them and showing in your website - you are being a bad person. Stop! Ideally you should discuss this with your lawyer, which we did, and get some thoughts on how to proceed.
Getting blocked
Stop sign

One of the decisions you will need to make is how you are going to identify the spider - you can either replicate normal browser’s headers or introduce the spider by its name (eg. ’googlebot’). First one will allow you to stay undetected, probably, while later one is considered to be the correct way. From my experience, for anything small Firefox headers will work just fine.

Websites might still decide to block you though, and it’s something you might want to be prepared for. If you are identifying the spider by a name, you should respect robots.txt and stop crawling if you are being denied by that file. However the most likely blocking mechanism is to block your IP address, which is going to happen if you are being stupid. Really stupid.

You see, when people are browsing the web they request 1 page each 3 or 4 seconds, hence if you have a list of 1000 urls to download and you just start iterating over them and issuing requests… Well, you are easy to catch by just looking at the access logs. Don’t do this. Rather have a queue of urls to download and issue requests with a random delay from a range of 1 to 5 seconds. It’s going to take longer, but it will help to avoid problems.

This doesn’t scale though, you might say. And in fact you are right, because 5 seconds delay between each request limits the amount of content you can download per day. Luckily for you, I have a tip here too - use proxy servers. It’s going to require writing a requests scheduler, but if you need to download the same 1000 urls you might as well distribute them over a list of proxies each with their own delay times. The more proxies you have the amount of content you can download increases linearly.
Extracting the data
XPath

Once you have the HTML you want to process (to extract links to follow or to extract actual data), you might wonder how to actually do it. There couple of ways and libraries for this, however if you want to keep it simple using XPath or CSS-like queries is going work just fine. If you feel like it, and believe me sometimes there is no other way, you might go with using regular expressions for this, but that’s got problems I’m going to talk about just in a second.

I tend to go with XPath because it’s very easy to write and to debug. Furthermore there are various extensions you get for your browser which will allow creating those queries and test them on the actual website. I have worked on spiders for over a 100 different websites and XPath worked fine all the time, as long as…

The problem you will need to solve is how to process invalid HTML or XHTML markup. And from my experience, I’m yet to see a website with all pages being 100% valid. The more invalid it is, the harder it’s to fix those problems. There are libraries though, most famously BeautifulSoap, which will try to process invalid markup. They do have performance implications, but keep them in mind because you won’t be able to issue XPath queries on invalid syntax.

Now let’s get back to regular expressions. Theoretically they might look awesome, because they can extract data even if the HTML markup is invalid, however the problem is that soon they get complicated and very easy to break. XPath allows you to work on a DOM tree, hence if the website structure change they just stop working completely. Regexps on the other hand might still work, but produce very unpredictable results.
Conclusion

We, as a company, have a lot of experience web scraping the data and it’s actually very very easy. As long as you follow the logical rules and don’t try to over-complicate the data extraction, you could easily extract all news items, products or blog posts in 30 or so code lines of spider. I can talk on this for hours or days, so I might write more on this soon, because this is just a top of an iceberg.



Source: http://blog.webspecies.co.uk/2011-07-27/web-scrapping-is-actually-pretty-easy.html

An introduction to data scraping with Scraperwiki

Last week I spent a day playing with the screen scraping website Scraperwiki with a class of MA Online Journalism students and a local blogger or two, led by Scraperwiki’s own Anna Powell-Smith. I thought I might take the opportunity to try to explain what screen scraping is through the functionality of Scraperwiki, in journalistic terms.

It’s pretty good.
Why screen scraping is useful for journalists

Screen scraping can cover a range of things but for journalists it, initially, boils down to a few things:

    Getting information from somewhere
    Storing it somewhere that you can get to it later
    And in a form that makes it easy (or easier) to analyse and interrogate

So, for instance, you might use a screen scraper to gather information from a local police authority website, and store it in a lovely spreadsheet that you can then sort through, average, total up, filter and so on – when the alternative may have been to print off 80 PDFs and get out the highlighter pens, Post-Its and back-of-a-fag-packet calculations.

But those are just the initial aspects of screen scraping. Screen scraping tools like Scraperwiki or scripts you might write yourself offer further benefits that are also worth outlining:

    Scheduling a scraper to run at regular intervals (Adrian Holovaty compares this to making regular virtual trips to the local police station)
    Re-formatting data to clarify it, filter it, or make it compatible with other sets of data (for example, converting lat-long coordinates to postcodes, or feet to metres)
    Visualising data (for example as a chart, or on a map)
    Combining data from more than one source (for example, scraping a list of company directors and comparing that against a list of donors)

If you can think of any more, let me know.


Source: http://onlinejournalismblog.com/2010/07/07/an-introduction-to-data-scraping-with-scraperwiki/

Best method for scraping data from Web using VB macro?

This is something of a conceptual question rather than on the specifics of code (ie am I going about this the right way in general or is there a better technique I could use?). I think that my problem represents a broad issue affecting many of the inexperienced people who post on this forum so an overview and sharing of best practice would also help many people.

My aim is to scrape statistical data from a website (here is an exemplar page: www.racingpost.com/horses/result_home.sd?race_id=572318&r_date=2013-02-26&popup=yes#results_top_tabs=re_&results_bottom_tabs=ANALYSIS

I have a very basic (if you pardon the pun) knowledge of VB which I use through excel but know nothing about other programming languages or conventions (SQL, HTML, XML etc.), however I am quite good at writing code to manipulate strings- that is, once I can scrape the data, even if it is in a very noisy form then I am expert at processing it. I am trying to build an automated process that will scrape up to 1000 pages in one hit. In one form or another, I have been working on this for years and the last few weeks have been very frustrating in that I have come up with several new methods which have taken days of work but have each had one fatal flaw that has stopped my progress.

Here are the methods I have tried (all using a VB macro run from Excel):
1) Control Firefox (as a shell application) - this was the poorest, I found that I could not interact with Firefox properly using a VB excel macro- i tried mainly using keystrokes etc.
2) Get inner text, outer text, inner html or outer html from internet explorer (IE)- this method was by far the most reliable but the data was, at times, very hard to parse and did not always contain everything I needed (good for some applications but bad for others)
3) automated Copy and Paste from IE- this was tantalisingly close to being perfect but is given to throwing up inexplicable errors whereby the type of information copied to the clipboard differs depending on whether it is done manually (ie CTRL+A, CTRL+C) or through the automated process (with the former I could get the HTML structure- ie tables, with the latter only text). The enigma here was that I could get the automated copy/paste to give me the right info IF I FIRST CLICKED INSIDE THE IE WINDOW USING MOUSE POINTER- however I was unable to automate this using a VB MACRO (I tried sendkeys and various other methods)
4) By automating an excel webquery- I recorded a macro of a query, this worked flawlessly giving me the structure of tables I needed. Snag was it was very very slow- even for a single page it might take 14 to 16 seconds (some of the other methods I used were near instantaneous). Also this method appears to encounter severe lagging/crashing problems when many refreshes are done (that may be because I don't know how to update the queries with different criteria, or properly extinguish them)
5) Loading the page as an XML document- I am investigating this method now- I know next-to-nothing about XML but have a hunch the sort of pages I am scraping (see example above) are suitable for this. I have managed to load the pages as an XML object but at present seem to be running into difficulties trying to parse the structure (ie various nodes) to extract text- keep running into object errors.

For the record I have posted highly specific questions with code relating to these individual methods without response so I am trying a broader question. What is the experience of others here? Which of these methods should I focus on? (bear in mind I am trying to keep everything to Excel VB Macros). I am getting to the point where I might look to get someone to code something for me and pay them (as this is taking hundreds of hours) - have people had good experiences employing others to write ad hoc code in this manner?


Source: http://www.mrexcel.com/forum/excel-questions/688229-best-method-scraping-data-web-using-vbulletin-macro.html

Scrape data from Store Locators

One of the many applications of web scraping is to capture the locations of store from a retailers store locator site. For many years, retail store planners wanting to relocate or open new branches of their stores often had to rely on finding a connection within the retail organization of which data was needed. Moreover, if data list exchanged many hands, the final end user ended up paying thousands of dollars. With many retail chains putting up-to-date information of their stores on the store locator engine, web scraping allows us to capture this data quickly, accurately and at a fraction of the cost of what you might have to pay if you are looking to obtain this data by finding a source within the said organization.

The information you can generally scrape is:

    Name of the store / Store ID
    Address
    City
    State
    Zip / Postal Code
    Phone Number
    Facilities Available
    Store Hours
    Latitude & Longitude

Utilizing our experience in custom web scraping, we can provide you with all the above information from the store locator site in Excel / CSV or in any other format as per requirement. For a quote on your web scraping project, send us your request along with the targeted store locator site address, the geographical region / area and format in which data is needed. Our expert team of programmers will evaluate the site and write scripts that will anonymously and with 100% accuracy scrape the data.

Source: http://blog.website-scraping.com/scrape-data-from-store-locators/

Website Data Scraping Services Helps Even Internet Solutions

In all over world “in the most credible and reliable” based on an outsourcing provider – data services, scratching the business Web Solutions. Scraping data services, high quality, and industry specific information and manually scraping and scraping the web site services to the lowest possible rate.

Scraping data from the services of an Indian company outsourcing data entry skills,derrick rose cards, data processing, research,washington capitals ovechkin, and the scraping of Web site data. Scraping a wide range of services, data entry, data conversion, document scanning and data services at a lower price to the scraping of the industry since 2005.

Our services cover the following areas; data entry, data mining, Web search, data conversion, data processing, web scraping, web harvesting and data collection, e mail.

Standard for high quality web search, data mining and web scraping services to provide a scraping action, the following information services. Our research on the web, data mining and data entry into the project formulation process is the quality standard.

Scraping industry information usually can be lawyers, doctors, hospitals, students, school,discount softball jerseys, university, chiropractor, dentist, hotels, real estate, real estate, pubs, bars, nightclubs, restaurants and IT business to follow a. The most common tool for database and email IDs of scratching the online business, twitter, Face book, social networking sites and the Google search.

Scratching more credible and reliable data services provider in the world service, data processing, Data scraping services offered, database development, web data mining, data extraction and business scraping. We are already ahead of many popular online business directories. We are skilled enough to be publicly available database of businesses from scratch in any directory.

Data scraping Services is one of the best and reliable web data scraping business in the world. Data scraping Services provide a wide range of web data scraping, scraping website data, web data mining, web data extraction, and automated data scraping and scraping manual data services.

Within a short time we have worked for various offshore clients, not just a few times (with many current customers since worked first project). We can one-stop solution provider for all your data entry, offline data mining, data capture, web data extraction and scraping work requirement. Try us for your project requirement in order to get quality results within a fast time.

Data scraping Services is India based on the most reliable company; the website offers data scraping solution for offshore clients. Try scraping with Data Services to your web search, data mining, data conversion, document scanning and web data scraping work to accomplish. Get free quote and sample work by sending your information and data scraping scraper development should be our

The forward thinking like this that from its competitors. It is not only our company. It is our passion. We have a lot of time and effort in making it easy and affordable for you to build and manage your own data capture scripts.

Scraping software turns the web into your own personal database with phenomenal ease. Once you know you will find there are no restrictions with a comprehensive suite of tool that you as an indispensable point after your first experience.

Source: http://horst6.co.nu/website-data-scraping-services-helps-even-internet-solutions/