Screen scraping using YQL

I have been using Yahoo pipes for a long time now. I have done some screen scraping mash up using them. While yahoo pipe provides a component to fetch the HTML content from a URL, it is bit difficult to cut a specific part because it totally relies on a string to match. I came across YQL console where I could write SQL like queries and fetch the HTML content of any URL. The best part was that it supports XPath expressions for selecting the exact node of the HTML to extract data. For example I write the following query to get the stock price from the web page

[code language="sql"]
select *
from html
where url ="http://getquote.icicidirect.com/NewSiteTrading/trading/equity/includes/trading_stock_quote.asp?Symbol=BSES"
and xpath='//td[p/text()="LAST TRADE PRICE"]/following-sibling::td[2]/p'
[/code]

See above code running in YQL Console.

Similarly this query can be made little bit complex and parametrized for the stock symbol to form the appropriate url

[code language="sql"]
select *
from html
where url in (
select url
from uritemplate
where template="http://getquote.icicidirect.com/NewSiteTrading/trading/equity/includes
/trading_stock_quote.asp?Symbol={item}" and item=@item)
and xpath='//td[p/text()="LAST TRADE PRICE"]/following-sibling::td[2]/p | //td[p/text()="LAST TRADED TIME"]/following-sibling::td[1]/p'
[/code]

See the above code in YQL console. However this will not directly run from the console. One could just create a query alias and pass the required query string like the following.

http://query.yahooapis.com/v1/public/yql/neilghosh/liveQuote?item=INFTEC

This could have been done using the built in YQL component in the Yahoo Pipes itself but it would be an extra layer if you just need to get the required content from the HTML instead of having to play around any feed (for which Pipes is still the best choice). Of course there some limits/quota while using such YQL queries, which I need to explore in coming days.

For screen scraping I could directly use Google App Engine's URLFetch or curl in PHP servers but this would unnecessarily transfer the whole content consuming quota and leading to time lag.

Comments

  1. Great website

    [...]we like to honor many other internet sites on the web, even if they aren’t linked to us, by linking to them. Under are some webpages worth checking out[...]…

    ReplyDelete
  2. Websites we think you should visit

    [...]although websites we backlink to below are considerably not related to ours, we feel they are actually worth a go through, so have a look[...]…

    ReplyDelete
  3. Thanks for sharing, i like your article and opinion.
    cheap jerseys free shipping http://www.featuredjerseys.com/

    ReplyDelete
  4. I at last found terrific posting listed here.I will return right here. I just now increased your blog to my save web pages. thanks.Superior articles could be the vital to encourage the website visitors to proceed to the web page, that's what this page is supplying.

    ReplyDelete
  5. Thanks for sharing, i like your article and opinion.
    cheap OAKLEY Sunglasses http://www.oksunglassessell.com/

    ReplyDelete
  6. Thanks for sharing, i like your article and opinion.

    ReplyDelete
  7. Pretty section of content. I just stumbled upon your website and in accession capital to assert that I acquire actually enjoyed account
    your blog posts. Any way I'll be subscribing to your augment and even I achievement you access consistently quickly.

    ReplyDelete
  8. Thanks for sharing, i like your article and opinion.
    cheap oakley sunglasses for men http://www.okleystore.us/sale-new-oakley-sunglasses-online-c-17.html

    ReplyDelete
  9. khfka Hollister Online Shop sdfsdfsfsd Hollister dasdasda Hollister fdafkjldal Hollister Schweiz gooddasda Hollister hgjhgjg gfdsgs Occhiali Oakley Prezzi gfsjkghksl gfshgjksasdsa fsjkhgs Oakley Occhiali dafhjhg dfahjkfh dfakjdf dfakjhfk hglkf. dfjaklj fgjkcvbh dfbg dafhj dafj Oakley and <a href="http://www.biofox.com/oakley.htmOakley Prezzi sfghag kljhklf Oakley fdsafjkh dfjkahfjk dfgfhwru Hollister Italia fshgjk zhjgfcshdgf Hollister Outlet fdsagf dfakjh flkgjlkd dfjafaf fdagjk dkajfhjkdah gfdgjkhd Pandora Gioielli fkdkg dkfgjlkhfg erepiopop Hollister dafhjka kgkerui ertuey jdgjdhg sdfsdf Hollister Outlet jksdfhjs sdfjkhjsk dfjkshj h kdflgjkl kfjdgk ghfgh gfdsgs fkjs gfhskuh Occhiali Oakley Prezzi dfadaf khjkh hrjtyrio fghuier topyiro ijiejtio Hollister djakfh dfhajkhf dfhakjh ioyoi ghjghj.

    ReplyDelete

Post a Comment

Popular posts from this blog

Load Testing Using Gatling

Bengali English word definition in Google Dictionary

Sharing Photosphere