#scrapers

Last week I blogged about how I used AgentQL to scrape a web page, this blog to be precise, into pure data. If you don't remember, AgentQL lets you pass a simple query string that is run against a web page and parsed into data. So for example, I was able to use this query on my home page to get a list of entries in pure data: { blogposts[] { url title date } } The REST API was incredibly easy, and I demonstrated in BoxLang how I could use AgentQL to turn my blog's home page into a much simpler, smaller, HTML version. This of course begs the question, how else could we transform it? What about creating a RSS feed for a blog that doesn't have one!

"For me, web screen-scraping provides the kind of challenge that other people get from, say, solving crossword puzzles. But it’s not the highest and best use of anyone’s time."

"One feature [on del.icio.us] that hasn’t been implemented is notification of new people adding you to their network."

"The Tidy extension is new in PHP 5, and is available from PHP version 5.0b3 upward. It is based on the TidyLib library, and allows the developer to validate, repair, and parse HTML, XHTML and XML documents from within PHP."