links.lmorchard.com - #scrapers

Using AgentQL and Pipedream to Fix Missing RSS Feeds

Tue, 29 Jul 2025 02:52:09 +0000

Last week I blogged about how I used AgentQL to scrape a web page, this blog to be precise, into pure data. If you don't remember, AgentQL lets you pass a simple query string that is run against a web page and parsed into data. So for example, I was able to use this query on my home page to get a list of entries in pure data: { blogposts[] { url title date } } The REST API was incredibly easy, and I demonstrated in BoxLang how I could use AgentQL to turn my blog's home page into a much simpler, smaller, HTML version. This of course begs the question, how else could we transform it? What about creating a RSS feed for a blog that doesn't have one!

Tags: #agentql #rss #scrapers

Crowbar - SIMILE

Fri, 23 Jan 2009 20:14:19 +0000

Tags: #crowbar #javascript #js #mozilla #scrapers #scraping #webdev #xulrunner

Overcoming data friction « Jon Udell

Fri, 22 Feb 2008 18:41:58 +0000

"For me, web screen-scraping provides the kind of challenge that other people get from, say, solving crossword puzzles. But it’s not the highest and best use of anyone’s time."

Tags: #data #datafriction #scrapers #webdev

Glibness — How to make an RSS feed where one never was

Tue, 15 Jan 2008 05:47:48 +0000

"One feature [on del.icio.us] that hasn’t been implemented is notification of new people adding you to their network."

Tags: #delicious #feeds #rss #scrapers

Zend Developer Zone | Tidying up your HTML with PHP 5

Sat, 19 Aug 2006 05:25:29 +0000

"The Tidy extension is new in PHP 5, and is available from PHP version 5.0b3 upward. It is based on the TidyLib library, and allows the developer to validate, repair, and parse HTML, XHTML and XML documents from within PHP."

Tags: #php #scrapers #scraping #tidy #webdev