links.lmorchard.com - #scraping

Continuing our infrastructure updates: What’s changing in the coming months : r/modnews

Wed, 07 Oct 2026 18:20:29 +0000

Because RSS is now a common surface for large-scale scraping and automated abuse, we'll stop supporting RSS feeds on November 13, 2026 while preserving moderation workflows that depend on it through supported alternatives.

Tags: #ai #reddit #rss #scraping

jacktuck/unfurl: Metadata scraper with support for oEmbed, Twitter Cards and Open Graph Protocol for Node.js :zap:

Wed, 21 Aug 2024 23:31:41 +0000

Highlight:Unfurl (spread out from a furled state) will take a url and some options, fetch the url, extract the metadata we care about and format the result in a sane way. It supports all major metadata providers and expanding it to work for any others should be trivial. Metadata scraper with support for oEmbed, Twitter Cards and Open Graph Protocol for Node.js :zap: - jacktuck/unfurl

Tags: #html #imported:raindrop #scraping #webdev

microlinkhq/metascraper: Get unified metadata from websites using Open Graph, Microdata, RDFa, Twitter Cards, JSON-LD, HTML, and more.

Wed, 21 Aug 2024 23:30:30 +0000

Highlight:The metascraper library allows you to easily scrape metadata from an article on the web using Open Graph metadata, regular HTML metadata, and series of fallbacks. Get unified metadata from websites using Open Graph, Microdata, RDFa, Twitter Cards, JSON-LD, HTML, and more. - microlinkhq/metascraper

Tags: #html #imported:raindrop #opengraph #scraping #webdev

Using SimpleXML with HTML | drewish.com

Wed, 21 Apr 2010 17:41:30 +0000

"PHP 5's SimpleXML module is one of the the biggest reasons to upgrade to 5. If you're parsing RSS feeds or the results of webservice requests it works beautifully and saves a ton of time. The only problem with it is that it'll only load valid XML. I banged my head against it for way to long before coming up with the following:"

Tags: #dom #html #php #scraping #simplexml #webdev #xml

Scrapy.org. an open source web scraping framework in Python

Wed, 18 Mar 2009 03:05:49 +0000

"Scrapy is a high level scraping and web crawling framework for writing spiders to crawl and parse web pages for all kinds of purposes, from information retrieval to monitoring or testing web sites. "

Tags: #python #scraping #webdev

Crowbar - SIMILE

Fri, 23 Jan 2009 20:14:19 +0000

Tags: #crowbar #javascript #js #mozilla #scrapers #scraping #webdev #xulrunner

Zend Developer Zone | Tidying up your HTML with PHP 5

Sat, 19 Aug 2006 05:25:29 +0000

"The Tidy extension is new in PHP 5, and is available from PHP version 5.0b3 upward. It is based on the TidyLib library, and allows the developer to validate, repair, and parse HTML, XHTML and XML documents from within PHP."

Tags: #php #scrapers #scraping #tidy #webdev

Microsummaries - MozillaWiki

Sun, 14 May 2006 22:49:08 +0000

"Microsummaries are regularly-updated succinct compilations of the most important information on web pages."

Tags: #firefox #mozilla #scraping #webdev #xsl

HTML Screen Scraping: A How-To Document

Sat, 21 Jan 2006 18:03:36 +0000

"This document explains how to do HTML screen scraping. In effect it shows how to treat the Web as a resource by enabling you to retrieve and extract data from HTML Web pages."

Tags: #programming #scraping #webdev

For GRDDL-heads: XSLT+Tidy from Mark Nottingham on 2005-10-19 (semantic-web@w3.org from October 2005)

Thu, 20 Oct 2005 13:46:16 +0000

"Let the scraping begin..."

Tags: #grddl #microformats #scraping #xslt

Read/Write Web: The danger of running a remix service

Sat, 17 Sep 2005 17:02:15 +0000

"Populicio.us still lost their service because their reliance on del.icio.us fell away, but the lesson here is that screen scraping HTML comes with those risks by nature."

Tags: #del.icio.us #hacks #scraping #webdev #webservices

miscoranda: Link in a Soupstack

Sun, 05 Jun 2005 20:19:41 +0000

"The problem with getting links from HTML is that the HTML you find lying about on the web is often quite broken..."

Tags: #html #python #scraping

Beautiful Soup

Mon, 21 Jun 2004 11:25:28 +0000

"You didn't write that awful page. You're just trying to get some data out of it. Right now, you don't really care what HTML is supposed to look like."

Tags: #python #scraping

Pop Goes the Gmail. SMTP/POP server for Gmail!

Tue, 01 Jun 2004 16:44:04 +0000

Tags: #mail #scraping #webdev