Pipes and Filters Architecture

Yahoo have recently launched pipes, a visual programming environment for creating a mashup RSS feed from user inputs and available RSS sources. XML languages for defining pipelines are emerging.

Here are some bloogers on the subject:

• John Musser (Programmeable Web)
• Tim O'Reilly.
• Fred Stutzman (about the need to change web applications to support fine-grained RSS feeds.)
• TechCrunch
• Kurt Cagle (on XML pipelines)
• Jeni Tennison's Xtech paper is an excellent overview of XML pipelines

Issues with these languages include

• whether the pipeline itself is expressed in XML (and thus processable with XML tools)
  
• whether non-XML data streams are allowed. For example where an intermediate file is non-XML (e.g. Graphviz dot) or the output is non XML (a GIF image)

The origin of Pipes as a concept in which the output of one process is conected to the input of another is in the Unix operating system - Unix Pipe

Ant is widely used in the Java community as a build tool, but can perform XML pipelining.

To use a pipe architecture, we need component filters to carry out standard transformations.

• Dapper is a tool for scraping HTML pages to create an XML or RSS feed . The neat thing about this tool is that you can give it a number of similar pages and Dapper will try to infer which data items differ page to page, and how to recognise each item. You then name the items you want to scrape and can form these into an HTML, XML or RSS feed.
• RSS or Atom to PDF e.g. BBC Bristol Weather
  
•