Backup Blogger Blog with Ruby Mechanize Script

After writing up a cURL script to download a backup of my Blogger blog in my previous post I started wondering what the best way to do this in Ruby would be. I looked at some Ruby HTTP clients and narrowed my choices down to two: Mechanize and HTTPClient. Both of these tools look pretty good but Mechanize seemed to offer a very simple interface to scripting a web interaction like this while HTTPClient would offer something that was more like my original cURL script. So I decided to write the same code with a Mechanize approach:



I need to beef this up with some error handling obviously - but I think you'll find Mechanize to be pretty slick if you try it.

Fix Broken Delimiter Separated Values Files

We get broken delimiter separated values files that are not properly quoted. Actually they don't use any type of quoting and expect the delimiter (a pipe character) to not appear in the output. Invariably these files end up having a handful or rows with fields with carriage returns in them. These rows end up getting dropped during later processing. Below is a simple script to fix such files. Note that the default delimiter is a pipe symbol and the default number of separators comes from counting the separators in the first row (typically the header row).



This simple script demonstrates optparse and writing a script that uses standard in / out or opens files according to arguments.

Sinatra is Kind of Cool - Passenger Tips

Sinatra is another very cool Ruby web framework that packs quite a punch in what is currently only 1,576 LOC. It has pretty decent routing, filters, layouts, and more. If you are building something that functions primarily as a web service it would definitely be an option worth checking out.

Sinatra is Rack compatible and thus deploying behind Phusion Passenger works well. Unfortunately, the documentation on the Passenger site leaves a bit to be desired and may leave you scratching your head as to why your Sinatra application isn't working correctly. Here is a sample config.ru script that will bring up your application behind passenger with an explanation coming after the code:



First off, my config.ru provides the ability to "freeze" to a certain version of Sinatra with the $:.unshift line near the top of the file. This line just checks for the existence of a vendor/sinatra directory in your application and if present pushes its lib directory to the front of the load path. The setup of the default options is where I add some information not given by the Passenger instructions which will otherwise cause your application to fall down go boom - that is you must tell Sinatra how to find its views directory (so that you can use views) and its app_file (so that it will reload correctly when running under :development mode). Also notice how the environment is picked up from the RackEnv variable coming from the conf file. The real guts of the application are defined in app.rb which is required at the bottom of the file.

This is my first post using gist to host code snippets and it seems to be a winner. I think I will also start hosting my Javascript example code straight out of gist raw as well.

Ruby Page Monitor

Here is a script I use to monitor pages for changes that don't supply feeds. It features the ability to check only a subset of the page using regular expression filters. You feed it a YAML file of the pages you want to monitor and optionally the email address you wish to send the results to (if you are running it as a cron job). The links file might originally look something like this:



I say "originally" because the script will add some information to this file to keep track of whether or not the page has changed on subsequent invocations. Here is the script:



The script tries to use last modified and etag headers if the server supports them, but will fall back to creating an md5 hash of the page if necessary. To scrape just a portion of the page, pass in "start" and "end" regular expressions in the YAML to instruct the script where to start and stop scraping.

Generating Combinations in Ruby and Javascript

Hey, time to dust off your discrete math text! (The one I used is Discrete Mathematics and Its Applications by Kenneth H. Rosen). The number of combinations of r items that can be selected from n items is given by the formula = n!/r!(n - r)! (note to self, render this with MathML). The following three code blocks show how to return those combinations in Ruby, Javascript, and Javascript with Prototype. Note that these functions yield (in Ruby) or do a callback with (Javascript) each combination. These could easily be modified to return an array of the combinations. After the three code snippets is a live example of the Javascript code. First the Ruby: Now the Javascript: And finally Javascript with Prototype (wow, now like Ruby): Wow, if that doesn't make you want to learn Prototype using Prototype and script.aculo.us: You Never Knew JavaScript Could Do This! then nothing will.

items: choose:

[ combinations ]

Converting from REXML to LibXML-Ruby

With the recent resurrection of LibXML-Ruby I decided to investigate converting one of our more XML processing heavy applications from REXML to LibXML-Ruby. LibXML-Ruby is touted to be much faster than REXML, and, I found this to be the case. In the process I kept track of some of the differences between the two that should help you if you decide to do the same. Here are some of the command equivalents between the two:

	REXML	LibXML-Ruby
create doc from filename	REXML::Document.new(filename)	XML::Document.file(filename)
create doc from file	REXML::Document.new(file)	XML::Parser.io(file).parse
grab to root of a doc	doc.root	doc.root
create doc from string	REXML::Document.new(string)	parser = XML::Parser.new parser.string = string doc = parser.parse (no, you can pass a StringIO to XML::Parser.io)
return all elements (not text nodes)	node.elements	node.find('*')
xpath from element	elem[xpath](annoyingly can return a single item or an array)	elem.find(xpath)
xpath to the first match	REXML::XPath.first(elem, xpath)	elem.find_first(xpath)
grab text content of node	elem.text	elem.content
working with attributes	elem.attributes[attr] (elem[...] reserved for xpath)	elem[attr]
creating nodes	REXML::Element.new(name) REXML::Element.new(name, attr_hash)	XML::Node.new(name) XML::Node.new(name, content) (can't set attrs on create)
deep clone a node	elem.deep_copy	elem.copy(true)
add a child element	node.elements.add(child) (child can be node or string) node.elements.add_element(name, attr_hash)	node << child_node node.child = child.node node.child_add(child)
removing elements	parent.remove(child) parent.delete_element(child) (child may be Element, String, or Integer)	child.remove!
jump to the next sibling	elem.next_element	elem.next
can XPath node not in a document?	yes	no
can add node directly from one document to another	yes	no

Ok, for those of you that actually read all the way through the table the bonus is right down here, because the biggest difference between REXML and LibXML-Ruby is in the handling of default namespaces. A default namespace is a namespace placed on an XML document that acts as the default, that is it doesn't use a prefix. A good example of this is KML documents which are often defined like this:



With REXML, you can use XPath expressions with the assumption that you are referencing the default namespace and they will just work - no prefix necessary. With LibXML-Ruby, this is not the case. Say you have a reference to a node with LibXML-Ruby, and you want to run some XPath on it, with LibXML-Ruby you will be forced to do something like this:



I found an approach of registering a prefix for the default namespace on Bogle's Blog. While this is nice, you still can't register this once for the whole document, but must do it on each node you will be running an XPath expression on.

(On another note, did I just remove all carriage returns from my table to make blogger happy? Why yes, yes I did.)

Ruby Singleton Methods Handy for Quick and Dirty Debugging

In a previous post I talked about needing to dump ActiveRecord models in favor of record sets when producing large reports from a Rails application. While this works well, the record set returned when doing a select_all is an array of hashes. And, being a hash, if you misspell the column name and type, for example:



nil will be returned and you will be none the wiser. Rubies singleton methods can quickly be used to detect such mistakes. Merely override the [] method with a method that will raise an error if the key can't be found:



By appending the singleton method override for [] you can quickly find out if you have made a typo somewhere in your view when specifying the column names of the record set.